From patchwork Fri May 17 04:21:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andriy Gelman X-Patchwork-Id: 13180 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 0E88D44876F for ; Fri, 17 May 2019 07:21:44 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E2F3D68832D; Fri, 17 May 2019 07:21:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1CB7F6807BC for ; Fri, 17 May 2019 07:21:37 +0300 (EEST) Received: by mail-qt1-f195.google.com with SMTP id a17so6614429qth.3 for ; Thu, 16 May 2019 21:21:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=gsmcncOjvkeUilzszlz+dbfhQo4/sK+/51JHfxH6Uf8=; b=k39hfpZJwIAbarOE4wRGRAR3UB6Af//O/pa5mLXkkyA7l+FSNLtOrDRxU96CvSShaL C+3oy7yRT6Pff/H88skDXTP13hnieA6c4xVxioxgUTHpRI/Y2+MPwISXVW7n9SFj1gF+ 380RWNFpAK8EcdnjxD78eAT0/0s2JbcGUOe5/NmBZQqdeP3w4JRBU5c+cvszbJLntKcS nbUeaMquJF5UT1j7C+5jbkhKyqxAv7hItCswnSDcBgnFIq0qYaAMUSIrHQV7lG6Hv/uj rHkvkm4exfZnT5szGMaxY4sWbUAsjgdhF+Iri3ml3dJ2ri22lU7v2B6a4648NB0vtQfj DxXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=gsmcncOjvkeUilzszlz+dbfhQo4/sK+/51JHfxH6Uf8=; b=UCL+93jwLgxiZ15JNQQejFV297b/NEwbGpoyGqBGsZkb3qlIRR3lqCmTVhQWF1M1Op ToQcjxOUAhm/pwdfAyMhujLanhg26f7NPssARgfvBdGzr2xukOceeBKgVt0b8pRRycmx S7OD3QE9xzxv6rrg/jKZwHJOnWFH9aNR2VZcWt0L6k12LKd/OkwAIMrvqG0hN+qIus3X k0dZjk0lbo0+MNO31RowAsI8H8UVU0/igSgiSAemy3x0Yxjrvsh24IhR548QYFBqTzFZ +rq3V+7O8ZDrvrELuMOb7MIvOs7Y0yOfbMBaIfdNXoUYkqDXtZy2sr976QUinN+MehLP tJeA== X-Gm-Message-State: APjAAAVwTzotTwWzhok5IzB3TafTUl9AQSgt+QjYt4Dg1AL7JgodS7zz LPr2DBjO40b2tP6ODFFuFpzzUELp X-Google-Smtp-Source: APXvYqwwmpzVgJlrSYIWnf4l2DuR8FAYvGrQVsW3E514TOJUeCkApKdwmib4pOoVC9XC3rCKuZH69g== X-Received: by 2002:a0c:c60d:: with SMTP id v13mr43370483qvi.22.1558066895292; Thu, 16 May 2019 21:21:35 -0700 (PDT) Received: from manj (c-24-60-253-205.hsd1.ma.comcast.net. [24.60.253.205]) by smtp.gmail.com with ESMTPSA id h28sm131845qkh.80.2019.05.16.21.21.34 for (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Thu, 16 May 2019 21:21:34 -0700 (PDT) Date: Fri, 17 May 2019 00:21:51 -0400 From: Andriy Gelman To: FFmpeg development discussions and patches Message-ID: <20190517042151.lhjrqi6lm2yayx43@manj> References: <20190516071222.30876-1-junli1026@gmail.com> <20190516071222.30876-2-junli1026@gmail.com> <20190516194850.urdbtayktpnfcrgh@manj> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716-1344-11488f-dirty Subject: Re: [FFmpeg-devel] [PATCH v3 2/2] fftools/ffmpeg: add support for per frame rotation and flip X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On Thu, 16. May 18:28, Jun Li wrote: > On Thu, May 16, 2019 at 12:54 PM Andriy Gelman > wrote: > > > Hi Jun, > > > > On Thu, 16. May 00:12, Jun Li wrote: > > > Fix #6945 > > > Current implementaion for autorotate works fine for stream > > > level rotataion but no support for frame level operation > > > and frame flip. This patch is for adding flip support and > > > per frame operations. > > > --- > > > fftools/cmdutils.c | 9 ++--- > > > fftools/cmdutils.h | 2 +- > > > fftools/ffmpeg.c | 21 ++++++++++- > > > fftools/ffmpeg.h | 2 + > > > fftools/ffmpeg_filter.c | 81 ++++++++++++++++++++++++++++++++++++++--- > > > fftools/ffplay.c | 28 +++++++++++--- > > > 6 files changed, 126 insertions(+), 17 deletions(-) > > > > > > diff --git a/fftools/cmdutils.c b/fftools/cmdutils.c > > > index 9cfbc45c2b..b8bb904419 100644 > > > --- a/fftools/cmdutils.c > > > +++ b/fftools/cmdutils.c > > > @@ -2172,13 +2172,12 @@ void *grow_array(void *array, int elem_size, int > > *size, int new_size) > > > return array; > > > } > > > > > > -double get_rotation(AVStream *st) > > > +double get_rotation_hflip(int32_t display_matrix[9], int* hflip) > > > { > > > - uint8_t* displaymatrix = av_stream_get_side_data(st, > > > - > > AV_PKT_DATA_DISPLAYMATRIX, NULL); > > > double theta = 0; > > > - if (displaymatrix) > > > - theta = -av_display_rotation_get((int32_t*) displaymatrix); > > > + > > > + if (display_matrix) > > > + theta = -av_display_rotation_hflip_get(display_matrix, hflip); > > > > > > theta -= 360*floor(theta/360 + 0.9/360); > > > > > > diff --git a/fftools/cmdutils.h b/fftools/cmdutils.h > > > index 6e2e0a2acb..dce44ed6e1 100644 > > > --- a/fftools/cmdutils.h > > > +++ b/fftools/cmdutils.h > > > @@ -643,6 +643,6 @@ void *grow_array(void *array, int elem_size, int > > *size, int new_size); > > > char name[128];\ > > > av_get_channel_layout_string(name, sizeof(name), 0, ch_layout); > > > > > > -double get_rotation(AVStream *st); > > > +double get_rotation_hflip(int32_t display_matrix[9], int* hflip); > > > > > > #endif /* FFTOOLS_CMDUTILS_H */ > > > diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c > > > index 01f04103cf..9ea1aaa930 100644 > > > --- a/fftools/ffmpeg.c > > > +++ b/fftools/ffmpeg.c > > > @@ -2126,6 +2126,24 @@ static int > > ifilter_has_all_input_formats(FilterGraph *fg) > > > return 1; > > > } > > > > > > +static int ifilter_display_matrix_need_from_frame(InputFilter *ifilter, > > AVFrame* frame) > > > +{ > > > + int32_t* stream_matrix = > > (int32_t*)av_stream_get_side_data(ifilter->ist->st, > > > + > > AV_PKT_DATA_DISPLAYMATRIX, NULL); > > > + // if we already have display matrix from stream, use it instead of > > extracting > > > + // from frame. > > > + if (stream_matrix) { > > > + memcpy(ifilter->display_matrix, stream_matrix, 4 * 9); > > > + return 0; > > > + } > > > + > > > + // cleanup the display matrix, may be from last frame > > > + memset(ifilter->display_matrix, 0, 4 * 9); > > > + av_display_rotation_set(ifilter->display_matrix, 0); > > > + > > > + return !!av_dict_get(frame->metadata, "Orientation", NULL, 0); > > > +} > > > + > > > static int ifilter_send_frame(InputFilter *ifilter, AVFrame *frame) > > > { > > > FilterGraph *fg = ifilter->graph; > > > @@ -2141,7 +2159,8 @@ static int ifilter_send_frame(InputFilter > > *ifilter, AVFrame *frame) > > > ifilter->channel_layout != frame->channel_layout; > > > break; > > > case AVMEDIA_TYPE_VIDEO: > > > - need_reinit |= ifilter->width != frame->width || > > > + need_reinit |= ifilter_display_matrix_need_from_frame(ifilter, > > frame) || > > > + ifilter->width != frame->width || > > > ifilter->height != frame->height; > > > break; > > > } > > > diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h > > > index eb1eaf6363..8331720663 100644 > > > --- a/fftools/ffmpeg.h > > > +++ b/fftools/ffmpeg.h > > > @@ -251,6 +251,8 @@ typedef struct InputFilter { > > > int channels; > > > uint64_t channel_layout; > > > > > > + int32_t display_matrix[9]; > > > + > > > AVBufferRef *hw_frames_ctx; > > > > > > int eof; > > > diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c > > > index 72838de1e2..1894f30561 100644 > > > --- a/fftools/ffmpeg_filter.c > > > +++ b/fftools/ffmpeg_filter.c > > > @@ -328,6 +328,7 @@ static void init_input_filter(FilterGraph *fg, > > AVFilterInOut *in) > > > fg->inputs[fg->nb_inputs - 1]->format = -1; > > > fg->inputs[fg->nb_inputs - 1]->type = ist->st->codecpar->codec_type; > > > fg->inputs[fg->nb_inputs - 1]->name = describe_filter_link(fg, in, > > 1); > > > + av_display_rotation_set(fg->inputs[fg->nb_inputs - > > 1]->display_matrix, 0); > > > > > > fg->inputs[fg->nb_inputs - 1]->frame_queue = av_fifo_alloc(8 * > > sizeof(AVFrame*)); > > > if (!fg->inputs[fg->nb_inputs - 1]->frame_queue) > > > @@ -807,22 +808,43 @@ static int > > configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, > > > last_filter = ifilter->filter; > > > > > > if (ist->autorotate) { > > > - double theta = get_rotation(ist->st); > > > + int hflip = 0; > > > + double theta = get_rotation_hflip(ifilter->display_matrix, > > &hflip); > > > > > > - if (fabs(theta - 90) < 1.0) { > > > + if (fabs(theta) < 1.0) { > > > + if (hflip) > > > + ret = insert_filter(&last_filter, &pad_idx, "hflip", > > NULL); > > > + } else if (fabs(theta - 90) < 1.0) { > > > ret = insert_filter(&last_filter, &pad_idx, "transpose", > > "clock"); > > > - } else if (fabs(theta - 180) < 1.0) { > > > - ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); > > > if (ret < 0) > > > return ret; > > > - ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL); > > > + if (hflip) > > > + ret = insert_filter(&last_filter, &pad_idx, "hflip", > > NULL); > > > + } else if (fabs(theta - 180) < 1.0) { > > > + if (hflip) { // rotate 180 and hflip equals vflip > > > + ret = insert_filter(&last_filter, &pad_idx, "vflip", > > NULL); > > > + } else { > > > + ret = insert_filter(&last_filter, &pad_idx, "hflip", > > NULL); > > > + if (ret < 0) > > > + return ret; > > > + ret = insert_filter(&last_filter, &pad_idx, "vflip", > > NULL); > > > + } > > > } else if (fabs(theta - 270) < 1.0) { > > > ret = insert_filter(&last_filter, &pad_idx, "transpose", > > "cclock"); > > > + if (ret < 0) > > > + return ret; > > > + if (hflip) > > > + ret = insert_filter(&last_filter, &pad_idx, "hflip", > > NULL); > > > } else if (fabs(theta) > 1.0) { > > > char rotate_buf[64]; > > > snprintf(rotate_buf, sizeof(rotate_buf), "%f*PI/180", > > theta); > > > ret = insert_filter(&last_filter, &pad_idx, "rotate", > > rotate_buf); > > > + if (ret < 0) > > > + return ret; > > > + if (hflip) > > > + ret = insert_filter(&last_filter, &pad_idx, "hflip", > > NULL); > > > } > > > > Just a thought, if you model your transformation matrix as M = RH, where H > > is > > the horizontal flip and R is the rotation, is it possible to simplify the > > logic > > here to: > > > > //your H transformation > > if (hflip) > > ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); > > > > //the R transformation > > same code as before.. > > > > Sorry Andriy, but I donot quite get it. Do you mean put this code > //your H transformation > if (hflip) > ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); > outside of the if check ? Sorry I wasn't clear. Before your patch, this part of the code was just applying a rotation. With your patch, you are proposing a horizontal mirror operation and a rotation. These are two separate operations. So, I do not see a reason to add extra logic to the rotation part of the code. I attach a diff applied to your patch set V2 to show what I mean. Note that I had to change a couple of things on how you define transformation for case 5 and 7 (since mirror horizontal is second argument). I tested on the examples from ticket #6945 and this worked. P.s. the diff is applied to your patch set V2 and not V3. I had errors applying V3 to master branch, but didn't have time to look into it. Andriy diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index 1894f30561..0b7f6a6089 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -811,40 +811,25 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, int hflip = 0; double theta = get_rotation_hflip(ifilter->display_matrix, &hflip); - if (fabs(theta) < 1.0) { - if (hflip) - ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); - } else if (fabs(theta - 90) < 1.0) { + //mirror part + if (hflip) + ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); + + //rotation part + if (fabs(theta - 90) < 1.0) { ret = insert_filter(&last_filter, &pad_idx, "transpose", "clock"); + } else if (fabs(theta - 180) < 1.0) { + ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); if (ret < 0) return ret; - if (hflip) - ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); - } else if (fabs(theta - 180) < 1.0) { - if (hflip) { // rotate 180 and hflip equals vflip - ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL); - } else { - ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); - if (ret < 0) - return ret; - ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL); - } + ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL); } else if (fabs(theta - 270) < 1.0) { ret = insert_filter(&last_filter, &pad_idx, "transpose", "cclock"); - if (ret < 0) - return ret; - if (hflip) - ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); } else if (fabs(theta) > 1.0) { char rotate_buf[64]; snprintf(rotate_buf, sizeof(rotate_buf), "%f*PI/180", theta); ret = insert_filter(&last_filter, &pad_idx, "rotate", rotate_buf); - if (ret < 0) - return ret; - if (hflip) - ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); } - if (ret < 0) return ret; } @@ -1232,14 +1217,14 @@ static void set_display_matrix_from_frame(const AVFrame *frame, int32_t m[9]) break; case 5: // mirror horizontal and rotate 270 CW av_display_rotation_set(m, 270.0); - av_display_matrix_flip(m, 0, 1); + av_display_matrix_flip(m, 1, 0); //mirror horizontal is the first arg? break; case 6: // rotate 90 CW av_display_rotation_set(m, 90.0); break; case 7: // mirror horizontal and rotate 90 CW av_display_rotation_set(m, 90.0); - av_display_matrix_flip(m, 0, 1); + av_display_matrix_flip(m, 1, 0); break; case 8: // rotate 270 CW av_display_rotation_set(m, 270.0);