From patchwork Wed May 15 05:36:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jun Li X-Patchwork-Id: 13119 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 15F974489DE for ; Wed, 15 May 2019 08:37:04 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id F3CB968978D; Wed, 15 May 2019 08:37:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6DF3A680D4C for ; Wed, 15 May 2019 08:36:56 +0300 (EEST) Received: by mail-pg1-f176.google.com with SMTP id c13so772359pgt.1 for ; Tue, 14 May 2019 22:36:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=cXW/2qZB501mQQzXE3l6gpFrvVWPS6V6RcbF76GQC7w=; b=davS5jEEDu2B7NnsnIGsynkaHUWLsqGsvUWHKY8Gc5eY7VEtFs79K6hLsP/+rZEXjY HN7zViNHknazd73RxhdQIPL1Ud1T4VY8A5pV4x5cS3s6f+UqfXkFooGgvupwV76pDyLk 9KvxlQ0vTkq2/yS5pIj3h6UsbQ+YdWpaSPyltxe9KwNDHx+nqAFFpZXqFE3THCrMzJ8I q0CHIK6qpcG2rCg1Ar35fScfl8vZPsQKn7x7L6PJSIEFaVBngttdxj+IJ8XUsmaYLB2X Zi9q02BOSf2jzaQH4OfXmjNSpnyLlukRWu2X7T0AQ8HOPpT7dObipSwElS6ORwUijnvM NyZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=cXW/2qZB501mQQzXE3l6gpFrvVWPS6V6RcbF76GQC7w=; b=Vq5CkJBv3JF7Nn2NxS2orXfT/XJxaxZ8RTsBWWrxVmYVH/nYG6OBh7MSLWkcaaYvtX 6BDNNwziQVvBkXSTOmGqW5z3FaKmo8RfuXhdIgWhgDjyE/LgUhW6YDFjQ6Fk574bq9bV 24K1UTpRENudhB/uufAZ7/Sh+QITtOjFP18EbCs5ki1tHSvTyD8nBybzE9PzWkgJ3zJa jwD8UMHkYliSCi7Udc/YQS+zkTdIUw6MIu/84/AqNDdqZp3qlpP+JR5vQFC4E3vCJRYY +zKG8BNgECGryqX2oe9k6F3MT+J5HbsbWEe4GSgEU//hfReu71yTej7RU7THvjvTbwyc tm3w== X-Gm-Message-State: APjAAAXouosyFW1w8NMb/7IYIkSDQMpASADuP7kHHRh6fhSpCPrIVDp/ XbaXaFDUsR4TXnxaAZ9I922ktUE+fmA= X-Google-Smtp-Source: APXvYqwEjVqEoqEiHdh43WfW0KE/IqwAp42aOYME7C6ZpHZQ6cSuuBibs+KxnnvUlmTarDI+zRzUtQ== X-Received: by 2002:a63:e451:: with SMTP id i17mr42410369pgk.312.1557898614457; Tue, 14 May 2019 22:36:54 -0700 (PDT) Received: from super.lan (c-73-193-9-32.hsd1.wa.comcast.net. [73.193.9.32]) by smtp.gmail.com with ESMTPSA id j12sm1009282pff.148.2019.05.14.22.36.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 22:36:53 -0700 (PDT) From: Jun Li To: ffmpeg-devel@ffmpeg.org Date: Tue, 14 May 2019 22:36:45 -0700 Message-Id: <20190515053645.21432-2-junli1026@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190515053645.21432-1-junli1026@gmail.com> References: <20190515053645.21432-1-junli1026@gmail.com> Subject: [FFmpeg-devel] [PATCH v2 2/2] fftools/ffmpeg: add support for per frame rotation and flip X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fix #6945 Current implementaion for autorotate works fine for stream level rotataion but no support for frame level operation and frame flip. This patch is for adding flip support and per frame operations. --- fftools/cmdutils.c | 9 ++--- fftools/cmdutils.h | 2 +- fftools/ffmpeg.c | 21 ++++++++++- fftools/ffmpeg.h | 2 + fftools/ffmpeg_filter.c | 81 ++++++++++++++++++++++++++++++++++++++--- fftools/ffplay.c | 28 +++++++++++--- 6 files changed, 126 insertions(+), 17 deletions(-) diff --git a/fftools/cmdutils.c b/fftools/cmdutils.c index 9cfbc45c2b..b8bb904419 100644 --- a/fftools/cmdutils.c +++ b/fftools/cmdutils.c @@ -2172,13 +2172,12 @@ void *grow_array(void *array, int elem_size, int *size, int new_size) return array; } -double get_rotation(AVStream *st) +double get_rotation_hflip(int32_t display_matrix[9], int* hflip) { - uint8_t* displaymatrix = av_stream_get_side_data(st, - AV_PKT_DATA_DISPLAYMATRIX, NULL); double theta = 0; - if (displaymatrix) - theta = -av_display_rotation_get((int32_t*) displaymatrix); + + if (display_matrix) + theta = -av_display_rotation_hflip_get(display_matrix, hflip); theta -= 360*floor(theta/360 + 0.9/360); diff --git a/fftools/cmdutils.h b/fftools/cmdutils.h index 6e2e0a2acb..dce44ed6e1 100644 --- a/fftools/cmdutils.h +++ b/fftools/cmdutils.h @@ -643,6 +643,6 @@ void *grow_array(void *array, int elem_size, int *size, int new_size); char name[128];\ av_get_channel_layout_string(name, sizeof(name), 0, ch_layout); -double get_rotation(AVStream *st); +double get_rotation_hflip(int32_t display_matrix[9], int* hflip); #endif /* FFTOOLS_CMDUTILS_H */ diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 01f04103cf..9ea1aaa930 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -2126,6 +2126,24 @@ static int ifilter_has_all_input_formats(FilterGraph *fg) return 1; } +static int ifilter_display_matrix_need_from_frame(InputFilter *ifilter, AVFrame* frame) +{ + int32_t* stream_matrix = (int32_t*)av_stream_get_side_data(ifilter->ist->st, + AV_PKT_DATA_DISPLAYMATRIX, NULL); + // if we already have display matrix from stream, use it instead of extracting + // from frame. + if (stream_matrix) { + memcpy(ifilter->display_matrix, stream_matrix, 4 * 9); + return 0; + } + + // cleanup the display matrix, may be from last frame + memset(ifilter->display_matrix, 0, 4 * 9); + av_display_rotation_set(ifilter->display_matrix, 0); + + return !!av_dict_get(frame->metadata, "Orientation", NULL, 0); +} + static int ifilter_send_frame(InputFilter *ifilter, AVFrame *frame) { FilterGraph *fg = ifilter->graph; @@ -2141,7 +2159,8 @@ static int ifilter_send_frame(InputFilter *ifilter, AVFrame *frame) ifilter->channel_layout != frame->channel_layout; break; case AVMEDIA_TYPE_VIDEO: - need_reinit |= ifilter->width != frame->width || + need_reinit |= ifilter_display_matrix_need_from_frame(ifilter, frame) || + ifilter->width != frame->width || ifilter->height != frame->height; break; } diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index eb1eaf6363..8331720663 100644 --- a/fftools/ffmpeg.h +++ b/fftools/ffmpeg.h @@ -251,6 +251,8 @@ typedef struct InputFilter { int channels; uint64_t channel_layout; + int32_t display_matrix[9]; + AVBufferRef *hw_frames_ctx; int eof; diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index 72838de1e2..1894f30561 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -328,6 +328,7 @@ static void init_input_filter(FilterGraph *fg, AVFilterInOut *in) fg->inputs[fg->nb_inputs - 1]->format = -1; fg->inputs[fg->nb_inputs - 1]->type = ist->st->codecpar->codec_type; fg->inputs[fg->nb_inputs - 1]->name = describe_filter_link(fg, in, 1); + av_display_rotation_set(fg->inputs[fg->nb_inputs - 1]->display_matrix, 0); fg->inputs[fg->nb_inputs - 1]->frame_queue = av_fifo_alloc(8 * sizeof(AVFrame*)); if (!fg->inputs[fg->nb_inputs - 1]->frame_queue) @@ -807,22 +808,43 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, last_filter = ifilter->filter; if (ist->autorotate) { - double theta = get_rotation(ist->st); + int hflip = 0; + double theta = get_rotation_hflip(ifilter->display_matrix, &hflip); - if (fabs(theta - 90) < 1.0) { + if (fabs(theta) < 1.0) { + if (hflip) + ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); + } else if (fabs(theta - 90) < 1.0) { ret = insert_filter(&last_filter, &pad_idx, "transpose", "clock"); - } else if (fabs(theta - 180) < 1.0) { - ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); if (ret < 0) return ret; - ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL); + if (hflip) + ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); + } else if (fabs(theta - 180) < 1.0) { + if (hflip) { // rotate 180 and hflip equals vflip + ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL); + } else { + ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); + if (ret < 0) + return ret; + ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL); + } } else if (fabs(theta - 270) < 1.0) { ret = insert_filter(&last_filter, &pad_idx, "transpose", "cclock"); + if (ret < 0) + return ret; + if (hflip) + ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); } else if (fabs(theta) > 1.0) { char rotate_buf[64]; snprintf(rotate_buf, sizeof(rotate_buf), "%f*PI/180", theta); ret = insert_filter(&last_filter, &pad_idx, "rotate", rotate_buf); + if (ret < 0) + return ret; + if (hflip) + ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL); } + if (ret < 0) return ret; } @@ -1182,6 +1204,53 @@ fail: return ret; } +static void set_display_matrix_from_frame(const AVFrame *frame, int32_t m[9]) +{ + AVDictionaryEntry *entry = NULL; + int orientation = 0; + + // read exif orientation data + entry = av_dict_get(frame->metadata, "Orientation", NULL, 0); + if (entry) { + orientation = atoi(entry->value); + memset(m, 0, 4 * 9); + switch (orientation) + { + case 1: // horizontal (normal) + av_display_rotation_set(m, 0.0); + break; + case 2: // mirror horizontal + av_display_rotation_set(m, 0.0); + av_display_matrix_flip(m, 1, 0); + break; + case 3: // rotate 180 + av_display_rotation_set(m, 180.0); + break; + case 4: // mirror vertical + av_display_rotation_set(m, 0.0); + av_display_matrix_flip(m, 0, 1); + break; + case 5: // mirror horizontal and rotate 270 CW + av_display_rotation_set(m, 270.0); + av_display_matrix_flip(m, 0, 1); + break; + case 6: // rotate 90 CW + av_display_rotation_set(m, 90.0); + break; + case 7: // mirror horizontal and rotate 90 CW + av_display_rotation_set(m, 90.0); + av_display_matrix_flip(m, 0, 1); + break; + case 8: // rotate 270 CW + av_display_rotation_set(m, 270.0); + break; + default: + av_display_rotation_set(m, 0.0); + break; + } + } +} + int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame) { av_buffer_unref(&ifilter->hw_frames_ctx); @@ -1202,6 +1271,8 @@ int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame) return AVERROR(ENOMEM); } + set_display_matrix_from_frame(frame, ifilter->display_matrix); + return 0; } diff --git a/fftools/ffplay.c b/fftools/ffplay.c index 8f050e16e6..717a9a830c 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -1914,19 +1914,37 @@ static int configure_video_filters(AVFilterGraph *graph, VideoState *is, const c } while (0) if (autorotate) { - double theta = get_rotation(is->video_st); - - if (fabs(theta - 90) < 1.0) { + int hflip = 0; + double theta = 0.0; + int32_t* display_matrix = (int32_t*)av_stream_get_side_data(is->video_st, + AV_PKT_DATA_DISPLAYMATRIX, NULL); + if (display_matrix) + theta = get_rotation_hflip(display_matrix, &hflip); + + if (fabs(theta) < 1.0) { + if (hflip) + INSERT_FILT("hflip", NULL); + } else if (fabs(theta - 90) < 1.0) { INSERT_FILT("transpose", "clock"); + if (hflip) + INSERT_FILT("hflip", NULL); } else if (fabs(theta - 180) < 1.0) { - INSERT_FILT("hflip", NULL); - INSERT_FILT("vflip", NULL); + if (hflip) { // rotate 180 and hflip equals vflip + INSERT_FILT("vflip", NULL); + } else { + INSERT_FILT("hflip", NULL); + INSERT_FILT("vflip", NULL); + } } else if (fabs(theta - 270) < 1.0) { INSERT_FILT("transpose", "cclock"); + if (hflip) + INSERT_FILT("hflip", NULL); } else if (fabs(theta) > 1.0) { char rotate_buf[64]; snprintf(rotate_buf, sizeof(rotate_buf), "%f*PI/180", theta); INSERT_FILT("rotate", rotate_buf); + if (hflip) + INSERT_FILT("hflip", NULL); } }