From patchwork Fri Feb 14 03:32:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 17781 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 85BBC44A76F for ; Fri, 14 Feb 2020 05:41:21 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5D41668A1C3; Fri, 14 Feb 2020 05:41:21 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 89CAC688390 for ; Fri, 14 Feb 2020 05:41:14 +0200 (EET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Feb 2020 19:41:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,439,1574150400"; d="scan'208";a="406867908" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.13.25]) by orsmga005.jf.intel.com with ESMTP; 13 Feb 2020 19:41:11 -0800 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Fri, 14 Feb 2020 11:32:05 +0800 Message-Id: <1581651125-25232-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 1/3] avutil/frame: add use_last_roi X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" For some cases, the regions of interest do not change, it is not convenient to always prepare the roi data for every frame. So, add use_last_roi to show it uses the same roi data as last frame. Since a new flag is added into AVFrame, the major version number of lavu is changed. Signed-off-by: Guo, Yejun --- doc/APIchanges | 3 +++ libavutil/frame.h | 8 ++++++++ libavutil/version.h | 20 ++++++++++---------- 3 files changed, 21 insertions(+), 10 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index 761f37f..eec0f2f 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -15,6 +15,9 @@ libavutil: 2017-10-21 API changes, most recent first: +2020-02-14 - xxxxxxxxxx - lavu 57.0.100 - frame.h + Add use_last_roi in AVFrame. + 2020-02-13 - xxxxxxxxxx - lavu 56.41.100 - tx.h Add AV_TX_INT32_FFT and AV_TX_INT32_MDCT diff --git a/libavutil/frame.h b/libavutil/frame.h index b5afb58..2c786f4 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -634,6 +634,14 @@ typedef struct AVFrame { AVBufferRef *hw_frames_ctx; /** + * It is set if the RegionOfInterest info is the same as last frame, + * so, we don't need to prepare the same thing in side_data. + * If there is roi info in side_data and this variable is also set, + * use the info in side_data. + */ + int8_t use_last_roi; + + /** * AVBufferRef for free use by the API user. FFmpeg will never check the * contents of the buffer ref. FFmpeg calls av_buffer_unref() on it when * the frame is unreferenced. av_frame_copy_props() calls create a new diff --git a/libavutil/version.h b/libavutil/version.h index 90cc55b..aade19b 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -78,8 +78,8 @@ * @{ */ -#define LIBAVUTIL_VERSION_MAJOR 56 -#define LIBAVUTIL_VERSION_MINOR 41 +#define LIBAVUTIL_VERSION_MAJOR 57 +#define LIBAVUTIL_VERSION_MINOR 0 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ @@ -106,28 +106,28 @@ */ #ifndef FF_API_VAAPI -#define FF_API_VAAPI (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_VAAPI (LIBAVUTIL_VERSION_MAJOR < 58) #endif #ifndef FF_API_FRAME_QP -#define FF_API_FRAME_QP (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_FRAME_QP (LIBAVUTIL_VERSION_MAJOR < 58) #endif #ifndef FF_API_PLUS1_MINUS1 -#define FF_API_PLUS1_MINUS1 (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_PLUS1_MINUS1 (LIBAVUTIL_VERSION_MAJOR < 58) #endif #ifndef FF_API_ERROR_FRAME -#define FF_API_ERROR_FRAME (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_ERROR_FRAME (LIBAVUTIL_VERSION_MAJOR < 58) #endif #ifndef FF_API_PKT_PTS -#define FF_API_PKT_PTS (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_PKT_PTS (LIBAVUTIL_VERSION_MAJOR < 58) #endif #ifndef FF_API_CRYPTO_SIZE_T -#define FF_API_CRYPTO_SIZE_T (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_CRYPTO_SIZE_T (LIBAVUTIL_VERSION_MAJOR < 58) #endif #ifndef FF_API_FRAME_GET_SET -#define FF_API_FRAME_GET_SET (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_FRAME_GET_SET (LIBAVUTIL_VERSION_MAJOR < 58) #endif #ifndef FF_API_PSEUDOPAL -#define FF_API_PSEUDOPAL (LIBAVUTIL_VERSION_MAJOR < 57) +#define FF_API_PSEUDOPAL (LIBAVUTIL_VERSION_MAJOR < 58) #endif From patchwork Fri Feb 14 03:32:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 17782 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 8179F44A76F for ; Fri, 14 Feb 2020 05:41:27 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6D82F68AE56; Fri, 14 Feb 2020 05:41:27 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BD33E68A101 for ; Fri, 14 Feb 2020 05:41:20 +0200 (EET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Feb 2020 19:41:18 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,439,1574150400"; d="scan'208";a="434655968" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.13.25]) by fmsmga006.fm.intel.com with ESMTP; 13 Feb 2020 19:41:18 -0800 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Fri, 14 Feb 2020 11:32:12 +0800 Message-Id: <1581651132-25279-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 2/3] libavcodec/libx264.c: refine code to extract function x264_encode_set_roi X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Guo, Yejun --- libavcodec/libx264.c | 148 ++++++++++++++++++++++++++++----------------------- 1 file changed, 80 insertions(+), 68 deletions(-) diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c index ca8f6c0..bad9351 100644 --- a/libavcodec/libx264.c +++ b/libavcodec/libx264.c @@ -289,6 +289,83 @@ static void reconfig_encoder(AVCodecContext *ctx, const AVFrame *frame) } } +static int x264_encode_set_roi(const AVFrame *frame, AVCodecContext *ctx, int bit_depth) +{ + AVFrameSideData *sd = av_frame_get_side_data(frame, AV_FRAME_DATA_REGIONS_OF_INTEREST); + X264Context *x4 = ctx->priv_data; + int mbx = (frame->width + MB_SIZE - 1) / MB_SIZE; + int mby = (frame->height + MB_SIZE - 1) / MB_SIZE; + int qp_range = 51 + 6 * (bit_depth - 8); + int nb_rois; + const AVRegionOfInterest *roi; + uint32_t roi_size; + float *qoffsets; + + if (!sd) { + return 0; + } + + if (x4->params.rc.i_aq_mode == X264_AQ_NONE) { + if (!x4->roi_warned) { + x4->roi_warned = 1; + av_log(ctx, AV_LOG_WARNING, "Adaptive quantization must be enabled to use ROI encoding, skipping ROI.\n"); + } + return 0; + } + + if (frame->interlaced_frame != 0) { + if (!x4->roi_warned) { + x4->roi_warned = 1; + av_log(ctx, AV_LOG_WARNING, "interlaced_frame not supported for ROI encoding yet, skipping ROI.\n"); + } + return 0; + } + + roi = (const AVRegionOfInterest*)sd->data; + roi_size = roi->self_size; + if (!roi_size || sd->size % roi_size != 0) { + av_log(ctx, AV_LOG_ERROR, "Invalid AVRegionOfInterest.self_size.\n"); + return AVERROR(EINVAL); + } + nb_rois = sd->size / roi_size; + + qoffsets = av_mallocz_array(mbx * mby, sizeof(*qoffsets)); + if (!qoffsets) + return AVERROR(ENOMEM); + + // This list must be iterated in reverse because the first + // region in the list applies when regions overlap. + for (int i = nb_rois - 1; i >= 0; i--) { + int startx, endx, starty, endy; + float qoffset; + + roi = (const AVRegionOfInterest*)(sd->data + roi_size * i); + + starty = FFMIN(mby, roi->top / MB_SIZE); + endy = FFMIN(mby, (roi->bottom + MB_SIZE - 1)/ MB_SIZE); + startx = FFMIN(mbx, roi->left / MB_SIZE); + endx = FFMIN(mbx, (roi->right + MB_SIZE - 1)/ MB_SIZE); + + if (roi->qoffset.den == 0) { + av_free(qoffsets); + av_log(ctx, AV_LOG_ERROR, "AVRegionOfInterest.qoffset.den must not be zero.\n"); + return AVERROR(EINVAL); + } + qoffset = roi->qoffset.num * 1.0f / roi->qoffset.den; + qoffset = av_clipf(qoffset * qp_range, -qp_range, +qp_range); + + for (int y = starty; y < endy; y++) { + for (int x = startx; x < endx; x++) { + qoffsets[x + y*mbx] = qoffset; + } + } + } + + x4->pic.prop.quant_offsets = qoffsets; + x4->pic.prop.quant_offsets_free = av_free; + return 0; +} + static int X264_frame(AVCodecContext *ctx, AVPacket *pkt, const AVFrame *frame, int *got_packet) { @@ -300,7 +377,6 @@ static int X264_frame(AVCodecContext *ctx, AVPacket *pkt, const AVFrame *frame, int bit_depth; int64_t wallclock = 0; X264Opaque *out_opaque; - AVFrameSideData *sd; x264_picture_init( &x4->pic ); x4->pic.img.i_csp = x4->params.i_csp; @@ -367,73 +443,9 @@ static int X264_frame(AVCodecContext *ctx, AVPacket *pkt, const AVFrame *frame, } } - sd = av_frame_get_side_data(frame, AV_FRAME_DATA_REGIONS_OF_INTEREST); - if (sd) { - if (x4->params.rc.i_aq_mode == X264_AQ_NONE) { - if (!x4->roi_warned) { - x4->roi_warned = 1; - av_log(ctx, AV_LOG_WARNING, "Adaptive quantization must be enabled to use ROI encoding, skipping ROI.\n"); - } - } else { - if (frame->interlaced_frame == 0) { - int mbx = (frame->width + MB_SIZE - 1) / MB_SIZE; - int mby = (frame->height + MB_SIZE - 1) / MB_SIZE; - int qp_range = 51 + 6 * (bit_depth - 8); - int nb_rois; - const AVRegionOfInterest *roi; - uint32_t roi_size; - float *qoffsets; - - roi = (const AVRegionOfInterest*)sd->data; - roi_size = roi->self_size; - if (!roi_size || sd->size % roi_size != 0) { - av_log(ctx, AV_LOG_ERROR, "Invalid AVRegionOfInterest.self_size.\n"); - return AVERROR(EINVAL); - } - nb_rois = sd->size / roi_size; - - qoffsets = av_mallocz_array(mbx * mby, sizeof(*qoffsets)); - if (!qoffsets) - return AVERROR(ENOMEM); - - // This list must be iterated in reverse because the first - // region in the list applies when regions overlap. - for (int i = nb_rois - 1; i >= 0; i--) { - int startx, endx, starty, endy; - float qoffset; - - roi = (const AVRegionOfInterest*)(sd->data + roi_size * i); - - starty = FFMIN(mby, roi->top / MB_SIZE); - endy = FFMIN(mby, (roi->bottom + MB_SIZE - 1)/ MB_SIZE); - startx = FFMIN(mbx, roi->left / MB_SIZE); - endx = FFMIN(mbx, (roi->right + MB_SIZE - 1)/ MB_SIZE); - - if (roi->qoffset.den == 0) { - av_free(qoffsets); - av_log(ctx, AV_LOG_ERROR, "AVRegionOfInterest.qoffset.den must not be zero.\n"); - return AVERROR(EINVAL); - } - qoffset = roi->qoffset.num * 1.0f / roi->qoffset.den; - qoffset = av_clipf(qoffset * qp_range, -qp_range, +qp_range); - - for (int y = starty; y < endy; y++) { - for (int x = startx; x < endx; x++) { - qoffsets[x + y*mbx] = qoffset; - } - } - } - - x4->pic.prop.quant_offsets = qoffsets; - x4->pic.prop.quant_offsets_free = av_free; - } else { - if (!x4->roi_warned) { - x4->roi_warned = 1; - av_log(ctx, AV_LOG_WARNING, "interlaced_frame not supported for ROI encoding yet, skipping ROI.\n"); - } - } - } - } + ret = x264_encode_set_roi(frame, ctx, bit_depth); + if (ret < 0) + return ret; } do { From patchwork Fri Feb 14 03:32:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 17783 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 6D72044A76F for ; Fri, 14 Feb 2020 05:41:35 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5448468AE8A; Fri, 14 Feb 2020 05:41:35 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CA8F668A101 for ; Fri, 14 Feb 2020 05:41:28 +0200 (EET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Feb 2020 19:41:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,439,1574150400"; d="scan'208";a="432893841" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.13.25]) by fmsmga005.fm.intel.com with ESMTP; 13 Feb 2020 19:41:26 -0800 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Fri, 14 Feb 2020 11:32:20 +0800 Message-Id: <1581651140-25325-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 3/3] libavcodec/libx264.c: add support for use_last_roi X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Guo, Yejun --- libavcodec/libx264.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c index bad9351..b275cc2 100644 --- a/libavcodec/libx264.c +++ b/libavcodec/libx264.c @@ -107,6 +107,12 @@ typedef struct X264Context { X264Opaque *reordered_opaque; /** + * quant_offsets that will be passed to encoder + */ + float *qoffsets; + int qsize; //in byte + + /** * If the encoder does not support ROI then warn the first time we * encounter a frame with ROI side data. */ @@ -299,9 +305,18 @@ static int x264_encode_set_roi(const AVFrame *frame, AVCodecContext *ctx, int bi int nb_rois; const AVRegionOfInterest *roi; uint32_t roi_size; - float *qoffsets; + int qsize = mbx * mby * sizeof(*x4->qoffsets); if (!sd) { + if (frame->use_last_roi) { + if (!x4->qoffsets) { + if (!x4->roi_warned) { + x4->roi_warned = 1; + av_log(ctx, AV_LOG_WARNING, "No previous data for ROI encoding, skipping ROI.\n"); + } + } + x4->pic.prop.quant_offsets = x4->qoffsets; + } return 0; } @@ -329,10 +344,18 @@ static int x264_encode_set_roi(const AVFrame *frame, AVCodecContext *ctx, int bi } nb_rois = sd->size / roi_size; - qoffsets = av_mallocz_array(mbx * mby, sizeof(*qoffsets)); - if (!qoffsets) + // re-allocate x4->qoffsets if size changed + if (x4->qsize != qsize) + av_freep(&x4->qoffsets); + + if (!x4->qoffsets) + x4->qoffsets = av_malloc(qsize); + if (!x4->qoffsets) return AVERROR(ENOMEM); + x4->qsize = qsize; + memset(x4->qoffsets, 0, x4->qsize); + // This list must be iterated in reverse because the first // region in the list applies when regions overlap. for (int i = nb_rois - 1; i >= 0; i--) { @@ -347,7 +370,7 @@ static int x264_encode_set_roi(const AVFrame *frame, AVCodecContext *ctx, int bi endx = FFMIN(mbx, (roi->right + MB_SIZE - 1)/ MB_SIZE); if (roi->qoffset.den == 0) { - av_free(qoffsets); + av_freep(&x4->qoffsets); av_log(ctx, AV_LOG_ERROR, "AVRegionOfInterest.qoffset.den must not be zero.\n"); return AVERROR(EINVAL); } @@ -356,13 +379,12 @@ static int x264_encode_set_roi(const AVFrame *frame, AVCodecContext *ctx, int bi for (int y = starty; y < endy; y++) { for (int x = startx; x < endx; x++) { - qoffsets[x + y*mbx] = qoffset; + x4->qoffsets[x + y*mbx] = qoffset; } } } - x4->pic.prop.quant_offsets = qoffsets; - x4->pic.prop.quant_offsets_free = av_free; + x4->pic.prop.quant_offsets = x4->qoffsets; return 0; } @@ -515,6 +537,7 @@ static av_cold int X264_close(AVCodecContext *avctx) av_freep(&avctx->extradata); av_freep(&x4->sei); av_freep(&x4->reordered_opaque); + av_freep(&x4->qoffsets); if (x4->enc) { x264_encoder_close(x4->enc);