From patchwork Tue Feb 25 09:15:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 17927 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 9F13C447FCA for ; Tue, 25 Feb 2020 11:24:38 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 89E8B68B667; Tue, 25 Feb 2020 11:24:38 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C7D5E68B656 for ; Tue, 25 Feb 2020 11:24:36 +0200 (EET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Feb 2020 01:24:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,483,1574150400"; d="scan'208";a="410181597" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga005.jf.intel.com with ESMTP; 25 Feb 2020 01:24:33 -0800 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Tue, 25 Feb 2020 17:15:14 +0800 Message-Id: <1582622114-6479-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH V2 3/3] avfilter/vf_dnn_processing.c: add frame size change support for planar yuv format X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" The Y channel is handled by dnn, and also resized by dnn. The UV channels are resized with swscale. The command to use espcn.pb (see vf_sr) looks like: ./ffmpeg -i 480p.jpg -vf format=yuv420p,dnn_processing=dnn_backend=tensorflow:model=espcn.pb:input=x:output=y -y tmp.espcn.jpg Signed-off-by: Guo, Yejun --- doc/filters.texi | 9 +++++++++ libavfilter/vf_dnn_processing.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 39 insertions(+), 7 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 33b7857..e3df8f9 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -9155,6 +9155,7 @@ ffmpeg -i INPUT -f lavfi -i nullsrc=hd720,geq='r=128+80*(sin(sqrt((X-W/2)*(X-W/2 @end example @end itemize +@anchor{dnn_processing} @section dnn_processing Do image processing with deep neural networks. It works together with another filter @@ -9216,6 +9217,12 @@ Handle the Y channel with srcnn.pb (see @ref{sr} filter) for frame with yuv420p ./ffmpeg -i 480p.jpg -vf format=yuv420p,scale=w=iw*2:h=ih*2,dnn_processing=dnn_backend=tensorflow:model=srcnn.pb:input=x:output=y -y srcnn.jpg @end example +@item +Handle the Y channel with espcn.pb (see @ref{sr} filter), which changes frame size, for format yuv420p (planar YUV formats supported): +@example +./ffmpeg -i 480p.jpg -vf format=yuv420p,dnn_processing=dnn_backend=tensorflow:model=espcn.pb:input=x:output=y -y tmp.espcn.jpg +@end example + @end itemize @section drawbox @@ -17369,6 +17376,8 @@ Default value is @code{2}. Scale factor is necessary for SRCNN model, because it input upscaled using bicubic upscaling with proper scale factor. @end table +This feature can also be finished with @ref{dnn_processing} filter. + @section ssim Obtain the SSIM (Structural SImilarity Metric) between two input videos. diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c index f9458f0..7f40f85 100644 --- a/libavfilter/vf_dnn_processing.c +++ b/libavfilter/vf_dnn_processing.c @@ -51,6 +51,8 @@ typedef struct DnnProcessingContext { struct SwsContext *sws_gray8_to_grayf32; struct SwsContext *sws_grayf32_to_gray8; + struct SwsContext *sws_uv_scale; + int sws_uv_height; } DnnProcessingContext; #define OFFSET(x) offsetof(DnnProcessingContext, x) @@ -274,6 +276,18 @@ static int prepare_sws_context(AVFilterLink *outlink) outlink->h, AV_PIX_FMT_GRAY8, 0, NULL, NULL, NULL); + + if (inlink->w != outlink->w || inlink->h != outlink->h) { + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(fmt); + int sws_src_h = AV_CEIL_RSHIFT(inlink->h, desc->log2_chroma_h); + int sws_src_w = AV_CEIL_RSHIFT(inlink->w, desc->log2_chroma_w); + int sws_dst_h = AV_CEIL_RSHIFT(outlink->h, desc->log2_chroma_h); + int sws_dst_w = AV_CEIL_RSHIFT(outlink->w, desc->log2_chroma_w); + ctx->sws_uv_scale = sws_getContext(sws_src_w, sws_src_h, AV_PIX_FMT_GRAY8, + sws_dst_w, sws_dst_h, AV_PIX_FMT_GRAY8, + SWS_BICUBIC, NULL, NULL, NULL); + ctx->sws_uv_height = sws_src_h; + } return 0; default: //do nothing @@ -404,13 +418,21 @@ static av_always_inline int isPlanarYUV(enum AVPixelFormat pix_fmt) static int copy_uv_planes(DnnProcessingContext *ctx, AVFrame *out, const AVFrame *in) { - const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(in->format); - int uv_height = AV_CEIL_RSHIFT(in->height, desc->log2_chroma_h); - for (int i = 1; i < 3; ++i) { - int bytewidth = av_image_get_linesize(in->format, in->width, i); - av_image_copy_plane(out->data[i], out->linesize[i], - in->data[i], in->linesize[i], - bytewidth, uv_height); + if (!ctx->sws_uv_scale) { + av_assert0(in->height == out->height && in->width == out->width); + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(in->format); + int uv_height = AV_CEIL_RSHIFT(in->height, desc->log2_chroma_h); + for (int i = 1; i < 3; ++i) { + int bytewidth = av_image_get_linesize(in->format, in->width, i); + av_image_copy_plane(out->data[i], out->linesize[i], + in->data[i], in->linesize[i], + bytewidth, uv_height); + } + } else { + sws_scale(ctx->sws_uv_scale, (const uint8_t **)(in->data + 1), in->linesize + 1, + 0, ctx->sws_uv_height, out->data + 1, out->linesize + 1); + sws_scale(ctx->sws_uv_scale, (const uint8_t **)(in->data + 2), in->linesize + 2, + 0, ctx->sws_uv_height, out->data + 2, out->linesize + 2); } return 0; @@ -455,6 +477,7 @@ static av_cold void uninit(AVFilterContext *ctx) sws_freeContext(context->sws_gray8_to_grayf32); sws_freeContext(context->sws_grayf32_to_gray8); + sws_freeContext(context->sws_uv_scale); if (context->dnn_module) (context->dnn_module->free_model)(&context->model);