From patchwork Fri May 14 08:47:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 27771 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp276745iof; Fri, 14 May 2021 01:57:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx6Ed7OW9bBPGsw1bFW7H+QWWrplPKzwML94xEdT5p3DOWyOyZlqRbqhAXmatOHtGAn8KOw X-Received: by 2002:a17:907:7355:: with SMTP id dq21mr47784747ejc.157.1620982622041; Fri, 14 May 2021 01:57:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1620982622; cv=none; d=google.com; s=arc-20160816; b=rs/ij6gmrcqILgbHFs0qxsSDLzKjMgP4ezSbY3fzT0eoZgFt8OnM7ZcMEd9RVZUimz aGHpJG4mIlPjhM2xNbEsyzgsgdey47ELvBxt7i5AbO8VH3lQOaB7YlLwZuNRMyhjTgAf 0TJDQ1B5C1d3r1WXID9ikMcyVj1DypG9WyXdedseVpedKmZDRrSYcbM1XN9IufnPioEl xSQKIVHw9jF3rUtyNEbLTpeNoHPhNMSw3M1EJKWtoZ8YCoPyYKul0Jw3YRAu7EeF/Hha oC675u6Ye4zQ5MpvxMJMKH0kedKJKdzI+nn/APcHqUUltzMITEcYV15yzCbcEUdxbn1z wtPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:ironport-sdr :ironport-sdr:delivered-to; bh=07rBTz2ZY5F82JuT1CnIXMw24CsFvc0yRanXnZomBLU=; b=P4TUPG125HjdwfrRdyTIdqjES2idkftzPhUJHtjxDL3lY3D5FYtzKxGUdLidGxymfj PyTaEl+sRpgzfAcgauUOJQDG29X046+OX/LTz1XY8bIeMfGEx5DOBqAOWcz9cUaqY+At Exv3aozamNW4T1Pzngn3jDLMoy8Wq2ZdS+yNumfwbAiswgPo1Ozo7mlx1d3rlZYwEY2E mXREjNkKMsZ9LDhXgVxfbIQV6oXg2hPChtLmQ2AEcScj3TgELvY8XqhyBdDu0REq1OqB BOQeFD8Hr7WMXEmKYEaMQMyOu3oT682owqvGcz6bNsQrFlHM0Pq+cgjWUygzb4Lw0xAz uc9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id sb14si5669561ejb.322.2021.05.14.01.57.01; Fri, 14 May 2021 01:57:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CF3876880D3; Fri, 14 May 2021 11:56:57 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DBD23687F2A for ; Fri, 14 May 2021 11:56:50 +0300 (EEST) IronPort-SDR: KfS2pc2OLX2ZscxD7+m1Mywr0AbSvMLJSxsUlnnh/AhHVBxKBoW2saJpxOTQf3Ia4O4GqiQUa7 wAQ3VYWF2lrA== X-IronPort-AV: E=McAfee;i="6200,9189,9983"; a="199831917" X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="199831917" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 01:56:41 -0700 IronPort-SDR: nCikOa1hga1tXeSpIy5Eeu2D96QPGiII54n5ktsaFSM7yUkGQ2CeeBHllG2p6kmVMQuWXEVuI8 H3KCgkZYW9DQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="393561439" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.83]) by orsmga006.jf.intel.com with ESMTP; 14 May 2021 01:56:40 -0700 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 14 May 2021 16:47:00 +0800 Message-Id: <20210514084702.21273-1-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH 1/3] lavfi/drawbox: refine code X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: JtdD+0fnGi9r Extract common code of filter_frame() and drawgrid_filter_frame() to draw_region(). Signed-off-by: Ting Fu --- libavfilter/vf_drawbox.c | 160 ++++++++++++++------------------------- 1 file changed, 58 insertions(+), 102 deletions(-) diff --git a/libavfilter/vf_drawbox.c b/libavfilter/vf_drawbox.c index 2794fc2520..95e26191bd 100644 --- a/libavfilter/vf_drawbox.c +++ b/libavfilter/vf_drawbox.c @@ -85,6 +85,61 @@ typedef struct DrawBoxContext { static const int NUM_EXPR_EVALS = 5; +typedef int (*PixelBelongsToRegion)(DrawBoxContext *s, int x, int y); + +#define ASSIGN_THREE_CHANNELS \ + row[0] = frame->data[0] + y * frame->linesize[0]; \ + row[1] = frame->data[1] + (y >> ctx->vsub) * frame->linesize[1]; \ + row[2] = frame->data[2] + (y >> ctx->vsub) * frame->linesize[2]; + +#define ASSIGN_FOUR_CHANNELS \ + ASSIGN_THREE_CHANNELS \ + row[3] = frame->data[3] + y * frame->linesize[3]; + +static void draw_region(AVFrame *frame, DrawBoxContext *ctx, int left, int top, int right, int down, + PixelBelongsToRegion pixel_belongs_to_region) +{ + unsigned char *row[4]; + int x, y; + if (ctx->have_alpha && ctx->replace) { + for (y = top; y < down; y++) { + ASSIGN_FOUR_CHANNELS + if (ctx->invert_color) { + for (x = left; x < right; x++) + if (pixel_belongs_to_region(ctx, x, y)) + row[0][x] = 0xff - row[0][x]; + } else { + for (x = left; x < right; x++) { + if (pixel_belongs_to_region(ctx, x, y)) { + row[0][x ] = ctx->yuv_color[Y]; + row[1][x >> ctx->hsub] = ctx->yuv_color[U]; + row[2][x >> ctx->hsub] = ctx->yuv_color[V]; + row[3][x ] = ctx->yuv_color[A]; + } + } + } + } + } else { + for (y = top; y < down; y++) { + ASSIGN_THREE_CHANNELS + if (ctx->invert_color) { + if (pixel_belongs_to_region(ctx, x, y)) + row[0][x] = 0xff - row[0][x]; + } else { + for (x = left; x < right; x++) { + double alpha = (double)ctx->yuv_color[A] / 255; + + if (pixel_belongs_to_region(ctx, x, y)) { + row[0][x ] = (1 - alpha) * row[0][x ] + alpha * ctx->yuv_color[Y]; + row[1][x >> ctx->hsub] = (1 - alpha) * row[1][x >> ctx->hsub] + alpha * ctx->yuv_color[U]; + row[2][x >> ctx->hsub] = (1 - alpha) * row[2][x >> ctx->hsub] + alpha * ctx->yuv_color[V]; + } + } + } + } + } +} + static av_cold int init(AVFilterContext *ctx) { DrawBoxContext *s = ctx->priv; @@ -217,58 +272,9 @@ static av_pure av_always_inline int pixel_belongs_to_box(DrawBoxContext *s, int static int filter_frame(AVFilterLink *inlink, AVFrame *frame) { DrawBoxContext *s = inlink->dst->priv; - int plane, x, y, xb = s->x, yb = s->y; - unsigned char *row[4]; - - if (s->have_alpha && s->replace) { - for (y = FFMAX(yb, 0); y < frame->height && y < (yb + s->h); y++) { - row[0] = frame->data[0] + y * frame->linesize[0]; - row[3] = frame->data[3] + y * frame->linesize[3]; - - for (plane = 1; plane < 3; plane++) - row[plane] = frame->data[plane] + - frame->linesize[plane] * (y >> s->vsub); - - if (s->invert_color) { - for (x = FFMAX(xb, 0); x < xb + s->w && x < frame->width; x++) - if (pixel_belongs_to_box(s, x, y)) - row[0][x] = 0xff - row[0][x]; - } else { - for (x = FFMAX(xb, 0); x < xb + s->w && x < frame->width; x++) { - if (pixel_belongs_to_box(s, x, y)) { - row[0][x ] = s->yuv_color[Y]; - row[1][x >> s->hsub] = s->yuv_color[U]; - row[2][x >> s->hsub] = s->yuv_color[V]; - row[3][x ] = s->yuv_color[A]; - } - } - } - } - } else { - for (y = FFMAX(yb, 0); y < frame->height && y < (yb + s->h); y++) { - row[0] = frame->data[0] + y * frame->linesize[0]; - for (plane = 1; plane < 3; plane++) - row[plane] = frame->data[plane] + - frame->linesize[plane] * (y >> s->vsub); - - if (s->invert_color) { - for (x = FFMAX(xb, 0); x < xb + s->w && x < frame->width; x++) - if (pixel_belongs_to_box(s, x, y)) - row[0][x] = 0xff - row[0][x]; - } else { - for (x = FFMAX(xb, 0); x < xb + s->w && x < frame->width; x++) { - double alpha = (double)s->yuv_color[A] / 255; - - if (pixel_belongs_to_box(s, x, y)) { - row[0][x ] = (1 - alpha) * row[0][x ] + alpha * s->yuv_color[Y]; - row[1][x >> s->hsub] = (1 - alpha) * row[1][x >> s->hsub] + alpha * s->yuv_color[U]; - row[2][x >> s->hsub] = (1 - alpha) * row[2][x >> s->hsub] + alpha * s->yuv_color[V]; - } - } - } - } - } + draw_region(frame, s, FFMAX(s->x, 0), FFMAX(s->y, 0), FFMIN(s->x + s->w, frame->width), + FFMIN(s->y + s->h, frame->height), pixel_belongs_to_box); return ff_filter_frame(inlink->dst->outputs[0], frame); } @@ -389,58 +395,8 @@ static av_pure av_always_inline int pixel_belongs_to_grid(DrawBoxContext *drawgr static int drawgrid_filter_frame(AVFilterLink *inlink, AVFrame *frame) { DrawBoxContext *drawgrid = inlink->dst->priv; - int plane, x, y; - uint8_t *row[4]; - - if (drawgrid->have_alpha && drawgrid->replace) { - for (y = 0; y < frame->height; y++) { - row[0] = frame->data[0] + y * frame->linesize[0]; - row[3] = frame->data[3] + y * frame->linesize[3]; - - for (plane = 1; plane < 3; plane++) - row[plane] = frame->data[plane] + - frame->linesize[plane] * (y >> drawgrid->vsub); - - if (drawgrid->invert_color) { - for (x = 0; x < frame->width; x++) - if (pixel_belongs_to_grid(drawgrid, x, y)) - row[0][x] = 0xff - row[0][x]; - } else { - for (x = 0; x < frame->width; x++) { - if (pixel_belongs_to_grid(drawgrid, x, y)) { - row[0][x ] = drawgrid->yuv_color[Y]; - row[1][x >> drawgrid->hsub] = drawgrid->yuv_color[U]; - row[2][x >> drawgrid->hsub] = drawgrid->yuv_color[V]; - row[3][x ] = drawgrid->yuv_color[A]; - } - } - } - } - } else { - for (y = 0; y < frame->height; y++) { - row[0] = frame->data[0] + y * frame->linesize[0]; - for (plane = 1; plane < 3; plane++) - row[plane] = frame->data[plane] + - frame->linesize[plane] * (y >> drawgrid->vsub); - - if (drawgrid->invert_color) { - for (x = 0; x < frame->width; x++) - if (pixel_belongs_to_grid(drawgrid, x, y)) - row[0][x] = 0xff - row[0][x]; - } else { - for (x = 0; x < frame->width; x++) { - double alpha = (double)drawgrid->yuv_color[A] / 255; - - if (pixel_belongs_to_grid(drawgrid, x, y)) { - row[0][x ] = (1 - alpha) * row[0][x ] + alpha * drawgrid->yuv_color[Y]; - row[1][x >> drawgrid->hsub] = (1 - alpha) * row[1][x >> drawgrid->hsub] + alpha * drawgrid->yuv_color[U]; - row[2][x >> drawgrid->hsub] = (1 - alpha) * row[2][x >> drawgrid->hsub] + alpha * drawgrid->yuv_color[V]; - } - } - } - } - } + draw_region(frame, drawgrid, 0, 0, frame->width, frame->height, pixel_belongs_to_grid); return ff_filter_frame(inlink->dst->outputs[0], frame); } From patchwork Fri May 14 08:47:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 27769 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp276863iof; Fri, 14 May 2021 01:57:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxcEm1MWR95xmYIXhX81aLNbfnMV5bocgMkjBnRC4aDdsKK+Pl53LIIdybPmTm4R4c8jkgB X-Received: by 2002:aa7:c782:: with SMTP id n2mr55825881eds.77.1620982631331; Fri, 14 May 2021 01:57:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1620982631; cv=none; d=google.com; s=arc-20160816; b=e1oileqoIZMDpfkQClxBpNpxRDKi2F6mwhwjxgEdA8IsyBe3psrjk0HCmm7kwIUZxd fWa5bDG3wdxzEejDhblBKzH6MH9vintaB9qLZyeYw7qGU6yds8HsRVLvB/wtJ8+WlKEJ xTVQEmzfaZwIx2/i7DRSJSJYYccsBkeKj/2qxSAKItDgnM47kszImyUKlk4jnK0uEV9V j1P3m+LGfvXWVOqajgZRsmkvDWtgKIMpWaci0ZI5wv6tZr5yfrhMS8zVWwTjbj+3D1VC ngpA1BlnaMN5n+EyNI9vFTHwOS5WtSXMQJLW7X56ZUzz9KklveR/BsFLQPbo24ovRuuE WWNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:ironport-sdr:ironport-sdr:delivered-to; bh=oOWuxxkA1FuqFqSPmSmGHBob2j6o87rUjWjTViuYE7E=; b=QgAquk4Nno8QH2Pyw7EaZ6heqSJj7+U8LGSHv8NhF32Kdl/C68Ug/hy7OquGO7LorO eMZCOdqAcYwzYHOrbWCL75IwtLjkpPf71JrQXByIvg7FrV94j4r+nfsrO3xxth1SbnWk PuetgQNkQ2IOu6iYtSLnwN1HYcUhX3EIFWi6xEzfCHZeiLiaejTnjOle+ufPWivxeNtc 3f1RELVIQkjYhSnrIcTV+cfNi7+IEx7YZ/hBrvlq8FuGB4p4EXNTRsTu3N4urb2IQ8H+ 1o9bWWORLkAvoV7RGbkvG22hdICauPb6exg/srhWmuLhufYB0yJlSvad1K1O2I8xuT0q QSpA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t15si5227031ejd.261.2021.05.14.01.57.10; Fri, 14 May 2021 01:57:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 14A0268818B; Fri, 14 May 2021 11:57:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C0B6A68809B for ; Fri, 14 May 2021 11:56:56 +0300 (EEST) IronPort-SDR: 7F2dvNMhRZS+SBMS2CVl9hO7pKbTOgQaWMf1+2txy2igUOSrV+Rk2zG8RYqCVMuPfBNy23ZNJG ouFL3nCB93kA== X-IronPort-AV: E=McAfee;i="6200,9189,9983"; a="199831920" X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="199831920" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 01:56:42 -0700 IronPort-SDR: LgdHVHyOLB674oKL7/V9NzJYie5ntvENqpwByT3URqyD7uKN04XTaQi0+bU/88T7ExL5se5laD YFbudyvAscTw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="393561452" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.83]) by orsmga006.jf.intel.com with ESMTP; 14 May 2021 01:56:41 -0700 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 14 May 2021 16:47:01 +0800 Message-Id: <20210514084702.21273-2-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514084702.21273-1-ting.fu@intel.com> References: <20210514084702.21273-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH 2/3] libavfilter: vf_drawbox filter support draw box with detection bounding boxes in side_data X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: COfsFRBj0vtQ This feature can be used with dnn detection by setting vf_drawbox's option box_source=side_data_detection_bboxes, for example: ./ffmpeg -i face.jpeg -vf dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:\ input=data:output=detection_out:labels=face-detection-adas-0001.label,\ drawbox=box_source=side_data_detection_bboxes -y face_detect.jpeg Signed-off-by: Ting Fu --- doc/filters.texi | 8 +++++++ libavfilter/vf_drawbox.c | 52 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index a218289ddd..f2ac8c4cc8 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -10356,6 +10356,14 @@ The x and y offset coordinates where the box is drawn. @item h The width and height of the drawn box. +@item box_source +Box source can be set as side_data_detection_bboxes if you want to use box data in +detection bboxes of side data. + +If @var{box_source} is set, the @var{x}, @var{y}, @var{width} and @var{height} will be ignored and +still use box data in detection bboxes of side data. So please do not use this parameter if you were +not sure about the box source. + @item t The thickness of the drawn box. diff --git a/libavfilter/vf_drawbox.c b/libavfilter/vf_drawbox.c index 95e26191bd..fff78862e9 100644 --- a/libavfilter/vf_drawbox.c +++ b/libavfilter/vf_drawbox.c @@ -31,6 +31,7 @@ #include "libavutil/eval.h" #include "libavutil/pixdesc.h" #include "libavutil/parseutils.h" +#include "libavutil/detection_bbox.h" #include "avfilter.h" #include "formats.h" #include "internal.h" @@ -79,8 +80,10 @@ typedef struct DrawBoxContext { char *x_expr, *y_expr; ///< expression for x and y char *w_expr, *h_expr; ///< expression for width and height char *t_expr; ///< expression for thickness + char *box_source_string; ///< string for box data source int have_alpha; int replace; + enum AVFrameSideDataType box_source; } DrawBoxContext; static const int NUM_EXPR_EVALS = 5; @@ -140,11 +143,30 @@ static void draw_region(AVFrame *frame, DrawBoxContext *ctx, int left, int top, } } +static enum AVFrameSideDataType box_source_string_parse(const char *box_source_string) +{ + av_assert0(box_source_string); + if (!strcmp(box_source_string, "side_data_detection_bboxes")) { + return AV_FRAME_DATA_DETECTION_BBOXES; + } else { + // will support side_data_regions_of_interest next + return AVERROR(EINVAL); + } +} + static av_cold int init(AVFilterContext *ctx) { DrawBoxContext *s = ctx->priv; uint8_t rgba_color[4]; + if (s->box_source_string) { + s->box_source = box_source_string_parse(s->box_source_string); + if ((int)s->box_source < 0) { + av_log(ctx, AV_LOG_ERROR, "Error box source: %s\n",s->box_source_string); + return AVERROR(EINVAL); + } + } + if (!strcmp(s->color_str, "invert")) s->invert_color = 1; else if (av_parse_color(rgba_color, s->color_str, -1, ctx) < 0) @@ -272,9 +294,34 @@ static av_pure av_always_inline int pixel_belongs_to_box(DrawBoxContext *s, int static int filter_frame(AVFilterLink *inlink, AVFrame *frame) { DrawBoxContext *s = inlink->dst->priv; + const AVDetectionBBoxHeader *header = NULL; + const AVDetectionBBox *bbox; + AVFrameSideData *sd; + int loop = 1; + + if (s->box_source == AV_FRAME_DATA_DETECTION_BBOXES) { + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES); + if (sd) { + header = (AVDetectionBBoxHeader *)sd->data; + loop = header->nb_bboxes; + } else { + av_log(s, AV_LOG_WARNING, "No detection bboxes.\n"); + return ff_filter_frame(inlink->dst->outputs[0], frame); + } + } - draw_region(frame, s, FFMAX(s->x, 0), FFMAX(s->y, 0), FFMIN(s->x + s->w, frame->width), - FFMIN(s->y + s->h, frame->height), pixel_belongs_to_box); + for (int i = 0; i < loop; i++) { + if (header) { + bbox = av_get_detection_bbox(header, i); + s->y = bbox->y; + s->x = bbox->x; + s->h = bbox->h; + s->w = bbox->w; + } + + draw_region(frame, s, FFMAX(s->x, 0), FFMAX(s->y, 0), FFMIN(s->x + s->w, frame->width), + FFMIN(s->y + s->h, frame->height), pixel_belongs_to_box); + } return ff_filter_frame(inlink->dst->outputs[0], frame); } @@ -329,6 +376,7 @@ static const AVOption drawbox_options[] = { { "thickness", "set the box thickness", OFFSET(t_expr), AV_OPT_TYPE_STRING, { .str="3" }, 0, 0, FLAGS }, { "t", "set the box thickness", OFFSET(t_expr), AV_OPT_TYPE_STRING, { .str="3" }, 0, 0, FLAGS }, { "replace", "replace color & alpha", OFFSET(replace), AV_OPT_TYPE_BOOL, { .i64=0 }, 0, 1, FLAGS }, + { "box_source", "use datas from bounding box in side data", OFFSET(box_source_string), AV_OPT_TYPE_STRING, { .str=NULL }, 0, 1, FLAGS }, { NULL } }; From patchwork Fri May 14 08:47:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 27770 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp276973iof; Fri, 14 May 2021 01:57:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyEvnaIzKNTgQEBPkTo5W2S8WRcmQtPblABARJPMrfvB0nJQUqh901/LDVXBquoZEiNDV0u X-Received: by 2002:a17:906:1110:: with SMTP id h16mr31392327eja.530.1620982640066; Fri, 14 May 2021 01:57:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1620982640; cv=none; d=google.com; s=arc-20160816; b=xXxnzuTi+rimLntLI7I4Oie7PQL9jx4O7PXMEyEJXKkYl8n9rbSt8pF3jxNk7pKeJU cj21nYF6lWQoLNRowAxULVI7roLTMJr5HJSpAMQyfx193SstHFSaMcZ53plESAiHfucF RYuIkQ5MS3YkVKlI2qNNsBYQjLqf/SCN0speu/uA8nOY9EMLVOweZ6Y52Lri/fgGmrDf haVFvnSm5Laj6r5GDGEEnnFocE4+OMt6gj1Ivm8K+oIQV8edd8GdNq43eKAsIy+Ek1jY 7G50AEnVKEY+QA8hwGFRMATZFhMsmUG0pusNeOVUIUsqzb3W9mRktp4ftEwYCvFtEjwq xfew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:ironport-sdr:ironport-sdr:delivered-to; bh=YG4snRR4UufA2dL4QaaRIEEBeZyV+IeGOnnGxZfYzbs=; b=YXsc0sE/CKXODCY0MhIW0rOnvhMvV0rYqGxh8Ku3TTS8Tlrb/POKlD4NXX3C6wvhxT BZii5AWM9CTbzPCaz+qtONSk4hOM9KUPztDOW6CjvL8t4aHwOXIzZSMoGDOGCsQIEgTE GJxlw7D+bXPyPcWCSMPSx/nbn+OEuptuUccoTdu0OnrAgBFnMq7lX0wxNjgDm2U0PMpT IY/jX75598AUhckb7XOYxRKygCB+GYsgheVCROP37foh8bB9NiFd4YIhV0sE5qv4nVOR IpNLOsF8VROKbvGkRRzYXRc86FB7NEH1ecdCCwZLoJ6Z4AIhUkPfGLvCdy+AiB7l1eFf 8wCA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y21si4703620eda.281.2021.05.14.01.57.19; Fri, 14 May 2021 01:57:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 371AE6881AC; Fri, 14 May 2021 11:57:06 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0E2EF68809B for ; Fri, 14 May 2021 11:56:57 +0300 (EEST) IronPort-SDR: ZX4XL9PN05/wq5nL4EpbNNu/cZs1XIAdT2WWaJSfqfmSKu8TGYGeOnqFF7JsP0pX6BwetTgIjz CMrAK29SdrDw== X-IronPort-AV: E=McAfee;i="6200,9189,9983"; a="199831922" X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="199831922" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 01:56:42 -0700 IronPort-SDR: GrhcOphwtvR/Plce06RrAjGwB/VUc2ZvcNdouhN2y9p7A2apqh6QKX79l9E4vxv9Aw77udC/et rdfMVaqoN/Wg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="393561462" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.83]) by orsmga006.jf.intel.com with ESMTP; 14 May 2021 01:56:42 -0700 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 14 May 2021 16:47:02 +0800 Message-Id: <20210514084702.21273-3-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514084702.21273-1-ting.fu@intel.com> References: <20210514084702.21273-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH 3/3] libavfilter: vf_drawtext filter support draw text with detection bounding boxes in side_data X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: KDVoEAzbVvV4 This feature can be used with dnn detection by setting vf_drawtext's option text_source=side_data_detection_bboxes, for example: ./ffmpeg -i face.jpeg -vf dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:\ input=data:output=detection_out:labels=face-detection-adas-0001.label,drawbox=box_source= side_data_detection_bboxes,drawtext=text_source=side_data_detection_bboxes:fontcolor=green:\ fontsize=40, -y face_detect.jpeg Please note, the default fontsize of vf_drawtext is 12, which may be too small to be seen clearly. Signed-off-by: Ting Fu --- doc/filters.texi | 8 ++++ libavfilter/vf_drawtext.c | 77 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 79 insertions(+), 6 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index f2ac8c4cc8..d10e6de03d 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -10788,6 +10788,14 @@ parameter @var{text}. If both @var{text} and @var{textfile} are specified, an error is thrown. +@item text_source +Text source should be set as side_data_detection_bboxes if you want to use text data in +detection bboxes of side data. + +If text source is set, @var{text} and @var{textfile} will be ignored and still use +text data in detection bboxes of side data. So please do not use this parameter +if you are not sure about the text source. + @item reload If set to 1, the @var{textfile} will be reloaded before each frame. Be sure to update it atomically, or it may be read partially, or even fail. diff --git a/libavfilter/vf_drawtext.c b/libavfilter/vf_drawtext.c index 7ea057b812..382d589e26 100644 --- a/libavfilter/vf_drawtext.c +++ b/libavfilter/vf_drawtext.c @@ -55,6 +55,7 @@ #include "libavutil/time_internal.h" #include "libavutil/tree.h" #include "libavutil/lfg.h" +#include "libavutil/detection_bbox.h" #include "avfilter.h" #include "drawutils.h" #include "formats.h" @@ -199,6 +200,8 @@ typedef struct DrawTextContext { int tc24hmax; ///< 1 if timecode is wrapped to 24 hours, 0 otherwise int reload; ///< reload text file for each frame int start_number; ///< starting frame number for n/frame_num var + char *text_source_string; ///< the string to specify text data source + enum AVFrameSideDataType text_source; #if CONFIG_LIBFRIBIDI int text_shaping; ///< 1 to shape the text before drawing it #endif @@ -246,6 +249,7 @@ static const AVOption drawtext_options[]= { { "alpha", "apply alpha while rendering", OFFSET(a_expr), AV_OPT_TYPE_STRING, { .str = "1" }, .flags = FLAGS }, {"fix_bounds", "check and fix text coords to avoid clipping", OFFSET(fix_bounds), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS}, {"start_number", "start frame number for n/frame_num variable", OFFSET(start_number), AV_OPT_TYPE_INT, {.i64=0}, 0, INT_MAX, FLAGS}, + {"text_source", "the source of text", OFFSET(text_source_string), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 1, FLAGS }, #if CONFIG_LIBFRIBIDI {"text_shaping", "attempt to shape text before drawing", OFFSET(text_shaping), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, FLAGS}, @@ -690,6 +694,16 @@ out: } #endif +static enum AVFrameSideDataType text_source_string_parse(const char *text_source_string) +{ + av_assert0(text_source_string); + if (!strcmp(text_source_string, "side_data_detection_bboxes")) { + return AV_FRAME_DATA_DETECTION_BBOXES; + } else { + return AVERROR(EINVAL); + } +} + static av_cold int init(AVFilterContext *ctx) { int err; @@ -731,9 +745,28 @@ static av_cold int init(AVFilterContext *ctx) s->text = av_strdup(""); } + if (s->text_source_string) { + s->text_source = text_source_string_parse(s->text_source_string); + if ((int)s->text_source < 0) { + av_log(ctx, AV_LOG_ERROR, "Error text source: %s\n", s->text_source_string); + return AVERROR(EINVAL); + } + } + + if (s->text_source == AV_FRAME_DATA_DETECTION_BBOXES) { + if (s->text) { + av_log(ctx, AV_LOG_WARNING, "Multiple texts provided, will use text_source only\n"); + av_free(s->text); + } + s->text = av_mallocz(AV_DETECTION_BBOX_LABEL_NAME_MAX_SIZE * + (AV_NUM_DETECTION_BBOX_CLASSIFY + 1)); + if (!s->text) + return AVERROR(ENOMEM); + } + if (!s->text) { av_log(ctx, AV_LOG_ERROR, - "Either text, a valid file or a timecode must be provided\n"); + "Either text, a valid file, a timecode or text source must be provided\n"); return AVERROR(EINVAL); } @@ -1440,10 +1473,15 @@ continue_on_invalid2: s->var_values[VAR_LINE_H] = s->var_values[VAR_LH] = s->max_glyph_h; - s->x = s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, &s->prng); - s->y = s->var_values[VAR_Y] = av_expr_eval(s->y_pexpr, s->var_values, &s->prng); - /* It is necessary if x is expressed from y */ - s->x = s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, &s->prng); + if (s->text_source == AV_FRAME_DATA_DETECTION_BBOXES) { + s->var_values[VAR_X] = s->x; + s->var_values[VAR_Y] = s->y; + } else { + s->x = s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, &s->prng); + s->y = s->var_values[VAR_Y] = av_expr_eval(s->y_pexpr, s->var_values, &s->prng); + /* It is necessary if x is expressed from y */ + s->x = s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, &s->prng); + } update_alpha(s); update_color_with_alpha(s, &fontcolor , s->fontcolor ); @@ -1511,6 +1549,21 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) AVFilterLink *outlink = ctx->outputs[0]; DrawTextContext *s = ctx->priv; int ret; + const AVDetectionBBoxHeader *header = NULL; + const AVDetectionBBox *bbox; + AVFrameSideData *sd; + int loop = 1; + + if (s->text_source == AV_FRAME_DATA_DETECTION_BBOXES && sd) { + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES); + if (sd) { + header = (AVDetectionBBoxHeader *)sd->data; + loop = header->nb_bboxes; + } else { + av_log(s, AV_LOG_WARNING, "No detection bboxes.\n"); + return ff_filter_frame(outlink, frame); + } + } if (s->reload) { if ((ret = load_textfile(ctx)) < 0) { @@ -1536,7 +1589,19 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) s->var_values[VAR_PKT_SIZE] = frame->pkt_size; s->metadata = frame->metadata; - draw_text(ctx, frame, frame->width, frame->height); + for (int i = 0; i < loop; i++) { + if (header) { + bbox = av_get_detection_bbox(header, i); + strcpy(s->text, bbox->detect_label); + for (int j = 0; j < bbox->classify_count; j++) { + strcat(s->text, ", "); + strcat(s->text, bbox->classify_labels[j]); + } + s->x = bbox->x; + s->y = bbox->y - s->fontsize; + } + draw_text(ctx, frame, frame->width, frame->height); + } av_log(ctx, AV_LOG_DEBUG, "n:%d t:%f text_w:%d text_h:%d x:%d y:%d\n", (int)s->var_values[VAR_N], s->var_values[VAR_T],