From patchwork Fri May 14 08:47:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 27769 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp276863iof; Fri, 14 May 2021 01:57:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxcEm1MWR95xmYIXhX81aLNbfnMV5bocgMkjBnRC4aDdsKK+Pl53LIIdybPmTm4R4c8jkgB X-Received: by 2002:aa7:c782:: with SMTP id n2mr55825881eds.77.1620982631331; Fri, 14 May 2021 01:57:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1620982631; cv=none; d=google.com; s=arc-20160816; b=e1oileqoIZMDpfkQClxBpNpxRDKi2F6mwhwjxgEdA8IsyBe3psrjk0HCmm7kwIUZxd fWa5bDG3wdxzEejDhblBKzH6MH9vintaB9qLZyeYw7qGU6yds8HsRVLvB/wtJ8+WlKEJ xTVQEmzfaZwIx2/i7DRSJSJYYccsBkeKj/2qxSAKItDgnM47kszImyUKlk4jnK0uEV9V j1P3m+LGfvXWVOqajgZRsmkvDWtgKIMpWaci0ZI5wv6tZr5yfrhMS8zVWwTjbj+3D1VC ngpA1BlnaMN5n+EyNI9vFTHwOS5WtSXMQJLW7X56ZUzz9KklveR/BsFLQPbo24ovRuuE WWNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:ironport-sdr:ironport-sdr:delivered-to; bh=oOWuxxkA1FuqFqSPmSmGHBob2j6o87rUjWjTViuYE7E=; b=QgAquk4Nno8QH2Pyw7EaZ6heqSJj7+U8LGSHv8NhF32Kdl/C68Ug/hy7OquGO7LorO eMZCOdqAcYwzYHOrbWCL75IwtLjkpPf71JrQXByIvg7FrV94j4r+nfsrO3xxth1SbnWk PuetgQNkQ2IOu6iYtSLnwN1HYcUhX3EIFWi6xEzfCHZeiLiaejTnjOle+ufPWivxeNtc 3f1RELVIQkjYhSnrIcTV+cfNi7+IEx7YZ/hBrvlq8FuGB4p4EXNTRsTu3N4urb2IQ8H+ 1o9bWWORLkAvoV7RGbkvG22hdICauPb6exg/srhWmuLhufYB0yJlSvad1K1O2I8xuT0q QSpA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t15si5227031ejd.261.2021.05.14.01.57.10; Fri, 14 May 2021 01:57:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 14A0268818B; Fri, 14 May 2021 11:57:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C0B6A68809B for ; Fri, 14 May 2021 11:56:56 +0300 (EEST) IronPort-SDR: 7F2dvNMhRZS+SBMS2CVl9hO7pKbTOgQaWMf1+2txy2igUOSrV+Rk2zG8RYqCVMuPfBNy23ZNJG ouFL3nCB93kA== X-IronPort-AV: E=McAfee;i="6200,9189,9983"; a="199831920" X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="199831920" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2021 01:56:42 -0700 IronPort-SDR: LgdHVHyOLB674oKL7/V9NzJYie5ntvENqpwByT3URqyD7uKN04XTaQi0+bU/88T7ExL5se5laD YFbudyvAscTw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="393561452" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.83]) by orsmga006.jf.intel.com with ESMTP; 14 May 2021 01:56:41 -0700 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 14 May 2021 16:47:01 +0800 Message-Id: <20210514084702.21273-2-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210514084702.21273-1-ting.fu@intel.com> References: <20210514084702.21273-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH 2/3] libavfilter: vf_drawbox filter support draw box with detection bounding boxes in side_data X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: COfsFRBj0vtQ This feature can be used with dnn detection by setting vf_drawbox's option box_source=side_data_detection_bboxes, for example: ./ffmpeg -i face.jpeg -vf dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:\ input=data:output=detection_out:labels=face-detection-adas-0001.label,\ drawbox=box_source=side_data_detection_bboxes -y face_detect.jpeg Signed-off-by: Ting Fu --- doc/filters.texi | 8 +++++++ libavfilter/vf_drawbox.c | 52 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index a218289ddd..f2ac8c4cc8 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -10356,6 +10356,14 @@ The x and y offset coordinates where the box is drawn. @item h The width and height of the drawn box. +@item box_source +Box source can be set as side_data_detection_bboxes if you want to use box data in +detection bboxes of side data. + +If @var{box_source} is set, the @var{x}, @var{y}, @var{width} and @var{height} will be ignored and +still use box data in detection bboxes of side data. So please do not use this parameter if you were +not sure about the box source. + @item t The thickness of the drawn box. diff --git a/libavfilter/vf_drawbox.c b/libavfilter/vf_drawbox.c index 95e26191bd..fff78862e9 100644 --- a/libavfilter/vf_drawbox.c +++ b/libavfilter/vf_drawbox.c @@ -31,6 +31,7 @@ #include "libavutil/eval.h" #include "libavutil/pixdesc.h" #include "libavutil/parseutils.h" +#include "libavutil/detection_bbox.h" #include "avfilter.h" #include "formats.h" #include "internal.h" @@ -79,8 +80,10 @@ typedef struct DrawBoxContext { char *x_expr, *y_expr; ///< expression for x and y char *w_expr, *h_expr; ///< expression for width and height char *t_expr; ///< expression for thickness + char *box_source_string; ///< string for box data source int have_alpha; int replace; + enum AVFrameSideDataType box_source; } DrawBoxContext; static const int NUM_EXPR_EVALS = 5; @@ -140,11 +143,30 @@ static void draw_region(AVFrame *frame, DrawBoxContext *ctx, int left, int top, } } +static enum AVFrameSideDataType box_source_string_parse(const char *box_source_string) +{ + av_assert0(box_source_string); + if (!strcmp(box_source_string, "side_data_detection_bboxes")) { + return AV_FRAME_DATA_DETECTION_BBOXES; + } else { + // will support side_data_regions_of_interest next + return AVERROR(EINVAL); + } +} + static av_cold int init(AVFilterContext *ctx) { DrawBoxContext *s = ctx->priv; uint8_t rgba_color[4]; + if (s->box_source_string) { + s->box_source = box_source_string_parse(s->box_source_string); + if ((int)s->box_source < 0) { + av_log(ctx, AV_LOG_ERROR, "Error box source: %s\n",s->box_source_string); + return AVERROR(EINVAL); + } + } + if (!strcmp(s->color_str, "invert")) s->invert_color = 1; else if (av_parse_color(rgba_color, s->color_str, -1, ctx) < 0) @@ -272,9 +294,34 @@ static av_pure av_always_inline int pixel_belongs_to_box(DrawBoxContext *s, int static int filter_frame(AVFilterLink *inlink, AVFrame *frame) { DrawBoxContext *s = inlink->dst->priv; + const AVDetectionBBoxHeader *header = NULL; + const AVDetectionBBox *bbox; + AVFrameSideData *sd; + int loop = 1; + + if (s->box_source == AV_FRAME_DATA_DETECTION_BBOXES) { + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES); + if (sd) { + header = (AVDetectionBBoxHeader *)sd->data; + loop = header->nb_bboxes; + } else { + av_log(s, AV_LOG_WARNING, "No detection bboxes.\n"); + return ff_filter_frame(inlink->dst->outputs[0], frame); + } + } - draw_region(frame, s, FFMAX(s->x, 0), FFMAX(s->y, 0), FFMIN(s->x + s->w, frame->width), - FFMIN(s->y + s->h, frame->height), pixel_belongs_to_box); + for (int i = 0; i < loop; i++) { + if (header) { + bbox = av_get_detection_bbox(header, i); + s->y = bbox->y; + s->x = bbox->x; + s->h = bbox->h; + s->w = bbox->w; + } + + draw_region(frame, s, FFMAX(s->x, 0), FFMAX(s->y, 0), FFMIN(s->x + s->w, frame->width), + FFMIN(s->y + s->h, frame->height), pixel_belongs_to_box); + } return ff_filter_frame(inlink->dst->outputs[0], frame); } @@ -329,6 +376,7 @@ static const AVOption drawbox_options[] = { { "thickness", "set the box thickness", OFFSET(t_expr), AV_OPT_TYPE_STRING, { .str="3" }, 0, 0, FLAGS }, { "t", "set the box thickness", OFFSET(t_expr), AV_OPT_TYPE_STRING, { .str="3" }, 0, 0, FLAGS }, { "replace", "replace color & alpha", OFFSET(replace), AV_OPT_TYPE_BOOL, { .i64=0 }, 0, 1, FLAGS }, + { "box_source", "use datas from bounding box in side data", OFFSET(box_source_string), AV_OPT_TYPE_STRING, { .str=NULL }, 0, 1, FLAGS }, { NULL } };