From patchwork Mon Dec 4 05:36:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chen, Wenbin" X-Patchwork-Id: 44897 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:a301:b0:181:818d:5e7f with SMTP id x1csp2566559pzk; Sun, 3 Dec 2023 21:37:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IFJFCbVdmLwDu6Vy/24K4J1V+78ewOLa6KVrQWtRSD/Bp8ZlAhYp6v6PB0PGRT4EJ1ko/oA X-Received: by 2002:ac2:504d:0:b0:50b:e76a:6ad1 with SMTP id a13-20020ac2504d000000b0050be76a6ad1mr1095726lfm.116.1701668236062; Sun, 03 Dec 2023 21:37:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701668236; cv=none; d=google.com; s=arc-20160816; b=mVdYaPJJtXLzUns3kWD+FHX5y/+TwMerruz0GML7DHptIBdszuoeOIvD1q0TfQjjfZ 6bzFGOLy6CdhRFV0o22lYCX86F1bn36GI6Ojdm3+52WDeSTfSYF4rb/zHZqsR3ys2mJI GzfPrRS/vKODdxvf8jEKFrMIHwl15ftb9WqjEsjJ06jqNPGQfVvm/ce29Hfv6/s80bW+ 2tqfRxOSLqvd7OgMTS3lHNPvpHgeZmU2Az6rnnpQVH+dih69FOMEU81NHX0LSGg4kZKe AU4aKK3V2BVehbvgkrzDVgO/AVA0Dp8mta0JLQT1y4FSig4jl6inKiUuCkbNAe1Do6DN m8gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=JaFIAot2qAm5NlN79ylv3YVppYUH+yq6OlUsoLdbQXM=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=C6vtI6ZRa1wWNefsWRdjqNr79DWuvUk8qCLFTO+hSJrDxg77NW10mVYi+i3GojSUFp IgH8BnPzH0oI1fnx/LOak7n3VHX5c+FOSG5sSCoQOqPkqNfiMRjLqLq46E1sfemKt8+K LE1nJ5QRKcz3M4RcApMoKYXyGpFZ/awTVN4vsgT2Gr9FXfYwaiK0YMhthWB48iSWN+lC n9D9QwjA3PxugQ7J3DQ0VFSXTc+r+/6/SmwOJtW6Hws7KqdxXkErpyWisarsVVHhUxoO Gg1kFxc5WS0Ios3wzDf8z01Uwq5H3E7sh9A7Kz1EQ0oialLxvjsRA/JhiMX9r5qk6Y3v fyRA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=kPG8XUNq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t17-20020a50c251000000b0054cc8d2c0d9si748672edf.565.2023.12.03.21.37.15; Sun, 03 Dec 2023 21:37:16 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=kPG8XUNq; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1A0C868CECE; Mon, 4 Dec 2023 07:36:53 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CF5E268CE38 for ; Mon, 4 Dec 2023 07:36:45 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1701668211; x=1733204211; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=FS8lqdjHwyeQoqINQEGfo36+QLLxv1E+XptoPNCgSRc=; b=kPG8XUNqn7T4A4cI0VMJPbyDeLpPkMjTsngdjSnmXuZ1O7489XjSh5/v Fjzlld0ClvGq0EE1aX+dGTIBLRwIpAn74wXf19v0ki+eHJr2xajccFYpy UKlkwATyJfil5wjdG8lSJEiK9Fr/WfqAbBzqFwj/+gIMetyWpHKUsenMm 7TeSX90XvBc/msFBtVq5VWyQiTKKJR/vk5XpZvcavBozYc+rk46a63cEA 0PiGt997Sk0m97n6QBqdfOQec6aUJqxv4YQ1Wl9/qn4I3Cl8ut8pFrevS W6mk4SOEQszoj6sT+sIXyiks291ewWLk/aY6XkSnYz0VuzJgZxgve2KAG w==; X-IronPort-AV: E=McAfee;i="6600,9927,10913"; a="397574033" X-IronPort-AV: E=Sophos;i="6.04,248,1695711600"; d="scan'208";a="397574033" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Dec 2023 21:36:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10913"; a="914308729" X-IronPort-AV: E=Sophos;i="6.04,249,1695711600"; d="scan'208";a="914308729" Received: from wenbin-z390-aorus-ultra.sh.intel.com ([10.239.156.43]) by fmsmga001.fm.intel.com with ESMTP; 03 Dec 2023 21:36:37 -0800 From: wenbin.chen-at-intel.com@ffmpeg.org To: ffmpeg-devel@ffmpeg.org Date: Mon, 4 Dec 2023 13:36:33 +0800 Message-Id: <20231204053633.1743228-4-wenbin.chen@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231204053633.1743228-1-wenbin.chen@intel.com> References: <20231204053633.1743228-1-wenbin.chen@intel.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/4] libavfilter/vf_dnn_detect: Add yolov4 support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: A6L/b/RpCzEu From: Wenbin Chen The difference of yolov4 is that sigmoid function needed to be applied on x, y coordinates. Also make it compatiple with NHWC output as the yolov4 model from openvino model zoo has NHWC output layout. Model refer to: https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v4-tf Signed-off-by: Wenbin Chen --- libavfilter/vf_dnn_detect.c | 71 ++++++++++++++++++++++++++++++------- 1 file changed, 59 insertions(+), 12 deletions(-) diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index 7a32b191c3..1b04a2cb98 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -35,7 +35,8 @@ typedef enum { DDMT_SSD, DDMT_YOLOV1V2, - DDMT_YOLOV3 + DDMT_YOLOV3, + DDMT_YOLOV4 } DNNDetectionModelType; typedef struct DnnDetectContext { @@ -75,6 +76,7 @@ static const AVOption dnn_detect_options[] = { { "ssd", "output shape [1, 1, N, 7]", 0, AV_OPT_TYPE_CONST, { .i64 = DDMT_SSD }, 0, 0, FLAGS, "model_type" }, { "yolo", "output shape [1, N*Cx*Cy*DetectionBox]", 0, AV_OPT_TYPE_CONST, { .i64 = DDMT_YOLOV1V2 }, 0, 0, FLAGS, "model_type" }, { "yolov3", "outputs shape [1, N*D, Cx, Cy]", 0, AV_OPT_TYPE_CONST, { .i64 = DDMT_YOLOV3 }, 0, 0, FLAGS, "model_type" }, + { "yolov4", "outputs shape [1, N*D, Cx, Cy]", 0, AV_OPT_TYPE_CONST, { .i64 = DDMT_YOLOV4 }, 0, 0, FLAGS, "model_type" }, { "cell_w", "cell width", OFFSET2(cell_w), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INTMAX_MAX, FLAGS }, { "cell_h", "cell height", OFFSET2(cell_h), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INTMAX_MAX, FLAGS }, { "nb_classes", "The number of class", OFFSET2(nb_classes), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INTMAX_MAX, FLAGS }, @@ -84,6 +86,14 @@ static const AVOption dnn_detect_options[] = { AVFILTER_DEFINE_CLASS(dnn_detect); +static inline float sigmoid(float x) { + return 1.f / (1.f + exp(-x)); +} + +static inline float linear(float x) { + return x; +} + static int dnn_detect_get_label_id(int nb_classes, int cell_size, float *label_data) { float max_prob = 0; @@ -142,6 +152,8 @@ static int dnn_detect_parse_yolo_output(AVFrame *frame, DNNData *output, int out float *output_data = output[output_index].data; float *anchors = ctx->anchors; AVDetectionBBox *bbox; + float (*post_process_raw_data)(float x); + int is_NHWC = 0; if (ctx->model_type == DDMT_YOLOV1V2) { cell_w = ctx->cell_w; @@ -149,13 +161,30 @@ static int dnn_detect_parse_yolo_output(AVFrame *frame, DNNData *output, int out scale_w = cell_w; scale_h = cell_h; } else { - cell_w = output[output_index].width; - cell_h = output[output_index].height; + if (output[output_index].height != output[output_index].width && + output[output_index].height == output[output_index].channels) { + is_NHWC = 1; + cell_w = output[output_index].height; + cell_h = output[output_index].channels; + } else { + cell_w = output[output_index].width; + cell_h = output[output_index].height; + } scale_w = ctx->scale_width; scale_h = ctx->scale_height; } box_size = nb_classes + 5; + switch (ctx->model_type) { + case DDMT_YOLOV1V2: + case DDMT_YOLOV3: + post_process_raw_data = linear; + break; + case DDMT_YOLOV4: + post_process_raw_data = sigmoid; + break; + } + if (!cell_h || !cell_w) { av_log(filter_ctx, AV_LOG_ERROR, "cell_w and cell_h are detected\n"); return AVERROR(EINVAL); @@ -193,19 +222,36 @@ static int dnn_detect_parse_yolo_output(AVFrame *frame, DNNData *output, int out float *detection_boxes_data; int label_id; - detection_boxes_data = output_data + box_id * box_size * cell_w * cell_h; - conf = detection_boxes_data[cy * cell_w + cx + 4 * cell_w * cell_h]; + if (is_NHWC) { + detection_boxes_data = output_data + + ((cy * cell_w + cx) * detection_boxes + box_id) * box_size; + conf = post_process_raw_data(detection_boxes_data[4]); + } else { + detection_boxes_data = output_data + box_id * box_size * cell_w * cell_h; + conf = post_process_raw_data( + detection_boxes_data[cy * cell_w + cx + 4 * cell_w * cell_h]); + } if (conf < conf_threshold) { continue; } - x = detection_boxes_data[cy * cell_w + cx]; - y = detection_boxes_data[cy * cell_w + cx + cell_w * cell_h]; - w = detection_boxes_data[cy * cell_w + cx + 2 * cell_w * cell_h]; - h = detection_boxes_data[cy * cell_w + cx + 3 * cell_w * cell_h]; - label_id = dnn_detect_get_label_id(ctx->nb_classes, cell_w * cell_h, - detection_boxes_data + cy * cell_w + cx + 5 * cell_w * cell_h); - conf = conf * detection_boxes_data[cy * cell_w + cx + (label_id + 5) * cell_w * cell_h]; + if (is_NHWC) { + x = post_process_raw_data(detection_boxes_data[0]); + y = post_process_raw_data(detection_boxes_data[1]); + w = detection_boxes_data[2]; + h = detection_boxes_data[3]; + label_id = dnn_detect_get_label_id(ctx->nb_classes, 1, detection_boxes_data + 5); + conf = conf * post_process_raw_data(detection_boxes_data[label_id + 5]); + } else { + x = post_process_raw_data(detection_boxes_data[cy * cell_w + cx]); + y = post_process_raw_data(detection_boxes_data[cy * cell_w + cx + cell_w * cell_h]); + w = detection_boxes_data[cy * cell_w + cx + 2 * cell_w * cell_h]; + h = detection_boxes_data[cy * cell_w + cx + 3 * cell_w * cell_h]; + label_id = dnn_detect_get_label_id(ctx->nb_classes, cell_w * cell_h, + detection_boxes_data + cy * cell_w + cx + 5 * cell_w * cell_h); + conf = conf * post_process_raw_data( + detection_boxes_data[cy * cell_w + cx + (label_id + 5) * cell_w * cell_h]); + } bbox = av_mallocz(sizeof(*bbox)); if (!bbox) @@ -404,6 +450,7 @@ static int dnn_detect_post_proc_ov(AVFrame *frame, DNNData *output, int nb_outpu if (ret < 0) return ret; case DDMT_YOLOV3: + case DDMT_YOLOV4: ret = dnn_detect_post_proc_yolov3(frame, output, filter_ctx, nb_outputs); if (ret < 0) return ret;