From patchwork Sat Apr 11 08:39:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18872 Delivered-To: andriy.gelman@gmail.com Received: by 2002:ad4:45d0:0:0:0:0:0 with SMTP id v16csp608244qvt; Sat, 11 Apr 2020 01:41:13 -0700 (PDT) X-Google-Smtp-Source: APiQypLx8y7uvJbTaaxL2qr6sqBOpM0l5evlaTnei3IUVdFLhsaQAtzu6NFoH5Zjw9erz6JmQQlc X-Received: by 2002:a1c:2506:: with SMTP id l6mr8623445wml.44.1586594473705; Sat, 11 Apr 2020 01:41:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586594473; cv=none; d=google.com; s=arc-20160816; b=Ff4BhgsiKivm5kEBMjZubEm+WufYiLdl0GC5RiapxU0oj21kjbLiZaMs7YAFOfMHwU A8M0CMp7y7K5q4Y8bR7gT3hBEebJg+l2i7hsOc8JUdz1WPY5BzdXYkcCUCdpHFNvZW9L xadcrAkOoAXDQkhIGlumEwOhN1cxelDtiVyVaviOlb33oeq8pO9tRMqgVb8vvRx6/Fvz kaKwsTXjQyZMDmfY2ngLR28XWyj7ad0a2jzNNggUxIsJ7oCX2W8hU1Jb5Vk5frZC16R2 dIleC1Pmmtqf9LhRa9QvMx71NMcs8foOJCpkXM/AcPnKk5u7zUuCV9VA4Mi0cNP7TywV PM9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:ironport-sdr :ironport-sdr:delivered-to; bh=Y/h0wP/VOrGqvoz/omlhgWTVEwbfdPlKXDe4bdz4CBs=; b=a4gdMvptUIWv6ZvKIf4Ng0S/73QjOmTtAN9kHZ+K+g3PXEy2hqsJkow3s13DwX6GV+ HIvs2hkxW/NJVj8rLLLyh9boatqv+V0eD4maWkaTUjN1z/L5i67yg4TTjq5dieZZ+F01 ua0x1Xe8fqNWx7wN6l5QLlw4X5mlYCoWfDQ2LuJyB9Us7OZElCPDVp7V+8b3fvtGmHX5 jGGkUpepTzooi9EogqDJm0XFhP6TBZMMDgOaaciK+I94OpWzKNxdnGZaEhRtdO6NRjFM P/NNMnyyWvOsFr9NOj0YT69y6CadcLlWmoyZbAUBJBbDb0U3ScHhqldPfVF4ip07m6um nrZw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z8si4569169wmc.146.2020.04.11.01.41.13; Sat, 11 Apr 2020 01:41:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8F0FB68B096; Sat, 11 Apr 2020 11:41:10 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4989468AAD8 for ; Sat, 11 Apr 2020 11:41:03 +0300 (EEST) IronPort-SDR: 9tH6XweLUMQCX5peQw+1X317FpIUdOMAwu9fC81H642oqu6puNUxfQzewSYscTzCQQDboTKfPS KTb3vcuNtXpw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Apr 2020 01:41:00 -0700 IronPort-SDR: dggZK/uIosEWYsyekd/A98DpRTUSn1KsfiHznpYx8x+X/yZWY6QID30xhWWimQRnwpVjpB5qJU GAN265ZZBX4Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,370,1580803200"; d="scan'208";a="426151464" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga005.jf.intel.com with ESMTP; 11 Apr 2020 01:40:59 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Sat, 11 Apr 2020 16:39:31 +0800 Message-Id: <1586594371-16799-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 1/6] dnn/native: add native support for add X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: nRweFpCl/SxW Content-Length: 7810 It can be tested with the model file generated with below python script: import tensorflow as tf import numpy as np import imageio in_img = imageio.imread('input.jpg') in_img = in_img.astype(np.float32)/255.0 in_data = in_img[np.newaxis, :] x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in') z1 = 0.039 + x z2 = x + 0.042 z3 = z1 + z2 z4 = z3 - 0.381 z5 = z4 - x y = tf.math.maximum(z5, 0.0, name='dnn_out') sess=tf.Session() sess.run(tf.global_variables_initializer()) graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ['dnn_out']) tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False) print("image_process.pb generated, please use \ path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n") output = sess.run(y, feed_dict={x: in_data}) imageio.imsave("out.jpg", np.squeeze(output)) Signed-off-by: Guo, Yejun --- libavfilter/dnn/dnn_backend_native_layer_mathbinary.c | 13 +++++++++++++ libavfilter/dnn/dnn_backend_native_layer_mathbinary.h | 1 + tools/python/convert_from_tensorflow.py | 15 +++++++-------- tools/python/convert_header.py | 2 +- 4 files changed, 22 insertions(+), 9 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c index 3b8bab8..3fe337f 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c @@ -107,6 +107,19 @@ int dnn_execute_layer_math_binary(DnnOperand *operands, const int32_t *input_ope } } return 0; + case DMBO_ADD: + if (params->input0_broadcast || params->input1_broadcast) { + for (int i = 0; i < dims_count; ++i) { + dst[i] = params->v + src[i]; + } + } else { + const DnnOperand *input1 = &operands[input_operand_indexes[1]]; + const float *src1 = input1->data; + for (int i = 0; i < dims_count; ++i) { + dst[i] = src[i] + src1[i]; + } + } + return 0; default: return -1; } diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h index 6b684d1..3c5bc6b 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h @@ -32,6 +32,7 @@ typedef enum { DMBO_SUB = 0, + DMBO_ADD = 1, DMBO_COUNT } DNNMathBinaryOperation; diff --git a/tools/python/convert_from_tensorflow.py b/tools/python/convert_from_tensorflow.py index 2485f16..9a495c0 100644 --- a/tools/python/convert_from_tensorflow.py +++ b/tools/python/convert_from_tensorflow.py @@ -71,7 +71,7 @@ class TFConverter: self.conv2d_scope_names = set() self.conv2d_scopename_inputname_dict = {} self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3, 'Maximum':4, 'MathBinary':5} - self.mathbin2code = {'Sub':0} + self.mathbin2code = {'Sub':0, 'Add':1} self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2} self.name_operand_dict = {} @@ -255,8 +255,7 @@ class TFConverter: np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f) - def dump_sub_to_file(self, node, f): - assert(node.op == 'Sub') + def dump_mathbinary_to_file(self, node, f): self.layer_number = self.layer_number + 1 self.converted_nodes.add(node.name) i0_node = self.name_node_dict[node.input[0]] @@ -264,15 +263,13 @@ class TFConverter: np.array([self.op2code['MathBinary'], self.mathbin2code[node.op]], dtype=np.uint32).tofile(f) if i0_node.op == 'Const': scalar = i0_node.attr['value'].tensor.float_val[0] - assert(i0_node.name.find('sub/x')) - np.array([1], dtype=np.uint32).tofile(f) + np.array([1], dtype=np.uint32).tofile(f) # broadcast: 1 np.array([scalar], dtype=np.float32).tofile(f) - np.array([0], dtype=np.uint32).tofile(f) + np.array([0], dtype=np.uint32).tofile(f) # broadcast: 0 input_operand_index = self.add_operand(i1_node.name, Operand.IOTYPE_INPUT) np.array([input_operand_index], dtype=np.uint32).tofile(f) elif i1_node.op == 'Const': scalar = i1_node.attr['value'].tensor.float_val[0] - assert(i1_node.name.find('sub/y')) np.array([0], dtype=np.uint32).tofile(f) input_operand_index = self.add_operand(i0_node.name, Operand.IOTYPE_INPUT) np.array([input_operand_index], dtype=np.uint32).tofile(f) @@ -309,7 +306,9 @@ class TFConverter: elif node.op == 'Maximum': self.dump_maximum_to_file(node, f) elif node.op == 'Sub': - self.dump_sub_to_file(node, f) + self.dump_mathbinary_to_file(node, f) + elif node.op == 'Add': + self.dump_mathbinary_to_file(node, f) def dump_operands_to_file(self, f): diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py index 6576fca..7027022 100644 --- a/tools/python/convert_header.py +++ b/tools/python/convert_header.py @@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE' major = 1 # increase minor when we don't have to re-convert the model file -minor = 1 +minor = 2 From patchwork Sat Apr 11 08:39:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18869 Delivered-To: andriy.gelman@gmail.com Received: by 2002:ad4:45d0:0:0:0:0:0 with SMTP id v16csp608344qvt; Sat, 11 Apr 2020 01:41:23 -0700 (PDT) X-Google-Smtp-Source: APiQypKTAjudWM8WhXdE6CkKvA49+YzjkLzWiUDAew9rbh9NMVavkju1XQ19H3NR+qJ5GauhZsuz X-Received: by 2002:adf:ee12:: with SMTP id y18mr8277462wrn.289.1586594482971; Sat, 11 Apr 2020 01:41:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586594482; cv=none; d=google.com; s=arc-20160816; b=wep7d7zrrc15MxyC7g4f7xpXcgdtw8iRDGUUcKpnr8t9k3Fv6JffCyA6lKXiakw67P YSpRitsOuxq6MW/Rp3kH5cLQcmHZAzweqmmK1KpGzfHMEBlsMtQ7F8f7vbW350dIkjcN pZ10wHVonMx4yZBc70CHuZuFsRrc7Eb1TOKEDi9tDX9N8agNKbfYpsA21+rWeURvwsKZ JLd38wlTfOawaJLO+QMRyGCDtTxRNCN/uXWK2V/qOh0xxdYF1JfWWCMrgMHv0PjjplBW u2gBLbJ/WN5wGgm/qmB/tfXIvZsi/1YkI7IDcgUNEhs0rjBrroqU7rRFwPllhOtDz+Zc hDUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:ironport-sdr :ironport-sdr:delivered-to; bh=KpnC5r+/NXTti50sxo16ZjChX5oUcNErTctLHJ6KFXU=; b=0gt/pAI7k4EEkWqJwIQ+zESSS7LkMsitZE2NFFuMFNYDzWM6OTrj3MJ5i4AltUehNJ j8kthPXdzEXSc2mhd6B4o074lp53SXd7AiuKSZnnImLq+yBcVxDjbThkpUjnYkzsaVp3 Xqd8i7u+e+0v73zY/AzJKoCkvB2oN/LajW1vJiqH8OnbCCEjIXs99C+fewZz39SbEIii 2ZBEJi39pxwRS4DFmAgCAwEucBsxEm5hzcU2DciwqlFWfZCI9ZEC+guF0uh9aKljikM8 tn8ZyZ11IftyGqYbT/i2mH+XMsF4FxMqVnHa2QnhTMoqQCqoLjgx3+yvlXsG2+2/0BVJ p3nw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t8si4470799wru.164.2020.04.11.01.41.22; Sat, 11 Apr 2020 01:41:22 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C6CAE68B0C5; Sat, 11 Apr 2020 11:41:18 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DC43C68AFF5 for ; Sat, 11 Apr 2020 11:41:11 +0300 (EEST) IronPort-SDR: UUbZ4kOKMshtgyltP02HENXaA2Wt4xUPKU+LhRBppxDFTJSy3Z1xrHKzMVyrlJ01vQsR6foOFM 0khWw2+813xg== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Apr 2020 01:41:10 -0700 IronPort-SDR: g+23mCmZ7c2TgxweTZ23ThTJiI0VHY6Mkmn0WH6G2wGD98UO9zEnltgUV/F6GWMjwGsdgZy9IU hRKPT0UvjMlw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,370,1580803200"; d="scan'208";a="276413592" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by fmsmga004.fm.intel.com with ESMTP; 11 Apr 2020 01:41:09 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Sat, 11 Apr 2020 16:39:41 +0800 Message-Id: <1586594381-16845-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 2/6] dnn-layer-mathbinary-test: add unit test for add X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ENx+Zv0DLfOJ Content-Length: 6987 Signed-off-by: Guo, Yejun --- tests/dnn/dnn-layer-mathbinary-test.c | 55 ++++++++++++++++++++++++----------- 1 file changed, 38 insertions(+), 17 deletions(-) diff --git a/tests/dnn/dnn-layer-mathbinary-test.c b/tests/dnn/dnn-layer-mathbinary-test.c index 1243784..fd8037f 100644 --- a/tests/dnn/dnn-layer-mathbinary-test.c +++ b/tests/dnn/dnn-layer-mathbinary-test.c @@ -22,10 +22,25 @@ #include #include #include "libavfilter/dnn/dnn_backend_native_layer_mathbinary.h" +#include "libavutil/avassert.h" #define EPSON 0.00001 -static int test_sub_broadcast_input0(void) +static float get_expected(float f1, float f2, DNNMathBinaryOperation op) +{ + switch (op) + { + case DMBO_SUB: + return f1 - f2; + case DMBO_ADD: + return f1 + f2; + default: + av_assert0(!"not supported yet"); + return 0.f; + } +} + +static int test_broadcast_input0(DNNMathBinaryOperation op) { DnnLayerMathBinaryParams params; DnnOperand operands[2]; @@ -35,7 +50,7 @@ static int test_sub_broadcast_input0(void) }; float *output; - params.bin_op = DMBO_SUB; + params.bin_op = op; params.input0_broadcast = 1; params.input1_broadcast = 0; params.v = 7.28; @@ -52,9 +67,10 @@ static int test_sub_broadcast_input0(void) output = operands[1].data; for (int i = 0; i < sizeof(input) / sizeof(float); i++) { - float expected_output = params.v - input[i]; + float expected_output = get_expected(params.v, input[i], op); if (fabs(output[i] - expected_output) > EPSON) { - printf("at index %d, output: %f, expected_output: %f\n", i, output[i], expected_output); + printf("op %d, at index %d, output: %f, expected_output: %f (%s:%d)\n", + op, i, output[i], expected_output, __FILE__, __LINE__); av_freep(&output); return 1; } @@ -64,7 +80,7 @@ static int test_sub_broadcast_input0(void) return 0; } -static int test_sub_broadcast_input1(void) +static int test_broadcast_input1(DNNMathBinaryOperation op) { DnnLayerMathBinaryParams params; DnnOperand operands[2]; @@ -74,7 +90,7 @@ static int test_sub_broadcast_input1(void) }; float *output; - params.bin_op = DMBO_SUB; + params.bin_op = op; params.input0_broadcast = 0; params.input1_broadcast = 1; params.v = 7.28; @@ -91,9 +107,10 @@ static int test_sub_broadcast_input1(void) output = operands[1].data; for (int i = 0; i < sizeof(input) / sizeof(float); i++) { - float expected_output = input[i] - params.v; + float expected_output = get_expected(input[i], params.v, op); if (fabs(output[i] - expected_output) > EPSON) { - printf("at index %d, output: %f, expected_output: %f\n", i, output[i], expected_output); + printf("op %d, at index %d, output: %f, expected_output: %f (%s:%d)\n", + op, i, output[i], expected_output, __FILE__, __LINE__); av_freep(&output); return 1; } @@ -103,7 +120,7 @@ static int test_sub_broadcast_input1(void) return 0; } -static int test_sub_no_broadcast(void) +static int test_no_broadcast(DNNMathBinaryOperation op) { DnnLayerMathBinaryParams params; DnnOperand operands[3]; @@ -116,7 +133,7 @@ static int test_sub_no_broadcast(void) }; float *output; - params.bin_op = DMBO_SUB; + params.bin_op = op; params.input0_broadcast = 0; params.input1_broadcast = 0; @@ -138,9 +155,10 @@ static int test_sub_no_broadcast(void) output = operands[2].data; for (int i = 0; i < sizeof(input0) / sizeof(float); i++) { - float expected_output = input0[i] - input1[i]; + float expected_output = get_expected(input0[i], input1[i], op); if (fabs(output[i] - expected_output) > EPSON) { - printf("at index %d, output: %f, expected_output: %f\n", i, output[i], expected_output); + printf("op %d, at index %d, output: %f, expected_output: %f (%s:%d)\n", + op, i, output[i], expected_output, __FILE__, __LINE__); av_freep(&output); return 1; } @@ -150,15 +168,15 @@ static int test_sub_no_broadcast(void) return 0; } -static int test_sub(void) +static int test(DNNMathBinaryOperation op) { - if (test_sub_broadcast_input0()) + if (test_broadcast_input0(op)) return 1; - if (test_sub_broadcast_input1()) + if (test_broadcast_input1(op)) return 1; - if (test_sub_no_broadcast()) + if (test_no_broadcast(op)) return 1; return 0; @@ -166,7 +184,10 @@ static int test_sub(void) int main(int argc, char **argv) { - if (test_sub()) + if (test(DMBO_SUB)) + return 1; + + if (test(DMBO_ADD)) return 1; return 0; From patchwork Sat Apr 11 08:39:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18870 Delivered-To: andriy.gelman@gmail.com Received: by 2002:ad4:45d0:0:0:0:0:0 with SMTP id v16csp608455qvt; Sat, 11 Apr 2020 01:41:33 -0700 (PDT) X-Google-Smtp-Source: APiQypIW3NxdvIKXVlPXTVF59ytvHNdR1tVp3vk7C/qe0p/ky7b9uG7Mi97+u5vlht18H5+oYYC1 X-Received: by 2002:a5d:4ecf:: with SMTP id s15mr9243993wrv.226.1586594493777; Sat, 11 Apr 2020 01:41:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586594493; cv=none; d=google.com; s=arc-20160816; b=iIEhRG2glSqf0mXAcCOBwdBarvFRuoubTRdYmv3JrQ5W10UGHA9EYanDZ0CCLhkj1T 3AESEpt1fGMaL0Tug4NWSqp+3+iIohEay2O+AgXyfE2pikklLNg23tW0Ozas4Eb+Q2w0 zFefOvZEuxqiu6z6JgzwNRDcu7OSCcsFAh/R0CXfXPOQxpNE4FNdae7k6y7YFejdEOW9 /Xdi2JEoRAX8bBK4UF5f5AeAhL2dWiiX+GiO6VeWbmPp8cPK8t8kChWBMNDFQuyuHbxb +9b3eXDiApzFeaJpqrrvF12QYq9fq7+MbT8eQv0Ln+JISAJnO0MaVUU06UPi331/nK0W co/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:ironport-sdr :ironport-sdr:delivered-to; bh=TWEXw5P3qxIrxjNKJD8rJmQh2U04TggbSlIiklHYIc8=; b=Mk5qgOnVfkKEPHfxuYtrc5HvMSq4mMic+8v7Scbs/6lVCnJNs/e2wYpEVYJDTDu3dF EdeN+MMSiNMibSj0kwNXiPOrkBV++kBLiGPxnf7+bT00Ec7PiK4pN3n0WqdAIIFf+nfD /2BdlRVfA6nf8+7Uky5Vksg/Yu/NJ0oHhe9cNiV2wPKQCVrNPjbids65zGHIeKTK4MPj +bx34UIW9oH2j23x8OR1zJ5GOkWpywz264h7rARLpRzYOVN1gQ/sJ22+T1CLKmRG3iqg g4xmuvSYjNHs3mvdUs7jsNNLq0ViB0Y2DoibfCyzcWkXzATC/Cr2aI3dfK45+j9vdUsc juPA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i127si4611755wma.83.2020.04.11.01.41.33; Sat, 11 Apr 2020 01:41:33 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CB48968B096; Sat, 11 Apr 2020 11:41:29 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7B9EA68ABB7 for ; Sat, 11 Apr 2020 11:41:22 +0300 (EEST) IronPort-SDR: JEsqncp6osEwvzbARO3bBCqZGJkNl/w3PesxYB7y4/N1S3z5gXV7cMhdHW50iPBtnU9wJWRMNM /cO1JHxgH9Qw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Apr 2020 01:41:20 -0700 IronPort-SDR: HolWEqGKPwwcXl+NZrZpc5xbuqwkFU+vwEJ2G0PKewWfYDsV1324OmqdprnqRxKi3V7+E9xHzQ F5AX3wY8yDjw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,370,1580803200"; d="scan'208";a="426151558" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga005.jf.intel.com with ESMTP; 11 Apr 2020 01:41:19 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Sat, 11 Apr 2020 16:39:51 +0800 Message-Id: <1586594391-16891-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 3/6] dnn/native: add native support for mul X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jBxRMULLeW02 Content-Length: 5585 it can be tested with model file generated from above python script: import tensorflow as tf import numpy as np import imageio in_img = imageio.imread('input.jpg') in_img = in_img.astype(np.float32)/255.0 in_data = in_img[np.newaxis, :] x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in') z1 = 0.5 + 0.3 * x z2 = z1 * 4 z3 = z2 - x - 2.0 y = tf.identity(z3, name='dnn_out') sess=tf.Session() sess.run(tf.global_variables_initializer()) graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ['dnn_out']) tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False) print("image_process.pb generated, please use \ path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n") output = sess.run(y, feed_dict={x: in_data}) imageio.imsave("out.jpg", np.squeeze(output)) Signed-off-by: Guo, Yejun --- libavfilter/dnn/dnn_backend_native_layer_mathbinary.c | 13 +++++++++++++ libavfilter/dnn/dnn_backend_native_layer_mathbinary.h | 1 + tools/python/convert_from_tensorflow.py | 4 +++- tools/python/convert_header.py | 2 +- 4 files changed, 18 insertions(+), 2 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c index 3fe337f..222941e 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c @@ -120,6 +120,19 @@ int dnn_execute_layer_math_binary(DnnOperand *operands, const int32_t *input_ope } } return 0; + case DMBO_MUL: + if (params->input0_broadcast || params->input1_broadcast) { + for (int i = 0; i < dims_count; ++i) { + dst[i] = params->v * src[i]; + } + } else { + const DnnOperand *input1 = &operands[input_operand_indexes[1]]; + const float *src1 = input1->data; + for (int i = 0; i < dims_count; ++i) { + dst[i] = src[i] * src1[i]; + } + } + return 0; default: return -1; } diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h index 3c5bc6b..d58b48c 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h @@ -33,6 +33,7 @@ typedef enum { DMBO_SUB = 0, DMBO_ADD = 1, + DMBO_MUL = 2, DMBO_COUNT } DNNMathBinaryOperation; diff --git a/tools/python/convert_from_tensorflow.py b/tools/python/convert_from_tensorflow.py index 9a495c0..dc3b4e3 100644 --- a/tools/python/convert_from_tensorflow.py +++ b/tools/python/convert_from_tensorflow.py @@ -71,7 +71,7 @@ class TFConverter: self.conv2d_scope_names = set() self.conv2d_scopename_inputname_dict = {} self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3, 'Maximum':4, 'MathBinary':5} - self.mathbin2code = {'Sub':0, 'Add':1} + self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2} self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2} self.name_operand_dict = {} @@ -309,6 +309,8 @@ class TFConverter: self.dump_mathbinary_to_file(node, f) elif node.op == 'Add': self.dump_mathbinary_to_file(node, f) + elif node.op == 'Mul': + self.dump_mathbinary_to_file(node, f) def dump_operands_to_file(self, f): diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py index 7027022..87899fe 100644 --- a/tools/python/convert_header.py +++ b/tools/python/convert_header.py @@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE' major = 1 # increase minor when we don't have to re-convert the model file -minor = 2 +minor = 3 From patchwork Sat Apr 11 08:40:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18868 Delivered-To: andriy.gelman@gmail.com Received: by 2002:ad4:45d0:0:0:0:0:0 with SMTP id v16csp608538qvt; Sat, 11 Apr 2020 01:41:43 -0700 (PDT) X-Google-Smtp-Source: APiQypLl/axRkj6BXG9Kqnx9vsiplWB9ap/6LL3PqCKPiL7RrTDz7DY1fIbGX+Tc6G4j4CbvLBll X-Received: by 2002:adf:e991:: with SMTP id h17mr8494916wrm.17.1586594503172; Sat, 11 Apr 2020 01:41:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586594503; cv=none; d=google.com; s=arc-20160816; b=lW/sCPkD7M3XdGfBe9ExUg4aXFdfTaLbmjtM/RN5pw44kCfEDS7bpiHnwkjheImoIb dBbWVt7yD5lao/Ckdv/D2jlsxiLpWux0js9RISbeEjvHUFhOpC4dR4z1qyTTHK8vJo4c rEGbdGdP12TXy2dC2wrkRo58mhsCpRHl/j0sKvnFWNRIlR8kMhKUQoOf80djDLOme2N7 DxkqFbqcf35Uk2FnpWaykKKVODv7fybaiHYipQqIRcNfEFzPt7gk2N91JTIltAKvnHnq obC9WvCG3Xvbe+6Jji16A5LPEWT036Kyq7DDb/rh8izVsJbZUSGL1X0MyeT+s8r1260G OA6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:ironport-sdr :ironport-sdr:delivered-to; bh=RJ4qm+Js7PtGptMPai2dA5bcalUZrgpR3TVDzdUM62M=; b=VR6BHCUrvyCGLvJIu/cFN9gEaVv6R5tYtZBk1P1bB1rVJbFosRJfBBh1zNw+NOJlDJ 6rda2Mmm5CSC5CSn7aXjzHQk4ctLwD/g/UtGYlavqq2jLeLwt14cJCdUgKSyKI5zyZRt 9yIAcCS6sKy9rnH7WSpHSyaXt3+ft4UX7Cxz0Msm2hhsf1NsYnccAPEuMg6yIueVkOKN pca76N2mZG7VQPCqmkWfG1+pEIiIiuYTlv1/F0dGDr3gqdnXRBmtVuM3+4xNOeei7o63 zEj6P9ug7u2b/s3L2Vr8yfwMGS1mgrcvSgGsc/la+CKXCz/Lt4Pr1lXPkpHihifUWEKt JPqw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z7si4297163wru.344.2020.04.11.01.41.42; Sat, 11 Apr 2020 01:41:43 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BD2E768B154; Sat, 11 Apr 2020 11:41:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0BCE868B0AD for ; Sat, 11 Apr 2020 11:41:30 +0300 (EEST) IronPort-SDR: gVM7IlQ2/89iRytiu2EaHdFl/gNzHWGhyuBrxZ3F+5Oufz8+zZS8qmDt8kKnlxu5kU8nkJJvJf nIewMIvhw53w== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Apr 2020 01:41:29 -0700 IronPort-SDR: CFt9eN4PczORKOCtz7GAumqCt7c18Kkv8xgg6/egu0DGxgBV4ksJWEDKGM6bIV02hzl+XAFER1 /kl8u9A8Kl5A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,370,1580803200"; d="scan'208";a="255704378" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga006.jf.intel.com with ESMTP; 11 Apr 2020 01:41:28 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Sat, 11 Apr 2020 16:40:00 +0800 Message-Id: <1586594400-16937-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 4/6] dnn-layer-mathbinary-test: add unit test for mul X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jiNq4HbIrfUs Content-Length: 1423 Signed-off-by: Guo, Yejun --- tests/dnn/dnn-layer-mathbinary-test.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/dnn/dnn-layer-mathbinary-test.c b/tests/dnn/dnn-layer-mathbinary-test.c index fd8037f..3b0f0c5 100644 --- a/tests/dnn/dnn-layer-mathbinary-test.c +++ b/tests/dnn/dnn-layer-mathbinary-test.c @@ -34,6 +34,8 @@ static float get_expected(float f1, float f2, DNNMathBinaryOperation op) return f1 - f2; case DMBO_ADD: return f1 + f2; + case DMBO_MUL: + return f1 * f2; default: av_assert0(!"not supported yet"); return 0.f; @@ -190,5 +192,8 @@ int main(int argc, char **argv) if (test(DMBO_ADD)) return 1; + if (test(DMBO_MUL)) + return 1; + return 0; } From patchwork Sat Apr 11 08:40:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18871 Delivered-To: andriy.gelman@gmail.com Received: by 2002:ad4:45d0:0:0:0:0:0 with SMTP id v16csp608643qvt; Sat, 11 Apr 2020 01:41:56 -0700 (PDT) X-Google-Smtp-Source: APiQypJ0qyRXRo8mY/3VBIuYX665igeQyw3z23I267FhAsKvN5EzZY4Ianrv9pZXhKZCRcPD7cHw X-Received: by 2002:a1c:ed1a:: with SMTP id l26mr8898955wmh.175.1586594515937; Sat, 11 Apr 2020 01:41:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586594515; cv=none; d=google.com; s=arc-20160816; b=sJVEXyB0wABoprxXea2bJRJHEd2bBFqGvjzdLwlzrrPsMXv32qIMJBJFuNHqj1gNHP 61j0/sWAgqmo3CNrxGTPA19hDxXXpNBYAc6NotRZ2VNhA+cxtvCrn5G84cN7o359BzT9 zmdRq1QSyWYLpwPgbn8BTqY6LgQ7HkXVtnVDJr/kRx5JRFsOGGvfCisHfjgb6zMi/KXb DHmP0InTGJQMV8Ow7cr4GBZ1/NoTaWGXl42aDN0sFMQAVet6+yk8OD6fLMVEWZh6B0ee SaO8vi4HdDDZzC/xk0T3quWtqxjiS3puNN+EpTcuGt5/oA3iHLSO3PuBYBvrZGca2THK ty0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:ironport-sdr :ironport-sdr:delivered-to; bh=ToB5n3uxnmUe+T6lK7NxqfQTFBLGNUuB0yo94wvUpk4=; b=s7OuGdaDCKLud8N2NcqOpz5dneiXI0gAIXpqjcw28HI38IxthMsip3gG4Qu2NUkrUT D2gQxRP1h1TlQyUD63Znvtlvg/zBD+xcx1jg6txoMp80nhTlxo9MMrUtoZ9aIFQBHv0w RdmKh8nMK5ZAOloNP4DCTlosTgRaiBzm3JaiwX1qIp6b6hdd+7f5qvCZ5w30CeahHFSG yfeUYlMfdVJgsKMcRPEUlyjbq+WDLCgPPKEIheBlYygPkX+VR304ZK6fZRWX6KGvW3jN bHqGCsOe9zec8CDdBi5UOoFqyZdZDmD4OcGs0Qmi6LxW9A0hteRjeRzfUWz+c2z4Jaq7 sswA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z15si4287781wro.162.2020.04.11.01.41.55; Sat, 11 Apr 2020 01:41:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C3BD968B0C5; Sat, 11 Apr 2020 11:41:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A498B68B0C5 for ; Sat, 11 Apr 2020 11:41:42 +0300 (EEST) IronPort-SDR: BRQ/N8aKzKfu1qB4YYPjtFFicHDyIU3ghwxui0NMRG93CEK88ZGIpbwzcTG4DHd6qTerIXufVl oOOmsvZ6XtHQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Apr 2020 01:41:41 -0700 IronPort-SDR: KyBGFfcT3AlqpaMbtUuetCkyTfplURREweJIQxLV2kUNgTk4yoeojj0mQYwJFjUJH+zfy1EfXe 9u/pv+G1bIYA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,370,1580803200"; d="scan'208";a="399137766" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga004.jf.intel.com with ESMTP; 11 Apr 2020 01:41:39 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Sat, 11 Apr 2020 16:40:11 +0800 Message-Id: <1586594411-17005-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 5/6] dnn/native: add native support for divide X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ubjYCke4XJ+X Content-Length: 5954 it can be tested with model file generated with below python script: import tensorflow as tf import numpy as np import imageio in_img = imageio.imread('input.jpg') in_img = in_img.astype(np.float32)/255.0 in_data = in_img[np.newaxis, :] x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in') z1 = 2 / x z2 = 1 / z1 z3 = z2 / 0.25 + 0.3 z4 = z3 - x * 1.5 - 0.3 y = tf.identity(z4, name='dnn_out') sess=tf.Session() sess.run(tf.global_variables_initializer()) graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ['dnn_out']) tf.train.write_graph(graph_def, '.', 'image_process.pb', as_text=False) print("image_process.pb generated, please use \ path_to_ffmpeg/tools/python/convert.py to generate image_process.model\n") output = sess.run(y, feed_dict={x: in_data}) imageio.imsave("out.jpg", np.squeeze(output)) Signed-off-by: Guo, Yejun --- libavfilter/dnn/dnn_backend_native_layer_mathbinary.c | 17 +++++++++++++++++ libavfilter/dnn/dnn_backend_native_layer_mathbinary.h | 1 + tools/python/convert_from_tensorflow.py | 5 +++-- tools/python/convert_header.py | 2 +- 4 files changed, 22 insertions(+), 3 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c index 222941e..c32a042 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c @@ -133,6 +133,23 @@ int dnn_execute_layer_math_binary(DnnOperand *operands, const int32_t *input_ope } } return 0; + case DMBO_REALDIV: + if (params->input0_broadcast) { + for (int i = 0; i < dims_count; ++i) { + dst[i] = params->v / src[i]; + } + } else if (params->input1_broadcast) { + for (int i = 0; i < dims_count; ++i) { + dst[i] = src[i] / params->v; + } + } else { + const DnnOperand *input1 = &operands[input_operand_indexes[1]]; + const float *src1 = input1->data; + for (int i = 0; i < dims_count; ++i) { + dst[i] = src[i] / src1[i]; + } + } + return 0; default: return -1; } diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h index d58b48c..2ffbb66 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h @@ -34,6 +34,7 @@ typedef enum { DMBO_SUB = 0, DMBO_ADD = 1, DMBO_MUL = 2, + DMBO_REALDIV = 3, DMBO_COUNT } DNNMathBinaryOperation; diff --git a/tools/python/convert_from_tensorflow.py b/tools/python/convert_from_tensorflow.py index dc3b4e3..a0fdad2 100644 --- a/tools/python/convert_from_tensorflow.py +++ b/tools/python/convert_from_tensorflow.py @@ -71,7 +71,7 @@ class TFConverter: self.conv2d_scope_names = set() self.conv2d_scopename_inputname_dict = {} self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3, 'Maximum':4, 'MathBinary':5} - self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2} + self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3} self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2} self.name_operand_dict = {} @@ -311,7 +311,8 @@ class TFConverter: self.dump_mathbinary_to_file(node, f) elif node.op == 'Mul': self.dump_mathbinary_to_file(node, f) - + elif node.op == 'RealDiv': + self.dump_mathbinary_to_file(node, f) def dump_operands_to_file(self, f): operands = sorted(self.name_operand_dict.values()) diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py index 87899fe..75d1ce8 100644 --- a/tools/python/convert_header.py +++ b/tools/python/convert_header.py @@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE' major = 1 # increase minor when we don't have to re-convert the model file -minor = 3 +minor = 4 From patchwork Sat Apr 11 08:40:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18866 Delivered-To: andriy.gelman@gmail.com Received: by 2002:ad4:45d0:0:0:0:0:0 with SMTP id v16csp608743qvt; Sat, 11 Apr 2020 01:42:06 -0700 (PDT) X-Google-Smtp-Source: APiQypIR706MYNA+QtkwS8RF21TSwBs2W4gmI/DbWoYmZzYrENs/ExM02LVWgmMxfF/WYdRR3OWi X-Received: by 2002:a1c:f60c:: with SMTP id w12mr8818307wmc.29.1586594525964; Sat, 11 Apr 2020 01:42:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586594525; cv=none; d=google.com; s=arc-20160816; b=rWxX50YYNz9FAOeYLsuNmVqzfjNw1FIhCpeSkgCqM5pMJ+A42j5d+usvDAGAjBszj/ t9+4NjrIxkZfxeNUx51kAGOMZmOCQAyvJQcDDIwLNZCuxEVZxCozTnJ+nnh9eC2MvgFI ETI4jxneSL1+UdI/XFCbvIIqQXSGFG1x19iWUlWbDkHI57aQk2RmASp2jyVitJm+yENr suU5U4x0kJUrLzPF+cEP3JbnuC4bAvtSjkJlTr7x4//90520bHSCDx/M6shTrHCNY+Ij dUrXe7RMdJTeaJXm1b491cCU5clQmDBmRQG9Z/DClEjFQ8hg7Wf+fsWV9xTlI9OpAQz3 aoDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:ironport-sdr :ironport-sdr:delivered-to; bh=UiThMIIAhdv379hH6+wxnJJYIhVtSuvsCPK4VdtUVjo=; b=v7dMpTPaGUxc0JM8ADTssNJDKIFtOaz29lF8Hn0uAkl1mVwxjYxR5HkUo8TX0mSeOT 7LOcHXcZ7rS0+BG5AvkfnwmRDqci3Yoz8i7pTIDZTYu80KLzOyN7u5zkkvMyghqSTTGt QD++ELt3TlQtjJX8Ei6u7Hq1GxSBm4aOpIJvSkascBIa7HOAMlJlnA0v6R3383TQPk+o 8nuN8mKUK69FIlls+k4/wW5iFVwz7VwbDoMnsdkNJQYDdxJ1MIP/xbvJO6mn+Blq86qX KUm/oJVNiIrZ5P3tQhq290zPLCZDySyCk2Bk2aRmAz1hMjv6n3xgFNtA6Srwi1JnAGCy lZmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s125si4462816wmf.124.2020.04.11.01.42.05; Sat, 11 Apr 2020 01:42:05 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C660A68B1E3; Sat, 11 Apr 2020 11:41:53 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ABF4868B0AD for ; Sat, 11 Apr 2020 11:41:50 +0300 (EEST) IronPort-SDR: TkNamp7eBvxsEHdEpfZaH1ro3FaEXEq4ED9MPOgASYi8FRJG1+N+hZFUshpo2noTAE1+Gfk5zh knin+ACLuz3A== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Apr 2020 01:41:49 -0700 IronPort-SDR: 2kXTWFsZ9wp47zkW759awsx8ghl6XZx8Ro1A/lVWtTMTt5cFpHgZ8MSxMOpB9V4JBMlZU4uyrY SGXLNdDmfKRQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,370,1580803200"; d="scan'208";a="252417071" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga003.jf.intel.com with ESMTP; 11 Apr 2020 01:41:48 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Sat, 11 Apr 2020 16:40:19 +0800 Message-Id: <1586594419-17053-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 6/6] dnn-layer-mathbinary-test: add unit test for divide X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: /CbVcsV9VzgY Content-Length: 1431 Signed-off-by: Guo, Yejun --- tests/dnn/dnn-layer-mathbinary-test.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/dnn/dnn-layer-mathbinary-test.c b/tests/dnn/dnn-layer-mathbinary-test.c index 3b0f0c5..f82d15b 100644 --- a/tests/dnn/dnn-layer-mathbinary-test.c +++ b/tests/dnn/dnn-layer-mathbinary-test.c @@ -36,6 +36,8 @@ static float get_expected(float f1, float f2, DNNMathBinaryOperation op) return f1 + f2; case DMBO_MUL: return f1 * f2; + case DMBO_REALDIV: + return f1 / f2; default: av_assert0(!"not supported yet"); return 0.f; @@ -195,5 +197,8 @@ int main(int argc, char **argv) if (test(DMBO_MUL)) return 1; + if (test(DMBO_REALDIV)) + return 1; + return 0; }