From patchwork Mon Mar 23 04:08:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18361 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 42B2744B13D for ; Mon, 23 Mar 2020 06:09:05 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 14F3F68B680; Mon, 23 Mar 2020 06:09:05 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B870C68B679 for ; Mon, 23 Mar 2020 06:08:57 +0200 (EET) IronPort-SDR: rq/DO0AuvQJ93RVBiNH30INGGMwjWR9Qz8joXA/Z7qMAKrbz/swqCkFtN1WVuySFW9eYPlrvb8 u2XEXW6j26Lw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Mar 2020 21:08:55 -0700 IronPort-SDR: U6hcTZ1TqjlrceQpiJcdSoxKb6U2vwtEJSiaInua+JqEkr1knw9DepqNvMR9QD3Hr2+S7PqI6E 8ylB811nSmkQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,295,1580803200"; d="scan'208";a="392781089" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga004.jf.intel.com with ESMTP; 22 Mar 2020 21:08:54 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Mar 2020 12:08:00 +0800 Message-Id: <1584936480-9033-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 1/4] avfilter/vf_dnn_processing.c: fix typo for the linesize of dnn data X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Guo, Yejun --- libavfilter/vf_dnn_processing.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c index fabe4f1..cf589ac 100644 --- a/libavfilter/vf_dnn_processing.c +++ b/libavfilter/vf_dnn_processing.c @@ -329,7 +329,7 @@ static int copy_from_frame_to_dnn(DnnProcessingContext *ctx, const AVFrame *fram if (dnn_input->dt == DNN_FLOAT) { sws_scale(ctx->sws_gray8_to_grayf32, (const uint8_t **)frame->data, frame->linesize, 0, frame->height, (uint8_t * const*)(&dnn_input->data), - (const int [4]){frame->linesize[0] * sizeof(float), 0, 0, 0}); + (const int [4]){frame->width * 3 * sizeof(float), 0, 0, 0}); } else { av_assert0(dnn_input->dt == DNN_UINT8); av_image_copy_plane(dnn_input->data, bytewidth, @@ -369,7 +369,7 @@ static int copy_from_dnn_to_frame(DnnProcessingContext *ctx, AVFrame *frame) case AV_PIX_FMT_BGR24: if (dnn_output->dt == DNN_FLOAT) { sws_scale(ctx->sws_grayf32_to_gray8, (const uint8_t *[4]){(const uint8_t *)dnn_output->data, 0, 0, 0}, - (const int[4]){frame->linesize[0] * sizeof(float), 0, 0, 0}, + (const int[4]){frame->width * 3 * sizeof(float), 0, 0, 0}, 0, frame->height, (uint8_t * const*)frame->data, frame->linesize); } else { From patchwork Mon Mar 23 04:08:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18362 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 6F58C44B13D for ; Mon, 23 Mar 2020 06:09:13 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5465868B6A5; Mon, 23 Mar 2020 06:09:13 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EFCAD68B6A3 for ; Mon, 23 Mar 2020 06:09:05 +0200 (EET) IronPort-SDR: tc3EwIhPFtw8FBjlX+vByP47Sk0T6S/n/Uxdht8Cx3oOfkWV1wURDdOH4Aw4/C5lwUZqU4xmfr bpyp2+1Ubouw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Mar 2020 21:09:03 -0700 IronPort-SDR: jmN9+DigxUP8/zHyGqBC/D0DmKGdCrHzBAcjsIxophaU79S8BO8ixENj3AzVlWYwcuxROVhhlW IxwXGgY2jsmQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,295,1580803200"; d="scan'208";a="249489344" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga006.jf.intel.com with ESMTP; 22 Mar 2020 21:09:02 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Mar 2020 12:08:08 +0800 Message-Id: <1584936488-9079-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 2/4] dnn_backend_native_layer_mathbinary: add sub support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" more math binary operations will be added here Signed-off-by: Guo, Yejun --- libavfilter/dnn/Makefile | 1 + libavfilter/dnn/dnn_backend_native.h | 1 + .../dnn/dnn_backend_native_layer_mathbinary.c | 113 +++++++++++++++++++++ .../dnn/dnn_backend_native_layer_mathbinary.h | 49 +++++++++ libavfilter/dnn/dnn_backend_native_layers.c | 2 + tools/python/convert_from_tensorflow.py | 55 +++++++++- tools/python/convert_header.py | 2 +- 7 files changed, 219 insertions(+), 4 deletions(-) create mode 100644 libavfilter/dnn/dnn_backend_native_layer_mathbinary.c create mode 100644 libavfilter/dnn/dnn_backend_native_layer_mathbinary.h diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile index 171f00e..ce52958 100644 --- a/libavfilter/dnn/Makefile +++ b/libavfilter/dnn/Makefile @@ -5,6 +5,7 @@ OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layer_pad OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layer_conv2d.o OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layer_depth2space.o OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layer_maximum.o +OBJS-$(CONFIG_DNN) += dnn/dnn_backend_native_layer_mathbinary.o DNN-OBJS-$(CONFIG_LIBTENSORFLOW) += dnn/dnn_backend_tf.o diff --git a/libavfilter/dnn/dnn_backend_native.h b/libavfilter/dnn/dnn_backend_native.h index 53ed22c..5d76d87 100644 --- a/libavfilter/dnn/dnn_backend_native.h +++ b/libavfilter/dnn/dnn_backend_native.h @@ -41,6 +41,7 @@ typedef enum { DLT_DEPTH_TO_SPACE = 2, DLT_MIRROR_PAD = 3, DLT_MAXIMUM = 4, + DLT_MATH_BINARY = 5, DLT_COUNT } DNNLayerType; diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c new file mode 100644 index 0000000..3b8bab8 --- /dev/null +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.c @@ -0,0 +1,113 @@ +/* + * Copyright (c) 2020 + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * DNN native backend implementation. + */ + +#include "dnn_backend_native.h" +#include "libavutil/avassert.h" +#include "dnn_backend_native_layer_mathbinary.h" + +int dnn_load_layer_math_binary(Layer *layer, AVIOContext *model_file_context, int file_size) +{ + DnnLayerMathBinaryParams *params; + int dnn_size = 0; + int input_index = 0; + params = av_malloc(sizeof(*params)); + if (!params) + return 0; + + params->bin_op = (int32_t)avio_rl32(model_file_context); + dnn_size += 4; + + params->input0_broadcast = (int32_t)avio_rl32(model_file_context); + dnn_size += 4; + if (params->input0_broadcast) { + params->v = av_int2float(avio_rl32(model_file_context)); + } else { + layer->input_operand_indexes[input_index] = (int32_t)avio_rl32(model_file_context); + input_index++; + } + dnn_size += 4; + + params->input1_broadcast = (int32_t)avio_rl32(model_file_context); + dnn_size += 4; + if (params->input1_broadcast) { + params->v = av_int2float(avio_rl32(model_file_context)); + } else { + layer->input_operand_indexes[input_index] = (int32_t)avio_rl32(model_file_context); + input_index++; + } + dnn_size += 4; + + layer->output_operand_index = (int32_t)avio_rl32(model_file_context); + dnn_size += 4; + layer->params = params; + + return dnn_size; +} + +int dnn_execute_layer_math_binary(DnnOperand *operands, const int32_t *input_operand_indexes, + int32_t output_operand_index, const void *parameters) +{ + const DnnOperand *input = &operands[input_operand_indexes[0]]; + DnnOperand *output = &operands[output_operand_index]; + const DnnLayerMathBinaryParams *params = (const DnnLayerMathBinaryParams *)parameters; + int dims_count; + const float *src; + float *dst; + + for (int i = 0; i < 4; ++i) + output->dims[i] = input->dims[i]; + + output->data_type = input->data_type; + output->length = calculate_operand_data_length(output); + output->data = av_realloc(output->data, output->length); + if (!output->data) + return DNN_ERROR; + + dims_count = calculate_operand_dims_count(output); + src = input->data; + dst = output->data; + + switch (params->bin_op) { + case DMBO_SUB: + if (params->input0_broadcast) { + for (int i = 0; i < dims_count; ++i) { + dst[i] = params->v - src[i]; + } + } else if (params->input1_broadcast) { + for (int i = 0; i < dims_count; ++i) { + dst[i] = src[i] - params->v; + } + } else { + const DnnOperand *input1 = &operands[input_operand_indexes[1]]; + const float *src1 = input1->data; + for (int i = 0; i < dims_count; ++i) { + dst[i] = src[i] - src1[i]; + } + } + return 0; + default: + return -1; + } +} diff --git a/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h new file mode 100644 index 0000000..6b684d1 --- /dev/null +++ b/libavfilter/dnn/dnn_backend_native_layer_mathbinary.h @@ -0,0 +1,49 @@ +/* + * Copyright (c) 2020 + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * DNN inference functions interface for native backend. + */ + + +#ifndef AVFILTER_DNN_DNN_BACKEND_NATIVE_LAYER_MATHBINARY_H +#define AVFILTER_DNN_DNN_BACKEND_NATIVE_LAYER_MATHBINARY_H + +#include "libavformat/avio.h" +#include "dnn_backend_native.h" + +typedef enum { + DMBO_SUB = 0, + DMBO_COUNT +} DNNMathBinaryOperation; + +typedef struct DnnLayerMathBinaryParams{ + DNNMathBinaryOperation bin_op; + int input0_broadcast; + int input1_broadcast; + float v; +} DnnLayerMathBinaryParams; + +int dnn_load_layer_math_binary(Layer *layer, AVIOContext *model_file_context, int file_size); +int dnn_execute_layer_math_binary(DnnOperand *operands, const int32_t *input_operand_indexes, + int32_t output_operand_index, const void *parameters); + +#endif diff --git a/libavfilter/dnn/dnn_backend_native_layers.c b/libavfilter/dnn/dnn_backend_native_layers.c index d659667..af18552 100644 --- a/libavfilter/dnn/dnn_backend_native_layers.c +++ b/libavfilter/dnn/dnn_backend_native_layers.c @@ -24,6 +24,7 @@ #include "dnn_backend_native_layer_conv2d.h" #include "dnn_backend_native_layer_depth2space.h" #include "dnn_backend_native_layer_maximum.h" +#include "dnn_backend_native_layer_mathbinary.h" LayerFunc layer_funcs[DLT_COUNT] = { {NULL, NULL}, @@ -31,4 +32,5 @@ LayerFunc layer_funcs[DLT_COUNT] = { {dnn_execute_layer_depth2space, dnn_load_layer_depth2space}, {dnn_execute_layer_pad, dnn_load_layer_pad}, {dnn_execute_layer_maximum, dnn_load_layer_maximum}, + {dnn_execute_layer_math_binary, dnn_load_layer_math_binary}, }; diff --git a/tools/python/convert_from_tensorflow.py b/tools/python/convert_from_tensorflow.py index 5e87e22..2485f16 100644 --- a/tools/python/convert_from_tensorflow.py +++ b/tools/python/convert_from_tensorflow.py @@ -70,7 +70,8 @@ class TFConverter: self.converted_nodes = set() self.conv2d_scope_names = set() self.conv2d_scopename_inputname_dict = {} - self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3, 'Maximum':4} + self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3, 'Maximum':4, 'MathBinary':5} + self.mathbin2code = {'Sub':0} self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2} self.name_operand_dict = {} @@ -113,6 +114,8 @@ class TFConverter: # if activation is None, and BiasAdd.next is the last op which is Identity if conv2d_scope_name + '/BiasAdd' in self.edges: anode = self.edges[conv2d_scope_name + '/BiasAdd'][0] + if anode.op not in self.conv_activations: + anode = None else: anode = None return knode, bnode, dnode, anode @@ -252,14 +255,47 @@ class TFConverter: np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f) + def dump_sub_to_file(self, node, f): + assert(node.op == 'Sub') + self.layer_number = self.layer_number + 1 + self.converted_nodes.add(node.name) + i0_node = self.name_node_dict[node.input[0]] + i1_node = self.name_node_dict[node.input[1]] + np.array([self.op2code['MathBinary'], self.mathbin2code[node.op]], dtype=np.uint32).tofile(f) + if i0_node.op == 'Const': + scalar = i0_node.attr['value'].tensor.float_val[0] + assert(i0_node.name.find('sub/x')) + np.array([1], dtype=np.uint32).tofile(f) + np.array([scalar], dtype=np.float32).tofile(f) + np.array([0], dtype=np.uint32).tofile(f) + input_operand_index = self.add_operand(i1_node.name, Operand.IOTYPE_INPUT) + np.array([input_operand_index], dtype=np.uint32).tofile(f) + elif i1_node.op == 'Const': + scalar = i1_node.attr['value'].tensor.float_val[0] + assert(i1_node.name.find('sub/y')) + np.array([0], dtype=np.uint32).tofile(f) + input_operand_index = self.add_operand(i0_node.name, Operand.IOTYPE_INPUT) + np.array([input_operand_index], dtype=np.uint32).tofile(f) + np.array([1], dtype=np.uint32).tofile(f) + np.array([scalar], dtype=np.float32).tofile(f) + else: + np.array([0], dtype=np.uint32).tofile(f) + input_operand_index = self.add_operand(i0_node.name, Operand.IOTYPE_INPUT) + np.array([input_operand_index], dtype=np.uint32).tofile(f) + np.array([0], dtype=np.uint32).tofile(f) + input_operand_index = self.add_operand(i1_node.name, Operand.IOTYPE_INPUT) + np.array([input_operand_index], dtype=np.uint32).tofile(f) + output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT) + np.array([output_operand_index], dtype=np.uint32).tofile(f) + + def dump_layers_to_file(self, f): for node in self.nodes: if node.name in self.converted_nodes: continue # conv2d with dilation generates very complex nodes, so handle it in special - scope_name = TFConverter.get_scope_name(node.name) - if scope_name in self.conv2d_scope_names: + if self.in_conv2d_scope(node.name): if node.op == 'Conv2D': self.dump_complex_conv2d_to_file(node, f) continue @@ -272,6 +308,8 @@ class TFConverter: self.dump_mirrorpad_to_file(node, f) elif node.op == 'Maximum': self.dump_maximum_to_file(node, f) + elif node.op == 'Sub': + self.dump_sub_to_file(node, f) def dump_operands_to_file(self, f): @@ -352,6 +390,17 @@ class TFConverter: return name[0:index] + def in_conv2d_scope(self, name): + inner_scope = TFConverter.get_scope_name(name) + if inner_scope == "": + return False; + for scope in self.conv2d_scope_names: + index = inner_scope.find(scope) + if index == 0: + return True + return False + + def generate_conv2d_scope_info(self): # mostly, conv2d is a sub block in graph, get the scope name for node in self.nodes: diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py index 67672b2..6576fca 100644 --- a/tools/python/convert_header.py +++ b/tools/python/convert_header.py @@ -23,4 +23,4 @@ str = 'FFMPEGDNNNATIVE' major = 1 # increase minor when we don't have to re-convert the model file -minor = 0 +minor = 1 From patchwork Mon Mar 23 04:08:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18363 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 7322E44B13D for ; Mon, 23 Mar 2020 06:09:26 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 61B5868B762; Mon, 23 Mar 2020 06:09:26 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3B17B68B755 for ; Mon, 23 Mar 2020 06:09:19 +0200 (EET) IronPort-SDR: viiMVlOiXBb1HCBzPI/86jQ6LiJIdsM6xOh5sYDSN5yRzvgIOAUKB4ZlaiGo8ddJgetreq0xVF BJNee1pdNIug== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Mar 2020 21:09:17 -0700 IronPort-SDR: 8O+Qz2+xc+JsNE0QRGM+yGlpJ/G/mjonUv1A0pdDkK/aPdHC8Y8lZXHCMiT4LE0OaMUmOu7RhD uF+qh0r8WYtg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,295,1580803200"; d="scan'208";a="246073131" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by orsmga003.jf.intel.com with ESMTP; 22 Mar 2020 21:09:16 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Mar 2020 12:08:23 +0800 Message-Id: <1584936503-9191-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 3/4] dnn-layer-mathbinary-test: add unit test for subtraction X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Guo, Yejun --- tests/dnn/.gitignore | 1 + tests/dnn/Makefile | 1 + tests/dnn/dnn-layer-mathbinary-test.c | 173 ++++++++++++++++++++++++++++++++++ tests/fate/dnn.mak | 5 + 4 files changed, 180 insertions(+) create mode 100644 tests/dnn/dnn-layer-mathbinary-test.c diff --git a/tests/dnn/.gitignore b/tests/dnn/.gitignore index 5eedaaa..d78c5c1 100644 --- a/tests/dnn/.gitignore +++ b/tests/dnn/.gitignore @@ -2,3 +2,4 @@ /dnn-layer-depth2space-test /dnn-layer-maximum-test /dnn-layer-pad-test +/dnn-layer-mathbinary-test diff --git a/tests/dnn/Makefile b/tests/dnn/Makefile index e1bfe3f..1f96710 100644 --- a/tests/dnn/Makefile +++ b/tests/dnn/Makefile @@ -1,6 +1,7 @@ DNNTESTPROGS += dnn-layer-pad DNNTESTPROGS += dnn-layer-conv2d DNNTESTPROGS += dnn-layer-depth2space +DNNTESTPROGS += dnn-layer-mathbinary DNNTESTPROGS += dnn-layer-maximum DNNTESTOBJS := $(DNNTESTOBJS:%=$(DNNTESTSDIR)%) $(DNNTESTPROGS:%=$(DNNTESTSDIR)/%-test.o) diff --git a/tests/dnn/dnn-layer-mathbinary-test.c b/tests/dnn/dnn-layer-mathbinary-test.c new file mode 100644 index 0000000..1243784 --- /dev/null +++ b/tests/dnn/dnn-layer-mathbinary-test.c @@ -0,0 +1,173 @@ +/* + * Copyright (c) 2020 + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include +#include +#include "libavfilter/dnn/dnn_backend_native_layer_mathbinary.h" + +#define EPSON 0.00001 + +static int test_sub_broadcast_input0(void) +{ + DnnLayerMathBinaryParams params; + DnnOperand operands[2]; + int32_t input_indexes[1]; + float input[1*1*2*3] = { + -3, 2.5, 2, -2.1, 7.8, 100 + }; + float *output; + + params.bin_op = DMBO_SUB; + params.input0_broadcast = 1; + params.input1_broadcast = 0; + params.v = 7.28; + + operands[0].data = input; + operands[0].dims[0] = 1; + operands[0].dims[1] = 1; + operands[0].dims[2] = 2; + operands[0].dims[3] = 3; + operands[1].data = NULL; + + input_indexes[0] = 0; + dnn_execute_layer_math_binary(operands, input_indexes, 1, ¶ms); + + output = operands[1].data; + for (int i = 0; i < sizeof(input) / sizeof(float); i++) { + float expected_output = params.v - input[i]; + if (fabs(output[i] - expected_output) > EPSON) { + printf("at index %d, output: %f, expected_output: %f\n", i, output[i], expected_output); + av_freep(&output); + return 1; + } + } + + av_freep(&output); + return 0; +} + +static int test_sub_broadcast_input1(void) +{ + DnnLayerMathBinaryParams params; + DnnOperand operands[2]; + int32_t input_indexes[1]; + float input[1*1*2*3] = { + -3, 2.5, 2, -2.1, 7.8, 100 + }; + float *output; + + params.bin_op = DMBO_SUB; + params.input0_broadcast = 0; + params.input1_broadcast = 1; + params.v = 7.28; + + operands[0].data = input; + operands[0].dims[0] = 1; + operands[0].dims[1] = 1; + operands[0].dims[2] = 2; + operands[0].dims[3] = 3; + operands[1].data = NULL; + + input_indexes[0] = 0; + dnn_execute_layer_math_binary(operands, input_indexes, 1, ¶ms); + + output = operands[1].data; + for (int i = 0; i < sizeof(input) / sizeof(float); i++) { + float expected_output = input[i] - params.v; + if (fabs(output[i] - expected_output) > EPSON) { + printf("at index %d, output: %f, expected_output: %f\n", i, output[i], expected_output); + av_freep(&output); + return 1; + } + } + + av_freep(&output); + return 0; +} + +static int test_sub_no_broadcast(void) +{ + DnnLayerMathBinaryParams params; + DnnOperand operands[3]; + int32_t input_indexes[2]; + float input0[1*1*2*3] = { + -3, 2.5, 2, -2.1, 7.8, 100 + }; + float input1[1*1*2*3] = { + -1, 2, 3, -21, 8, 10.0 + }; + float *output; + + params.bin_op = DMBO_SUB; + params.input0_broadcast = 0; + params.input1_broadcast = 0; + + operands[0].data = input0; + operands[0].dims[0] = 1; + operands[0].dims[1] = 1; + operands[0].dims[2] = 2; + operands[0].dims[3] = 3; + operands[1].data = input1; + operands[1].dims[0] = 1; + operands[1].dims[1] = 1; + operands[1].dims[2] = 2; + operands[1].dims[3] = 3; + operands[2].data = NULL; + + input_indexes[0] = 0; + input_indexes[1] = 1; + dnn_execute_layer_math_binary(operands, input_indexes, 2, ¶ms); + + output = operands[2].data; + for (int i = 0; i < sizeof(input0) / sizeof(float); i++) { + float expected_output = input0[i] - input1[i]; + if (fabs(output[i] - expected_output) > EPSON) { + printf("at index %d, output: %f, expected_output: %f\n", i, output[i], expected_output); + av_freep(&output); + return 1; + } + } + + av_freep(&output); + return 0; +} + +static int test_sub(void) +{ + if (test_sub_broadcast_input0()) + return 1; + + if (test_sub_broadcast_input1()) + return 1; + + if (test_sub_no_broadcast()) + return 1; + + return 0; +} + +int main(int argc, char **argv) +{ + if (test_sub()) + return 1; + + return 0; +} diff --git a/tests/fate/dnn.mak b/tests/fate/dnn.mak index ec60b07..5a8e629 100644 --- a/tests/fate/dnn.mak +++ b/tests/fate/dnn.mak @@ -13,6 +13,11 @@ fate-dnn-layer-depth2space: $(DNNTESTSDIR)/dnn-layer-depth2space-test$(EXESUF) fate-dnn-layer-depth2space: CMD = run $(DNNTESTSDIR)/dnn-layer-depth2space-test$(EXESUF) fate-dnn-layer-depth2space: CMP = null +FATE_DNN += fate-dnn-layer-mathbinary +fate-dnn-layer-mathbinary: $(DNNTESTSDIR)/dnn-layer-mathbinary-test$(EXESUF) +fate-dnn-layer-mathbinary: CMD = run $(DNNTESTSDIR)/dnn-layer-mathbinary-test$(EXESUF) +fate-dnn-layer-mathbinary: CMP = null + FATE_DNN += fate-dnn-layer-maximum fate-dnn-layer-maximum: $(DNNTESTSDIR)/dnn-layer-maximum-test$(EXESUF) fate-dnn-layer-maximum: CMD = run $(DNNTESTSDIR)/dnn-layer-maximum-test$(EXESUF) From patchwork Mon Mar 23 04:08:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 18364 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 55FC744B13D for ; Mon, 23 Mar 2020 06:09:29 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3CF4968B796; Mon, 23 Mar 2020 06:09:29 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8D2AE68B787 for ; Mon, 23 Mar 2020 06:09:27 +0200 (EET) IronPort-SDR: 81Ghq1VCDvHVkz7Stzu8eyoS7YSWkWybCk5qBBXjwJEcJVeWh00jzVhrSp9R9mk5I5HFUS3H7Z WNEV2zn2SLJw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Mar 2020 21:09:27 -0700 IronPort-SDR: 63bzrl1Uh+LnR8ADIo0AWYDkjCwqtXa3gMX55ROzJxVD7nJ9Joq5jTE8gJc+0meNKmd/Wo9yIX +ZJYqKRmH7ug== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,295,1580803200"; d="scan'208";a="447269583" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by fmsmga006.fm.intel.com with ESMTP; 22 Mar 2020 21:09:25 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Mon, 23 Mar 2020 12:08:32 +0800 Message-Id: <1584936512-9306-1-git-send-email-yejun.guo@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH 4/4] avfilter/vf_derain.c: put all the calculation in model file. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" currently, the model outputs the rain, and so need a subtraction in filter c code to get the final derain result. I've sent a PR to update the model file and accepted, see at https://github.com/XueweiMeng/derain_filter/pull/3 Signed-off-by: Guo, Yejun --- doc/filters.texi | 7 +++++-- libavfilter/vf_derain.c | 6 +----- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 328e984..3dc7715 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -8878,6 +8878,7 @@ delogo=x=0:y=0:w=100:h=77:band=10 @end itemize +@anchor{derain} @section derain Remove the rain in the input image/video by applying the derain methods based on @@ -8932,6 +8933,8 @@ Note that different backends use different file formats. TensorFlow and native backend can load files for only its format. @end table +It can also be finished with @ref{dnn_processing} filter. + @section deshake Attempt to fix small changes in horizontal and/or vertical shift. This @@ -9201,9 +9204,9 @@ Set the output name of the dnn network. @itemize @item -Halve the red channle of the frame with format rgb24: +Remove rain in rgb24 frame with can.pb (see @ref{derain} filter): @example -ffmpeg -i input.jpg -vf format=rgb24,dnn_processing=model=halve_first_channel.model:input=dnn_in:output=dnn_out:dnn_backend=native out.native.png +./ffmpeg -i rain.jpg -vf format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y derain.jpg @end example @item diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c index 89f9d5a..7432260 100644 --- a/libavfilter/vf_derain.c +++ b/libavfilter/vf_derain.c @@ -100,7 +100,6 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) AVFilterLink *outlink = ctx->outputs[0]; DRContext *dr_context = ctx->priv; DNNReturnType dnn_result; - int pad_size; AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); if (!out) { @@ -129,15 +128,12 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) out->width = dr_context->output.width; outlink->h = dr_context->output.height; outlink->w = dr_context->output.width; - pad_size = (in->height - out->height) >> 1; for (int i = 0; i < out->height; i++){ for(int j = 0; j < out->width * 3; j++){ int k = i * out->linesize[0] + j; int t = i * out->width * 3 + j; - - int t_in = (i + pad_size) * in->width * 3 + j + pad_size * 3; - out->data[0][k] = CLIP((int)((((float *)dr_context->input.data)[t_in] - ((float *)dr_context->output.data)[t]) * 255), 0, 255); + out->data[0][k] = CLIP((int)((((float *)dr_context->output.data)[t]) * 255), 0, 255); } }