From patchwork Sat May 18 07:19:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuewei Meng X-Patchwork-Id: 13189 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D2FDD44922D for ; Sat, 18 May 2019 10:26:19 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AABD868A742; Sat, 18 May 2019 10:26:19 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E83DA680818 for ; Sat, 18 May 2019 10:26:12 +0300 (EEST) Received: by mail-pg1-f195.google.com with SMTP id a3so4365126pgb.3 for ; Sat, 18 May 2019 00:26:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6i49LtXi+pcD/FsaE9pONP0wa1s5xdgttL9oR4M7wAA=; b=mpeTz7RLTl7GACISxT69D/dLikI2n1UkIPm7qNtrl3dNEInnVUrXh4tY1z9e8r2pRq cOdtnEJAIoBFd5vv3ptQrapm3Bx6oFGMQ7ZHvD9bJ/WAdhP1gpX37FTGP/vfOQRAaoWb XDoTEQlxNL+dKlddOMFDSllwNr04A0g1WzBHq3q8IHFDzzHNy+5zvvbHUEl/OCOETfK8 4g4qTyb6TsFrwkTko4ekpUx08Pjjs8K/cocWBsyugfgoVjQZ15agD3SZxMAAAGpogCzG oGgWki5YUQSNsEKMgaeTUdaRT7TYjoPEyFsFA1qBGZCl4bbq7DUJtzxAaUUshkNTgZkJ N95w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6i49LtXi+pcD/FsaE9pONP0wa1s5xdgttL9oR4M7wAA=; b=bP1p4E+nqrAAKX43r3D1LGf5DWAqDksUObvf3EY1ZvBrGyYdsPEtvmnKPDxCMd6egP gr1xzT6F7S36NSTvsCl0sWi4du6F0nq75HGCiVe1UmAxiw4ROuCsn6XzFiyzIJlO45BC fDYDxnBcg5Wg/B5orVG8RpCem7JwmUAMDivsHT1MWeNpvRMYu/m/MjeZi+UqkllM7BEO 8t/gAQkzHD3NGVWvdp7TjDRdrQn9GeY957V7hxy04wXubCHQsWJalsR5wSWIxacriOp3 7FCqLZsH+U1Z2LvLLqxAuybiAvoX/o/j+fs3ugIKqik5Dw+To3iVqvenck83cSXN0kXb Gpeg== X-Gm-Message-State: APjAAAXQ8nlOBMMqKUmeHQmjFsXIf73rkAuvzfuUDJwfu9EVkvgN9ksu RxJLWbCDe8seTWX0ezEXMizzpGKEZquuaQ== X-Google-Smtp-Source: APXvYqwa8dBXsd7W8SOIneeJrFjWhnaio1dKusRLKX+yeS+O+qEJIu9T3lO9xEukGX7enQNAj8QYaQ== X-Received: by 2002:a62:6dc6:: with SMTP id i189mr65218410pfc.155.1558163970757; Sat, 18 May 2019 00:19:30 -0700 (PDT) Received: from DESKTOP-IACK8OK.localdomain ([2001:da8:201:3474:80cc:829:3853:a14e]) by smtp.gmail.com with ESMTPSA id c142sm21542264pfb.171.2019.05.18.00.19.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 18 May 2019 00:19:30 -0700 (PDT) From: Xuewei Meng To: ffmpeg-devel@ffmpeg.org Date: Sat, 18 May 2019 15:19:18 +0800 Message-Id: <20190518071918.525-1-xwmeng96@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <854E8DBA9F41904AB047E03BB6963AE55FAE14D5@SHSMSX101.ccr.corp.intel.com> References: <854E8DBA9F41904AB047E03BB6963AE55FAE14D5@SHSMSX101.ccr.corp.intel.com> Subject: [FFmpeg-devel] [PATCH v3] libavfilter/dnn_native: Add multiple padding methods in dnn native X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Xuewei Meng MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Add another two padding methods "VALID" and "SAME" as tensorflow, and keep the existing "SAME_CLAMP_TO_EDGE" method suggested by sr filter. As "SAME_CLAMP_TO_EDGE"can keep the output with the same size as original input, and gives a slight better result as mentioned by sr filter. Signed-off-by: Xuewei Meng --- libavfilter/dnn_backend_native.c | 52 ++++++++++++++++++++++++-------- libavfilter/dnn_backend_native.h | 3 ++ 2 files changed, 43 insertions(+), 12 deletions(-) diff --git a/libavfilter/dnn_backend_native.c b/libavfilter/dnn_backend_native.c index 06fbdf368b..171a756385 100644 --- a/libavfilter/dnn_backend_native.c +++ b/libavfilter/dnn_backend_native.c @@ -61,6 +61,12 @@ static DNNReturnType set_input_output_native(void *model, DNNInputData *input, c return DNN_ERROR; } cur_channels = conv_params->output_num; + + if(conv_params->padding_method == VALID){ + int pad_size = conv_params->kernel_size - 1; + cur_height -= pad_size; + cur_width -= pad_size; + } break; case DEPTH_TO_SPACE: depth_to_space_params = (DepthToSpaceParams *)network->layers[layer].params; @@ -77,6 +83,10 @@ static DNNReturnType set_input_output_native(void *model, DNNInputData *input, c if (network->layers[layer].output){ av_freep(&network->layers[layer].output); } + + if(cur_height <= 0 || cur_width <= 0) + return DNN_ERROR; + network->layers[layer].output = av_malloc(cur_height * cur_width * cur_channels * sizeof(float)); if (!network->layers[layer].output){ return DNN_ERROR; @@ -154,13 +164,14 @@ DNNModel *ff_dnn_load_model_native(const char *model_filename) ff_dnn_free_model_native(&model); return NULL; } + conv_params->padding_method = (int32_t)avio_rl32(model_file_context); conv_params->activation = (int32_t)avio_rl32(model_file_context); conv_params->input_num = (int32_t)avio_rl32(model_file_context); conv_params->output_num = (int32_t)avio_rl32(model_file_context); conv_params->kernel_size = (int32_t)avio_rl32(model_file_context); kernel_size = conv_params->input_num * conv_params->output_num * conv_params->kernel_size * conv_params->kernel_size; - dnn_size += 16 + (kernel_size + conv_params->output_num << 2); + dnn_size += 20 + (kernel_size + conv_params->output_num << 2); if (dnn_size > file_size || conv_params->input_num <= 0 || conv_params->output_num <= 0 || conv_params->kernel_size <= 0){ avio_closep(&model_file_context); @@ -218,23 +229,35 @@ DNNModel *ff_dnn_load_model_native(const char *model_filename) static void convolve(const float *input, float *output, const ConvolutionalParams *conv_params, int width, int height) { - int y, x, n_filter, ch, kernel_y, kernel_x; int radius = conv_params->kernel_size >> 1; int src_linesize = width * conv_params->input_num; int filter_linesize = conv_params->kernel_size * conv_params->input_num; int filter_size = conv_params->kernel_size * filter_linesize; + int pad_size = (conv_params->padding_method == VALID) ? (conv_params->kernel_size - 1) / 2 : 0; - for (y = 0; y < height; ++y){ - for (x = 0; x < width; ++x){ - for (n_filter = 0; n_filter < conv_params->output_num; ++n_filter){ + for (int y = pad_size; y < height - pad_size; ++y){ + for (int x = pad_size; x < width - pad_size; ++x){ + for (int n_filter = 0; n_filter < conv_params->output_num; ++n_filter){ output[n_filter] = conv_params->biases[n_filter]; - for (ch = 0; ch < conv_params->input_num; ++ch){ - for (kernel_y = 0; kernel_y < conv_params->kernel_size; ++kernel_y){ - for (kernel_x = 0; kernel_x < conv_params->kernel_size; ++kernel_x){ - output[n_filter] += input[CLAMP_TO_EDGE(y + kernel_y - radius, height) * src_linesize + - CLAMP_TO_EDGE(x + kernel_x - radius, width) * conv_params->input_num + ch] * - conv_params->kernel[n_filter * filter_size + kernel_y * filter_linesize + - kernel_x * conv_params->input_num + ch]; + + for (int ch = 0; ch < conv_params->input_num; ++ch){ + for (int kernel_y = 0; kernel_y < conv_params->kernel_size; ++kernel_y){ + for (int kernel_x = 0; kernel_x < conv_params->kernel_size; ++kernel_x){ + float input_pel; + if(conv_params->padding_method == SAME_CLAMP_TO_EDGE){ + int y_pos = CLAMP_TO_EDGE(y + kernel_y - radius, height); + int x_pos = CLAMP_TO_EDGE(x + kernel_x - radius, width); + input_pel = input[y_pos * src_linesize + x_pos * conv_params->input_num + ch]; + }else{ + int y_pos = y + kernel_y - radius; + int x_pos = x + kernel_x - radius; + input_pel = (x_pos < 0 || x_pos >= width || y_pos < 0 || y_pos >= height) ? 0.0 : + input[y_pos * src_linesize + x_pos * conv_params->input_num + ch]; + } + + + output[n_filter] += input_pel * conv_params->kernel[n_filter * filter_size + kernel_y * filter_linesize + + kernel_x * conv_params->input_num + ch]; } } } @@ -305,6 +328,11 @@ DNNReturnType ff_dnn_execute_model_native(const DNNModel *model, DNNData *output conv_params = (ConvolutionalParams *)network->layers[layer].params; convolve(network->layers[layer - 1].output, network->layers[layer].output, conv_params, cur_width, cur_height); cur_channels = conv_params->output_num; + if(conv_params->padding_method == VALID){ + int pad_size = conv_params->kernel_size - 1; + cur_height -= pad_size; + cur_width -= pad_size; + } break; case DEPTH_TO_SPACE: depth_to_space_params = (DepthToSpaceParams *)network->layers[layer].params; diff --git a/libavfilter/dnn_backend_native.h b/libavfilter/dnn_backend_native.h index e13a68a168..d70cd16387 100644 --- a/libavfilter/dnn_backend_native.h +++ b/libavfilter/dnn_backend_native.h @@ -34,6 +34,8 @@ typedef enum {INPUT, CONV, DEPTH_TO_SPACE} DNNLayerType; typedef enum {RELU, TANH, SIGMOID} DNNActivationFunc; +typedef enum {VALID, SAME, SAME_CLAMP_TO_EDGE} DNNConvPaddingParam; + typedef struct Layer{ DNNLayerType type; float *output; @@ -43,6 +45,7 @@ typedef struct Layer{ typedef struct ConvolutionalParams{ int32_t input_num, output_num, kernel_size; DNNActivationFunc activation; + DNNConvPaddingParam padding_method; float *kernel; float *biases; } ConvolutionalParams;