[FFmpeg-devel,1/2] dnn: add layer pad which is equivalent to tf.pad

the reason to add this layer first is that vf_sr uses it in its
tensorflow model, and the next plan is to update the python script
to convert tf.pad into native model.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
---
 libavfilter/dnn/Makefile                       |   1 +
 libavfilter/dnn/dnn_backend_native_layer_pad.c | 211 +++++++++++++++++++++++++
 libavfilter/dnn/dnn_backend_native_layer_pad.h |  40 +++++
 3 files changed, 252 insertions(+)
 create mode 100644 libavfilter/dnn/dnn_backend_native_layer_pad.c
 create mode 100644 libavfilter/dnn/dnn_backend_native_layer_pad.h

> -----Original Message-----
> From: Guo, Yejun
> Sent: Monday, July 01, 2019 4:08 PM
> To: ffmpeg-devel@ffmpeg.org
> Cc: Guo, Yejun <yejun.guo@intel.com>
> Subject: [PATCH 1/2] dnn: add layer pad which is equivalent to tf.pad
> 
> the reason to add this layer first is that vf_sr uses it in its
> tensorflow model, and the next plan is to update the python script
> to convert tf.pad into native model.
> 
> Signed-off-by: Guo, Yejun <yejun.guo@intel.com>

this patch set is based on a previous patch at http://ffmpeg.org/pipermail/ffmpeg-devel/2019-June/245481.html 
(libavfilter/dnn: move dnn files from	libavfilter to libavfilter/dnn)

> ---
>  libavfilter/dnn/Makefile                       |   1 +
>  libavfilter/dnn/dnn_backend_native_layer_pad.c | 211
> +++++++++++++++++++++++++
>  libavfilter/dnn/dnn_backend_native_layer_pad.h |  40 +++++
>  3 files changed, 252 insertions(+)
>  create mode 100644 libavfilter/dnn/dnn_backend_native_layer_pad.c
>  create mode 100644 libavfilter/dnn/dnn_backend_native_layer_pad.h
>

> > -----Original Message-----

> > From: Guo, Yejun

> > Sent: Monday, July 01, 2019 4:08 PM

> > To: ffmpeg-devel@ffmpeg.org

> > Cc: Guo, Yejun <yejun.guo@intel.com>

> > Subject: [PATCH 1/2] dnn: add layer pad which is equivalent to tf.pad

> >

> > the reason to add this layer first is that vf_sr uses it in its

> > tensorflow model, and the next plan is to update the python script

> > to convert tf.pad into native model.

> >

> > Signed-off-by: Guo, Yejun <yejun.guo@intel.com>

> 

> this patch set is based on a previous patch at

> http://ffmpeg.org/pipermail/ffmpeg-devel/2019-June/245481.html

> (libavfilter/dnn: move dnn files from	libavfilter to libavfilter/dnn)


hello, I've finished 10+ new patches locally to improve dnn module, including one new patch relative to pad.

I'll resend this patch set, together with the new pad relative patch.

I'll send more patches set by set, once the previous patch set is pushed, since the patches have dependency.

Just in case you are interested in my new patches, I've uploaded to https://github.com/guoyejun/ffmpeg/tree/dnn0716. 
for your convenient, I also copy the oneline log here for each patch (from newer to older) with 3 patch sets.

50a3353 fate: add unit test for dnn depth_to_space layer
af9e3ab dnn: separate depth_to_space layer from dnn_backend_native.c to a new file
41b97e4 fate: add unit test for dnn conv2d layer
4143485 dnn: separate conv2d layer from dnn_backend_native.c to a new file

870383e dnn: export operand info in python script and load in c code
650d576 dnn: change .model file format to put layer number at the end of file
d029bf8 dnn: introduce dnn operand (in c code) to hold operand infos within network

c9b9e1c doc/filters: update how to generate native model for derain filter
064aa45 convert_from_tensorflow.py: support conv2d with dilation
1c419a5 convert_from_tensorflow.py: add option to dump graph for visualization in tensorboard


I'll continue to improve dnn module, such as:
- update dnn interface since we have more operand info (names, types, dims) in native mode.
- optimize native conv2d layer
- add more native layers to support at least mobile net as a milestone
- add a general filter which generates analysis result (in a general side data for AVFrame) with dnn network.
- add a general filter which changes the content of AVFrame with dnn network.

Hi,

Em seg, 1 de jul de 2019 às 05:10, Guo, Yejun <yejun.guo@intel.com> escreveu:
>
> the reason to add this layer first is that vf_sr uses it in its
> tensorflow model, and the next plan is to update the python script
> to convert tf.pad into native model.
>
> Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
> ---
>  libavfilter/dnn/Makefile                       |   1 +
>  libavfilter/dnn/dnn_backend_native_layer_pad.c | 211 +++++++++++++++++++++++++
>  libavfilter/dnn/dnn_backend_native_layer_pad.h |  40 +++++
>  3 files changed, 252 insertions(+)
>  create mode 100644 libavfilter/dnn/dnn_backend_native_layer_pad.c
>  create mode 100644 libavfilter/dnn/dnn_backend_native_layer_pad.h
>
> diff --git a/libavfilter/dnn/Makefile b/libavfilter/dnn/Makefile
> index 1d12ade..83938e5 100644
> --- a/libavfilter/dnn/Makefile
> +++ b/libavfilter/dnn/Makefile
> @@ -1,5 +1,6 @@
>  OBJS-$(CONFIG_DNN)                           += dnn/dnn_interface.o
>  OBJS-$(CONFIG_DNN)                           += dnn/dnn_backend_native.o
> +OBJS-$(CONFIG_DNN)                           += dnn/dnn_backend_native_layer_pad.o
>
>  DNN-OBJS-$(CONFIG_LIBTENSORFLOW)             += dnn/dnn_backend_tf.o
>
> diff --git a/libavfilter/dnn/dnn_backend_native_layer_pad.c b/libavfilter/dnn/dnn_backend_native_layer_pad.c
> new file mode 100644
> index 0000000..aa12f7f
> --- /dev/null
> +++ b/libavfilter/dnn/dnn_backend_native_layer_pad.c
> @@ -0,0 +1,211 @@
> +/*
> + * Copyright (c) 2019 Guo Yejun
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <string.h>
> +#include "libavutil/avassert.h"
> +#include "dnn_backend_native_layer_pad.h"
> +
> +static int before_get_buddy(int given, int paddings, LayerPadModeParam mode)
> +{
> +    if (mode == LPMP_SYMMETRIC) {
> +        return (2 * paddings - 1 - given);
> +    } else if (mode == LPMP_REFLECT) {
> +        return (2 * paddings - given);
> +    } else {
> +        av_assert0(!"should not reach here");
> +        return 0;
> +    }
> +}
> +
> +static int after_get_buddy(int given, int border, LayerPadModeParam mode)
> +{
> +    if (mode == LPMP_SYMMETRIC) {
> +        int offset = given - border;
> +        return (border - 1 - offset);
> +    } else if (mode == LPMP_REFLECT) {
> +        int offset = given - border;
> +        return (border - 2 - offset);
> +    } else {
> +        av_assert0(!"should not reach here");
> +        return 0;
> +    }
> +}
> +
> +void dnn_execute_layer_pad(const float *input, float *output, const LayerPadParams *params, int number, int height, int width, int channel)
> +{
> +    int32_t before_paddings;
> +    int32_t after_paddings;
> +
> +    // suppose format is <N, H, W, C>
> +    int new_number = number + params->paddings[0][0] + params->paddings[0][1];
> +    int new_height = height + params->paddings[1][0] + params->paddings[1][1];
> +    int new_width = width + params->paddings[2][0] + params->paddings[2][1];
> +    int new_channel = channel + params->paddings[3][0] + params->paddings[3][1];
> +
> +    int c_stride = channel;
> +    int wc_stride = c_stride * width;
> +    int hwc_stride = wc_stride * height;
> +
> +    int new_c_stride = new_channel;
> +    int new_wc_stride = new_c_stride * new_width;
> +    int new_hwc_stride = new_wc_stride * new_height;
> +
> +    // copy the original data
> +    for (int n = 0; n < number; n++) {
> +        for (int h = 0; h < height; h++) {
> +            for (int w = 0; w < width; w++) {
> +                const float *src = input + n * hwc_stride + h * wc_stride + w * c_stride;
> +                float *dst = output + (n + params->paddings[0][0]) * new_hwc_stride
> +                                    + (h + params->paddings[1][0]) * new_wc_stride
> +                                    + (w + params->paddings[2][0]) * new_c_stride
> +                                    + params->paddings[3][0];
> +                memcpy(dst, src, channel * sizeof(float));
> +            }
> +        }
> +    }
> +
> +    // handle the first dimension
> +    before_paddings = params->paddings[0][0];
> +    after_paddings = params->paddings[0][1];
> +    for (int n = 0; n < before_paddings; n++) {
> +        float *dst = output + n * new_hwc_stride;
> +        if (params->mode == LPMP_CONSTANT) {
> +            for (int i = 0; i < new_hwc_stride; i++) {
> +                *dst = params->constant_values;
I suppose it is missing a pointer increase in dst ptr?
I found this same pattern in all loops below.


> +            }
> +        }
> +        else {
> +            int buddy = before_get_buddy(n, before_paddings, params->mode);
> +            float *src = output + buddy * new_hwc_stride;
> +            memcpy(dst, src, new_hwc_stride * sizeof(float));
> +        }
> +    }
> +    for (int n = 0; n < after_paddings; n++) {
> +        int given = number + before_paddings + n;
> +        float *dst = output + given * new_hwc_stride;
> +        if (params->mode == LPMP_CONSTANT) {
> +            for (int i = 0; i < new_hwc_stride; i++) {
> +                *dst = params->constant_values;
there

> +            }
> +        } else {
> +            int buddy = after_get_buddy(given, number + before_paddings, params->mode);
> +            float *src = output + buddy * new_hwc_stride;
> +            memcpy(dst, src, new_hwc_stride * sizeof(float));
> +        }
> +    }
> +
> +    // handle the second dimension
> +    before_paddings = params->paddings[1][0];
> +    after_paddings = params->paddings[1][1];
> +    for (int n = 0; n < new_number; n++) {
> +        float *start = output + n * new_hwc_stride;
> +        for (int h = 0; h < before_paddings; h++) {
> +            float *dst = start + h * new_wc_stride;
> +            if (params->mode == LPMP_CONSTANT) {
> +                for (int i = 0; i < new_wc_stride; i++) {
> +                    *dst = params->constant_values;
there

> +                }
> +            } else {
> +                int buddy = before_get_buddy(h, before_paddings, params->mode);
> +                float *src = start + buddy * new_wc_stride;
> +                memcpy(dst, src, new_wc_stride * sizeof(float));
> +            }
> +        }
> +        for (int h = 0; h < after_paddings; h++) {
> +            int given = height + before_paddings + h;
> +            float *dst = start + given * new_wc_stride;
> +            if (params->mode == LPMP_CONSTANT) {
> +                for (int i = 0; i < new_wc_stride; i++) {
> +                    *dst = params->constant_values;
there

> +                }
> +            } else {
> +                int buddy = after_get_buddy(given, height + before_paddings, params->mode);
> +                float *src = start + buddy * new_wc_stride;
> +                memcpy(dst, src, new_wc_stride * sizeof(float));
> +            }
> +        }
> +    }
> +
> +    // handle the third dimension
> +    before_paddings = params->paddings[2][0];
> +    after_paddings = params->paddings[2][1];
> +    for (int n = 0; n < new_number; n++) {
> +        for (int h = 0; h < new_height; h++) {
> +            float *start = output + n * new_hwc_stride + h * new_wc_stride;
> +            for (int w = 0; w < before_paddings; w++) {
> +                float *dst = start + w * new_c_stride;
> +                if (params->mode == LPMP_CONSTANT) {
> +                    for (int i = 0; i < new_c_stride; i++) {
> +                        *dst = params->constant_values;
there

> +                    }
> +                } else {
> +                    int buddy = before_get_buddy(w, before_paddings, params->mode);
> +                    float *src = start + buddy * new_c_stride;
> +                    memcpy(dst, src, new_c_stride * sizeof(float));
> +                }
> +            }
> +            for (int w = 0; w < after_paddings; w++) {
> +                int given = width + before_paddings + w;
> +                float *dst = start + given * new_c_stride;
> +                if (params->mode == LPMP_CONSTANT) {
> +                    for (int i = 0; i < new_c_stride; i++) {
> +                        *dst = params->constant_values;
there

> +                    }
> +                } else {
> +                    int buddy = after_get_buddy(given, width + before_paddings, params->mode);
> +                    float *src = start + buddy * new_c_stride;
> +                    memcpy(dst, src, new_c_stride * sizeof(float));
> +                }
> +            }
> +        }
> +    }
> +
> +    // handle the fourth dimension
> +    before_paddings = params->paddings[3][0];
> +    after_paddings = params->paddings[3][1];
> +    for (int n = 0; n < new_number; n++) {
> +        for (int h = 0; h < new_height; h++) {
> +            for (int w = 0; w < new_width; w++) {
> +                float *start = output + n * new_hwc_stride + h * new_wc_stride + w * new_c_stride;
> +                for (int c = 0; c < before_paddings; c++) {
> +                    float *dst = start + c;
> +                    if (params->mode == LPMP_CONSTANT) {
> +                        *dst = params->constant_values;
> +                    } else {
> +                        int buddy = before_get_buddy(c, before_paddings, params->mode);
> +                        float *src = start + buddy;
> +                        *dst = *src;
> +                    }
> +                }
> +                for (int c = 0; c < after_paddings; c++) {
> +                    int given = channel + before_paddings + c;
> +                    float *dst = start + given;
> +                    if (params->mode == LPMP_CONSTANT) {
> +                        *dst = params->constant_values;
> +                    } else {
> +                        int buddy = after_get_buddy(given, channel + before_paddings, params->mode);
> +                        float *src = start + buddy;
> +                        *dst = *src;
> +                    }
> +                }
> +            }
> +        }
> +    }
> +}
> diff --git a/libavfilter/dnn/dnn_backend_native_layer_pad.h b/libavfilter/dnn/dnn_backend_native_layer_pad.h
> new file mode 100644
> index 0000000..465e96a
> --- /dev/null
> +++ b/libavfilter/dnn/dnn_backend_native_layer_pad.h
> @@ -0,0 +1,40 @@
> +/*
> + * Copyright (c) 2019 Guo Yejun
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +/**
> + * @file
> + * layer pad (equivalent to tf.pad) for native backend.
> + */
> +#include <stdint.h>
> +
> +#ifndef AVFILTER_DNN_BACKEND_NATIVE_LAYER_PAD_H
> +#define AVFILTER_DNN_BACKEND_NATIVE_LAYER_PAD_H
> +
> +typedef enum {LPMP_CONSTANT, LPMP_REFLECT, LPMP_SYMMETRIC} LayerPadModeParam;
> +
> +typedef struct LayerPadParams{
> +    int32_t paddings[4][2];
> +    LayerPadModeParam mode;
> +    float constant_values;
> +} LayerPadParams;
> +
> +void dnn_execute_layer_pad(const float *input, float *output, const LayerPadParams *params, int number, int height, int width, int channel);
> +
> +#endif
> --
> 2.7.4
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

> -----Original Message-----

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf Of

> Pedro Arthur

> Sent: Friday, July 26, 2019 10:36 PM

> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>

> Subject: Re: [FFmpeg-devel] [PATCH 1/2] dnn: add layer pad which is equivalent

> to tf.pad

> 

> Hi,

> 

> Em seg, 1 de jul de 2019 às 05:10, Guo, Yejun <yejun.guo@intel.com>

> escreveu:

> >


> > +    // handle the first dimension

> > +    before_paddings = params->paddings[0][0];

> > +    after_paddings = params->paddings[0][1];

> > +    for (int n = 0; n < before_paddings; n++) {

> > +        float *dst = output + n * new_hwc_stride;

> > +        if (params->mode == LPMP_CONSTANT) {

> > +            for (int i = 0; i < new_hwc_stride; i++) {

> > +                *dst = params->constant_values;

> I suppose it is missing a pointer increase in dst ptr?

> I found this same pattern in all loops below.


thanks, nice catch.

I just checked my unit test for LPMP_CONSTANT and found it happened to test the fourth dimension which does not need the pointer increasing.

will send out a new patch together with unit test improved.