diff mbox series

[FFmpeg-devel] avfilter: add http video filter.

Message ID 1599782008.655533000.a5lxms1z@frv55.fwdcdn.com
State New
Headers show
Series [FFmpeg-devel] avfilter: add http video filter. | expand

Checks

Context Check Description
andriy/default pending
andriy/make success Make finished
andriy/make_fate success Make fate finished

Commit Message

Alex Sept. 10, 2020, 11:57 p.m. UTC
Signed-off-by: alex_qt <3.14pi@ukr.net>
---
 Changelog                |   1 +
 configure                |   4 +
 doc/filters.texi         |  28 +++++
 libavfilter/Makefile     |   1 +
 libavfilter/allfilters.c |   1 +
 libavfilter/version.h    |   2 +-
 libavfilter/vf_http.c    | 221 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 257 insertions(+), 1 deletion(-)
 create mode 100644 libavfilter/vf_http.c

Comments

Timo Rothenpieler Sept. 11, 2020, 1:06 a.m. UTC | #1
Entirely outside of this filter being acceptable at all,
pulling in libcurl is not an option without major safeguards for every 
single overlapping tls library both ffmpeg and curl support.
If both are using the same library, there is a huge potential for 
libcurl being linked at all breaks ffmpegs ability to talk TLS.

On top of that, ffmpeg already has code to talk to http servers, so 
pulling in a library to do it is even less acceptable.

I also really fail to see the utility of this filter. What's stopping 
you from just making ffmpeg output raw frames and then sending them off 
via the curl cli tool or whatever else?
Nicolas George Sept. 11, 2020, 9:40 a.m. UTC | #2
Timo Rothenpieler (12020-09-11):
> Entirely outside of this filter being acceptable at all,
> pulling in libcurl is not an option without major safeguards for every
> single overlapping tls library both ffmpeg and curl support.
> If both are using the same library, there is a huge potential for libcurl
> being linked at all breaks ffmpegs ability to talk TLS.
> 
> On top of that, ffmpeg already has code to talk to http servers, so pulling
> in a library to do it is even less acceptable.

I completely agree with this.

> I also really fail to see the utility of this filter. What's stopping you
> from just making ffmpeg output raw frames and then sending them off via the
> curl cli tool or whatever else?

This would happen at the end of the filter graph. This patch is for
something in the middle. For a complex graph, I do not think there is an
easy way of implementing with just command-line tools. We would need
some kind of movie sink for that.

But this is way too specific to be accepted. Hard-coded URL parameters,
nothing to set the formats, not even the possibility of a filter that
changes the resolution. This is exactly what I was warning about in this
mail:
https://ffmpeg.org/pipermail/ffmpeg-devel/2020-September/269348.html

And do not let us forget that the coding style is not at all what we do.

Regards,
Alex Sept. 11, 2020, 11:02 a.m. UTC | #3
> Hard-coded URL parameters,
>nothing to set the formats, not even the possibility of a filter that
>changes the resolution.

This filter has options: url and content-type header for requests and it's not hardcoded. Format and resolution can be changed after this filter later in filter graph, it's not a job for this filter, for example:
ffmpeg -i video.mp4 -vf scale=1920:-1,format=rgb24,http=url="http://localhost:3000/frame?type=ffmpeg":content_type="application/octet-stream",scale=1280:-1,format=yuv420p -c:v h264 -y out.mp4

And it will be send raw frames to servers with additional parameters in url query, like that: "POST http://localhost:3000/frame?type=ffmpeg&width=1280&height=720&format=2&linesize=344336064&size=2764800&pts=22012

For my opportunity this filter open door for postprocess frames outside of ffmpeg, so filter can be written or at least prototyped in other languages: js, python, go, etc without touching ffmpeg code and quick start for people who don't know c language or may be script is to big for porting to ffmpeg code.

For my use case I will be used it to "connect" ffmpeg to python (this script is huge and complicated to porting to c language and ffmpeg) script that postprocess frame/image in different process (docker container). 

Without this http filter I must do the same stuff in multiple steps, for example (it's typical usage):

#1 use ffmpeg to extract jpg's images for every frame from video and save it to folder on disk
#2 run custom python script on every images in folder and create output images
#3 use ffmpeg to marge images into video and do something to sink movie

But instead of multistage flow I can convert video in single step with injection of "http" filter in filter graph like in example.


--- Original message ---
From: "Nicolas George" <george@nsup.org>
Date: 11 September 2020, 12:40:23

Timo Rothenpieler (12020-09-11):
> Entirely outside of this filter being acceptable at all,
> pulling in libcurl is not an option without major safeguards for every
> single overlapping tls library both ffmpeg and curl support.
> If both are using the same library, there is a huge potential for libcurl
> being linked at all breaks ffmpegs ability to talk TLS.
> 
> On top of that, ffmpeg already has code to talk to http servers, so pulling
> in a library to do it is even less acceptable.

I completely agree with this.

> I also really fail to see the utility of this filter. What's stopping you
> from just making ffmpeg output raw frames and then sending them off via the
> curl cli tool or whatever else?

This would happen at the end of the filter graph. This patch is for
something in the middle. For a complex graph, I do not think there is an
easy way of implementing with just command-line tools. We would need
some kind of movie sink for that.

But this is way too specific to be accepted. Hard-coded URL parameters,
nothing to set the formats, not even the possibility of a filter that
changes the resolution. This is exactly what I was warning about in this
mail:
https://ffmpeg.org/pipermail/ffmpeg-devel/2020-September/269348.html

And do not let us forget that the coding style is not at all what we do.

Regards,
Nicolas George Sept. 11, 2020, 11:09 a.m. UTC | #4
Alex (12020-09-11):
> This filter has options: url and content-type header for requests and it's not hardcoded.

The format for the parameters is hard-coded.

> Format and resolution can be changed after this filter later in filter
> graph, it's not a job for this filter, for example:

It is a job for this filter if the external processing requires it. As
it is, the external processing must return frames with the same format
and resolution as its input, which is a completely unacceptable
limitation.

> For my opportunity this filter open door for postprocess frames outside of ffmpeg

We do not accept patches for somebody's opportunity. Patches need to be
generic enough to work for many people. The hard-coding of HTTP, the
hard-coding of the parameters passing, the arbitrary limitations make
this unacceptable.

The sooner you accept it and start thinking more globally, the sooner
you can consider making something that would be accepted. I have given
an indication: what you need a movie sink.

Regards,
Alex Sept. 11, 2020, 11:41 a.m. UTC | #5
>The format for the parameters is hard-coded.Please, tell me more I don't understand that do You mean?

--- Original message ---
From: "Nicolas George" <george@nsup.org>
Date: 11 September 2020, 14:09:47

Alex (12020-09-11):
> This filter has options: url and content-type header for requests and it's not hardcoded.

The format for the parameters is hard-coded.

> Format and resolution can be changed after this filter later in filter
> graph, it's not a job for this filter, for example:

It is a job for this filter if the external processing requires it. As
it is, the external processing must return frames with the same format
and resolution as its input, which is a completely unacceptable
limitation.

> For my opportunity this filter open door for postprocess frames outside of ffmpeg

We do not accept patches for somebody's opportunity. Patches need to be
generic enough to work for many people. The hard-coding of HTTP, the
hard-coding of the parameters passing, the arbitrary limitations make
this unacceptable.

The sooner you accept it and start thinking more globally, the sooner
you can consider making something that would be accepted. I have given
an indication: what you need a movie sink.

Regards,
Alex Sept. 11, 2020, 1:01 p.m. UTC | #6
> The hard-coding of HTTP, the hard-coding of the parameters passing, the arbitrary limitations make> this unacceptable.

Can You explain more, because I not fully understand "hard-coding " params, http.

> It is a job for this filter if the external processing requires it. As
> it is, the external processing must return frames with the same format
> and resolution as its input, which is a completely unacceptable
> limitation

I found way for fixing that, server must set headers with new image params in response and then it's possible to create new frame with right width, height, pix format, etc. I will fix it.

 

--- Original message ---
From: "Nicolas George" <george@nsup.org>
Date: 11 September 2020, 14:09:47

Alex (12020-09-11):
> This filter has options: url and content-type header for requests and it's not hardcoded.

The format for the parameters is hard-coded.

> Format and resolution can be changed after this filter later in filter
> graph, it's not a job for this filter, for example:

It is a job for this filter if the external processing requires it. As
it is, the external processing must return frames with the same format
and resolution as its input, which is a completely unacceptable
limitation.

> For my opportunity this filter open door for postprocess frames outside of ffmpeg

We do not accept patches for somebody's opportunity. Patches need to be
generic enough to work for many people. The hard-coding of HTTP, the
hard-coding of the parameters passing, the arbitrary limitations make
this unacceptable.

The sooner you accept it and start thinking more globally, the sooner
you can consider making something that would be accepted. I have given
an indication: what you need a movie sink.

Regards,
Nicolas George Sept. 11, 2020, 1:49 p.m. UTC | #7
Alex (12020-09-11):
> Can You explain more, because I not fully understand "hard-coding " params, http.

Where does it say that the size and format of the frame needs to be
passed as "width=1280&height=720&format=1" rather than
"1280x720/yuv420p"?

In fact, where did you find that using the numeric value of a codec id
like that was a good idea?

You are not adding a generic filter, not even a generic HTTP filter, you
are defining your own private protocol op top of HTTP, without even
realizing it, without properly documenting it, and making it
unacceptably limited (where is the time base? where is the aspect
ratio?).

There are already many protocols and formats that already do all of it,
properly.

Regards,
diff mbox series

Patch

diff --git a/Changelog b/Changelog
index cd8be931ef..777cca679a 100644
--- a/Changelog
+++ b/Changelog
@@ -22,6 +22,7 @@  version <next>:
 - MODS demuxer
 - PhotoCD decoder
 - MCA demuxer
+- http video filter, send raw frames to remote url for postprocessing
 
 
 version 4.3:
diff --git a/configure b/configure
index ae8c6e61c8..f5131d4669 100755
--- a/configure
+++ b/configure
@@ -325,6 +325,7 @@  External library support:
   --enable-vulkan          enable Vulkan code [no]
   --disable-xlib           disable xlib [autodetect]
   --disable-zlib           disable zlib [autodetect]
+  --enable-libcurl         enable http filter that send raw frames to remote server
 
   The following libraries provide various hardware acceleration features:
   --disable-amf            disable AMF video encoding code [autodetect]
@@ -1827,6 +1828,7 @@  EXTERNAL_LIBRARY_LIST="
     opengl
     pocketsphinx
     vapoursynth
+    libcurl
 "
 
 HWACCEL_AUTODETECT_LIBRARY_LIST="
@@ -3650,6 +3652,7 @@  vpp_qsv_filter_select="qsvvpp"
 xfade_opencl_filter_deps="opencl"
 yadif_cuda_filter_deps="ffnvcodec"
 yadif_cuda_filter_deps_any="cuda_nvcc cuda_llvm"
+http_filter_deps="libcurl"
 
 # examples
 avio_list_dir_deps="avformat avutil"
@@ -6367,6 +6370,7 @@  enabled libmysofa         && { check_pkg_config libmysofa libmysofa mysofa.h mys
 enabled libnpp            && { check_lib libnpp npp.h nppGetLibVersion -lnppig -lnppicc -lnppc -lnppidei ||
                                check_lib libnpp npp.h nppGetLibVersion -lnppi -lnppc -lnppidei ||
                                die "ERROR: libnpp not found"; }
+enabled libcurl           && require "libcurl >= 7.68.0" curl/curl.h curl_easy_init -lcurl
 enabled libopencore_amrnb && require libopencore_amrnb opencore-amrnb/interf_dec.h Decoder_Interface_init -lopencore-amrnb
 enabled libopencore_amrwb && require libopencore_amrwb opencore-amrwb/dec_if.h D_IF_init -lopencore-amrwb
 enabled libopencv         && { check_headers opencv2/core/core_c.h &&
diff --git a/doc/filters.texi b/doc/filters.texi
index cbb16f22b2..660ef8b4d9 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -12078,6 +12078,34 @@  For example, to horizontally flip the input video with @command{ffmpeg}:
 ffmpeg -i in.avi -vf "hflip" out.avi
 @end example
 
+@anchor{http}
+@section http
+
+Send raw frame data to the remote server for postprocessing and await response as new frame in same format and size. To enable filter configure ffmpeg with @code{./configure --enable-libcurl}.
+
+The filter accepts the following options:
+
+@table @option
+@item url
+Specify remote server url location.
+
+@item content_type
+Specify content-type header in request header, default to "application/octet-stream".
+@end table
+
+Simple demo http server for postprocessing frames can be found here: @url{https://github.com/devalexqt/simple_ffmpeg_http_filter_server}
+
+@subsection Examples
+@itemize
+
+@item
+Send raw frames to "http://localhost:3000/frame?param=abc"
+
+@example
+ffmpeg -i input.mp4 -vf format=rgb24,http=url="http\\\://localhost\\\:3000/frame?param=abc":content_type=application/octet-stream -t 10 out.mp4
+@end example
+@end itemize
+
 @section histeq
 This filter applies a global color histogram equalization on a
 per-frame basis.
diff --git a/libavfilter/Makefile b/libavfilter/Makefile
index e6d3c283da..38eb1f4204 100644
--- a/libavfilter/Makefile
+++ b/libavfilter/Makefile
@@ -467,6 +467,7 @@  OBJS-$(CONFIG_YAEPBLUR_FILTER)               += vf_yaepblur.o
 OBJS-$(CONFIG_ZMQ_FILTER)                    += f_zmq.o
 OBJS-$(CONFIG_ZOOMPAN_FILTER)                += vf_zoompan.o
 OBJS-$(CONFIG_ZSCALE_FILTER)                 += vf_zscale.o
+OBJS-$(CONFIG_HTTP_FILTER)                   += vf_http.o
 
 OBJS-$(CONFIG_ALLRGB_FILTER)                 += vsrc_testsrc.o
 OBJS-$(CONFIG_ALLYUV_FILTER)                 += vsrc_testsrc.o
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
index fa91e608e4..8626fbc331 100644
--- a/libavfilter/allfilters.c
+++ b/libavfilter/allfilters.c
@@ -445,6 +445,7 @@  extern AVFilter ff_vf_yaepblur;
 extern AVFilter ff_vf_zmq;
 extern AVFilter ff_vf_zoompan;
 extern AVFilter ff_vf_zscale;
+extern AVFilter ff_vf_http;
 
 extern AVFilter ff_vsrc_allrgb;
 extern AVFilter ff_vsrc_allyuv;
diff --git a/libavfilter/version.h b/libavfilter/version.h
index 308fbe07c3..b8ba489da7 100644
--- a/libavfilter/version.h
+++ b/libavfilter/version.h
@@ -30,7 +30,7 @@ 
 #include "libavutil/version.h"
 
 #define LIBAVFILTER_VERSION_MAJOR   7
-#define LIBAVFILTER_VERSION_MINOR  87
+#define LIBAVFILTER_VERSION_MINOR  88
 #define LIBAVFILTER_VERSION_MICRO 100
 
 
diff --git a/libavfilter/vf_http.c b/libavfilter/vf_http.c
new file mode 100644
index 0000000000..f4290ca254
--- /dev/null
+++ b/libavfilter/vf_http.c
@@ -0,0 +1,221 @@ 
+/*
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * http video filter
+ */
+
+#include "libavutil/avassert.h"
+#include "libavutil/opt.h"
+#include "libavutil/internal.h"
+#include "libavutil/imgutils.h"
+#include "libavutil/avstring.h"
+#include "avfilter.h"
+#include "formats.h"
+#include "internal.h"
+#include "video.h"
+#include "stdio.h"
+#include "curl/curl.h"
+
+struct MemoryStruct {
+  uint8_t *memory;
+  size_t size;
+};
+
+typedef struct HttpContext {
+    const AVClass *class;
+    char *url;
+    char *content_type;
+    CURL *curl;
+    struct curl_slist *headers;
+} HttpContext;
+
+#define OFFSET(x) offsetof(HttpContext, x)
+#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM
+
+static const AVOption http_options[] = {
+    { "url", "set remote url address", OFFSET(url), AV_OPT_TYPE_STRING, {.str=NULL}, FLAGS },
+    { "content_type", "set 'Content-Type' request header", OFFSET(content_type), AV_OPT_TYPE_STRING, {.str="application/octet-stream"}, FLAGS },
+    { NULL }
+};
+
+AVFILTER_DEFINE_CLASS(http);
+
+static av_cold int init(AVFilterContext *ctx)
+{
+    HttpContext *http = ctx->priv;
+    http->curl = curl_easy_init();
+
+    /* check if remote server url is valid formated */
+    CURLU *url= curl_url();
+    CURLUcode result;
+
+    /* parse a full URL */ 
+    result = curl_url_set(url, CURLUPART_URL, http->url, 0);
+    if(result){
+        av_log(NULL, AV_LOG_ERROR, "http filter failed: invalid input url!\n");
+    return AVERROR(EINVAL); 
+    }
+
+    /* set request headers */
+    http->headers=NULL;
+    http->headers = curl_slist_append(http->headers, av_asprintf("Content-Type: %s",http->content_type));
+    http->headers = curl_slist_append(http->headers, "Expect:");
+    curl_easy_setopt(http->curl, CURLOPT_HTTPHEADER, http->headers);
+    curl_easy_setopt(http->curl, CURLOPT_FOLLOWLOCATION, 1L);
+
+    curl_url_cleanup(url);
+    return 0;
+}
+
+static av_cold void uninit(AVFilterContext *ctx)
+{
+    HttpContext *http = ctx->priv;
+    
+    curl_easy_cleanup(http->curl);
+    curl_slist_free_all(http->headers);
+    curl_global_cleanup();
+}
+
+static int query_formats(AVFilterContext *ctx)
+{
+    const HttpContext *http = ctx->priv;
+    return ff_default_query_formats(ctx);
+}
+
+static size_t WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)
+{
+    size_t realsize = size * nmemb;
+    struct MemoryStruct *mem = (struct MemoryStruct *)userp;
+ 
+    char *ptr = realloc(mem->memory, mem->size + realsize + 1);
+    if(!ptr) {
+      /* out of memory! */ 
+      return AVERROR(ENOMEM); 
+    }
+ 
+    mem->memory = ptr;
+    memcpy(&(mem->memory[mem->size]), contents, realsize);
+    mem->size += realsize;
+    mem->memory[mem->size] = 0;
+ 
+  return realsize;
+}
+
+static int filter_frame(AVFilterLink *inlink, AVFrame *in)
+{
+    AVFilterContext *ctx = inlink->dst;
+    AVFilterLink *outlink = ctx->outputs[0];
+    HttpContext *http = inlink->dst->priv;
+    AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h);
+    if (!out) {
+       av_frame_free(&in);
+       return AVERROR(ENOMEM); 
+    }
+    av_frame_copy_props(out, in);
+
+    CURL *curl=http->curl;
+    CURLU *url= curl_url();
+    CURLcode res;
+    struct MemoryStruct chunk;
+       chunk.memory = av_malloc(1);
+       chunk.size = 0;    
+
+    /* create buffer and copy input frame */
+    int buff_size=av_image_get_buffer_size(in->format, in->width, in->height, 1);
+    uint8_t *buffer=av_malloc(buff_size);    
+    av_image_copy_to_buffer(buffer, buff_size, in->data, in->linesize, in->format, in->width, in->height, 1);
+
+    /* update url query */
+    CURLUcode result;
+    const char *url_params=av_asprintf("width=%d&height=%d&format=%d&size=%d&pts=%ld",in->width, in->height, in->format, buff_size, in->pts);
+    curl_url_set(url, CURLUPART_URL, http->url, 0);
+    result = curl_url_set(url, CURLUPART_QUERY, url_params, CURLU_APPENDQUERY);
+    if(result){
+        av_log(NULL, AV_LOG_ERROR, "http filter failed: failed to setup url query params!\n");
+    return AVERROR(EINVAL); 
+    }
+
+    /* specify the POST data */ 
+    curl_easy_setopt(http->curl, CURLOPT_CURLU,url);
+    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, buffer);
+    curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, buff_size);
+
+    /* send all data to this function  */ 
+    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
+        
+    /* we pass our 'chunk' struct to the callback function */ 
+    curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&chunk);
+    curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/1.0");
+
+    /* Perform the request, res will get the return code */ 
+    res = curl_easy_perform(curl);
+
+    /* Check for errors */ 
+    if(res != CURLE_OK){
+        av_log(NULL, AV_LOG_ERROR, "http filter failed: %s\n", curl_easy_strerror(res));
+        av_frame_free(&in);
+        av_frame_free(&out);
+        free(buffer);
+    return AVERROR(EINVAL);
+    }
+    else{
+        if(chunk.size!=buff_size){
+           av_log(NULL, AV_LOG_ERROR, "http filter failed: size of the input and received frames must be equal!\n");
+        return AVERROR(EINVAL);
+        }
+        /* fill frame data from received buffer */
+        av_image_fill_arrays(out->data, out->linesize, chunk.memory, out->format, out->width, out->height, 1);
+    }
+       
+        /* cleanup */ 
+        curl_url_cleanup(url);
+        av_free(buffer);
+        av_frame_free(&in);
+    return ff_filter_frame(outlink, out);
+} 
+
+static const AVFilterPad avfilter_vf_http_inputs[] = {
+    {
+        .name = "default",
+        .type = AVMEDIA_TYPE_VIDEO,
+        .filter_frame = filter_frame,        
+    },
+    { NULL }
+};
+
+static const AVFilterPad avfilter_vf_http_outputs[] = {
+    {
+        .name = "default",
+        .type = AVMEDIA_TYPE_VIDEO,
+    },
+    { NULL }
+};
+
+AVFilter ff_vf_http = {
+    .name          = "http",
+    .description   = NULL_IF_CONFIG_SMALL("Send raw frame data to the remote server for postprocessing and await response as new frame in same format and size."),
+    .priv_size     = sizeof(HttpContext),
+    .init          = init,
+    .uninit        = uninit,
+    .query_formats = query_formats,
+    .inputs        = avfilter_vf_http_inputs,
+    .outputs       = avfilter_vf_http_outputs,
+    .priv_class    = &http_class,
+};