Message ID | 1599782008.655533000.a5lxms1z@frv55.fwdcdn.com |
---|---|
State | New |
Headers | show |
Series | [FFmpeg-devel] avfilter: add http video filter. | expand |
Context | Check | Description |
---|---|---|
andriy/default | pending | |
andriy/make | success | Make finished |
andriy/make_fate | success | Make fate finished |
Entirely outside of this filter being acceptable at all, pulling in libcurl is not an option without major safeguards for every single overlapping tls library both ffmpeg and curl support. If both are using the same library, there is a huge potential for libcurl being linked at all breaks ffmpegs ability to talk TLS. On top of that, ffmpeg already has code to talk to http servers, so pulling in a library to do it is even less acceptable. I also really fail to see the utility of this filter. What's stopping you from just making ffmpeg output raw frames and then sending them off via the curl cli tool or whatever else?
Timo Rothenpieler (12020-09-11): > Entirely outside of this filter being acceptable at all, > pulling in libcurl is not an option without major safeguards for every > single overlapping tls library both ffmpeg and curl support. > If both are using the same library, there is a huge potential for libcurl > being linked at all breaks ffmpegs ability to talk TLS. > > On top of that, ffmpeg already has code to talk to http servers, so pulling > in a library to do it is even less acceptable. I completely agree with this. > I also really fail to see the utility of this filter. What's stopping you > from just making ffmpeg output raw frames and then sending them off via the > curl cli tool or whatever else? This would happen at the end of the filter graph. This patch is for something in the middle. For a complex graph, I do not think there is an easy way of implementing with just command-line tools. We would need some kind of movie sink for that. But this is way too specific to be accepted. Hard-coded URL parameters, nothing to set the formats, not even the possibility of a filter that changes the resolution. This is exactly what I was warning about in this mail: https://ffmpeg.org/pipermail/ffmpeg-devel/2020-September/269348.html And do not let us forget that the coding style is not at all what we do. Regards,
> Hard-coded URL parameters, >nothing to set the formats, not even the possibility of a filter that >changes the resolution. This filter has options: url and content-type header for requests and it's not hardcoded. Format and resolution can be changed after this filter later in filter graph, it's not a job for this filter, for example: ffmpeg -i video.mp4 -vf scale=1920:-1,format=rgb24,http=url="http://localhost:3000/frame?type=ffmpeg":content_type="application/octet-stream",scale=1280:-1,format=yuv420p -c:v h264 -y out.mp4 And it will be send raw frames to servers with additional parameters in url query, like that: "POST http://localhost:3000/frame?type=ffmpeg&width=1280&height=720&format=2&linesize=344336064&size=2764800&pts=22012 For my opportunity this filter open door for postprocess frames outside of ffmpeg, so filter can be written or at least prototyped in other languages: js, python, go, etc without touching ffmpeg code and quick start for people who don't know c language or may be script is to big for porting to ffmpeg code. For my use case I will be used it to "connect" ffmpeg to python (this script is huge and complicated to porting to c language and ffmpeg) script that postprocess frame/image in different process (docker container). Without this http filter I must do the same stuff in multiple steps, for example (it's typical usage): #1 use ffmpeg to extract jpg's images for every frame from video and save it to folder on disk #2 run custom python script on every images in folder and create output images #3 use ffmpeg to marge images into video and do something to sink movie But instead of multistage flow I can convert video in single step with injection of "http" filter in filter graph like in example. --- Original message --- From: "Nicolas George" <george@nsup.org> Date: 11 September 2020, 12:40:23 Timo Rothenpieler (12020-09-11): > Entirely outside of this filter being acceptable at all, > pulling in libcurl is not an option without major safeguards for every > single overlapping tls library both ffmpeg and curl support. > If both are using the same library, there is a huge potential for libcurl > being linked at all breaks ffmpegs ability to talk TLS. > > On top of that, ffmpeg already has code to talk to http servers, so pulling > in a library to do it is even less acceptable. I completely agree with this. > I also really fail to see the utility of this filter. What's stopping you > from just making ffmpeg output raw frames and then sending them off via the > curl cli tool or whatever else? This would happen at the end of the filter graph. This patch is for something in the middle. For a complex graph, I do not think there is an easy way of implementing with just command-line tools. We would need some kind of movie sink for that. But this is way too specific to be accepted. Hard-coded URL parameters, nothing to set the formats, not even the possibility of a filter that changes the resolution. This is exactly what I was warning about in this mail: https://ffmpeg.org/pipermail/ffmpeg-devel/2020-September/269348.html And do not let us forget that the coding style is not at all what we do. Regards,
Alex (12020-09-11): > This filter has options: url and content-type header for requests and it's not hardcoded. The format for the parameters is hard-coded. > Format and resolution can be changed after this filter later in filter > graph, it's not a job for this filter, for example: It is a job for this filter if the external processing requires it. As it is, the external processing must return frames with the same format and resolution as its input, which is a completely unacceptable limitation. > For my opportunity this filter open door for postprocess frames outside of ffmpeg We do not accept patches for somebody's opportunity. Patches need to be generic enough to work for many people. The hard-coding of HTTP, the hard-coding of the parameters passing, the arbitrary limitations make this unacceptable. The sooner you accept it and start thinking more globally, the sooner you can consider making something that would be accepted. I have given an indication: what you need a movie sink. Regards,
>The format for the parameters is hard-coded.Please, tell me more I don't understand that do You mean? --- Original message --- From: "Nicolas George" <george@nsup.org> Date: 11 September 2020, 14:09:47 Alex (12020-09-11): > This filter has options: url and content-type header for requests and it's not hardcoded. The format for the parameters is hard-coded. > Format and resolution can be changed after this filter later in filter > graph, it's not a job for this filter, for example: It is a job for this filter if the external processing requires it. As it is, the external processing must return frames with the same format and resolution as its input, which is a completely unacceptable limitation. > For my opportunity this filter open door for postprocess frames outside of ffmpeg We do not accept patches for somebody's opportunity. Patches need to be generic enough to work for many people. The hard-coding of HTTP, the hard-coding of the parameters passing, the arbitrary limitations make this unacceptable. The sooner you accept it and start thinking more globally, the sooner you can consider making something that would be accepted. I have given an indication: what you need a movie sink. Regards,
> The hard-coding of HTTP, the hard-coding of the parameters passing, the arbitrary limitations make> this unacceptable. Can You explain more, because I not fully understand "hard-coding " params, http. > It is a job for this filter if the external processing requires it. As > it is, the external processing must return frames with the same format > and resolution as its input, which is a completely unacceptable > limitation I found way for fixing that, server must set headers with new image params in response and then it's possible to create new frame with right width, height, pix format, etc. I will fix it. --- Original message --- From: "Nicolas George" <george@nsup.org> Date: 11 September 2020, 14:09:47 Alex (12020-09-11): > This filter has options: url and content-type header for requests and it's not hardcoded. The format for the parameters is hard-coded. > Format and resolution can be changed after this filter later in filter > graph, it's not a job for this filter, for example: It is a job for this filter if the external processing requires it. As it is, the external processing must return frames with the same format and resolution as its input, which is a completely unacceptable limitation. > For my opportunity this filter open door for postprocess frames outside of ffmpeg We do not accept patches for somebody's opportunity. Patches need to be generic enough to work for many people. The hard-coding of HTTP, the hard-coding of the parameters passing, the arbitrary limitations make this unacceptable. The sooner you accept it and start thinking more globally, the sooner you can consider making something that would be accepted. I have given an indication: what you need a movie sink. Regards,
Alex (12020-09-11):
> Can You explain more, because I not fully understand "hard-coding " params, http.
Where does it say that the size and format of the frame needs to be
passed as "width=1280&height=720&format=1" rather than
"1280x720/yuv420p"?
In fact, where did you find that using the numeric value of a codec id
like that was a good idea?
You are not adding a generic filter, not even a generic HTTP filter, you
are defining your own private protocol op top of HTTP, without even
realizing it, without properly documenting it, and making it
unacceptably limited (where is the time base? where is the aspect
ratio?).
There are already many protocols and formats that already do all of it,
properly.
Regards,
diff --git a/Changelog b/Changelog index cd8be931ef..777cca679a 100644 --- a/Changelog +++ b/Changelog @@ -22,6 +22,7 @@ version <next>: - MODS demuxer - PhotoCD decoder - MCA demuxer +- http video filter, send raw frames to remote url for postprocessing version 4.3: diff --git a/configure b/configure index ae8c6e61c8..f5131d4669 100755 --- a/configure +++ b/configure @@ -325,6 +325,7 @@ External library support: --enable-vulkan enable Vulkan code [no] --disable-xlib disable xlib [autodetect] --disable-zlib disable zlib [autodetect] + --enable-libcurl enable http filter that send raw frames to remote server The following libraries provide various hardware acceleration features: --disable-amf disable AMF video encoding code [autodetect] @@ -1827,6 +1828,7 @@ EXTERNAL_LIBRARY_LIST=" opengl pocketsphinx vapoursynth + libcurl " HWACCEL_AUTODETECT_LIBRARY_LIST=" @@ -3650,6 +3652,7 @@ vpp_qsv_filter_select="qsvvpp" xfade_opencl_filter_deps="opencl" yadif_cuda_filter_deps="ffnvcodec" yadif_cuda_filter_deps_any="cuda_nvcc cuda_llvm" +http_filter_deps="libcurl" # examples avio_list_dir_deps="avformat avutil" @@ -6367,6 +6370,7 @@ enabled libmysofa && { check_pkg_config libmysofa libmysofa mysofa.h mys enabled libnpp && { check_lib libnpp npp.h nppGetLibVersion -lnppig -lnppicc -lnppc -lnppidei || check_lib libnpp npp.h nppGetLibVersion -lnppi -lnppc -lnppidei || die "ERROR: libnpp not found"; } +enabled libcurl && require "libcurl >= 7.68.0" curl/curl.h curl_easy_init -lcurl enabled libopencore_amrnb && require libopencore_amrnb opencore-amrnb/interf_dec.h Decoder_Interface_init -lopencore-amrnb enabled libopencore_amrwb && require libopencore_amrwb opencore-amrwb/dec_if.h D_IF_init -lopencore-amrwb enabled libopencv && { check_headers opencv2/core/core_c.h && diff --git a/doc/filters.texi b/doc/filters.texi index cbb16f22b2..660ef8b4d9 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -12078,6 +12078,34 @@ For example, to horizontally flip the input video with @command{ffmpeg}: ffmpeg -i in.avi -vf "hflip" out.avi @end example +@anchor{http} +@section http + +Send raw frame data to the remote server for postprocessing and await response as new frame in same format and size. To enable filter configure ffmpeg with @code{./configure --enable-libcurl}. + +The filter accepts the following options: + +@table @option +@item url +Specify remote server url location. + +@item content_type +Specify content-type header in request header, default to "application/octet-stream". +@end table + +Simple demo http server for postprocessing frames can be found here: @url{https://github.com/devalexqt/simple_ffmpeg_http_filter_server} + +@subsection Examples +@itemize + +@item +Send raw frames to "http://localhost:3000/frame?param=abc" + +@example +ffmpeg -i input.mp4 -vf format=rgb24,http=url="http\\\://localhost\\\:3000/frame?param=abc":content_type=application/octet-stream -t 10 out.mp4 +@end example +@end itemize + @section histeq This filter applies a global color histogram equalization on a per-frame basis. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index e6d3c283da..38eb1f4204 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -467,6 +467,7 @@ OBJS-$(CONFIG_YAEPBLUR_FILTER) += vf_yaepblur.o OBJS-$(CONFIG_ZMQ_FILTER) += f_zmq.o OBJS-$(CONFIG_ZOOMPAN_FILTER) += vf_zoompan.o OBJS-$(CONFIG_ZSCALE_FILTER) += vf_zscale.o +OBJS-$(CONFIG_HTTP_FILTER) += vf_http.o OBJS-$(CONFIG_ALLRGB_FILTER) += vsrc_testsrc.o OBJS-$(CONFIG_ALLYUV_FILTER) += vsrc_testsrc.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index fa91e608e4..8626fbc331 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -445,6 +445,7 @@ extern AVFilter ff_vf_yaepblur; extern AVFilter ff_vf_zmq; extern AVFilter ff_vf_zoompan; extern AVFilter ff_vf_zscale; +extern AVFilter ff_vf_http; extern AVFilter ff_vsrc_allrgb; extern AVFilter ff_vsrc_allyuv; diff --git a/libavfilter/version.h b/libavfilter/version.h index 308fbe07c3..b8ba489da7 100644 --- a/libavfilter/version.h +++ b/libavfilter/version.h @@ -30,7 +30,7 @@ #include "libavutil/version.h" #define LIBAVFILTER_VERSION_MAJOR 7 -#define LIBAVFILTER_VERSION_MINOR 87 +#define LIBAVFILTER_VERSION_MINOR 88 #define LIBAVFILTER_VERSION_MICRO 100 diff --git a/libavfilter/vf_http.c b/libavfilter/vf_http.c new file mode 100644 index 0000000000..f4290ca254 --- /dev/null +++ b/libavfilter/vf_http.c @@ -0,0 +1,221 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * http video filter + */ + +#include "libavutil/avassert.h" +#include "libavutil/opt.h" +#include "libavutil/internal.h" +#include "libavutil/imgutils.h" +#include "libavutil/avstring.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" +#include "stdio.h" +#include "curl/curl.h" + +struct MemoryStruct { + uint8_t *memory; + size_t size; +}; + +typedef struct HttpContext { + const AVClass *class; + char *url; + char *content_type; + CURL *curl; + struct curl_slist *headers; +} HttpContext; + +#define OFFSET(x) offsetof(HttpContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM + +static const AVOption http_options[] = { + { "url", "set remote url address", OFFSET(url), AV_OPT_TYPE_STRING, {.str=NULL}, FLAGS }, + { "content_type", "set 'Content-Type' request header", OFFSET(content_type), AV_OPT_TYPE_STRING, {.str="application/octet-stream"}, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(http); + +static av_cold int init(AVFilterContext *ctx) +{ + HttpContext *http = ctx->priv; + http->curl = curl_easy_init(); + + /* check if remote server url is valid formated */ + CURLU *url= curl_url(); + CURLUcode result; + + /* parse a full URL */ + result = curl_url_set(url, CURLUPART_URL, http->url, 0); + if(result){ + av_log(NULL, AV_LOG_ERROR, "http filter failed: invalid input url!\n"); + return AVERROR(EINVAL); + } + + /* set request headers */ + http->headers=NULL; + http->headers = curl_slist_append(http->headers, av_asprintf("Content-Type: %s",http->content_type)); + http->headers = curl_slist_append(http->headers, "Expect:"); + curl_easy_setopt(http->curl, CURLOPT_HTTPHEADER, http->headers); + curl_easy_setopt(http->curl, CURLOPT_FOLLOWLOCATION, 1L); + + curl_url_cleanup(url); + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + HttpContext *http = ctx->priv; + + curl_easy_cleanup(http->curl); + curl_slist_free_all(http->headers); + curl_global_cleanup(); +} + +static int query_formats(AVFilterContext *ctx) +{ + const HttpContext *http = ctx->priv; + return ff_default_query_formats(ctx); +} + +static size_t WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp) +{ + size_t realsize = size * nmemb; + struct MemoryStruct *mem = (struct MemoryStruct *)userp; + + char *ptr = realloc(mem->memory, mem->size + realsize + 1); + if(!ptr) { + /* out of memory! */ + return AVERROR(ENOMEM); + } + + mem->memory = ptr; + memcpy(&(mem->memory[mem->size]), contents, realsize); + mem->size += realsize; + mem->memory[mem->size] = 0; + + return realsize; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + HttpContext *http = inlink->dst->priv; + AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) { + av_frame_free(&in); + return AVERROR(ENOMEM); + } + av_frame_copy_props(out, in); + + CURL *curl=http->curl; + CURLU *url= curl_url(); + CURLcode res; + struct MemoryStruct chunk; + chunk.memory = av_malloc(1); + chunk.size = 0; + + /* create buffer and copy input frame */ + int buff_size=av_image_get_buffer_size(in->format, in->width, in->height, 1); + uint8_t *buffer=av_malloc(buff_size); + av_image_copy_to_buffer(buffer, buff_size, in->data, in->linesize, in->format, in->width, in->height, 1); + + /* update url query */ + CURLUcode result; + const char *url_params=av_asprintf("width=%d&height=%d&format=%d&size=%d&pts=%ld",in->width, in->height, in->format, buff_size, in->pts); + curl_url_set(url, CURLUPART_URL, http->url, 0); + result = curl_url_set(url, CURLUPART_QUERY, url_params, CURLU_APPENDQUERY); + if(result){ + av_log(NULL, AV_LOG_ERROR, "http filter failed: failed to setup url query params!\n"); + return AVERROR(EINVAL); + } + + /* specify the POST data */ + curl_easy_setopt(http->curl, CURLOPT_CURLU,url); + curl_easy_setopt(curl, CURLOPT_POSTFIELDS, buffer); + curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, buff_size); + + /* send all data to this function */ + curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback); + + /* we pass our 'chunk' struct to the callback function */ + curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&chunk); + curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/1.0"); + + /* Perform the request, res will get the return code */ + res = curl_easy_perform(curl); + + /* Check for errors */ + if(res != CURLE_OK){ + av_log(NULL, AV_LOG_ERROR, "http filter failed: %s\n", curl_easy_strerror(res)); + av_frame_free(&in); + av_frame_free(&out); + free(buffer); + return AVERROR(EINVAL); + } + else{ + if(chunk.size!=buff_size){ + av_log(NULL, AV_LOG_ERROR, "http filter failed: size of the input and received frames must be equal!\n"); + return AVERROR(EINVAL); + } + /* fill frame data from received buffer */ + av_image_fill_arrays(out->data, out->linesize, chunk.memory, out->format, out->width, out->height, 1); + } + + /* cleanup */ + curl_url_cleanup(url); + av_free(buffer); + av_frame_free(&in); + return ff_filter_frame(outlink, out); +} + +static const AVFilterPad avfilter_vf_http_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + }, + { NULL } +}; + +static const AVFilterPad avfilter_vf_http_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, + { NULL } +}; + +AVFilter ff_vf_http = { + .name = "http", + .description = NULL_IF_CONFIG_SMALL("Send raw frame data to the remote server for postprocessing and await response as new frame in same format and size."), + .priv_size = sizeof(HttpContext), + .init = init, + .uninit = uninit, + .query_formats = query_formats, + .inputs = avfilter_vf_http_inputs, + .outputs = avfilter_vf_http_outputs, + .priv_class = &http_class, +};
Signed-off-by: alex_qt <3.14pi@ukr.net> --- Changelog | 1 + configure | 4 + doc/filters.texi | 28 +++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/version.h | 2 +- libavfilter/vf_http.c | 221 +++++++++++++++++++++++++++++++++++++++ 7 files changed, 257 insertions(+), 1 deletion(-) create mode 100644 libavfilter/vf_http.c