From patchwork Wed Jul 15 08:37:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21033 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id EE8BA44B9FC for ; Wed, 15 Jul 2020 11:37:45 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C548768B4B5; Wed, 15 Jul 2020 11:37:45 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 72DD1688082 for ; Wed, 15 Jul 2020 11:37:38 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 2E5401008CBCD; Wed, 15 Jul 2020 16:37:34 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 2B37A200B4499; Wed, 15 Jul 2020 16:37:34 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id B0NgwXpUHmnO; Wed, 15 Jul 2020 16:37:33 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id CB129200B4496; Wed, 15 Jul 2020 16:37:32 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Jul 2020 16:37:28 +0800 Message-Id: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v1 1/6] avformat/abr: Adaptive Bitrate support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add abr module for hls/dash. v1 fixed: 1. add an "ff" prefix to the protocol name to mark it internal. 2. use 1.2f for float constant 1.2. 3. simplify abr_seek for we just need AVSEEK_SIZE only. Signed-off-by: spartazhc --- doc/protocols.texi | 7 ++ libavformat/Makefile | 1 + libavformat/abr.c | 250 ++++++++++++++++++++++++++++++++++++++++ libavformat/protocols.c | 1 + 4 files changed, 259 insertions(+) create mode 100644 libavformat/abr.c diff --git a/doc/protocols.texi b/doc/protocols.texi index 64ad3f05d6..ffbb36147e 100644 --- a/doc/protocols.texi +++ b/doc/protocols.texi @@ -51,6 +51,13 @@ in microseconds. A description of the currently available protocols follows. +@section abr + +Adaptive bitrate sub-protocol work for hls/dash. + +The abr protocol takes stream information from hls/dash as input, +use bandwidth estimation to decide whether to switch or not. + @section amqp Advanced Message Queueing Protocol (AMQP) version 0-9-1 is a broker based diff --git a/libavformat/Makefile b/libavformat/Makefile index 26af859a28..3c08289d5e 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -588,6 +588,7 @@ OBJS-$(CONFIG_LIBOPENMPT_DEMUXER) += libopenmpt.o OBJS-$(CONFIG_VAPOURSYNTH_DEMUXER) += vapoursynth.o # protocols I/O +OBJS-$(CONFIG_ABR_PROTOCOL) += abr.o OBJS-$(CONFIG_ASYNC_PROTOCOL) += async.o OBJS-$(CONFIG_APPLEHTTP_PROTOCOL) += hlsproto.o OBJS-$(CONFIG_BLURAY_PROTOCOL) += bluray.o diff --git a/libavformat/abr.c b/libavformat/abr.c new file mode 100644 index 0000000000..e201d3edda --- /dev/null +++ b/libavformat/abr.c @@ -0,0 +1,250 @@ +/* + * Adaptive Bitrate Module for HLS / DASH + * Copyright (c) 2020 Hongcheng Zhong + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "avformat.h" +#include "libavutil/opt.h" +#include "libavutil/time.h" +#include "libavutil/avstring.h" +#include "url.h" +#include + +enum ABRFormatType { + ABR_TYPE_HLS, + ABR_TYPE_DASH +}; + +typedef struct variant_bitrate { + int value; + int index; +} variant_bitrate; + +typedef struct ABRContext { + AVClass *class; + URLContext *hd; + AVDictionary *abr_params; + AVDictionary *abr_metadata; + enum ABRFormatType format; + int cur_pls; + int can_switch; + int n_variants; + variant_bitrate *variants_bitrate; + int index; + int n_throughputs; + float *throughputs; +} ABRContext; + +static float harmonic_mean(int num, float* arr) +{ + float tmp = 0; + + if (num <= 0) return 0; + + for (size_t i = 0; i < num; i++) { + tmp += 1 / arr[i]; + } + + return num / tmp; +} + +static int hls_param_parse(ABRContext *c, const char *key, const char *value) +{ + if (!av_strcasecmp(key, "cur_pls")) { + c->cur_pls = atoi(value); + } else if (!av_strcasecmp(key, "can_switch")) { + c->can_switch = atoi(value); + } else if (!av_strcasecmp(key, "n_variants")) { + c->n_variants = atoi(value); + c->variants_bitrate = av_mallocz(sizeof(variant_bitrate) * c->n_variants); + if (!c->variants_bitrate) + return -1; + } else if (av_strstart(key, "variant_bitrate", NULL)) { + c->variants_bitrate[c->index].value = atoi(value); + c->variants_bitrate[c->index].index = c->index; + c->index++; + } else if (!av_strcasecmp(key, "n_throughputs")) { + c->n_throughputs = atoi(value); + c->index = 0; + if (c->n_throughputs > 0) { + c->throughputs = av_malloc(sizeof(float) * c->n_throughputs); + if (!c->throughputs) + return -1; + } + } else if (av_strstart(key, "throughputs", NULL)) + c->throughputs[c->index++] = atof(value); + return 0; +} + +static int dash_param_parse(ABRContext *c, const char *key, const char *value) +{ + return 0; +} + +static int abr_param_parse(ABRContext *c, enum ABRFormatType type, const char *key, const char *value) +{ + if (type == ABR_TYPE_HLS) { + hls_param_parse(c, key, value); + } else if (type == ABR_TYPE_DASH) { + dash_param_parse(c, key, value); + } + return 0; +} + +static int compare_vb(const void *a, const void *b) +{ + variant_bitrate *a1 = (variant_bitrate *)a; + variant_bitrate *a2 = (variant_bitrate *)b; + return (*a2).value - (*a1).value; +} + +static int abr_rule(ABRContext *c, float bw_estimate) +{ + int ret = -1; + + if (c->n_throughputs > 6) { + if (bw_estimate < c->variants_bitrate[c->cur_pls].value / 1000 * 1.2f && + bw_estimate > c->variants_bitrate[c->cur_pls].value / 1000 * 0.8f) + return -1; + qsort(c->variants_bitrate, c->n_variants, sizeof(variant_bitrate), compare_vb); + for (int i = 0; i < c->n_variants; i++) { + if (bw_estimate > c->variants_bitrate[i].value / 1000) { + ret = i; + break; + } + } + } + if (ret == c->cur_pls) + ret = -1; + av_log(c, AV_LOG_VERBOSE, "[switch] bwe=%.2fkbps, cur=%d, switch=%d\n", bw_estimate, c->cur_pls, ret); + return ret; +} + +static int abr_open(URLContext *h, const char *uri, int flags, AVDictionary **options) +{ + const char *nested_url; + uint64_t start, end; + float bw_estimation; + int switch_request = -1; + int ret = 0; + ABRContext *c = h->priv_data; + + if (!av_strstart(uri, "ffabr+", &nested_url) && + !av_strstart(uri, "ffabr:", &nested_url)) { + av_log(h, AV_LOG_ERROR, "Unsupported url %s\n", uri); + ret = AVERROR(EINVAL); + goto err; + } + + AVDictionaryEntry *en = NULL; + en = av_dict_get(c->abr_params, "", en, AV_DICT_IGNORE_SUFFIX); + if (!en) + return -1; + if (!av_strcasecmp(en->key, "format")) { + if (!av_strcasecmp(en->value, "hls")) { + c->format = ABR_TYPE_HLS; + } else if (!av_strcasecmp(en->value, "dash")) { + c->format = ABR_TYPE_DASH; + } + av_log(h, AV_LOG_VERBOSE, "%s is using ABR\n", en->value); + } + + while (en = av_dict_get(c->abr_params, "", en, AV_DICT_IGNORE_SUFFIX)) { + if (abr_param_parse(c, c->format, en->key, en->value) < 0) + av_log(h, AV_LOG_WARNING,"Error parsing option '%s = %s'.\n", + en->key, en->value); + } + + start = av_gettime(); + if ((ret = ffurl_open_whitelist(&c->hd, nested_url, flags, + &h->interrupt_callback, options, + h->protocol_whitelist, h->protocol_blacklist, h)) < 0) { + av_log(h, AV_LOG_ERROR, "Unable to open resource: %s\n", nested_url); + goto err; + } + end = av_gettime(); + + bw_estimation = harmonic_mean(c->n_throughputs, c->throughputs); + + if (c->can_switch == 1) + switch_request = abr_rule(c, bw_estimation); + + av_dict_set_int(&c->abr_metadata, "download_time", (end - start), 0); + av_dict_set_int(&c->abr_metadata, "switch_request", switch_request, 0); + +err: + return ret; +} + + +static int abr_read(URLContext *h, uint8_t *buf, int size) +{ + ABRContext *c = h->priv_data; + + return ffurl_read(c->hd, buf, size); +} + +static int64_t abr_seek(URLContext *h, int64_t pos, int whence) +{ + ABRContext *c = h->priv_data; + int64_t ret; + + if (whence == AVSEEK_SIZE) { + ret = ffurl_seek(c->hd, pos, AVSEEK_SIZE); + return ret < 0 ? AVERROR(errno) : ret; + } + + return AVERROR(errno); +} + +static int abr_close(URLContext *h) +{ + ABRContext *c = h->priv_data; + int ret = 0; + + ffurl_closep(&c->hd); + av_free(c->variants_bitrate); + av_free(c->throughputs); + return ret; +} + +#define OFFSET(x) offsetof(ABRContext, x) +#define D AV_OPT_FLAG_DECODING_PARAM +static const AVOption ffabr_options[] = { + { "abr-params", "Informations ABR needed, using a :-separated list of key=value parameters", OFFSET(abr_params), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { "abr-metadata", "Metadata return from abr, including switch signal and network bandwidth", OFFSET(abr_metadata), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { NULL } +}; + +static const AVClass ffabr_class = { + .class_name = "ffabr", + .item_name = av_default_item_name, + .option = ffabr_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +const URLProtocol ff_ffabr_protocol = { + .name = "ffabr", + .url_open2 = abr_open, + .url_read = abr_read, + .url_seek = abr_seek, + .url_close = abr_close, + .priv_data_size = sizeof(ABRContext), + .priv_data_class = &ffabr_class, +}; diff --git a/libavformat/protocols.c b/libavformat/protocols.c index 7df18fbb3b..1d6af8e380 100644 --- a/libavformat/protocols.c +++ b/libavformat/protocols.c @@ -29,6 +29,7 @@ extern const URLProtocol ff_cache_protocol; extern const URLProtocol ff_concat_protocol; extern const URLProtocol ff_crypto_protocol; extern const URLProtocol ff_data_protocol; +extern const URLProtocol ff_ffabr_protocol; extern const URLProtocol ff_ffrtmpcrypt_protocol; extern const URLProtocol ff_ffrtmphttp_protocol; extern const URLProtocol ff_file_protocol; From patchwork Wed Jul 15 08:37:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21034 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id ABFBB44B9FC for ; Wed, 15 Jul 2020 11:37:46 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 996C968B4AC; Wed, 15 Jul 2020 11:37:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7546F68B485 for ; Wed, 15 Jul 2020 11:37:38 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 22CE01008CBCE; Wed, 15 Jul 2020 16:37:35 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 1F1F9200B448D; Wed, 15 Jul 2020 16:37:35 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id w4x_J_NGyxKB; Wed, 15 Jul 2020 16:37:35 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 4CCD2200B4498; Wed, 15 Jul 2020 16:37:33 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Jul 2020 16:37:29 +0800 Message-Id: <20200715083733.101880-2-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v1 2/6] avformat/http: Add abr to whitelist X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc add abr protocol to http's whitelist Signed-off-by: spartazhc --- libavformat/http.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavformat/http.c b/libavformat/http.c index 6c39da1a8b..b77517ff51 100644 --- a/libavformat/http.c +++ b/libavformat/http.c @@ -1815,7 +1815,7 @@ const URLProtocol ff_http_protocol = { .priv_data_size = sizeof(HTTPContext), .priv_data_class = &http_context_class, .flags = URL_PROTOCOL_FLAG_NETWORK, - .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data" + .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data,ffabr" }; #endif /* CONFIG_HTTP_PROTOCOL */ From patchwork Wed Jul 15 08:37:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21035 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 8B63E44B9FC for ; Wed, 15 Jul 2020 11:37:48 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 69B4068B4FF; Wed, 15 Jul 2020 11:37:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 40F7268B485 for ; Wed, 15 Jul 2020 11:37:39 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id A88211008CBCF; Wed, 15 Jul 2020 16:37:36 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id A613F200B4496; Wed, 15 Jul 2020 16:37:36 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id wltf-kAlUPVi; Wed, 15 Jul 2020 16:37:36 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id CDAEA200B448D; Wed, 15 Jul 2020 16:37:35 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Jul 2020 16:37:30 +0800 Message-Id: <20200715083733.101880-3-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v1 3/6] avformat/hls: use abr to switch streams X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc When abr is enable, it will take over the task to call http to download segments, and will return a switch-request for hls to switch streams. For reason not to waste segments that have been downloaded, switch will become effective after old segments is used out. Abr cannot work with http_persistent option, and currently use http_multiple. v1 fixed: 1. fix memory leak Signed-off-by: spartazhc --- doc/demuxers.texi | 3 + libavformat/hls.c | 232 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 229 insertions(+), 6 deletions(-) diff --git a/doc/demuxers.texi b/doc/demuxers.texi index 3c15ab9eee..4cdbd95962 100644 --- a/doc/demuxers.texi +++ b/doc/demuxers.texi @@ -321,6 +321,9 @@ available in a metadata key named "variant_bitrate". It accepts the following options: @table @option +@item abr +enable abr to switch streams. + @item live_start_index segment index to start live streams at (negative values are from the end). diff --git a/libavformat/hls.c b/libavformat/hls.c index ba17c4ed96..877bd8c677 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -189,6 +189,17 @@ struct variant { char subtitles_group[MAX_FIELD_LEN]; }; +struct throughput { + int n_throughputs; + + /* throughputs are in kbps, always record the last 20 times + * set record 20 times refer to paper FESTIVE https://doi.org/10.1145/2413176.2413189 + */ + float throughput_fifo[20]; + int head; + int tail; +}; + typedef struct HLSContext { AVClass *class; AVFormatContext *ctx; @@ -213,8 +224,36 @@ typedef struct HLSContext { int http_multiple; int http_seekable; AVIOContext *playlist_pb; + + int abr; + struct throughput *throughputs; + int can_switch; + int switch_request2; + int switch_delay; + int64_t switch_timestamp; + int64_t delta_timestamp; + int cur_pls; } HLSContext; +static struct segment *next_segment(struct playlist *pls); +static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in); + +static void sync_cur_seq(HLSContext *c) { + int i; + for (i = 0; i < c->n_playlists; i++) { + struct playlist *pls = c->playlists[i]; + pls->cur_seq_no = c->cur_seq_no; + } +} + +static struct segment *next2_segment(struct playlist *pls) +{ + int n = pls->cur_seq_no - pls->start_seq_no + 2; + if (n >= pls->n_segments) + return NULL; + return pls->segments[n]; +} + static void free_segment_dynarray(struct segment **segments, int n_segments) { int i; @@ -624,6 +663,31 @@ static int open_url_keepalive(AVFormatContext *s, AVIOContext **pb, #endif } +static int update_throughputs(struct throughput *thr, float time, int pb_size) +{ + if (pb_size <= 0 || time <= 0) + return -1; + if (thr->n_throughputs < 20) { + ++thr->n_throughputs; + } else { + ++thr->head; + } + thr->throughput_fifo[thr->tail] = (float)(pb_size) / time; + thr->tail = (thr->tail + 1) % 20; + return 0; +} + +static int64_t get_switch_timestamp(HLSContext *c, struct playlist *pls) +{ + int64_t pos = c->first_timestamp == AV_NOPTS_VALUE ? + 0 : c->first_timestamp; + + for (int i = 0; i < pls->cur_seq_no + 2; i++) { + pos += pls->segments[i]->duration; + } + return pos; +} + static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, AVDictionary *opts, AVDictionary *opts2, int *is_http_out) { @@ -639,6 +703,9 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, } else if (av_strstart(url, "data", NULL)) { if (url[4] == '+' || url[4] == ':') proto_name = avio_find_protocol_name(url + 5); + } else if (av_strstart(url, "ffabr", NULL)) { + if (url[5] == '+' || url[5] == ':') + proto_name = avio_find_protocol_name(url + 6); } if (!proto_name) @@ -669,6 +736,8 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, ; else if (av_strstart(url, "data", NULL) && !strncmp(proto_name, url + 5, strlen(proto_name)) && url[5 + strlen(proto_name)] == ':') ; + else if (av_strstart(url, "ffabr", NULL) && !strncmp(proto_name, url + 6, strlen(proto_name)) && url[6 + strlen(proto_name)] == ':') + ; else if (strcmp(proto_name, "file") || !strncmp(url, "file,", 5)) return AVERROR_INVALIDDATA; @@ -690,6 +759,43 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, } else { ret = s->io_open(s, pb, url, AVIO_FLAG_READ, &tmp); } + if (c->abr && ret >= 0) { + AVDictionary *abr_ret = NULL; + AVDictionaryEntry *en = NULL; + struct segment *seg; + int pb_size, switch_request; + av_opt_get_dict_val(*pb, "abr-metadata", AV_OPT_SEARCH_CHILDREN, &abr_ret); + if (abr_ret) { + en = av_dict_get(abr_ret, "download_time", NULL, 0); + if (en) { + pb_size = avio_size(*pb); + update_throughputs(c->throughputs, atoi(en->value) / 1000.0, pb_size); + av_log(s, AV_LOG_VERBOSE, "[abr] time=%.2fms, size=%.2fkb\n", atoi(en->value) / 1000.0, pb_size / 1000.0); + } + en = av_dict_get(abr_ret, "switch_request", NULL, 0); + if (en) { + switch_request = atoi(en->value); + av_log(s, AV_LOG_VERBOSE, "[abr] switch request: %s\n", en->value); + } + if (switch_request != -1) { + c->switch_request2 = switch_request; + c->switch_delay = 2; + c->can_switch = 0; + sync_cur_seq(c); + seg = next2_segment(c->playlists[c->switch_request2]); + if (!seg) { + c->switch_request2 = -1; + av_log(s, AV_LOG_INFO, "[abr] no more segment need to switch\n"); + } else { + c->switch_timestamp = get_switch_timestamp(c, c->playlists[c->switch_request2]); + av_log(s, AV_LOG_VERBOSE, "[abr] switch timestamp: %ld\n", c->switch_timestamp); + c->playlists[c->switch_request2]->input_next_requested = 1; + ret = open_input(c, c->playlists[c->switch_request2], seg, &c->playlists[c->switch_request2]->input_next); + } + } + av_dict_free(&abr_ret); + } + } if (ret >= 0) { // update cookies on http response with setcookies. char *new_cookies = NULL; @@ -1219,6 +1325,33 @@ static void intercept_id3(struct playlist *pls, uint8_t *buf, pls->is_id3_timestamped = (pls->id3_mpegts_timestamp != AV_NOPTS_VALUE); } +static int abrinfo_to_dict(HLSContext *c, char **abr_info) +{ + struct throughput *thr = c->throughputs; + char buffer[MAX_URL_SIZE]; + int size, i; + size = snprintf(buffer, sizeof(buffer), "format=hls:"); + size += snprintf(buffer + size, sizeof(buffer) - size, "cur_pls=%d:", c->cur_pls); + size += snprintf(buffer + size, sizeof(buffer) - size, "can_switch=%d:", c->can_switch); + size += snprintf(buffer + size, sizeof(buffer) - size, "n_variants=%d:", c->n_variants); + for (i = 0; i < c->n_variants; i++) { + struct variant *v = c->variants[i]; + size += snprintf(buffer + size, sizeof(buffer) - size, "variant_bitrate%d=%d:", i, v->bandwidth); + } + size += snprintf(buffer + size, sizeof(buffer) - size, "n_throughputs=%d:", thr->n_throughputs); + if (thr->n_throughputs > 0) { + i = thr->head; + do { + size += snprintf(buffer + size, sizeof(buffer) - size, "throughputs%d=%.2f:", i, thr->throughput_fifo[i]); + i = (i + 1) % 20; + } while (i != thr->tail); + } + *abr_info = av_malloc(size); + snprintf(*abr_info, size, "%s", buffer); + av_log(c, AV_LOG_DEBUG, "[abr] abr_info: %s\n", *abr_info); + return 0; +} + static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in) { AVDictionary *opts = NULL; @@ -1238,8 +1371,20 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, av_log(pls->parent, AV_LOG_VERBOSE, "HLS request for url '%s', offset %"PRId64", playlist %d\n", seg->url, seg->url_offset, pls->index); + if (c->abr) { + char *abr_opts; + abrinfo_to_dict(c, &abr_opts); + av_dict_set(&opts, "abr-params", abr_opts, 0); // how to setup flag? + av_free(abr_opts); + } if (seg->key_type == KEY_NONE) { - ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + char url_abr[MAX_URL_SIZE]; + if (c->abr) { + snprintf(url_abr, sizeof(url_abr), "ffabr:%s", seg->url); // + or : tbd + ret = open_url(pls->parent, in, url_abr, c->avio_opts, opts, &is_http); + } else { + ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + } } else if (seg->key_type == KEY_AES_128) { char iv[33], key[33], url[MAX_URL_SIZE]; if (strcmp(seg->key, pls->key_url)) { @@ -1273,6 +1418,7 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, goto cleanup; } ret = 0; + // TODO: Add abr prefix for crypto } else if (seg->key_type == KEY_SAMPLE_AES) { av_log(pls->parent, AV_LOG_ERROR, "SAMPLE-AES encryption is not supported yet\n"); @@ -1435,7 +1581,17 @@ restart: /* Check that the playlist is still needed before opening a new * segment. */ v->needed = playlist_needed(v); - + if (c->abr) { + if (c->switch_request2 == -1) + ; + else if (v->needed && c->switch_request2 != v->index && c->switch_delay <= 0) { + av_log(v->parent, AV_LOG_VERBOSE, "read_data: needed but not download playlist %d ('%s')\n", + v->index, v->url); + return AVERROR_EOF; + } + if (!c->can_switch) + c->can_switch = 1; + } if (!v->needed) { av_log(v->parent, AV_LOG_INFO, "No longer receiving playlist %d ('%s')\n", v->index, v->url); @@ -1530,7 +1686,12 @@ reload: } seg = next_segment(v); - if (c->http_multiple == 1 && !v->input_next_requested && + + // when get switch_request, old stream should stop downloading next seg + if (c->abr && c->switch_request2 != v->index && c->switch_timestamp != AV_NOPTS_VALUE + && ((c->cur_timestamp + seg->duration * 2) >= c->switch_timestamp) ) + ; + else if (c->http_multiple == 1 && !v->input_next_requested && seg && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { ret = open_input(c, v, seg, &v->input_next); if (ret < 0) { @@ -1563,11 +1724,17 @@ reload: return ret; } - if (c->http_persistent && + if (c->http_persistent && !c->abr && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { v->input_read_done = 1; } else { ff_format_io_close(v->parent, &v->input); + if (c->abr) { + if (c->switch_delay > 0) + c->switch_delay--; + av_log(v->parent, AV_LOG_VERBOSE, "read_data: close pls[%d]->input, v->cur_seq_no=%d, c->switch_delay= %d\n", + v->index, v->cur_seq_no, c->switch_delay); + } } v->cur_seq_no++; @@ -1860,6 +2027,17 @@ static int hls_read_header(AVFormatContext *s) c->first_packet = 1; c->first_timestamp = AV_NOPTS_VALUE; c->cur_timestamp = AV_NOPTS_VALUE; + c->switch_request2 = -1; + c->switch_delay = 0; + c->switch_timestamp = AV_NOPTS_VALUE; + c->delta_timestamp = AV_NOPTS_VALUE; + c->can_switch = -1; + + if (c->abr) { + c->http_persistent = 0; + c->http_multiple = 1; + c->throughputs = av_mallocz(sizeof(struct throughput)); + } if ((ret = save_avio_options(s)) < 0) goto fail; @@ -2049,6 +2227,9 @@ static int hls_read_header(AVFormatContext *s) add_metadata_from_renditions(s, pls, AVMEDIA_TYPE_SUBTITLE); } + if (c->abr && c->can_switch == -1) + c->can_switch = 1; + update_noheader_flag(s); return 0; @@ -2057,6 +2238,13 @@ fail: return ret; } +static void change_discard_flags(struct playlist *pls, int flag) +{ + for (int i = 0; i < pls->n_main_streams; i++) { + pls->main_streams[i]->discard = flag; + } +} + static int recheck_discard_flags(AVFormatContext *s, int first) { HLSContext *c = s->priv_data; @@ -2067,6 +2255,22 @@ static int recheck_discard_flags(AVFormatContext *s, int first) for (i = 0; i < c->n_playlists; i++) { struct playlist *pls = c->playlists[i]; + if (c->abr) { + if (c->switch_request2 == -1) + ; + else if (c->switch_request2 == i && c->switch_delay <= 0 + && c->cur_timestamp + c->delta_timestamp >= c->switch_timestamp + && c->switch_timestamp != AV_NOPTS_VALUE) { + av_log(s, AV_LOG_VERBOSE, "[switch point] cur_timestamp:%ld\n", c->cur_timestamp); + change_discard_flags(pls, AVDISCARD_DEFAULT); + c->switch_timestamp = AV_NOPTS_VALUE; + } else if (c->switch_request2 != -1 && c->switch_request2 != i && c->switch_delay <= 0 + && c->cur_timestamp + c->delta_timestamp >= c->switch_timestamp + && c->switch_timestamp != AV_NOPTS_VALUE) { + change_discard_flags(pls, AVDISCARD_ALL); + } + } + cur_needed = playlist_needed(c->playlists[i]); if (pls->broken) { @@ -2077,6 +2281,11 @@ static int recheck_discard_flags(AVFormatContext *s, int first) changed = 1; pls->cur_seq_no = select_cur_seq_no(c, pls); pls->pb.eof_reached = 0; + if (c->abr) { + pls->cur_seq_no = select_cur_seq_no(c, pls) + 1; + avio_flush(&pls->pb); + avformat_flush(pls->ctx); + } if (c->cur_timestamp != AV_NOPTS_VALUE) { /* catch up */ pls->seek_timestamp = c->cur_timestamp; @@ -2084,7 +2293,7 @@ static int recheck_discard_flags(AVFormatContext *s, int first) pls->seek_stream_index = -1; } av_log(s, AV_LOG_INFO, "Now receiving playlist %d, segment %d\n", i, pls->cur_seq_no); - } else if (first && !cur_needed && pls->needed) { + } else if ((first || c->abr) && !cur_needed && pls->needed) { ff_format_io_close(pls->parent, &pls->input); pls->input_read_done = 0; ff_format_io_close(pls->parent, &pls->input_next); @@ -2093,6 +2302,9 @@ static int recheck_discard_flags(AVFormatContext *s, int first) changed = 1; av_log(s, AV_LOG_INFO, "No longer receiving playlist %d\n", i); } + + if (c->abr && changed && cur_needed) + c->cur_pls = i; } return changed; } @@ -2167,7 +2379,13 @@ static int hls_read_packet(AVFormatContext *s, AVPacket *pkt) /* audio elementary streams are id3 timestamped */ fill_timing_for_id3_timestamped_stream(pls); } - + if (c->abr && c->first_timestamp != AV_NOPTS_VALUE && c->delta_timestamp == AV_NOPTS_VALUE && !i) { + c->delta_timestamp = av_rescale_q(pls->pkt.dts, + get_timebase(pls), AV_TIME_BASE_Q) - c->first_timestamp + 1; + av_log(s, AV_LOG_VERBOSE, "[abr] delta_timestamp=%ld, %ld, %ld\n", + c->delta_timestamp, c->first_timestamp,av_rescale_q(pls->pkt.dts, + get_timebase(pls), AV_TIME_BASE_Q)); + } if (c->first_timestamp == AV_NOPTS_VALUE && pls->pkt.dts != AV_NOPTS_VALUE) c->first_timestamp = av_rescale_q(pls->pkt.dts, @@ -2375,6 +2593,8 @@ static int hls_probe(const AVProbeData *p) #define OFFSET(x) offsetof(HLSContext, x) #define FLAGS AV_OPT_FLAG_DECODING_PARAM static const AVOption hls_options[] = { + {"abr", "enable abr to switch streams", + OFFSET(abr), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, FLAGS }, {"live_start_index", "segment index to start live streams at (negative values are from the end)", OFFSET(live_start_index), AV_OPT_TYPE_INT, {.i64 = -3}, INT_MIN, INT_MAX, FLAGS}, {"allowed_extensions", "List of file extensions that hls is allowed to access", From patchwork Wed Jul 15 08:37:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21036 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 820D144B9FC for ; Wed, 15 Jul 2020 11:37:49 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7085368B510; Wed, 15 Jul 2020 11:37:49 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ADEE368921A for ; Wed, 15 Jul 2020 11:37:40 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 2DECB1008CBD3; Wed, 15 Jul 2020 16:37:38 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 2B061200B4496; Wed, 15 Jul 2020 16:37:38 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Antox6CY9-Nl; Wed, 15 Jul 2020 16:37:38 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 5BF11200B448D; Wed, 15 Jul 2020 16:37:36 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Jul 2020 16:37:31 +0800 Message-Id: <20200715083733.101880-4-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v1 4/6] ffplay: add an option to enable abr X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add abr option, ffplay can play hls using abr by: ffplay -i http://xxx/master.m3u8 -abr Structure ABRList is added to save stream type and index, it is used to allow packet_queue_put function to put pkt which from same type(for example: video pkt) but different stream index to queue. Signed-off-by: spartazhc --- doc/ffplay.texi | 2 + fftools/ffplay.c | 145 ++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 138 insertions(+), 9 deletions(-) diff --git a/doc/ffplay.texi b/doc/ffplay.texi index f3761bb12e..6a24542cda 100644 --- a/doc/ffplay.texi +++ b/doc/ffplay.texi @@ -46,6 +46,8 @@ Disable audio. Disable video. @item -sn Disable subtitles. +@item -abr +Enable adaptive bitrate for hls/dash. @item -ss @var{pos} Seek to @var{pos}. Note that in most formats it is not possible to seek exactly, so @command{ffplay} will seek to the nearest seek point to diff --git a/fftools/ffplay.c b/fftools/ffplay.c index d673b8049a..b17b75fa8f 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -201,6 +201,15 @@ typedef struct Decoder { SDL_Thread *decoder_tid; } Decoder; +typedef struct ABRList { + int **audio_list; + int audios; + int **video_list; + int videos; + int **sub_list; + int subs; +} ABRList; + typedef struct VideoState { SDL_Thread *read_tid; AVInputFormat *iformat; @@ -305,6 +314,8 @@ typedef struct VideoState { int last_video_stream, last_audio_stream, last_subtitle_stream; SDL_cond *continue_read_thread; + + ABRList *abr_list; } VideoState; /* options specified by the user */ @@ -356,6 +367,7 @@ static char *afilters = NULL; static int autorotate = 1; static int find_stream_info = 1; static int filter_nbthreads = 0; +static int abr = 0; /* current context */ static int is_full_screen; @@ -1262,6 +1274,29 @@ static void stream_component_close(VideoState *is, int stream_index) } } +static void free_abr_dynarray(int **list, int num) +{ + for (int i = 0; i < num; i++) { + av_free(list[i]); + } +} + +static void free_abr_list(ABRList *abrlist) +{ + if (abrlist->audios) { + free_abr_dynarray(abrlist->audio_list, abrlist->audios); + av_freep(&abrlist->audio_list); + } + if (abrlist->videos) { + free_abr_dynarray(abrlist->video_list, abrlist->videos); + av_freep(&abrlist->video_list); + } + if (abrlist->subs) { + free_abr_dynarray(abrlist->sub_list, abrlist->subs); + av_freep(&abrlist->sub_list); + } +} + static void stream_close(VideoState *is) { /* XXX: use a special url_shutdown call to abort parse cleanly */ @@ -2753,6 +2788,67 @@ static int is_realtime(AVFormatContext *s) return 0; } +static av_cold int abr_init_list(VideoState *is) +{ + int stream_index, *tmp; + AVStream *st; + int nb_streams = is->ic->nb_streams; + ABRList *abrlist = is->abr_list; + + for (stream_index = 0; stream_index < nb_streams; stream_index++) { + st = is->ic->streams[stream_index]; + tmp = av_memdup(&stream_index, sizeof(int)); + if (!tmp) + return -1; + switch (st->codecpar->codec_type) { + case AVMEDIA_TYPE_AUDIO: + av_dynarray_add(&abrlist->audio_list, &abrlist->audios, tmp); + break; + case AVMEDIA_TYPE_VIDEO: + av_dynarray_add(&abrlist->video_list, &abrlist->videos, tmp); + break; + case AVMEDIA_TYPE_SUBTITLE: + av_dynarray_add(&abrlist->sub_list, &abrlist->subs, tmp); + break; + default: + av_free(tmp); + break; + } + } + return 0; +} + +static int abr_check_list(ABRList *abr_list, enum AVMediaType type, int st) +{ + int **st_list; + int n_st; + switch (type) { + case AVMEDIA_TYPE_AUDIO: + st_list = abr_list->audio_list; + n_st = abr_list->audios; + break; + case AVMEDIA_TYPE_VIDEO: + st_list = abr_list->video_list; + n_st = abr_list->videos; + break; + case AVMEDIA_TYPE_SUBTITLE: + st_list = abr_list->sub_list; + n_st = abr_list->subs; + break; + default: + break; + } + if (!st_list) + return 0; + for (int i = 0; i < n_st; i++) { + if (*st_list[i] == st) + return 1; + } + return 0; +} + + + /* this thread gets the stream from the disk or the network */ static int read_thread(void *arg) { @@ -2789,6 +2885,8 @@ static int read_thread(void *arg) av_dict_set(&format_opts, "scan_all_pmts", "1", AV_DICT_DONT_OVERWRITE); scan_all_pmts_set = 1; } + if (abr) + av_dict_set(&format_opts, "abr", "1", 0); err = avformat_open_input(&ic, is->filename, is->iformat, &format_opts); if (err < 0) { print_error(is->filename, err); @@ -2918,6 +3016,15 @@ static int read_thread(void *arg) stream_component_open(is, st_index[AVMEDIA_TYPE_SUBTITLE]); } + /* clean packet list filled in hls_read_header if abr is enabled */ + if (abr) { + is->abr_list = av_mallocz(sizeof(ABRList)); + ret = abr_init_list(is); + if (ret < 0) { + av_log(NULL, AV_LOG_WARNING, "Failed to initiate abr_list\n"); + } + } + if (is->video_stream < 0 && is->audio_stream < 0) { av_log(NULL, AV_LOG_FATAL, "Failed to open file '%s' or configure filtergraph\n", is->filename); @@ -3045,15 +3152,31 @@ static int read_thread(void *arg) av_q2d(ic->streams[pkt->stream_index]->time_base) - (double)(start_time != AV_NOPTS_VALUE ? start_time : 0) / 1000000 <= ((double)duration / 1000000); - if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { - packet_queue_put(&is->audioq, pkt); - } else if (pkt->stream_index == is->video_stream && pkt_in_play_range - && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { - packet_queue_put(&is->videoq, pkt); - } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { - packet_queue_put(&is->subtitleq, pkt); + if (abr) { + if ((pkt->stream_index == is->audio_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_AUDIO, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if ((pkt->stream_index == is->video_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_VIDEO, pkt->stream_index)) && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if ((pkt->stream_index == is->subtitle_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_SUBTITLE, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } else { - av_packet_unref(pkt); + if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if (pkt->stream_index == is->video_stream && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } } @@ -3061,7 +3184,10 @@ static int read_thread(void *arg) fail: if (ic && !is->ic) avformat_close_input(&ic); - + if (abr && is->abr_list) { + free_abr_list(is->abr_list); + av_freep(&is->abr_list); + } if (ret != 0) { SDL_Event event; @@ -3588,6 +3714,7 @@ static const OptionDef options[] = { { "an", OPT_BOOL, { &audio_disable }, "disable audio" }, { "vn", OPT_BOOL, { &video_disable }, "disable video" }, { "sn", OPT_BOOL, { &subtitle_disable }, "disable subtitling" }, + { "abr", OPT_BOOL, { &abr }, "enable adaptive bitrate for hls/dash" }, { "ast", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_AUDIO] }, "select desired audio stream", "stream_specifier" }, { "vst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_VIDEO] }, "select desired video stream", "stream_specifier" }, { "sst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_SUBTITLE] }, "select desired subtitle stream", "stream_specifier" }, From patchwork Wed Jul 15 08:37:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21037 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 5047544B9FC for ; Wed, 15 Jul 2020 11:37:50 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3BB8A68B51E; Wed, 15 Jul 2020 11:37:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 39DD068B4B2 for ; Wed, 15 Jul 2020 11:37:42 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id A44531008CBD4; Wed, 15 Jul 2020 16:37:39 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id A1A44200B4496; Wed, 15 Jul 2020 16:37:39 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id CWLt66mo2Mgr; Wed, 15 Jul 2020 16:37:39 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id CC9A5200B448D; Wed, 15 Jul 2020 16:37:38 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Jul 2020 16:37:32 +0800 Message-Id: <20200715083733.101880-5-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v1 5/6] avformat/utils: add av_packet_buffer_filter to filter packet_buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add av_packet_buffer_filter to remove AVPackets whose stream_index is not in st_index list. st_index has length of AVMEDIA_TYPE_NB, contains the stream_index of all these media types. Generally s->internal->packet_buffer may have pkts from different stream, and stream_index will be used to discard pkt that is not needed. But in case of abr, several streams may pass the stream_index check. So we need a function to remove AVPackets not needed in pktl added by hls_read_header. v1 fixed: 1. rename function name *_clean to *_filter 2. fix memory leak in ff_packet_buffer_filter 3. update the doc Signed-off-by: spartazhc --- libavformat/avformat.h | 12 ++++++++++ libavformat/internal.h | 16 +++++++++++++ libavformat/utils.c | 54 ++++++++++++++++++++++++++++++++++++++++++ libavformat/version.h | 2 +- 4 files changed, 83 insertions(+), 1 deletion(-) diff --git a/libavformat/avformat.h b/libavformat/avformat.h index e91e7f1d33..c796fd0391 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -2474,6 +2474,18 @@ int avformat_seek_file(AVFormatContext *s, int stream_index, int64_t min_ts, int */ int avformat_flush(AVFormatContext *s); +/** + * Filter the packet buffer list of the AVFormatContext, remove the AVPackets + * do not need according to st_index. + * Only filter the packet_buffer list. + * + * @param s media file handle + * @param st_index the stream_index list which is needed + * st_index has length of AVMEDIA_TYPE_NB + * in index AVMEDIA_TYPE_XXX contains the stream_index needed of type XXX + */ +int av_packet_buffer_filter(AVFormatContext *s, int *st_index); + /** * Start playing a network-based stream (e.g. RTSP stream) at the * current position. diff --git a/libavformat/internal.h b/libavformat/internal.h index 17a6ab07d3..58ebcb2e35 100644 --- a/libavformat/internal.h +++ b/libavformat/internal.h @@ -772,6 +772,22 @@ int ff_packet_list_get(AVPacketList **head, AVPacketList **tail, */ void ff_packet_list_free(AVPacketList **head, AVPacketList **tail); +/** + * Remove the AVPackets do not need in the packet buffer list. + * For each type in AVMediaType, at most keep one stream and + * the others will be removed. + * + * @param head List head element + * @param tail List tail element + * @param st_index the stream_index list which is needed + * st_index has length of AVMEDIA_TYPE_NB + * in index AVMEDIA_TYPE_XXX contains the stream_index needed of type XXX + * @return 0 on success. Success is guaranteed + * if the packet list is not empty. + */ +int ff_packet_buffer_filter(AVPacketList **head, AVPacketList **tail, + int *st_index); + void avpriv_register_devices(const AVOutputFormat * const o[], const AVInputFormat * const i[]); #endif /* AVFORMAT_INTERNAL_H */ diff --git a/libavformat/utils.c b/libavformat/utils.c index 807d9f10cb..7674e4ea3d 100644 --- a/libavformat/utils.c +++ b/libavformat/utils.c @@ -1565,6 +1565,60 @@ int ff_packet_list_get(AVPacketList **pkt_buffer, return 0; } +/** + * return 1 if needed + */ +static int ff_check_st_index(int st, int *st_index) +{ + for (int i = 0; i < AVMEDIA_TYPE_NB; ++i) { + if (st_index[i] == st) + return 1; + } + return 0; +} + +int ff_packet_buffer_filter(AVPacketList **pkt_buffer, + AVPacketList **pkt_buffer_end, + int *st_index) +{ + AVPacketList *pktl, *pktn; + av_assert0(*pkt_buffer); + pktl = *pkt_buffer; + pktn = pktl->next; + + /* num >= 2 */ + while (pktn) { + if (!ff_check_st_index(pktn->pkt.stream_index, st_index)) { + av_packet_unref(&pktn->pkt); + pktl->next = pktn->next; + av_freep(&pktn); + pktn = pktl->next; + } else { + pktl = pktn; + pktn = pktn->next; + } + } + *pkt_buffer_end = pktl; + /* first one*/ + pktl = *pkt_buffer; + if (!ff_check_st_index(pktl->pkt.stream_index, st_index)) { + av_packet_unref(&pktl->pkt); + *pkt_buffer = pktl->next; + if (!pktl->next) + *pkt_buffer_end = NULL; + av_freep(&pktl); + } + + return 0; +} + +int av_packet_buffer_filter(AVFormatContext *s, int *st_index) +{ + int ret = ff_packet_buffer_filter(&s->internal->packet_buffer, + &s->internal->packet_buffer_end, st_index); + return ret; +} + static int64_t ts_to_samples(AVStream *st, int64_t ts) { return av_rescale(ts, st->time_base.num * st->codecpar->sample_rate, st->time_base.den); diff --git a/libavformat/version.h b/libavformat/version.h index 75c03fde0a..33cebed85e 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -32,7 +32,7 @@ // Major bumping may affect Ticket5467, 5421, 5451(compatibility with Chromium) // Also please add any ticket numbers that you believe might be affected here #define LIBAVFORMAT_VERSION_MAJOR 58 -#define LIBAVFORMAT_VERSION_MINOR 48 +#define LIBAVFORMAT_VERSION_MINOR 49 #define LIBAVFORMAT_VERSION_MICRO 100 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ From patchwork Wed Jul 15 08:37:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21038 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 25D0D44B9FC for ; Wed, 15 Jul 2020 11:37:53 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0C16968B529; Wed, 15 Jul 2020 11:37:53 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 60F3268B4F3 for ; Wed, 15 Jul 2020 11:37:46 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 47FCB1008CBD0; Wed, 15 Jul 2020 16:37:41 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 45510200B4498; Wed, 15 Jul 2020 16:37:41 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id KCBupcOSiOzy; Wed, 15 Jul 2020 16:37:41 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 5FFA2200B448D; Wed, 15 Jul 2020 16:37:39 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Jul 2020 16:37:33 +0800 Message-Id: <20200715083733.101880-6-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200715083733.101880-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v1 6/6] ffplay: add av_packet_buffer_filter to filter packet buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc hls_read_header will add all streams to s->internal->packet_buffer. Use av_packet_buffer_filter to remove the AVPackets from other streams that are not needed, otherwise abr will allow them to be added to ffplay's packet_queue. Signed-off-by: spartazhc --- fftools/ffplay.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index b17b75fa8f..832e97d910 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -3018,6 +3018,10 @@ static int read_thread(void *arg) /* clean packet list filled in hls_read_header if abr is enabled */ if (abr) { + ret = av_packet_buffer_filter(ic, st_index); + if (ret < 0) { + av_log(NULL, AV_LOG_WARNING, "Failed to clean av_packet\n"); + } is->abr_list = av_mallocz(sizeof(ABRList)); ret = abr_init_list(is); if (ret < 0) {