From patchwork Sun Aug 23 12:23:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21839 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 8AC80449B15 for ; Sun, 23 Aug 2020 15:24:18 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7063868975D; Sun, 23 Aug 2020 15:24:18 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 920CF687F29 for ; Sun, 23 Aug 2020 15:24:10 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 183671008CBC2; Sun, 23 Aug 2020 20:24:05 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 0C545200B449C; Sun, 23 Aug 2020 20:24:05 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id gDAMHvgOiagW; Sun, 23 Aug 2020 20:24:04 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 1AE90200B44A1; Sun, 23 Aug 2020 20:24:02 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 23 Aug 2020 20:23:49 +0800 Message-Id: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC v3 1/7] avformat/abr: Adaptive Bitrate support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add abr module for hls/dash. Signed-off-by: spartazhc v1 fixed: 1. add an "ff" prefix to the protocol name to mark it internal. 2. use 1.2f for float constant 1.2. 3. simplify abr_seek for we just need AVSEEK_SIZE only. v2 fixed: 1. fix error return 2. simplify abr_seek v3 fixed: 1. rewrite hls_param_parse function 2. fix error code return 3. use unsigned type and const prefix 4. fix documentation 5. fix abr_rule 6. rename cur_pls to cur_var 7. add type input and output --- doc/protocols.texi | 7 ++ libavformat/Makefile | 1 + libavformat/ffabr.c | 271 ++++++++++++++++++++++++++++++++++++++++ libavformat/protocols.c | 1 + 4 files changed, 280 insertions(+) create mode 100644 libavformat/ffabr.c diff --git a/doc/protocols.texi b/doc/protocols.texi index 7b3df96fda..e31de80ab6 100644 --- a/doc/protocols.texi +++ b/doc/protocols.texi @@ -232,6 +232,13 @@ For example, to convert a GIF file given inline with @command{ffmpeg}: ffmpeg -i "data:image/gif;base64,R0lGODdhCAAIAMIEAAAAAAAA//8AAP//AP///////////////ywAAAAACAAIAAADF0gEDLojDgdGiJdJqUX02iB4E8Q9jUMkADs=" smiley.png @end example +@section ffabr + +Adaptive bitrate sub-protocol work for hls/dash, ffabr is internal. + +The ffabr protocol takes stream information from hls/dash as input, +use bandwidth estimation to decide whether to switch or not. + @section file File access protocol. diff --git a/libavformat/Makefile b/libavformat/Makefile index cbb33fe37c..68b004eee0 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -598,6 +598,7 @@ OBJS-$(CONFIG_CACHE_PROTOCOL) += cache.o OBJS-$(CONFIG_CONCAT_PROTOCOL) += concat.o OBJS-$(CONFIG_CRYPTO_PROTOCOL) += crypto.o OBJS-$(CONFIG_DATA_PROTOCOL) += data_uri.o +OBJS-$(CONFIG_FFABR_PROTOCOL) += ffabr.o OBJS-$(CONFIG_FFRTMPCRYPT_PROTOCOL) += rtmpcrypt.o rtmpdigest.o rtmpdh.o OBJS-$(CONFIG_FFRTMPHTTP_PROTOCOL) += rtmphttp.o OBJS-$(CONFIG_FILE_PROTOCOL) += file.o diff --git a/libavformat/ffabr.c b/libavformat/ffabr.c new file mode 100644 index 0000000000..1785ef5643 --- /dev/null +++ b/libavformat/ffabr.c @@ -0,0 +1,271 @@ +/* + * Adaptive Bitrate Module for HLS / DASH + * Copyright (c) 2020 + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include "avformat.h" +#include "libavutil/avassert.h" +#include "libavutil/avstring.h" +#include "libavutil/common.h" +#include "libavutil/opt.h" +#include "libavutil/time.h" +#include "url.h" + +#define ABR_NOT_SWITCH -1 + +enum ABRFormatType { + ABR_TYPE_HLS, + ABR_TYPE_DASH +}; + +typedef struct Variant { + uint32_t bitrate; + size_t index; +} variant; + +typedef struct ABRContext { + const AVClass *class; + URLContext *hd; + AVDictionary *abr_params; + AVDictionary *abr_metadata; + enum ABRFormatType format; + uint8_t cur_var; + uint8_t type; + int8_t can_switch; + size_t n_variants; + variant *variants; + size_t n_throughputs; + float *throughputs; +} ABRContext; + +static float harmonic_mean(const float *arr, size_t num) +{ + float tmp = 0; + + if (!num) return 0; + + for (size_t i = 0; i < num; i++) { + tmp += 1 / arr[i]; + } + + return num / tmp; +} + +static int hls_param_parse(ABRContext *c, const AVDictionaryEntry *entry) +{ + AVDictionaryEntry *en; + size_t index; + char key_tmp[20]; + + + en = av_dict_get(c->abr_params, "cur_var", entry, AV_DICT_IGNORE_SUFFIX); + if (en) { + c->cur_var = strtol(en->value, NULL, 10); + } + en = av_dict_get(c->abr_params, "type", entry, AV_DICT_IGNORE_SUFFIX); + if (en) { + c->type = strtol(en->value, NULL, 10); + } + en = av_dict_get(c->abr_params, "can_switch", entry, AV_DICT_IGNORE_SUFFIX); + if (en) { + c->can_switch = strtol(en->value, NULL, 10); + } + en = av_dict_get(c->abr_params, "n_variants", entry, AV_DICT_IGNORE_SUFFIX); + if (en) { + c->n_variants = strtol(en->value, NULL, 10); + c->variants = av_mallocz(sizeof(variant) * c->n_variants); + if (!c->variants) + return AVERROR(ENOMEM); + index = 0; + snprintf(key_tmp, sizeof(key_tmp), "variant_bitrate%ld", index); + while ((en = av_dict_get(c->abr_params, key_tmp, entry, AV_DICT_IGNORE_SUFFIX)) + && index < c->n_variants) { + c->variants[index].bitrate = strtol(en->value, NULL, 10); + c->variants[index].index = index; + index++; + snprintf(key_tmp, sizeof(key_tmp), "variant_bitrate%ld", index); + } + } + en = av_dict_get(c->abr_params, "n_throughputs", entry, AV_DICT_IGNORE_SUFFIX); + if (en) { + c->n_throughputs = strtol(en->value, NULL, 10); + if (!c->n_throughputs) + return 0; + c->throughputs = av_malloc(sizeof(float) * c->n_throughputs); + if (!c->throughputs) + return AVERROR(ENOMEM); + index = 0; + snprintf(key_tmp, sizeof(key_tmp), "throughputs%ld", index); + while ((en = av_dict_get(c->abr_params, key_tmp, entry, AV_DICT_IGNORE_SUFFIX)) + && index < c->n_throughputs) { + c->throughputs[index++] = strtol(en->value, NULL, 10); + snprintf(key_tmp, sizeof(key_tmp), "throughputs%ld", index); + } + } + + return 0; +} + +static int abr_param_parse(ABRContext *c, enum ABRFormatType type, const AVDictionaryEntry *en) +{ + int ret; + if (type == ABR_TYPE_HLS) { + ret = hls_param_parse(c, en); + } + return ret; +} + +static int compare_vb(const void *a, const void *b) +{ + return FFDIFFSIGN((*(const variant *)b).bitrate, (*(const variant *)a).bitrate); +} + +static int abr_rule(URLContext *h, float bw_estimate) +{ + int ret = ABR_NOT_SWITCH; + ABRContext *c = h->priv_data; + + if (bw_estimate < c->variants[c->cur_var].bitrate / 1000 * 1.2f && + bw_estimate > c->variants[c->cur_var].bitrate / 1000 * 0.8f) + return ABR_NOT_SWITCH; + qsort(c->variants, c->n_variants, sizeof(variant), compare_vb); + for (int i = 0; i < c->n_variants; i++) { + if (bw_estimate > c->variants[i].bitrate / 1000) { + ret = c->variants[i].index; + break; + } + } + if (ret == ABR_NOT_SWITCH) + ret = c->variants[c->n_variants - 1].index; + else if (ret == c->cur_var) + ret = ABR_NOT_SWITCH; + + av_log(h, AV_LOG_VERBOSE, "[switch] bwe=%.2fkbps, cur=%d, switch=%d\n", bw_estimate, c->cur_var, ret); + return ret; +} + +static int abr_open(URLContext *h, const char *uri, int flags, AVDictionary **options) +{ + const char *nested_url; + int64_t start, end; + float bw_estimation; + int switch_request = ABR_NOT_SWITCH; + int ret = 0; + ABRContext *c = h->priv_data; + AVDictionaryEntry *en = NULL; + + if (!av_strstart(uri, "ffabr+", &nested_url) && + !av_strstart(uri, "ffabr:", &nested_url)) { + av_log(h, AV_LOG_ERROR, "Unsupported url %s\n", uri); + return AVERROR(EINVAL); + } + + en = av_dict_get(c->abr_params, "format", en, AV_DICT_IGNORE_SUFFIX); + if (en) { + if (!av_strcasecmp(en->value, "hls")) { + c->format = ABR_TYPE_HLS; + } else if (!av_strcasecmp(en->value, "dash")) { + c->format = ABR_TYPE_DASH; + } + av_log(h, AV_LOG_VERBOSE, "%s is using ABR\n", en->value); + } else { + return AVERROR(EINVAL); + } + + if (ret = abr_param_parse(c, c->format, en) < 0) { + av_log(h, AV_LOG_ERROR,"Error parsing abr params.\n"); + return ret; + } + + start = av_gettime(); + if ((ret = ffurl_open_whitelist(&c->hd, nested_url, flags, + &h->interrupt_callback, options, + h->protocol_whitelist, h->protocol_blacklist, h)) < 0) { + av_log(h, AV_LOG_ERROR, "Unable to open resource: %s\n", nested_url); + return ret; + } + end = av_gettime(); + + bw_estimation = harmonic_mean(c->throughputs, c->n_throughputs); + + if (c->can_switch == 1) + switch_request = abr_rule(h, bw_estimation); + + av_dict_set_int(&c->abr_metadata, "download_time", (end - start), 0); + av_dict_set_int(&c->abr_metadata, "switch_request", switch_request, 0); + av_dict_set_int(&c->abr_metadata, "type", c->type, 0); + + return ret; +} + + +static int abr_read(URLContext *h, uint8_t *buf, int size) +{ + ABRContext *c = h->priv_data; + + return ffurl_read(c->hd, buf, size); +} + +static int64_t abr_seek(URLContext *h, int64_t pos, int whence) +{ + ABRContext *c = h->priv_data; + + if (whence == AVSEEK_SIZE) { + return ffurl_seek(c->hd, pos, AVSEEK_SIZE); + } else { + return AVERROR(errno); + } +} + +static int abr_close(URLContext *h) +{ + ABRContext *c = h->priv_data; + int ret = 0; + + ffurl_closep(&c->hd); + av_free(c->variants); + av_free(c->throughputs); + return ret; +} + +#define OFFSET(x) offsetof(ABRContext, x) +#define D AV_OPT_FLAG_DECODING_PARAM +static const AVOption ffabr_options[] = { + { "abr-params", "Informations ABR needed, using a :-separated list of key=value parameters", OFFSET(abr_params), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { "abr-metadata", "Metadata return from abr, including switch signal and network bandwidth", OFFSET(abr_metadata), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { NULL } +}; + +static const AVClass ffabr_class = { + .class_name = "ffabr", + .item_name = av_default_item_name, + .option = ffabr_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +const URLProtocol ff_ffabr_protocol = { + .name = "ffabr", + .url_open2 = abr_open, + .url_read = abr_read, + .url_seek = abr_seek, + .url_close = abr_close, + .priv_data_size = sizeof(ABRContext), + .priv_data_class = &ffabr_class, +}; diff --git a/libavformat/protocols.c b/libavformat/protocols.c index 7df18fbb3b..1d6af8e380 100644 --- a/libavformat/protocols.c +++ b/libavformat/protocols.c @@ -29,6 +29,7 @@ extern const URLProtocol ff_cache_protocol; extern const URLProtocol ff_concat_protocol; extern const URLProtocol ff_crypto_protocol; extern const URLProtocol ff_data_protocol; +extern const URLProtocol ff_ffabr_protocol; extern const URLProtocol ff_ffrtmpcrypt_protocol; extern const URLProtocol ff_ffrtmphttp_protocol; extern const URLProtocol ff_file_protocol; From patchwork Sun Aug 23 12:23:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21838 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id A39B8449B15 for ; Sun, 23 Aug 2020 15:24:17 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7C47B687FAE; Sun, 23 Aug 2020 15:24:17 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 94AEA687FAE for ; Sun, 23 Aug 2020 15:24:10 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id CFD861008CBC4; Sun, 23 Aug 2020 20:24:06 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id AF18320089C8D; Sun, 23 Aug 2020 20:24:06 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Ef6MKiQ_2ecA; Sun, 23 Aug 2020 20:24:05 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 0C7EA200B44A4; Sun, 23 Aug 2020 20:24:04 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 23 Aug 2020 20:23:50 +0800 Message-Id: <20200823122355.188611-2-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC v3 2/7] avformat/http: Add ffabr to whitelist X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc add ffabr protocol to the default_whitelist of http and https Signed-off-by: spartazhc --- libavformat/http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavformat/http.c b/libavformat/http.c index 6c39da1a8b..88e69114a2 100644 --- a/libavformat/http.c +++ b/libavformat/http.c @@ -1815,7 +1815,7 @@ const URLProtocol ff_http_protocol = { .priv_data_size = sizeof(HTTPContext), .priv_data_class = &http_context_class, .flags = URL_PROTOCOL_FLAG_NETWORK, - .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data" + .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data,ffabr" }; #endif /* CONFIG_HTTP_PROTOCOL */ @@ -1835,7 +1835,7 @@ const URLProtocol ff_https_protocol = { .priv_data_size = sizeof(HTTPContext), .priv_data_class = &https_context_class, .flags = URL_PROTOCOL_FLAG_NETWORK, - .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy" + .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,ffabr" }; #endif /* CONFIG_HTTPS_PROTOCOL */ From patchwork Sun Aug 23 12:23:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21840 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id C04F9449B15 for ; Sun, 23 Aug 2020 15:24:20 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A691E6897CD; Sun, 23 Aug 2020 15:24:20 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2D01668973D for ; Sun, 23 Aug 2020 15:24:13 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 9AACD1008CBCD; Sun, 23 Aug 2020 20:24:10 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 644202008CB41; Sun, 23 Aug 2020 20:24:10 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id dgw9WxW3nc9y; Sun, 23 Aug 2020 20:24:10 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id AF59D200A4B00; Sun, 23 Aug 2020 20:24:05 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 23 Aug 2020 20:23:51 +0800 Message-Id: <20200823122355.188611-3-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC v3 3/7] avformat/hls: use abr to switch streams X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc When abr is enable, it will take over the task to call http to download segments, and will return a switch-request for hls to switch streams. For reason not to waste segments that have been downloaded, switch will become effective after old segments is used out. Abr cannot work with http_persistent option, and currently use http_multiple. Signed-off-by: spartazhc v1 fixed: 1. fix memory leak v2 fixed: 1. check malloc result and return AVERROR(ENOMEM) 2. define ABR_THROUGHPUT_FIFO_LEN instead of hardcode v3 fixed: 1. hls should switch variant other than playlist, two structure switch_info and switch_task are added to support arbitrary switch request in different manifest 2. bug fix in throughput update 3. added switch_step to support http_multiple option 4. fix several segment faults --- doc/demuxers.texi | 3 + libavformat/hls.c | 405 +++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 402 insertions(+), 6 deletions(-) diff --git a/doc/demuxers.texi b/doc/demuxers.texi index 3c15ab9eee..4cdbd95962 100644 --- a/doc/demuxers.texi +++ b/doc/demuxers.texi @@ -321,6 +321,9 @@ available in a metadata key named "variant_bitrate". It accepts the following options: @table @option +@item abr +enable abr to switch streams. + @item live_start_index segment index to start live streams at (negative values are from the end). diff --git a/libavformat/hls.c b/libavformat/hls.c index 3ab07f1b3f..4e760f8e8a 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -47,6 +47,7 @@ #define MPEG_TIME_BASE 90000 #define MPEG_TIME_BASE_Q (AVRational){1, MPEG_TIME_BASE} +#define ABR_THROUGHPUT_FIFO_LEN 20 /* * An apple http stream consists of a playlist with media segment files, * played sequentially. There may be several playlists with the same @@ -65,6 +66,13 @@ enum KeyType { KEY_SAMPLE_AES }; +enum SwitchType { + SWITCH_VIDEO, + SWITCH_AUDIO, + SWITCH_VIDEO_AUDIO, + SWITCH_TYPES +}; + struct segment { int64_t duration; int64_t url_offset; @@ -189,6 +197,26 @@ struct variant { char subtitles_group[MAX_FIELD_LEN]; }; +struct throughput { + int n_throughputs; + + /* throughputs are in kbps */ + float throughput_fifo[ABR_THROUGHPUT_FIFO_LEN]; + int head; + int tail; +}; + +struct switch_task { + enum SwitchType type; + int64_t switch_timestamp; +}; + +struct switch_info { + int pls_index; + int64_t first_timestamp; + int64_t delta_timestamp; +}; + typedef struct HLSContext { AVClass *class; AVFormatContext *ctx; @@ -213,8 +241,58 @@ typedef struct HLSContext { int http_multiple; int http_seekable; AVIOContext *playlist_pb; + + int abr; + struct throughput *throughputs; + struct switch_task *switch_tasks; + struct switch_info *switch_info; + int switch_inited; + int can_switch; + int switch_request; + int switch_step; + int cur_var; } HLSContext; +static struct segment *next_segment(struct playlist *pls); +static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in); +static int select_cur_seq_no(HLSContext *c, struct playlist *pls); +static AVRational get_timebase(struct playlist *pls); + +static struct segment *next2_segment(struct playlist *pls) +{ + int n = pls->cur_seq_no - pls->start_seq_no + 2; + if (n >= pls->n_segments) + return NULL; + return pls->segments[n]; +} + +static int playlist_type_full(struct playlist *pls) +{ + if (pls->n_main_streams == 1) { + return (enum SwitchType)pls->main_streams[0]->codecpar->codec_type; + } else { + return SWITCH_VIDEO_AUDIO; + } +} + +static int playlist_type_simple(struct playlist *pls) +{ + int type = playlist_type_full(pls); + if (type == SWITCH_VIDEO_AUDIO) + type = SWITCH_VIDEO; + return type; +} + +static int is_pls_switch_to(HLSContext *c, int index) { + if (c->switch_request == -1) + return 0; + for (int i = 0; i < c->variants[c->switch_request]->n_playlists; i++) { + if (c->variants[c->switch_request]->playlists[i]->index == index) + return 1; + } + return 0; +} + static void free_segment_dynarray(struct segment **segments, int n_segments) { int i; @@ -616,6 +694,38 @@ static int open_url_keepalive(AVFormatContext *s, AVIOContext **pb, #endif } +static int update_throughputs(struct throughput *thr, float time, int pb_size) +{ + if (pb_size <= 0 || time <= 0) + return AVERROR(EINVAL); + if (thr->n_throughputs < ABR_THROUGHPUT_FIFO_LEN) { + ++thr->n_throughputs; + } else { + thr->head = (thr->head + 1) % ABR_THROUGHPUT_FIFO_LEN; + } + thr->throughput_fifo[thr->tail] = (float)(pb_size) / time; + thr->tail = (thr->tail + 1) % ABR_THROUGHPUT_FIFO_LEN; + return 0; +} + +static int64_t get_switch_timestamp(HLSContext *c, struct playlist *pls) +{ + int64_t first_timestamp, pos; + int type; + int n = pls->cur_seq_no + c->switch_step; + if (n >= pls->n_segments) + return -1; + + type = playlist_type_simple(pls); + first_timestamp = c->switch_info[type].first_timestamp; + pos = first_timestamp == AV_NOPTS_VALUE ? 0 : first_timestamp; + + for (int i = 0; i < n; i++) { + pos += pls->segments[i]->duration; + } + return pos; +} + static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, AVDictionary *opts, AVDictionary *opts2, int *is_http_out) { @@ -631,6 +741,9 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, } else if (av_strstart(url, "data", NULL)) { if (url[4] == '+' || url[4] == ':') proto_name = avio_find_protocol_name(url + 5); + } else if (av_strstart(url, "ffabr", NULL)) { + if (url[5] == '+' || url[5] == ':') + proto_name = avio_find_protocol_name(url + 6); } if (!proto_name) @@ -661,6 +774,8 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, ; else if (av_strstart(url, "data", NULL) && !strncmp(proto_name, url + 5, strlen(proto_name)) && url[5 + strlen(proto_name)] == ':') ; + else if (av_strstart(url, "ffabr", NULL) && !strncmp(proto_name, url + 6, strlen(proto_name)) && url[6 + strlen(proto_name)] == ':') + ; else if (strcmp(proto_name, "file") || !strncmp(url, "file,", 5)) return AVERROR_INVALIDDATA; @@ -682,6 +797,69 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, } else { ret = s->io_open(s, pb, url, AVIO_FLAG_READ, &tmp); } + if (c->abr && ret >= 0) { + AVDictionary *abr_ret = NULL; + AVDictionaryEntry *en = NULL; + struct segment *seg; + int pb_size, switch_request; + enum SwitchType type; + av_opt_get_dict_val(*pb, "abr-metadata", AV_OPT_SEARCH_CHILDREN, &abr_ret); + if (abr_ret) { + en = av_dict_get(abr_ret, "download_time", NULL, 0); + if (en) { + pb_size = avio_size(*pb); + update_throughputs(c->throughputs, strtol(en->value, NULL, 10) / 1000.0, pb_size); + av_log(s, AV_LOG_VERBOSE, "[abr] time=%.2fms, size=%.2fkb\n", + strtol(en->value, NULL, 10) / 1000.0, pb_size / 1000.0); + } + en = av_dict_get(abr_ret, "switch_request", NULL, 0); + if (en) { + switch_request = strtol(en->value, NULL, 10); + if (switch_request != -1) + av_log(s, AV_LOG_INFO, "[abr] switch request: %s\n", en->value); + } + en = av_dict_get(abr_ret, "type", NULL, 0); + if (en) { + type = strtol(en->value, NULL, 10); + } + if (switch_request != -1) { + struct variant *var = c->variants[switch_request]; + c->switch_request = switch_request; + c->can_switch = 0; + for (int i = 0; i < var->n_playlists; i++) { + struct playlist *pls = var->playlists[i]; + int64_t switch_timestamp; + pls->cur_seq_no = select_cur_seq_no(c, pls); + // if pls has same type to the segment just downloaded, switch should be delayed + if (type == SWITCH_VIDEO_AUDIO || playlist_type_full(pls) == type) { + pls->cur_seq_no++; + } + if (c->switch_step == 2) { + seg = next2_segment(pls); + } else { + seg = next_segment(pls); + } + switch_timestamp = get_switch_timestamp(c, pls); + if (!seg || switch_timestamp == -1) { + c->switch_request = -1; + av_log(s, AV_LOG_INFO, "[abr] no more segment need to switch\n"); + } else { + int ptype; + ptype = playlist_type_simple(pls); + c->switch_tasks[pls->index].type = ptype; + c->switch_tasks[pls->index].switch_timestamp = switch_timestamp - c->switch_info[ptype].delta_timestamp * 1.5; + av_log(s, AV_LOG_INFO, "[abr] pls%d, switch type: %d timestamp: %ld\n", + pls->index, c->switch_tasks[pls->index].type, c->switch_tasks[pls->index].switch_timestamp); + if (c->switch_step == 2) { + pls->input_next_requested = 1; + ret = open_input(c, pls, seg, &pls->input_next); + } + } + } + } + av_dict_free(&abr_ret); + } + } if (ret >= 0) { // update cookies on http response with setcookies. char *new_cookies = NULL; @@ -1211,6 +1389,38 @@ static void intercept_id3(struct playlist *pls, uint8_t *buf, pls->is_id3_timestamped = (pls->id3_mpegts_timestamp != AV_NOPTS_VALUE); } +static int abrinfo_to_dict(HLSContext *c, enum SwitchType type, char **abr_info) +{ + struct throughput *thr = c->throughputs; + char buffer[MAX_URL_SIZE]; + int size, i, j; + size = snprintf(buffer, sizeof(buffer), "format=hls:"); + size += snprintf(buffer + size, sizeof(buffer) - size, "cur_var=%d:", c->cur_var); + size += snprintf(buffer + size, sizeof(buffer) - size, "type=%d:", type); + size += snprintf(buffer + size, sizeof(buffer) - size, "can_switch=%d:", c->can_switch); + size += snprintf(buffer + size, sizeof(buffer) - size, "n_variants=%d:", c->n_variants); + for (i = 0; i < c->n_variants; i++) { + struct variant *v = c->variants[i]; + size += snprintf(buffer + size, sizeof(buffer) - size, "variant_bitrate%d=%d:", i, v->bandwidth); + } + size += snprintf(buffer + size, sizeof(buffer) - size, "n_throughputs=%d:", thr->n_throughputs); + if (thr->n_throughputs > 0) { + i = thr->head; + j = 0; + do { + size += snprintf(buffer + size, sizeof(buffer) - size, "throughputs%d=%.2f:", j, thr->throughput_fifo[i]); + j++; + i = (i + 1) % ABR_THROUGHPUT_FIFO_LEN; + } while (i != thr->tail); + } + *abr_info = av_malloc(size); + if (!abr_info) + return AVERROR(ENOMEM); + snprintf(*abr_info, size, "%s", buffer); + av_log(c, AV_LOG_VERBOSE, "[abr] abr_info: %s\n", *abr_info); + return 0; +} + static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in) { AVDictionary *opts = NULL; @@ -1230,8 +1440,20 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, av_log(pls->parent, AV_LOG_VERBOSE, "HLS request for url '%s', offset %"PRId64", playlist %d\n", seg->url, seg->url_offset, pls->index); + if (c->abr) { + char *abr_opts; + abrinfo_to_dict(c, playlist_type_full(pls), &abr_opts); + av_dict_set(&opts, "abr-params", abr_opts, 0); + av_free(abr_opts); + } if (seg->key_type == KEY_NONE) { - ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + char url_abr[MAX_URL_SIZE]; + if (c->abr) { + snprintf(url_abr, sizeof(url_abr), "ffabr:%s", seg->url); // + or : tbd + ret = open_url(pls->parent, in, url_abr, c->avio_opts, opts, &is_http); + } else { + ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + } } else if (seg->key_type == KEY_AES_128) { char iv[33], key[33], url[MAX_URL_SIZE]; if (strcmp(seg->key, pls->key_url)) { @@ -1427,7 +1649,15 @@ restart: /* Check that the playlist is still needed before opening a new * segment. */ v->needed = playlist_needed(v); - + if (c->abr) { + if (c->switch_request == -1) + ; + else if (v->needed && !is_pls_switch_to(c, v->index) && !v->input_next_requested) { + av_log(v->parent, AV_LOG_VERBOSE, "read_data: needed but not download playlist %d ('%s')\n", + v->index, v->url); + return AVERROR_EOF; + } + } if (!v->needed) { av_log(v->parent, AV_LOG_INFO, "No longer receiving playlist %d ('%s')\n", v->index, v->url); @@ -1517,12 +1747,22 @@ reload: int r = av_opt_get(v->input, "http_version", AV_OPT_SEARCH_CHILDREN, &http_version_opt); if (r >= 0) { c->http_multiple = (!strncmp((const char *)http_version_opt, "1.1", 3) || !strncmp((const char *)http_version_opt, "2.0", 3)); + c->switch_step = c->http_multiple + 1; av_freep(&http_version_opt); + } else { + c->switch_step = 1; } + } else { + c->switch_step = c->http_multiple + 1; } seg = next_segment(v); - if (c->http_multiple == 1 && !v->input_next_requested && + + // when get switch_request, old stream should stop downloading next seg + if (c->abr && c->switch_inited && c->switch_request != -1 && c->http_multiple == 1 && !is_pls_switch_to(c, v->index) + && ((c->cur_timestamp + seg->duration * c->switch_step) >= c->switch_tasks[v->index].switch_timestamp)) + av_log(v->parent, AV_LOG_VERBOSE, "old playlist %d should stop downloading next seg\n", v->index); + else if (c->http_multiple == 1 && !v->input_next_requested && seg && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { ret = open_input(c, v, seg, &v->input_next); if (ret < 0) { @@ -1555,7 +1795,7 @@ reload: return ret; } - if (c->http_persistent && + if (c->http_persistent && !c->abr && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { v->input_read_done = 1; } else { @@ -1833,7 +2073,11 @@ static int hls_close(AVFormatContext *s) free_playlist_list(c); free_variant_list(c); free_rendition_list(c); - + if (c->abr) { + av_free(c->throughputs); + av_free(c->switch_tasks); + av_free(c->switch_info); + } av_dict_free(&c->avio_opts); ff_format_io_close(c->ctx, &c->playlist_pb); @@ -1852,6 +2096,19 @@ static int hls_read_header(AVFormatContext *s) c->first_packet = 1; c->first_timestamp = AV_NOPTS_VALUE; c->cur_timestamp = AV_NOPTS_VALUE; + c->cur_var = -1; + c->switch_request = -1; + c->switch_step = 1; + c->can_switch = -1; + + if (c->abr) { + c->http_persistent = 0; + c->throughputs = av_mallocz(sizeof(struct throughput)); + if (!c->throughputs) { + ret = AVERROR(ENOMEM); + goto fail; + } + } if ((ret = save_avio_options(s)) < 0) goto fail; @@ -2042,6 +2299,37 @@ static int hls_read_header(AVFormatContext *s) add_metadata_from_renditions(s, pls, AVMEDIA_TYPE_SUBTITLE); } + if (c->abr) { + c->switch_tasks = av_malloc(c->n_playlists * sizeof(struct switch_task)); + if (!c->switch_tasks) { + ret = AVERROR(ENOMEM); + goto fail; + } + c->switch_info = av_malloc((SWITCH_TYPES - 1) * sizeof(struct switch_info)); + if (!c->switch_info) { + ret = AVERROR(ENOMEM); + goto fail; + } + for (i = 0; i < SWITCH_TYPES - 1; i++) { + c->switch_info[i].pls_index = -1; + c->switch_info[i].first_timestamp = AV_NOPTS_VALUE; + c->switch_info[i].delta_timestamp = AV_NOPTS_VALUE; + } + for (i = 0; i < c->n_playlists; i++) { + struct playlist *pls = c->playlists[i]; + int type; + + c->switch_tasks[i].switch_timestamp = AV_NOPTS_VALUE; + type = playlist_type_simple(pls); + if (type == SWITCH_AUDIO) { + if (c->switch_info[SWITCH_AUDIO].pls_index == -1) + c->switch_info[SWITCH_AUDIO].pls_index = pls->index; + } else { + if (c->switch_info[SWITCH_VIDEO].pls_index == -1) + c->switch_info[SWITCH_VIDEO].pls_index = pls->index; + } + } + } update_noheader_flag(s); return 0; @@ -2050,12 +2338,71 @@ fail: return ret; } +static void change_discard_flags(struct playlist *pls, enum SwitchType type, int flag) +{ + int process_all = 0; + if (type == SWITCH_VIDEO_AUDIO) + process_all = 1; + for (int j = 0; j < pls->n_main_streams; j++) { + if (process_all || pls->main_streams[j]->codecpar->codec_type == (enum AVMediaType)type) + pls->main_streams[j]->discard = flag; + } +} + +static int find_current_variant(HLSContext *c) +{ + int i, j, needed; + for (i = 0; i < c->n_variants; i++) { + needed = 1; + for (j = 0; j < c->variants[i]->n_playlists; j++) { + struct playlist *pls = c->variants[i]->playlists[j]; + if (!playlist_needed(pls)) { + needed = 0; + } + } + if (needed) + return i; + } + return -1; +} + static int recheck_discard_flags(AVFormatContext *s, int first) { HLSContext *c = s->priv_data; int i, changed = 0; int cur_needed; + if (c->abr && c->switch_request != -1) { + struct variant *var = c->variants[c->switch_request]; + int no_switch_task = 1; + for (int i = 0; i < var->n_playlists; i++) { + struct playlist *pls = var->playlists[i]; + if (c->switch_tasks[pls->index].switch_timestamp != AV_NOPTS_VALUE + && c->cur_timestamp >= c->switch_tasks[pls->index].switch_timestamp) { + av_log(s, AV_LOG_INFO, "[switch point] cur_timestamp:%ld\n", c->cur_timestamp); + change_discard_flags(pls, c->switch_tasks[pls->index].type, AVDISCARD_DEFAULT); + c->switch_tasks[pls->index].switch_timestamp = AV_NOPTS_VALUE; + + for (int j = 0; j < c->n_playlists; j++) { + struct playlist *cpls = c->playlists[j]; + if (cpls == pls) + continue; + change_discard_flags(cpls, c->switch_tasks[pls->index].type, AVDISCARD_ALL); + } + } + } + + for (int k = 0; k < c->n_playlists; ++k) { + if (c->switch_tasks[k].switch_timestamp != AV_NOPTS_VALUE) { + no_switch_task = 0; + } + } + if (no_switch_task) { + c->can_switch = 1; + c->cur_var = c->switch_request; + } + } + /* Check if any new streams are needed */ for (i = 0; i < c->n_playlists; i++) { struct playlist *pls = c->playlists[i]; @@ -2070,6 +2417,11 @@ static int recheck_discard_flags(AVFormatContext *s, int first) changed = 1; pls->cur_seq_no = select_cur_seq_no(c, pls); pls->pb.eof_reached = 0; + if (c->abr) { + pls->cur_seq_no++; + avio_flush(&pls->pb); + avformat_flush(pls->ctx); + } if (c->cur_timestamp != AV_NOPTS_VALUE) { /* catch up */ pls->seek_timestamp = c->cur_timestamp; @@ -2077,7 +2429,7 @@ static int recheck_discard_flags(AVFormatContext *s, int first) pls->seek_stream_index = -1; } av_log(s, AV_LOG_INFO, "Now receiving playlist %d, segment %d\n", i, pls->cur_seq_no); - } else if (first && !cur_needed && pls->needed) { + } else if ((first || c->abr) && !cur_needed && pls->needed) { ff_format_io_close(pls->parent, &pls->input); pls->input_read_done = 0; ff_format_io_close(pls->parent, &pls->input_next); @@ -2087,6 +2439,13 @@ static int recheck_discard_flags(AVFormatContext *s, int first) av_log(s, AV_LOG_INFO, "No longer receiving playlist %d\n", i); } } + + if (c->abr && changed && c->cur_var == -1) { + c->cur_var = find_current_variant(c); + av_log(s, AV_LOG_INFO, "Find current variant %d\n", c->cur_var); + if (c->cur_var != -1 && c->can_switch == -1) + c->can_switch = 1; + } return changed; } @@ -2159,7 +2518,39 @@ static int hls_read_packet(AVFormatContext *s, AVPacket *pkt) /* audio elementary streams are id3 timestamped */ fill_timing_for_id3_timestamped_stream(pls); } + if (c->abr && !c->switch_inited && pls->pkt.dts != AV_NOPTS_VALUE) { + for (int j = 0; j < SWITCH_TYPES - 1; j++) { + if (c->switch_info[j].first_timestamp != AV_NOPTS_VALUE + && c->switch_info[j].delta_timestamp == AV_NOPTS_VALUE + && c->switch_info[j].pls_index == pls->index) { + c->switch_info[j].delta_timestamp = + av_rescale_q(pls->pkt.dts, get_timebase(pls), AV_TIME_BASE_Q) + - c->switch_info[j].first_timestamp; + av_log(s, AV_LOG_VERBOSE, "[abr_delta] pls%d dts=%ld\n", pls->index, pls->pkt.dts); + } + if (c->switch_info[j].first_timestamp == AV_NOPTS_VALUE + && c->switch_info[j].pls_index == pls->index) { + c->switch_info[j].first_timestamp = av_rescale_q(pls->pkt.dts, + get_timebase(pls), AV_TIME_BASE_Q); + av_log(s, AV_LOG_VERBOSE, "[abr_first] pls%d dts=%ld\n", pls->index, pls->pkt.dts); + } + } + if ((c->switch_info[SWITCH_AUDIO].pls_index == -1 || + c->switch_info[SWITCH_AUDIO].delta_timestamp != AV_NOPTS_VALUE) + && (c->switch_info[SWITCH_VIDEO].pls_index == -1 || + c->switch_info[SWITCH_VIDEO].delta_timestamp != AV_NOPTS_VALUE)) { + c->switch_inited = 1; + if (c->switch_info[SWITCH_VIDEO].pls_index != -1) + av_log(s, AV_LOG_VERBOSE, "[abr_init] video:first=%ld, delta=%ld\n", + c->switch_info[SWITCH_VIDEO].first_timestamp, + c->switch_info[SWITCH_VIDEO].delta_timestamp); + if (c->switch_info[SWITCH_AUDIO].pls_index != -1) + av_log(s, AV_LOG_VERBOSE, "[abr_init] audio:first=%ld, delta=%ld\n", + c->switch_info[SWITCH_AUDIO].first_timestamp, + c->switch_info[SWITCH_AUDIO].delta_timestamp); + } + } if (c->first_timestamp == AV_NOPTS_VALUE && pls->pkt.dts != AV_NOPTS_VALUE) c->first_timestamp = av_rescale_q(pls->pkt.dts, @@ -2367,6 +2758,8 @@ static int hls_probe(const AVProbeData *p) #define OFFSET(x) offsetof(HLSContext, x) #define FLAGS AV_OPT_FLAG_DECODING_PARAM static const AVOption hls_options[] = { + {"abr", "enable abr to switch streams", + OFFSET(abr), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, FLAGS }, {"live_start_index", "segment index to start live streams at (negative values are from the end)", OFFSET(live_start_index), AV_OPT_TYPE_INT, {.i64 = -3}, INT_MIN, INT_MAX, FLAGS}, {"allowed_extensions", "List of file extensions that hls is allowed to access", From patchwork Sun Aug 23 12:23:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21841 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 10CBB449B15 for ; Sun, 23 Aug 2020 15:24:26 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DD40C6899A8; Sun, 23 Aug 2020 15:24:25 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 832B968827C for ; Sun, 23 Aug 2020 15:24:18 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id D74261008CBCE; Sun, 23 Aug 2020 20:24:15 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id E0311200B44A4; Sun, 23 Aug 2020 20:24:14 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id tRW8siZiBNGc; Sun, 23 Aug 2020 20:24:14 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id DD107200A4B0C; Sun, 23 Aug 2020 20:24:10 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 23 Aug 2020 20:23:52 +0800 Message-Id: <20200823122355.188611-4-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC v3 4/7] avformat/hls allow crypto works as ffabr's subprotocol X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc For example, ffabr:crypto+http://xxx.m3u8 Signed-off-by: spartazhc --- libavformat/hls.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/libavformat/hls.c b/libavformat/hls.c index 4e760f8e8a..37a5a017b1 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -765,6 +765,8 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, is_http = 1; } else if (av_strstart(proto_name, "data", NULL)) { ; + } else if (av_strstart(proto_name, "crypto", NULL)) { + ; } else return AVERROR_INVALIDDATA; @@ -774,7 +776,8 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, ; else if (av_strstart(url, "data", NULL) && !strncmp(proto_name, url + 5, strlen(proto_name)) && url[5 + strlen(proto_name)] == ':') ; - else if (av_strstart(url, "ffabr", NULL) && !strncmp(proto_name, url + 6, strlen(proto_name)) && url[6 + strlen(proto_name)] == ':') + else if (av_strstart(url, "ffabr", NULL) && !strncmp(proto_name, url + 6, strlen(proto_name)) + && (url[6 + strlen(proto_name)] == ':' || url[6 + strlen(proto_name)] == '+')) ; else if (strcmp(proto_name, "file") || !strncmp(url, "file,", 5)) return AVERROR_INVALIDDATA; @@ -1455,7 +1458,7 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); } } else if (seg->key_type == KEY_AES_128) { - char iv[33], key[33], url[MAX_URL_SIZE]; + char iv[33], key[33], url[MAX_URL_SIZE], prefix[13]; if (strcmp(seg->key, pls->key_url)) { AVIOContext *pb = NULL; if (open_url(pls->parent, &pb, seg->key, c->avio_opts, opts, NULL) == 0) { @@ -1474,10 +1477,15 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, ff_data_to_hex(iv, seg->iv, sizeof(seg->iv), 0); ff_data_to_hex(key, pls->key, sizeof(pls->key), 0); iv[32] = key[32] = '\0'; + if (c->abr) + sprintf(prefix, "ffabr:crypto"); + else + sprintf(prefix, "crypto"); + if (strstr(seg->url, "://")) - snprintf(url, sizeof(url), "crypto+%s", seg->url); + snprintf(url, sizeof(url), "%s+%s", prefix, seg->url); else - snprintf(url, sizeof(url), "crypto:%s", seg->url); + snprintf(url, sizeof(url), "%s:%s", prefix, seg->url); av_dict_set(&opts, "key", key, 0); av_dict_set(&opts, "iv", iv, 0); From patchwork Sun Aug 23 12:23:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21842 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 17B56449B15 for ; Sun, 23 Aug 2020 15:24:27 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EE5A16883D0; Sun, 23 Aug 2020 15:24:26 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0B91D688336 for ; Sun, 23 Aug 2020 15:24:20 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 4CF441008CBC2; Sun, 23 Aug 2020 20:24:17 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 4709B200B448D; Sun, 23 Aug 2020 20:24:17 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 8z7SOVzqr9lx; Sun, 23 Aug 2020 20:24:16 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id A32D8200A4B01; Sun, 23 Aug 2020 20:24:14 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 23 Aug 2020 20:23:53 +0800 Message-Id: <20200823122355.188611-5-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC v3 5/7] ffplay: add an option to enable abr X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add abr option, ffplay can play hls using abr by: ffplay -i http://xxx/master.m3u8 -abr Structure ABRList is added to save stream type and index, it is used to allow packet_queue_put function to put pkt which from same type(for example: video pkt) but different stream index to queue. Signed-off-by: spartazhc v1 fixed: None. v2 fixed: 1. check malloc result and return error message Signed-off-by: spartazhc --- doc/ffplay.texi | 2 + fftools/ffplay.c | 151 ++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 144 insertions(+), 9 deletions(-) diff --git a/doc/ffplay.texi b/doc/ffplay.texi index f3761bb12e..6a24542cda 100644 --- a/doc/ffplay.texi +++ b/doc/ffplay.texi @@ -46,6 +46,8 @@ Disable audio. Disable video. @item -sn Disable subtitles. +@item -abr +Enable adaptive bitrate for hls/dash. @item -ss @var{pos} Seek to @var{pos}. Note that in most formats it is not possible to seek exactly, so @command{ffplay} will seek to the nearest seek point to diff --git a/fftools/ffplay.c b/fftools/ffplay.c index 6c9c041e9a..30944ba4be 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -201,6 +201,15 @@ typedef struct Decoder { SDL_Thread *decoder_tid; } Decoder; +typedef struct ABRList { + int **audio_list; + int audios; + int **video_list; + int videos; + int **sub_list; + int subs; +} ABRList; + typedef struct VideoState { SDL_Thread *read_tid; AVInputFormat *iformat; @@ -305,6 +314,8 @@ typedef struct VideoState { int last_video_stream, last_audio_stream, last_subtitle_stream; SDL_cond *continue_read_thread; + + ABRList *abr_list; } VideoState; /* options specified by the user */ @@ -356,6 +367,7 @@ static char *afilters = NULL; static int autorotate = 1; static int find_stream_info = 1; static int filter_nbthreads = 0; +static int abr = 0; /* current context */ static int is_full_screen; @@ -1262,6 +1274,29 @@ static void stream_component_close(VideoState *is, int stream_index) } } +static void free_abr_dynarray(int **list, int num) +{ + for (int i = 0; i < num; i++) { + av_free(list[i]); + } +} + +static void free_abr_list(ABRList *abrlist) +{ + if (abrlist->audios) { + free_abr_dynarray(abrlist->audio_list, abrlist->audios); + av_freep(&abrlist->audio_list); + } + if (abrlist->videos) { + free_abr_dynarray(abrlist->video_list, abrlist->videos); + av_freep(&abrlist->video_list); + } + if (abrlist->subs) { + free_abr_dynarray(abrlist->sub_list, abrlist->subs); + av_freep(&abrlist->sub_list); + } +} + static void stream_close(VideoState *is) { /* XXX: use a special url_shutdown call to abort parse cleanly */ @@ -2753,6 +2788,67 @@ static int is_realtime(AVFormatContext *s) return 0; } +static av_cold int abr_init_list(VideoState *is) +{ + int stream_index, *tmp; + AVStream *st; + int nb_streams = is->ic->nb_streams; + ABRList *abrlist = is->abr_list; + + for (stream_index = 0; stream_index < nb_streams; stream_index++) { + st = is->ic->streams[stream_index]; + tmp = av_memdup(&stream_index, sizeof(int)); + if (!tmp) + return AVERROR(ENOMEM); + switch (st->codecpar->codec_type) { + case AVMEDIA_TYPE_AUDIO: + av_dynarray_add(&abrlist->audio_list, &abrlist->audios, tmp); + break; + case AVMEDIA_TYPE_VIDEO: + av_dynarray_add(&abrlist->video_list, &abrlist->videos, tmp); + break; + case AVMEDIA_TYPE_SUBTITLE: + av_dynarray_add(&abrlist->sub_list, &abrlist->subs, tmp); + break; + default: + av_free(tmp); + break; + } + } + return 0; +} + +static int abr_check_list(ABRList *abr_list, enum AVMediaType type, int st) +{ + int **st_list; + int n_st; + switch (type) { + case AVMEDIA_TYPE_AUDIO: + st_list = abr_list->audio_list; + n_st = abr_list->audios; + break; + case AVMEDIA_TYPE_VIDEO: + st_list = abr_list->video_list; + n_st = abr_list->videos; + break; + case AVMEDIA_TYPE_SUBTITLE: + st_list = abr_list->sub_list; + n_st = abr_list->subs; + break; + default: + break; + } + if (!st_list) + return 0; + for (int i = 0; i < n_st; i++) { + if (*st_list[i] == st) + return 1; + } + return 0; +} + + + /* this thread gets the stream from the disk or the network */ static int read_thread(void *arg) { @@ -2789,6 +2885,8 @@ static int read_thread(void *arg) av_dict_set(&format_opts, "scan_all_pmts", "1", AV_DICT_DONT_OVERWRITE); scan_all_pmts_set = 1; } + if (abr) + av_dict_set(&format_opts, "abr", "1", 0); err = avformat_open_input(&ic, is->filename, is->iformat, &format_opts); if (err < 0) { print_error(is->filename, err); @@ -2918,6 +3016,21 @@ static int read_thread(void *arg) stream_component_open(is, st_index[AVMEDIA_TYPE_SUBTITLE]); } + /* clean packet list filled in hls_read_header if abr is enabled */ + if (abr) { + is->abr_list = av_mallocz(sizeof(ABRList)); + if (!is->abr_list) { + ret = AVERROR(ENOMEM); + goto fail; + } + ret = abr_init_list(is); + if (ret < 0) { + av_log(NULL, AV_LOG_FATAL, "Failed to initiate abr_list\n"); + ret = -1; + goto fail; + } + } + if (is->video_stream < 0 && is->audio_stream < 0) { av_log(NULL, AV_LOG_FATAL, "Failed to open file '%s' or configure filtergraph\n", is->filename); @@ -3045,15 +3158,31 @@ static int read_thread(void *arg) av_q2d(ic->streams[pkt->stream_index]->time_base) - (double)(start_time != AV_NOPTS_VALUE ? start_time : 0) / 1000000 <= ((double)duration / 1000000); - if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { - packet_queue_put(&is->audioq, pkt); - } else if (pkt->stream_index == is->video_stream && pkt_in_play_range - && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { - packet_queue_put(&is->videoq, pkt); - } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { - packet_queue_put(&is->subtitleq, pkt); + if (abr) { + if ((pkt->stream_index == is->audio_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_AUDIO, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if ((pkt->stream_index == is->video_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_VIDEO, pkt->stream_index)) && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if ((pkt->stream_index == is->subtitle_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_SUBTITLE, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } else { - av_packet_unref(pkt); + if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if (pkt->stream_index == is->video_stream && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } } @@ -3061,7 +3190,10 @@ static int read_thread(void *arg) fail: if (ic && !is->ic) avformat_close_input(&ic); - + if (abr && is->abr_list) { + free_abr_list(is->abr_list); + av_freep(&is->abr_list); + } if (ret != 0) { SDL_Event event; @@ -3588,6 +3720,7 @@ static const OptionDef options[] = { { "an", OPT_BOOL, { &audio_disable }, "disable audio" }, { "vn", OPT_BOOL, { &video_disable }, "disable video" }, { "sn", OPT_BOOL, { &subtitle_disable }, "disable subtitling" }, + { "abr", OPT_BOOL, { &abr }, "enable adaptive bitrate for hls/dash" }, { "ast", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_AUDIO] }, "select desired audio stream", "stream_specifier" }, { "vst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_VIDEO] }, "select desired video stream", "stream_specifier" }, { "sst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_SUBTITLE] }, "select desired subtitle stream", "stream_specifier" }, From patchwork Sun Aug 23 12:23:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21843 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 285E8449B15 for ; Sun, 23 Aug 2020 15:24:29 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0C12B689EA6; Sun, 23 Aug 2020 15:24:29 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 94D2968827C for ; Sun, 23 Aug 2020 15:24:21 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id EB1BB1008CBC4; Sun, 23 Aug 2020 20:24:18 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id E71B8200B449C; Sun, 23 Aug 2020 20:24:18 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id tQB8_ayXvIIB; Sun, 23 Aug 2020 20:24:18 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 472C0200B4496; Sun, 23 Aug 2020 20:24:16 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 23 Aug 2020 20:23:54 +0800 Message-Id: <20200823122355.188611-6-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC v3 6/7] avformat/hls add metadata "abr_initial" for ffplay X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add an AVDictionary option "abr_initial", which could be used to send message to ffplay. Currently, the first entry "abr_init_duration" is added. Signed-off-by: spartazhc --- doc/demuxers.texi | 3 +++ libavformat/hls.c | 14 ++++++++++++++ 2 files changed, 17 insertions(+) diff --git a/doc/demuxers.texi b/doc/demuxers.texi index 4cdbd95962..761a372b86 100644 --- a/doc/demuxers.texi +++ b/doc/demuxers.texi @@ -324,6 +324,9 @@ It accepts the following options: @item abr enable abr to switch streams. +@item abr_initial +metadata used by abr for ffplay. + @item live_start_index segment index to start live streams at (negative values are from the end). diff --git a/libavformat/hls.c b/libavformat/hls.c index 37a5a017b1..5c251cd002 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -243,6 +243,7 @@ typedef struct HLSContext { AVIOContext *playlist_pb; int abr; + AVDictionary *abr_initial; struct throughput *throughputs; struct switch_task *switch_tasks; struct switch_info *switch_info; @@ -2308,6 +2309,7 @@ static int hls_read_header(AVFormatContext *s) } if (c->abr) { + int64_t abr_init_duration = AV_NOPTS_VALUE; c->switch_tasks = av_malloc(c->n_playlists * sizeof(struct switch_task)); if (!c->switch_tasks) { ret = AVERROR(ENOMEM); @@ -2326,6 +2328,15 @@ static int hls_read_header(AVFormatContext *s) for (i = 0; i < c->n_playlists; i++) { struct playlist *pls = c->playlists[i]; int type; + int64_t start = 0; + for (int i = 0; i < c->cur_seq_no + 1; i++) { + start += pls->segments[i]->duration; + } + start = av_rescale_q(start, + AV_TIME_BASE_Q, + get_timebase(pls)); + if (abr_init_duration == AV_NOPTS_VALUE || abr_init_duration > start) + abr_init_duration = start; c->switch_tasks[i].switch_timestamp = AV_NOPTS_VALUE; type = playlist_type_simple(pls); @@ -2337,6 +2348,7 @@ static int hls_read_header(AVFormatContext *s) c->switch_info[SWITCH_VIDEO].pls_index = pls->index; } } + av_dict_set_int(&c->abr_initial, "abr_init_duration", abr_init_duration, 0); } update_noheader_flag(s); @@ -2768,6 +2780,8 @@ static int hls_probe(const AVProbeData *p) static const AVOption hls_options[] = { {"abr", "enable abr to switch streams", OFFSET(abr), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, FLAGS }, + {"abr_initial", "metadata used by abr for ffplay", + OFFSET(abr_initial), AV_OPT_TYPE_DICT, { 0 }, 0, 0, FLAGS }, {"live_start_index", "segment index to start live streams at (negative values are from the end)", OFFSET(live_start_index), AV_OPT_TYPE_INT, {.i64 = -3}, INT_MIN, INT_MAX, FLAGS}, {"allowed_extensions", "List of file extensions that hls is allowed to access", From patchwork Sun Aug 23 12:23:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21844 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 40D78449B15 for ; Sun, 23 Aug 2020 15:24:30 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 24383689F10; Sun, 23 Aug 2020 15:24:30 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 53D21689C71 for ; Sun, 23 Aug 2020 15:24:22 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id D02FC1008CBCD; Sun, 23 Aug 2020 20:24:19 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id CCED1200B448D; Sun, 23 Aug 2020 20:24:19 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id WKtlG-0Z_Vfi; Sun, 23 Aug 2020 20:24:19 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id E80DB200A4B00; Sun, 23 Aug 2020 20:24:18 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 23 Aug 2020 20:23:55 +0800 Message-Id: <20200823122355.188611-7-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200823122355.188611-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC v3 7/7] ffplay: fix for unwanted pkts when using abr X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc HLS use abr may switch streams, ffplay should allow pkt whic same type (for example: video pkt) but different stream index to queue. But in the begining, avformat_find_stream_info() will add pkts to packet_buffer from all streams to find info. This will cause problem because these unwanted pkts can be added to ffplay's packet_queue now. The exactly packet number added is depend on stream, but it should be less than the packet number in the segments which are downloaded during hls_read_headeri(). So abr_init_duration will be passed to ffplay via abr_initial, pkts do not needed will be unref as usual. Signed-off-by: spartazhc --- fftools/ffplay.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index 30944ba4be..5ec2a8578c 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -2858,6 +2858,7 @@ static int read_thread(void *arg) int st_index[AVMEDIA_TYPE_NB]; AVPacket pkt1, *pkt = &pkt1; int64_t stream_start_time; + int64_t abr_init_duration = 0; int pkt_in_play_range = 0; AVDictionaryEntry *t; SDL_mutex *wait_mutex = SDL_CreateMutex(); @@ -3018,6 +3019,21 @@ static int read_thread(void *arg) /* clean packet list filled in hls_read_header if abr is enabled */ if (abr) { + AVDictionary *abr_initial = NULL; + AVDictionaryEntry *en = NULL; + av_opt_get_dict_val(ic, "abr_initial", AV_OPT_SEARCH_CHILDREN, &abr_initial); + en = av_dict_get(abr_initial, "abr_init_duration", NULL, 0); + if (en) { + if (st_index[AVMEDIA_TYPE_VIDEO] != -1) { + abr_init_duration = strtol(en->value, NULL, 10); + if (abr_init_duration < 0) { + ret = AVERROR(EINVAL); + goto fail; + } + } + av_log(NULL, AV_LOG_VERBOSE, "[ffplay-abr]: abr_init_duration=%ld\n", abr_init_duration); + } + av_dict_free(&abr_initial); is->abr_list = av_mallocz(sizeof(ABRList)); if (!is->abr_list) { ret = AVERROR(ENOMEM); @@ -3158,12 +3174,13 @@ static int read_thread(void *arg) av_q2d(ic->streams[pkt->stream_index]->time_base) - (double)(start_time != AV_NOPTS_VALUE ? start_time : 0) / 1000000 <= ((double)duration / 1000000); - if (abr) { + if (abr && pkt_ts >= (stream_start_time + abr_init_duration)) { + int test = abr_check_list(is->abr_list, AVMEDIA_TYPE_VIDEO, pkt->stream_index); if ((pkt->stream_index == is->audio_stream || abr_check_list(is->abr_list, AVMEDIA_TYPE_AUDIO, pkt->stream_index)) && pkt_in_play_range) { packet_queue_put(&is->audioq, pkt); } else if ((pkt->stream_index == is->video_stream - || abr_check_list(is->abr_list, AVMEDIA_TYPE_VIDEO, pkt->stream_index)) && pkt_in_play_range + || (is->video_stream != -1 && abr_check_list(is->abr_list, AVMEDIA_TYPE_VIDEO, pkt->stream_index))) && pkt_in_play_range && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { packet_queue_put(&is->videoq, pkt); } else if ((pkt->stream_index == is->subtitle_stream