From patchwork Sun Jul 5 11:34:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 20814 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 8736544BDAF for ; Sun, 5 Jul 2020 14:35:30 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 704EA68AFA5; Sun, 5 Jul 2020 14:35:30 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8DCA7687F29 for ; Sun, 5 Jul 2020 14:35:22 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id C3C9E1008CBC2; Sun, 5 Jul 2020 19:35:17 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id A613B200B4496; Sun, 5 Jul 2020 19:35:17 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7HtvFCygUJaP; Sun, 5 Jul 2020 19:35:17 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id B9EB6200B4497; Sun, 5 Jul 2020 19:35:16 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Jul 2020 19:34:54 +0800 Message-Id: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC 1/6] avformat/abr: Adaptive Bitrate support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add abr module for hls/dash. Signed-off-by: spartazhc --- doc/protocols.texi | 7 + libavformat/Makefile | 1 + libavformat/abr.c | 282 ++++++++++++++++++++++++++++++++++++++++ libavformat/protocols.c | 1 + 4 files changed, 291 insertions(+) create mode 100644 libavformat/abr.c diff --git a/doc/protocols.texi b/doc/protocols.texi index 644a17963d..fc80209884 100644 --- a/doc/protocols.texi +++ b/doc/protocols.texi @@ -51,6 +51,13 @@ in microseconds. A description of the currently available protocols follows. +@section abr + +Adaptive bitrate sub-protocol work for hls/dash. + +The abr protocol takes stream information from hls/dash as input, +use bandwidth estimation to decide whether to switch or not. + @section amqp Advanced Message Queueing Protocol (AMQP) version 0-9-1 is a broker based diff --git a/libavformat/Makefile b/libavformat/Makefile index 26af859a28..3c08289d5e 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -588,6 +588,7 @@ OBJS-$(CONFIG_LIBOPENMPT_DEMUXER) += libopenmpt.o OBJS-$(CONFIG_VAPOURSYNTH_DEMUXER) += vapoursynth.o # protocols I/O +OBJS-$(CONFIG_ABR_PROTOCOL) += abr.o OBJS-$(CONFIG_ASYNC_PROTOCOL) += async.o OBJS-$(CONFIG_APPLEHTTP_PROTOCOL) += hlsproto.o OBJS-$(CONFIG_BLURAY_PROTOCOL) += bluray.o diff --git a/libavformat/abr.c b/libavformat/abr.c new file mode 100644 index 0000000000..b7d00efcae --- /dev/null +++ b/libavformat/abr.c @@ -0,0 +1,282 @@ +/* + * Adaptive Bitrate Module for HLS / DASH + * Copyright (c) 2020 Hongcheng Zhong + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "avformat.h" +#include "libavutil/opt.h" +#include "libavutil/time.h" +#include "libavutil/avstring.h" +#include "url.h" +#include + +enum ABRFormatType { + ABR_TYPE_HLS, + ABR_TYPE_DASH +}; + +typedef struct variant_bitrate { + int value; + int index; +} variant_bitrate; + +typedef struct ABRContext { + AVClass *class; + URLContext *hd; + AVDictionary *abr_params; + AVDictionary *abr_metadata; + enum ABRFormatType format; + int cur_pls; + int can_switch; + int n_variants; + variant_bitrate *variants_bitrate; + int index; + int n_throughputs; + float *throughputs; + int64_t position; +} ABRContext; + +static float harmonic_mean(int num, float* arr) +{ + float tmp = 0; + + if (num <= 0) return 0; + + for (size_t i = 0; i < num; i++) { + tmp += 1 / arr[i]; + } + + return num / tmp; +} + +static int hls_param_parse(ABRContext *c, const char *key, const char *value) +{ + if (!av_strcasecmp(key, "cur_pls")) { + c->cur_pls = atoi(value); + } else if (!av_strcasecmp(key, "can_switch")) { + c->can_switch = atoi(value); + } else if (!av_strcasecmp(key, "n_variants")) { + c->n_variants = atoi(value); + c->variants_bitrate = av_mallocz(sizeof(variant_bitrate) * c->n_variants); + if (!c->variants_bitrate) + return -1; + } else if (av_strstart(key, "variant_bitrate", NULL)) { + c->variants_bitrate[c->index].value = atoi(value); + c->variants_bitrate[c->index].index = c->index; + c->index++; + } else if (!av_strcasecmp(key, "n_throughputs")) { + c->n_throughputs = atoi(value); + c->index = 0; + if (c->n_throughputs > 0) { + c->throughputs = av_malloc(sizeof(float) * c->n_throughputs); + if (!c->throughputs) + return -1; + } + } else if (av_strstart(key, "throughputs", NULL)) + c->throughputs[c->index++] = atof(value); + return 0; +} + +static int dash_param_parse(ABRContext *c, const char *key, const char *value) +{ + return 0; +} + +static int abr_param_parse(ABRContext *c, enum ABRFormatType type, const char *key, const char *value) +{ + if (type == ABR_TYPE_HLS) { + hls_param_parse(c, key, value); + } else if (type == ABR_TYPE_DASH) { + dash_param_parse(c, key, value); + } + return 0; +} + +static int compare_vb(const void *a, const void *b) +{ + variant_bitrate *a1 = (variant_bitrate *)a; + variant_bitrate *a2 = (variant_bitrate *)b; + return (*a2).value - (*a1).value; +} + +static int abr_rule(ABRContext *c, float bw_estimate) +{ + int ret = -1; + + if (c->n_throughputs > 6) { + if (bw_estimate < c->variants_bitrate[c->cur_pls].value / 1000 * 1.2 && + bw_estimate > c->variants_bitrate[c->cur_pls].value / 1000 * 0.8) + return -1; + qsort(c->variants_bitrate, c->n_variants, sizeof(variant_bitrate), compare_vb); + for (int i = 0; i < c->n_variants; i++) { + if (bw_estimate > c->variants_bitrate[i].value / 1000) { + ret = i; + break; + } + } + } + if (ret == c->cur_pls) + ret = -1; + av_log(c, AV_LOG_VERBOSE, "[switch] bwe=%.2fkbps, cur=%d, switch=%d\n", bw_estimate, c->cur_pls, ret); + return ret; +} + +static int abr_open(URLContext *h, const char *uri, int flags, AVDictionary **options) +{ + const char *nested_url; + uint64_t start, end; + float bw_estimation; + int switch_request = -1; + int ret = 0; + ABRContext *c = h->priv_data; + + if (!av_strstart(uri, "abr+", &nested_url) && + !av_strstart(uri, "abr:", &nested_url)) { + av_log(h, AV_LOG_ERROR, "Unsupported url %s\n", uri); + ret = AVERROR(EINVAL); + goto err; + } + + { + AVDictionaryEntry *en = NULL; + en = av_dict_get(c->abr_params, "", en, AV_DICT_IGNORE_SUFFIX); + if (!en) + return -1; + if (!av_strcasecmp(en->key, "format")) { + if (!av_strcasecmp(en->value, "hls")) { + c->format = ABR_TYPE_HLS; + } else if (!av_strcasecmp(en->value, "dash")) { + c->format = ABR_TYPE_DASH; + } + av_log(h, AV_LOG_VERBOSE, "%s is using ABR\n", en->value); + } + + while (en = av_dict_get(c->abr_params, "", en, AV_DICT_IGNORE_SUFFIX)) { + if (abr_param_parse(c, c->format, en->key, en->value) < 0) + av_log(h, AV_LOG_WARNING, + "Error parsing option '%s = %s'.\n", + en->key, en->value); + } + } + + start = av_gettime(); + if ((ret = ffurl_open_whitelist(&c->hd, nested_url, flags, + &h->interrupt_callback, options, + h->protocol_whitelist, h->protocol_blacklist, h)) < 0) { + av_log(h, AV_LOG_ERROR, "Unable to open resource: %s\n", nested_url); + goto err; + } + end = av_gettime(); + + bw_estimation = harmonic_mean(c->n_throughputs, c->throughputs); + + if (c->can_switch == 1) + switch_request = abr_rule(c, bw_estimation); + + av_dict_set_int(&c->abr_metadata, "download_time", (end - start), 0); + av_dict_set_int(&c->abr_metadata, "switch_request", switch_request, 0); + +err: + return ret; +} + + +static int abr_read(URLContext *h, uint8_t *buf, int size) +{ + ABRContext *c = h->priv_data; + + return ffurl_read(c->hd, buf, size); +} + +static int64_t abr_seek(URLContext *h, int64_t pos, int whence) +{ + ABRContext *c = h->priv_data; + int64_t newpos; + + switch (whence) { + case SEEK_SET: + break; + case SEEK_CUR: + pos = pos + c->position; + break; + case SEEK_END: { + int64_t newpos = ffurl_seek( c->hd, pos, AVSEEK_SIZE ); + if (newpos < 0) { + av_log(h, AV_LOG_ERROR, + "ABR: seek_end - can't get file size (pos=%lld)\r\n", (long long int)pos); + return newpos; + } + pos = newpos - pos; + } + break; + case AVSEEK_SIZE: { + int64_t newpos = ffurl_seek( c->hd, pos, AVSEEK_SIZE ); + return newpos; + } + break; + default: + av_log(h, AV_LOG_ERROR, + "ABR: no support for seek where 'whence' is %d\r\n", whence); + return AVERROR(EINVAL); + } + + newpos = ffurl_seek( c->hd, c->position, SEEK_SET ); + if (newpos < 0) { + av_log(h, AV_LOG_ERROR, + "ABR: nested protocol no support for seek or seek failed\n"); + return newpos; + } + return c->position; +} + +static int abr_close(URLContext *h) +{ + ABRContext *c = h->priv_data; + int ret = 0; + + ffurl_closep(&c->hd); + av_free(c->variants_bitrate); + av_free(c->throughputs); + return ret; +} + +#define OFFSET(x) offsetof(ABRContext, x) +#define D AV_OPT_FLAG_DECODING_PARAM +static const AVOption options[] = { + { "abr-params", "Informations ABR needed, using a :-separated list of key=value parameters", OFFSET(abr_params), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { "abr-metadata", "Metadata return from abr, including switch signal and network bandwidth", OFFSET(abr_metadata), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { NULL } +}; + +static const AVClass abr_class = { + .class_name = "abr", + .item_name = av_default_item_name, + .option = options, + .version = LIBAVUTIL_VERSION_INT, +}; + +const URLProtocol ff_abr_protocol = { + .name = "abr", + .url_open2 = abr_open, + .url_read = abr_read, + .url_seek = abr_seek, + .url_close = abr_close, + .priv_data_size = sizeof(ABRContext), + .priv_data_class = &abr_class, +}; diff --git a/libavformat/protocols.c b/libavformat/protocols.c index 7df18fbb3b..16c844c914 100644 --- a/libavformat/protocols.c +++ b/libavformat/protocols.c @@ -23,6 +23,7 @@ #include "url.h" +extern const URLProtocol ff_abr_protocol; extern const URLProtocol ff_async_protocol; extern const URLProtocol ff_bluray_protocol; extern const URLProtocol ff_cache_protocol; From patchwork Sun Jul 5 11:34:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 20813 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 891D444BDAF for ; Sun, 5 Jul 2020 14:35:29 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6667E689A4B; Sun, 5 Jul 2020 14:35:29 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C89DA687F9F for ; Sun, 5 Jul 2020 14:35:22 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 081161008CBC4; Sun, 5 Jul 2020 19:35:19 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id E526B200B4497; Sun, 5 Jul 2020 19:35:19 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id a8hX0M5_YXpW; Sun, 5 Jul 2020 19:35:19 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 5B777200B4496; Sun, 5 Jul 2020 19:35:17 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Jul 2020 19:34:55 +0800 Message-Id: <20200705113459.77467-2-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC 2/6] avformat/http: Add abr to whitelist X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc add abr protocol to http's whitelist Signed-off-by: spartazhc --- libavformat/http.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavformat/http.c b/libavformat/http.c index 6c39da1a8b..6a58c9afef 100644 --- a/libavformat/http.c +++ b/libavformat/http.c @@ -1815,7 +1815,7 @@ const URLProtocol ff_http_protocol = { .priv_data_size = sizeof(HTTPContext), .priv_data_class = &http_context_class, .flags = URL_PROTOCOL_FLAG_NETWORK, - .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data" + .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data,abr" }; #endif /* CONFIG_HTTP_PROTOCOL */ From patchwork Sun Jul 5 11:34:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 20815 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 6B16244BDAF for ; Sun, 5 Jul 2020 14:35:31 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3EB0968B0C2; Sun, 5 Jul 2020 14:35:31 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0438468AA64 for ; Sun, 5 Jul 2020 14:35:25 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 2CB7A1008CBCD; Sun, 5 Jul 2020 19:35:22 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 1B6F2200B4497; Sun, 5 Jul 2020 19:35:22 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id IETI3w61eoTk; Sun, 5 Jul 2020 19:35:22 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id A7FF6200B4496; Sun, 5 Jul 2020 19:35:19 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Jul 2020 19:34:56 +0800 Message-Id: <20200705113459.77467-3-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC 3/6] avformat/hls: use abr to switch streams X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc When abr is enable, it will take over the task to call http to download segments, and will return a switch-request for hls to switch streams. For reason not to waste segments that have been downloaded, switch will become effective after old segments is used out. Abr cannot work with http_persistent option, and currently use http_multiple. Signed-off-by: spartazhc --- doc/demuxers.texi | 3 + libavformat/hls.c | 222 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 218 insertions(+), 7 deletions(-) diff --git a/doc/demuxers.texi b/doc/demuxers.texi index 3c15ab9eee..4cdbd95962 100644 --- a/doc/demuxers.texi +++ b/doc/demuxers.texi @@ -321,6 +321,9 @@ available in a metadata key named "variant_bitrate". It accepts the following options: @table @option +@item abr +enable abr to switch streams. + @item live_start_index segment index to start live streams at (negative values are from the end). diff --git a/libavformat/hls.c b/libavformat/hls.c index 3798bdd5d1..b8f202e6a8 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -189,6 +189,15 @@ struct variant { char subtitles_group[MAX_FIELD_LEN]; }; +struct throughput { + int n_throughputs; + + /* throughputs are in kbps, always record the last 20 times */ + float throughput_fifo[20]; + int head; + int tail; +}; + typedef struct HLSContext { AVClass *class; AVFormatContext *ctx; @@ -213,8 +222,36 @@ typedef struct HLSContext { int http_multiple; int http_seekable; AVIOContext *playlist_pb; + + int abr; + struct throughput *throughputs; + int can_switch; + int switch_request2; + int switch_delay; + int64_t switch_timestamp; + int64_t delta_timestamp; + int cur_pls; } HLSContext; +static struct segment *next_segment(struct playlist *pls); +static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in); + +static void sync_cur_seq(HLSContext *c) { + int i; + for (i = 0; i < c->n_playlists; i++) { + struct playlist *pls = c->playlists[i]; + pls->cur_seq_no = c->cur_seq_no; + } +} + +static struct segment *next2_segment(struct playlist *pls) +{ + int n = pls->cur_seq_no - pls->start_seq_no + 2; + if (n >= pls->n_segments) + return NULL; + return pls->segments[n]; +} + static void free_segment_dynarray(struct segment **segments, int n_segments) { int i; @@ -624,6 +661,33 @@ static int open_url_keepalive(AVFormatContext *s, AVIOContext **pb, #endif } +static int update_throughputs(struct throughput *thr, float time, int pb_size) +{ + if (pb_size <= 0 || time <= 0) + return -1; + if (thr->n_throughputs < 20) { + ++thr->n_throughputs; + thr->throughput_fifo[thr->tail] = (float)(pb_size) / time; + } else { + thr->throughput_fifo[thr->tail] = (float)(pb_size) / time; + ++thr->head; + } + + thr->tail = (thr->tail + 1) % 20; + return 0; +} + +static int64_t get_switch_timestamp(HLSContext *c, struct playlist *pls) +{ + int64_t pos = c->first_timestamp == AV_NOPTS_VALUE ? + 0 : c->first_timestamp; + + for (int i = 0; i < pls->cur_seq_no + 2; i++) { + pos += pls->segments[i]->duration; + } + return pos; +} + static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, AVDictionary *opts, AVDictionary *opts2, int *is_http_out) { @@ -633,7 +697,10 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, int ret; int is_http = 0; - if (av_strstart(url, "crypto", NULL)) { + if (av_strstart(url, "abr", NULL)) { + if (url[3] == '+' || url[3] == ':') + proto_name = avio_find_protocol_name(url + 4); + } else if (av_strstart(url, "crypto", NULL)) { if (url[6] == '+' || url[6] == ':') proto_name = avio_find_protocol_name(url + 7); } else if (av_strstart(url, "data", NULL)) { @@ -665,6 +732,8 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, if (!strncmp(proto_name, url, strlen(proto_name)) && url[strlen(proto_name)] == ':') ; + else if (av_strstart(url, "abr", NULL) && !strncmp(proto_name, url + 4, strlen(proto_name)) && url[4 + strlen(proto_name)] == ':') + ; else if (av_strstart(url, "crypto", NULL) && !strncmp(proto_name, url + 7, strlen(proto_name)) && url[7 + strlen(proto_name)] == ':') ; else if (av_strstart(url, "data", NULL) && !strncmp(proto_name, url + 5, strlen(proto_name)) && url[5 + strlen(proto_name)] == ':') @@ -690,6 +759,31 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, } else { ret = s->io_open(s, pb, url, AVIO_FLAG_READ, &tmp); } + if (c->abr) { + AVDictionary *abr_ret = NULL; + AVDictionaryEntry *en = NULL; + struct segment *seg; + int pb_size, switch_request; + av_opt_get_dict_val(*pb, "abr-metadata", AV_OPT_SEARCH_CHILDREN, &abr_ret); + en = av_dict_get(abr_ret, "download_time", NULL, 0); + pb_size = avio_size(*pb); + update_throughputs(c->throughputs, atoi(en->value) / 1000.0, pb_size); + av_log(s, AV_LOG_VERBOSE, "[abr] time=%.2fms, size=%.2fkb\n", atoi(en->value) / 1000.0, pb_size / 1000.0); + en = av_dict_get(abr_ret, "switch_request", NULL, 0); + switch_request = atoi(en->value); + av_log(s, AV_LOG_VERBOSE, "[abr] switch request: %s\n", en->value); + if (switch_request != -1) { + c->switch_request2 = switch_request; + c->switch_delay = 2; + c->can_switch = 0; + sync_cur_seq(c); + seg = next2_segment(c->playlists[c->switch_request2]); + c->switch_timestamp = get_switch_timestamp(c, c->playlists[c->switch_request2]); + av_log(s, AV_LOG_VERBOSE, "[abr] switch timestamp: %ld\n", c->switch_timestamp); + c->playlists[c->switch_request2]->input_next_requested = 1; + ret = open_input(c, c->playlists[c->switch_request2], seg, &c->playlists[c->switch_request2]->input_next); + } + } if (ret >= 0) { // update cookies on http response with setcookies. char *new_cookies = NULL; @@ -1219,6 +1313,33 @@ static void intercept_id3(struct playlist *pls, uint8_t *buf, pls->is_id3_timestamped = (pls->id3_mpegts_timestamp != AV_NOPTS_VALUE); } +static int abrinfo_to_dict(HLSContext *c, char **abr_info) +{ + struct throughput *thr = c->throughputs; + char buffer[MAX_URL_SIZE]; + int size, i; + size = snprintf(buffer, sizeof(buffer), "format=hls:"); + size += snprintf(buffer + size, sizeof(buffer) - size, "cur_pls=%d:", c->cur_pls); + size += snprintf(buffer + size, sizeof(buffer) - size, "can_switch=%d:", c->can_switch); + size += snprintf(buffer + size, sizeof(buffer) - size, "n_variants=%d:", c->n_variants); + for (i = 0; i < c->n_variants; i++) { + struct variant *v = c->variants[i]; + size += snprintf(buffer + size, sizeof(buffer) - size, "variant_bitrate%d=%d:", i, v->bandwidth); + } + size += snprintf(buffer + size, sizeof(buffer) - size, "n_throughputs=%d:", thr->n_throughputs); + if (thr->n_throughputs > 0) { + i = thr->head; + do { + size += snprintf(buffer + size, sizeof(buffer) - size, "throughputs%d=%.2f:", i, thr->throughput_fifo[i]); + i = (i + 1) % 20; + } while (i != thr->tail); + } + *abr_info = av_malloc(size); + snprintf(*abr_info, size, "%s", buffer); + av_log(c, AV_LOG_DEBUG, "[abr] abr_info: %s\n", *abr_info); + return 0; +} + static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in) { AVDictionary *opts = NULL; @@ -1238,8 +1359,20 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, av_log(pls->parent, AV_LOG_VERBOSE, "HLS request for url '%s', offset %"PRId64", playlist %d\n", seg->url, seg->url_offset, pls->index); + if (c->abr) { + char *abr_opts; + abrinfo_to_dict(c, &abr_opts); + av_dict_set(&opts, "abr-params", abr_opts, 0); // how to setup flag? + av_free(abr_opts); + } if (seg->key_type == KEY_NONE) { - ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + char url_abr[MAX_URL_SIZE]; + if (c->abr) { + snprintf(url_abr, sizeof(url_abr), "abr:%s", seg->url); // + or : tbd + ret = open_url(pls->parent, in, url_abr, c->avio_opts, opts, &is_http); + } else { + ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + } } else if (seg->key_type == KEY_AES_128) { char iv[33], key[33], url[MAX_URL_SIZE]; if (strcmp(seg->key, pls->key_url)) { @@ -1273,6 +1406,7 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, goto cleanup; } ret = 0; + // TODO: Add abr prefix for crypto } else if (seg->key_type == KEY_SAMPLE_AES) { av_log(pls->parent, AV_LOG_ERROR, "SAMPLE-AES encryption is not supported yet\n"); @@ -1435,7 +1569,17 @@ restart: /* Check that the playlist is still needed before opening a new * segment. */ v->needed = playlist_needed(v); - + if (c->abr) { + if (c->switch_request2 == -1) + ; + else if (v->needed && c->switch_request2 != v->index && c->switch_delay <= 0) { + av_log(v->parent, AV_LOG_VERBOSE, "read_data: needed but not download playlist %d ('%s')\n", + v->index, v->url); + return AVERROR_EOF; + } + if (!c->can_switch) + c->can_switch = 1; + } if (!v->needed) { av_log(v->parent, AV_LOG_INFO, "No longer receiving playlist %d ('%s')\n", v->index, v->url); @@ -1530,7 +1674,12 @@ reload: } seg = next_segment(v); - if (c->http_multiple == 1 && !v->input_next_requested && + + // when get switch_request, old stream should stop downloading next seg + if (c->abr && c->switch_request2 != v->index && c->switch_timestamp != AV_NOPTS_VALUE + && ((c->cur_timestamp + seg->duration * 2) >= c->switch_timestamp) ) + ; + else if (c->http_multiple == 1 && !v->input_next_requested && seg && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { ret = open_input(c, v, seg, &v->input_next); if (ret < 0) { @@ -1563,11 +1712,17 @@ reload: return ret; } - if (c->http_persistent && + if (c->http_persistent && !c->abr && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { v->input_read_done = 1; } else { ff_format_io_close(v->parent, &v->input); + if (c->abr) { + if (c->switch_delay > 0) + c->switch_delay--; + av_log(v->parent, AV_LOG_VERBOSE, "read_data: close pls[%d]->input, v->cur_seq_no=%d, c->switch_delay= %d\n", + v->index, v->cur_seq_no, c->switch_delay); + } } v->cur_seq_no++; @@ -1860,6 +2015,17 @@ static int hls_read_header(AVFormatContext *s) c->first_packet = 1; c->first_timestamp = AV_NOPTS_VALUE; c->cur_timestamp = AV_NOPTS_VALUE; + c->switch_request2 = -1; + c->switch_delay = 0; + c->switch_timestamp = AV_NOPTS_VALUE; + c->delta_timestamp = AV_NOPTS_VALUE; + c->can_switch = -1; + + if (c->abr) { + c->http_persistent = 0; + c->http_multiple = 1; + c->throughputs = av_mallocz(sizeof(struct throughput)); + } if ((ret = save_avio_options(s)) < 0) goto fail; @@ -2047,6 +2213,9 @@ static int hls_read_header(AVFormatContext *s) add_metadata_from_renditions(s, pls, AVMEDIA_TYPE_SUBTITLE); } + if (c->abr && c->can_switch == -1) + c->can_switch = 1; + update_noheader_flag(s); return 0; @@ -2055,6 +2224,13 @@ fail: return ret; } +static void change_discard_flags(struct playlist *pls, int flag) +{ + for (int i = 0; i < pls->n_main_streams; i++) { + pls->main_streams[i]->discard = flag; + } +} + static int recheck_discard_flags(AVFormatContext *s, int first) { HLSContext *c = s->priv_data; @@ -2065,6 +2241,22 @@ static int recheck_discard_flags(AVFormatContext *s, int first) for (i = 0; i < c->n_playlists; i++) { struct playlist *pls = c->playlists[i]; + if (c->abr) { + if (c->switch_request2 == -1) + ; + else if (c->switch_request2 == i && c->switch_delay <= 0 + && c->cur_timestamp + c->delta_timestamp >= c->switch_timestamp + && c->switch_timestamp != AV_NOPTS_VALUE) { + av_log(s, AV_LOG_VERBOSE, "[switch point] cur_timestamp:%ld\n", c->cur_timestamp); + change_discard_flags(pls, AVDISCARD_DEFAULT); + c->switch_timestamp = AV_NOPTS_VALUE; + } else if (c->switch_request2 != -1 && c->switch_request2 != i && c->switch_delay <= 0 + && c->cur_timestamp + c->delta_timestamp >= c->switch_timestamp + && c->switch_timestamp != AV_NOPTS_VALUE) { + change_discard_flags(pls, AVDISCARD_ALL); + } + } + cur_needed = playlist_needed(c->playlists[i]); if (pls->broken) { @@ -2075,6 +2267,11 @@ static int recheck_discard_flags(AVFormatContext *s, int first) changed = 1; pls->cur_seq_no = select_cur_seq_no(c, pls); pls->pb.eof_reached = 0; + if (c->abr) { + pls->cur_seq_no = select_cur_seq_no(c, pls) + 1; + avio_flush(&pls->pb); + avformat_flush(pls->ctx); + } if (c->cur_timestamp != AV_NOPTS_VALUE) { /* catch up */ pls->seek_timestamp = c->cur_timestamp; @@ -2082,7 +2279,7 @@ static int recheck_discard_flags(AVFormatContext *s, int first) pls->seek_stream_index = -1; } av_log(s, AV_LOG_INFO, "Now receiving playlist %d, segment %d\n", i, pls->cur_seq_no); - } else if (first && !cur_needed && pls->needed) { + } else if ((first || c->abr) && !cur_needed && pls->needed) { ff_format_io_close(pls->parent, &pls->input); pls->input_read_done = 0; ff_format_io_close(pls->parent, &pls->input_next); @@ -2091,6 +2288,9 @@ static int recheck_discard_flags(AVFormatContext *s, int first) changed = 1; av_log(s, AV_LOG_INFO, "No longer receiving playlist %d\n", i); } + + if (c->abr && changed && cur_needed) + c->cur_pls = i; } return changed; } @@ -2165,7 +2365,13 @@ static int hls_read_packet(AVFormatContext *s, AVPacket *pkt) /* audio elementary streams are id3 timestamped */ fill_timing_for_id3_timestamped_stream(pls); } - + if (c->abr && c->first_timestamp != AV_NOPTS_VALUE && c->delta_timestamp == AV_NOPTS_VALUE && !i) { + c->delta_timestamp = av_rescale_q(pls->pkt.dts, + get_timebase(pls), AV_TIME_BASE_Q) - c->first_timestamp + 1; + av_log(s, AV_LOG_VERBOSE, "[abr] delta_timestamp=%ld, %ld, %ld\n", + c->delta_timestamp, c->first_timestamp,av_rescale_q(pls->pkt.dts, + get_timebase(pls), AV_TIME_BASE_Q)); + } if (c->first_timestamp == AV_NOPTS_VALUE && pls->pkt.dts != AV_NOPTS_VALUE) c->first_timestamp = av_rescale_q(pls->pkt.dts, @@ -2373,6 +2579,8 @@ static int hls_probe(const AVProbeData *p) #define OFFSET(x) offsetof(HLSContext, x) #define FLAGS AV_OPT_FLAG_DECODING_PARAM static const AVOption hls_options[] = { + {"abr", "enable abr to switch streams", + OFFSET(abr), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, FLAGS }, {"live_start_index", "segment index to start live streams at (negative values are from the end)", OFFSET(live_start_index), AV_OPT_TYPE_INT, {.i64 = -3}, INT_MIN, INT_MAX, FLAGS}, {"allowed_extensions", "List of file extensions that hls is allowed to access", From patchwork Sun Jul 5 11:34:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 20816 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 6C5E544BDAF for ; Sun, 5 Jul 2020 14:35:33 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 525F468B2F3; Sun, 5 Jul 2020 14:35:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7010868B213 for ; Sun, 5 Jul 2020 14:35:26 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id B2B931008CBCE; Sun, 5 Jul 2020 19:35:23 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id AB87D200B4497; Sun, 5 Jul 2020 19:35:23 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 9CQe4biLHcFU; Sun, 5 Jul 2020 19:35:23 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id C3DDC200B4496; Sun, 5 Jul 2020 19:35:22 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Jul 2020 19:34:57 +0800 Message-Id: <20200705113459.77467-4-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC 4/6] ffplay: add an option to enable abr X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add abr option, ffplay can play hls using abr by: ffplay -i http://xxx/master.m3u8 -abr Structure ABRList is added to save stream type and index, it is used to allow packet_queue_put function to put pkt which from same type(for example: video pkt) but different stream index to queue. Signed-off-by: spartazhc --- doc/ffplay.texi | 2 + fftools/ffplay.c | 145 ++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 138 insertions(+), 9 deletions(-) diff --git a/doc/ffplay.texi b/doc/ffplay.texi index f3761bb12e..6a24542cda 100644 --- a/doc/ffplay.texi +++ b/doc/ffplay.texi @@ -46,6 +46,8 @@ Disable audio. Disable video. @item -sn Disable subtitles. +@item -abr +Enable adaptive bitrate for hls/dash. @item -ss @var{pos} Seek to @var{pos}. Note that in most formats it is not possible to seek exactly, so @command{ffplay} will seek to the nearest seek point to diff --git a/fftools/ffplay.c b/fftools/ffplay.c index d673b8049a..b17b75fa8f 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -201,6 +201,15 @@ typedef struct Decoder { SDL_Thread *decoder_tid; } Decoder; +typedef struct ABRList { + int **audio_list; + int audios; + int **video_list; + int videos; + int **sub_list; + int subs; +} ABRList; + typedef struct VideoState { SDL_Thread *read_tid; AVInputFormat *iformat; @@ -305,6 +314,8 @@ typedef struct VideoState { int last_video_stream, last_audio_stream, last_subtitle_stream; SDL_cond *continue_read_thread; + + ABRList *abr_list; } VideoState; /* options specified by the user */ @@ -356,6 +367,7 @@ static char *afilters = NULL; static int autorotate = 1; static int find_stream_info = 1; static int filter_nbthreads = 0; +static int abr = 0; /* current context */ static int is_full_screen; @@ -1262,6 +1274,29 @@ static void stream_component_close(VideoState *is, int stream_index) } } +static void free_abr_dynarray(int **list, int num) +{ + for (int i = 0; i < num; i++) { + av_free(list[i]); + } +} + +static void free_abr_list(ABRList *abrlist) +{ + if (abrlist->audios) { + free_abr_dynarray(abrlist->audio_list, abrlist->audios); + av_freep(&abrlist->audio_list); + } + if (abrlist->videos) { + free_abr_dynarray(abrlist->video_list, abrlist->videos); + av_freep(&abrlist->video_list); + } + if (abrlist->subs) { + free_abr_dynarray(abrlist->sub_list, abrlist->subs); + av_freep(&abrlist->sub_list); + } +} + static void stream_close(VideoState *is) { /* XXX: use a special url_shutdown call to abort parse cleanly */ @@ -2753,6 +2788,67 @@ static int is_realtime(AVFormatContext *s) return 0; } +static av_cold int abr_init_list(VideoState *is) +{ + int stream_index, *tmp; + AVStream *st; + int nb_streams = is->ic->nb_streams; + ABRList *abrlist = is->abr_list; + + for (stream_index = 0; stream_index < nb_streams; stream_index++) { + st = is->ic->streams[stream_index]; + tmp = av_memdup(&stream_index, sizeof(int)); + if (!tmp) + return -1; + switch (st->codecpar->codec_type) { + case AVMEDIA_TYPE_AUDIO: + av_dynarray_add(&abrlist->audio_list, &abrlist->audios, tmp); + break; + case AVMEDIA_TYPE_VIDEO: + av_dynarray_add(&abrlist->video_list, &abrlist->videos, tmp); + break; + case AVMEDIA_TYPE_SUBTITLE: + av_dynarray_add(&abrlist->sub_list, &abrlist->subs, tmp); + break; + default: + av_free(tmp); + break; + } + } + return 0; +} + +static int abr_check_list(ABRList *abr_list, enum AVMediaType type, int st) +{ + int **st_list; + int n_st; + switch (type) { + case AVMEDIA_TYPE_AUDIO: + st_list = abr_list->audio_list; + n_st = abr_list->audios; + break; + case AVMEDIA_TYPE_VIDEO: + st_list = abr_list->video_list; + n_st = abr_list->videos; + break; + case AVMEDIA_TYPE_SUBTITLE: + st_list = abr_list->sub_list; + n_st = abr_list->subs; + break; + default: + break; + } + if (!st_list) + return 0; + for (int i = 0; i < n_st; i++) { + if (*st_list[i] == st) + return 1; + } + return 0; +} + + + /* this thread gets the stream from the disk or the network */ static int read_thread(void *arg) { @@ -2789,6 +2885,8 @@ static int read_thread(void *arg) av_dict_set(&format_opts, "scan_all_pmts", "1", AV_DICT_DONT_OVERWRITE); scan_all_pmts_set = 1; } + if (abr) + av_dict_set(&format_opts, "abr", "1", 0); err = avformat_open_input(&ic, is->filename, is->iformat, &format_opts); if (err < 0) { print_error(is->filename, err); @@ -2918,6 +3016,15 @@ static int read_thread(void *arg) stream_component_open(is, st_index[AVMEDIA_TYPE_SUBTITLE]); } + /* clean packet list filled in hls_read_header if abr is enabled */ + if (abr) { + is->abr_list = av_mallocz(sizeof(ABRList)); + ret = abr_init_list(is); + if (ret < 0) { + av_log(NULL, AV_LOG_WARNING, "Failed to initiate abr_list\n"); + } + } + if (is->video_stream < 0 && is->audio_stream < 0) { av_log(NULL, AV_LOG_FATAL, "Failed to open file '%s' or configure filtergraph\n", is->filename); @@ -3045,15 +3152,31 @@ static int read_thread(void *arg) av_q2d(ic->streams[pkt->stream_index]->time_base) - (double)(start_time != AV_NOPTS_VALUE ? start_time : 0) / 1000000 <= ((double)duration / 1000000); - if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { - packet_queue_put(&is->audioq, pkt); - } else if (pkt->stream_index == is->video_stream && pkt_in_play_range - && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { - packet_queue_put(&is->videoq, pkt); - } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { - packet_queue_put(&is->subtitleq, pkt); + if (abr) { + if ((pkt->stream_index == is->audio_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_AUDIO, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if ((pkt->stream_index == is->video_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_VIDEO, pkt->stream_index)) && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if ((pkt->stream_index == is->subtitle_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_SUBTITLE, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } else { - av_packet_unref(pkt); + if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if (pkt->stream_index == is->video_stream && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } } @@ -3061,7 +3184,10 @@ static int read_thread(void *arg) fail: if (ic && !is->ic) avformat_close_input(&ic); - + if (abr && is->abr_list) { + free_abr_list(is->abr_list); + av_freep(&is->abr_list); + } if (ret != 0) { SDL_Event event; @@ -3588,6 +3714,7 @@ static const OptionDef options[] = { { "an", OPT_BOOL, { &audio_disable }, "disable audio" }, { "vn", OPT_BOOL, { &video_disable }, "disable video" }, { "sn", OPT_BOOL, { &subtitle_disable }, "disable subtitling" }, + { "abr", OPT_BOOL, { &abr }, "enable adaptive bitrate for hls/dash" }, { "ast", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_AUDIO] }, "select desired audio stream", "stream_specifier" }, { "vst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_VIDEO] }, "select desired video stream", "stream_specifier" }, { "sst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_SUBTITLE] }, "select desired subtitle stream", "stream_specifier" }, From patchwork Sun Jul 5 11:34:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 20817 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 63A1844BDAF for ; Sun, 5 Jul 2020 14:35:34 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4BD7068B31C; Sun, 5 Jul 2020 14:35:34 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EA37A68B205 for ; Sun, 5 Jul 2020 14:35:27 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 3E0541008CBCF; Sun, 5 Jul 2020 19:35:25 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 37C17200B4497; Sun, 5 Jul 2020 19:35:25 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id JdzbI0CM3f-L; Sun, 5 Jul 2020 19:35:25 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 6062E200B4496; Sun, 5 Jul 2020 19:35:23 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Jul 2020 19:34:58 +0800 Message-Id: <20200705113459.77467-5-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC 5/6] avformat/utils: add av_packet_clean to remove AVPackets not needed in pktl X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc Add av_packet_clean to remove AVPackets whose stream_index is not in st_index list. Generally s->internal->packet_buffer may have pkts from different stream, and stream_index will be used to discard pkt that is not needed. But in case of abr, several streams may pass the stream_index check. So we need a function to remove AVPackets not needed in pktl added by hls_read_header. Signed-off-by: spartazhc --- libavformat/avformat.h | 9 +++++++ libavformat/internal.h | 15 +++++++++++ libavformat/utils.c | 59 ++++++++++++++++++++++++++++++++++++++++++ libavformat/version.h | 2 +- 4 files changed, 84 insertions(+), 1 deletion(-) diff --git a/libavformat/avformat.h b/libavformat/avformat.h index e91e7f1d33..90dee0d075 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -2474,6 +2474,15 @@ int avformat_seek_file(AVFormatContext *s, int stream_index, int64_t min_ts, int */ int avformat_flush(AVFormatContext *s); +/** + * Remove the AVPackets do not needed in the packet list. + * The behaviour is undefined if the packet list is empty. + * + * @param s media file handle + * @param st_index the stream_index list which is needed + */ +int av_packet_clean(AVFormatContext *s, int *st_index); + /** * Start playing a network-based stream (e.g. RTSP stream) at the * current position. diff --git a/libavformat/internal.h b/libavformat/internal.h index 17a6ab07d3..ac943b1441 100644 --- a/libavformat/internal.h +++ b/libavformat/internal.h @@ -764,6 +764,21 @@ int ff_packet_list_put(AVPacketList **head, AVPacketList **tail, int ff_packet_list_get(AVPacketList **head, AVPacketList **tail, AVPacket *pkt); +/** + * Remove the AVPackets do not needed in the packet list. + * The behaviour is undefined if the packet list is empty. + * + * @note The pkt will be overwritten completely. The caller owns the + * packet and must unref it by itself. + * + * @param head List head element + * @param tail List tail element + * @param st_index the stream_index list which is needed + * @return 0 on success. Success is guaranteed + * if the packet list is not empty. + */ +int ff_packet_list_clean(AVPacketList **head, AVPacketList **tail, + int *st_index); /** * Wipe the list and unref all the packets in it. * diff --git a/libavformat/utils.c b/libavformat/utils.c index 45a4179552..26b5a08a19 100644 --- a/libavformat/utils.c +++ b/libavformat/utils.c @@ -1565,6 +1565,65 @@ int ff_packet_list_get(AVPacketList **pkt_buffer, return 0; } +/** + * return 1 if needed + */ +static int ff_check_st_index(int st, int *st_index) +{ + for (int i = 0; i < AVMEDIA_TYPE_NB; ++i) { + if (st_index[i] == st) + return 1; + } + return 0; +} + +int ff_packet_list_clean(AVPacketList **pkt_buffer, + AVPacketList **pkt_buffer_end, + int *st_index) +{ + AVPacketList *pktl, *pktn; + av_assert0(*pkt_buffer); + pktl = *pkt_buffer; + pktn = pktl->next; + + /* num >= 3 */ + while (pktn && pktn != *pkt_buffer_end) { + if (!ff_check_st_index(pktn->pkt.stream_index, st_index)) { + av_packet_unref(&pktn->pkt); + pktl->next = pktn->next; + av_freep(pktn); + pktn = pktl->next; + } else { + pktl = pktn; + pktn = pktn->next; + } + } + /* last one*/ + if (pktn && !ff_check_st_index(pktn->pkt.stream_index, st_index)) { + av_packet_unref(&pktn->pkt); + av_freep(pktn); + pktl->next = NULL; + *pkt_buffer_end = pktl; + } + /* first one*/ + pktl = *pkt_buffer; + if (!ff_check_st_index(pktl->pkt.stream_index, st_index)) { + av_packet_unref(&pktl->pkt); + *pkt_buffer = pktl->next; + if (!pktl->next) + *pkt_buffer_end = NULL; + av_freep(&pktl); + } + + return 0; +} + +int av_packet_clean(AVFormatContext *s, int *st_index) { + int ret = ff_packet_list_clean(&s->internal->packet_buffer, + &s->internal->packet_buffer_end, st_index); + return ret; +} + static int64_t ts_to_samples(AVStream *st, int64_t ts) { return av_rescale(ts, st->time_base.num * st->codecpar->sample_rate, st->time_base.den); diff --git a/libavformat/version.h b/libavformat/version.h index 3c1957b00c..75c03fde0a 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -32,7 +32,7 @@ // Major bumping may affect Ticket5467, 5421, 5451(compatibility with Chromium) // Also please add any ticket numbers that you believe might be affected here #define LIBAVFORMAT_VERSION_MAJOR 58 -#define LIBAVFORMAT_VERSION_MINOR 47 +#define LIBAVFORMAT_VERSION_MINOR 48 #define LIBAVFORMAT_VERSION_MICRO 100 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ From patchwork Sun Jul 5 11:34:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 20818 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 462D544BDAF for ; Sun, 5 Jul 2020 14:35:37 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2F53368B310; Sun, 5 Jul 2020 14:35:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 12355687F9F for ; Sun, 5 Jul 2020 14:35:31 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id C6EDE1008CBD0; Sun, 5 Jul 2020 19:35:26 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id BE679200B4497; Sun, 5 Jul 2020 19:35:26 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id iVj1ZDfCVsCR; Sun, 5 Jul 2020 19:35:26 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id EC083200B4496; Sun, 5 Jul 2020 19:35:25 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Jul 2020 19:34:59 +0800 Message-Id: <20200705113459.77467-6-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200705113459.77467-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] [GSoC 6/6] ffplay: add av_packet_clean to remove AVPackets not needed X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: spartazhc hls_read_header will add all streams to s->internal->packet_buffer. Use av_packet_clean to remove the AVPackets from other streams that are not needed, otherwise abr will allow them to be added to ffplay's packet_queue. Signed-off-by: spartazhc --- fftools/ffplay.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index b17b75fa8f..019dc7f32e 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -3018,6 +3018,10 @@ static int read_thread(void *arg) /* clean packet list filled in hls_read_header if abr is enabled */ if (abr) { + ret = av_packet_clean(ic, st_index); + if (ret < 0) { + av_log(NULL, AV_LOG_WARNING, "Failed to clean av_packet\n"); + } is->abr_list = av_mallocz(sizeof(ABRList)); ret = abr_init_list(is); if (ret < 0) {