From patchwork Thu Jul 16 12:51:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21102 Delivered-To: andriy.gelman@gmail.com Received: by 2002:a25:80ca:0:0:0:0:0 with SMTP id c10csp1198003ybm; Thu, 16 Jul 2020 05:51:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzWgEhF0lKoqJxbZs6HS/SN0M9pTURw9acMF8gZTfj+oj4S5mLjv5kxRwEBNwyEtqYVgMmr X-Received: by 2002:adf:aace:: with SMTP id i14mr4841868wrc.236.1594903894795; Thu, 16 Jul 2020 05:51:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594903894; cv=none; d=google.com; s=arc-20160816; b=Pez/43bZOwNNO6efFfhuYah2s5o1rULCHGKd0yC9RCI2OWN6XrhVELbCW8Kr5K/IRK Xp7xp9KFSzLPW3KS1JsvsjF0VYd5gZNWQ9o32GcNh20JX/xz6mmhQp5vln46bcXo3w0S PKm2ydalJibBOiwemxiABtXW5QF8C+VpPBZAWfBIMxDgoSMzMr5l+4F/mCFUX+onsRD4 T8x6OOuwY6NcoK88fLlMc7BddZXUu5M0eA7AnUwGgtazKSrcU63u5r4d8sI88XXCdcQE 2VrZa7xG+bCvt3lFLhgTucpD1Vg8/EznadGvEKigK4FAH1d+9hAij0HV1PH80gw+9e1W LFkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=Zeuta0/rwpAfB4IMvjXHlwaQeTPnQYJY3+AKhO7h7Cw=; b=BS+G4p4BSlmBLsoOaQRjCsnDwIVRIl6av5cxJt/a89Fml6DlDYG9xexZ57LP+ByvPo JyAUfPHksNV3rdMwbQsCqIcpPWi7sasMcwMKtPtWuelvY8Ac1VaWwti/pk7L+rTD7sVY 6KaSACIwfBCW7kQzJPujrAdNB1CyocDDduwHIBstWLAFu1tVPoY0tIQrQkQGe6Ao3fty IKxMRqZ6YroH4d90pcoxbyARuDRubj8FurTPXkbPS7d7uLznp5tX6aaSp/7bWg8TBlql fKRbHDRob6YmYaSeel2VrZKpKDhjJUC6kwL7xmAOqnyYtjt5i7QE5jMUQk5Jc647azT3 O2AQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y6si4382487wrn.519.2020.07.16.05.51.33; Thu, 16 Jul 2020 05:51:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E14B268B5AD; Thu, 16 Jul 2020 15:51:28 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DC14B68B4FF for ; Thu, 16 Jul 2020 15:51:21 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 263101008CBD0; Thu, 16 Jul 2020 20:51:18 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 6F57D200B4498; Thu, 16 Jul 2020 20:51:17 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id MaXfaRKKKLjf; Thu, 16 Jul 2020 20:51:17 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 96838200B449C; Thu, 16 Jul 2020 20:51:15 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Jul 2020 20:51:11 +0800 Message-Id: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v2 1/6] avformat/abr: Adaptive Bitrate support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: O6o+JAaRDUBC Content-Length: 14083 From: spartazhc Add abr module for hls/dash. v1 fixed: 1. add an "ff" prefix to the protocol name to mark it internal. 2. use 1.2f for float constant 1.2. 3. simplify abr_seek for we just need AVSEEK_SIZE only. v2 fixed: 1. fix error return 2. simplify abr_seek Signed-off-by: spartazhc --- doc/protocols.texi | 7 ++ libavformat/Makefile | 1 + libavformat/abr.c | 249 ++++++++++++++++++++++++++++++++++++++++ libavformat/protocols.c | 1 + 4 files changed, 258 insertions(+) create mode 100644 libavformat/abr.c diff --git a/doc/protocols.texi b/doc/protocols.texi index 64ad3f05d6..ffbb36147e 100644 --- a/doc/protocols.texi +++ b/doc/protocols.texi @@ -51,6 +51,13 @@ in microseconds. A description of the currently available protocols follows. +@section abr + +Adaptive bitrate sub-protocol work for hls/dash. + +The abr protocol takes stream information from hls/dash as input, +use bandwidth estimation to decide whether to switch or not. + @section amqp Advanced Message Queueing Protocol (AMQP) version 0-9-1 is a broker based diff --git a/libavformat/Makefile b/libavformat/Makefile index 26af859a28..7d74e45d2a 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -595,6 +595,7 @@ OBJS-$(CONFIG_CACHE_PROTOCOL) += cache.o OBJS-$(CONFIG_CONCAT_PROTOCOL) += concat.o OBJS-$(CONFIG_CRYPTO_PROTOCOL) += crypto.o OBJS-$(CONFIG_DATA_PROTOCOL) += data_uri.o +OBJS-$(CONFIG_FFABR_PROTOCOL) += abr.o OBJS-$(CONFIG_FFRTMPCRYPT_PROTOCOL) += rtmpcrypt.o rtmpdigest.o rtmpdh.o OBJS-$(CONFIG_FFRTMPHTTP_PROTOCOL) += rtmphttp.o OBJS-$(CONFIG_FILE_PROTOCOL) += file.o diff --git a/libavformat/abr.c b/libavformat/abr.c new file mode 100644 index 0000000000..7699b9baef --- /dev/null +++ b/libavformat/abr.c @@ -0,0 +1,249 @@ +/* + * Adaptive Bitrate Module for HLS / DASH + * Copyright (c) 2020 Hongcheng Zhong + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "avformat.h" +#include "libavutil/opt.h" +#include "libavutil/time.h" +#include "libavutil/avstring.h" +#include "url.h" +#include + +enum ABRFormatType { + ABR_TYPE_HLS, + ABR_TYPE_DASH +}; + +typedef struct variant_bitrate { + int value; + int index; +} variant_bitrate; + +typedef struct ABRContext { + AVClass *class; + URLContext *hd; + AVDictionary *abr_params; + AVDictionary *abr_metadata; + enum ABRFormatType format; + int cur_pls; + int can_switch; + int n_variants; + variant_bitrate *variants_bitrate; + int index; + int n_throughputs; + float *throughputs; +} ABRContext; + +static float harmonic_mean(int num, float* arr) +{ + float tmp = 0; + + if (num <= 0) return 0; + + for (size_t i = 0; i < num; i++) { + tmp += 1 / arr[i]; + } + + return num / tmp; +} + +static int hls_param_parse(ABRContext *c, const char *key, const char *value) +{ + if (!av_strcasecmp(key, "cur_pls")) { + c->cur_pls = atoi(value); + } else if (!av_strcasecmp(key, "can_switch")) { + c->can_switch = atoi(value); + } else if (!av_strcasecmp(key, "n_variants")) { + c->n_variants = atoi(value); + c->variants_bitrate = av_mallocz(sizeof(variant_bitrate) * c->n_variants); + if (!c->variants_bitrate) + return AVERROR(ENOMEM); + } else if (av_strstart(key, "variant_bitrate", NULL)) { + c->variants_bitrate[c->index].value = atoi(value); + c->variants_bitrate[c->index].index = c->index; + c->index++; + } else if (!av_strcasecmp(key, "n_throughputs")) { + c->n_throughputs = atoi(value); + c->index = 0; + if (c->n_throughputs > 0) { + c->throughputs = av_malloc(sizeof(float) * c->n_throughputs); + if (!c->throughputs) + return AVERROR(ENOMEM); + } + } else if (av_strstart(key, "throughputs", NULL)) + c->throughputs[c->index++] = atof(value); + return 0; +} + +static int dash_param_parse(ABRContext *c, const char *key, const char *value) +{ + return 0; +} + +static int abr_param_parse(ABRContext *c, enum ABRFormatType type, const char *key, const char *value) +{ + if (type == ABR_TYPE_HLS) { + hls_param_parse(c, key, value); + } else if (type == ABR_TYPE_DASH) { + dash_param_parse(c, key, value); + } + return 0; +} + +static int compare_vb(const void *a, const void *b) +{ + variant_bitrate *a1 = (variant_bitrate *)a; + variant_bitrate *a2 = (variant_bitrate *)b; + return (*a2).value - (*a1).value; +} + +static int abr_rule(ABRContext *c, float bw_estimate) +{ + int ret = -1; + + if (c->n_throughputs > 6) { + if (bw_estimate < c->variants_bitrate[c->cur_pls].value / 1000 * 1.2f && + bw_estimate > c->variants_bitrate[c->cur_pls].value / 1000 * 0.8f) + return -1; + qsort(c->variants_bitrate, c->n_variants, sizeof(variant_bitrate), compare_vb); + for (int i = 0; i < c->n_variants; i++) { + if (bw_estimate > c->variants_bitrate[i].value / 1000) { + ret = i; + break; + } + } + } + if (ret == c->cur_pls) + ret = -1; + av_log(c, AV_LOG_VERBOSE, "[switch] bwe=%.2fkbps, cur=%d, switch=%d\n", bw_estimate, c->cur_pls, ret); + return ret; +} + +static int abr_open(URLContext *h, const char *uri, int flags, AVDictionary **options) +{ + const char *nested_url; + uint64_t start, end; + float bw_estimation; + int switch_request = -1; + int ret = 0; + ABRContext *c = h->priv_data; + + if (!av_strstart(uri, "ffabr+", &nested_url) && + !av_strstart(uri, "ffabr:", &nested_url)) { + av_log(h, AV_LOG_ERROR, "Unsupported url %s\n", uri); + ret = AVERROR(EINVAL); + goto err; + } + + AVDictionaryEntry *en = NULL; + en = av_dict_get(c->abr_params, "", en, AV_DICT_IGNORE_SUFFIX); + if (!en) + return -1; + if (!av_strcasecmp(en->key, "format")) { + if (!av_strcasecmp(en->value, "hls")) { + c->format = ABR_TYPE_HLS; + } else if (!av_strcasecmp(en->value, "dash")) { + c->format = ABR_TYPE_DASH; + } + av_log(h, AV_LOG_VERBOSE, "%s is using ABR\n", en->value); + } + + while (en = av_dict_get(c->abr_params, "", en, AV_DICT_IGNORE_SUFFIX)) { + if (abr_param_parse(c, c->format, en->key, en->value) < 0) + av_log(h, AV_LOG_WARNING,"Error parsing option '%s = %s'.\n", + en->key, en->value); + } + + start = av_gettime(); + if ((ret = ffurl_open_whitelist(&c->hd, nested_url, flags, + &h->interrupt_callback, options, + h->protocol_whitelist, h->protocol_blacklist, h)) < 0) { + av_log(h, AV_LOG_ERROR, "Unable to open resource: %s\n", nested_url); + goto err; + } + end = av_gettime(); + + bw_estimation = harmonic_mean(c->n_throughputs, c->throughputs); + + if (c->can_switch == 1) + switch_request = abr_rule(c, bw_estimation); + + av_dict_set_int(&c->abr_metadata, "download_time", (end - start), 0); + av_dict_set_int(&c->abr_metadata, "switch_request", switch_request, 0); + +err: + return ret; +} + + +static int abr_read(URLContext *h, uint8_t *buf, int size) +{ + ABRContext *c = h->priv_data; + + return ffurl_read(c->hd, buf, size); +} + +static int64_t abr_seek(URLContext *h, int64_t pos, int whence) +{ + ABRContext *c = h->priv_data; + int64_t ret = 0; + + if (whence == AVSEEK_SIZE) { + ret = ffurl_seek(c->hd, pos, AVSEEK_SIZE); + } + + return ret; +} + +static int abr_close(URLContext *h) +{ + ABRContext *c = h->priv_data; + int ret = 0; + + ffurl_closep(&c->hd); + av_free(c->variants_bitrate); + av_free(c->throughputs); + return ret; +} + +#define OFFSET(x) offsetof(ABRContext, x) +#define D AV_OPT_FLAG_DECODING_PARAM +static const AVOption ffabr_options[] = { + { "abr-params", "Informations ABR needed, using a :-separated list of key=value parameters", OFFSET(abr_params), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { "abr-metadata", "Metadata return from abr, including switch signal and network bandwidth", OFFSET(abr_metadata), AV_OPT_TYPE_DICT, { 0 }, 0, 0, D }, + { NULL } +}; + +static const AVClass ffabr_class = { + .class_name = "ffabr", + .item_name = av_default_item_name, + .option = ffabr_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +const URLProtocol ff_ffabr_protocol = { + .name = "ffabr", + .url_open2 = abr_open, + .url_read = abr_read, + .url_seek = abr_seek, + .url_close = abr_close, + .priv_data_size = sizeof(ABRContext), + .priv_data_class = &ffabr_class, +}; diff --git a/libavformat/protocols.c b/libavformat/protocols.c index 7df18fbb3b..1d6af8e380 100644 --- a/libavformat/protocols.c +++ b/libavformat/protocols.c @@ -29,6 +29,7 @@ extern const URLProtocol ff_cache_protocol; extern const URLProtocol ff_concat_protocol; extern const URLProtocol ff_crypto_protocol; extern const URLProtocol ff_data_protocol; +extern const URLProtocol ff_ffabr_protocol; extern const URLProtocol ff_ffrtmpcrypt_protocol; extern const URLProtocol ff_ffrtmphttp_protocol; extern const URLProtocol ff_file_protocol; From patchwork Thu Jul 16 12:51:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21112 Delivered-To: andriy.gelman@gmail.com Received: by 2002:a25:80ca:0:0:0:0:0 with SMTP id c10csp1198066ybm; Thu, 16 Jul 2020 05:51:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyC3Y/J5S/GFilY+6CRa9wsBVbYFSIzRIi5WtvVeKzAroiY3qaxCayqntGTcOs6aCLHBxTS X-Received: by 2002:a1c:2e17:: with SMTP id u23mr4168923wmu.142.1594903901765; Thu, 16 Jul 2020 05:51:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594903901; cv=none; d=google.com; s=arc-20160816; b=y629dMSzQlJztRJkhMSg6Tj3DWtrriHlWgFsxiDYbQ/Q+HBFh899vdhbFgUFWsSHjy o5ByJnMdt2ebviswhGOX/lxcE4fRyVwNtnPgiG5pcMC3tvtum1d8dyGhrGCmhDXxhn8i ixi3SrhfeWnyZ/8TnTb7zyy2U62glRh+B/pbQSqX7Mo4vPoj9G/ix9B+1bB4JOGYT2tS WqEL1yAZo0Ab/msY0pgbgtkE5XZWHqwUN8z6E8lh+9q6p9o7GMHnjqAV2SZCQB6Y8oEg QQHbXQ0t5Ogm83FTN1D1nFlWNi3LEgTN1nffIaxkfp2Dh2KWYXUf1+ky6TB5icYwhkmv mQ/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=9+bgPaQhQOEOGzSm1Kbkpc0exJomn4M6BsGzrxPVyHU=; b=q1vJ/lh4HpXAEroo9N8ezp+6ndFwbq9PYoCBTEzFhtoSlpt6E5TOfLCvZo0lNXw3o5 5caXwcj1zIMPVhft+mzef/kIg72nqf44sQJL99JmdY0Phu0wTc7ZgNVIOSE5nx+WhPH4 9QzfaQ/jpqUfPNurAUZm2I+ktSKO7m7T3w9CXVwEp8jr/v9X62SgffO0e48V7oVehZbb SN4A6xnU0XslA+jrlWk+B4IdqwQkeMhVzp3/nbrRHnbEJKPdHSukB2X79Nb80qXUkCle VdBJ47hKmh4flrVbXnB6svAOIEX7RFc61Dz+5CGaNcCf6gwB3YTjvsElxHW+uahy7P+7 pXfA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z14si4239823wml.78.2020.07.16.05.51.41; Thu, 16 Jul 2020 05:51:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 936AF68B5B2; Thu, 16 Jul 2020 15:51:29 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 434DD68B51A for ; Thu, 16 Jul 2020 15:51:22 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id A039C1008CBD2; Thu, 16 Jul 2020 20:51:19 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id EC022200B448D; Thu, 16 Jul 2020 20:51:18 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id LkTK1QBA-83F; Thu, 16 Jul 2020 20:51:18 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 224D7200B4497; Thu, 16 Jul 2020 20:51:17 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Jul 2020 20:51:12 +0800 Message-Id: <20200716125116.187031-2-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v2 2/6] avformat/http: Add abr to whitelist X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ykQGq5RljC9y Content-Length: 1382 From: spartazhc add abr protocol to http's whitelist Signed-off-by: spartazhc --- libavformat/http.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavformat/http.c b/libavformat/http.c index 6c39da1a8b..b77517ff51 100644 --- a/libavformat/http.c +++ b/libavformat/http.c @@ -1815,7 +1815,7 @@ const URLProtocol ff_http_protocol = { .priv_data_size = sizeof(HTTPContext), .priv_data_class = &http_context_class, .flags = URL_PROTOCOL_FLAG_NETWORK, - .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data" + .default_whitelist = "http,https,tls,rtp,tcp,udp,crypto,httpproxy,data,ffabr" }; #endif /* CONFIG_HTTP_PROTOCOL */ From patchwork Thu Jul 16 12:51:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21106 Delivered-To: andriy.gelman@gmail.com Received: by 2002:a25:80ca:0:0:0:0:0 with SMTP id c10csp1198170ybm; Thu, 16 Jul 2020 05:51:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxfKVIGC9GFqGU8lyY4D6q+RP8wfbsx2/uw3k2hVcvB8Tj97gqs3tQQhPTDtd2wlr7mpWq4 X-Received: by 2002:adf:efcc:: with SMTP id i12mr4871161wrp.349.1594903915196; Thu, 16 Jul 2020 05:51:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594903915; cv=none; d=google.com; s=arc-20160816; b=j5liUxxhUfbHOcmjebXOFBYbbMB8lt1DtFaDPUqlQDrOvYcAuWhMMjpqe2dcBehck0 Yfdw2PjJ68EeZJJb5R+sVYPoofM42SkoX60qBsantG4o4wrmdfYKz7d9a4G1mrQrYePo 73HVBh0InPgbB7Uq6THp/KcKztn4xpVXL7oCDY94OU2LngK0mcIfrSoX6SRw64zV+ENx 7lc6brRmukmfYHu2FMBNIt+EH/pxMpLyv0/8ddRQOGVmclrHoA3V0umCtucNkQ2zpZvT nDLlgLdw30C1UYyNdcm252twArPeg2bp2Vw0HB9etGHb1AZfVaRJ0379Q/JoRJ9HzQGI nIAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=An/VEXEEVP5jCOj3tzCWvBnkvSKyAC0mlnz+qP3LvM8=; b=pjC3X7rwsGvjd0QbckpxzQdPDvQ7kganC3CRhQfamMXR23V6S0FYrgpt1+PsIsWERw TYuMsNgyOj4YBZtnbOC8Q4fsdHcD3Nmhu83oc4M3xv/G9ppdAxrbJvsuwLfCX3JTWcPi MhxAdrsNb4V/scbPqAZVSwsX4Rud5iFLkEDDZcvoxoRr/VPrarQcBu8MJGnN4InBsj2R NYOtFzydPcqisBE5xbqj7Hmp5U30K6kBJrqMRDrXtB2JJQ+9fb3sJE4f6i/IM/uP8KFE 42kKteGoLsbG/p0xi/xID1GET6voVbmsqRHN6KbPyANqC/A9uxzxBofdsmXh+pZwSPLP aKpg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c81si4207845wmd.132.2020.07.16.05.51.54; Thu, 16 Jul 2020 05:51:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 023DB68B575; Thu, 16 Jul 2020 15:51:31 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F33AC68B5AF for ; Thu, 16 Jul 2020 15:51:23 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 36F351008CBD3; Thu, 16 Jul 2020 20:51:21 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 77D2E200B448D; Thu, 16 Jul 2020 20:51:20 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id mipA2Rp7fTgD; Thu, 16 Jul 2020 20:51:20 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 9C1F2200B4496; Thu, 16 Jul 2020 20:51:18 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Jul 2020 20:51:13 +0800 Message-Id: <20200716125116.187031-3-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v2 3/6] avformat/hls: use abr to switch streams X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6IGlKxvEn2in Content-Length: 23943 From: spartazhc When abr is enable, it will take over the task to call http to download segments, and will return a switch-request for hls to switch streams. For reason not to waste segments that have been downloaded, switch will become effective after old segments is used out. Abr cannot work with http_persistent option, and currently use http_multiple. v1 fixed: 1. fix memory leak v2 fixed: 1. check malloc result and return AVERROR(ENOMEM) 2. define ABR_THROUGHPUT_FIFO_LEN instead of hardcode Signed-off-by: spartazhc --- doc/demuxers.texi | 3 + libavformat/hls.c | 237 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 234 insertions(+), 6 deletions(-) diff --git a/doc/demuxers.texi b/doc/demuxers.texi index 3c15ab9eee..4cdbd95962 100644 --- a/doc/demuxers.texi +++ b/doc/demuxers.texi @@ -321,6 +321,9 @@ available in a metadata key named "variant_bitrate". It accepts the following options: @table @option +@item abr +enable abr to switch streams. + @item live_start_index segment index to start live streams at (negative values are from the end). diff --git a/libavformat/hls.c b/libavformat/hls.c index ba17c4ed96..0b297303d8 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -47,6 +47,7 @@ #define MPEG_TIME_BASE 90000 #define MPEG_TIME_BASE_Q (AVRational){1, MPEG_TIME_BASE} +#define ABR_THROUGHPUT_FIFO_LEN 20 /* * An apple http stream consists of a playlist with media segment files, * played sequentially. There may be several playlists with the same @@ -189,6 +190,15 @@ struct variant { char subtitles_group[MAX_FIELD_LEN]; }; +struct throughput { + int n_throughputs; + + /* throughputs are in kbps */ + float throughput_fifo[ABR_THROUGHPUT_FIFO_LEN]; + int head; + int tail; +}; + typedef struct HLSContext { AVClass *class; AVFormatContext *ctx; @@ -213,8 +223,36 @@ typedef struct HLSContext { int http_multiple; int http_seekable; AVIOContext *playlist_pb; + + int abr; + struct throughput *throughputs; + int can_switch; + int switch_request2; + int switch_delay; + int64_t switch_timestamp; + int64_t delta_timestamp; + int cur_pls; } HLSContext; +static struct segment *next_segment(struct playlist *pls); +static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in); + +static void sync_cur_seq(HLSContext *c) { + int i; + for (i = 0; i < c->n_playlists; i++) { + struct playlist *pls = c->playlists[i]; + pls->cur_seq_no = c->cur_seq_no; + } +} + +static struct segment *next2_segment(struct playlist *pls) +{ + int n = pls->cur_seq_no - pls->start_seq_no + 2; + if (n >= pls->n_segments) + return NULL; + return pls->segments[n]; +} + static void free_segment_dynarray(struct segment **segments, int n_segments) { int i; @@ -624,6 +662,31 @@ static int open_url_keepalive(AVFormatContext *s, AVIOContext **pb, #endif } +static int update_throughputs(struct throughput *thr, float time, int pb_size) +{ + if (pb_size <= 0 || time <= 0) + return AVERROR(EINVAL); + if (thr->n_throughputs < ABR_THROUGHPUT_FIFO_LEN) { + ++thr->n_throughputs; + } else { + ++thr->head; + } + thr->throughput_fifo[thr->tail] = (float)(pb_size) / time; + thr->tail = (thr->tail + 1) % ABR_THROUGHPUT_FIFO_LEN; + return 0; +} + +static int64_t get_switch_timestamp(HLSContext *c, struct playlist *pls) +{ + int64_t pos = c->first_timestamp == AV_NOPTS_VALUE ? + 0 : c->first_timestamp; + + for (int i = 0; i < pls->cur_seq_no + 2; i++) { + pos += pls->segments[i]->duration; + } + return pos; +} + static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, AVDictionary *opts, AVDictionary *opts2, int *is_http_out) { @@ -639,6 +702,9 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, } else if (av_strstart(url, "data", NULL)) { if (url[4] == '+' || url[4] == ':') proto_name = avio_find_protocol_name(url + 5); + } else if (av_strstart(url, "ffabr", NULL)) { + if (url[5] == '+' || url[5] == ':') + proto_name = avio_find_protocol_name(url + 6); } if (!proto_name) @@ -669,6 +735,8 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, ; else if (av_strstart(url, "data", NULL) && !strncmp(proto_name, url + 5, strlen(proto_name)) && url[5 + strlen(proto_name)] == ':') ; + else if (av_strstart(url, "ffabr", NULL) && !strncmp(proto_name, url + 6, strlen(proto_name)) && url[6 + strlen(proto_name)] == ':') + ; else if (strcmp(proto_name, "file") || !strncmp(url, "file,", 5)) return AVERROR_INVALIDDATA; @@ -690,6 +758,43 @@ static int open_url(AVFormatContext *s, AVIOContext **pb, const char *url, } else { ret = s->io_open(s, pb, url, AVIO_FLAG_READ, &tmp); } + if (c->abr && ret >= 0) { + AVDictionary *abr_ret = NULL; + AVDictionaryEntry *en = NULL; + struct segment *seg; + int pb_size, switch_request; + av_opt_get_dict_val(*pb, "abr-metadata", AV_OPT_SEARCH_CHILDREN, &abr_ret); + if (abr_ret) { + en = av_dict_get(abr_ret, "download_time", NULL, 0); + if (en) { + pb_size = avio_size(*pb); + update_throughputs(c->throughputs, atoi(en->value) / 1000.0, pb_size); + av_log(s, AV_LOG_VERBOSE, "[abr] time=%.2fms, size=%.2fkb\n", atoi(en->value) / 1000.0, pb_size / 1000.0); + } + en = av_dict_get(abr_ret, "switch_request", NULL, 0); + if (en) { + switch_request = atoi(en->value); + av_log(s, AV_LOG_VERBOSE, "[abr] switch request: %s\n", en->value); + } + if (switch_request != -1) { + c->switch_request2 = switch_request; + c->switch_delay = 2; + c->can_switch = 0; + sync_cur_seq(c); + seg = next2_segment(c->playlists[c->switch_request2]); + if (!seg) { + c->switch_request2 = -1; + av_log(s, AV_LOG_INFO, "[abr] no more segment need to switch\n"); + } else { + c->switch_timestamp = get_switch_timestamp(c, c->playlists[c->switch_request2]); + av_log(s, AV_LOG_VERBOSE, "[abr] switch timestamp: %ld\n", c->switch_timestamp); + c->playlists[c->switch_request2]->input_next_requested = 1; + ret = open_input(c, c->playlists[c->switch_request2], seg, &c->playlists[c->switch_request2]->input_next); + } + } + av_dict_free(&abr_ret); + } + } if (ret >= 0) { // update cookies on http response with setcookies. char *new_cookies = NULL; @@ -1219,6 +1324,35 @@ static void intercept_id3(struct playlist *pls, uint8_t *buf, pls->is_id3_timestamped = (pls->id3_mpegts_timestamp != AV_NOPTS_VALUE); } +static int abrinfo_to_dict(HLSContext *c, char **abr_info) +{ + struct throughput *thr = c->throughputs; + char buffer[MAX_URL_SIZE]; + int size, i; + size = snprintf(buffer, sizeof(buffer), "format=hls:"); + size += snprintf(buffer + size, sizeof(buffer) - size, "cur_pls=%d:", c->cur_pls); + size += snprintf(buffer + size, sizeof(buffer) - size, "can_switch=%d:", c->can_switch); + size += snprintf(buffer + size, sizeof(buffer) - size, "n_variants=%d:", c->n_variants); + for (i = 0; i < c->n_variants; i++) { + struct variant *v = c->variants[i]; + size += snprintf(buffer + size, sizeof(buffer) - size, "variant_bitrate%d=%d:", i, v->bandwidth); + } + size += snprintf(buffer + size, sizeof(buffer) - size, "n_throughputs=%d:", thr->n_throughputs); + if (thr->n_throughputs > 0) { + i = thr->head; + do { + size += snprintf(buffer + size, sizeof(buffer) - size, "throughputs%d=%.2f:", i, thr->throughput_fifo[i]); + i = (i + 1) % ABR_THROUGHPUT_FIFO_LEN; + } while (i != thr->tail); + } + *abr_info = av_malloc(size); + if (!abr_info) + return AVERROR(ENOMEM); + snprintf(*abr_info, size, "%s", buffer); + av_log(c, AV_LOG_DEBUG, "[abr] abr_info: %s\n", *abr_info); + return 0; +} + static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, AVIOContext **in) { AVDictionary *opts = NULL; @@ -1238,8 +1372,20 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, av_log(pls->parent, AV_LOG_VERBOSE, "HLS request for url '%s', offset %"PRId64", playlist %d\n", seg->url, seg->url_offset, pls->index); + if (c->abr) { + char *abr_opts; + abrinfo_to_dict(c, &abr_opts); + av_dict_set(&opts, "abr-params", abr_opts, 0); // how to setup flag? + av_free(abr_opts); + } if (seg->key_type == KEY_NONE) { - ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + char url_abr[MAX_URL_SIZE]; + if (c->abr) { + snprintf(url_abr, sizeof(url_abr), "ffabr:%s", seg->url); // + or : tbd + ret = open_url(pls->parent, in, url_abr, c->avio_opts, opts, &is_http); + } else { + ret = open_url(pls->parent, in, seg->url, c->avio_opts, opts, &is_http); + } } else if (seg->key_type == KEY_AES_128) { char iv[33], key[33], url[MAX_URL_SIZE]; if (strcmp(seg->key, pls->key_url)) { @@ -1273,6 +1419,7 @@ static int open_input(HLSContext *c, struct playlist *pls, struct segment *seg, goto cleanup; } ret = 0; + // TODO: Add abr prefix for crypto } else if (seg->key_type == KEY_SAMPLE_AES) { av_log(pls->parent, AV_LOG_ERROR, "SAMPLE-AES encryption is not supported yet\n"); @@ -1435,7 +1582,17 @@ restart: /* Check that the playlist is still needed before opening a new * segment. */ v->needed = playlist_needed(v); - + if (c->abr) { + if (c->switch_request2 == -1) + ; + else if (v->needed && c->switch_request2 != v->index && c->switch_delay <= 0) { + av_log(v->parent, AV_LOG_VERBOSE, "read_data: needed but not download playlist %d ('%s')\n", + v->index, v->url); + return AVERROR_EOF; + } + if (!c->can_switch) + c->can_switch = 1; + } if (!v->needed) { av_log(v->parent, AV_LOG_INFO, "No longer receiving playlist %d ('%s')\n", v->index, v->url); @@ -1530,7 +1687,12 @@ reload: } seg = next_segment(v); - if (c->http_multiple == 1 && !v->input_next_requested && + + // when get switch_request, old stream should stop downloading next seg + if (c->abr && c->switch_request2 != v->index && c->switch_timestamp != AV_NOPTS_VALUE + && ((c->cur_timestamp + seg->duration * 2) >= c->switch_timestamp) ) + ; + else if (c->http_multiple && !v->input_next_requested && seg && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { ret = open_input(c, v, seg, &v->input_next); if (ret < 0) { @@ -1563,11 +1725,17 @@ reload: return ret; } - if (c->http_persistent && + if (c->http_persistent && !c->abr && seg->key_type == KEY_NONE && av_strstart(seg->url, "http", NULL)) { v->input_read_done = 1; } else { ff_format_io_close(v->parent, &v->input); + if (c->abr) { + if (c->switch_delay > 0) + c->switch_delay--; + av_log(v->parent, AV_LOG_VERBOSE, "read_data: close pls[%d]->input, v->cur_seq_no=%d, c->switch_delay= %d\n", + v->index, v->cur_seq_no, c->switch_delay); + } } v->cur_seq_no++; @@ -1860,6 +2028,21 @@ static int hls_read_header(AVFormatContext *s) c->first_packet = 1; c->first_timestamp = AV_NOPTS_VALUE; c->cur_timestamp = AV_NOPTS_VALUE; + c->switch_request2 = -1; + c->switch_delay = 0; + c->switch_timestamp = AV_NOPTS_VALUE; + c->delta_timestamp = AV_NOPTS_VALUE; + c->can_switch = -1; + + if (c->abr) { + c->http_persistent = 0; + c->http_multiple = 1; + c->throughputs = av_mallocz(sizeof(struct throughput)); + if (!c->throughputs) { + ret = AVERROR(ENOMEM); + goto fail; + } + } if ((ret = save_avio_options(s)) < 0) goto fail; @@ -2049,6 +2232,9 @@ static int hls_read_header(AVFormatContext *s) add_metadata_from_renditions(s, pls, AVMEDIA_TYPE_SUBTITLE); } + if (c->abr && c->can_switch == -1) + c->can_switch = 1; + update_noheader_flag(s); return 0; @@ -2057,6 +2243,13 @@ fail: return ret; } +static void change_discard_flags(struct playlist *pls, int flag) +{ + for (int i = 0; i < pls->n_main_streams; i++) { + pls->main_streams[i]->discard = flag; + } +} + static int recheck_discard_flags(AVFormatContext *s, int first) { HLSContext *c = s->priv_data; @@ -2067,6 +2260,22 @@ static int recheck_discard_flags(AVFormatContext *s, int first) for (i = 0; i < c->n_playlists; i++) { struct playlist *pls = c->playlists[i]; + if (c->abr) { + if (c->switch_request2 == -1) + ; + else if (c->switch_request2 == i && c->switch_delay <= 0 + && c->cur_timestamp + c->delta_timestamp >= c->switch_timestamp + && c->switch_timestamp != AV_NOPTS_VALUE) { + av_log(s, AV_LOG_VERBOSE, "[switch point] cur_timestamp:%ld\n", c->cur_timestamp); + change_discard_flags(pls, AVDISCARD_DEFAULT); + c->switch_timestamp = AV_NOPTS_VALUE; + } else if (c->switch_request2 != -1 && c->switch_request2 != i && c->switch_delay <= 0 + && c->cur_timestamp + c->delta_timestamp >= c->switch_timestamp + && c->switch_timestamp != AV_NOPTS_VALUE) { + change_discard_flags(pls, AVDISCARD_ALL); + } + } + cur_needed = playlist_needed(c->playlists[i]); if (pls->broken) { @@ -2077,6 +2286,11 @@ static int recheck_discard_flags(AVFormatContext *s, int first) changed = 1; pls->cur_seq_no = select_cur_seq_no(c, pls); pls->pb.eof_reached = 0; + if (c->abr) { + pls->cur_seq_no = select_cur_seq_no(c, pls) + 1; + avio_flush(&pls->pb); + avformat_flush(pls->ctx); + } if (c->cur_timestamp != AV_NOPTS_VALUE) { /* catch up */ pls->seek_timestamp = c->cur_timestamp; @@ -2084,7 +2298,7 @@ static int recheck_discard_flags(AVFormatContext *s, int first) pls->seek_stream_index = -1; } av_log(s, AV_LOG_INFO, "Now receiving playlist %d, segment %d\n", i, pls->cur_seq_no); - } else if (first && !cur_needed && pls->needed) { + } else if ((first || c->abr) && !cur_needed && pls->needed) { ff_format_io_close(pls->parent, &pls->input); pls->input_read_done = 0; ff_format_io_close(pls->parent, &pls->input_next); @@ -2093,6 +2307,9 @@ static int recheck_discard_flags(AVFormatContext *s, int first) changed = 1; av_log(s, AV_LOG_INFO, "No longer receiving playlist %d\n", i); } + + if (c->abr && changed && cur_needed) + c->cur_pls = i; } return changed; } @@ -2167,7 +2384,13 @@ static int hls_read_packet(AVFormatContext *s, AVPacket *pkt) /* audio elementary streams are id3 timestamped */ fill_timing_for_id3_timestamped_stream(pls); } - + if (c->abr && c->first_timestamp != AV_NOPTS_VALUE && c->delta_timestamp == AV_NOPTS_VALUE && !i) { + c->delta_timestamp = av_rescale_q(pls->pkt.dts, + get_timebase(pls), AV_TIME_BASE_Q) - c->first_timestamp + 1; + av_log(s, AV_LOG_VERBOSE, "[abr] delta_timestamp=%ld, %ld, %ld\n", + c->delta_timestamp, c->first_timestamp,av_rescale_q(pls->pkt.dts, + get_timebase(pls), AV_TIME_BASE_Q)); + } if (c->first_timestamp == AV_NOPTS_VALUE && pls->pkt.dts != AV_NOPTS_VALUE) c->first_timestamp = av_rescale_q(pls->pkt.dts, @@ -2375,6 +2598,8 @@ static int hls_probe(const AVProbeData *p) #define OFFSET(x) offsetof(HLSContext, x) #define FLAGS AV_OPT_FLAG_DECODING_PARAM static const AVOption hls_options[] = { + {"abr", "enable abr to switch streams", + OFFSET(abr), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, FLAGS }, {"live_start_index", "segment index to start live streams at (negative values are from the end)", OFFSET(live_start_index), AV_OPT_TYPE_INT, {.i64 = -3}, INT_MIN, INT_MAX, FLAGS}, {"allowed_extensions", "List of file extensions that hls is allowed to access", From patchwork Thu Jul 16 12:51:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21107 Delivered-To: andriy.gelman@gmail.com Received: by 2002:a25:80ca:0:0:0:0:0 with SMTP id c10csp1198280ybm; Thu, 16 Jul 2020 05:52:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwwdeNZ+m8ASpW/GST5bw8ST/v0Zh+fOPJuKiQF+Zn/FPe1MGpyEb9n6m543/ejpZdjjqDQ X-Received: by 2002:adf:fac8:: with SMTP id a8mr4949320wrs.368.1594903927556; Thu, 16 Jul 2020 05:52:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594903927; cv=none; d=google.com; s=arc-20160816; b=jhntkbU1vTWqpsUUuBR2hRu5NzQ7C6VmwBrTMvvj1embu6QP2FIqwgKSmqeeM7d7lX I9rWZFaxPwfhV4s1/WBjF8Syb8kQt+a08QJ8MzuoNxH6Zrdg+LpW+Gbbj1YmLGJJngyW P0tJJpNsNUbYIjvqKAXF89iB85RbY+/U04rN/02MLVJT47FHkM1OeG9B4k0T1t2d4YP+ /sUV2/xZn+FrOKNQ8/p/Rpx0bXHFg/wJbPcH7EarGZVCgQ2OMwmGuF3dUNGB6o6fO29n vX/mz4Nd03f7sZ36eYMql/HK+1MUZUphEkEGbJeVhYAToYhozz7+2Db65xn9YikULNKr BdfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=3BYUCMh0I4bsg3hZluPWs0IPG0OJfz4QJ/aRG8AHFcY=; b=FhnOhhunTJJNOQCHJA5ZudHRdMpAlzScbGY6VUXbI12+ZSP7INlhr6TVG9QdDWnbx/ xQAy53XHldIW9BETCxWkHqH9LFCi5LJCmL6RQlez7YY2Zwfw4SA8UceGCd0IsvVPo6nK F6sAM7yp9I9oCmxqHD1MkHTvxftIGoAPfxGtVwAbfPgxiEAEIYGUsCFcDy7woznBooNy uSobF/MB0TJsuzJ2bVdp8qz1Cy9wLMr2QBl03nbi4TdjwLuJzWbGsjZU+oOQXXGzZZXe Fu+YMPl/ZanH/1WTyvxTGqeBCAtY4yeE/Hi67osdq0zgiEivaMrtCr1wecV+A985Xnno 37tA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c15si4519914wrp.233.2020.07.16.05.52.06; Thu, 16 Jul 2020 05:52:07 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 415A668B588; Thu, 16 Jul 2020 15:51:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7284068B575 for ; Thu, 16 Jul 2020 15:51:25 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id C6E921008CBD4; Thu, 16 Jul 2020 20:51:22 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 210E3200B448D; Thu, 16 Jul 2020 20:51:22 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id RczknSITlHUd; Thu, 16 Jul 2020 20:51:22 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 2B690200B4496; Thu, 16 Jul 2020 20:51:20 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Jul 2020 20:51:14 +0800 Message-Id: <20200716125116.187031-4-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v2 4/6] ffplay: add an option to enable abr X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: h7hiMXCMpSOI Content-Length: 12730 From: spartazhc Add abr option, ffplay can play hls using abr by: ffplay -i http://xxx/master.m3u8 -abr Structure ABRList is added to save stream type and index, it is used to allow packet_queue_put function to put pkt which from same type(for example: video pkt) but different stream index to queue. v1 fixed: None. v2 fixed: 1. check malloc result and return error message Signed-off-by: spartazhc --- doc/ffplay.texi | 2 + fftools/ffplay.c | 151 ++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 144 insertions(+), 9 deletions(-) diff --git a/doc/ffplay.texi b/doc/ffplay.texi index f3761bb12e..6a24542cda 100644 --- a/doc/ffplay.texi +++ b/doc/ffplay.texi @@ -46,6 +46,8 @@ Disable audio. Disable video. @item -sn Disable subtitles. +@item -abr +Enable adaptive bitrate for hls/dash. @item -ss @var{pos} Seek to @var{pos}. Note that in most formats it is not possible to seek exactly, so @command{ffplay} will seek to the nearest seek point to diff --git a/fftools/ffplay.c b/fftools/ffplay.c index d673b8049a..bcee507f40 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -201,6 +201,15 @@ typedef struct Decoder { SDL_Thread *decoder_tid; } Decoder; +typedef struct ABRList { + int **audio_list; + int audios; + int **video_list; + int videos; + int **sub_list; + int subs; +} ABRList; + typedef struct VideoState { SDL_Thread *read_tid; AVInputFormat *iformat; @@ -305,6 +314,8 @@ typedef struct VideoState { int last_video_stream, last_audio_stream, last_subtitle_stream; SDL_cond *continue_read_thread; + + ABRList *abr_list; } VideoState; /* options specified by the user */ @@ -356,6 +367,7 @@ static char *afilters = NULL; static int autorotate = 1; static int find_stream_info = 1; static int filter_nbthreads = 0; +static int abr = 0; /* current context */ static int is_full_screen; @@ -1262,6 +1274,29 @@ static void stream_component_close(VideoState *is, int stream_index) } } +static void free_abr_dynarray(int **list, int num) +{ + for (int i = 0; i < num; i++) { + av_free(list[i]); + } +} + +static void free_abr_list(ABRList *abrlist) +{ + if (abrlist->audios) { + free_abr_dynarray(abrlist->audio_list, abrlist->audios); + av_freep(&abrlist->audio_list); + } + if (abrlist->videos) { + free_abr_dynarray(abrlist->video_list, abrlist->videos); + av_freep(&abrlist->video_list); + } + if (abrlist->subs) { + free_abr_dynarray(abrlist->sub_list, abrlist->subs); + av_freep(&abrlist->sub_list); + } +} + static void stream_close(VideoState *is) { /* XXX: use a special url_shutdown call to abort parse cleanly */ @@ -2753,6 +2788,67 @@ static int is_realtime(AVFormatContext *s) return 0; } +static av_cold int abr_init_list(VideoState *is) +{ + int stream_index, *tmp; + AVStream *st; + int nb_streams = is->ic->nb_streams; + ABRList *abrlist = is->abr_list; + + for (stream_index = 0; stream_index < nb_streams; stream_index++) { + st = is->ic->streams[stream_index]; + tmp = av_memdup(&stream_index, sizeof(int)); + if (!tmp) + return AVERROR(ENOMEM); + switch (st->codecpar->codec_type) { + case AVMEDIA_TYPE_AUDIO: + av_dynarray_add(&abrlist->audio_list, &abrlist->audios, tmp); + break; + case AVMEDIA_TYPE_VIDEO: + av_dynarray_add(&abrlist->video_list, &abrlist->videos, tmp); + break; + case AVMEDIA_TYPE_SUBTITLE: + av_dynarray_add(&abrlist->sub_list, &abrlist->subs, tmp); + break; + default: + av_free(tmp); + break; + } + } + return 0; +} + +static int abr_check_list(ABRList *abr_list, enum AVMediaType type, int st) +{ + int **st_list; + int n_st; + switch (type) { + case AVMEDIA_TYPE_AUDIO: + st_list = abr_list->audio_list; + n_st = abr_list->audios; + break; + case AVMEDIA_TYPE_VIDEO: + st_list = abr_list->video_list; + n_st = abr_list->videos; + break; + case AVMEDIA_TYPE_SUBTITLE: + st_list = abr_list->sub_list; + n_st = abr_list->subs; + break; + default: + break; + } + if (!st_list) + return 0; + for (int i = 0; i < n_st; i++) { + if (*st_list[i] == st) + return 1; + } + return 0; +} + + + /* this thread gets the stream from the disk or the network */ static int read_thread(void *arg) { @@ -2789,6 +2885,8 @@ static int read_thread(void *arg) av_dict_set(&format_opts, "scan_all_pmts", "1", AV_DICT_DONT_OVERWRITE); scan_all_pmts_set = 1; } + if (abr) + av_dict_set(&format_opts, "abr", "1", 0); err = avformat_open_input(&ic, is->filename, is->iformat, &format_opts); if (err < 0) { print_error(is->filename, err); @@ -2918,6 +3016,21 @@ static int read_thread(void *arg) stream_component_open(is, st_index[AVMEDIA_TYPE_SUBTITLE]); } + /* clean packet list filled in hls_read_header if abr is enabled */ + if (abr) { + is->abr_list = av_mallocz(sizeof(ABRList)); + if (!is->abr_list) { + ret = AVERROR(ENOMEM); + goto fail; + } + ret = abr_init_list(is); + if (ret < 0) { + av_log(NULL, AV_LOG_FATAL, "Failed to initiate abr_list\n"); + ret = -1; + goto fail; + } + } + if (is->video_stream < 0 && is->audio_stream < 0) { av_log(NULL, AV_LOG_FATAL, "Failed to open file '%s' or configure filtergraph\n", is->filename); @@ -3045,15 +3158,31 @@ static int read_thread(void *arg) av_q2d(ic->streams[pkt->stream_index]->time_base) - (double)(start_time != AV_NOPTS_VALUE ? start_time : 0) / 1000000 <= ((double)duration / 1000000); - if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { - packet_queue_put(&is->audioq, pkt); - } else if (pkt->stream_index == is->video_stream && pkt_in_play_range - && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { - packet_queue_put(&is->videoq, pkt); - } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { - packet_queue_put(&is->subtitleq, pkt); + if (abr) { + if ((pkt->stream_index == is->audio_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_AUDIO, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if ((pkt->stream_index == is->video_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_VIDEO, pkt->stream_index)) && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if ((pkt->stream_index == is->subtitle_stream + || abr_check_list(is->abr_list, AVMEDIA_TYPE_SUBTITLE, pkt->stream_index)) && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } else { - av_packet_unref(pkt); + if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { + packet_queue_put(&is->audioq, pkt); + } else if (pkt->stream_index == is->video_stream && pkt_in_play_range + && !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) { + packet_queue_put(&is->videoq, pkt); + } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { + packet_queue_put(&is->subtitleq, pkt); + } else { + av_packet_unref(pkt); + } } } @@ -3061,7 +3190,10 @@ static int read_thread(void *arg) fail: if (ic && !is->ic) avformat_close_input(&ic); - + if (abr && is->abr_list) { + free_abr_list(is->abr_list); + av_freep(&is->abr_list); + } if (ret != 0) { SDL_Event event; @@ -3588,6 +3720,7 @@ static const OptionDef options[] = { { "an", OPT_BOOL, { &audio_disable }, "disable audio" }, { "vn", OPT_BOOL, { &video_disable }, "disable video" }, { "sn", OPT_BOOL, { &subtitle_disable }, "disable subtitling" }, + { "abr", OPT_BOOL, { &abr }, "enable adaptive bitrate for hls/dash" }, { "ast", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_AUDIO] }, "select desired audio stream", "stream_specifier" }, { "vst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_VIDEO] }, "select desired video stream", "stream_specifier" }, { "sst", OPT_STRING | HAS_ARG | OPT_EXPERT, { &wanted_stream_spec[AVMEDIA_TYPE_SUBTITLE] }, "select desired subtitle stream", "stream_specifier" }, From patchwork Thu Jul 16 12:51:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21115 Delivered-To: andriy.gelman@gmail.com Received: by 2002:a25:80ca:0:0:0:0:0 with SMTP id c10csp1198356ybm; Thu, 16 Jul 2020 05:52:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwgltQ1WHLlUeGZM0QR4b2U7G9IJTmRtjLuVH/VCY4mhOIMFeRh7+ugtrkgaVfN6u2ObAF8 X-Received: by 2002:a7b:c218:: with SMTP id x24mr4071603wmi.109.1594903940372; Thu, 16 Jul 2020 05:52:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594903940; cv=none; d=google.com; s=arc-20160816; b=dO51y4qNn40Ci1Q1nEQTwYYsC/8r4aOfKeY7CKhT1nqTqe5u5hckS5joclajKNkr8H U9SSe4e5HDFe+SFI0DflCALGczuq1UgrdbdE0BB9CpQZVHUPxJ2r/BD/i7whgjkDyH6S cfZjDE8Ec3uzYQoGZXEdlzRYY8gZSYE45EDqd1TDn4O2JsA2GkU6o5Kn1GfH1gbLs2yS xa/rFpWr6mgaf0Jnbt1PUEkDLbcrGl8dQdYb9BhnkmhnQGJ6CT27U/EEtZPy8C/6tzSj 0pYxnVWGnL9+XRPmb/Qz7kOPi+OBVzqgL/VOzJWpEteLBPbhC63BIZbDIxNEe6/0fBqY BaIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=5nmI6wLYO0ZGYF2Co6c3M51r0imqAmr6V94vhlExuac=; b=TVKVfZ7uXiv+Z0VzKO60WyF/D4gUusUcxqhjqpx0esT3Oh2YQV43QO4o84lqKai6Xo h2YzCegu7SqiUkOLiOpDXMuFjUI8Njk09QajWXYCipCd0qRoA6BqsKRUH0kFQYb+U0c+ tx+Oq6ihnwoq7UV4oq4s4qaZo5GBmFpTXB8x/O9tUkHNBavt4cTz8OiSnOyXKt8w5Bdq 19dKEWOCrD9J4QKNUWmY3h9Bk7bxvwjYI/j5nSrF9V2kwFOLkKrEr8pIIMFGGBo7YO7H dnZ83TDrbN1BLTZJjxhM2KXSt0bzPyOXIOR9gZu2yyT2KH6G3KLQDubAsofPmuBRG5PG F/ew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w8si4329159wrn.267.2020.07.16.05.52.20; Thu, 16 Jul 2020 05:52:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 25EA868B5CF; Thu, 16 Jul 2020 15:51:35 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 21C9068B5B7 for ; Thu, 16 Jul 2020 15:51:27 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 805D51008CBD6; Thu, 16 Jul 2020 20:51:24 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id CD069200B448D; Thu, 16 Jul 2020 20:51:23 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 5dge18lYTOva; Thu, 16 Jul 2020 20:51:23 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id C42EF200B4496; Thu, 16 Jul 2020 20:51:22 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Jul 2020 20:51:15 +0800 Message-Id: <20200716125116.187031-5-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v2 5/6] avformat/utils: add av_packet_buffer_filter to filter packet_buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: TiUV/hBjJxQa Content-Length: 7437 From: spartazhc Add av_packet_buffer_filter to remove AVPackets whose stream_index is not in st_index list. st_index has length of AVMEDIA_TYPE_NB, contains the stream_index of all these media types. Generally s->internal->packet_buffer may have pkts from different stream, and stream_index will be used to discard pkt that is not needed. But in case of abr, several streams may pass the stream_index check. So we need a function to remove AVPackets not needed in pktl added by hls_read_header. v1 fixed: 1. rename function name *_clean to *_filter 2. fix memory leak in ff_packet_buffer_filter 3. update the doc Signed-off-by: spartazhc --- libavformat/avformat.h | 12 ++++++++++ libavformat/internal.h | 16 +++++++++++++ libavformat/utils.c | 54 ++++++++++++++++++++++++++++++++++++++++++ libavformat/version.h | 2 +- 4 files changed, 83 insertions(+), 1 deletion(-) diff --git a/libavformat/avformat.h b/libavformat/avformat.h index e91e7f1d33..c796fd0391 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -2474,6 +2474,18 @@ int avformat_seek_file(AVFormatContext *s, int stream_index, int64_t min_ts, int */ int avformat_flush(AVFormatContext *s); +/** + * Filter the packet buffer list of the AVFormatContext, remove the AVPackets + * do not need according to st_index. + * Only filter the packet_buffer list. + * + * @param s media file handle + * @param st_index the stream_index list which is needed + * st_index has length of AVMEDIA_TYPE_NB + * in index AVMEDIA_TYPE_XXX contains the stream_index needed of type XXX + */ +int av_packet_buffer_filter(AVFormatContext *s, int *st_index); + /** * Start playing a network-based stream (e.g. RTSP stream) at the * current position. diff --git a/libavformat/internal.h b/libavformat/internal.h index 17a6ab07d3..58ebcb2e35 100644 --- a/libavformat/internal.h +++ b/libavformat/internal.h @@ -772,6 +772,22 @@ int ff_packet_list_get(AVPacketList **head, AVPacketList **tail, */ void ff_packet_list_free(AVPacketList **head, AVPacketList **tail); +/** + * Remove the AVPackets do not need in the packet buffer list. + * For each type in AVMediaType, at most keep one stream and + * the others will be removed. + * + * @param head List head element + * @param tail List tail element + * @param st_index the stream_index list which is needed + * st_index has length of AVMEDIA_TYPE_NB + * in index AVMEDIA_TYPE_XXX contains the stream_index needed of type XXX + * @return 0 on success. Success is guaranteed + * if the packet list is not empty. + */ +int ff_packet_buffer_filter(AVPacketList **head, AVPacketList **tail, + int *st_index); + void avpriv_register_devices(const AVOutputFormat * const o[], const AVInputFormat * const i[]); #endif /* AVFORMAT_INTERNAL_H */ diff --git a/libavformat/utils.c b/libavformat/utils.c index 807d9f10cb..7674e4ea3d 100644 --- a/libavformat/utils.c +++ b/libavformat/utils.c @@ -1565,6 +1565,60 @@ int ff_packet_list_get(AVPacketList **pkt_buffer, return 0; } +/** + * return 1 if needed + */ +static int ff_check_st_index(int st, int *st_index) +{ + for (int i = 0; i < AVMEDIA_TYPE_NB; ++i) { + if (st_index[i] == st) + return 1; + } + return 0; +} + +int ff_packet_buffer_filter(AVPacketList **pkt_buffer, + AVPacketList **pkt_buffer_end, + int *st_index) +{ + AVPacketList *pktl, *pktn; + av_assert0(*pkt_buffer); + pktl = *pkt_buffer; + pktn = pktl->next; + + /* num >= 2 */ + while (pktn) { + if (!ff_check_st_index(pktn->pkt.stream_index, st_index)) { + av_packet_unref(&pktn->pkt); + pktl->next = pktn->next; + av_freep(&pktn); + pktn = pktl->next; + } else { + pktl = pktn; + pktn = pktn->next; + } + } + *pkt_buffer_end = pktl; + /* first one*/ + pktl = *pkt_buffer; + if (!ff_check_st_index(pktl->pkt.stream_index, st_index)) { + av_packet_unref(&pktl->pkt); + *pkt_buffer = pktl->next; + if (!pktl->next) + *pkt_buffer_end = NULL; + av_freep(&pktl); + } + + return 0; +} + +int av_packet_buffer_filter(AVFormatContext *s, int *st_index) +{ + int ret = ff_packet_buffer_filter(&s->internal->packet_buffer, + &s->internal->packet_buffer_end, st_index); + return ret; +} + static int64_t ts_to_samples(AVStream *st, int64_t ts) { return av_rescale(ts, st->time_base.num * st->codecpar->sample_rate, st->time_base.den); diff --git a/libavformat/version.h b/libavformat/version.h index 75c03fde0a..c17727cf73 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -33,7 +33,7 @@ // Also please add any ticket numbers that you believe might be affected here #define LIBAVFORMAT_VERSION_MAJOR 58 #define LIBAVFORMAT_VERSION_MINOR 48 -#define LIBAVFORMAT_VERSION_MICRO 100 +#define LIBAVFORMAT_VERSION_MICRO 101 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ LIBAVFORMAT_VERSION_MINOR, \ From patchwork Thu Jul 16 12:51:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongcheng Zhong X-Patchwork-Id: 21111 Delivered-To: andriy.gelman@gmail.com Received: by 2002:a25:80ca:0:0:0:0:0 with SMTP id c10csp1198453ybm; Thu, 16 Jul 2020 05:52:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyAbt29LLuEmykzG4jT2QuLIqrl+K8jUYb+nyG4kvSAUwOcpxpCZQCScT/2PmFu1qluZ9hs X-Received: by 2002:a05:600c:2187:: with SMTP id e7mr4191086wme.171.1594903953884; Thu, 16 Jul 2020 05:52:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594903953; cv=none; d=google.com; s=arc-20160816; b=tuL415Dh4zHuyajTOTtnPb5up5B427uHPl8zJ7cYjdQU7OzMdvLdnelV1B63H61BHe 3wNIGC2oVu6azBAV4+81samTuUheZBoqD8TsBTnWDlIcEKGC9RNph6elP7FHRXaFw57/ 8OU7+7ClY4KnOsX/fLU+fcJ+OGex87nbhEkdeM8p8mepB4wZ4inF7m0MuarOi/do+aWI RlkyeonRPpbE6LegFh0Czl9Qr83yBxf61aP9qNvcrqYJ4ddEK8c/mWcJCD93KMv2TeYD u8p03L71yj8iQZngeT2dPNv3aNV5S061xWvyC0SIXaHg7rfPZrskbwHmh9rxBdMaYYs2 nBtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=mXVof2HQXuUHeBy7kPztTbGe45JgOeyx1JdwVArk2s0=; b=smuVXYtblVvkUO7PpkQPPYxA/JH458Yn69I/e8QeaLaQdWt1QVx6GJurOArJofeKsk 1wSvteg2cFvu/XEyGXvOQam944WaXEh/PmN+qmmEbaexZ48mSUTX18EP/hn1Cv872Lm+ jrSt+xblSe4W1L6wfyohx4JTNqC54nM2sZQoDJc3SraTFQ2dmssBTm39Q8OE41c00n7P SNtFOZRYEnOH7wjJar7ysX1b9+t0DYWQNSYTuii1SWpX75x3hUB5ki93Tw7LZp+ZFBBn tLdEuNcXAtDwSMvlNBYrX82G7US7z2WDrwUCjqX9cgXU3lS6N9d5hEZSvCyJR82a/ElI iLMQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id o18si4296217wrq.8.2020.07.16.05.52.33; Thu, 16 Jul 2020 05:52:33 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E8E8468B5D7; Thu, 16 Jul 2020 15:51:36 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp181.sjtu.edu.cn (smtp181.sjtu.edu.cn [202.120.2.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 060C768B5C6 for ; Thu, 16 Jul 2020 15:51:30 +0300 (EEST) Received: from proxy02.sjtu.edu.cn (smtp188.sjtu.edu.cn [202.120.2.188]) by smtp181.sjtu.edu.cn (Postfix) with ESMTPS id 7008B1008D5F4; Thu, 16 Jul 2020 20:51:25 +0800 (CST) Received: from localhost (localhost.localdomain [127.0.0.1]) by proxy02.sjtu.edu.cn (Postfix) with ESMTP id 6B0EC200B448D; Thu, 16 Jul 2020 20:51:25 +0800 (CST) X-Virus-Scanned: amavisd-new at Received: from proxy02.sjtu.edu.cn ([127.0.0.1]) by localhost (proxy02.sjtu.edu.cn [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id RLpjOdnBZREP; Thu, 16 Jul 2020 20:51:25 +0800 (CST) Received: from localhost.localdomain (unknown [10.162.157.92]) (Authenticated sender: sj.hc_Zhong@sjtu.edu.cn) by proxy02.sjtu.edu.cn (Postfix) with ESMTPSA id 7D934200B4496; Thu, 16 Jul 2020 20:51:23 +0800 (CST) From: Hongcheng Zhong To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Jul 2020 20:51:16 +0800 Message-Id: <20200716125116.187031-6-sj.hc_Zhong@sjtu.edu.cn> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> References: <20200716125116.187031-1-sj.hc_Zhong@sjtu.edu.cn> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC][GSoC][PATCH v2 6/6] ffplay: add av_packet_buffer_filter to filter packet buffer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: spartazhc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: WG92d4qU8Z/p Content-Length: 1626 From: spartazhc hls_read_header will add all streams to s->internal->packet_buffer. Use av_packet_buffer_filter to remove the AVPackets from other streams that are not needed, otherwise abr will allow them to be added to ffplay's packet_queue. Signed-off-by: spartazhc --- fftools/ffplay.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/fftools/ffplay.c b/fftools/ffplay.c index bcee507f40..ba71dbb38e 100644 --- a/fftools/ffplay.c +++ b/fftools/ffplay.c @@ -3018,6 +3018,10 @@ static int read_thread(void *arg) /* clean packet list filled in hls_read_header if abr is enabled */ if (abr) { + ret = av_packet_buffer_filter(ic, st_index); + if (ret < 0) { + av_log(NULL, AV_LOG_WARNING, "Failed to clean av_packet\n"); + } is->abr_list = av_mallocz(sizeof(ABRList)); if (!is->abr_list) { ret = AVERROR(ENOMEM);