From patchwork Wed Aug 1 20:41:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liu Steven X-Patchwork-Id: 9857 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:104:0:0:0:0:0 with SMTP id c4-v6csp1274232jad; Wed, 1 Aug 2018 13:42:34 -0700 (PDT) X-Google-Smtp-Source: AAOMgpdzMZ4S3axuKqcoLszKAVCvBLMYMspwe39Ck1JEz58NU76V74x3ym+0plJT2sjhTgfS4R4a X-Received: by 2002:a1c:af53:: with SMTP id y80-v6mr35147wme.55.1533156154673; Wed, 01 Aug 2018 13:42:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533156154; cv=none; d=google.com; s=arc-20160816; b=r6HNKbrnnF+B0Osv0z5ks1Nos8WFZH2DO0BNUVmFbbQesRjPfRsEpvBMVxVSasQMMS GGYs+iA6g4hiKIx7DEkFMgRI8CSA8Z2OWrgFOIUSQEtoIivU6j7KjEU6eFB7gDoy/zyc cw8eJStjSXt2cNV/hzFaFXpnF/q6opgabjCTipFFAyl1lJeNT/ua9KD+DpSFasKPe0IZ NHFLMNaiaiShUX2uNf8c9ZbP8f95JJFFEtmbkfCi6ZeYFclIiP/pRoABVbbW/ivOXCsI 60IzjnnVqNp4wMuvnSgICJbpQYgmvMB+44cbnk0FzS0FopPfhGWPFfagbJMc/9V509tt HQGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:feedback-id:to:references:message-id :date:in-reply-to:from:mime-version:delivered-to :arc-authentication-results; bh=ywjILC8sLOIfBiPICSvIfw1pdEOLYaCG2YgnhLZd4Gc=; b=VAVFOuveoca5EQtgRxeSbTco7OTe8JAUkDhQFLG3Kwf8qXcqGGgIdEy9C9elwiH330 UEmIHBFzzA+rjsMnavbC9pQkbqPK5zaFYvs9GBFMBO7uX9YPmESWdk/E2Qq5lGIn3SJa pjZcLauGOzC3CUPc1yeZ+AROZUQ0PXBnNPN2vNPxPdoGmn7ZhtHj3YFCCOlnF0ZR5e0Z 8CcYGbuyPNuZIKdzZrw7PJ8zMf3iLNC5RKzAOWQjM8A9SqMBL86czxd+phdpmphiRuYh IdZrELy9LKczBEio+Cjo+pPfD9jwm3kHOPDLXAcNByUgyphRe7dZdT5LL1aakFr+FHQO Hiog== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e7-v6si7800651wri.119.2018.08.01.13.42.33; Wed, 01 Aug 2018 13:42:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 39ED4689FF7; Wed, 1 Aug 2018 23:42:14 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtpbgsg2.qq.com (smtpbgsg2.qq.com [54.254.200.128]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EBDE268990E for ; Wed, 1 Aug 2018 23:42:06 +0300 (EEST) X-QQ-mid: bizesmtp7t1533156137tbjl53bqo Received: from [192.168.3.5] (unknown [123.115.198.36]) by esmtp4.qq.com (ESMTP) with id ; Thu, 02 Aug 2018 04:42:16 +0800 (CST) X-QQ-SSF: 01100000008000F0FLF0B00A0000000 X-QQ-FEAT: gpnk31ZXf4FSBzHrAkH5wtzcKKVGMkD+a4v7GNfVSnYY9YOu60HARM9RZSmif 7CEkqEvmJiHu0w28FC/Sq8DL/NATSJu5kDlT7rX8Zothx27t78lTBW7Rd2MvQbta/Sg2BJ8 i4TwW3uH7yLnIV8hJCqVwvEcy9Teuxd+mRjcveUH/ZcfbMnQ4b0pE3cccFkBcOaPizCzp2C xslObFL/7Sh2OxB0UnDqoy1PGOojk49b5ckc00PuAnzmJHXjq5COsD4tnsIQzSx2MeEk6d6 YkW3Z4T21h8pcqC26CSxMyhKtTHnn4F9Rm0c4Xh35lT2lw X-QQ-GoodBg: 0 Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) From: Steven Liu In-Reply-To: Date: Thu, 2 Aug 2018 04:41:47 +0800 Message-Id: References: <0AF864E1-BEEF-41B7-98F1-49A8C6B04A10@yahoo.com> <81C29710-A67A-4C1E-B8B0-2B03262A7F89@yahoo.com> To: FFmpeg development discussions and patches X-Mailer: Apple Mail (2.3445.6.18) X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:chinaffmpeg.org:qybgforeign:qybgforeign4 X-QQ-Bgrelay: 1 Subject: Re: [FFmpeg-devel] First Patch for hlsenc.c for https://trac.ffmpeg.org/ticket/7281 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Steven Liu Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" > On Aug 2, 2018, at 03:50, Ronak wrote: > > <0001-libavformat-hlsenc-Fix-HLS-Manifest-Generation-from-.patch> From e7ff03ed52c709d647a112833427b44c41e3ed12 Mon Sep 17 00:00:00 2001 From: "Ronak Patel (Audible)" Date: Tue, 31 Jul 2018 19:05:18 -0400 Subject: [PATCH] libavformat/hlsenc: Fix HLS Manifest Generation from an N^2 algorithm to N. This fixes the creation of the hls manifest in hlsenc.c by writing the header once in hls_write_header, and the segments individually in the hls_window method. Files that would previously take over a week to fragment now take 1 minute on the same hardware. This was a 153 hour audio file (2.2GB of audio). Signed-off-by: Ronak Patel --- libavformat/hlsenc.c | 143 +++++++++++++++++++++++++++------------------------ 1 file changed, 76 insertions(+), 67 deletions(-) diff --git a/libavformat/hlsenc.c b/libavformat/hlsenc.c index b5644f0..b15645d 100644 --- a/libavformat/hlsenc.c +++ b/libavformat/hlsenc.c @@ -488,7 +488,6 @@ static int hls_delete_old_segments(AVFormatContext *s, HLSContext *hls, } p = (char *)av_basename(dirname); *p = '\0'; - } while (segment) { @@ -1365,63 +1364,37 @@ fail: return ret; } -static int hls_window(AVFormatContext *s, int last, VariantStream *vs) +static int hls_write_manifest_segment(AVFormatContext *s, int last, VariantStream *vs) { HLSContext *hls = s->priv_data; HLSSegment *en; - int target_duration = 0; int ret = 0; - char temp_filename[1024]; - int64_t sequence = FFMAX(hls->start_sequence, vs->sequence - vs->nb_entries); - const char *proto = avio_find_protocol_name(s->url); - int use_rename = proto && !strcmp(proto, "file"); - static unsigned warned_non_file; + int byterange_mode; char *key_uri = NULL; char *iv_string = NULL; AVDictionary *options = NULL; double prog_date_time = vs->initial_prog_date_time; double *prog_date_time_p = (hls->flags & HLS_PROGRAM_DATE_TIME) ? &prog_date_time : NULL; - int byterange_mode = (hls->flags & HLS_SINGLE_FILE) || (hls->max_seg_size > 0); - - hls->version = 3; - if (byterange_mode) { - hls->version = 4; - sequence = 0; - } - - if (hls->flags & HLS_INDEPENDENT_SEGMENTS) { - hls->version = 6; - } - if (hls->segment_type == SEGMENT_TYPE_FMP4) { - hls->version = 7; - } + if (last) { - if (!use_rename && !warned_non_file++) - av_log(s, AV_LOG_ERROR, "Cannot use rename on non file protocol, this may lead to races and temporary partial files\n"); + if ((hls->flags & HLS_OMIT_ENDLIST==0)) { + ff_hls_write_end_list(hls->m3u8_out); + } - set_http_options(s, &options, hls); - snprintf(temp_filename, sizeof(temp_filename), use_rename ? "%s.tmp" : "%s", vs->m3u8_name); - if ((ret = hlsenc_io_open(s, &hls->m3u8_out, temp_filename, &options)) < 0) - goto fail; + if (vs->vtt_m3u8_name) { + ff_hls_write_end_list(hls->sub_m3u8_out); + } + } else { - for (en = vs->segments; en; en = en->next) { - if (target_duration <= en->duration) - target_duration = lrint(en->duration); - } + byterange_mode = (hls->flags & HLS_SINGLE_FILE) || (hls->max_seg_size > 0); - vs->discontinuity_set = 0; - ff_hls_write_playlist_header(hls->m3u8_out, hls->version, hls->allowcache, - target_duration, sequence, hls->pl_type); + en = vs->last_segment; + if (en == NULL) { + ret = 1; + goto fail; + } - if((hls->flags & HLS_DISCONT_START) && sequence==hls->start_sequence && vs->discontinuity_set==0 ){ - avio_printf(hls->m3u8_out, "#EXT-X-DISCONTINUITY\n"); - vs->discontinuity_set = 1; - } - if (vs->has_video && (hls->flags & HLS_INDEPENDENT_SEGMENTS)) { - avio_printf(hls->m3u8_out, "#EXT-X-INDEPENDENT-SEGMENTS\n"); - } - for (en = vs->segments; en; en = en->next) { if ((hls->encrypt || hls->key_info_file) && (!key_uri || strcmp(en->key_uri, key_uri) || av_strcasecmp(en->iv_string, iv_string))) { avio_printf(hls->m3u8_out, "#EXT-X-KEY:METHOD=AES-128,URI=\"%s\"", en->key_uri); @@ -1444,17 +1417,8 @@ static int hls_window(AVFormatContext *s, int last, VariantStream *vs) if (ret < 0) { av_log(s, AV_LOG_WARNING, "ff_hls_write_file_entry get error\n"); } - } - - if (last && (hls->flags & HLS_OMIT_ENDLIST)==0) - ff_hls_write_end_list(hls->m3u8_out); - if( vs->vtt_m3u8_name ) { - if ((ret = hlsenc_io_open(s, &hls->sub_m3u8_out, vs->vtt_m3u8_name, &options)) < 0) - goto fail; - ff_hls_write_playlist_header(hls->sub_m3u8_out, hls->version, hls->allowcache, - target_duration, sequence, PLAYLIST_TYPE_NONE); - for (en = vs->segments; en; en = en->next) { + if( vs->vtt_m3u8_name ) { ret = ff_hls_write_file_entry(hls->sub_m3u8_out, 0, byterange_mode, en->duration, 0, en->size, en->pos, vs->baseurl, en->sub_filename, NULL); @@ -1462,22 +1426,16 @@ static int hls_window(AVFormatContext *s, int last, VariantStream *vs) av_log(s, AV_LOG_WARNING, "ff_hls_write_file_entry get error\n"); } } - - if (last) - ff_hls_write_end_list(hls->sub_m3u8_out); - } fail: av_dict_free(&options); - hlsenc_io_close(s, &hls->m3u8_out, temp_filename); - hlsenc_io_close(s, &hls->sub_m3u8_out, vs->vtt_m3u8_name); - if (ret >= 0 && use_rename) - ff_rename(temp_filename, vs->m3u8_name, s); - if (ret >= 0 && hls->master_pl_name) - if (create_master_playlist(s, vs) < 0) + if (ret >= 0 && hls->master_pl_name) { + if (create_master_playlist(s, vs) < 0) { av_log(s, AV_LOG_WARNING, "Master playlist creation failed\n"); + } + } return ret; } @@ -2078,6 +2036,11 @@ static int hls_write_header(AVFormatContext *s) int ret, i, j; AVDictionary *options = NULL; VariantStream *vs = NULL; + int target_duration = 0; + int byterange_mode = 0; + + target_duration = lrint(hls->time); + byterange_mode = (hls->flags & HLS_SINGLE_FILE) || (hls->max_seg_size > 0); for (i = 0; i < hls->nb_varstreams; i++) { vs = &hls->var_streams[i]; @@ -2091,7 +2054,7 @@ static int hls_write_header(AVFormatContext *s) goto fail; } av_dict_free(&options); - //av_assert0(s->nb_streams == hls->avf->nb_streams); + for (j = 0; j < vs->nb_streams; j++) { AVStream *inner_st; AVStream *outer_st = vs->streams[j]; @@ -2130,6 +2093,47 @@ static int hls_write_header(AVFormatContext *s) } } } + + // write the header for the main hls playlist + int64_t sequence = FFMAX(hls->start_sequence, vs->sequence - vs->nb_entries); + hls->version = 3; + if (byterange_mode) { + hls->version = 4; + sequence = 0; + } + + if (hls->flags & HLS_INDEPENDENT_SEGMENTS) { + hls->version = 6; + } + + if (hls->segment_type == SEGMENT_TYPE_FMP4) { + hls->version = 7; + } + + set_http_options(s, &options, hls); + if ((ret = hlsenc_io_open(s, &hls->m3u8_out, vs->m3u8_name, &options)) < 0) { + goto fail; + } + + vs->discontinuity_set = 0; + ff_hls_write_playlist_header(hls->m3u8_out, hls->version, hls->allowcache, target_duration, sequence, hls->pl_type); + + if((hls->flags & HLS_DISCONT_START) && sequence==hls->start_sequence && vs->discontinuity_set==0 ){ + avio_printf(hls->m3u8_out, "#EXT-X-DISCONTINUITY\n"); + vs->discontinuity_set = 1; + } + if (vs->has_video && (hls->flags & HLS_INDEPENDENT_SEGMENTS)) { + avio_printf(hls->m3u8_out, "#EXT-X-INDEPENDENT-SEGMENTS\n"); + } + + if( vs->vtt_m3u8_name ) { + if ((ret = hlsenc_io_open(s, &hls->sub_m3u8_out, vs->vtt_m3u8_name, &options)) < 0) { + goto fail; + } + + ff_hls_write_playlist_header(hls->sub_m3u8_out, hls->version, hls->allowcache, target_duration, sequence, PLAYLIST_TYPE_NONE); + } + fail: return ret; @@ -2334,10 +2338,11 @@ static int hls_write_packet(AVFormatContext *s, AVPacket *pkt) return ret; } - if (!vs->fmp4_init_mode || byterange_mode) - if ((ret = hls_window(s, 0, vs)) < 0) { + if (!vs->fmp4_init_mode || byterange_mode) { + if ((ret = hls_write_manifest_segment(s, 0, vs)) < 0) { return ret; } + } } vs->packets_written++; @@ -2421,7 +2426,7 @@ failed: avformat_free_context(oc); vs->avf = NULL; - hls_window(s, 1, vs); + hls_write_manifest_segment(s, 1, vs); av_freep(&vs->fmp4_init_filename); if (vtt_oc) { @@ -2447,6 +2452,9 @@ failed: av_freep(&ccs->language); } + hlsenc_io_close(s, &hls->m3u8_out, vs->m3u8_name); + hlsenc_io_close(s, &hls->sub_m3u8_out, vs->vtt_m3u8_name); + ff_format_io_close(s, &hls->m3u8_out); ff_format_io_close(s, &hls->sub_m3u8_out); av_freep(&hls->key_basename); @@ -2884,3 +2892,4 @@ AVOutputFormat ff_hls_muxer = { .write_trailer = hls_write_trailer, .priv_class = &hls_class, }; +