From patchwork Fri Aug 17 13:01:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ronak Patel X-Patchwork-Id: 10021 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:104:0:0:0:0:0 with SMTP id c4-v6csp729789jad; Fri, 17 Aug 2018 06:02:17 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxX7gIeiUZGbokN+V4GFGRlmvspZocOlXMx+82ihgghbC1TYmaeNiUdT7mbW9Z7tzPG6I3Q X-Received: by 2002:adf:dbc3:: with SMTP id e3-v6mr21113150wrj.217.1534510937874; Fri, 17 Aug 2018 06:02:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534510937; cv=none; d=google.com; s=arc-20160816; b=F2WgDdEOk/P51P+AvLMNVrzNm9HZIQNhOhGHaR9OZ2I6qlv3bWTW/tzH5u6K3twKT4 KVF6k5TIe4fEGnxqyqkk7pqkHUNlnLMKTCHP5FEMBVBn4L/h7rQ3s1bNAK4LEKKfeI+S buEA/mrm4pfL6pTAsBJMCe1fjIHS5gDB9HMnRfTzguwlObeKIstHv52ngBAdQJZ4ilW3 bAR51Jnf5uYd/rD4tpztZDYD+ZmOr59o7/JRJI/ACdecnEcd9OgcC07nxRvQUKvpfTrV wEBOzFqvK4406aBZzLZ9/NXGjNxePkk97W7WSGvZs2HI+ulG7FDeV79kEULvDuzOG+Xy j5cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :dkim-signature:delivered-to:arc-authentication-results; bh=1JW4rPNTFLM/qpqu9WdMqouoB0LSduHzGT45RALxpoE=; b=mmCW6GnHK8RlAkpEsWH9+ABYhfVQWydgjjR63klYXQx0Ux6BCkFw1kYCBctITvuo9P U7QOzFcUMl+t/3OJ7a3v6CG+PvkhhlX0WrWHDrhRz8cv61sF1m59FNZSCMb3ATQCm3aG Xv3quaZg4agCTvYe2ET8QF54Zukgy618oZi2N2rq/Fx7Yh2dZ/FpMVpl5IWNM26p6ot1 sWXKdPwWDCSEVkRII7VzYPWZztFKAfKpb2eq6yeFFdUA89ngAZP4LLIEZ5tpTSWy8ONB LEhMopx3XWoY/MS7aJU0Ohb94IJmx2q3UZgUP3Psdgegwv4t6ESn51Vy9XvURjFUPLxj RNQQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@yahoo.com header.s=s2048 header.b=jNLNMMQK; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z15-v6si1630707wro.40.2018.08.17.06.02.17; Fri, 17 Aug 2018 06:02:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@yahoo.com header.s=s2048 header.b=jNLNMMQK; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 47345689FD7; Fri, 17 Aug 2018 16:01:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from sonic303-22.consmr.mail.gq1.yahoo.com (sonic303-22.consmr.mail.gq1.yahoo.com [98.137.64.203]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 830FC689FB3 for ; Fri, 17 Aug 2018 16:01:43 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1534510927; bh=85snLJQdEYSpxywg2CGqsUwbTewnaZd4AEYbDoILojo=; h=From:Subject:Date:To:From:Subject; b=jNLNMMQK7kx9jL/R2QQAgjlI6jPKe8p45OU51f3KGzIwGBOLrdaf3b0LIREtJstKCbaGzJFNmqvYuNSzN/ZMGRESTSxzWzC5aFwcPETWOj4+ivqC4gLn3iI1B5go04j3yeiJ/Qlvbp2YT1WHok0KxwtJYAAMUCOpIeaUwEltw6EAWk9w5OdviiwD/fWc3trbAWqQcFWXk6711LMsAb1NlwCbkE01mma1CK61PVL9GEhg//mZzx8iVwd1Pzl1oBOnr7ODIoDe8dBRxpGkbRls5Qjvrvtyep8fS8QadiRlbByrSffGATTfBHH0J9OCqLiW9JCRRMHpOBhITMc95pveTQ== X-YMail-OSG: 72u9crEVM1nj9QgM1iSfunxXo5Q1Wz1cAiSlgm5J3in1hXRz4RrVQZ2FZfGWk2w bES5hmQ162ZmjLwtzAsagNTTovMdLw8lPFKCrB2A3LKbQx1qLIitcOQUYOBfqE37vHrxvz7p.SV7 Q5t9yvlqIuhYUFXDbYbnXFnh.rUAeMaX_h4HKz3g6.lOAlR6bCEHYBdnF1LbtXvEq1aqTdYE3HBs UH67iNr7Tom2OR_w0K3tLJX5Da4JfHj7kxXzznrwfHsUmT8ewBRry08K9UwynJ8W1bCvO2QSRRA_ WJ_s3eokkBr0UWqH5I3VmYoREASapRjG7aszFuk5gOEzkLVKrLvop5YdiTsn8EU0elTa7doXWRty 44cZZytlg_hcVqSPdOi3GhhwjUynBIx2U2T4WUAKnDeRHTOj7HmIjk3ZiQRdiR0x9_cZP69Iex2M ruzhK2HPUhLxI8Sfo2SnmcGdZui9Val8LwC_j4sPS_9Y4IB.JWAZZIv4CgQujSA2PrFVHE421Amh FZaQ7mCj9Vy7k_dPNjSHJHx2SdgA1FOQayisBcPYTaRm0s.khS2PWmeUSIcr4JWdYNAnHy6YgIlU 0b4tFUovVePg_LTqYeQfCFe2ks0wxBY5pjKrlqElOcFYNtuBmWdRS4XBzbx0EkuvYBXm2Vmm6aok pZc_cu8R2OeoWAcpshUL_2k06oECtoxmxiZd_WrOwEfZXXhmzFI4.XBiPfez2oiEuuaw9Khah2Ck 4IuXbZ2cSfkbU4Ab72E0kTBDWtqWtS5JSRaYUqA55iTl.ckGgaofgKEsh2RxcWU2VzEOlSOri0Fj wvixfb4we4XhCPYjPmPulRVDK.uQyynTqjhK7vxcUKwwBU0_ToOk19peTRXOztIzqVWYUr7KFvs6 tj_CtG_qNH6SEQsRiHMZwfZ1dHdsV4CqiAFHlNFXOzR07MAhDDVnmxxXWOvS0_jsOsi3Dri7CMDW 3OmBch8W2chPTUjL43tXt8jvYnYn6Ihoyl61yyQ-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Fri, 17 Aug 2018 13:02:07 +0000 Received: from inet-64-112-176-1.bos.netblazr.com (EHLO [192.168.1.13]) ([64.112.176.1]) by smtp418.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 44d163cfdf24692450d8953e2ffc6e46 for ; Fri, 17 Aug 2018 13:02:02 +0000 (UTC) From: Ronak Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Message-Id: Date: Fri, 17 Aug 2018 09:01:53 -0400 To: FFmpeg development discussions and patches X-Mailer: Apple Mail (2.3445.9.1) Subject: [FFmpeg-devel] Resending Patch for hlsenc.c fixes for https://trac.ffmpeg.org/ticket/7281 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From 99e28c807a49cecf6ceb47ee44a85a3fdf78af64 Mon Sep 17 00:00:00 2001 From: "Ronak Patel" Date: Thu, 2 Aug 2018 09:25:12 -0400 Subject: [PATCH v3] libavformat/hlsenc: Fix HLS Manifest Generation from an N^2 algorithm to N. This fixes the creation of the hls manifest in hlsenc.c by writing the entire manifest at the end for VOD playlists. Live & Event Playlists are unaffected. This also fixes the behavior with HLS_TEMP_FILE to work correctly when -hlsflags temp_file is specified, instead of always relying on use_rename, which caused these problems. Files that would previously take over a week to fragment now take 1 minute on the same hardware. This was a 153 hour audio file (2.2GB of audio). Signed-off-by: Ronak Patel --- libavformat/hlsenc.c | 51 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 15 deletions(-) diff --git a/libavformat/hlsenc.c b/libavformat/hlsenc.c index b5644f0..55983a1 100644 --- a/libavformat/hlsenc.c +++ b/libavformat/hlsenc.c @@ -1168,7 +1168,6 @@ static int hls_rename_temp_file(AVFormatContext *s, AVFormatContext *oc) return AVERROR(ENOMEM); final_filename[len-4] = '\0'; ret = ff_rename(oc->url, final_filename, s); - oc->url[len-4] = '\0'; av_freep(&final_filename); return ret; } @@ -1374,7 +1373,7 @@ static int hls_window(AVFormatContext *s, int last, VariantStream *vs) char temp_filename[1024]; int64_t sequence = FFMAX(hls->start_sequence, vs->sequence - vs->nb_entries); const char *proto = avio_find_protocol_name(s->url); - int use_rename = proto && !strcmp(proto, "file"); + int use_temp_file = proto && !strcmp(proto, "file") && (s->flags & HLS_TEMP_FILE); static unsigned warned_non_file; char *key_uri = NULL; char *iv_string = NULL; @@ -1397,11 +1396,11 @@ static int hls_window(AVFormatContext *s, int last, VariantStream *vs) hls->version = 7; } - if (!use_rename && !warned_non_file++) + if (!use_temp_file && !warned_non_file++) av_log(s, AV_LOG_ERROR, "Cannot use rename on non file protocol, this may lead to races and temporary partial files\n"); set_http_options(s, &options, hls); - snprintf(temp_filename, sizeof(temp_filename), use_rename ? "%s.tmp" : "%s", vs->m3u8_name); + snprintf(temp_filename, sizeof(temp_filename), use_temp_file ? "%s.tmp" : "%s", vs->m3u8_name); if ((ret = hlsenc_io_open(s, &hls->m3u8_out, temp_filename, &options)) < 0) goto fail; @@ -1472,8 +1471,8 @@ fail: av_dict_free(&options); hlsenc_io_close(s, &hls->m3u8_out, temp_filename); hlsenc_io_close(s, &hls->sub_m3u8_out, vs->vtt_m3u8_name); - if (ret >= 0 && use_rename) - ff_rename(temp_filename, vs->m3u8_name, s); + if (use_temp_file) + ff_rename(temp_filename, vs->m3u8_name, s); if (ret >= 0 && hls->master_pl_name) if (create_master_playlist(s, vs) < 0) @@ -1488,6 +1487,8 @@ static int hls_start(AVFormatContext *s, VariantStream *vs) AVFormatContext *oc = vs->avf; AVFormatContext *vtt_oc = vs->vtt_avf; AVDictionary *options = NULL; + const char *proto = avio_find_protocol_name(s->url); + int use_temp_file = proto && !strcmp(proto, "file") && (s->flags & HLS_TEMP_FILE); char *filename, iv_string[KEYSIZE*2 + 1]; int err = 0; @@ -1583,7 +1584,7 @@ static int hls_start(AVFormatContext *s, VariantStream *vs) set_http_options(s, &options, c); - if (c->flags & HLS_TEMP_FILE) { + if (use_temp_file) { char *new_name = av_asprintf("%s.tmp", oc->url); if (!new_name) return AVERROR(ENOMEM); @@ -2145,6 +2146,8 @@ static int hls_write_packet(AVFormatContext *s, AVPacket *pkt) int ret = 0, can_split = 1, i, j; int stream_index = 0; int range_length = 0; + const char *proto = avio_find_protocol_name(s->url); + int use_temp_file = proto && !strcmp(proto, "file") && (s->flags & HLS_TEMP_FILE); uint8_t *buffer = NULL; VariantStream *vs = NULL; AVDictionary *options = NULL; @@ -2214,10 +2217,10 @@ static int hls_write_packet(AVFormatContext *s, AVPacket *pkt) } + char *old_filename = NULL; if (vs->packets_written && can_split && av_compare_ts(pkt->pts - vs->start_pts, st->time_base, end_pts, AV_TIME_BASE_Q) >= 0) { int64_t new_start_pos; - char *old_filename = NULL; int byterange_mode = (hls->flags & HLS_SINGLE_FILE) || (hls->max_seg_size > 0); av_write_frame(vs->avf, NULL); /* Flush any buffered data */ @@ -2253,11 +2256,13 @@ static int hls_write_packet(AVFormatContext *s, AVPacket *pkt) hlsenc_io_close(s, &vs->vtt_avf->pb, vs->vtt_avf->url); } } - if ((hls->flags & HLS_TEMP_FILE) && oc->url[0]) { + + // look to rename the asset name + if (use_temp_file && oc->url[0]) { if (!(hls->flags & HLS_SINGLE_FILE) || (hls->max_seg_size <= 0)) - if ((vs->avf->oformat->priv_class && vs->avf->priv_data) && hls->segment_type != SEGMENT_TYPE_FMP4) - av_opt_set(vs->avf->priv_data, "mpegts_flags", "resend_headers", 0); - hls_rename_temp_file(s, oc); + if ((vs->avf->oformat->priv_class && vs->avf->priv_data) && hls->segment_type != SEGMENT_TYPE_FMP4) { + av_opt_set(vs->avf->priv_data, "mpegts_flags", "resend_headers", 0); + } } if (vs->fmp4_init_mode) { @@ -2286,6 +2291,17 @@ static int hls_write_packet(AVFormatContext *s, AVPacket *pkt) return ret; } ff_format_io_close(s, &vs->out); + + // rename that segment from .tmp to the real one + if (use_temp_file && oc->url[0]) { + hls_rename_temp_file(s, oc); + av_free(old_filename); + old_filename = av_strdup(vs->avf->url); + + if (!old_filename) { + return AVERROR(ENOMEM); + } + } } } @@ -2334,10 +2350,12 @@ static int hls_write_packet(AVFormatContext *s, AVPacket *pkt) return ret; } - if (!vs->fmp4_init_mode || byterange_mode) + // if we're building a VOD playlist, skip writing the manifest multiple times, and just wait until the end + if (hls->pl_type != PLAYLIST_TYPE_VOD) { if ((ret = hls_window(s, 0, vs)) < 0) { return ret; } + } } vs->packets_written++; @@ -2352,6 +2370,8 @@ static int hls_write_trailer(struct AVFormatContext *s) AVFormatContext *oc = NULL; AVFormatContext *vtt_oc = NULL; char *old_filename = NULL; + const char *proto = avio_find_protocol_name(s->url); + int use_temp_file = proto && !strcmp(proto, "file") && (s->flags & HLS_TEMP_FILE); int i; int ret = 0; VariantStream *vs = NULL; @@ -2394,8 +2414,9 @@ failed: if (hls->segment_type != SEGMENT_TYPE_FMP4) ff_format_io_close(s, &oc->pb); - if ((hls->flags & HLS_TEMP_FILE) && oc->url[0]) { - hls_rename_temp_file(s, oc); + // rename that segment from .tmp to the real one + if (use_temp_file && oc->url[0] && !(hls->flags & HLS_SINGLE_FILE)) { + hls_rename_temp_file(s, oc); av_free(old_filename); old_filename = av_strdup(vs->avf->url);