From patchwork Tue Nov 15 19:11:11 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Franklin Phillips X-Patchwork-Id: 1432 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.90.1 with SMTP id o1csp1766850vsb; Tue, 15 Nov 2016 11:11:47 -0800 (PST) X-Received: by 10.28.137.81 with SMTP id l78mr5372760wmd.36.1479237107594; Tue, 15 Nov 2016 11:11:47 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s129si4126917wmb.120.2016.11.15.11.11.47; Tue, 15 Nov 2016 11:11:47 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@inbox.lv; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE dis=NONE) header.from=inbox.lv Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D4721689F22; Tue, 15 Nov 2016 21:11:44 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from shark2.inbox.lv (shark2.inbox.lv [194.152.32.82]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 52230689B97 for ; Tue, 15 Nov 2016 21:11:38 +0200 (EET) Received: from shark2.inbox.lv (localhost [127.0.0.1]) by shark2-out.inbox.lv (Postfix) with ESMTP id 07EA8243F0; Tue, 15 Nov 2016 21:11:38 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=inbox.lv; s=30062014; t=1479237098; bh=HkeP8gsQLPBQJR7nnI1CszrokrfTOpbU4cGZhfuuIxA=; l=1024; h=From:To:Cc:Subject:Date; b=lRiDB1046a4Un8KbaU4SGZZoRN8yKTTIKouJCmMIvYNVuBJ2f+y33tqwvbZJ5RWIo SKM1EY1zseXSttB0VRKVTjh4zQ/VvEwwj2euV1dy0l9KHd8upCUW7ohqFX6jzIuDpm W3bwDIH2D6kK22EVQqqgMdy7GmDzytpc9q4/l0w8= Received: from localhost (localhost [127.0.0.1]) by shark2-in.inbox.lv (Postfix) with ESMTP id F1767243C2; Tue, 15 Nov 2016 21:11:37 +0200 (EET) Received: from shark2.inbox.lv ([127.0.0.1]) by localhost (shark2.inbox.lv [127.0.0.1]) (spamfilter, port 35) with ESMTP id 115XWMAzve4y; Tue, 15 Nov 2016 21:11:33 +0200 (EET) Received: from mail.inbox.lv (pop1 [10.0.1.111]) by shark2-in.inbox.lv (Postfix) with ESMTP id D2C8E24306; Tue, 15 Nov 2016 21:11:33 +0200 (EET) Received: from localhost (unknown [61.74.121.198]) (Authenticated sender: franklinphillips9p8@inbox.lv) by mail.inbox.lv (Postfix) with ESMTPSA id C15CBDA1A1; Tue, 15 Nov 2016 21:11:31 +0200 (EET) From: Franklin Phillips To: ffmpeg-devel@ffmpeg.org Date: Tue, 15 Nov 2016 19:11:11 +0000 Message-Id: <1479237071-3239-1-git-send-email-franklinphillips9p8@inbox.lv> X-Mailer: git-send-email 2.1.4 X-Virus-Scanned: OK X-ESPOL: EZeEAiZdhQAmtca/NYNo+eLmx8iyX1I/31T3zKFSnX9YsrTDtNhzc2uQB/ecFHnb Subject: [FFmpeg-devel] [PATCH] avformat/hls: Add subtitle support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Franklin Phillips MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This patch is a fix for ticket #2833. Each subtitle segment is its own WebVTT file and has to demuxed individually. The timing of the subtitles are not perfect but it is the best I could do and it also does not take into account the X-TIMESTAMP-MAP header in the WebVTT files which is needed to conform to the specification (https://tools.ietf.org/html/draft-pantos-http-live-streaming-20#section-3.5). Signed-off-by: Franklin Phillips --- libavformat/hls.c | 178 ++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 159 insertions(+), 19 deletions(-) diff --git a/libavformat/hls.c b/libavformat/hls.c index 3ae3c7c..bf13be4 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -153,6 +153,8 @@ struct playlist { * playlist, if any. */ int n_init_sections; struct segment **init_sections; + + int is_subtitle; /* Indicates if the playlist is for subtitles */ }; /* @@ -312,6 +314,8 @@ static struct playlist *new_playlist(HLSContext *c, const char *url, pls->is_id3_timestamped = -1; pls->id3_mpegts_timestamp = AV_NOPTS_VALUE; + pls->is_subtitle = 0; + dynarray_add(&c->playlists, &c->n_playlists, pls); return pls; } @@ -482,11 +486,6 @@ static struct rendition *new_rendition(HLSContext *c, struct rendition_info *inf if (type == AVMEDIA_TYPE_SUBTITLE && !info->uri[0]) return NULL; - /* TODO: handle subtitles (each segment has to parsed separately) */ - if (c->strict_std_compliance > FF_COMPLIANCE_EXPERIMENTAL) - if (type == AVMEDIA_TYPE_SUBTITLE) - return NULL; - rend = av_mallocz(sizeof(struct rendition)); if (!rend) return NULL; @@ -501,9 +500,14 @@ static struct rendition *new_rendition(HLSContext *c, struct rendition_info *inf /* add the playlist if this is an external rendition */ if (info->uri[0]) { rend->playlist = new_playlist(c, info->uri, url_base); - if (rend->playlist) + if (rend->playlist) { dynarray_add(&rend->playlist->renditions, &rend->playlist->n_renditions, rend); + if (type == AVMEDIA_TYPE_SUBTITLE) { + rend->playlist->is_subtitle = 1; + rend->playlist->is_id3_timestamped = 0; + } + } } if (info->assoc_language[0]) { @@ -1349,6 +1353,136 @@ reload: goto restart; } +static int nested_io_open(AVFormatContext *s, AVIOContext **pb, const char *url, + int flags, AVDictionary **opts) +{ + av_log(s, AV_LOG_ERROR, + "A HLS playlist item '%s' referred to an external file '%s'. " + "Opening this file was forbidden for security reasons\n", + s->filename, url); + return AVERROR(EPERM); +} + +static int read_data_simple(void *opaque, uint8_t *buf, int buf_size) +{ + struct playlist *v = opaque; + HLSContext *c = v->parent->priv_data; + struct segment *seg; + + if (v->cur_seq_no >= v->start_seq_no + v->n_segments) { + return AVERROR_EOF; + } else { + seg = current_segment(v); + } + + if (!v->input) { + open_input(c, v, seg); + } + + return read_from_url(v, seg, buf, buf_size, READ_NORMAL); +} + +static int read_packet_subtitle(struct playlist *v, AVPacket *pkt) +{ + HLSContext *c = v->parent->priv_data; + int ret, i; + +restart: + if (!v->needed) + return AVERROR_EOF; + + if (!v->input) { + int64_t reload_interval; + + /* Check that the playlist is still needed before opening a new + * segment. */ + if (v->ctx && v->ctx->nb_streams) { + v->needed = 0; + for (i = 0; i < v->n_main_streams; i++) { + if (v->main_streams[i]->discard < AVDISCARD_ALL) { + v->needed = 1; + break; + } + } + } + if (!v->needed) { + av_log(v->parent, AV_LOG_INFO, "No longer receiving playlist %d\n", + v->index); + return AVERROR_EOF; + } + + /* If this is a live stream and the reload interval has elapsed since + * the last playlist reload, reload the playlists now. */ + reload_interval = default_reload_interval(v); + + if (!v->finished && + av_gettime_relative() - v->last_load_time >= reload_interval) { + if ((ret = parse_playlist(c, v->url, v, NULL)) < 0) { + av_log(v->parent, AV_LOG_WARNING, "Failed to reload playlist %d\n", + v->index); + return ret; + } + /* If we need to reload the playlist again below (if + * there's still no more segments), switch to a reload + * interval of half the target duration. */ + reload_interval = v->target_duration / 2; + } + if (v->cur_seq_no < v->start_seq_no) { + av_log(NULL, AV_LOG_WARNING, + "skipping %d segments ahead, expired from subtitle playlists\n", + v->start_seq_no - v->cur_seq_no); + v->cur_seq_no = v->start_seq_no; + } + if (v->cur_seq_no >= v->start_seq_no + v->n_segments) { + return AVERROR_EOF; + } + } + + if (v->ctx == NULL) { + AVInputFormat *in_fmt; + + if (!(v->ctx = avformat_alloc_context())) { + return AVERROR(ENOMEM); + } + + v->read_buffer = av_malloc(INITIAL_BUFFER_SIZE); + if (!v->read_buffer){ + avformat_free_context(v->ctx); + v->ctx = NULL; + return AVERROR(ENOMEM); + } + + ffio_init_context(&v->pb, v->read_buffer, INITIAL_BUFFER_SIZE, 0, v, + read_data_simple, NULL, NULL); + v->pb.seekable = 0; + v->ctx->pb = &v->pb; + v->ctx->io_open = nested_io_open; + + ret = ff_copy_whiteblacklists(v->ctx, v->parent); + if (ret < 0) { + return ret; + } + + in_fmt = av_find_input_format("webvtt"); + ret = avformat_open_input(&v->ctx, current_segment(v)->url, in_fmt, NULL); + if (ret < 0) { + return ret; + } + } + + ret = av_read_frame(v->ctx, pkt); + if (ret < 0) { + ff_format_io_close(v->parent, &v->input); + avformat_close_input(&v->ctx); + if (ret == AVERROR_EOF) { + v->cur_seq_no++; + goto restart; + } + } + + return ret; +} + static void add_renditions_to_variant(HLSContext *c, struct variant *var, enum AVMediaType type, const char *group_id) { @@ -1492,16 +1626,6 @@ static int save_avio_options(AVFormatContext *s) return ret; } -static int nested_io_open(AVFormatContext *s, AVIOContext **pb, const char *url, - int flags, AVDictionary **opts) -{ - av_log(s, AV_LOG_ERROR, - "A HLS playlist item '%s' referred to an external file '%s'. " - "Opening this file was forbidden for security reasons\n", - s->filename, url); - return AVERROR(EPERM); -} - static void add_stream_to_programs(AVFormatContext *s, struct playlist *pls, AVStream *stream) { HLSContext *c = s->priv_data; @@ -1744,8 +1868,14 @@ static int hls_read_header(AVFormatContext *s) pls->ctx = NULL; goto fail; } - ffio_init_context(&pls->pb, pls->read_buffer, INITIAL_BUFFER_SIZE, 0, pls, - read_data, NULL, NULL); + + if (pls->is_subtitle) { + ffio_init_context(&pls->pb, pls->read_buffer, INITIAL_BUFFER_SIZE, 0, pls, + read_data_simple, NULL, NULL); + } else { + ffio_init_context(&pls->pb, pls->read_buffer, INITIAL_BUFFER_SIZE, 0, pls, + read_data, NULL, NULL); + } pls->pb.seekable = 0; ret = av_probe_input_buffer(&pls->pb, &in_fmt, pls->segments[0]->url, NULL, 0, 0); @@ -1843,6 +1973,8 @@ static int recheck_discard_flags(AVFormatContext *s, int first) } else if (first && !pls->cur_needed && pls->needed) { if (pls->input) ff_format_io_close(pls->parent, &pls->input); + if (pls->is_subtitle) + avformat_close_input(&pls->ctx); pls->needed = 0; changed = 1; av_log(s, AV_LOG_INFO, "No longer receiving playlist %d\n", i); @@ -1909,7 +2041,12 @@ static int hls_read_packet(AVFormatContext *s, AVPacket *pkt) while (1) { int64_t ts_diff; AVRational tb; - ret = av_read_frame(pls->ctx, &pls->pkt); + if (pls->is_subtitle) { + ret = read_packet_subtitle(pls, &pls->pkt); + } else { + ret = av_read_frame(pls->ctx, &pls->pkt); + } + if (ret < 0) { if (!avio_feof(&pls->pb) && ret != AVERROR_EOF) return ret; @@ -2087,6 +2224,9 @@ static int hls_read_seek(AVFormatContext *s, int stream_index, /* Flush the packet queue of the subdemuxer. */ ff_read_frame_flush(pls->ctx); + if (pls->is_subtitle) + avformat_close_input(&pls->ctx); + pls->seek_timestamp = seek_timestamp; pls->seek_flags = flags;