From patchwork Fri Dec 30 20:10:05 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Franklin Phillips X-Patchwork-Id: 1995 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.89.21 with SMTP id n21csp2995122vsb; Fri, 30 Dec 2016 12:10:46 -0800 (PST) X-Received: by 10.28.13.9 with SMTP id 9mr39075511wmn.50.1483128646540; Fri, 30 Dec 2016 12:10:46 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r3si63579760wjs.183.2016.12.30.12.10.46; Fri, 30 Dec 2016 12:10:46 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4721A689CB2; Fri, 30 Dec 2016 22:10:32 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7DDF6689BE7 for ; Fri, 30 Dec 2016 22:10:25 +0200 (EET) Received: from localhost.localdomain ([59.26.118.28]) by mail.gmx.com (mrgmx103 [212.227.17.174]) with ESMTPSA (Nemesis) id 0MfVzj-1cBoyy3NL1-00P6lU; Fri, 30 Dec 2016 21:10:26 +0100 From: Franklin Phillips To: ffmpeg-devel@ffmpeg.org Date: Fri, 30 Dec 2016 20:10:05 +0000 Message-Id: <20161230201005.7097-2-franklinphillips@gmx.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20161230201005.7097-1-franklinphillips@gmx.com> References: <20161216171350.3a4f064b@debian> <20161230201005.7097-1-franklinphillips@gmx.com> X-Provags-ID: V03:K0:DR8VuCAxmDNRHLg55z2SSCM7+tp9XL2IYpaKIccgk3EtR363IOr ijQzRwfB48Lq5/EBW+YlwA+UJOiKV8qR7RCQ27klVEpa87VvvtOG83WayU2VZMIe5MQG9HG TF+LZXeHhRVdHtIqCVt8sv6/N9k5YGgzExHDThrdhqy2/burwmt2v5bcrUdZVFMZAl71hcg g3Ua0UI7U3VDIcfLFcvzg== X-UI-Out-Filterresults: notjunk:1; V01:K0:TlVxfxXrAew=:pJX5P0lQWjigCIq0ioWG9i PT2C0gCF0xd0yazAz9+ld6bYn2IPa615ZFwUCUvuFaEabiZxn7oWnF1eoNIr28QPASU4r1RkH dGf74b7kxLgADqVV3Xt3y7x5JQzR2e1IC+/XLQgDcJlAD5XFG+JacFkq3m0kYfUdeFmVNsIxy 03Ge9z9J0XLGxnqpSdkCkSjfilvM+yO20VOcTHkxZxj6xVNTnmouvvRi+BDp5683uyuo1oIZ7 9PowlRp2Ieh+2bNsjZ+WhZrjDXRJ+FAb4BT3v+/8nQCm3vRVvqjKR6bET4RktU0yGnWKfTT/j 0I9ej5MWkRUN1fH+lXAwGVhytZoym5GzfFAsPOX7zWPldRzfnOSiJ+cD0CPSy6qd24p3mgObd 4gzD1V01X5Ktmro3qtm5VHOcnQ9ITp+FpLVcmUjksokYKIyBFpP0vScvhWStevDxYvctqXV2p uTHLe8zQg/V7C+CBegaS7Nh0cAOLrAInEvLjBY2MrwjKhXNVpBx1KczQvU9w67AhfHd+jvCZ6 Ds0X794UWJsGMTir0UZyr2pOOopDYcik8MMwS2iMRRwGGOtFa2KZvTOinLY4NID97WbD419nv OVb17MSBD1k8VhUDOyT6JcLFJzoLuOuqbNr+jaa2+rFkWgsSOixZAOTgY0JXKSnGU0q/AG80i PodxeOX3qSRI0r9Dj13hMpkR1abpv/OxnKy7ltjKN6b36bD/wGOBRkouCrqdUJK4ojTDzJv2J ldOx/ssnd2hrZuxVWd14wI1I9wv/X6kpnYLcoD+Zl2cu0a2s8G0hf5mau/JR1aUhV4DsboVY0 dULX8VYaa2Ry7cjIIFBw9KKUOCUyv5wwuhPKbMtUhkGFGhaeppeNeXrm6T8t4eJcnSbyvbEDi J3vAVBew//P3pvgoCyYLBRr1c8V1z5Uq98MAYoMUuz7a22fsuFi5g5syLXxtxivyHvB/pSR4Q OXubNCoIVS7hFly8Rbx8HOtOX1rK4Piaq6o2zXzxZpTl9/3dG59+KqxNa0LtGCFopuQt6NuO1 pmBg7t0DMGXXWpzyHCxG1lrke0ZtRWt2jCxXsqPt5Msm Subject: [FFmpeg-devel] [PATCH 2/2] avformat/hls: Add subtitle support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Franklin Phillips MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Each subtile segment is a WebVTT file and needs to be demuxed separately. These segments also contain a header to synchronize their timing with the MPEG TS stream so those timestamps are requested from the WebVTT demuxer through an AVOption. Signed-off-by: Franklin Phillips --- libavformat/hls.c | 196 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 138 insertions(+), 58 deletions(-) diff --git a/libavformat/hls.c b/libavformat/hls.c index 3ae3c7c..278c3b2 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -153,6 +153,8 @@ struct playlist { * playlist, if any. */ int n_init_sections; struct segment **init_sections; + + int is_subtitle; /* Indicates if it's a subtitle playlist */ }; /* @@ -203,6 +205,7 @@ typedef struct HLSContext { char *headers; ///< holds HTTP headers set as an AVOption to the HTTP protocol context char *http_proxy; ///< holds the address of the HTTP proxy server AVDictionary *avio_opts; + AVDictionary *demuxer_opts; int strict_std_compliance; } HLSContext; @@ -309,6 +312,8 @@ static struct playlist *new_playlist(HLSContext *c, const char *url, ff_make_absolute_url(pls->url, sizeof(pls->url), base, url); pls->seek_timestamp = AV_NOPTS_VALUE; + pls->is_subtitle = 0; + pls->is_id3_timestamped = -1; pls->id3_mpegts_timestamp = AV_NOPTS_VALUE; @@ -482,11 +487,6 @@ static struct rendition *new_rendition(HLSContext *c, struct rendition_info *inf if (type == AVMEDIA_TYPE_SUBTITLE && !info->uri[0]) return NULL; - /* TODO: handle subtitles (each segment has to parsed separately) */ - if (c->strict_std_compliance > FF_COMPLIANCE_EXPERIMENTAL) - if (type == AVMEDIA_TYPE_SUBTITLE) - return NULL; - rend = av_mallocz(sizeof(struct rendition)); if (!rend) return NULL; @@ -501,9 +501,14 @@ static struct rendition *new_rendition(HLSContext *c, struct rendition_info *inf /* add the playlist if this is an external rendition */ if (info->uri[0]) { rend->playlist = new_playlist(c, info->uri, url_base); - if (rend->playlist) + if (rend->playlist) { dynarray_add(&rend->playlist->renditions, &rend->playlist->n_renditions, rend); + if (type == AVMEDIA_TYPE_SUBTITLE) { + rend->playlist->is_subtitle = 1; + rend->playlist->is_id3_timestamped = 0; + } + } } if (info->assoc_language[0]) { @@ -1241,8 +1246,103 @@ static int read_data(void *opaque, uint8_t *buf, int buf_size) { struct playlist *v = opaque; HLSContext *c = v->parent->priv_data; - int ret, i; - int just_opened = 0; + int ret, just_opened = 0; + struct segment *seg; + + if (v->cur_seq_no - v->start_seq_no >= v->n_segments) { + return AVERROR_EOF; + } else { + seg = current_segment(v); + } + + if (!v->input) { + /* load/update Media Initialization Section, if any */ + ret = update_init_section(v, seg); + if (ret < 0) + return ret; + + ret = open_input(c, v, seg); + if (ret < 0) { + if (ff_check_interrupt(c->interrupt_callback)) + return AVERROR_EXIT; + av_log(v->parent, AV_LOG_WARNING, "Failed to open segment of playlist %d\n", + v->index); + return ret; + } + just_opened = 1; + } + + if (v->init_sec_buf_read_offset < v->init_sec_data_len) { + /* Push init section out first before first actual segment */ + int copy_size = FFMIN(v->init_sec_data_len - v->init_sec_buf_read_offset, buf_size); + memcpy(buf, v->init_sec_buf, copy_size); + v->init_sec_buf_read_offset += copy_size; + return copy_size; + } + + ret = read_from_url(v, seg, buf, buf_size, READ_NORMAL); + if (ret > 0) { + if (just_opened && v->is_id3_timestamped != 0) { + /* Intercept ID3 tags here, elementary audio streams are required + * to convey timestamps using them in the beginning of each segment. */ + intercept_id3(v, buf, buf_size, &ret); + } + } + + return ret; +} + +static int nested_io_open(AVFormatContext *s, AVIOContext **pb, const char *url, + int flags, AVDictionary **opts) +{ + av_log(s, AV_LOG_ERROR, + "A HLS playlist item '%s' referred to an external file '%s'. " + "Opening this file was forbidden for security reasons\n", + s->filename, url); + return AVERROR(EPERM); +} + +static int init_subtitle_context(struct playlist *pls) +{ + HLSContext *c = pls->parent->priv_data; + AVInputFormat *in_fmt; + AVDictionary *opts = NULL; + int ret = 0; + + if (!(pls->ctx = avformat_alloc_context())) + return AVERROR(ENOMEM); + + pls->read_buffer = av_malloc(INITIAL_BUFFER_SIZE); + if (!pls->read_buffer) { + avformat_free_context(pls->ctx); + pls->ctx = NULL; + return AVERROR(ENOMEM); + } + + ffio_init_context(&pls->pb, pls->read_buffer, INITIAL_BUFFER_SIZE, 0, pls, + read_data, NULL, NULL); + pls->pb.seekable = 0; + pls->ctx->pb = &pls->pb; + pls->ctx->io_open = nested_io_open; + + ret = ff_copy_whiteblacklists(pls->ctx, pls->parent); + if (ret < 0) + return ret; + + in_fmt = av_find_input_format("webvtt"); + av_dict_copy(&opts, c->demuxer_opts, 0); + ret = avformat_open_input(&pls->ctx, current_segment(pls)->url, in_fmt, &opts); + av_dict_free(&opts); + if (ret < 0) + return ret; + + return ret; +} + +static int read_packet(struct playlist *v, AVPacket *pkt) +{ + HLSContext *c = v->parent->priv_data; + int ret; restart: if (!v->needed) @@ -1250,11 +1350,11 @@ restart: if (!v->input) { int64_t reload_interval; - struct segment *seg; /* Check that the playlist is still needed before opening a new * segment. */ - if (v->ctx && v->ctx->nb_streams) { + if ((v->ctx && v->ctx->nb_streams) || v->is_subtitle) { + int i; v->needed = 0; for (i = 0; i < v->n_main_streams; i++) { if (v->main_streams[i]->discard < AVDISCARD_ALL) { @@ -1270,7 +1370,7 @@ restart: } /* If this is a live stream and the reload interval has elapsed since - * the last playlist reload, reload the playlists now. */ + * the last playlist reload, reload the playlists now. */ reload_interval = default_reload_interval(v); reload: @@ -1293,7 +1393,7 @@ reload: v->cur_seq_no = v->start_seq_no; } if (v->cur_seq_no >= v->start_seq_no + v->n_segments) { - if (v->finished) + if (v->finished || v->is_subtitle) return AVERROR_EOF; while (av_gettime_relative() - v->last_load_time < reload_interval) { if (ff_check_interrupt(c->interrupt_callback)) @@ -1304,48 +1404,26 @@ reload: goto reload; } - seg = current_segment(v); - - /* load/update Media Initialization Section, if any */ - ret = update_init_section(v, seg); - if (ret) - return ret; - - ret = open_input(c, v, seg); - if (ret < 0) { - if (ff_check_interrupt(c->interrupt_callback)) - return AVERROR_EXIT; - av_log(v->parent, AV_LOG_WARNING, "Failed to open segment of playlist %d\n", - v->index); - v->cur_seq_no += 1; - goto reload; + if (!v->ctx && v->is_subtitle) { + ret = init_subtitle_context(v); + if (ret < 0) + return ret; } - just_opened = 1; } - if (v->init_sec_buf_read_offset < v->init_sec_data_len) { - /* Push init section out first before first actual segment */ - int copy_size = FFMIN(v->init_sec_data_len - v->init_sec_buf_read_offset, buf_size); - memcpy(buf, v->init_sec_buf, copy_size); - v->init_sec_buf_read_offset += copy_size; - return copy_size; - } - - ret = read_from_url(v, current_segment(v), buf, buf_size, READ_NORMAL); - if (ret > 0) { - if (just_opened && v->is_id3_timestamped != 0) { - /* Intercept ID3 tags here, elementary audio streams are required - * to convey timestamps using them in the beginning of each segment. */ - intercept_id3(v, buf, buf_size, &ret); - } - + /* EOF flag may get set when the end of a segment is reached */ + v->pb.eof_reached = 0; + ret = av_read_frame(v->ctx, &v->pkt); + if (!ret) { return ret; } ff_format_io_close(v->parent, &v->input); v->cur_seq_no++; - c->cur_seq_no = v->cur_seq_no; + if (v->is_subtitle) + avformat_close_input(&v->ctx); + goto restart; } @@ -1492,16 +1570,6 @@ static int save_avio_options(AVFormatContext *s) return ret; } -static int nested_io_open(AVFormatContext *s, AVIOContext **pb, const char *url, - int flags, AVDictionary **opts) -{ - av_log(s, AV_LOG_ERROR, - "A HLS playlist item '%s' referred to an external file '%s'. " - "Opening this file was forbidden for security reasons\n", - s->filename, url); - return AVERROR(EPERM); -} - static void add_stream_to_programs(AVFormatContext *s, struct playlist *pls, AVStream *stream) { HLSContext *c = s->priv_data; @@ -1602,6 +1670,7 @@ static int hls_close(AVFormatContext *s) free_rendition_list(c); av_dict_free(&c->avio_opts); + av_dict_free(&c->demuxer_opts); return 0; } @@ -1708,10 +1777,13 @@ static int hls_read_header(AVFormatContext *s) highest_cur_seq_no = FFMAX(highest_cur_seq_no, pls->cur_seq_no); } + av_dict_set(&c->demuxer_opts, "prefer_hls_mpegts_pts", "1", 0); + /* Open the demuxer for each playlist */ for (i = 0; i < c->n_playlists; i++) { struct playlist *pls = c->playlists[i]; AVInputFormat *in_fmt = NULL; + AVDictionary *opts = NULL; if (!(pls->ctx = avformat_alloc_context())) { ret = AVERROR(ENOMEM); @@ -1765,7 +1837,9 @@ static int hls_read_header(AVFormatContext *s) if ((ret = ff_copy_whiteblacklists(pls->ctx, s)) < 0) goto fail; - ret = avformat_open_input(&pls->ctx, pls->segments[0]->url, in_fmt, NULL); + av_dict_copy(&opts, c->demuxer_opts, 0); + ret = avformat_open_input(&pls->ctx, pls->segments[0]->url, in_fmt, &opts); + av_dict_free(&opts); if (ret < 0) goto fail; @@ -1843,6 +1917,8 @@ static int recheck_discard_flags(AVFormatContext *s, int first) } else if (first && !pls->cur_needed && pls->needed) { if (pls->input) ff_format_io_close(pls->parent, &pls->input); + if (pls->is_subtitle) + avformat_close_input(&pls->ctx); pls->needed = 0; changed = 1; av_log(s, AV_LOG_INFO, "No longer receiving playlist %d\n", i); @@ -1909,7 +1985,7 @@ static int hls_read_packet(AVFormatContext *s, AVPacket *pkt) while (1) { int64_t ts_diff; AVRational tb; - ret = av_read_frame(pls->ctx, &pls->pkt); + ret = read_packet(pls, &pls->pkt); if (ret < 0) { if (!avio_feof(&pls->pb) && ret != AVERROR_EOF) return ret; @@ -2085,7 +2161,11 @@ static int hls_read_seek(AVFormatContext *s, int stream_index, /* Reset the pos, to let the mpegts demuxer know we've seeked. */ pls->pb.pos = 0; /* Flush the packet queue of the subdemuxer. */ - ff_read_frame_flush(pls->ctx); + if (pls->ctx) + ff_read_frame_flush(pls->ctx); + + if (pls->is_subtitle) + avformat_close_input(&pls->ctx); pls->seek_timestamp = seek_timestamp; pls->seek_flags = flags;