From patchwork Sun Sep 13 21:33:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Jan_Ekstr=C3=B6m?= X-Patchwork-Id: 22343 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 929D1449D85 for ; Mon, 14 Sep 2020 00:33:23 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 500D168BBD6; Mon, 14 Sep 2020 00:33:23 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f66.google.com (mail-lf1-f66.google.com [209.85.167.66]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5094968BAEB for ; Mon, 14 Sep 2020 00:33:16 +0300 (EEST) Received: by mail-lf1-f66.google.com with SMTP id z19so11288087lfr.4 for ; Sun, 13 Sep 2020 14:33:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ln5BRzbNAWoG2ZlY7RqPzmEv95xOq51lDRlksf9p/4M=; b=KGxb3p/gob5aO9TrPq2KQdOU/oviCxvy7zFn4fOwENDWZNo1TGMci5ttN0MgJaaaER 6Mt3n0uVctig6HrcFk6OUVYrj09wgWfoLG6g+1Peya8hKXtLZOtHQ5p1J9LqZA6tGM4/ gPfo1RB3vBAlrYIgE+agIRpeSsczvu9TFEAXgkx60LA515p3XQ0VvbIPsxugw8Cn8mjM Wia67lS+iUHkwjpcT78slohB6quLfL4SxMN9LUlfbCZ+clOYXGL6ite1grrREEShuyjl PMU47FEvNNHQZCN9Fgqe8RSzqVnT7ux/DMKHcUIffdwe/g2QeSjWpUPYsDSx8pz92ZIN QGlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ln5BRzbNAWoG2ZlY7RqPzmEv95xOq51lDRlksf9p/4M=; b=T58frJXz22z/UVqMpQsVR7yh62FUxd+/kgbj7K35skbawv0tuf5MtRaDnMawjo56Zl oMgMtGpB3aEHResU1rPkdG2+stLPKhbeareJd7EQaS2ZD3OLXf95ZkDKVatAL3SwX3BL FG5WD5sx2wvFcxDazk6bVne/sPWfe2gTGl0xU1v5qcaWcet+nY7iZYyqPznjdti75J1/ 7Y5/mVIbgmtHEcIpHhCt0gCbjPodXpedE2CGOVg/lm0m2ZhHvUCa8bVaKcAfDorIW0lK CY7ADENbOzaIcTjgEHBJpa3627/9rzoeJgeEwVH2xH2wTxR95QBk8U5YIaoPPUyN4WTP OfEg== X-Gm-Message-State: AOAM532J7Muq7zmNEW9uxfihS3q8vj8pAGw2aOQjRKBgORhUBUONirPg JKYz2wH4aYc7viZxpNkDxJg/1onQlII= X-Google-Smtp-Source: ABdhPJzuQJCt4Jj8CkD4YK6KKERa7fBexJe7hekAPwavtggnVsuwNZcFLeFOQgCbKnmPKaIYpYcN0A== X-Received: by 2002:a19:b8a:: with SMTP id 132mr2992225lfl.249.1600032795314; Sun, 13 Sep 2020 14:33:15 -0700 (PDT) Received: from localhost.localdomain (91-159-194-103.elisa-laajakaista.fi. [91.159.194.103]) by smtp.gmail.com with ESMTPSA id o27sm2772301lfb.306.2020.09.13.14.33.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Sep 2020 14:33:14 -0700 (PDT) From: =?utf-8?q?Jan_Ekstr=C3=B6m?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 14 Sep 2020 00:33:14 +0300 Message-Id: <20200913213314.247143-1-jeebjp@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200913102622.168011-4-jeebjp@gmail.com> References: <20200913102622.168011-4-jeebjp@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/4 v2] ffmpeg: move A/V non-streamcopy initialization to a later point X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" - For video, this means a single initialization point in do_video_out. - For audio we unfortunately need to do it in two places just before the buffer sink is utilized (if av_buffersink_get_samples would still work according to its specification after a call to avfilter_graph_request_oldest was made, we could at least remove the one in transcode_step). Other adjustments to make things work: - As the AVFrame PTS adjustment to encoder time base needs the encoder to be initialized, so it is now moved to do_{video,audio}_out, right after the encoder has been initialized. Due to this, the additional parameter in do_video_out is removed as it is no longer necessary. --- fftools/ffmpeg.c | 112 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 77 insertions(+), 35 deletions(-) diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c index 54802a8ec3..8658dc0ca4 100644 --- a/fftools/ffmpeg.c +++ b/fftools/ffmpeg.c @@ -938,6 +938,28 @@ early_exit: return float_pts; } +static int init_output_stream(OutputStream *ost, char *error, int error_len); + +static int init_output_stream_wrapper(OutputStream *ost, unsigned int fatal) +{ + int ret = AVERROR_BUG; + char error[1024] = {0}; + + if (ost->initialized) + return 0; + + ret = init_output_stream(ost, error, sizeof(error)); + if (ret < 0) { + av_log(NULL, AV_LOG_ERROR, "Error initializing output stream %d:%d -- %s\n", + ost->file_index, ost->index, error); + + if (fatal) + exit_program(1); + } + + return ret; +} + static void do_audio_out(OutputFile *of, OutputStream *ost, AVFrame *frame) { @@ -949,6 +971,8 @@ static void do_audio_out(OutputFile *of, OutputStream *ost, pkt.data = NULL; pkt.size = 0; + adjust_frame_pts_to_encoder_tb(of, ost, frame); + if (!check_recording_time(ost)) return; @@ -1083,8 +1107,7 @@ static void do_subtitle_out(OutputFile *of, static void do_video_out(OutputFile *of, OutputStream *ost, - AVFrame *next_picture, - double sync_ipts) + AVFrame *next_picture) { int ret, format_video_sync; AVPacket pkt; @@ -1094,10 +1117,14 @@ static void do_video_out(OutputFile *of, int nb_frames, nb0_frames, i; double delta, delta0; double duration = 0; + double sync_ipts = AV_NOPTS_VALUE; int frame_size = 0; InputStream *ist = NULL; AVFilterContext *filter = ost->filter->filter; + init_output_stream_wrapper(ost, 1); + sync_ipts = adjust_frame_pts_to_encoder_tb(of, ost, next_picture); + if (ost->source_index >= 0) ist = input_streams[ost->source_index]; @@ -1431,28 +1458,6 @@ static void do_video_stats(OutputStream *ost, int frame_size) } } -static int init_output_stream(OutputStream *ost, char *error, int error_len); - -static int init_output_stream_wrapper(OutputStream *ost, unsigned int fatal) -{ - int ret = AVERROR_BUG; - char error[1024] = {0}; - - if (ost->initialized) - return 0; - - ret = init_output_stream(ost, error, sizeof(error)); - if (ret < 0) { - av_log(NULL, AV_LOG_ERROR, "Error initializing output stream %d:%d -- %s\n", - ost->file_index, ost->index, error); - - if (fatal) - exit_program(1); - } - - return ret; -} - static void finish_output_stream(OutputStream *ost) { OutputFile *of = output_files[ost->file_index]; @@ -1489,7 +1494,17 @@ static int reap_filters(int flush) continue; filter = ost->filter->filter; - init_output_stream_wrapper(ost, 1); + /* + * Unlike video, with audio the audio frame size matters. + * Currently we are fully reliant on the lavfi filter chain to + * do the buffering deed for us, and thus the frame size parameter + * needs to be set accordingly. Where does one get the required + * frame size? From the initialized AVCodecContext of an audio + * encoder. Thus, if we have gotten to an audio stream, initialize + * the encoder earlier than receiving the first AVFrame. + */ + if (av_buffersink_get_type(filter) == AVMEDIA_TYPE_AUDIO) + init_output_stream_wrapper(ost, 1); if (!ost->filtered_frame && !(ost->filtered_frame = av_frame_alloc())) { return AVERROR(ENOMEM); @@ -1497,7 +1512,6 @@ static int reap_filters(int flush) filtered_frame = ost->filtered_frame; while (1) { - double float_pts = AV_NOPTS_VALUE; // this is identical to filtered_frame.pts but with higher precision ret = av_buffersink_get_frame_flags(filter, filtered_frame, AV_BUFFERSINK_FLAG_NO_REQUEST); if (ret < 0) { @@ -1506,7 +1520,7 @@ static int reap_filters(int flush) "Error in av_buffersink_get_frame_flags(): %s\n", av_err2str(ret)); } else if (flush && ret == AVERROR_EOF) { if (av_buffersink_get_type(filter) == AVMEDIA_TYPE_VIDEO) - do_video_out(of, ost, NULL, AV_NOPTS_VALUE); + do_video_out(of, ost, NULL); } break; } @@ -1515,15 +1529,12 @@ static int reap_filters(int flush) continue; } - float_pts = adjust_frame_pts_to_encoder_tb(of, ost, - filtered_frame); - switch (av_buffersink_get_type(filter)) { case AVMEDIA_TYPE_VIDEO: if (!ost->frame_aspect_ratio.num) enc->sample_aspect_ratio = filtered_frame->sample_aspect_ratio; - do_video_out(of, ost, filtered_frame, float_pts); + do_video_out(of, ost, filtered_frame); break; case AVMEDIA_TYPE_AUDIO: if (!(enc->codec->capabilities & AV_CODEC_CAP_PARAM_CHANGE) && @@ -3688,10 +3699,19 @@ static int transcode_init(void) goto dump_format; } - /* open each encoder */ + /* + * initialize stream copy and subtitle/data streams. + * Encoded AVFrame based streams will get initialized as follows: + * - when the first AVFrame is received in do_video_out + * - just before the first AVFrame is received in either transcode_step + * or reap_filters due to us requiring the filter chain buffer sink + * to be configured with the correct audio frame size, which is only + * known after the encoder is initialized. + */ for (i = 0; i < nb_output_streams; i++) { - // skip streams fed from filtergraphs until we have a frame for them - if (output_streams[i]->filter) + if (!output_streams[i]->stream_copy && + (output_streams[i]->enc_ctx->codec_type == AVMEDIA_TYPE_VIDEO || + output_streams[i]->enc_ctx->codec_type == AVMEDIA_TYPE_AUDIO)) continue; ret = init_output_stream_wrapper(output_streams[i], 0); @@ -4605,7 +4625,29 @@ static int transcode_step(void) } if (ost->filter && ost->filter->graph->graph) { - init_output_stream_wrapper(ost, 1); + /* + * Similar case to the early audio initialization in reap_filters. + * Audio is special in ffmpeg.c currently as we depend on lavfi's + * audio frame buffering/creation to get the output audio frame size + * in samples correct. The audio frame size for the filter chain is + * configured during the output stream initialization. + * + * Apparently avfilter_graph_request_oldest (called in + * transcode_from_filter just down the line) peeks. Peeking already + * puts one frame "ready to be given out", which means that any + * update in filter buffer sink configuration afterwards will not + * help us. And yes, even if it would be utilized, + * av_buffersink_get_samples is affected, as it internally utilizes + * the same early exit for peeked frames. + * + * In other words, if avfilter_graph_request_oldest would not make + * further filter chain configuration or usage of + * av_buffersink_get_samples useless (by just causing the return + * of the peeked AVFrame as-is), we could get rid of this additional + * early encoder initialization. + */ + if (av_buffersink_get_type(ost->filter->filter) == AVMEDIA_TYPE_AUDIO) + init_output_stream_wrapper(ost, 1); if ((ret = transcode_from_filter(ost->filter->graph, &ist)) < 0) return ret;