From patchwork Mon May 14 22:35:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Steinar H. Gunderson" X-Patchwork-Id: 8974 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:155:0:0:0:0:0 with SMTP id c82-v6csp1072580jad; Tue, 15 May 2018 00:21:03 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpHUsqAl6ZjcGepk+ZDwXscbADFZU5JZpM1jy/S1CIc2UoaGDl3/5MYxAdhq5jkEnNq/jD3 X-Received: by 2002:adf:8248:: with SMTP id 66-v6mr9790232wrb.127.1526368863661; Tue, 15 May 2018 00:21:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526368863; cv=none; d=google.com; s=arc-20160816; b=j7lnnQpVI8CB+2lxG3KCS+5SiAK0j91oLkxG6wZArkqoW90ZRQ83EQPxaETAfkgD7F 5/5hsI/KEjJf7t8w67K7C0GkwokpX6SMTLWjC1QnuZFFvcJKeHUWi2PpxgTej9UyOxE9 xkCwbaMdc1yEMGHurv09WSjOb6XuE9ffde1ljCNs8RVBNfygorl9IxxPaMwPIYgSXFJf VZIlpUG8WMi5qiq3BA6v7PAk9J2mpiiCACF8rlsl/drFqsyVs5MDEuXrXToyNbhneWhr NenEhhq5GmrIvicsLotzcRuQWjAQaPjACttxv60qqJ9lSqjY+KsjcXEealoBIxhVk9+G b9xQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:to:date:from:delivered-to :arc-authentication-results; bh=wIDBamLSYOq17GGm28Np07yw+5oGdoIkJwVJapudbbE=; b=UFUnlLieC0dBfV1oPeNyw/gJ2czyfzMAqHvZWbcwDyJyIUmik20+he0/WnlhqnsHpz nTdgwe6V3GIF0WSDoVqy4eHoKTnEg+HZE1Mq3s40xPxfDIsLDPThNXHhJoTF2v6r9NTV 0Kxi88FQCVbtVfv84Z3xGcJ3bb3mJT8Ls2qUNXJJH0oBgaudGE6knue+sqJiWBiCHoWn tX86xx+CHXUOOP/KqENi+sis/ix0UUFWc1upcZhCSRkTeSdedKohreYhLXY7AEfeEwae ZONyMVvz2oFYlW+IBXQLVj868We0Oc3srEDVXP4l4QfNJPyHk8L4CMOgzU8zUC68+U/D pnEA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q48-v6si9620567wrb.209.2018.05.15.00.21.02; Tue, 15 May 2018 00:21:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 165C0680643; Tue, 15 May 2018 10:20:23 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from cassarossa.samfundet.no (cassarossa.samfundet.no [193.35.52.29]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F17DD6802CF for ; Tue, 15 May 2018 10:20:16 +0300 (EEST) Received: from pannekake.samfundet.no ([2001:67c:29f4::50]) by cassarossa.samfundet.no with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1fIUGT-0006Py-Ue for ffmpeg-devel@ffmpeg.org; Tue, 15 May 2018 09:20:54 +0200 Received: from sesse by pannekake.samfundet.no with local (Exim 4.89) (envelope-from ) id 1fIUGT-0003tH-Ql for ffmpeg-devel@ffmpeg.org; Tue, 15 May 2018 09:20:53 +0200 From: "Steinar H. Gunderson" Date: Tue, 15 May 2018 00:35:45 +0200 To: ffmpeg-devel@ffmpeg.org Message-Id: Subject: [FFmpeg-devel] [PATCH] [WIP] libopusdec: Enable FEC/PLC X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Whenever we detect a discontinuity in the incoming stream, ask libopus to make up intermediate frames based on the first one we actually have after the discontinuity. If the stream contains FEC data (basically a low-quality side stream that is delayed by one packet), libopus will decode that. If not, it will activate PLC (packet loss concealment), which tries to synthesize some reasonable-sounding frame based on the previous audio. It will usually be audible, but it's much better than just playing silence. Do note that libopus 1.2.1 has a bug that affects PLC for CELT streams, so you probably want to use Opus from git if you want to test this. This is a work in progress; in particular, I'm unsure about: - Are the samples_to_timebase()/timebase_to_samples() functions correct? I've seen avc->pkt_timebase be 0/1 in certain situations, which indicates it isn't. - Is pts discontinuity the right way of knowing whether packets were lost, or can the RTP demuxer signal this somehow? What if the timebase conversion is inexact; could we get false positives? - Do we need to worry about pkt->pts == AV_NOPTS_VALUE, or can I delete the tests in question? Signed-off-by: Steinar H. Gunderson --- libavcodec/libopusdec.c | 86 +++++++++++++++++++++++++++++++++++------ 1 file changed, 75 insertions(+), 11 deletions(-) diff --git a/libavcodec/libopusdec.c b/libavcodec/libopusdec.c index 2a97811d18..40ee7b8fec 100644 --- a/libavcodec/libopusdec.c +++ b/libavcodec/libopusdec.c @@ -43,8 +43,21 @@ struct libopus_context { #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST int apply_phase_inv; #endif + int64_t expected_next_pts; }; +const AVRational opus_timebase = { 1, 48000 }; + +static int samples_to_timebase(const AVCodecContext *avc, int nb_samples) +{ + return av_rescale_q(nb_samples, avc->pkt_timebase, opus_timebase); +} + +static int timebase_to_samples(const AVCodecContext *avc, int64_t pts) +{ + return av_rescale_q(pts, opus_timebase, avc->pkt_timebase); +} + #define OPUS_HEAD_SIZE 19 static av_cold int libopus_decode_init(AVCodecContext *avc) @@ -153,6 +166,8 @@ static av_cold int libopus_decode_init(AVCodecContext *avc) /* Decoder delay (in samples) at 48kHz */ avc->delay = avc->internal->skip_samples = opus->pre_skip; + opus->expected_next_pts = AV_NOPTS_VALUE; + return 0; } @@ -174,25 +189,74 @@ static int libopus_decode(AVCodecContext *avc, void *data, { struct libopus_context *opus = avc->priv_data; AVFrame *frame = data; - int ret, nb_samples; + uint8_t *outptr; + int ret, nb_samples = 0, nb_lost_samples = 0, nb_samples_left; + + if (opus->expected_next_pts != AV_NOPTS_VALUE && + pkt->pts != AV_NOPTS_VALUE && + pkt->pts != opus->expected_next_pts) { + // Cap at recovering 120 ms of lost audio. + nb_lost_samples = timebase_to_samples(avc, pkt->pts - opus->expected_next_pts); + nb_lost_samples = FFMIN(nb_lost_samples, MAX_FRAME_SIZE); + } - frame->nb_samples = MAX_FRAME_SIZE; + frame->nb_samples = MAX_FRAME_SIZE + nb_lost_samples; if ((ret = ff_get_buffer(avc, frame, 0)) < 0) return ret; + outptr = frame->data[0]; + nb_samples_left = frame->nb_samples; + + if (nb_lost_samples) { + // Try to recover the lost samples with FEC data from this one. + // If there's no FEC data, the decoder will do loss concealment instead. + if (avc->sample_fmt == AV_SAMPLE_FMT_S16) + nb_samples = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_lost_samples, 1); + else + nb_samples = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_lost_samples, 1); + + if (nb_samples < 0) { + av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", + opus_strerror(nb_samples)); + return ff_opus_error_to_averror(nb_samples); + } + + av_log(avc, AV_LOG_INFO, "Recovered %d samples\n", nb_samples); + + outptr += nb_samples * avc->channels * av_get_bytes_per_sample(avc->sample_fmt); + nb_samples_left -= nb_samples; + if (pkt->pts != AV_NOPTS_VALUE) { + pkt->pts -= samples_to_timebase(avc, nb_samples); + frame->pts = pkt->pts; + } + } + + // Decode the actual, non-lost data. if (avc->sample_fmt == AV_SAMPLE_FMT_S16) - nb_samples = opus_multistream_decode(opus->dec, pkt->data, pkt->size, - (opus_int16 *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_samples_left, 0); else - nb_samples = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, - (float *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_samples_left, 0); - if (nb_samples < 0) { + if (ret < 0) { av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", - opus_strerror(nb_samples)); - return ff_opus_error_to_averror(nb_samples); + opus_strerror(ret)); + return ff_opus_error_to_averror(ret); + } + + nb_samples += ret; + + if (pkt->pts == AV_NOPTS_VALUE) { + opus->expected_next_pts = AV_NOPTS_VALUE; + } else { + opus->expected_next_pts = pkt->pts + samples_to_timebase(avc, nb_samples); } #ifndef OPUS_SET_GAIN