From patchwork Mon Jul 4 14:13:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philip-Dylan Gleonec X-Patchwork-Id: 36652 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp2860811pzh; Mon, 4 Jul 2022 07:14:52 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tbsxoh8QW8lcNqzDWglEEW4cR6O80NBGsIlirobfU8GLCQnKPE+KOVmgT1NETKgEFLiUB5 X-Received: by 2002:a17:907:1629:b0:72a:d6b6:8c96 with SMTP id hb41-20020a170907162900b0072ad6b68c96mr1578164ejc.423.1656944091975; Mon, 04 Jul 2022 07:14:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656944091; cv=none; d=google.com; s=arc-20160816; b=RBDE1pxGUKZ/byPpEVh8uuhOlMoD2NyUBmXBgVpUdHZISFKhkDmX+heSUIiosdem5E pZabquCasME+JN1SFhip5MReiqaAFTaiwtUxfCLHMdD1HmdTahkY0J2/PPFgN1wVgc5e 9W2okMJgDbfYAa3Qvk8gTkzCWA4aAwEtYQ8beoGno/FMcXRrNKDFaLJUiO4fPfKh8l1C BYFdWSAf10cwSJK9e7hvt2h7ZxJguKc2lL76izDk6g4mlttpJXbYztM5PLlY+2f/TeeZ 700Do9aICEiNW/t5qFkMsFwNJyDH9JPziaU9ye31dIlcooJs60aN4FRuraY1RPT3LNil y+wQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:dkim-filter:delivered-to; bh=3s7Zs+202UdM4yvWFcejwg92DUwhosokJWvxUDDhr/g=; b=vj+b1JwnkYIl6VaLRyBYsWWYSt/Jjvi8P7wsve+082mx1uS+Kscxe/iJ9EaRLEn42f F9y5j3LrV1oFk61R94HyFBhBYqKEeuubmNTq36S1akj7JcifVbmSlTiiEaHgiYn9ENlh XLkjDC6+j8MffcyysebhRIftlNNvq52StecnmgkGKDM0bzVNtlDyNgz7ZD/s9ImaMnu7 uaOdcT2MV8iVckY4qrw7dnrRhQ/lBoKVXv1RYSmj6Xqvv4F1jUDjXBmJs4irgY8H12sp NwoyUgIwvxjutwXjTu3CinsCTJnx1hy+Vk1aQ1rSalo97s9eyNtHJkil97HmIp5xuK9/ uQYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=HS5RL4+M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q16-20020a170906771000b007120b53e117si11014975ejm.26.2022.07.04.07.14.51; Mon, 04 Jul 2022 07:14:51 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=HS5RL4+M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1D12868B8E8; Mon, 4 Jul 2022 17:14:41 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A238A68B8BD for ; Mon, 4 Jul 2022 17:14:34 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id BC54F9C0356; Mon, 4 Jul 2022 10:14:33 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id L2GvH5vz6H7b; Mon, 4 Jul 2022 10:14:33 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id F1B659C0353; Mon, 4 Jul 2022 10:14:32 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com F1B659C0353 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1656944073; bh=k0xZvl5KbIQuX96/CztefOzwt/XF1tH9v3RvO9WJ00E=; h=From:To:Date:Message-Id:MIME-Version; b=HS5RL4+MgnoJOT3b3VxymyFLcLr3SmyPMk9oJW5y30UVGS5v1SYYWZhNn2X9J2GSX eibuKLpPK7pZgIBWR+v69O/a0tOVOKQh+4oAadzVE4rYLfTOHzLP0svX4BOfPkW1Pe rdxQq9e2QW1Kf6OacmTuxzwtcV/ynpnHoFdXurA88eb+FV/U/FpH9xyhyG7DgauBI9 y7ulHentEXr+sQNFKPqjgy5woOuoWTbgk0MVfGNRULPD3t2/Ob78KlOxvJ7bK6lHGw wD1f09dOo6E/XJ4wsOVqn8g2M3Nxu6n4VjvSoTA+A2tTew0U23IrjRNeUo/uwD4hMM UrWG6dXUY/tjQ== X-Virus-Scanned: amavisd-new at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 5yn0sxKnAP0n; Mon, 4 Jul 2022 10:14:32 -0400 (EDT) Received: from T14-AMD.biamp.com (lfbn-ren-1-676-174.w81-53.abo.wanadoo.fr [81.53.245.174]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 560189C034D; Mon, 4 Jul 2022 10:14:32 -0400 (EDT) From: Philip-Dylan Gleonec To: ffmpeg-devel@ffmpeg.org Date: Mon, 4 Jul 2022 16:13:40 +0200 Message-Id: <20220704141340.493931-3-philip-dylan.gleonec@savoirfairelinux.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220704141340.493931-1-philip-dylan.gleonec@savoirfairelinux.com> References: <20220617162150.GW396728@pb2> <20220704141340.493931-1-philip-dylan.gleonec@savoirfairelinux.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 2/2] avcodec/libopusdec: Enable FEC/PLC X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Philip-Dylan Gleonec , "Steinar H . Gunderson" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: x1cYkz7ETwwA Adds FEC/PLC support to libopus. The lost packets are detected as a discontinuity in the audio stream. When a discontinuity is used, this patch tries to decode the FEC data. If FEC data is present in the packet, it is decoded, otherwise audio is re-created through PLC. This patch is based on Steinar H. Gunderson contribution, and corrects the pts computation: all pts are expressed in samples instead of time. This patch also adds an option "decode_fec" which enables or disables FEC decoding. This option is disabled by default to keep consistent behaviour with former versions. A number of checks are made to ensure compatibility with different containers. Indeed, video containers seem to have a pts expressed in ms while it is expressed in samples for audio containers. It also manages the cases where pkt->duration is 0, in some RTP streams. This patch ignores data it can not reconstruct, i.e. packets received twice and packets with a length that is not a multiple of 2.5ms. Signed-off-by: Philip-Dylan Gleonec Co-developed-by: Steinar H. Gunderson --- libavcodec/libopusdec.c | 105 +++++++++++++++++++++++++++++++++++----- 1 file changed, 94 insertions(+), 11 deletions(-) diff --git a/libavcodec/libopusdec.c b/libavcodec/libopusdec.c index 316ab0f2a7..f5d0e95fc8 100644 --- a/libavcodec/libopusdec.c +++ b/libavcodec/libopusdec.c @@ -44,10 +44,15 @@ struct libopus_context { #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST int apply_phase_inv; #endif + int decode_fec; + int64_t expected_next_pts; }; #define OPUS_HEAD_SIZE 19 +// Sample rate is constant as libopus always output at 48kHz +static const AVRational opus_timebase = { 1, 48000 }; + static av_cold int libopus_decode_init(AVCodecContext *avc) { struct libopus_context *opus = avc->priv_data; @@ -140,6 +145,8 @@ static av_cold int libopus_decode_init(AVCodecContext *avc) /* Decoder delay (in samples) at 48kHz */ avc->delay = avc->internal->skip_samples = opus->pre_skip; + opus->expected_next_pts = AV_NOPTS_VALUE; + return 0; } @@ -160,25 +167,100 @@ static int libopus_decode(AVCodecContext *avc, AVFrame *frame, int *got_frame_ptr, AVPacket *pkt) { struct libopus_context *opus = avc->priv_data; - int ret, nb_samples; + uint8_t *outptr; + int ret, nb_samples = 0, nb_lost_samples = 0, nb_samples_left; + + // If FEC is enabled, calculate number of lost samples + if (opus->decode_fec && + opus->expected_next_pts != AV_NOPTS_VALUE && + pkt->pts != AV_NOPTS_VALUE && + pkt->pts != opus->expected_next_pts) { + // Cap at recovering 120 ms of lost audio. + nb_lost_samples = pkt->pts - opus->expected_next_pts; + nb_lost_samples = FFMIN(nb_lost_samples, MAX_FRAME_SIZE); + // pts is expressed in ms for some containers (e.g. mkv) + // FEC only works for SILK frames (> 10ms) + // Detect if nb_lost_samples is in ms, and convert in samples if it is + if (nb_lost_samples > 0) { + if (avc->pkt_timebase.den != 48000) { + nb_lost_samples = av_rescale_q(nb_lost_samples, avc->pkt_timebase, opus_timebase); + } + // For FEC/PLC, frame_size has to be to have a multiple of 2.5 ms + if (nb_lost_samples % (5LL * opus_timebase.den / 2000)) { + nb_lost_samples -= nb_lost_samples % (5LL * opus_timebase.den / 2000); + } + } + } - frame->nb_samples = MAX_FRAME_SIZE; + frame->nb_samples = MAX_FRAME_SIZE + nb_lost_samples; if ((ret = ff_get_buffer(avc, frame, 0)) < 0) return ret; + outptr = frame->data[0]; + nb_samples_left = frame->nb_samples; + + if (opus->decode_fec && nb_lost_samples > 0) { + // Try to recover the lost samples with FEC data from this one. + // If there's no FEC data, the decoder will do loss concealment instead. + if (avc->sample_fmt == AV_SAMPLE_FMT_S16) + ret = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_lost_samples, 1); + else + ret = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_lost_samples, 1); + + if (ret < 0) { + if (opus->decode_fec) opus->expected_next_pts = pkt->pts + pkt->duration; + av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", + opus_strerror(ret)); + return ff_opus_error_to_averror(ret); + } + + av_log(avc, AV_LOG_WARNING, "Recovered %d samples with FEC/PLC\n", + ret); + + outptr += ret * avc->ch_layout.nb_channels * av_get_bytes_per_sample(avc->sample_fmt); + nb_samples_left -= ret; + nb_samples += ret; + if (pkt->pts != AV_NOPTS_VALUE) { + frame->pts = pkt->pts - ret; + } + } + + // Decode the actual, non-lost data. if (avc->sample_fmt == AV_SAMPLE_FMT_S16) - nb_samples = opus_multistream_decode(opus->dec, pkt->data, pkt->size, - (opus_int16 *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_samples_left, 0); else - nb_samples = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, - (float *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_samples_left, 0); - if (nb_samples < 0) { + if (ret < 0) { + if (opus->decode_fec) opus->expected_next_pts = pkt->pts + pkt->duration; av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", - opus_strerror(nb_samples)); - return ff_opus_error_to_averror(nb_samples); + opus_strerror(ret)); + return ff_opus_error_to_averror(ret); + } + nb_samples += ret; + + if (opus->decode_fec) + { + // Calculate the next expected pts + if (pkt->pts == AV_NOPTS_VALUE) { + opus->expected_next_pts = AV_NOPTS_VALUE; + } else { + if (pkt->duration) { + opus->expected_next_pts = pkt->pts + pkt->duration; + } else if (avc->pkt_timebase.num) { + opus->expected_next_pts = pkt->pts + av_rescale_q(ret, opus_timebase, avc->pkt_timebase); + } else { + opus->expected_next_pts = pkt->pts + ret; + } + } } #ifndef OPUS_SET_GAIN @@ -219,6 +301,7 @@ static const AVOption libopusdec_options[] = { #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST { "apply_phase_inv", "Apply intensity stereo phase inversion", OFFSET(apply_phase_inv), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, #endif + { "decode_fec", "Decode FEC data or use PLC", OFFSET(decode_fec), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { NULL }, };