From patchwork Mon Jul 4 14:13:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philip-Dylan Gleonec X-Patchwork-Id: 36651 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp2860735pzh; Mon, 4 Jul 2022 07:14:42 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tElxvvDGYgaY6Lj3OP1seZeQ2rlKW8OHqvr8Tup0y5lUz+05fHuNSjbhu1nDVROUhlemi2 X-Received: by 2002:a05:6402:158b:b0:43a:6cae:a029 with SMTP id c11-20020a056402158b00b0043a6caea029mr3999792edv.201.1656944082492; Mon, 04 Jul 2022 07:14:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656944082; cv=none; d=google.com; s=arc-20160816; b=SrXTKz2gbMoEyYGL1Tx2hP4/wnysWTeIkHDzLe8EpEWGcH0sDx+wq911aXckRcLWNO fNXHqIv5gT+UE1KM93x/lZcU8X+2uc4RgoiDDKO7FHHCN2h0qYhHpMj/L0IhOgM4j2X5 rLNMt4XOPaMcS30iRLJy3dpoAG61KedmQZ3gOAxSBfikeIHVMLQ19o6QMf+q17betG4W vIwJWoU7NhsoBFedpB1ejkyfMvEsGQMRfYzLLL2atduL3v2n/7kJUebBqQYy/YVSj8No eWvkV7oXNhpUveAl+CkMaKgfb/XsBWqgRq8AAXyWUijILzhLQDIDwdsa8ElnRQaTChCz 0WpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:dkim-filter:delivered-to; bh=pG3oEWrvqDVG8YPZFclzXqrOAA3OG4rxuMDwQndPrFg=; b=lOU737bk232rhyr3Bp10AzRCc0fwTCR+Qsvlsr9bBMlQFMJxm7FwAS+0my5u5npRR4 0zazxfTuiTJBdoyQNiDU3hATABS27bFw1Fv2ZYW/lDz9kjOIE7mSS5qBB6hCQmPQmlMU QmlBgUI/yFc4Fdh2AkPC/QEq/uWDz0Mh9ukBB4fYzegXe6ql+IVYit11TXixrx1s0oLV D0IiPt929GKrTEgJm03Wf0PkchJNNcb2M6lwd8/gpFHvwQNInaCLpjgbyOCjUJWzn4uU NDwbGRA1CAuUsVjW+2l8tCAp6DV832RJxUB+bRcacrdLnpIZ74e4SezR+8vCSzosqSCC fDSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=E9CzSVrt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id mf21-20020a1709071a5500b0072ab8afaac1si5114843ejc.508.2022.07.04.07.14.42; Mon, 04 Jul 2022 07:14:42 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=E9CzSVrt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EFD4868B8C6; Mon, 4 Jul 2022 17:14:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E2CC268B8B4 for ; Mon, 4 Jul 2022 17:14:31 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 0BAF49C033B for ; Mon, 4 Jul 2022 10:14:31 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 4IGSWmuQEnRe; Mon, 4 Jul 2022 10:14:30 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 879C39C034D; Mon, 4 Jul 2022 10:14:30 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 879C39C034D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1656944070; bh=ujiLSV5QIs8OZrOkEJFmZhXiwgeD2TaCTo7pjbQDhOk=; h=From:To:Date:Message-Id:MIME-Version; b=E9CzSVrtaa2nFGqlMXhEPdj8cAebVoxOEiURnnbAdT9FvSpxdRT8f2Lw3TV0iNqpP P9Emmabm3WfqTPgaKooTOwPNSJXzza26IfYjQs8Q5/R+AN0+9W8tbcZBCm5oNy88+M sT+2iezLCkoS8HXnUlexbtYEfvPFrRo5pab3XTALeyMWZ9pYaBaOeB/wO0yEvvLHcz FCjnysuzpuDvs4HiocRpc46I10Dam/icZV9An1vImv/O3ymWnAoazarEDvba/hYOyh 5QF15QzMF8C0N60y6gjcHfQa/2ofshDCaCzO6wnKj3SdYN8HzWl2LGdVR59DhIkka0 ObKF4/QVX06ew== X-Virus-Scanned: amavisd-new at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 2TTyAeB5_Kyk; Mon, 4 Jul 2022 10:14:30 -0400 (EDT) Received: from T14-AMD.biamp.com (lfbn-ren-1-676-174.w81-53.abo.wanadoo.fr [81.53.245.174]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 0399A9C033B; Mon, 4 Jul 2022 10:14:29 -0400 (EDT) From: Philip-Dylan Gleonec To: ffmpeg-devel@ffmpeg.org Date: Mon, 4 Jul 2022 16:13:39 +0200 Message-Id: <20220704141340.493931-2-philip-dylan.gleonec@savoirfairelinux.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220704141340.493931-1-philip-dylan.gleonec@savoirfairelinux.com> References: <20220617162150.GW396728@pb2> <20220704141340.493931-1-philip-dylan.gleonec@savoirfairelinux.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 1/2] avcodec/libopusenc: reload packet loss at encode X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Philip-Dylan Gleonec Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: pDQbvrwsARlA An estimation of packet loss is required by libopus to compute its FEC data. Currently, this estimation is constant, and can not be changed after configuration. This means an application using libopus through ffmpeg can not adapt the packet loss estimation when the network quality degrades. This patch makes the encoder reload the packet_loss AVOption before encoding samples, if fec is enabled and the packet loss estimation set is different than the current one. This way an application can modify the packet loss estimation by changing the AVOption. Typical use-case is a RTP stream, where packet loss can be estimated from RTCP packets. Signed-off-by: Philip-Dylan Gleonec --- libavcodec/libopusenc.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/libavcodec/libopusenc.c b/libavcodec/libopusenc.c index c884075ffe..26d2082ffa 100644 --- a/libavcodec/libopusenc.c +++ b/libavcodec/libopusenc.c @@ -462,6 +462,23 @@ static int libopus_encode(AVCodecContext *avctx, AVPacket *avpkt, uint8_t *audio; int ret; int discard_padding; + int32_t opus_packet_loss = 0; + + ret = opus_multistream_encoder_ctl(opus->enc, + OPUS_GET_PACKET_LOSS_PERC(&opus_packet_loss)); + if (ret != OPUS_OK) + av_log(avctx, AV_LOG_WARNING, + "Unable to get expected packet loss percentage: %s\n", + opus_strerror(ret)); + + if (opus->opts.fec && (opus_packet_loss != opus->opts.packet_loss)) { + ret = opus_multistream_encoder_ctl(opus->enc, + OPUS_SET_PACKET_LOSS_PERC(opus->opts.packet_loss)); + if (ret != OPUS_OK) + av_log(avctx, AV_LOG_WARNING, + "Unable to set expected packet loss percentage: %s\n", + opus_strerror(ret)); + } if (frame) { ret = ff_af_queue_add(&opus->afq, frame); From patchwork Mon Jul 4 14:13:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philip-Dylan Gleonec X-Patchwork-Id: 36652 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp2860811pzh; Mon, 4 Jul 2022 07:14:52 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tbsxoh8QW8lcNqzDWglEEW4cR6O80NBGsIlirobfU8GLCQnKPE+KOVmgT1NETKgEFLiUB5 X-Received: by 2002:a17:907:1629:b0:72a:d6b6:8c96 with SMTP id hb41-20020a170907162900b0072ad6b68c96mr1578164ejc.423.1656944091975; Mon, 04 Jul 2022 07:14:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656944091; cv=none; d=google.com; s=arc-20160816; b=RBDE1pxGUKZ/byPpEVh8uuhOlMoD2NyUBmXBgVpUdHZISFKhkDmX+heSUIiosdem5E pZabquCasME+JN1SFhip5MReiqaAFTaiwtUxfCLHMdD1HmdTahkY0J2/PPFgN1wVgc5e 9W2okMJgDbfYAa3Qvk8gTkzCWA4aAwEtYQ8beoGno/FMcXRrNKDFaLJUiO4fPfKh8l1C BYFdWSAf10cwSJK9e7hvt2h7ZxJguKc2lL76izDk6g4mlttpJXbYztM5PLlY+2f/TeeZ 700Do9aICEiNW/t5qFkMsFwNJyDH9JPziaU9ye31dIlcooJs60aN4FRuraY1RPT3LNil y+wQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:dkim-filter:delivered-to; bh=3s7Zs+202UdM4yvWFcejwg92DUwhosokJWvxUDDhr/g=; b=vj+b1JwnkYIl6VaLRyBYsWWYSt/Jjvi8P7wsve+082mx1uS+Kscxe/iJ9EaRLEn42f F9y5j3LrV1oFk61R94HyFBhBYqKEeuubmNTq36S1akj7JcifVbmSlTiiEaHgiYn9ENlh XLkjDC6+j8MffcyysebhRIftlNNvq52StecnmgkGKDM0bzVNtlDyNgz7ZD/s9ImaMnu7 uaOdcT2MV8iVckY4qrw7dnrRhQ/lBoKVXv1RYSmj6Xqvv4F1jUDjXBmJs4irgY8H12sp NwoyUgIwvxjutwXjTu3CinsCTJnx1hy+Vk1aQ1rSalo97s9eyNtHJkil97HmIp5xuK9/ uQYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=HS5RL4+M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q16-20020a170906771000b007120b53e117si11014975ejm.26.2022.07.04.07.14.51; Mon, 04 Jul 2022 07:14:51 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=HS5RL4+M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1D12868B8E8; Mon, 4 Jul 2022 17:14:41 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A238A68B8BD for ; Mon, 4 Jul 2022 17:14:34 +0300 (EEST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id BC54F9C0356; Mon, 4 Jul 2022 10:14:33 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id L2GvH5vz6H7b; Mon, 4 Jul 2022 10:14:33 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id F1B659C0353; Mon, 4 Jul 2022 10:14:32 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com F1B659C0353 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1656944073; bh=k0xZvl5KbIQuX96/CztefOzwt/XF1tH9v3RvO9WJ00E=; h=From:To:Date:Message-Id:MIME-Version; b=HS5RL4+MgnoJOT3b3VxymyFLcLr3SmyPMk9oJW5y30UVGS5v1SYYWZhNn2X9J2GSX eibuKLpPK7pZgIBWR+v69O/a0tOVOKQh+4oAadzVE4rYLfTOHzLP0svX4BOfPkW1Pe rdxQq9e2QW1Kf6OacmTuxzwtcV/ynpnHoFdXurA88eb+FV/U/FpH9xyhyG7DgauBI9 y7ulHentEXr+sQNFKPqjgy5woOuoWTbgk0MVfGNRULPD3t2/Ob78KlOxvJ7bK6lHGw wD1f09dOo6E/XJ4wsOVqn8g2M3Nxu6n4VjvSoTA+A2tTew0U23IrjRNeUo/uwD4hMM UrWG6dXUY/tjQ== X-Virus-Scanned: amavisd-new at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 5yn0sxKnAP0n; Mon, 4 Jul 2022 10:14:32 -0400 (EDT) Received: from T14-AMD.biamp.com (lfbn-ren-1-676-174.w81-53.abo.wanadoo.fr [81.53.245.174]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 560189C034D; Mon, 4 Jul 2022 10:14:32 -0400 (EDT) From: Philip-Dylan Gleonec To: ffmpeg-devel@ffmpeg.org Date: Mon, 4 Jul 2022 16:13:40 +0200 Message-Id: <20220704141340.493931-3-philip-dylan.gleonec@savoirfairelinux.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220704141340.493931-1-philip-dylan.gleonec@savoirfairelinux.com> References: <20220617162150.GW396728@pb2> <20220704141340.493931-1-philip-dylan.gleonec@savoirfairelinux.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 2/2] avcodec/libopusdec: Enable FEC/PLC X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Philip-Dylan Gleonec , "Steinar H . Gunderson" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: x1cYkz7ETwwA Adds FEC/PLC support to libopus. The lost packets are detected as a discontinuity in the audio stream. When a discontinuity is used, this patch tries to decode the FEC data. If FEC data is present in the packet, it is decoded, otherwise audio is re-created through PLC. This patch is based on Steinar H. Gunderson contribution, and corrects the pts computation: all pts are expressed in samples instead of time. This patch also adds an option "decode_fec" which enables or disables FEC decoding. This option is disabled by default to keep consistent behaviour with former versions. A number of checks are made to ensure compatibility with different containers. Indeed, video containers seem to have a pts expressed in ms while it is expressed in samples for audio containers. It also manages the cases where pkt->duration is 0, in some RTP streams. This patch ignores data it can not reconstruct, i.e. packets received twice and packets with a length that is not a multiple of 2.5ms. Signed-off-by: Philip-Dylan Gleonec Co-developed-by: Steinar H. Gunderson --- libavcodec/libopusdec.c | 105 +++++++++++++++++++++++++++++++++++----- 1 file changed, 94 insertions(+), 11 deletions(-) diff --git a/libavcodec/libopusdec.c b/libavcodec/libopusdec.c index 316ab0f2a7..f5d0e95fc8 100644 --- a/libavcodec/libopusdec.c +++ b/libavcodec/libopusdec.c @@ -44,10 +44,15 @@ struct libopus_context { #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST int apply_phase_inv; #endif + int decode_fec; + int64_t expected_next_pts; }; #define OPUS_HEAD_SIZE 19 +// Sample rate is constant as libopus always output at 48kHz +static const AVRational opus_timebase = { 1, 48000 }; + static av_cold int libopus_decode_init(AVCodecContext *avc) { struct libopus_context *opus = avc->priv_data; @@ -140,6 +145,8 @@ static av_cold int libopus_decode_init(AVCodecContext *avc) /* Decoder delay (in samples) at 48kHz */ avc->delay = avc->internal->skip_samples = opus->pre_skip; + opus->expected_next_pts = AV_NOPTS_VALUE; + return 0; } @@ -160,25 +167,100 @@ static int libopus_decode(AVCodecContext *avc, AVFrame *frame, int *got_frame_ptr, AVPacket *pkt) { struct libopus_context *opus = avc->priv_data; - int ret, nb_samples; + uint8_t *outptr; + int ret, nb_samples = 0, nb_lost_samples = 0, nb_samples_left; + + // If FEC is enabled, calculate number of lost samples + if (opus->decode_fec && + opus->expected_next_pts != AV_NOPTS_VALUE && + pkt->pts != AV_NOPTS_VALUE && + pkt->pts != opus->expected_next_pts) { + // Cap at recovering 120 ms of lost audio. + nb_lost_samples = pkt->pts - opus->expected_next_pts; + nb_lost_samples = FFMIN(nb_lost_samples, MAX_FRAME_SIZE); + // pts is expressed in ms for some containers (e.g. mkv) + // FEC only works for SILK frames (> 10ms) + // Detect if nb_lost_samples is in ms, and convert in samples if it is + if (nb_lost_samples > 0) { + if (avc->pkt_timebase.den != 48000) { + nb_lost_samples = av_rescale_q(nb_lost_samples, avc->pkt_timebase, opus_timebase); + } + // For FEC/PLC, frame_size has to be to have a multiple of 2.5 ms + if (nb_lost_samples % (5LL * opus_timebase.den / 2000)) { + nb_lost_samples -= nb_lost_samples % (5LL * opus_timebase.den / 2000); + } + } + } - frame->nb_samples = MAX_FRAME_SIZE; + frame->nb_samples = MAX_FRAME_SIZE + nb_lost_samples; if ((ret = ff_get_buffer(avc, frame, 0)) < 0) return ret; + outptr = frame->data[0]; + nb_samples_left = frame->nb_samples; + + if (opus->decode_fec && nb_lost_samples > 0) { + // Try to recover the lost samples with FEC data from this one. + // If there's no FEC data, the decoder will do loss concealment instead. + if (avc->sample_fmt == AV_SAMPLE_FMT_S16) + ret = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_lost_samples, 1); + else + ret = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_lost_samples, 1); + + if (ret < 0) { + if (opus->decode_fec) opus->expected_next_pts = pkt->pts + pkt->duration; + av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", + opus_strerror(ret)); + return ff_opus_error_to_averror(ret); + } + + av_log(avc, AV_LOG_WARNING, "Recovered %d samples with FEC/PLC\n", + ret); + + outptr += ret * avc->ch_layout.nb_channels * av_get_bytes_per_sample(avc->sample_fmt); + nb_samples_left -= ret; + nb_samples += ret; + if (pkt->pts != AV_NOPTS_VALUE) { + frame->pts = pkt->pts - ret; + } + } + + // Decode the actual, non-lost data. if (avc->sample_fmt == AV_SAMPLE_FMT_S16) - nb_samples = opus_multistream_decode(opus->dec, pkt->data, pkt->size, - (opus_int16 *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_samples_left, 0); else - nb_samples = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, - (float *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_samples_left, 0); - if (nb_samples < 0) { + if (ret < 0) { + if (opus->decode_fec) opus->expected_next_pts = pkt->pts + pkt->duration; av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", - opus_strerror(nb_samples)); - return ff_opus_error_to_averror(nb_samples); + opus_strerror(ret)); + return ff_opus_error_to_averror(ret); + } + nb_samples += ret; + + if (opus->decode_fec) + { + // Calculate the next expected pts + if (pkt->pts == AV_NOPTS_VALUE) { + opus->expected_next_pts = AV_NOPTS_VALUE; + } else { + if (pkt->duration) { + opus->expected_next_pts = pkt->pts + pkt->duration; + } else if (avc->pkt_timebase.num) { + opus->expected_next_pts = pkt->pts + av_rescale_q(ret, opus_timebase, avc->pkt_timebase); + } else { + opus->expected_next_pts = pkt->pts + ret; + } + } } #ifndef OPUS_SET_GAIN @@ -219,6 +301,7 @@ static const AVOption libopusdec_options[] = { #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST { "apply_phase_inv", "Apply intensity stereo phase inversion", OFFSET(apply_phase_inv), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, #endif + { "decode_fec", "Decode FEC data or use PLC", OFFSET(decode_fec), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { NULL }, };