From patchwork Wed Mar 16 14:00:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philip-Dylan Gleonec X-Patchwork-Id: 34792 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:3486:0:0:0:0 with SMTP id ek6csp51155nkb; Wed, 16 Mar 2022 07:02:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzybMjUKHM6gDrvvnEUCVYJ4FCqyjeVem5eNdwA8kVFEgKSG/FI39wS7ng6aYWrHETbPJGH X-Received: by 2002:a17:906:a210:b0:6d5:9fa:11ce with SMTP id r16-20020a170906a21000b006d509fa11cemr153232ejy.172.1647439317869; Wed, 16 Mar 2022 07:01:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647439317; cv=none; d=google.com; s=arc-20160816; b=aYVE5cWJ1x5gcC571JPq2/of7fKjJLwD6pgWd6gn1yU9EHadSYWHh+dhMaLObzErL9 eAywDp5682PyurFU4X22/bkLSyggfEY/RKoA8qugM787Ungf6aaBID/zKx0DeE1/57Ft kc2PJsRm5yrTPt7iXF578gOswweC7PYbE9dsBWXlE6kG+pS8nJkr/3jKOhfzbVb1YZwc U09kklwteRtD0rLUjbm4zxZI6BCGtoq0NTSv88UWCtHT1QG0tTvICiCSzTJu0Xz0umxf UgDM/KPzb7f8AjWFZuGhn83+VrYTyzEvd6+N8Z2A5jWIq4HhRZOOHVsOFXmhTo9GRlMa hSww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=GHZmI7xA6w2AVkIdaWRROqaPRgLGIMKYoC09Cog241A=; b=eL/8oq9F5kYof1XJHsysnlqUTzyCtUW7KTuUSJUKQlP+lGCj81AVt/hQaoIpURlOgj zjAHpOPl435/ukKdwTmSbzRZlaTYf8aoqi+8T+SFKF41SDs2yMEWVzK5JGX/fiXus9ix Vr1JlakWR4P6soXLp/ufCd3rv7ltKQETHyjA+9ZsPKQ8VVp9RNK76NxJCRLG5wQzpU6O zcRDXXVA77FxXIhcleo7QoMNm7/SFNYX6vpv512C5cTdVtZpPmhdGW7Ja8/VRwNi42P6 +2CJWgWjzGWrFQ8pW4iRav08qa6fmnP2GZOO24WswAMkPh7TQltoWYeoikhktY+gr1p2 +TQg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q23-20020a170906771700b006b775f31a5bsi1147635ejm.673.2022.03.16.07.01.57; Wed, 16 Mar 2022 07:01:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CBA5D68B00F; Wed, 16 Mar 2022 16:01:53 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3B27168AEE9 for ; Wed, 16 Mar 2022 16:01:47 +0200 (EET) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 56FA09C0383 for ; Wed, 16 Mar 2022 10:01:46 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id MWt_cDIphzdr; Wed, 16 Mar 2022 10:01:46 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id F078A9C0382; Wed, 16 Mar 2022 10:01:45 -0400 (EDT) X-Virus-Scanned: amavisd-new at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id tpgEnSGFaw4S; Wed, 16 Mar 2022 10:01:45 -0400 (EDT) Received: from localhost.localdomain (lfbn-ren-1-221-128.w83-205.abo.wanadoo.fr [83.205.67.128]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 6E6BB9C036E; Wed, 16 Mar 2022 10:01:45 -0400 (EDT) From: Philip-Dylan Gleonec To: ffmpeg-devel@ffmpeg.org Date: Wed, 16 Mar 2022 15:00:44 +0100 Message-Id: <20220316140045.369016-2-philip-dylan.gleonec@savoirfairelinux.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220316140045.369016-1-philip-dylan.gleonec@savoirfairelinux.com> References: <20220316140045.369016-1-philip-dylan.gleonec@savoirfairelinux.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] avcodec/libopusenc: reload packet loss at encode X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Philip-Dylan Gleonec Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: BObgfC5qRkzx An estimation of packet loss is required by libopus to compute its FEC data. Currently, this estimation is constant, and can not be changed after configuration. This means an application using libopus through ffmpeg can not adapt the packet loss estimation when the network quality degrades. This patch makes the encoder reload the packet_loss AVOption before encoding samples, if fec is enabled and the packet loss estimation set is different than the current one. This way an application can modify the packet loss estimation by changing the AVOption. Typical use-case is a RTP stream, where packet loss can be estimated from RTCP packets. Signed-off-by: Philip-Dylan Gleonec --- libavcodec/libopusenc.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/libavcodec/libopusenc.c b/libavcodec/libopusenc.c index 45b23fcbb5..b9e2fc45e3 100644 --- a/libavcodec/libopusenc.c +++ b/libavcodec/libopusenc.c @@ -460,6 +460,23 @@ static int libopus_encode(AVCodecContext *avctx, AVPacket *avpkt, uint8_t *audio; int ret; int discard_padding; + int32_t opus_packet_loss = 0; + + ret = opus_multistream_encoder_ctl(opus->enc, + OPUS_GET_PACKET_LOSS_PERC(&opus_packet_loss)); + if (ret != OPUS_OK) + av_log(avctx, AV_LOG_WARNING, + "Unable to get expected packet loss percentage: %s\n", + opus_strerror(ret)); + + if (opus->opts.fec && (opus_packet_loss != opus->opts.packet_loss)) { + ret = opus_multistream_encoder_ctl(opus->enc, + OPUS_SET_PACKET_LOSS_PERC(opus->opts.packet_loss)); + if (ret != OPUS_OK) + av_log(avctx, AV_LOG_WARNING, + "Unable to set expected packet loss percentage: %s\n", + opus_strerror(ret)); + } if (frame) { ret = ff_af_queue_add(&opus->afq, frame); From patchwork Wed Mar 16 14:00:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philip-Dylan Gleonec X-Patchwork-Id: 34793 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:3486:0:0:0:0 with SMTP id ek6csp51355nkb; Wed, 16 Mar 2022 07:02:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwNFJeVatvBIGQLStH7eHkNUEbHG+dmJTPaJ607LcyEdbDBGAJBldM1DOIAKN6TYKaOAmtC X-Received: by 2002:a05:6402:518a:b0:416:b50e:3c7d with SMTP id q10-20020a056402518a00b00416b50e3c7dmr28162904edd.177.1647439331689; Wed, 16 Mar 2022 07:02:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647439331; cv=none; d=google.com; s=arc-20160816; b=srR5dTlkSS6BW1aYwQ9LQZQIWfkkH50cnKaODIDZ0PkXAjlmt+KPL1jf5SFhDyk6l2 0h1Woq58jtJlh1xUH8DPUxm/qHVtVapRC6x60Fv0Anlwq2HEbLMO7+LsnScbct4Q69l9 2taY79/1KcMIbmG1egAvOX1gzFavehTWDzRS72k8+MDQdno+FoSk+5AZJVnqQxcKhyE0 MbMjbLRNctFEUSo3iPFoJfDH3dTnRyt3+ykv5WKV1C1jsI4tVlUCuWeeRaizkMQfdp0x m4W4/Ym6ksqk8VWcBspOmUMamMBuLli8cB3F34EYFx7+UyTQsK6tgq7/t9jtZQhCny6c b5mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=6B6UYVfjCz7XSc0fvui8+roavsrFuBuMzi4SsghCmTE=; b=geBZEtGL8Efh4SmgN4cS9vKjn2jz4Ly4fDIbY3B6PiiKUEltTGrzJnkK90Jkrp2GuE GOinI9lZp+pb0x7srHpb/xccix4ZSt8LhpF+bUHXcOHttDFGfH+/r7wuSkr0aeiDY/lQ ITt/0JoDaaKZEkPESs9H4iThwO/8Qrd/Rus5T7Na6iuDAInv1IsUL7KkQxnXejw//SKH pQ1IFPciuZX0f6SJL7OhBEofuBrw+q/6CwWGHbFJu+zNdzparOx7YWCBE/EPRErmpXtx A5/MbYfVK73FLiKQSlf8Y2v9KefA91tgR5vhz/DRFDa4QM0zp8YV8I54MSk94EbrSn1m wbiA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w11-20020a170906d20b00b006ceb94fbcf0si1196169ejz.271.2022.03.16.07.02.10; Wed, 16 Mar 2022 07:02:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 52DFF68030F; Wed, 16 Mar 2022 16:01:59 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EA9F968AFCC for ; Wed, 16 Mar 2022 16:01:56 +0200 (EET) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 5EBE59C036E; Wed, 16 Mar 2022 10:01:55 -0400 (EDT) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id JgbIxhtxctNp; Wed, 16 Mar 2022 10:01:54 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id AAE899C0354; Wed, 16 Mar 2022 10:01:54 -0400 (EDT) X-Virus-Scanned: amavisd-new at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id x_CWZ-vYDyQN; Wed, 16 Mar 2022 10:01:54 -0400 (EDT) Received: from localhost.localdomain (lfbn-ren-1-221-128.w83-205.abo.wanadoo.fr [83.205.67.128]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 0A8329C036E; Wed, 16 Mar 2022 10:01:53 -0400 (EDT) From: Philip-Dylan Gleonec To: ffmpeg-devel@ffmpeg.org Date: Wed, 16 Mar 2022 15:00:45 +0100 Message-Id: <20220316140045.369016-3-philip-dylan.gleonec@savoirfairelinux.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220316140045.369016-1-philip-dylan.gleonec@savoirfairelinux.com> References: <20220316140045.369016-1-philip-dylan.gleonec@savoirfairelinux.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] avcodec/libopusdec: Enable FEC/PLC X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Philip-Dylan Gleonec , "Steinar H . Gunderson" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: cq82qrYw2+br Adds FEC/PLC support to libopus. The lost packets are detected as a discontinuity in the audio stream. When a discontinuity is used, this patch tries to decode the FEC data. If FEC data is present in the packet, it is decoded, otherwise audio is re-created through PLC. This patch is based on Steinar H. Gunderson contribution, and corrects the pts computation: all pts are expressed in samples instead of time. This patch also adds an option "decode_fec" which enables or disables FEC decoding. This option is disabled by default to keep consistent behaviour with former versions. A number of checks are made to ensure compatibility with different containers. Indeed, video containers seem to have a pts expressed in ms while it is expressed in samples for audio containers. It also manages the cases where pkt->duration is 0, in some RTP streams. This patch ignores data it can not reconstruct, i.e. packets received twice and packets with a length that is not a multiple of 2.5ms. Signed-off-by: Philip-Dylan Gleonec Co-developed-by: Steinar H. Gunderson --- libavcodec/libopusdec.c | 105 +++++++++++++++++++++++++++++++++++----- 1 file changed, 94 insertions(+), 11 deletions(-) diff --git a/libavcodec/libopusdec.c b/libavcodec/libopusdec.c index 86ef715205..66134300d2 100644 --- a/libavcodec/libopusdec.c +++ b/libavcodec/libopusdec.c @@ -43,10 +43,15 @@ struct libopus_context { #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST int apply_phase_inv; #endif + int decode_fec; + int64_t expected_next_pts; }; #define OPUS_HEAD_SIZE 19 +// Sample rate is constant as libopus always output at 48kHz +const AVRational opus_timebase = { 1, 48000 }; + static av_cold int libopus_decode_init(AVCodecContext *avc) { struct libopus_context *opus = avc->priv_data; @@ -134,6 +139,8 @@ static av_cold int libopus_decode_init(AVCodecContext *avc) /* Decoder delay (in samples) at 48kHz */ avc->delay = avc->internal->skip_samples = opus->pre_skip; + opus->expected_next_pts = AV_NOPTS_VALUE; + return 0; } @@ -155,25 +162,100 @@ static int libopus_decode(AVCodecContext *avc, void *data, { struct libopus_context *opus = avc->priv_data; AVFrame *frame = data; - int ret, nb_samples; + uint8_t *outptr; + int ret, nb_samples = 0, nb_lost_samples = 0, nb_samples_left; + + // If FEC is enabled, calculate number of lost samples + if (opus->decode_fec && + opus->expected_next_pts != AV_NOPTS_VALUE && + pkt->pts != AV_NOPTS_VALUE && + pkt->pts != opus->expected_next_pts) { + // Cap at recovering 120 ms of lost audio. + nb_lost_samples = pkt->pts - opus->expected_next_pts; + nb_lost_samples = FFMIN(nb_lost_samples, MAX_FRAME_SIZE); + // pts is expressed in ms for some containers (e.g. mkv) + // FEC only works for SILK frames (> 10ms) + // Detect if nb_lost_samples is in ms, and convert in samples if it is + if (nb_lost_samples > 0) { + if (avc->pkt_timebase.den != 48000) { + nb_lost_samples = av_rescale_q(nb_lost_samples, avc->pkt_timebase, opus_timebase); + } + // For FEC/PLC, frame_size has to be to have a multiple of 2.5 ms + if (nb_lost_samples % (int)(2.5 / 1000 * opus_timebase.den)) { + nb_lost_samples -= nb_lost_samples % (int)(2.5 / 1000 * opus_timebase.den); + } + } + } - frame->nb_samples = MAX_FRAME_SIZE; + frame->nb_samples = MAX_FRAME_SIZE + nb_lost_samples; if ((ret = ff_get_buffer(avc, frame, 0)) < 0) return ret; + outptr = frame->data[0]; + nb_samples_left = frame->nb_samples; + + if (opus->decode_fec && nb_lost_samples > 0) { + // Try to recover the lost samples with FEC data from this one. + // If there's no FEC data, the decoder will do loss concealment instead. + if (avc->sample_fmt == AV_SAMPLE_FMT_S16) + ret = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_lost_samples, 1); + else + ret = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_lost_samples, 1); + + if (ret < 0) { + if (opus->decode_fec) opus->expected_next_pts = pkt->pts + pkt->duration; + av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", + opus_strerror(ret)); + return ff_opus_error_to_averror(ret); + } + + av_log(avc, AV_LOG_WARNING, "Recovered %d samples with FEC/PLC\n", + ret); + + outptr += ret * avc->channels * av_get_bytes_per_sample(avc->sample_fmt); + nb_samples_left -= ret; + nb_samples += ret; + if (pkt->pts != AV_NOPTS_VALUE) { + frame->pts = pkt->pts - ret; + } + } + + // Decode the actual, non-lost data. if (avc->sample_fmt == AV_SAMPLE_FMT_S16) - nb_samples = opus_multistream_decode(opus->dec, pkt->data, pkt->size, - (opus_int16 *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode(opus->dec, pkt->data, pkt->size, + (opus_int16 *)outptr, + nb_samples_left, 0); else - nb_samples = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, - (float *)frame->data[0], - frame->nb_samples, 0); + ret = opus_multistream_decode_float(opus->dec, pkt->data, pkt->size, + (float *)outptr, + nb_samples_left, 0); - if (nb_samples < 0) { + if (ret < 0) { + if (opus->decode_fec) opus->expected_next_pts = pkt->pts + pkt->duration; av_log(avc, AV_LOG_ERROR, "Decoding error: %s\n", - opus_strerror(nb_samples)); - return ff_opus_error_to_averror(nb_samples); + opus_strerror(ret)); + return ff_opus_error_to_averror(ret); + } + nb_samples += ret; + + if (opus->decode_fec) + { + // Calculate the next expected pts + if (pkt->pts == AV_NOPTS_VALUE) { + opus->expected_next_pts = AV_NOPTS_VALUE; + } else { + if (pkt->duration) { + opus->expected_next_pts = pkt->pts + pkt->duration; + } else if (avc->pkt_timebase.num) { + opus->expected_next_pts = pkt->pts + av_rescale_q(ret, opus_timebase, avc->pkt_timebase); + } else { + opus->expected_next_pts = pkt->pts + ret; + } + } } #ifndef OPUS_SET_GAIN @@ -214,6 +296,7 @@ static const AVOption libopusdec_options[] = { #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST { "apply_phase_inv", "Apply intensity stereo phase inversion", OFFSET(apply_phase_inv), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, #endif + { "decode_fec", "Decode FEC data or use PLC", OFFSET(decode_fec), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { NULL }, };