From patchwork Sun Jul 17 19:26:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 36816 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1649:b0:8b:613a:194d with SMTP id no9csp706160pzb; Sun, 17 Jul 2022 12:28:20 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tffeUsT2HbV1+lGT0BeGJ2SNRPsd5GnisCadF8m+0AObHjgwbSkl2tY1QyZeTemalNwuvl X-Received: by 2002:a05:6402:2b88:b0:43a:6c58:6c64 with SMTP id fj8-20020a0564022b8800b0043a6c586c64mr33792191edb.348.1658086099928; Sun, 17 Jul 2022 12:28:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1658086099; cv=none; d=google.com; s=arc-20160816; b=olDujIcOTE5fsSJz2aMcNl8lfYhP5liNb3eoKrUwlnxQFbCWvp39GcOXYWlt13iLUA R0x8d0SiVlL866GDswXJVr3HAIroIOiJ6PzUIjOlpTG+xVhaLyvb9HZZFfV3RlU9/E1r ECue9XbVDGhvzMHieRGbTsTReWvfFF77BDW41Il3AxMBV/bA17KgFiMthBLE5DrhC4Ox 2Hi1KMRMjMxmGwG+pcOczzUGTmZZ9ts47rb6Z88nIAhdtW0gcoAKw7lkErd5KtX4sxt/ jr0nzO080rlCoa1VQLeX+BwxsNSIKwASyPHVB5XrdyBS8XXT9e/3Kux68c74OSnOWLGb S04w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=xQWjC7KKGO5sLbd18GioK010tp43SwZM/XBOXZWkmPk=; b=iJBAOO5pDijFmbnAnGqMfnwcyXOVBh8460GaKjhYB74+cBRaG2FSr1UEH/2NTH2WU1 bNwMH/gCFUxfx2mp2SwDuZM6vW907KE9bniHQk+duJz5xe7N/fuWY1yQ8NPwehOcL9pJ TLWOZqtRE9n4tLrf2+QVQrYYpk3GYh6zjkmEIWZqLtFOX8X6pRZhq5tGu6JBwyGDNKbm a3MCmGcqMNN1Sv+eEElfThjS62fjyuxD6Ltng0VxCRN51p5/ha8jWgYCJb3NQ2kVxCUR PM5bJFbh4kno5X7cJJ10ABeO+hGBzqwLnLRMK3Sa69bx9esvt4xLPchQ2a/BmpCtlhbX lLrw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d5-20020a170906174500b00711e3fede57si9913247eje.190.2022.07.17.12.28.18; Sun, 17 Jul 2022 12:28:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BFDAD68B7A4; Sun, 17 Jul 2022 22:27:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail0.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8F44168B5FB for ; Sun, 17 Jul 2022 22:27:41 +0300 (EEST) Received: from localhost (localhost [IPv6:::1]) by mail0.khirnov.net (Postfix) with ESMTP id 1DC2824017E for ; Sun, 17 Jul 2022 21:27:41 +0200 (CEST) Received: from mail0.khirnov.net ([IPv6:::1]) by localhost (mail0.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id urCXqpyIGQ2E for ; Sun, 17 Jul 2022 21:27:39 +0200 (CEST) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail0.khirnov.net (Postfix) with ESMTPS id E66622400F5 for ; Sun, 17 Jul 2022 21:27:38 +0200 (CEST) Received: by libav.khirnov.net (Postfix, from userid 1000) id 533DF3A03D8; Sun, 17 Jul 2022 21:27:36 +0200 (CEST) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Sun, 17 Jul 2022 21:26:58 +0200 Message-Id: <20220717192700.1077-2-anton@khirnov.net> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220717192700.1077-1-anton@khirnov.net> References: <20220717192700.1077-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/4] lavc: add API for exporting reconstructed frames from encoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: f3r+3QA2+JEt --- doc/APIchanges | 5 +++++ libavcodec/avcodec.c | 10 ++++++++++ libavcodec/avcodec.h | 20 ++++++++++++++++---- libavcodec/codec.h | 8 ++++++++ libavcodec/decode.c | 2 +- libavcodec/decode.h | 5 +++++ libavcodec/encode.c | 25 +++++++++++++++++++++++++ libavcodec/encode.h | 5 +++++ libavcodec/internal.h | 8 ++++++++ libavcodec/options_table.h | 1 + libavcodec/version.h | 2 +- 11 files changed, 85 insertions(+), 6 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index 4e218af94e..b50fece404 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -14,6 +14,11 @@ libavutil: 2021-04-27 API changes, most recent first: +2022-07-xx - xxxxxxxxxx - lavc 59.40.100 - avcodec.h codec.h + Add AV_CODEC_FLAG_RECON_FRAME and AV_CODEC_CAP_ENCODER_RECON_FRAME. + avcodec_receive_frame() may now be used on encoders when + AV_CODEC_FLAG_RECON_FRAME is active. + 2022-07-xx - xxxxxxxxxx - lavu 57.30.100 - frame.h av_frame_make_writable() may now be called on non-refcounted frames and will make a refcounted copy out of them. diff --git a/libavcodec/avcodec.c b/libavcodec/avcodec.c index 5f6e71a39e..e25273cb4e 100644 --- a/libavcodec/avcodec.c +++ b/libavcodec/avcodec.c @@ -390,6 +390,8 @@ void avcodec_flush_buffers(AVCodecContext *avctx) } if (avci->in_frame) av_frame_unref(avci->in_frame); + if (avci->recon_frame) + av_frame_unref(avci->recon_frame); } else { av_packet_unref(avci->last_pkt_props); while (av_fifo_read(avci->pkt_props, avci->last_pkt_props, 1) >= 0) @@ -470,6 +472,7 @@ av_cold int avcodec_close(AVCodecContext *avctx) av_packet_free(&avci->in_pkt); av_frame_free(&avci->in_frame); + av_frame_free(&avci->recon_frame); av_buffer_unref(&avci->pool); @@ -716,3 +719,10 @@ int avcodec_is_open(AVCodecContext *s) { return !!s->internal; } + +int attribute_align_arg avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame) +{ + if (av_codec_is_decoder(avctx->codec)) + return ff_decode_receive_frame(avctx, frame); + return ff_encode_receive_frame(avctx, frame); +} diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index cb5c25bf63..04e9d33493 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -231,6 +231,16 @@ typedef struct RcOverride{ * decoded frame in stream. */ #define AV_CODEC_FLAG_DROPCHANGED (1 << 5) +/** + * Request the encoder to output reconstructed frames, i.e. frames that would be + * produced by decoding the encoded bistream. These frames may be retrieved by + * calling avcodec_receive_frame() immediately after a successful call to + * avcodec_receive_packet(). + * + * Should only be used with encoders flagged with the + * AV_CODEC_CAP_ENCODER_RECON_FRAME capability. + */ +#define AV_CODEC_FLAG_RECON_FRAME (1 << 6) /** * Use internal 2pass ratecontrol in first pass mode. */ @@ -2595,21 +2605,23 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, int avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt); /** - * Return decoded output data from a decoder. + * Return decoded output data from a decoder or encoder (when the + * AV_CODEC_FLAG_RECON_FRAME flag is used). * * @param avctx codec context * @param frame This will be set to a reference-counted video or audio * frame (depending on the decoder type) allocated by the - * decoder. Note that the function will always call + * codec. Note that the function will always call * av_frame_unref(frame) before doing anything else. * * @return * 0: success, a frame was returned * AVERROR(EAGAIN): output is not available in this state - user must try * to send new input - * AVERROR_EOF: the decoder has been fully flushed, and there will be + * AVERROR_EOF: the codec has been fully flushed, and there will be * no more output frames - * AVERROR(EINVAL): codec not opened, or it is an encoder + * AVERROR(EINVAL): codec not opened, or it is an encoder without + * the AV_CODEC_FLAG_RECON_FRAME flag enabled * AVERROR_INPUT_CHANGED: current decoded frame has changed parameters * with respect to first decoded frame. Applicable * when flag AV_CODEC_FLAG_DROPCHANGED is set. diff --git a/libavcodec/codec.h b/libavcodec/codec.h index 03e8be90a2..77a1a3f5a2 100644 --- a/libavcodec/codec.h +++ b/libavcodec/codec.h @@ -182,6 +182,14 @@ */ #define AV_CODEC_CAP_ENCODER_FLUSH (1 << 21) +/** + * The encoder is able to output reconstructed frame data, i.e. raw frames that + * would be produced by decoding the encoded bitstream. + * + * Reconstructed frame output is enabled by the AV_CODEC_FLAG_RECON_FRAME flag. + */ +#define AV_CODEC_CAP_ENCODER_RECON_FRAME (1 << 22) + /** * AVProfile. */ diff --git a/libavcodec/decode.c b/libavcodec/decode.c index 1893caa6a6..03f3d4e3a2 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -636,7 +636,7 @@ static int apply_cropping(AVCodecContext *avctx, AVFrame *frame) AV_FRAME_CROP_UNALIGNED : 0); } -int attribute_align_arg avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame) +int ff_decode_receive_frame(AVCodecContext *avctx, AVFrame *frame) { AVCodecInternal *avci = avctx->internal; int ret, changed; diff --git a/libavcodec/decode.h b/libavcodec/decode.h index 1b40f714e1..f7828fbff4 100644 --- a/libavcodec/decode.h +++ b/libavcodec/decode.h @@ -53,6 +53,11 @@ typedef struct FrameDecodeData { void (*hwaccel_priv_free)(void *priv); } FrameDecodeData; +/** + * avcodec_receive_frame() implementation for decoders. + */ +int ff_decode_receive_frame(AVCodecContext *avctx, AVFrame *frame); + /** * Called by decoders to get the next packet for decoding. * diff --git a/libavcodec/encode.c b/libavcodec/encode.c index 1f39ab1a2f..f15309ce09 100644 --- a/libavcodec/encode.c +++ b/libavcodec/encode.c @@ -594,6 +594,18 @@ int ff_encode_preinit(AVCodecContext *avctx) return AVERROR(ENOMEM); } + if ((avctx->flags & AV_CODEC_FLAG_RECON_FRAME)) { + if (!(avctx->codec->capabilities & AV_CODEC_CAP_ENCODER_RECON_FRAME)) { + av_log(avctx, AV_LOG_ERROR, "Reconstructed frame output requested " + "from an encoder not supporting it\n"); + return AVERROR(ENOSYS); + } + + avci->recon_frame = av_frame_alloc(); + if (!avci->recon_frame) + return AVERROR(ENOMEM); + } + return 0; } @@ -630,3 +642,16 @@ int ff_encode_alloc_frame(AVCodecContext *avctx, AVFrame *frame) return 0; } + +int ff_encode_receive_frame(AVCodecContext *avctx, AVFrame *frame) +{ + AVCodecInternal *avci = avctx->internal; + + if (!avci->recon_frame) + return AVERROR(EINVAL); + if (!avci->recon_frame->buf[0]) + return avci->draining_done ? AVERROR_EOF : AVERROR(EAGAIN); + + av_frame_move_ref(frame, avci->recon_frame); + return 0; +} diff --git a/libavcodec/encode.h b/libavcodec/encode.h index b2536bf0f3..bc77918d8f 100644 --- a/libavcodec/encode.h +++ b/libavcodec/encode.h @@ -26,6 +26,11 @@ #include "avcodec.h" #include "packet.h" +/** + * avcodec_receive_frame() implementation for encoders. + */ +int ff_encode_receive_frame(AVCodecContext *avctx, AVFrame *frame); + /** * Called by encoders to get the next frame for encoding. * diff --git a/libavcodec/internal.h b/libavcodec/internal.h index 17e1de8127..fb93d42aa1 100644 --- a/libavcodec/internal.h +++ b/libavcodec/internal.h @@ -103,6 +103,14 @@ typedef struct AVCodecInternal { */ AVFrame *in_frame; + /** + * When the AV_CODEC_FLAG_RECON_FRAME flag is used. the encoder should store + * here the reconstructed frame corresponding to the last returned packet. + * + * Not allocated in other cases. + */ + AVFrame *recon_frame; + /** * If this is set, then FFCodec->close (if existing) needs to be called * for the parent AVCodecContext. diff --git a/libavcodec/options_table.h b/libavcodec/options_table.h index e72b4d12b6..35a94c0227 100644 --- a/libavcodec/options_table.h +++ b/libavcodec/options_table.h @@ -57,6 +57,7 @@ static const AVOption avcodec_options[] = { {"qpel", "use 1/4-pel motion compensation", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_QPEL }, INT_MIN, INT_MAX, V|E, "flags"}, {"loop", "use loop filter", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_LOOP_FILTER }, INT_MIN, INT_MAX, V|E, "flags"}, {"qscale", "use fixed qscale", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_QSCALE }, INT_MIN, INT_MAX, 0, "flags"}, +{"recon_frame", "export reconstructed frames", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_RECON_FRAME}, .unit = "flags"}, {"pass1", "use internal 2-pass ratecontrol in first pass mode", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_PASS1 }, INT_MIN, INT_MAX, 0, "flags"}, {"pass2", "use internal 2-pass ratecontrol in second pass mode", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_PASS2 }, INT_MIN, INT_MAX, 0, "flags"}, {"gray", "only decode/encode grayscale", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_GRAY }, INT_MIN, INT_MAX, V|E|D, "flags"}, diff --git a/libavcodec/version.h b/libavcodec/version.h index f2f14eaed1..19f3f4a272 100644 --- a/libavcodec/version.h +++ b/libavcodec/version.h @@ -29,7 +29,7 @@ #include "version_major.h" -#define LIBAVCODEC_VERSION_MINOR 39 +#define LIBAVCODEC_VERSION_MINOR 40 #define LIBAVCODEC_VERSION_MICRO 100 #define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \