From patchwork Mon Jul 18 18:05:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 36824 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1649:b0:8b:613a:194d with SMTP id no9csp1265686pzb; Mon, 18 Jul 2022 11:06:49 -0700 (PDT) X-Google-Smtp-Source: AGRyM1s519whsQXTx4LTu9aFTpzrqpEjyg2+nDlo+rlGVSmL62ilNwA7nmE3/Lsj7o9e68ycQtUr X-Received: by 2002:a05:6402:34c5:b0:43a:8f90:e643 with SMTP id w5-20020a05640234c500b0043a8f90e643mr37773721edc.88.1658167609237; Mon, 18 Jul 2022 11:06:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1658167609; cv=none; d=google.com; s=arc-20160816; b=oOeZ1RjoPDAGrJpzSEuhPgnsCh4/f7Y7N7VwiyrHwMZPBWo08eHwv3Ithp0YWaUycm prebGMnH9GqPbvEvGkQS9BQC5pPNK3VIPboK/3A5/Q0tLlBPySbGTwWfjkXXNXZFSYi/ sIVUs2aTmsRZOvJwYe0ryS7FuAwKVLIWYeTMDMozshNBR6o25osm57EF5vJtkjOSoqRP KmN4+IovGfRM7VzG/2xSp2PcvlmMTNUXlho2sJI0u2FUinoOe+GiEqvcQ4dkPZu7V7r+ BTP39y0ZYAXJklpcXqaqkEuTqt8yVw/hESrXYynLVDna3weskbB88QnzJQQ9HbE9wUIY avcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=ZPK+jItqCOe4rUuwTzimZeiC/r8CwQTJVgGW7JQXUhE=; b=DECx+wBArJW11IR8zvdlJ+Zg3ik7qiPizK3hcUVp76C0Gm7toX7xIPQcanyWQL/bXd 00P7cb4DrDYSSvlLbahZziqfYavVmcNLcDyK2Hny1iBx1kHjMEZSzqOCx+4XkS2scdp6 jpGFh1B8TaXB7/M2NP5w+J5T9L+ufAA37SzKKIwGwBuK7pNM78mqVsAtRIcv/Crh8MKP zgBE1GTr0dHeFTPqavLSxoUboWcly9bmPSXJXpeFRCkl9aOMD4/t5BL3PyOmd89MokzU EGsGMVn3qTFnxFUQVUdgr3CtMHMcS1yAiJAIXcX2Z56ZooiD7CRFf3Fl7MOLTriKDpdt JF9w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dm20-20020a170907949400b0072f3a9f36b8si3546057ejc.945.2022.07.18.11.06.45; Mon, 18 Jul 2022 11:06:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id ACB9368B944; Mon, 18 Jul 2022 21:06:42 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail0.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 369DC68B70E for ; Mon, 18 Jul 2022 21:06:36 +0300 (EEST) Received: from localhost (localhost [IPv6:::1]) by mail0.khirnov.net (Postfix) with ESMTP id 711DB240179 for ; Mon, 18 Jul 2022 20:06:35 +0200 (CEST) Received: from mail0.khirnov.net ([IPv6:::1]) by localhost (mail0.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id j8WOSJdKM1US for ; Mon, 18 Jul 2022 20:06:33 +0200 (CEST) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail0.khirnov.net (Postfix) with ESMTPS id 999892400F5 for ; Mon, 18 Jul 2022 20:06:33 +0200 (CEST) Received: by libav.khirnov.net (Postfix, from userid 1000) id A51B03A03A9; Mon, 18 Jul 2022 20:06:33 +0200 (CEST) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Mon, 18 Jul 2022 20:05:40 +0200 Message-Id: <20220718180540.15070-1-anton@khirnov.net> X-Mailer: git-send-email 2.34.1 In-Reply-To: <7132588d-a8c2-6d23-4c9e-ac7539866cdb@gmail.com> References: <7132588d-a8c2-6d23-4c9e-ac7539866cdb@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] lavc: add API for exporting reconstructed frames from encoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: n6UnBPgQ/xGA --- doc/APIchanges | 5 +++++ libavcodec/avcodec.c | 12 ++++++++++++ libavcodec/avcodec.h | 20 ++++++++++++++++---- libavcodec/codec.h | 8 ++++++++ libavcodec/decode.c | 4 +--- libavcodec/decode.h | 5 +++++ libavcodec/encode.c | 25 +++++++++++++++++++++++++ libavcodec/encode.h | 5 +++++ libavcodec/internal.h | 8 ++++++++ libavcodec/options_table.h | 1 + libavcodec/version.h | 2 +- 11 files changed, 87 insertions(+), 8 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index 4e218af94e..b50fece404 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -14,6 +14,11 @@ libavutil: 2021-04-27 API changes, most recent first: +2022-07-xx - xxxxxxxxxx - lavc 59.40.100 - avcodec.h codec.h + Add AV_CODEC_FLAG_RECON_FRAME and AV_CODEC_CAP_ENCODER_RECON_FRAME. + avcodec_receive_frame() may now be used on encoders when + AV_CODEC_FLAG_RECON_FRAME is active. + 2022-07-xx - xxxxxxxxxx - lavu 57.30.100 - frame.h av_frame_make_writable() may now be called on non-refcounted frames and will make a refcounted copy out of them. diff --git a/libavcodec/avcodec.c b/libavcodec/avcodec.c index 5f6e71a39e..9dd4157750 100644 --- a/libavcodec/avcodec.c +++ b/libavcodec/avcodec.c @@ -390,6 +390,8 @@ void avcodec_flush_buffers(AVCodecContext *avctx) } if (avci->in_frame) av_frame_unref(avci->in_frame); + if (avci->recon_frame) + av_frame_unref(avci->recon_frame); } else { av_packet_unref(avci->last_pkt_props); while (av_fifo_read(avci->pkt_props, avci->last_pkt_props, 1) >= 0) @@ -470,6 +472,7 @@ av_cold int avcodec_close(AVCodecContext *avctx) av_packet_free(&avci->in_pkt); av_frame_free(&avci->in_frame); + av_frame_free(&avci->recon_frame); av_buffer_unref(&avci->pool); @@ -716,3 +719,12 @@ int avcodec_is_open(AVCodecContext *s) { return !!s->internal; } + +int attribute_align_arg avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame) +{ + av_frame_unref(frame); + + if (av_codec_is_decoder(avctx->codec)) + return ff_decode_receive_frame(avctx, frame); + return ff_encode_receive_frame(avctx, frame); +} diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index cb5c25bf63..04e9d33493 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -231,6 +231,16 @@ typedef struct RcOverride{ * decoded frame in stream. */ #define AV_CODEC_FLAG_DROPCHANGED (1 << 5) +/** + * Request the encoder to output reconstructed frames, i.e. frames that would be + * produced by decoding the encoded bistream. These frames may be retrieved by + * calling avcodec_receive_frame() immediately after a successful call to + * avcodec_receive_packet(). + * + * Should only be used with encoders flagged with the + * AV_CODEC_CAP_ENCODER_RECON_FRAME capability. + */ +#define AV_CODEC_FLAG_RECON_FRAME (1 << 6) /** * Use internal 2pass ratecontrol in first pass mode. */ @@ -2595,21 +2605,23 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, int avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt); /** - * Return decoded output data from a decoder. + * Return decoded output data from a decoder or encoder (when the + * AV_CODEC_FLAG_RECON_FRAME flag is used). * * @param avctx codec context * @param frame This will be set to a reference-counted video or audio * frame (depending on the decoder type) allocated by the - * decoder. Note that the function will always call + * codec. Note that the function will always call * av_frame_unref(frame) before doing anything else. * * @return * 0: success, a frame was returned * AVERROR(EAGAIN): output is not available in this state - user must try * to send new input - * AVERROR_EOF: the decoder has been fully flushed, and there will be + * AVERROR_EOF: the codec has been fully flushed, and there will be * no more output frames - * AVERROR(EINVAL): codec not opened, or it is an encoder + * AVERROR(EINVAL): codec not opened, or it is an encoder without + * the AV_CODEC_FLAG_RECON_FRAME flag enabled * AVERROR_INPUT_CHANGED: current decoded frame has changed parameters * with respect to first decoded frame. Applicable * when flag AV_CODEC_FLAG_DROPCHANGED is set. diff --git a/libavcodec/codec.h b/libavcodec/codec.h index 03e8be90a2..77a1a3f5a2 100644 --- a/libavcodec/codec.h +++ b/libavcodec/codec.h @@ -182,6 +182,14 @@ */ #define AV_CODEC_CAP_ENCODER_FLUSH (1 << 21) +/** + * The encoder is able to output reconstructed frame data, i.e. raw frames that + * would be produced by decoding the encoded bitstream. + * + * Reconstructed frame output is enabled by the AV_CODEC_FLAG_RECON_FRAME flag. + */ +#define AV_CODEC_CAP_ENCODER_RECON_FRAME (1 << 22) + /** * AVProfile. */ diff --git a/libavcodec/decode.c b/libavcodec/decode.c index 1893caa6a6..0a40b207e8 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -636,13 +636,11 @@ static int apply_cropping(AVCodecContext *avctx, AVFrame *frame) AV_FRAME_CROP_UNALIGNED : 0); } -int attribute_align_arg avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame) +int ff_decode_receive_frame(AVCodecContext *avctx, AVFrame *frame) { AVCodecInternal *avci = avctx->internal; int ret, changed; - av_frame_unref(frame); - if (!avcodec_is_open(avctx) || !av_codec_is_decoder(avctx->codec)) return AVERROR(EINVAL); diff --git a/libavcodec/decode.h b/libavcodec/decode.h index 1b40f714e1..f7828fbff4 100644 --- a/libavcodec/decode.h +++ b/libavcodec/decode.h @@ -53,6 +53,11 @@ typedef struct FrameDecodeData { void (*hwaccel_priv_free)(void *priv); } FrameDecodeData; +/** + * avcodec_receive_frame() implementation for decoders. + */ +int ff_decode_receive_frame(AVCodecContext *avctx, AVFrame *frame); + /** * Called by decoders to get the next packet for decoding. * diff --git a/libavcodec/encode.c b/libavcodec/encode.c index 1f39ab1a2f..f15309ce09 100644 --- a/libavcodec/encode.c +++ b/libavcodec/encode.c @@ -594,6 +594,18 @@ int ff_encode_preinit(AVCodecContext *avctx) return AVERROR(ENOMEM); } + if ((avctx->flags & AV_CODEC_FLAG_RECON_FRAME)) { + if (!(avctx->codec->capabilities & AV_CODEC_CAP_ENCODER_RECON_FRAME)) { + av_log(avctx, AV_LOG_ERROR, "Reconstructed frame output requested " + "from an encoder not supporting it\n"); + return AVERROR(ENOSYS); + } + + avci->recon_frame = av_frame_alloc(); + if (!avci->recon_frame) + return AVERROR(ENOMEM); + } + return 0; } @@ -630,3 +642,16 @@ int ff_encode_alloc_frame(AVCodecContext *avctx, AVFrame *frame) return 0; } + +int ff_encode_receive_frame(AVCodecContext *avctx, AVFrame *frame) +{ + AVCodecInternal *avci = avctx->internal; + + if (!avci->recon_frame) + return AVERROR(EINVAL); + if (!avci->recon_frame->buf[0]) + return avci->draining_done ? AVERROR_EOF : AVERROR(EAGAIN); + + av_frame_move_ref(frame, avci->recon_frame); + return 0; +} diff --git a/libavcodec/encode.h b/libavcodec/encode.h index b2536bf0f3..bc77918d8f 100644 --- a/libavcodec/encode.h +++ b/libavcodec/encode.h @@ -26,6 +26,11 @@ #include "avcodec.h" #include "packet.h" +/** + * avcodec_receive_frame() implementation for encoders. + */ +int ff_encode_receive_frame(AVCodecContext *avctx, AVFrame *frame); + /** * Called by encoders to get the next frame for encoding. * diff --git a/libavcodec/internal.h b/libavcodec/internal.h index 17e1de8127..fb93d42aa1 100644 --- a/libavcodec/internal.h +++ b/libavcodec/internal.h @@ -103,6 +103,14 @@ typedef struct AVCodecInternal { */ AVFrame *in_frame; + /** + * When the AV_CODEC_FLAG_RECON_FRAME flag is used. the encoder should store + * here the reconstructed frame corresponding to the last returned packet. + * + * Not allocated in other cases. + */ + AVFrame *recon_frame; + /** * If this is set, then FFCodec->close (if existing) needs to be called * for the parent AVCodecContext. diff --git a/libavcodec/options_table.h b/libavcodec/options_table.h index e72b4d12b6..35a94c0227 100644 --- a/libavcodec/options_table.h +++ b/libavcodec/options_table.h @@ -57,6 +57,7 @@ static const AVOption avcodec_options[] = { {"qpel", "use 1/4-pel motion compensation", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_QPEL }, INT_MIN, INT_MAX, V|E, "flags"}, {"loop", "use loop filter", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_LOOP_FILTER }, INT_MIN, INT_MAX, V|E, "flags"}, {"qscale", "use fixed qscale", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_QSCALE }, INT_MIN, INT_MAX, 0, "flags"}, +{"recon_frame", "export reconstructed frames", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_RECON_FRAME}, .unit = "flags"}, {"pass1", "use internal 2-pass ratecontrol in first pass mode", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_PASS1 }, INT_MIN, INT_MAX, 0, "flags"}, {"pass2", "use internal 2-pass ratecontrol in second pass mode", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_PASS2 }, INT_MIN, INT_MAX, 0, "flags"}, {"gray", "only decode/encode grayscale", 0, AV_OPT_TYPE_CONST, {.i64 = AV_CODEC_FLAG_GRAY }, INT_MIN, INT_MAX, V|E|D, "flags"}, diff --git a/libavcodec/version.h b/libavcodec/version.h index f2f14eaed1..19f3f4a272 100644 --- a/libavcodec/version.h +++ b/libavcodec/version.h @@ -29,7 +29,7 @@ #include "version_major.h" -#define LIBAVCODEC_VERSION_MINOR 39 +#define LIBAVCODEC_VERSION_MINOR 40 #define LIBAVCODEC_VERSION_MICRO 100 #define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \