From patchwork Sun Jul 9 11:05:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 42555 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6da1:b0:131:a7d0:bc6d with SMTP id gl33csp53868pzb; Sun, 9 Jul 2023 04:05:44 -0700 (PDT) X-Google-Smtp-Source: APBJJlH54wxOwV4DCRWgKd3KiG3a1e70dw7mTpg+/sxIF+Qsl3gnVTcAk9ZRBp/z9lyuMhQgUu40 X-Received: by 2002:a17:906:105e:b0:993:e809:b9ff with SMTP id j30-20020a170906105e00b00993e809b9ffmr4628909ejj.21.1688900744538; Sun, 09 Jul 2023 04:05:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688900744; cv=none; d=google.com; s=arc-20160816; b=oj7Of8/gDDTbuAF/EYmAnSW9NS8wtPl7J5HHgusYFEiLemrQnK7I/R9x8aGMPeALpc QZjK747d6h/RGeyj9HW5RnCEu1oPTJvbuXpnSH7PybzSRWGE6fdXYNlSU6USKGWL0n4n mviYvzFx8GBfflMRUz1VHKWHgiYKM6CF5M6DgJFmq3zzqFb7C8/f+IZ2Gy1gtrQMcIfP yP6QeV/OQ+rI5RjbkIw2zqODR7o5N9DEFXvNzjM/ifXeQT6BsZOaDF7j4vOpdIqLv5k+ e/BmIyAIPun7j5H9nWRAREtXlvtVbjjLj8DUJadoGt6kSIi2jqE1VV4Ie04ROfq9SF6F b4zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=MiFHbZDsHt+GaPTDSb1Pj8se3AoJ1LhlfWY7S0ky9Ug=; fh=8ZF2QtiKOjBtyn9wuO2cMeZaLsE4aq02Bq77gGK61E8=; b=BrikoxShG8MjyqpBUuIv0e8dOt9YcHxcDr08yTBSeyWJXRyRvzOJ6yAFchGsy0keK4 8aMr5mLQO3Lu7YpL2Xfa+vrTGQjuy7yNofnmQSzTzrSSIIY8JIluGWDY6AUyPhwnP+rC fnFofx70z6n7IPf6RI77SBtY2hVkRPwwQNgSoyykCOhALHVPYIrkEXd/GQpJGmsaNHZj zmkAgZQ6RridGe+b3sGokMC805yIC7OfjFeKlZfUmN89Q99Vy41wofK62geCrbiwfKKE HyXnRJdZkjCeAclNVSNBIzaOLezLrfaDRUIumkx/P1BD0uz9604UDU9Jn6KRAvwr0e7N Q5yA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gv15-20020a170906f10f00b009924806cd02si6233710ejb.392.2023.07.09.04.05.43; Sun, 09 Jul 2023 04:05:44 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7EE7B68C5C9; Sun, 9 Jul 2023 14:05:40 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail0.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0724A68C532 for ; Sun, 9 Jul 2023 14:05:34 +0300 (EEST) Received: from localhost (localhost [IPv6:::1]) by mail0.khirnov.net (Postfix) with ESMTP id AC7CE2404EC for ; Sun, 9 Jul 2023 13:05:33 +0200 (CEST) Received: from mail0.khirnov.net ([IPv6:::1]) by localhost (mail0.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id FbUpsTleOMwD for ; Sun, 9 Jul 2023 13:05:32 +0200 (CEST) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail0.khirnov.net (Postfix) with ESMTPS id 54A542404EA for ; Sun, 9 Jul 2023 13:05:32 +0200 (CEST) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id 059EC3A0153 for ; Sun, 9 Jul 2023 13:05:32 +0200 (CEST) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Sun, 9 Jul 2023 13:05:29 +0200 Message-Id: <20230709110529.29490-1-anton@khirnov.net> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] lavu: add AVVideoHint API X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ce8B9/2d3C/n From: Elias Carotti Add side data type to provide hint to the video encoders about unchanged portions of each frame. Signed-off-by: Anton Khirnov Signed-off-by: Anton Khirnov --- I've made couple small changes: * rebased against current master * consistently refer to rectangles or AVVideoRect, not blocks * use size_t instead of int for AVVideoHint.nb_rects * build unconditionally -- this is public API, it must always be available regardless of what encoders are or are not available * tweaked documentation If you have no objections to the changes, I'll push this in a few days. --- doc/APIchanges | 3 ++ libavutil/Makefile | 2 + libavutil/frame.h | 10 ++++ libavutil/version.h | 2 +- libavutil/video_hint.c | 82 +++++++++++++++++++++++++++++++ libavutil/video_hint.h | 108 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 libavutil/video_hint.c create mode 100644 libavutil/video_hint.h diff --git a/doc/APIchanges b/doc/APIchanges index 27f835cfce..0cda51fdee 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -2,6 +2,9 @@ The last version increases of all libraries were on 2023-02-09 API changes, most recent first: +2023-07-xx - xxxxxxxxxx - lavu 58.15.100 - video_hint.h + Add AVVideoHint API. + 2023-07-05 - xxxxxxxxxx - lavu 58.14.100 - random_seed.h Add av_random_bytes() diff --git a/libavutil/Makefile b/libavutil/Makefile index bd9c6f9e32..7828c94dc5 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -91,6 +91,7 @@ HEADERS = adler32.h \ tea.h \ tx.h \ film_grain_params.h \ + video_hint.h ARCH_HEADERS = bswap.h \ intmath.h \ @@ -181,6 +182,7 @@ OBJS = adler32.o \ uuid.o \ version.o \ video_enc_params.o \ + video_hint.o \ film_grain_params.o \ diff --git a/libavutil/frame.h b/libavutil/frame.h index a491315f25..c0c1b23db7 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -214,6 +214,16 @@ enum AVFrameSideDataType { * Ambient viewing environment metadata, as defined by H.274. */ AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT, + + /** + * Provide encoder-specific hinting information about changed/unchanged + * portions of a frame. It can be used to pass information about which + * macroblocks can be skipped because they didn't change from the + * corresponding ones in the previous frame. This could be useful for + * applications which know this information in advance to speed up + * encoding. + */ + AV_FRAME_DATA_VIDEO_HINT, }; enum AVActiveFormatDescription { diff --git a/libavutil/version.h b/libavutil/version.h index 24af520e08..9e798b0e3f 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 58 -#define LIBAVUTIL_VERSION_MINOR 14 +#define LIBAVUTIL_VERSION_MINOR 15 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ diff --git a/libavutil/video_hint.c b/libavutil/video_hint.c new file mode 100644 index 0000000000..5730ed6cfb --- /dev/null +++ b/libavutil/video_hint.c @@ -0,0 +1,82 @@ +/* + * Copyright 2023 Elias Carotti + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include + +#include "avstring.h" +#include "frame.h" +#include "macros.h" +#include "mem.h" +#include "video_hint.h" + +AVVideoHint *av_video_hint_alloc(size_t nb_rects, + size_t *out_size) +{ + struct TestStruct { + AVVideoHint hint; + AVVideoRect rect; + }; + const size_t rect_offset = offsetof(struct TestStruct, rect); + size_t size = rect_offset; + AVVideoHint *hint; + + *out_size = 0; + if (nb_rects > (SIZE_MAX - size) / sizeof(AVVideoRect)) + return NULL; + size += sizeof(AVVideoRect) * nb_rects; + + hint = av_mallocz(size); + if (!hint) + return NULL; + + hint->nb_rects = nb_rects; + hint->rect_offset = rect_offset; + hint->rect_size = sizeof(AVVideoRect); + + *out_size = size; + + return hint; +} + +AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, + size_t nb_rects) +{ + AVVideoHint *hint; + AVBufferRef *buf; + size_t size = 0; + + hint = av_video_hint_alloc(nb_rects, &size); + if (!hint) + return NULL; + + buf = av_buffer_create((uint8_t *)hint, size, NULL, NULL, 0); + if (!buf) { + av_freep(&hint); + return NULL; + } + + if (!av_frame_new_side_data_from_buf(frame, AV_FRAME_DATA_VIDEO_HINT, buf)) { + av_buffer_unref(&buf); + return NULL; + } + + return hint; +} + diff --git a/libavutil/video_hint.h b/libavutil/video_hint.h new file mode 100644 index 0000000000..86addf5ceb --- /dev/null +++ b/libavutil/video_hint.h @@ -0,0 +1,108 @@ +/** + * Copyright 2023 Elias Carotti + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_VIDEO_HINT_H +#define AVUTIL_VIDEO_HINT_H + +#include +#include +#include "libavutil/avassert.h" +#include "libavutil/frame.h" + +typedef struct AVVideoRect { + uint32_t x, y; + uint32_t width, height; +} AVVideoRect; + +typedef enum AVVideoHintType { + /* rectangled delimit the constant areas (unchanged), default is changed */ + AV_VIDEO_HINT_CONSTANT, + + /* rectangled delimit the constant areas (changed), default is not changed */ + AV_VIDEO_HINT_CHANGED, +} AVVideoHintType; + +typedef struct AVVideoHint { + /** + * Number of AVVideoRect present. + * + * May be 0, in which case no per-rectangle information is present. In this + * case the values of rect_offset / rect_size are unspecified and should + * not be accessed. + */ + size_t nb_rects; + + /** + * Offset in bytes from the beginning of this structure at which the array + * of AVVideoRect starts. + */ + size_t rect_offset; + + /** + * Size in bytes of AVVideoRect. + */ + size_t rect_size; + + AVVideoHintType type; +} AVVideoHint; + +static av_always_inline AVVideoRect* +av_video_hint_rects(const AVVideoHint *hints) +{ + return (AVVideoRect *)((uint8_t *)hints + hints->rect_offset); +} + +static av_always_inline AVVideoRect* +av_video_hint_get_rect(const AVVideoHint *hints, size_t idx) +{ + return (AVVideoRect *)((uint8_t *)hints + hints->rect_offset + idx * hints->rect_size); +} + +/** + * Allocate memory for the AVVideoHint struct along with an nb_rects-sized + * arrays of AVVideoRect. + * + * The side data contains a list of rectangles for the portions of the frame + * which changed from the last encoded one (and the remainder are assumed to be + * changed), or, alternately (depending on the type parameter) the unchanged + * ones (and the remanining ones are those which changed). + * Macroblocks will thus be hinted either to be P_SKIP-ped or go through the + * regular encoding procedure. + * + * It's responsibility of the caller to fill the AVRects accordingly, and to set + * the proper AVVideoHintType field. + * + * @param out_size if non-NULL, the size in bytes of the resulting data array is + * written here + * + * @return newly allocated AVVideoHint struct (must be freed by the caller using + * av_free()) on success, NULL on memory allocation failure + */ +AVVideoHint *av_video_hint_alloc(size_t nb_rects, + size_t *out_size); +/** + * Same as av_video_hint_alloc(), except newly-allocated AVVideoHint is attached + * as side data of type AV_FRAME_DATA_VIDEO_HINT_INFO to frame. + */ +AVVideoHint *av_video_hint_create_side_data(AVFrame *frame, + size_t nb_rects); + + +#endif /* AVUTIL_VIDEO_HINT_H */