From patchwork Fri Jun 10 10:11:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Carotti, Elias" X-Patchwork-Id: 36141 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp855486pzj; Fri, 10 Jun 2022 03:12:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw8pVbz9GJsMeiLYFuc7IXYbLVp3I5Gfy1JmK6GeejGGZQHCriXVSA7c0KB1QMdYxJIBRNx X-Received: by 2002:aa7:cb05:0:b0:431:95b9:6d31 with SMTP id s5-20020aa7cb05000000b0043195b96d31mr19319684edt.56.1654855962120; Fri, 10 Jun 2022 03:12:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654855962; cv=none; d=google.com; s=arc-20160816; b=W/lYA0yro+hTRZhI2uTl9NWxzl+KGMxqxj3kw4IdOSbav77SyD4CmltfE9hUICGmwo vWlRp349wu/vnbttSFj0fwaZIcEehRfIU8J+EguYhbpqDkpn7fx44TysI5Ifq7qeJ8pC wzx4WLc6iW1f9ur+Vq+HsWKv+q+4+KoY3koOcUoACg9ZSfpBextwFfNRRsQyXHfYybCd fLDhHtadYmZVLz+/MRMzg8a79BYEVPQKFpBw3mIWpCv8j5J3OEYxFYy3vxQ3lKz6wJiZ JO/TOElPug/AhTXRDWBdedK7gu/mM87q17xG7oiUZ82a3jLr2v5O88QbKKnppAg/sLKh SIQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:content-language:accept-language:message-id:date :thread-index:thread-topic:to:from:dkim-signature:delivered-to; bh=C+w0pN8HYHs19tlR1Y6L4ekFp7rdVDDn8AjG5S4x1Dg=; b=bv6y6nm1daZPJP+qZOEExwZDkms9KPlOZ8P97mf+Qv94oaJwXgjqEPaHNiFM8KA1n7 WGqZmyEYuVCDEmzh6vLPOxuyiwlmlUkjiLE68/BYHi9KZVOYaGg4o1U9o2PBolLGKxqQ Sd3lAQ1ro6nTV+naWjhW5CAn4by65V3fk8udAel+J/SiCHmXpwIHPaaWaeYZ8BS8Ck6V zuac5iwq39FqdGGgoXaeo/+af0m2ZNEr+yeDT3ur06z9YZ9W1Rh635j/bml9ajH3ip7F XDhAEDwsffZzzw8HxksoWICUTab5VUq9rMTiIRCOAvnEo7uqQqlEV4nIYUgqcBQ0KjrZ 8tfw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.it header.s=amazon201209 header.b=m1tKMf88; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.it Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p7-20020a17090653c700b00711ad88fde8si7285668ejo.369.2022.06.10.03.12.41; Fri, 10 Jun 2022 03:12:42 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.it header.s=amazon201209 header.b=m1tKMf88; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.it Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 94FB768B788; Fri, 10 Jun 2022 13:12:38 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-fw-2101.amazon.com (smtp-fw-2101.amazon.com [72.21.196.25]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8DC4D68B745 for ; Fri, 10 Jun 2022 13:12:31 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.it; i=@amazon.it; q=dns/txt; s=amazon201209; t=1654855957; x=1686391957; h=from:to:subject:date:message-id:mime-version; bh=0lN1F3IiAP66X8xrc+8oQlf8TT7CLN1KRw5QidzBO4M=; b=m1tKMf88M4iFPLbvjf6DpFwZAA5oXSsJ6W/xBSIWbnJsOybYR/xdAeGk Dq0kh9swwhvezwLZhTAuTgdSXwGJLsQEq9Y6hWr9AdIPTLhX3S5qVEMev ddMiSYPw0kfJsb5+efh4OTY1jrgw+MnZfZNRspuV0CQBat59hRexOUj/L g=; X-Amazon-filename: 0001-Added-support-for-MB_INFO.patch X-IronPort-AV: E=Sophos;i="5.91,288,1647302400"; d="scan'208,223";a="206566201" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-iad-1d-1c3c2014.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-2101.iad2.amazon.com with ESMTP; 10 Jun 2022 10:12:18 +0000 Received: from EX13D25EUA002.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan2.iad.amazon.com [10.40.163.34]) by email-inbound-relay-iad-1d-1c3c2014.us-east-1.amazon.com (Postfix) with ESMTPS id BEE9FCABCE for ; Fri, 10 Jun 2022 10:12:17 +0000 (UTC) Received: from EX13D25EUA003.ant.amazon.com (10.43.165.117) by EX13D25EUA002.ant.amazon.com (10.43.165.40) with Microsoft SMTP Server (TLS) id 15.0.1497.36; Fri, 10 Jun 2022 10:11:19 +0000 Received: from EX13D25EUA003.ant.amazon.com ([10.43.165.117]) by EX13D25EUA003.ant.amazon.com ([10.43.165.117]) with mapi id 15.00.1497.036; Fri, 10 Jun 2022 10:11:19 +0000 From: "Carotti, Elias" To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH] Added support for MB_INFO Thread-Index: AQHYfLJt5ZoAjIylI0i9kT+cl/Xc0A== Date: Fri, 10 Jun 2022 10:11:19 +0000 Message-ID: <9642852e9da11714c8643833562b6e86133ce1a1.camel@amazon.it> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.43.162.158] MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] Added support for MB_INFO X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: AVaxyX6RjAXP Hi, patch attached to add support for passing down to libx264 information about which macroblock could be eligible for being coded as P_SKIP. NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico From b658ba5e34afeb59cd07b4d4110ddfce749847b0 Mon Sep 17 00:00:00 2001 From: Elias Carotti Date: Thu, 26 May 2022 21:17:43 +0000 Subject: [PATCH] Added support for MB_INFO --- doc/APIchanges | 4 ++++ libavcodec/libx264.c | 22 +++++++++++++++++++ libavutil/Makefile | 4 ++++ libavutil/frame.h | 10 +++++++++ libavutil/mb_info.c | 43 +++++++++++++++++++++++++++++++++++++ libavutil/mb_info.h | 50 ++++++++++++++++++++++++++++++++++++++++++++ libavutil/version.h | 2 +- 7 files changed, 134 insertions(+), 1 deletion(-) create mode 100644 libavutil/mb_info.c create mode 100644 libavutil/mb_info.h diff --git a/doc/APIchanges b/doc/APIchanges index 337f1466d8..57dbf65551 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -14,6 +14,10 @@ libavutil: 2021-04-27 API changes, most recent first: +2022-06-08 - xxxxxxxxx - lavu 57.27.100 - frame.h + Add AV_FRAME_DATA_MB_INFO to AVFrameSideDataType and the supporting AVMBInfo + API. + 2022-05-23 - xxxxxxxxx - lavu 57.25.100 - avutil.h Deprecate av_fopen_utf8() without replacement. diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c index 4ce3791ae8..c4ac06c428 100644 --- a/libavcodec/libx264.c +++ b/libavcodec/libx264.c @@ -29,6 +29,7 @@ #include "libavutil/stereo3d.h" #include "libavutil/time.h" #include "libavutil/intreadwrite.h" +#include "libavutil/mb_info.h" #include "avcodec.h" #include "codec_internal.h" #include "encode.h" @@ -119,6 +120,8 @@ typedef struct X264Context { * encounter a frame with ROI side data. */ int roi_warned; + + int mb_info; } X264Context; static void X264_log(void *p, int level, const char *fmt, va_list args) @@ -495,6 +498,20 @@ static int X264_frame(AVCodecContext *ctx, AVPacket *pkt, const AVFrame *frame, } } + if (frame && x4->mb_info) { + AVFrameSideData *mbinfo_sd = av_frame_get_side_data(frame, AV_FRAME_DATA_MB_INFO); + + if (mbinfo_sd) { + AVMBInfo *par = (AVMBInfo *)mbinfo_sd->data; + + x4->pic.prop.mb_info = par->mb_info; + x4->pic.prop.mb_info_free = par->mb_info_free; + } else if (!mbinfo_sd || !mbinfo_sd->data) { + av_log(ctx, AV_LOG_WARNING, + "mb_info flag set but no actual MB info was provided\n"); + } + } + do { if (x264_encoder_encode(x4->enc, &nal, &nnal, frame? &x4->pic: NULL, &pic_out) < 0) return AVERROR_EXTERNAL; @@ -969,6 +986,10 @@ static av_cold int X264_init(AVCodecContext *avctx) } } + x4->params.analyse.b_mb_info = x4->mb_info; + x4->params.analyse.b_fast_pskip = 1; + printf("MB info is: %s\n", x4->mb_info ? "enabled" : "disabled"); + // update AVCodecContext with x264 parameters avctx->has_b_frames = x4->params.i_bframe ? x4->params.i_bframe_pyramid ? 2 : 1 : 0; @@ -1176,6 +1197,7 @@ static const AVOption options[] = { { "noise_reduction", "Noise reduction", OFFSET(noise_reduction), AV_OPT_TYPE_INT, { .i64 = -1 }, INT_MIN, INT_MAX, VE }, { "udu_sei", "Use user data unregistered SEI if available", OFFSET(udu_sei), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, { "x264-params", "Override the x264 configuration using a :-separated list of key=value parameters", OFFSET(x264_params), AV_OPT_TYPE_DICT, { 0 }, 0, 0, VE }, + { "mb_info", "Set mb_info data through AVSideData, only useful when used from the API", OFFSET(mb_info), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, { NULL }, }; diff --git a/libavutil/Makefile b/libavutil/Makefile index 74d21a8103..4890da77f0 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -89,6 +89,7 @@ HEADERS = adler32.h \ tea.h \ tx.h \ film_grain_params.h \ + mb_info.h \ ARCH_HEADERS = bswap.h \ intmath.h \ @@ -193,6 +194,7 @@ OBJS-$(CONFIG_VAAPI) += hwcontext_vaapi.o OBJS-$(CONFIG_VIDEOTOOLBOX) += hwcontext_videotoolbox.o OBJS-$(CONFIG_VDPAU) += hwcontext_vdpau.o OBJS-$(CONFIG_VULKAN) += hwcontext_vulkan.o +OBJS-$(CONFIG_LIBX264) += mb_info.o OBJS += $(COMPAT_OBJS:%=../compat/%) @@ -214,6 +216,8 @@ SKIPHEADERS-$(CONFIG_VULKAN) += hwcontext_vulkan.h vulkan.h \ vulkan_functions.h \ vulkan_loader.h +SKIPHEADERS-$(CONFIG_LIBX264) += mb_info.h + TESTPROGS = adler32 \ aes \ aes_ctr \ diff --git a/libavutil/frame.h b/libavutil/frame.h index 33fac2054c..08c84a1b63 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -209,6 +209,16 @@ enum AVFrameSideDataType { * volume transform - CUVA 005.1-2021. */ AV_FRAME_DATA_DYNAMIC_HDR_VIVID, + + /** + * Provide macro block encoder-specific hinting information for the encoder + * processing. It can be used to pass information about which macroblock + * can be skipped because it hasn't changed from the corresponding one in + * the previous frame. This is useful for applications which know in + * advance this information to speed up real-time encoding. Currently only + * used by libx264. + */ + AV_FRAME_DATA_MB_INFO, }; enum AVActiveFormatDescription { diff --git a/libavutil/mb_info.c b/libavutil/mb_info.c new file mode 100644 index 0000000000..d27d8dc412 --- /dev/null +++ b/libavutil/mb_info.c @@ -0,0 +1,43 @@ +/* + * Copyright (c) 2022 Elias Carotti eliascrt _AT_ amazon.it + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include + +#include "avstring.h" +#include "frame.h" +#include "macros.h" +#include "mem.h" +#include "mb_info.h" + + +AVMBInfo *av_mb_info_create_side_data(AVFrame *frame, uint8_t* mb_info, + void (*mb_info_free)(void *)) +{ + AVFrameSideData *side_data = av_frame_new_side_data(frame, + AV_FRAME_DATA_MB_INFO, + sizeof(AVMBInfo)); + AVMBInfo *par = (AVMBInfo *)side_data->data; + + par->mb_info = mb_info; + par->mb_info_free = mb_info_free; + + return par; +} + diff --git a/libavutil/mb_info.h b/libavutil/mb_info.h new file mode 100644 index 0000000000..007f276dc4 --- /dev/null +++ b/libavutil/mb_info.h @@ -0,0 +1,50 @@ +/** + * Copyright (c) 2022 Elias Carotti eliascrt _AT_ amazon.it + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_MB_INFO_H +#define AVUTIL_MB_INFO_H + +#include +#include + +#include "libavutil/avassert.h" +#include "libavutil/frame.h" + + +typedef struct AVMBInfo { + /** + * The actual mb_info data: one uint8_t per macroblock in raster-scan order. + * Currently the only flag defined in x264.h is X264_MB_INFO_CONSTANT + */ + uint8_t *mb_info; + + /* An optional pointer (may be NULL) to a de-allocator for the mb_info data */ + void (*mb_info_free)(void *); +} AVMBInfo; + +/** + * Allocate memory for AVMBInfo in the given AVFrame {@code frame} + * as AVFrameSideData of type AV_FRAME_DATA_MB_INFO + * and initializes the variables. + */ +AVMBInfo *av_mb_info_create_side_data(AVFrame *frame, uint8_t* mb_info, + void (*mb_info_free)(void *)); + +#endif /* AVUTIL_MB_INFO_H */ diff --git a/libavutil/version.h b/libavutil/version.h index 2c7f4f6b37..2e9e02dda8 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 57 -#define LIBAVUTIL_VERSION_MINOR 26 +#define LIBAVUTIL_VERSION_MINOR 27 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ -- 2.25.1