From patchwork Fri May 19 10:19:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Carotti, Elias" X-Patchwork-Id: 41727 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:ba91:b0:105:feb:71f2 with SMTP id fb17csp1185505pzb; Fri, 19 May 2023 03:19:26 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4Hd+wiw+5zvhXVTCc6EopT76jCuA76Ovqz+tBbP5chNhj12mmRD5QVLLPCxbK7QzKPJLQh X-Received: by 2002:a17:907:9493:b0:94e:e293:cf34 with SMTP id dm19-20020a170907949300b0094ee293cf34mr1211513ejc.6.1684491566385; Fri, 19 May 2023 03:19:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684491566; cv=none; d=google.com; s=arc-20160816; b=UNMk8CoKgv1etwjs46AbhneAeP8PQMNDoxiBN9IpvFMiPZtcBKQkl0kkOXJSbTwNs3 iQKsmg8HBzN6lxcn9yZIf+dwiGPjDWJqfimeHRY/9LlhwjVN+LPLrjAbkfF8wn6UksiB rptcI7C2A9DmqesbxhE7jJQmPINxKbH+2ehCI7ADcmAnMvGVT7IlUbaGDwL1wdoXduQy BY4pphWyMhAJNSDwUZ4sOS4ZBkgagetTP06q8HVBd6P1SWkUEgO6QrDO1/UtAVjlPQ8f 99bMA9pQmBPTlY49U39Re0XWf5sDJJ4ix5T/oA5yEvrYyU1jFgAOxr//pTmCALBVWkfs uSDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:content-language:accept-language:message-id:date :thread-index:thread-topic:to:from:dkim-signature:delivered-to; bh=HL3dMIowpWMkUVQe1LvaovuoKy+4miTlEVhyPZKqdZg=; b=eeRIp+OfG2Yz6KDSQDz+CZ0RYvAU1bR/7wgBV6x9e4LKERfc4Msaok+b2zc9emI5zC ZjBdhSh6vdl830oD/zUng5Ke7+6K+IrmiKerhWUlDEuaZCRsTcTUh0Kzejbfhae0e6g2 +25btoY5WDOiQU8VM9OIf8pVsI7WbXvlnnmZZuyFi81Jm5UX2B+t1kGLFSRBVtg8C0cf zOuGVLxnZqGq5XNX/UGxmXmlZzCqDQAP6PbmTo+sBQ/fVLr7LW8102pX4J/ZfugAzrL+ cl8HWPXK87L8rp711g6SezM3qkN/V/7MTIDKstfwp5Ve2t5k+bVU9Q35QGBcEvcCKqRi ga9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.it header.s=amazon201209 header.b=VMSaorhe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.it Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j2-20020aa7ca42000000b00510e67aa1ddsi2477978edt.658.2023.05.19.03.19.25; Fri, 19 May 2023 03:19:26 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.it header.s=amazon201209 header.b=VMSaorhe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.it Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 80F7F68BB8E; Fri, 19 May 2023 13:19:21 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-fw-9102.amazon.com (unknown [207.171.184.29]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 03BEA68C08B for ; Fri, 19 May 2023 13:19:13 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.it; i=@amazon.it; q=dns/txt; s=amazon201209; t=1684491559; x=1716027559; h=from:to:subject:date:message-id:mime-version; bh=P5871SMz6UqPbjteFD7qD458o3rlZqURaPHSLZiqKkU=; b=VMSaorheisu9x2wMbiI+SdHfsWOjiNwKVMCaxV38w81kiEpXCead9PDE GRr0gpmwKhg7x6VyNvYUfES4+MD/f7BJcnF4vziyaIh3ZzQHRJVK0EYjz xt7AjncYVY5mZ+Yrq9PhAm6eLmohESHmWDJxtdfXLbSrKngZfT2SFRZ/y Q=; X-Amazon-filename: 0001-Add-support-for-x264-s-MB_INFO.patch X-IronPort-AV: E=Sophos;i="6.00,176,1681171200"; d="scan'208";a="339937622" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO email-inbound-relay-iad-1e-m6i4x-6e7a78d7.us-east-1.amazon.com) ([10.25.36.210]) by smtp-border-fw-9102.sea19.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:19:06 +0000 Received: from EX19D014EUA002.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan2.iad.amazon.com [10.40.163.34]) by email-inbound-relay-iad-1e-m6i4x-6e7a78d7.us-east-1.amazon.com (Postfix) with ESMTPS id 03D2C80F86 for ; Fri, 19 May 2023 10:19:04 +0000 (UTC) Received: from EX19D025EUC003.ant.amazon.com (10.252.61.205) by EX19D014EUA002.ant.amazon.com (10.252.50.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.26; Fri, 19 May 2023 10:19:03 +0000 Received: from EX19D025EUC002.ant.amazon.com (10.252.61.247) by EX19D025EUC003.ant.amazon.com (10.252.61.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.26; Fri, 19 May 2023 10:19:03 +0000 Received: from EX19D025EUC002.ant.amazon.com ([fe80::c9fe:e3a8:dd4f:e087]) by EX19D025EUC002.ant.amazon.com ([fe80::c9fe:e3a8:dd4f:e087%3]) with mapi id 15.02.1118.026; Fri, 19 May 2023 10:19:03 +0000 From: "Carotti, Elias" To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH] Optimization: support for libx264's mb_info Thread-Index: AQHZijtVd7oHq6FyvEydeeIQMmGvDg== Date: Fri, 19 May 2023 10:19:03 +0000 Message-ID: <8eb9290f3c9cdae75b38c45a5d49ee624951a849.camel@amazon.it> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [10.1.213.23] MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] Optimization: support for libx264's mb_info X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: VvJYcalz+E4O Hi again, I am sending this patch again (I had missed a check for NULL), could somebody please have a look at it?  It is mainly an optimization when the encoder knows in advance that only portions of the whole frame changed and which areas actually did. The patch allows a user to pass down information to libx264 information about which parts of a frame changed (with respect to the preceding one) to be exploited as a hint for P_SKIP-ing macroblocks which didn't change.  This information is encoded in the mb_info field of the x264_param_t much like the quant_offsets which are already used for the AV_FRAME_DATA_REGIONS_OF_INTEREST side information. Best, Elias NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c index cfdd422236..55c64daafa 100644 --- a/libavcodec/libx264.c +++ b/libavcodec/libx264.c @@ -30,6 +30,7 @@ #include "libavutil/stereo3d.h" #include "libavutil/time.h" #include "libavutil/intreadwrite.h" +#include "libavutil/mb_info.h" #include "avcodec.h" #include "codec_internal.h" #include "encode.h" @@ -48,6 +49,9 @@ // from x264.h, for quant_offsets, Macroblocks are 16x16 // blocks of pixels (with respect to the luma plane) #define MB_SIZE 16 +#define MB_LSIZE 4 +#define MB_FLOOR(x) ((x) >> (MB_LSIZE)) +#define MB_CEIL(x) MB_FLOOR((x) + (MB_SIZE - 1)) typedef struct X264Opaque { #if FF_API_REORDERED_OPAQUE @@ -123,6 +127,8 @@ typedef struct X264Context { * encounter a frame with ROI side data. */ int roi_warned; + + int mb_info; } X264Context; static void X264_log(void *p, int level, const char *fmt, va_list args) @@ -295,6 +301,7 @@ static void free_picture(x264_picture_t *pic) av_free(pic->extra_sei.payloads[i].payload); av_freep(&pic->extra_sei.payloads); av_freep(&pic->prop.quant_offsets); + av_freep(&pic->prop.mb_info); pic->extra_sei.num_payloads = 0; } @@ -320,6 +327,45 @@ static enum AVPixelFormat csp_to_pixfmt(int csp) return AV_PIX_FMT_NONE; } +static int setup_mb_info(AVCodecContext *ctx, x264_picture_t *pic, + const AVFrame *frame, const uint8_t *data, + size_t size) +{ + int mb_width = (frame->width + MB_SIZE - 1) / MB_SIZE; + int mb_height = (frame->height + MB_SIZE - 1) / MB_SIZE; + const AVMBInfoRect *mbinfo_rects; + size_t mbinfo_count; + uint8_t *mbinfo; + + mbinfo_rects = (const AVMBInfoRect *)data; + mbinfo_count = size / sizeof(AVMBInfoRect); + + mbinfo = av_calloc(mb_width * mb_height, sizeof(*mbinfo)); + if (!mbinfo) + return AVERROR(ENOMEM); + + /* Sets the default as constant, i.e. P_SKIP-able, then selectively resets the flag */ + memset(mbinfo, X264_MBINFO_CONSTANT, sizeof(*mbinfo) * mb_width * mb_height); + + for (int i = 0; i < mbinfo_count; i++) { + int min_y = MB_FLOOR(mbinfo_rects->y); + int max_y = MB_CEIL(mbinfo_rects->y + mbinfo_rects->height); + int min_x = MB_FLOOR(mbinfo_rects->x); + int max_x = MB_CEIL(mbinfo_rects->x + mbinfo_rects->width); + + for (int mb_y = min_y; mb_y < max_y; ++mb_y) { + memset(mbinfo + mb_y * mb_width + min_x, 0, max_x - min_x); + } + + mbinfo_rects++; + } + + pic->prop.mb_info = mbinfo; + pic->prop.mb_info_free = av_free; + + return 0; +} + static int setup_roi(AVCodecContext *ctx, x264_picture_t *pic, int bit_depth, const AVFrame *frame, const uint8_t *data, size_t size) { @@ -404,6 +450,7 @@ static int setup_frame(AVCodecContext *ctx, const AVFrame *frame, int64_t wallclock = 0; int bit_depth, ret; AVFrameSideData *sd; + AVFrameSideData *mbinfo_sd; *ppic = NULL; if (!frame) @@ -499,6 +546,17 @@ FF_ENABLE_DEPRECATION_WARNINGS goto fail; } + mbinfo_sd = av_frame_get_side_data(frame, AV_FRAME_DATA_MB_INFO); + if (mbinfo_sd) { + int ret = setup_mb_info(ctx, pic, frame, mbinfo_sd->data, mbinfo_sd->size); + if (ret < 0) { + /* No need to fail here, this is not fatal. We just proceed with no + * mb_info and log a message */ + + av_log(ctx, AV_LOG_WARNING, "mb_info setup failure\n"); + } + } + if (x4->udu_sei) { for (int j = 0; j < frame->nb_side_data; j++) { AVFrameSideData *side_data = frame->side_data[j]; @@ -1096,6 +1154,9 @@ static av_cold int X264_init(AVCodecContext *avctx) } } + x4->params.analyse.b_mb_info = x4->mb_info; + x4->params.analyse.b_fast_pskip = 1; + // update AVCodecContext with x264 parameters avctx->has_b_frames = x4->params.i_bframe ? x4->params.i_bframe_pyramid ? 2 : 1 : 0; @@ -1305,6 +1366,7 @@ static const AVOption options[] = { { "noise_reduction", "Noise reduction", OFFSET(noise_reduction), AV_OPT_TYPE_INT, { .i64 = -1 }, INT_MIN, INT_MAX, VE }, { "udu_sei", "Use user data unregistered SEI if available", OFFSET(udu_sei), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, { "x264-params", "Override the x264 configuration using a :-separated list of key=value parameters", OFFSET(x264_params), AV_OPT_TYPE_DICT, { 0 }, 0, 0, VE }, + { "mb_info", "Set mb_info data through AVSideData, only useful when used from the API", OFFSET(mb_info), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, { NULL }, }; diff --git a/libavutil/Makefile b/libavutil/Makefile index dc9012f9a8..e99f448213 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -91,6 +91,7 @@ HEADERS = adler32.h \ tea.h \ tx.h \ film_grain_params.h \ + mb_info.h \ ARCH_HEADERS = bswap.h \ intmath.h \ @@ -196,6 +197,7 @@ OBJS-$(CONFIG_VAAPI) += hwcontext_vaapi.o OBJS-$(CONFIG_VIDEOTOOLBOX) += hwcontext_videotoolbox.o OBJS-$(CONFIG_VDPAU) += hwcontext_vdpau.o OBJS-$(CONFIG_VULKAN) += hwcontext_vulkan.o +OBJS-$(CONFIG_LIBX264) += mb_info.o OBJS-$(!CONFIG_VULKAN) += hwcontext_stub.o @@ -219,6 +221,8 @@ SKIPHEADERS-$(CONFIG_VULKAN) += hwcontext_vulkan.h vulkan.h \ vulkan_functions.h \ vulkan_loader.h +SKIPHEADERS-$(CONFIG_LIBX264) += mb_info.h + TESTPROGS = adler32 \ aes \ aes_ctr \ diff --git a/libavutil/frame.h b/libavutil/frame.h index f85d630c5c..9c0fdcf25d 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -214,6 +214,16 @@ enum AVFrameSideDataType { * Ambient viewing environment metadata, as defined by H.274. */ AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT, + + /** + * Provide macro block encoder-specific hinting information for the encoder + * processing. It can be used to pass information about which macroblock + * can be skipped because it hasn't changed from the corresponding one in + * the previous frame. This is useful for applications which know in + * advance this information to speed up real-time encoding. Currently only + * used by libx264. + */ + AV_FRAME_DATA_MB_INFO, }; enum AVActiveFormatDescription { diff --git a/libavutil/mb_info.c b/libavutil/mb_info.c new file mode 100644 index 0000000000..1e4a57b8d4 --- /dev/null +++ b/libavutil/mb_info.c @@ -0,0 +1,51 @@ +/* + * Copyright 2023 Elias Carotti + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include + +#include "avstring.h" +#include "frame.h" +#include "macros.h" +#include "mem.h" +#include "mb_info.h" + + +AVMBInfoRect *av_mb_info_create_side_data(AVFrame *frame, + AVMBInfoRect *rects, + size_t num_rects) +{ + AVFrameSideData *side_data; + AVMBInfoRect *par; + + side_data = av_frame_new_side_data(frame, + AV_FRAME_DATA_MB_INFO, + num_rects * sizeof(AVMBInfoRect)); + + if (!side_data) + return NULL; + + par = (AVMBInfoRect *)side_data->data; + + /* Just copies the rects over the newly allocated buffer */ + memcpy(par, rects, sizeof(AVMBInfoRect) * num_rects); + + return par; +} + diff --git a/libavutil/mb_info.h b/libavutil/mb_info.h new file mode 100644 index 0000000000..918cf167aa --- /dev/null +++ b/libavutil/mb_info.h @@ -0,0 +1,46 @@ +/** + * Copyright 2023 Elias Carotti + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_MB_INFO_H +#define AVUTIL_MB_INFO_H + +#include +#include +#include "libavutil/avassert.h" +#include "libavutil/frame.h" + +typedef struct _AVMBInfoRect { + uint32_t x, y; + uint32_t width, height; +} AVMBInfoRect; + +/** + * Allocate memory for a vector of AVMBInfoRect in the given AVFrame + * {@code frame} as AVFrameSideData of type AV_FRAME_DATA_MB_INFO. + * The side data contains a list of rectangles for the portions of the frame + * which changed from the last encoded one. The rest will be hinted to be + * P_SKIP-ped. Portions of the rects which are not on macroblock boundaries + * are not handled as P_SKIPS. + */ +AVMBInfoRect *av_mb_info_create_side_data(AVFrame *frame, + AVMBInfoRect *rects, + size_t num_rects); + +#endif /* AVUTIL_MB_INFO_H */