From patchwork Wed Jun 21 15:57:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Carotti, Elias" X-Patchwork-Id: 42265 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:be15:b0:121:b37c:e101 with SMTP id ge21csp2232497pzb; Wed, 21 Jun 2023 08:58:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4TNuW9QF2618ZGRM7VhdU1SKNQOJTNAbCoBuaps4XXRJXmVzp6wk+jVxELcsjPS9OaLetL X-Received: by 2002:a17:907:a42:b0:96a:48ed:5333 with SMTP id be2-20020a1709070a4200b0096a48ed5333mr14062985ejc.50.1687363082746; Wed, 21 Jun 2023 08:58:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687363082; cv=none; d=google.com; s=arc-20160816; b=nNDV3yIh2DUFb+c2QVmhuEdwE67cFKzJi0jhNf3Yn5s2/eURBAjxCwuL9sb/fS1CBF OIxmg64C1kEu8+mVFacwJpoOntn+GqqEce31BQXNgRmhoj9+aI8+0srOIPFttILoe+E4 6arlpyMgeYaMfMKK2F0uxcrJhVxC+cmY4i2s+F3v8V8+lKUST8oIvG3J3wXu6uD7N3ve aT5t0eLeemaSmO84L2dy83DWifFgCV1K9dMKeOARLx6FPb/r1oO9w2jVwOsfvdzuGRdj TRdj2pUz0CuF1KL9pI30ErFlNkDY8lFJslwDKTA/+U+LSU5hBEw8SJInKnsINBgS9an7 qNjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:content-language:accept-language:message-id:date :thread-index:thread-topic:to:from:dkim-signature:delivered-to; bh=F535BbBFA75oV4cu4oXqbuS/Anl7ii+05bFyERAtyNc=; b=b2FoB6VxPj626E1ERsrjkiDn9g70A4BvbJX/sZcl17dmxTrhrb7LYgyJft2QAXzMKM p89qOVY74T4+XGrtrq0Om5ohKsVyhNS9FshRoe+k0eQ3eCrCzSQrks9/mf4eUrxc5eq9 GXh5Xk4iTaolKwRM7xixEU28cf1YDyA1dFHslfq+kBLWjK0geVtRgjza55LyOODH6khJ D/JjEKywFj/CSIlS6dat8PDsXkyb2KqzAX/3q1r5KCItmCnBiArrFS1uYyWO54G5D+Vy nIKc4iQB+Mr7e52KOiaV2cbN75WTYiPktUoWfQVp/DLDLso68WqWN1n7V1akRZyN3auT +WqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.it header.s=amazon201209 header.b=E9Awgb6t; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.it Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id n17-20020a17090625d100b009886fc18ad6si438357ejb.885.2023.06.21.08.58.02; Wed, 21 Jun 2023 08:58:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@amazon.it header.s=amazon201209 header.b=E9Awgb6t; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.it Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2665A68C054; Wed, 21 Jun 2023 18:57:59 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp-fw-52005.amazon.com (smtp-fw-52005.amazon.com [52.119.213.156]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2AB0868BF4A for ; Wed, 21 Jun 2023 18:57:52 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.it; i=@amazon.it; q=dns/txt; s=amazon201209; t=1687363077; x=1718899077; h=from:to:subject:date:message-id:mime-version; bh=fPH237eq4Fe7upwbkV6uFUlRtHDJSflzuxDhb1yVUzY=; b=E9Awgb6tcKEXxXV65l2nNbEAIDFd6CHpyh+Pl8E54dRf7A8B1zimllWJ l97BXPWZDI0yAoo8dy/Opw0hgm3DT9MwUowcmQWYhXZNhUhKt2GXWC1cC S+3iPQ7FUD6wJ8YZqzG8Q6mPd7k3jFHfxQjful/uagW/sTywXgCAWlIx0 4=; X-Amazon-filename: 0002-lavc-libx264-add-mb_info-option.patch X-IronPort-AV: E=Sophos;i="6.00,260,1681171200"; d="scan'208,223";a="588646776" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-pdx-2a-m6i4x-3ef535ca.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-52005.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Jun 2023 15:57:49 +0000 Received: from EX19D008EUA001.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan3.pdx.amazon.com [10.236.137.198]) by email-inbound-relay-pdx-2a-m6i4x-3ef535ca.us-west-2.amazon.com (Postfix) with ESMTPS id 3CC2C60A2C for ; Wed, 21 Jun 2023 15:57:47 +0000 (UTC) Received: from EX19D025EUC003.ant.amazon.com (10.252.61.205) by EX19D008EUA001.ant.amazon.com (10.252.50.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.26; Wed, 21 Jun 2023 15:57:47 +0000 Received: from EX19D025EUC002.ant.amazon.com (10.252.61.247) by EX19D025EUC003.ant.amazon.com (10.252.61.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.26; Wed, 21 Jun 2023 15:57:46 +0000 Received: from EX19D025EUC002.ant.amazon.com ([fe80::c9fe:e3a8:dd4f:e087]) by EX19D025EUC002.ant.amazon.com ([fe80::c9fe:e3a8:dd4f:e087%3]) with mapi id 15.02.1118.026; Wed, 21 Jun 2023 15:57:46 +0000 From: "Carotti, Elias" To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH 2/2] lavc/libx264: add mb_info option Thread-Index: AQHZpFkf0wniyBZwyEaKKXbWaPmS1g== Date: Wed, 21 Jun 2023 15:57:46 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [10.1.212.11] MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] lavc/libx264: add mb_info option X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: skuh5s2mxE51 Hi all, please find the second part of the patch set. Best, Elias NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico From 2afd999f38fc3a7c3f56aba1d874de00bc9575ba Mon Sep 17 00:00:00 2001 From: Elias Carotti Date: Tue, 20 Jun 2023 19:29:08 +0200 Subject: [PATCH 2/2] lavc/libx264: add mb_info option Pass the information about unchanged parts of the frame by means of the AVVideoHint side data. --- Changelog | 1 + doc/APIchanges | 3 ++ libavcodec/libx264.c | 91 ++++++++++++++++++++++++++++++++++++++++++++ libavcodec/version.h | 2 +- 4 files changed, 96 insertions(+), 1 deletion(-) diff --git a/Changelog b/Changelog index 9cf3df8d6f..ec1b848b04 100644 --- a/Changelog +++ b/Changelog @@ -21,6 +21,7 @@ version : - Essential Video Coding frame merge bsf - bwdif_cuda filter - Microsoft RLE video encoder +- support for the P_SKIP hinting to speed up libx264 encoding version 6.0: - Radiance HDR image support diff --git a/doc/APIchanges b/doc/APIchanges index bfe04556d2..02723f5db7 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -2,6 +2,9 @@ The last version increases of all libraries were on 2023-02-09 API changes, most recent first: +2023-06-21 - xxxxxxxxxx - lavc 60.22.100 - libx264.c + Add mb_info option. + 2023-06-21 - xxxxxxxxxx - lavu 58.14.100 - video_hint.h Add AVVideoHint API. diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c index 5736f1efa7..c42e7ad39d 100644 --- a/libavcodec/libx264.c +++ b/libavcodec/libx264.c @@ -30,6 +30,7 @@ #include "libavutil/stereo3d.h" #include "libavutil/time.h" #include "libavutil/intreadwrite.h" +#include "libavutil/video_hint.h" #include "avcodec.h" #include "codec_internal.h" #include "encode.h" @@ -48,6 +49,9 @@ // from x264.h, for quant_offsets, Macroblocks are 16x16 // blocks of pixels (with respect to the luma plane) #define MB_SIZE 16 +#define MB_LSIZE 4 +#define MB_FLOOR(x) ((x) >> (MB_LSIZE)) +#define MB_CEIL(x) MB_FLOOR((x) + (MB_SIZE - 1)) typedef struct X264Opaque { #if FF_API_REORDERED_OPAQUE @@ -123,6 +127,8 @@ typedef struct X264Context { * encounter a frame with ROI side data. */ int roi_warned; + + int mb_info; } X264Context; static void X264_log(void *p, int level, const char *fmt, va_list args) @@ -295,6 +301,7 @@ static void free_picture(x264_picture_t *pic) av_free(pic->extra_sei.payloads[i].payload); av_freep(&pic->extra_sei.payloads); av_freep(&pic->prop.quant_offsets); + av_freep(&pic->prop.mb_info); pic->extra_sei.num_payloads = 0; } @@ -320,6 +327,74 @@ static enum AVPixelFormat csp_to_pixfmt(int csp) return AV_PIX_FMT_NONE; } +static void av_always_inline mbinfo_compute_changed_coords(const AVVideoRect *rect, + int *min_x, + int *max_x, + int *min_y, + int *max_y) +{ + *min_y = MB_FLOOR(rect->y); + *max_y = MB_CEIL(rect->y + rect->height); + *min_x = MB_FLOOR(rect->x); + *max_x = MB_CEIL(rect->x + rect->width); +} + +static void av_always_inline mbinfo_compute_constant_coords(const AVVideoRect *rect, + int *min_x, + int *max_x, + int *min_y, + int *max_y) +{ + *min_y = MB_CEIL(rect->y); + *max_y = MB_FLOOR(rect->y + rect->height); + *min_x = MB_CEIL(rect->x); + *max_x = MB_FLOOR(rect->x + rect->width); +} + +static int setup_mb_info(AVCodecContext *ctx, x264_picture_t *pic, + const AVFrame *frame, + const AVVideoHint *info) +{ + int mb_width = (frame->width + MB_SIZE - 1) / MB_SIZE; + int mb_height = (frame->height + MB_SIZE - 1) / MB_SIZE; + + const AVVideoRect *mbinfo_rects; + int nb_rects; + uint8_t *mbinfo; + + mbinfo_rects = (const AVVideoRect *)av_video_hint_rects(info); + nb_rects = info->nb_rects; + + mbinfo = av_calloc(mb_width * mb_height, sizeof(*mbinfo)); + if (!mbinfo) + return AVERROR(ENOMEM); + +#define COMPUTE_MBINFO(mbinfo_filler_, mbinfo_marker_, compute_coords_fn_) \ + memset(mbinfo, mbinfo_filler_, sizeof(*mbinfo) * mb_width * mb_height); \ + \ + for (int i = 0; i < nb_rects; i++) { \ + int min_x, max_x, min_y, max_y; \ + \ + compute_coords_fn_(mbinfo_rects, &min_x, &max_x, &min_y, &max_y); \ + for (int mb_y = min_y; mb_y < max_y; ++mb_y) { \ + memset(mbinfo + mb_y * mb_width + min_x, mbinfo_marker_, max_x - min_x); \ + } \ + \ + mbinfo_rects++; \ + } \ + + if (info->type == AV_VIDEO_HINT_CHANGED) { + COMPUTE_MBINFO(X264_MBINFO_CONSTANT, 0, mbinfo_compute_changed_coords); + } else /* if (info->type == AV_VIDEO_HINT_CHANGED) */ { + COMPUTE_MBINFO(0, X264_MBINFO_CONSTANT, mbinfo_compute_constant_coords); + } + + pic->prop.mb_info = mbinfo; + pic->prop.mb_info_free = av_free; + + return 0; +} + static int setup_roi(AVCodecContext *ctx, x264_picture_t *pic, int bit_depth, const AVFrame *frame, const uint8_t *data, size_t size) { @@ -404,6 +479,7 @@ static int setup_frame(AVCodecContext *ctx, const AVFrame *frame, int64_t wallclock = 0; int bit_depth, ret; AVFrameSideData *sd; + AVFrameSideData *mbinfo_sd; *ppic = NULL; if (!frame) @@ -499,6 +575,17 @@ FF_ENABLE_DEPRECATION_WARNINGS goto fail; } + mbinfo_sd = av_frame_get_side_data(frame, AV_FRAME_DATA_VIDEO_HINT); + if (mbinfo_sd) { + int ret = setup_mb_info(ctx, pic, frame, (const AVVideoHint *)mbinfo_sd->data); + if (ret < 0) { + /* No need to fail here, this is not fatal. We just proceed with no + * mb_info and log a message */ + + av_log(ctx, AV_LOG_WARNING, "setup_mb_info failed with error: %s\n", av_err2str(ret)); + } + } + if (x4->udu_sei) { for (int j = 0; j < frame->nb_side_data; j++) { AVFrameSideData *side_data = frame->side_data[j]; @@ -1102,6 +1189,9 @@ FF_ENABLE_DEPRECATION_WARNINGS } } + x4->params.analyse.b_mb_info = x4->mb_info; + x4->params.analyse.b_fast_pskip = 1; + // update AVCodecContext with x264 parameters avctx->has_b_frames = x4->params.i_bframe ? x4->params.i_bframe_pyramid ? 2 : 1 : 0; @@ -1311,6 +1401,7 @@ static const AVOption options[] = { { "noise_reduction", "Noise reduction", OFFSET(noise_reduction), AV_OPT_TYPE_INT, { .i64 = -1 }, INT_MIN, INT_MAX, VE }, { "udu_sei", "Use user data unregistered SEI if available", OFFSET(udu_sei), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, { "x264-params", "Override the x264 configuration using a :-separated list of key=value parameters", OFFSET(x264_params), AV_OPT_TYPE_DICT, { 0 }, 0, 0, VE }, + { "mb_info", "Set mb_info data through AVSideData, only useful when used from the API", OFFSET(mb_info), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE }, { NULL }, }; diff --git a/libavcodec/version.h b/libavcodec/version.h index da6f3a84ac..9411511e04 100644 --- a/libavcodec/version.h +++ b/libavcodec/version.h @@ -29,7 +29,7 @@ #include "version_major.h" -#define LIBAVCODEC_VERSION_MINOR 21 +#define LIBAVCODEC_VERSION_MINOR 22 #define LIBAVCODEC_VERSION_MICRO 100 #define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \ -- 2.34.1