From patchwork Sun Nov 20 02:59:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Dong, Ruijing" X-Patchwork-Id: 39344 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp1341404pzb; Sat, 19 Nov 2022 19:00:20 -0800 (PST) X-Google-Smtp-Source: AA0mqf728EyZhGAU7DCbCNBaTAALtDHSdm83VICxQPdJuzwrR2tlpMwpFCKiEYGY/TYPi2nC7tjS X-Received: by 2002:a05:6402:389:b0:459:2515:b27b with SMTP id o9-20020a056402038900b004592515b27bmr11475300edv.338.1668913220375; Sat, 19 Nov 2022 19:00:20 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id h4-20020a1709063c0400b007add0123aacsi4829141ejg.1006.2022.11.19.19.00.19; Sat, 19 Nov 2022 19:00:20 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@amd.com header.s=selector1 header.b=lsMBs84X; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 34BCB68A778; Sun, 20 Nov 2022 05:00:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2088.outbound.protection.outlook.com [40.107.101.88]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4ED6D68AFB5 for ; Sun, 20 Nov 2022 05:00:09 +0200 (EET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dM6A+ad9hQdE6sB1EzEDOL8ng062GOsmNe9+UjRdLu2YVVGwkZnIfXj2IahyXfL9anW+IQGK/iQZrTxjkyO1hhh18ADydUs3/kQ/g0qKtH7ZKvXAqSgXrQsYGla/LgZbENoQ4gVB1/5/xPstirDOc57pFVtOnh1TQWyRxK4cHSgtOCFgZXQYyAvyD5FdGQFwnFAxyJRLcijD240hzbV1bQlG7NWkG3GU1bbCRYO/GDLHc4bu1r4f5tRd/d+w0zd7lAJOEAau1RPTr8HWtMeEqH/U2JM1HgLlYDLtEWk2oROAI1xy6ndp3oQW5aFDSEAJSbonEtihLZa0n4jW2Z1N/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AzsVHY9BlIJBttr6vmmcFL45HrMzH84swUmHeJEUAqo=; b=nqYQIZg+pX9wfXXB/kZzASxXxe2dKebDxkqNd1CzQ2RIMFWlmTlmiBjzGPQf99eLGNEe8FVjQEA+DS6xTBhOxhbqby4UchByJImCuTO5TiJhXMhew1rzfEviCUniGIMO53oFji3aJ629nK4lwTkji+4ZQsD2q4XZent0EUAdX23OqBYiOK4bWz35LB5NwnAgpaDO4Pum7ET5aI2AyYLtcTdfB6B13VdthghlFo8c8j/0cMh3Zo+tJpHng94mVoHxEQ3PVSdDfgNv4SRngJXU/SDhW1XyoQEBTWMstGr8EtDa+Ck8TqGGwao5SP0K2TaZ2Kuuzf6eHo6SNtApEt5vCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=ffmpeg.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AzsVHY9BlIJBttr6vmmcFL45HrMzH84swUmHeJEUAqo=; b=lsMBs84XOhH0t9ZgvTwZW2Nl8alszzpgBV2hCz7QA8dQMU1u6CpzogjEMiSxY19R6Ya0UkBWDUIUOnRo54J80nqZ6lB051e0+i+OqRWY+viklWD0Iuvr3aRRui8gDL1XyjaIbLsQ4UFNGtafv1YSIPhZ9NRSKzMxwESRVHtOnSo= Received: from BN0PR02CA0032.namprd02.prod.outlook.com (2603:10b6:408:e5::7) by LV2PR12MB5797.namprd12.prod.outlook.com (2603:10b6:408:17b::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5834.11; Sun, 20 Nov 2022 03:00:01 +0000 Received: from BN8NAM11FT094.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e5:cafe::c) by BN0PR02CA0032.outlook.office365.com (2603:10b6:408:e5::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5834.9 via Frontend Transport; Sun, 20 Nov 2022 03:00:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT094.mail.protection.outlook.com (10.13.176.131) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5834.8 via Frontend Transport; Sun, 20 Nov 2022 03:00:01 +0000 Received: from ruijing-ubuntu-pc.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Sat, 19 Nov 2022 20:59:54 -0600 From: Ruijing Dong To: Date: Sat, 19 Nov 2022 21:59:14 -0500 Message-ID: <20221120025914.39732-1-ruijing.dong@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221118153422.67632-1-ruijing.dong@amd.com> References: <20221118153422.67632-1-ruijing.dong@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT094:EE_|LV2PR12MB5797:EE_ X-MS-Office365-Filtering-Correlation-Id: 00fc7c1b-bd65-4a13-d773-08dacaa350da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PeB7Q5sNfA9LREqYfkZi6hi1VeLJXhEEgXIZJLK+Y3qP4HKlFlZchgj/CiAzydsCUFnv8XtKqR/vAuOgcX1q5YJofCrtWcB11LGGSvPnM8WEQmu+nzykmLTRZiUlImCsBVAbFhfvTGEg5PCPnGpWImmrdo3KSH+e1sE909oNMIhcVdCh28M++NRBvGKec37+gqKCjcB+d1aXhbdSrxoIgjYfGi69G3evfDIZ0WOyRgBGmLbFX0WE17LuQzDb1hJoxwPThV7nBr+Z9yOKChs7PWqpy0/NdKe22gbI74Y42v+tGLg838ZbqWCMFuYLUkeB8GkRQAnZBHmbS/g/2hi4Se70oV8Hetz6/iDNUZ+AeGcqt46d10qhMkVnMXizoEDm+qFVTyjL4fA6zAWKfXIx3+M4sktpgAW7uIXhRInGbVYjnVMX793VffxNYixkWnjrQ1gnJ7nfp+fjfbT7zhLO5GAlu0cPCOx11xTw0yQ68ByTcgvoBkoTMXbPkz2c+d3p1fTkIyvzfbIJ0YCTj+bOrnDcixuMAXfcmLX1GMjyVHKMEw6oLE84RG64uGo2c3Lrn7PYchPy/QTmJb08PP0dXlaWCGs0+mvfvqNbAd7Smf6yWj2Zu281XxmU0C4D3irn2dCDLb3Sj3QXHvXmVoDSxhJP88hHksp+OvjSFuiOOXaP69TPu8Ws1oa0lL1+RqG/ X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230022)(4636009)(376002)(39860400002)(396003)(136003)(346002)(451199015)(36840700001)(46966006)(40470700004)(6666004)(7696005)(40460700003)(8936002)(44832011)(36756003)(41300700001)(186003)(966005)(478600001)(5660300002)(26005)(2906002)(82310400005)(356005)(81166007)(36860700001)(82740400003)(2616005)(1076003)(47076005)(16526019)(426003)(336012)(83380400001)(86362001)(40480700001)(70586007)(316002)(70206006)(6916009)(54906003)(8676002)(4326008)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Nov 2022 03:00:01.0877 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 00fc7c1b-bd65-4a13-d773-08dacaa350da X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT094.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5797 Subject: [FFmpeg-devel] [PATCH v2] avcodec/av1_vaapi: add direct film grain mode X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: ruijing.dong@amd.com, fei.w.wang@intel.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8sPDu2GFfRl6 Adding direct film grain mode for av1 decoder, which outputs alongside film grain. AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN is the new flag introduced to enable this path. issue: By using AMD av1 decoder via VAAPI, when used with film grain content, the output displays black screen with incorrect frame order. The issue being discussed in here: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6903#note_1613807 example: This flag can be used in ffmpeg command: ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_flags 8 -i input_with_film_gram.obu output_with_film_grain.yuv Signed-off-by: Ruijing Dong --- update: add new option direct_film_grain in optons_table.h libavcodec/avcodec.h | 17 +++++++++++++++++ libavcodec/options_table.h | 1 + libavcodec/vaapi_av1.c | 6 ++++-- 3 files changed, 22 insertions(+), 2 deletions(-) -- 2.25.1 diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index 3edd8e2636..9420e7820d 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -2253,6 +2253,23 @@ typedef struct AVHWAccel { */ #define AV_HWACCEL_FLAG_ALLOW_PROFILE_MISMATCH (1 << 2) +/** + * The film grain synthesis could be seperate from decoding process. + * The downstream device would apply the film grain parameters seperately. + * The desired film grain parameters can be found in SEI section in H264 + * or H265 bitstreams. + * + * In AV1, film grain is mandatorily specified, av1 decoders like AMD + * av1 decoder process film grain content internally, and the output + * includes applied film grain. For the purpose of supporting these av1 + * decoders, this flag needs to be utilized. + * + * @warning If the stream has no film grain content, this flag will + * be ignored in the supported av1 decoders. It is advised + * that this flag should only be used in av1 decoders + * that support it. + */ +#define AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN (1 << 3) /** * @} */ diff --git a/libavcodec/options_table.h b/libavcodec/options_table.h index cd02f5096f..0302f89280 100644 --- a/libavcodec/options_table.h +++ b/libavcodec/options_table.h @@ -399,6 +399,7 @@ static const AVOption avcodec_options[] = { {"ignore_level", "ignore level even if the codec level used is unknown or higher than the maximum supported level reported by the hardware driver", 0, AV_OPT_TYPE_CONST, { .i64 = AV_HWACCEL_FLAG_IGNORE_LEVEL }, INT_MIN, INT_MAX, V | D, "hwaccel_flags" }, {"allow_high_depth", "allow to output YUV pixel formats with a different chroma sampling than 4:2:0 and/or other than 8 bits per component", 0, AV_OPT_TYPE_CONST, {.i64 = AV_HWACCEL_FLAG_ALLOW_HIGH_DEPTH }, INT_MIN, INT_MAX, V | D, "hwaccel_flags"}, {"allow_profile_mismatch", "attempt to decode anyway if HW accelerated decoder's supported profiles do not exactly match the stream", 0, AV_OPT_TYPE_CONST, {.i64 = AV_HWACCEL_FLAG_ALLOW_PROFILE_MISMATCH }, INT_MIN, INT_MAX, V | D, "hwaccel_flags"}, +{"direct_film_grain", "allow decoder to directly apply film grain to the output", 0, AV_OPT_TYPE_CONST, {.i64 = AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN }, INT_MIN, INT_MAX, V | D, "hwaccel_flags"}, {"extra_hw_frames", "Number of extra hardware frames to allocate for the user", OFFSET(extra_hw_frames), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, V|D }, {"discard_damaged_percentage", "Percentage of damaged samples to discard a frame", OFFSET(discard_damaged_percentage), AV_OPT_TYPE_INT, {.i64 = 95 }, 0, 100, V|D }, {NULL}, diff --git a/libavcodec/vaapi_av1.c b/libavcodec/vaapi_av1.c index d0339b2705..6db910f2bf 100644 --- a/libavcodec/vaapi_av1.c +++ b/libavcodec/vaapi_av1.c @@ -127,6 +127,7 @@ static int vaapi_av1_start_frame(AVCodecContext *avctx, int8_t bit_depth_idx; int err = 0; int apply_grain = !(avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) && film_grain->apply_grain; + int direct_film_grain = avctx->hwaccel_flags & AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN; uint8_t remap_lr_type[4] = {AV1_RESTORE_NONE, AV1_RESTORE_SWITCHABLE, AV1_RESTORE_WIENER, AV1_RESTORE_SGRPROJ}; uint8_t segmentation_feature_signed[AV1_SEG_LVL_MAX] = {1, 1, 1, 1, 1, 0, 0, 0}; uint8_t segmentation_feature_max[AV1_SEG_LVL_MAX] = {255, AV1_MAX_LOOP_FILTER, @@ -136,7 +137,7 @@ static int vaapi_av1_start_frame(AVCodecContext *avctx, if (bit_depth_idx < 0) goto fail; - if (apply_grain) { + if (apply_grain && !direct_film_grain) { if (ctx->tmp_frame->buf[0]) ff_thread_release_buffer(avctx, ctx->tmp_frame); err = ff_thread_get_buffer(avctx, ctx->tmp_frame, AV_GET_BUFFER_FLAG_REF); @@ -375,6 +376,7 @@ static int vaapi_av1_end_frame(AVCodecContext *avctx) VAAPIAV1DecContext *ctx = avctx->internal->hwaccel_priv_data; int apply_grain = !(avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) && film_grain->apply_grain; + int direct_film_grain = avctx->hwaccel_flags & AV_HWACCEL_FLAG_DIRECT_FILM_GRAIN; int ret; ret = ff_vaapi_decode_issue(avctx, pic); if (ret < 0) @@ -385,7 +387,7 @@ static int vaapi_av1_end_frame(AVCodecContext *avctx) if (ctx->ref_tab[i].frame->buf[0]) ff_thread_release_buffer(avctx, ctx->ref_tab[i].frame); - if (apply_grain) { + if (apply_grain && !direct_film_grain) { ret = av_frame_ref(ctx->ref_tab[i].frame, ctx->tmp_frame); if (ret < 0) return ret;