From patchwork Fri Mar 11 09:00:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Fei W" X-Patchwork-Id: 34681 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:3486:0:0:0:0 with SMTP id ek6csp1511673nkb; Fri, 11 Mar 2022 01:08:37 -0800 (PST) X-Google-Smtp-Source: ABdhPJzbRcd9LqsX/qlthkf3nfJ6MDPo3xrnujfLtBmjsMwpnMNIxWmrRFlAlygsGjTHbkyWCCd9 X-Received: by 2002:a05:6402:1941:b0:413:2822:9c8 with SMTP id f1-20020a056402194100b00413282209c8mr8054398edz.13.1646989716892; Fri, 11 Mar 2022 01:08:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1646989716; cv=none; d=google.com; s=arc-20160816; b=B7jcuROo1Un1mpS3Gmg7reXt/O4gHBJRo/nj/gpfy+SGtEq5sXiU9Pqu826jsAcl7v VidQk/EGaij58tIVws0c3XW0S7I4B5vSo7KpeI07Z5Nfm/tPAwnhg1uNOKv04lwnKU8Z opb45bLo/1OuNQn3r2CPmQOeJVTBcYGHsn3CFoefQ8nh4FM34Vt58l2Bxq45u/Hbt5Ci tY1qaCEKJtHa2KCuS9LVU2emM7Ef/SN2DnAMlJC6sV5sSmY+tEzKK+QzzQLoE5UWXar2 0G+IujVXFNT7chmJa/4CjepCrAEKJ0GSZNjCle4cBPQny8SJHO9w+3Hc6TdjnmbJEYio h4WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=BXeH/6yjdkS7EmCfBqJVYc+03gVJci0O9BI5RJzCQ1g=; b=xkGUpDYdGypqhWHZ0Nj7jM+VPNH9AYtMOhxYyHizj9YP3YT4hFM20hZUPEm0feyuTM OpW6B6N9yP/0Jq08tonoSt+pyvaOqaM4Ft5iwkkvms2H+GYjKo5re6i1FckeCSOOjKr9 D63TPJm/xC98MBVBZs6ZaMGVaG3c32KO+iZ2b2T6l6j88hGuArL/yBVdSJvs4kumahIW dmTKrBY96WA6FV626KC755YeVeUE3xzjKLcEEpTBJYAWEttJq9nZ9UOk7ielWGQlh33z Ck95CUHbrFvhbWM2NvxGEC3ZbYS8Vg5n/7+D+OovfOl4SXhEsdjqt+8Ank9gEeQyrnda MAxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=eSyZ0GMe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q8-20020a170906770800b006b7faebc433si4480408ejm.853.2022.03.11.01.08.33; Fri, 11 Mar 2022 01:08:36 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=eSyZ0GMe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BCCCC68A61A; Fri, 11 Mar 2022 11:08:26 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D2F5168B279 for ; Fri, 11 Mar 2022 11:08:19 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646989705; x=1678525705; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=TSOKZTYpzwP+aZzGn0Ln+8YNXEsOz9rGM6GgoqGz2EA=; b=eSyZ0GMe502FIv5uDDXpZxhkIZBCqPQEolCPMF4p0lmOhMDBGYyy1m8A UsVi+3nWbjx6exT42FotEjXnx8EYQtL7SkS6weweG3aNOAZt9oIeWu51p qKCfJpQvgLuFRlYXxlL3kDTBb5B8c/AMfrWGZo9QowPa09iwWZeQmfmKg WsbJVg0aAWSU0ZIYBbPX8mh0x/c0Aq+zBR5PJSGmEdUOgi+8kWgW1lPAc wlt/NnLq1tCqBXa5WaBv+p9wNwtMPtMCBPFoaks6Sy4vOs0cbvTldqFiq d4YCf7m5Qs/9J3PbLe1nrXrl6UFZMyGrmrrKe9IaMevzdo+3RwKB2lfsu w==; X-IronPort-AV: E=McAfee;i="6200,9189,10282"; a="255483333" X-IronPort-AV: E=Sophos;i="5.90,173,1643702400"; d="scan'208";a="255483333" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Mar 2022 01:06:40 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,173,1643702400"; d="scan'208";a="644872426" Received: from t.sh.intel.com ([10.239.159.147]) by orsmga004.jf.intel.com with ESMTP; 11 Mar 2022 01:06:39 -0800 From: Fei Wang To: ffmpeg-devel@ffmpeg.org Date: Fri, 11 Mar 2022 17:00:43 +0800 Message-Id: <20220311090046.262777-1-fei.w.wang@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 1/4] lavc/vaapi_encode_h265: Add GPB frame support for hevc_vaapi X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Fei Wang , Linjie Fu Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ryuKYabL5NFg From: Linjie Fu Use GPB frames to replace regular P/B frames if backend driver does not support it. - GPB: Generalized P and B picture. Regular P/B frames replaced by B frames with previous-predict only, L0 == L1. Normal B frames still have 2 different ref_lists and allow bi-prediction Signed-off-by: Linjie Fu Signed-off-by: Fei Wang --- update: 1. Add b to gpb. 2. Optimise debug message. libavcodec/vaapi_encode.c | 74 +++++++++++++++++++++++++++++++--- libavcodec/vaapi_encode.h | 2 + libavcodec/vaapi_encode_h265.c | 24 ++++++++++- 3 files changed, 93 insertions(+), 7 deletions(-) diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c index 3bf379b1a0..bdba9726b2 100644 --- a/libavcodec/vaapi_encode.c +++ b/libavcodec/vaapi_encode.c @@ -848,9 +848,13 @@ static void vaapi_encode_set_b_pictures(AVCodecContext *avctx, pic->b_depth = current_depth; vaapi_encode_add_ref(avctx, pic, start, 1, 1, 0); - vaapi_encode_add_ref(avctx, pic, end, 1, 1, 0); vaapi_encode_add_ref(avctx, pic, prev, 0, 0, 1); + if (!ctx->b_to_gpb) + vaapi_encode_add_ref(avctx, pic, end, 1, 1, 0); + else + vaapi_encode_add_ref(avctx, pic, end, 0, 1, 0); + for (ref = end->refs[1]; ref; ref = ref->refs[1]) vaapi_encode_add_ref(avctx, pic, ref, 0, 1, 0); } @@ -871,8 +875,11 @@ static void vaapi_encode_set_b_pictures(AVCodecContext *avctx, vaapi_encode_add_ref(avctx, pic, pic, 0, 1, 0); vaapi_encode_add_ref(avctx, pic, start, 1, 1, 0); - vaapi_encode_add_ref(avctx, pic, end, 1, 1, 0); vaapi_encode_add_ref(avctx, pic, prev, 0, 0, 1); + if (!ctx->b_to_gpb) + vaapi_encode_add_ref(avctx, pic, end, 1, 1, 0); + else + vaapi_encode_add_ref(avctx, pic, end, 0, 1, 0); for (ref = end->refs[1]; ref; ref = ref->refs[1]) vaapi_encode_add_ref(avctx, pic, ref, 0, 1, 0); @@ -1845,6 +1852,51 @@ static av_cold int vaapi_encode_init_gop_structure(AVCodecContext *avctx) ref_l1 = attr.value >> 16 & 0xffff; } + ctx->p_to_gpb = 0; + ctx->b_to_gpb = 0; + +#if VA_CHECK_VERSION(1, 9, 0) + if (!(ctx->codec->flags & FLAG_INTRA_ONLY || + avctx->gop_size <= 1)) { + attr = (VAConfigAttrib) { VAConfigAttribPredictionDirection }; + vas = vaGetConfigAttributes(ctx->hwctx->display, + ctx->va_profile, + ctx->va_entrypoint, + &attr, 1); + if (vas != VA_STATUS_SUCCESS) { + av_log(avctx, AV_LOG_WARNING, "Failed to query prediction direction " + "attribute: %d (%s).\n", vas, vaErrorStr(vas)); + return AVERROR_EXTERNAL; + } else if (attr.value == VA_ATTRIB_NOT_SUPPORTED) { + av_log(avctx, AV_LOG_VERBOSE, "Driver does not report any additional " + "prediction constraints.\n"); + } else { + if (((ref_l0 > 0 || ref_l1 > 0) && !(attr.value & VA_PREDICTION_DIRECTION_PREVIOUS)) || + ((ref_l1 == 0) && (attr.value & (VA_PREDICTION_DIRECTION_FUTURE | VA_PREDICTION_DIRECTION_BI_NOT_EMPTY)))) { + av_log(avctx, AV_LOG_ERROR, "Driver report incorrect prediction " + "direction attribute.\n"); + return AVERROR_EXTERNAL; + } + + if (!(attr.value & VA_PREDICTION_DIRECTION_FUTURE)) { + if (ref_l0 > 0 && ref_l1 > 0) { + ctx->b_to_gpb = 1; + av_log(avctx, AV_LOG_VERBOSE, "Driver support previous prediction " + "only for B-frames.\n"); + } + } + + if (attr.value & VA_PREDICTION_DIRECTION_BI_NOT_EMPTY) { + if (ref_l0 > 0 && ref_l1 > 0) { + ctx->p_to_gpb = 1; + av_log(avctx, AV_LOG_VERBOSE, "Driver does not support P-frames, " + "replacing them with previous prediction only B-frames.\n"); + } + } + } + } +#endif + if (ctx->codec->flags & FLAG_INTRA_ONLY || avctx->gop_size <= 1) { av_log(avctx, AV_LOG_VERBOSE, "Using intra frames only.\n"); @@ -1855,14 +1907,24 @@ static av_cold int vaapi_encode_init_gop_structure(AVCodecContext *avctx) return AVERROR(EINVAL); } else if (!(ctx->codec->flags & FLAG_B_PICTURES) || ref_l1 < 1 || avctx->max_b_frames < 1) { - av_log(avctx, AV_LOG_VERBOSE, "Using intra and P-frames " - "(supported references: %d / %d).\n", ref_l0, ref_l1); + if (ctx->p_to_gpb) + av_log(avctx, AV_LOG_VERBOSE, "Using intra and B-frames " + "(supported references: %d / %d).\n", + ref_l0, ref_l1); + else + av_log(avctx, AV_LOG_VERBOSE, "Using intra and P-frames " + "(supported references: %d / %d).\n", ref_l0, ref_l1); ctx->gop_size = avctx->gop_size; ctx->p_per_i = INT_MAX; ctx->b_per_p = 0; } else { - av_log(avctx, AV_LOG_VERBOSE, "Using intra, P- and B-frames " - "(supported references: %d / %d).\n", ref_l0, ref_l1); + if (ctx->p_to_gpb) + av_log(avctx, AV_LOG_VERBOSE, "Using intra and B-frames " + "(supported references: %d / %d).\n", + ref_l0, ref_l1); + else + av_log(avctx, AV_LOG_VERBOSE, "Using intra, P- and B-frames " + "(supported references: %d / %d).\n", ref_l0, ref_l1); ctx->gop_size = avctx->gop_size; ctx->p_per_i = INT_MAX; ctx->b_per_p = avctx->max_b_frames; diff --git a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h index b41604a883..4ce1df0c6f 100644 --- a/libavcodec/vaapi_encode.h +++ b/libavcodec/vaapi_encode.h @@ -328,6 +328,8 @@ typedef struct VAAPIEncodeContext { int idr_counter; int gop_counter; int end_of_stream; + int p_to_gpb; + int b_to_gpb; // Whether the driver supports ROI at all. int roi_allowed; diff --git a/libavcodec/vaapi_encode_h265.c b/libavcodec/vaapi_encode_h265.c index e1dc53dfa9..25b9a707d2 100644 --- a/libavcodec/vaapi_encode_h265.c +++ b/libavcodec/vaapi_encode_h265.c @@ -717,7 +717,10 @@ static int vaapi_encode_h265_init_picture_params(AVCodecContext *avctx, hpic->pic_type = 1; } else { VAAPIEncodePicture *irap_ref; - av_assert0(pic->refs[0] && pic->refs[1]); + if (!ctx->b_to_gpb) + av_assert0(pic->refs[0] && pic->refs[1]); + else + av_assert0(pic->refs[0]); for (irap_ref = pic; irap_ref; irap_ref = irap_ref->refs[1]) { if (irap_ref->type == PICTURE_TYPE_I) break; @@ -886,6 +889,7 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, VAAPIEncodePicture *pic, VAAPIEncodeSlice *slice) { + VAAPIEncodeContext *ctx = avctx->priv_data; VAAPIEncodeH265Context *priv = avctx->priv_data; VAAPIEncodeH265Picture *hpic = pic->priv_data; const H265RawSPS *sps = &priv->raw_sps; @@ -908,6 +912,9 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, sh->slice_type = hpic->slice_type; + if (sh->slice_type == HEVC_SLICE_P && ctx->p_to_gpb) + sh->slice_type = HEVC_SLICE_B; + sh->slice_pic_order_cnt_lsb = hpic->pic_order_cnt & (1 << (sps->log2_max_pic_order_cnt_lsb_minus4 + 4)) - 1; @@ -1066,11 +1073,26 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, av_assert0(pic->type == PICTURE_TYPE_P || pic->type == PICTURE_TYPE_B); vslice->ref_pic_list0[0] = vpic->reference_frames[0]; + if (ctx->p_to_gpb && pic->type == PICTURE_TYPE_P) + // Reference for GPB B-frame, L0 == L1 + vslice->ref_pic_list1[0] = vpic->reference_frames[0]; } if (pic->nb_refs >= 2) { // Forward reference for B-frame. av_assert0(pic->type == PICTURE_TYPE_B); vslice->ref_pic_list1[0] = vpic->reference_frames[1]; + if (ctx->b_to_gpb && pic->type == PICTURE_TYPE_B) + // Reference for GPB B-frame, L0 == L1 + vslice->ref_pic_list1[0] = vpic->reference_frames[0]; + } + + if ((pic->type == PICTURE_TYPE_P && ctx->p_to_gpb) || + (pic->type == PICTURE_TYPE_B && ctx->b_to_gpb)) { + vslice->slice_type = HEVC_SLICE_B; + for (i = 0; i < FF_ARRAY_ELEMS(vslice->ref_pic_list0); i++) { + vslice->ref_pic_list1[i].picture_id = vslice->ref_pic_list0[i].picture_id; + vslice->ref_pic_list1[i].flags = vslice->ref_pic_list0[i].flags; + } } return 0;