From patchwork Wed Aug 22 23:45:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Thompson X-Patchwork-Id: 10092 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:12c4:0:0:0:0:0 with SMTP id 65-v6csp1443407jap; Wed, 22 Aug 2018 16:49:52 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb3lKtNXorf3GOP5ecD6RLPEWnQryabMxS1z3sgT6PnrkjFb7tM2WRFQFG0eSERwEMcIoM6 X-Received: by 2002:a1c:e595:: with SMTP id c143-v6mr3720425wmh.85.1534981792157; Wed, 22 Aug 2018 16:49:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534981792; cv=none; d=google.com; s=arc-20160816; b=iPwf+/XHPRG9f7HsZFII5AVpo69Ymq6rYUD77rKz1N9Uxk2BTwIH1goTVMZY9m2ah3 LuglwrfAsj202hDZazu0by5ZoPpoH4rI0eWGe1No+mnWgeP2fLHYbfn5o+b3e6jiYhtd Kkbp1e7ArLQsEk2Da+V9k8FD4SqV9JsP0r/obevLq5n1qAif7HdQmhQElkGAHWlE1cKM +t1mU07al6E2dT9iyUM/QUFtqHGiOCixwgWBNyL21xXLDMvPIy9YU2rQXCIBK3RTmvmy tX7cPk6Y6Pww5i0KgwMugV3T1vAbnHz1ubKIXFiDYSv7SRrlxhI36DswyMmzz50GVj5U RSxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=ye7C2xC//jhYVmOGXVlB7ZTZdc0gU/7pagrMpr+4WyA=; b=spIbD1axYNb8s3pSjlN/VxG7cBcPJDcKCyu0+Wj7wL+lV+6LpELMk2JqiHUSUuzyJn S0h7LMUienrwenNKETXtiAdgG5qODMgaZhweZJVH1fhb03KFlzXvZBjNjQ5SnG3t+6Dg RtF6jUs3DLxK7Ce5kKMgA1sj9MOnbp0OpAkGE2vEZJYjFjiDg/kimep6kI4LkMi+7JE3 ZPeyBATNESNaIRbunvH8m8zTI1pzhl+6RJKCxU8npII0A9cTgBUOEgfCcb52TOGWWOe+ WSDdJj+kXL80rh0KygYdt/CSB75X1KkrNvyZUMR90uEJa41Tw1GCwTgSuFwdDZejepr3 klYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=0vhF8eob; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 35-v6si2523835wrq.201.2018.08.22.16.49.51; Wed, 22 Aug 2018 16:49:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=0vhF8eob; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5F31B689FC7; Thu, 23 Aug 2018 02:46:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B11F8689FB8 for ; Thu, 23 Aug 2018 02:46:01 +0300 (EEST) Received: by mail-wr1-f48.google.com with SMTP id m27-v6so2999238wrf.3 for ; Wed, 22 Aug 2018 16:46:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jkqxz-net.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=66QfrcP+QvnZa4BwyuvUG/ZkQH4pUSXHnT03/2u1ZJs=; b=0vhF8eob7t4w33IMMXcfI7lfaTnda9uXSJwaPu9O4jD6LMWyrv9Ngvpl6r52iCIjgz +0YAavjF1ULwVLKcmoiHCvIXw31EVSQB03ezK2AnIPKVB1O6BtJ0Bjp2/EI7Ru7/7amW s+Dg6UxlPB93vLymWapoWJhyK8+75R13eRd6APEWeMTRC4Bse5PnKI002IYCD/EvSC9u D7UYJ6nc7xNUow6iM/20f3wC7LN/PhXa8qw0BOoBPo42COzWXKpsRh6XRNda/bWxIfzi Uie6jT7h9VMuVxGDsjMcQohFqdX8pWYAW5qTcDlWuu8SZttR0wu95F+tSNLRIqrH2mIY 1qtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=66QfrcP+QvnZa4BwyuvUG/ZkQH4pUSXHnT03/2u1ZJs=; b=EzYpNDs8k0Mgqn+rIqOgVjHkkJ+AcMUvxOHCXU76eBd+Nvy3XdGB1yrow066P3mJrD JGxuR7SSS/vVMwrMFdpR93uEbHXHG57JC/mxaO5GHxXLgDIG9YuwyrlhRV6ItWtfOKw5 1hvbIozPgY0yOjyMZhOVnrD6PyY8UDmtxMJ9KZXREdAqdY/P8O1Nu2lg/dpR/UD8Mvsg 6l2nbfJRFOIedsZ8LUszxxxaZh089VqXK92XqLuwxuADqAHA/s80n+JtSzO9vN2aRZYO +h3YPXXWIy8lTQcZ6oFWjAuh/RBbWTuqQ5745n/7XMN5/FxYPWq5LstU4Z0FZ0sIQ/fr lKtQ== X-Gm-Message-State: APzg51DhO3tikVfTwnUhHU3qe6oqhAPm6YY+t/ieJq1KUE1gGmvJaVy3 BWM2TIS+Sr5l8jYt4P8Oz0Kiw4HpcBQ= X-Received: by 2002:a5d:62c2:: with SMTP id o2-v6mr8549607wrv.83.1534981561950; Wed, 22 Aug 2018 16:46:01 -0700 (PDT) Received: from rywe.jkqxz.net (cpc91242-cmbg18-2-0-cust650.5-4.cable.virginm.net. [82.8.130.139]) by smtp.gmail.com with ESMTPSA id x82-v6sm8184537wmd.11.2018.08.22.16.46.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 16:46:01 -0700 (PDT) From: Mark Thompson To: ffmpeg-devel@ffmpeg.org Date: Thu, 23 Aug 2018 00:45:13 +0100 Message-Id: <20180822234514.10571-41-sw@jkqxz.net> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180822234514.10571-1-sw@jkqxz.net> References: <20180822234514.10571-1-sw@jkqxz.net> Subject: [FFmpeg-devel] [PATCH v3 40/41] vaapi_encode_h265: Support more complex reference structures X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" The reference picture sets are now constructed directly from the DPB information. --- An obvious useful change (not done) would be the move the RPSs to the SPS where possible, since they are now even larger. libavcodec/vaapi_encode_h265.c | 191 +++++++++++++++++++-------------- 1 file changed, 111 insertions(+), 80 deletions(-) diff --git a/libavcodec/vaapi_encode_h265.c b/libavcodec/vaapi_encode_h265.c index f25ba7edf0..b3c60523b6 100644 --- a/libavcodec/vaapi_encode_h265.c +++ b/libavcodec/vaapi_encode_h265.c @@ -42,6 +42,16 @@ enum { SEI_CONTENT_LIGHT_LEVEL = 0x10, }; +typedef struct VAAPIEncodeH265Picture { + int pic_order_cnt; + + int64_t last_idr_frame; + + int slice_nal_unit; + int slice_type; + int pic_type; +} VAAPIEncodeH265Picture; + typedef struct VAAPIEncodeH265Context { VAAPIEncodeContext common; @@ -61,14 +71,6 @@ typedef struct VAAPIEncodeH265Context { int fixed_qp_p; int fixed_qp_b; - // Stream state. - int64_t last_idr_frame; - int pic_order_cnt; - - int slice_nal_unit; - int slice_type; - int pic_type; - // Writer structures. H265RawAUD raw_aud; H265RawVPS raw_vps; @@ -362,8 +364,8 @@ static int vaapi_encode_h265_init_sequence_params(AVCodecContext *avctx) } vps->vps_sub_layer_ordering_info_present_flag = 0; - vps->vps_max_dec_pic_buffering_minus1[0] = (ctx->b_per_p > 0) + 1; - vps->vps_max_num_reorder_pics[0] = (ctx->b_per_p > 0); + vps->vps_max_dec_pic_buffering_minus1[0] = ctx->max_b_depth + 1; + vps->vps_max_num_reorder_pics[0] = ctx->max_b_depth; vps->vps_max_latency_increase_plus1[0] = 0; vps->vps_max_layer_id = 0; @@ -672,41 +674,54 @@ static int vaapi_encode_h265_init_sequence_params(AVCodecContext *avctx) static int vaapi_encode_h265_init_picture_params(AVCodecContext *avctx, VAAPIEncodePicture *pic) { + VAAPIEncodeContext *ctx = avctx->priv_data; VAAPIEncodeH265Context *priv = avctx->priv_data; + VAAPIEncodeH265Picture *hpic = pic->priv_data; + VAAPIEncodePicture *prev = pic->prev; + VAAPIEncodeH265Picture *hprev = prev ? prev->priv_data : NULL; VAEncPictureParameterBufferHEVC *vpic = pic->codec_picture_params; int i; if (pic->type == PICTURE_TYPE_IDR) { av_assert0(pic->display_order == pic->encode_order); - priv->last_idr_frame = pic->display_order; + hpic->last_idr_frame = pic->display_order; - priv->slice_nal_unit = HEVC_NAL_IDR_W_RADL; - priv->slice_type = HEVC_SLICE_I; - priv->pic_type = 0; + hpic->slice_nal_unit = HEVC_NAL_IDR_W_RADL; + hpic->slice_type = HEVC_SLICE_I; + hpic->pic_type = 0; } else { - av_assert0(pic->encode_order > priv->last_idr_frame); + av_assert0(prev); + hpic->last_idr_frame = hprev->last_idr_frame; if (pic->type == PICTURE_TYPE_I) { - priv->slice_nal_unit = HEVC_NAL_CRA_NUT; - priv->slice_type = HEVC_SLICE_I; - priv->pic_type = 0; + hpic->slice_nal_unit = HEVC_NAL_CRA_NUT; + hpic->slice_type = HEVC_SLICE_I; + hpic->pic_type = 0; } else if (pic->type == PICTURE_TYPE_P) { av_assert0(pic->refs[0]); - priv->slice_nal_unit = HEVC_NAL_TRAIL_R; - priv->slice_type = HEVC_SLICE_P; - priv->pic_type = 1; + hpic->slice_nal_unit = HEVC_NAL_TRAIL_R; + hpic->slice_type = HEVC_SLICE_P; + hpic->pic_type = 1; } else { + VAAPIEncodePicture *irap_ref; av_assert0(pic->refs[0] && pic->refs[1]); - if (pic->refs[1]->type == PICTURE_TYPE_I) - priv->slice_nal_unit = HEVC_NAL_RASL_N; - else - priv->slice_nal_unit = HEVC_NAL_TRAIL_N; - priv->slice_type = HEVC_SLICE_B; - priv->pic_type = 2; + for (irap_ref = pic; irap_ref; irap_ref = irap_ref->refs[1]) { + if (irap_ref->type == PICTURE_TYPE_I) + break; + } + if (pic->b_depth == ctx->max_b_depth) { + hpic->slice_nal_unit = irap_ref ? HEVC_NAL_RASL_N + : HEVC_NAL_TRAIL_N; + } else { + hpic->slice_nal_unit = irap_ref ? HEVC_NAL_RASL_R + : HEVC_NAL_TRAIL_R; + } + hpic->slice_type = HEVC_SLICE_B; + hpic->pic_type = 2; } } - priv->pic_order_cnt = pic->display_order - priv->last_idr_frame; + hpic->pic_order_cnt = pic->display_order - hpic->last_idr_frame; if (priv->aud) { priv->aud_needed = 1; @@ -716,7 +731,7 @@ static int vaapi_encode_h265_init_picture_params(AVCodecContext *avctx, .nuh_layer_id = 0, .nuh_temporal_id_plus1 = 1, }, - .pic_type = priv->pic_type, + .pic_type = hpic->pic_type, }; } else { priv->aud_needed = 0; @@ -796,17 +811,20 @@ static int vaapi_encode_h265_init_picture_params(AVCodecContext *avctx, vpic->decoded_curr_pic = (VAPictureHEVC) { .picture_id = pic->recon_surface, - .pic_order_cnt = priv->pic_order_cnt, + .pic_order_cnt = hpic->pic_order_cnt, .flags = 0, }; for (i = 0; i < pic->nb_refs; i++) { - VAAPIEncodePicture *ref = pic->refs[i]; + VAAPIEncodePicture *ref = pic->refs[i]; + VAAPIEncodeH265Picture *href; + av_assert0(ref && ref->encode_order < pic->encode_order); + href = ref->priv_data; vpic->reference_frames[i] = (VAPictureHEVC) { .picture_id = ref->recon_surface, - .pic_order_cnt = ref->display_order - priv->last_idr_frame, + .pic_order_cnt = href->pic_order_cnt, .flags = (ref->display_order < pic->display_order ? VA_PICTURE_HEVC_RPS_ST_CURR_BEFORE : 0) | (ref->display_order > pic->display_order ? @@ -822,7 +840,7 @@ static int vaapi_encode_h265_init_picture_params(AVCodecContext *avctx, vpic->coded_buf = pic->output_buffer; - vpic->nal_unit_type = priv->slice_nal_unit; + vpic->nal_unit_type = hpic->slice_nal_unit; switch (pic->type) { case PICTURE_TYPE_IDR: @@ -858,8 +876,8 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, VAAPIEncodePicture *pic, VAAPIEncodeSlice *slice) { - VAAPIEncodeContext *ctx = avctx->priv_data; VAAPIEncodeH265Context *priv = avctx->priv_data; + VAAPIEncodeH265Picture *hpic = pic->priv_data; const H265RawSPS *sps = &priv->raw_sps; const H265RawPPS *pps = &priv->raw_pps; H265RawSliceHeader *sh = &priv->raw_slice.header; @@ -868,7 +886,7 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, int i; sh->nal_unit_header = (H265RawNALUnitHeader) { - .nal_unit_type = priv->slice_nal_unit, + .nal_unit_type = hpic->slice_nal_unit, .nuh_layer_id = 0, .nuh_temporal_id_plus1 = 1, }; @@ -879,64 +897,74 @@ static int vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, sh->first_slice_segment_in_pic_flag = 1; sh->slice_segment_address = 0; - sh->slice_type = priv->slice_type; + sh->slice_type = hpic->slice_type; - sh->slice_pic_order_cnt_lsb = priv->pic_order_cnt & + sh->slice_pic_order_cnt_lsb = hpic->pic_order_cnt & (1 << (sps->log2_max_pic_order_cnt_lsb_minus4 + 4)) - 1; if (pic->type != PICTURE_TYPE_IDR) { H265RawSTRefPicSet *rps; - VAAPIEncodePicture *st; - int used; + int rps_poc[MAX_DPB_SIZE]; + int rps_used[MAX_DPB_SIZE]; + int i, j, poc, rps_pics; sh->short_term_ref_pic_set_sps_flag = 0; rps = &sh->short_term_ref_pic_set; memset(rps, 0, sizeof(*rps)); - for (st = ctx->pic_start; st; st = st->next) { - if (st->encode_order >= pic->encode_order) { - // Not yet in DPB. + rps_pics = 0; + for (i = 0; i < pic->nb_dpb_pics; i++) { + VAAPIEncodeH265Picture *strp; + if (pic->dpb[i] == pic) continue; + strp = pic->dpb[i]->priv_data; + rps_poc[rps_pics] = strp->pic_order_cnt; + rps_used[rps_pics] = 0; + for (j = 0; j < pic->nb_refs; j++) + if (pic->dpb[i] == pic->refs[j]) + rps_used[rps_pics] = 1; + ++rps_pics; + } + + for (i = 1; i < rps_pics; i++) { + for (j = i; j > 0; j--) { + if (rps_poc[j] > rps_poc[j - 1]) + break; + av_assert0(rps_poc[j] != rps_poc[j - 1]); + FFSWAP(int, rps_poc[j], rps_poc[j - 1]); + FFSWAP(int, rps_used[j], rps_used[j - 1]); } - used = 0; - for (i = 0; i < pic->nb_refs; i++) { - if (pic->refs[i] == st) - used = 1; - } - if (!used) { - // Usually each picture always uses all of the others in the - // DPB as references. The one case we have to treat here is - // a non-IDR IRAP picture, which may need to hold unused - // references across itself to be used for the decoding of - // following RASL pictures. This looks for such an RASL - // picture, and keeps the reference if there is one. - VAAPIEncodePicture *rp; - for (rp = ctx->pic_start; rp; rp = rp->next) { - if (rp->encode_order < pic->encode_order) - continue; - if (rp->type != PICTURE_TYPE_B) - continue; - if (rp->refs[0] == st && rp->refs[1] == pic) - break; - } - if (!rp) - continue; - } - // This only works for one instance of each (delta_poc_sN_minus1 - // is relative to the previous frame in the list, not relative to - // the current frame directly). - if (st->display_order < pic->display_order) { - rps->delta_poc_s0_minus1[rps->num_negative_pics] = - pic->display_order - st->display_order - 1; - rps->used_by_curr_pic_s0_flag[rps->num_negative_pics] = used; - ++rps->num_negative_pics; - } else { - rps->delta_poc_s1_minus1[rps->num_positive_pics] = - st->display_order - pic->display_order - 1; - rps->used_by_curr_pic_s1_flag[rps->num_positive_pics] = used; - ++rps->num_positive_pics; - } + } + + av_log(avctx, AV_LOG_DEBUG, "RPS for POC %d:", + hpic->pic_order_cnt); + for (i = 0; i < rps_pics; i++) { + av_log(avctx, AV_LOG_DEBUG, " (%d,%d)", + rps_poc[i], rps_used[i]); + } + av_log(avctx, AV_LOG_DEBUG, "\n"); + + for (i = 0; i < rps_pics; i++) { + av_assert0(rps_poc[i] != hpic->pic_order_cnt); + if (rps_poc[i] > hpic->pic_order_cnt) + break; + } + + rps->num_negative_pics = i; + poc = hpic->pic_order_cnt; + for (j = i - 1; j >= 0; j--) { + rps->delta_poc_s0_minus1[i - 1 - j] = poc - rps_poc[j] - 1; + rps->used_by_curr_pic_s0_flag[i - 1 - j] = rps_used[j]; + poc = rps_poc[j]; + } + + rps->num_positive_pics = rps_pics - i; + poc = hpic->pic_order_cnt; + for (j = i; j < rps_pics; j++) { + rps->delta_poc_s1_minus1[j - i] = rps_poc[j] - poc - 1; + rps->used_by_curr_pic_s1_flag[j - i] = rps_used[j]; + poc = rps_poc[j]; } sh->num_long_term_sps = 0; @@ -1092,10 +1120,13 @@ static const VAAPIEncodeType vaapi_encode_type_h265 = { .profiles = vaapi_encode_h265_profiles, .flags = FLAG_B_PICTURES | + FLAG_B_PICTURE_REFERENCES | FLAG_NON_IDR_KEY_PICTURES, .configure = &vaapi_encode_h265_configure, + .picture_priv_data_size = sizeof(VAAPIEncodeH265Picture), + .sequence_params_size = sizeof(VAEncSequenceParameterBufferHEVC), .init_sequence_params = &vaapi_encode_h265_init_sequence_params,