From patchwork Wed Aug 22 23:45:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Thompson X-Patchwork-Id: 10090 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:12c4:0:0:0:0:0 with SMTP id 65-v6csp1443334jap; Wed, 22 Aug 2018 16:49:45 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxEd2hQa+S6+jgO52BSAi6JKvXFdf7mjjdp3KkJWl/lAYlf4lksd2O+X8jBde+8BjRZYQ9T X-Received: by 2002:adf:82c3:: with SMTP id 61-v6mr18654592wrc.131.1534981785045; Wed, 22 Aug 2018 16:49:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534981785; cv=none; d=google.com; s=arc-20160816; b=zKA5cgB2N95MbQpMyOY6ZL8cz+PM69Ha8xHDxSsUI1nvrMccloPq9uqsd1aZR+8axF EypPTgUSrG0PlrRzWNfcMZ2EP1h1gpRNkBHquLwPilWhUQqaVGt2MZaItQoG0WL+nZ8R Lf+krfZMdcKQBIMbcef5Bi4BhV0hcuD8BZ/uKxavz5nMjn2ZWN3Jj4zoVAbg196m1XjL V9cFqPQyLV8wPhmT+PF6gdD2rqDxqye4tNzL2eojmCbY/ejOWGGHh1tuw+gjbrPSzs5T vt/CuEHhxVa7RtvLvmY2mHlt3DFy7U+CmokBnPi34VDO4LrWoqIgIY6sa3whof3GUek6 XYcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=Z+J02QJEimhzeCbZJyK8+Vfuf7noiwVPTxGdcHjjB54=; b=JQ3e+5NRNbCMBDpJL7dbMiv/wDZ4V6O1x+ia9BPoH7NIAu4083aKTGVXjMhAdL0PrL Nz2LqZ31FP/uCFbjpN4asLiVoigBZcYzG8D2ewUDoCEfJBAphTYNjJa9ZRCwoSqcOHr5 jPFHMnnv3LgEJEhQUcbEo+WW7tnpKvCN3bwvVrR3SdBdR6OhJbX+Tvdrt8B0JNloE3wH LTtf/PyxykzHidit560XLH/pvi+79vbHSlPbyesJjFcnDXFizSSKEm75j1Jlz/bKqO4b FCI6wKueuVG3tlp73MMYedi0e+L5zhEg+l2mBRSzeu4qPClMsVSCf6hQUxvqe6q7ee/X PJGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=Qlr8QjuA; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y2-v6si749491wrr.433.2018.08.22.16.49.44; Wed, 22 Aug 2018 16:49:45 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=Qlr8QjuA; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CE832689FBC; Thu, 23 Aug 2018 02:46:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 888C7689F62 for ; Thu, 23 Aug 2018 02:46:00 +0300 (EEST) Received: by mail-wr1-f68.google.com with SMTP id w11-v6so2993371wrc.5 for ; Wed, 22 Aug 2018 16:46:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jkqxz-net.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=qOC4UWX6zpjpDmHV3sQxEWK0UsnyUQTWOwCbdk1gORA=; b=Qlr8QjuAlT9BcJPtdRRy8JIab4QZwdiptlo6eqKOMcmfRN3WfAq8tR6IwgbesGi6NM iHwIB1SwWwzF2pCu09HMP3novZunkNU9uhoQRnAtEy4R+pshwvu+RtyOmQsT8irGp5Ql 0iJVoMXRoOj299eBDFDQ+DPeAweeQ706ppgSydVtmjKxbYrYjEGnxCAaM/qv+NS2Nt0H tJoKPYOC42GmtmrvLS+bXqVzdcUtMnM0LQl18JU+IY1zc+Rq71F5vQhTBiHKf980LjoJ AXGGdM/bTlZlbeOgOpIhkgmPkU0KdOKgwigaVVaemL3+2e21H5X4lQiIkfcSLPdukX2X j+aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=qOC4UWX6zpjpDmHV3sQxEWK0UsnyUQTWOwCbdk1gORA=; b=dpr/b5oDhiiIR07J5IPxxXerDdFpWJ/mvGorY4E7PHS0YmoV4xMHPmvyiNGVXEA7MU rdLdmqDlVy/0SUTbynEyxF0xwevNL8Owvr6drOCOuhuLveROnkRhjHqzgujQZO6AvNDr UOaCW1tclvLqDbFkEoZ2WF1YpKHHUQvr/+hymhMStuwJz8Lr8AgwE74/vXItRGN1FaiJ KOjbQ5YFgVwDVoFWWsP8BAsYu4kCpIWjh1S+4v51fJVx8kw218ePp+URyVg+3tv2w0qU toEWbJsIcbISQ8EzOYcCFGa1v8t/I5ElYszCrpnswGJQdmP60ags9UeKjB6FD+GJA+bt VuKA== X-Gm-Message-State: APzg51D5EERDwUOgP1oThzs1250dvpFRF5kxw04a7gZjjqq813U18Udb vy3n7/pBOTvoou6lG1lg5taYOeP/Ucg= X-Received: by 2002:a5d:488c:: with SMTP id g12-v6mr8432627wrq.0.1534981560888; Wed, 22 Aug 2018 16:46:00 -0700 (PDT) Received: from rywe.jkqxz.net (cpc91242-cmbg18-2-0-cust650.5-4.cable.virginm.net. [82.8.130.139]) by smtp.gmail.com with ESMTPSA id x82-v6sm8184537wmd.11.2018.08.22.16.45.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 16:46:00 -0700 (PDT) From: Mark Thompson To: ffmpeg-devel@ffmpeg.org Date: Thu, 23 Aug 2018 00:45:12 +0100 Message-Id: <20180822234514.10571-40-sw@jkqxz.net> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180822234514.10571-1-sw@jkqxz.net> References: <20180822234514.10571-1-sw@jkqxz.net> Subject: [FFmpeg-devel] [PATCH v3 39/41] vaapi_encode_h264: Support more complex reference structures X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" --- Some future changes: * Don't use r-p-l-m if the list is already correct. * Support more than one reference frame in either direction. libavcodec/vaapi_encode_h264.c | 180 ++++++++++++++++++++++----------- 1 file changed, 120 insertions(+), 60 deletions(-) diff --git a/libavcodec/vaapi_encode_h264.c b/libavcodec/vaapi_encode_h264.c index bf6e7dfb98..2313201a42 100644 --- a/libavcodec/vaapi_encode_h264.c +++ b/libavcodec/vaapi_encode_h264.c @@ -47,6 +47,20 @@ static const uint8_t vaapi_encode_h264_sei_identifier_uuid[16] = { 0x96, 0x75, 0x19, 0xd4, 0x1f, 0xea, 0xa9, 0x4d, }; +typedef struct VAAPIEncodeH264Picture { + int frame_num; + int pic_order_cnt; + + int64_t last_idr_frame; + uint16_t idr_pic_id; + + int primary_pic_type; + int slice_type; + + int cpb_delay; + int dpb_delay; +} VAAPIEncodeH264Picture; + typedef struct VAAPIEncodeH264Context { VAAPIEncodeContext common; @@ -67,19 +81,6 @@ typedef struct VAAPIEncodeH264Context { int fixed_qp_p; int fixed_qp_b; - // Stream state. - int frame_num; - int pic_order_cnt; - int next_frame_num; - int64_t last_idr_frame; - int64_t idr_pic_count; - - int primary_pic_type; - int slice_type; - - int cpb_delay; - int dpb_delay; - // Writer structures. CodedBitstreamContext *cbc; CodedBitstreamFragment current_access_unit; @@ -324,7 +325,7 @@ static int vaapi_encode_h264_init_sequence_params(AVCodecContext *avctx) if (ctx->gop_size == 1) dpb_frames = 0; else - dpb_frames = 1 + (ctx->b_per_p > 0); + dpb_frames = 1 + ctx->max_b_depth; if (avctx->level != FF_LEVEL_UNKNOWN) { sps->level_idc = avctx->level; @@ -353,8 +354,7 @@ static int vaapi_encode_h264_init_sequence_params(AVCodecContext *avctx) sps->log2_max_frame_num_minus4 = 4; sps->pic_order_cnt_type = 0; - sps->log2_max_pic_order_cnt_lsb_minus4 = - av_clip(av_log2(ctx->b_per_p + 1) - 2, 0, 12); + sps->log2_max_pic_order_cnt_lsb_minus4 = 4; sps->max_num_ref_frames = dpb_frames; @@ -493,8 +493,8 @@ static int vaapi_encode_h264_init_sequence_params(AVCodecContext *avctx) sps->vui.motion_vectors_over_pic_boundaries_flag = 1; sps->vui.log2_max_mv_length_horizontal = 15; sps->vui.log2_max_mv_length_vertical = 15; - sps->vui.max_num_reorder_frames = (ctx->b_per_p > 0); - sps->vui.max_dec_frame_buffering = sps->max_num_ref_frames; + sps->vui.max_num_reorder_frames = ctx->max_b_depth; + sps->vui.max_dec_frame_buffering = ctx->max_b_depth + 1; pps->nal_unit_header.nal_ref_idc = 3; pps->nal_unit_header.nal_unit_type = H264_NAL_PPS; @@ -615,6 +615,9 @@ static int vaapi_encode_h264_init_picture_params(AVCodecContext *avctx, { VAAPIEncodeContext *ctx = avctx->priv_data; VAAPIEncodeH264Context *priv = avctx->priv_data; + VAAPIEncodeH264Picture *hpic = pic->priv_data; + VAAPIEncodePicture *prev = pic->prev; + VAAPIEncodeH264Picture *hprev = prev ? prev->priv_data : NULL; H264RawSPS *sps = &priv->raw_sps; VAEncPictureParameterBufferH264 *vpic = pic->codec_picture_params; int i; @@ -624,37 +627,39 @@ static int vaapi_encode_h264_init_picture_params(AVCodecContext *avctx, if (pic->type == PICTURE_TYPE_IDR) { av_assert0(pic->display_order == pic->encode_order); - priv->frame_num = 0; - priv->next_frame_num = 1; - priv->cpb_delay = 0; - priv->last_idr_frame = pic->display_order; - ++priv->idr_pic_count; - - priv->slice_type = 7; - priv->primary_pic_type = 0; + + hpic->frame_num = 0; + hpic->last_idr_frame = pic->display_order; + hpic->idr_pic_id = hprev ? hprev->idr_pic_id + 1 : 0; + + hpic->primary_pic_type = 0; + hpic->slice_type = 7; + + hpic->cpb_delay = 0; } else { - priv->frame_num = priv->next_frame_num; + av_assert0(prev); - if (pic->type != PICTURE_TYPE_B) { - // Reference picture, so frame_num advances. - priv->next_frame_num = (priv->frame_num + 1) & - ((1 << (4 + sps->log2_max_frame_num_minus4)) - 1); - } - ++priv->cpb_delay; + hpic->frame_num = hprev->frame_num + prev->is_reference & + ((1 << (4 + sps->log2_max_frame_num_minus4)) - 1); + + hpic->last_idr_frame = hprev->last_idr_frame; + hpic->idr_pic_id = hprev->idr_pic_id; if (pic->type == PICTURE_TYPE_I) { - priv->slice_type = 7; - priv->primary_pic_type = 0; + hpic->slice_type = 7; + hpic->primary_pic_type = 0; } else if (pic->type == PICTURE_TYPE_P) { - priv->slice_type = 5; - priv->primary_pic_type = 1; + hpic->slice_type = 5; + hpic->primary_pic_type = 1; } else { - priv->slice_type = 6; - priv->primary_pic_type = 2; + hpic->slice_type = 6; + hpic->primary_pic_type = 2; } + + ++hpic->cpb_delay; } - priv->pic_order_cnt = pic->display_order - priv->last_idr_frame; - priv->dpb_delay = pic->display_order - pic->encode_order + 1; + hpic->pic_order_cnt = pic->display_order - hpic->last_idr_frame; + hpic->dpb_delay = pic->display_order - pic->encode_order + 1; if (priv->aud) { priv->aud_needed = 1; @@ -662,7 +667,7 @@ static int vaapi_encode_h264_init_picture_params(AVCodecContext *avctx, .nal_unit_header = { .nal_unit_type = H264_NAL_AUD, }, - .primary_pic_type = priv->primary_pic_type, + .primary_pic_type = hpic->primary_pic_type, }; } else { priv->aud_needed = 0; @@ -679,8 +684,8 @@ static int vaapi_encode_h264_init_picture_params(AVCodecContext *avctx, if (priv->sei & SEI_TIMING) { priv->sei_pic_timing = (H264RawSEIPicTiming) { - .cpb_removal_delay = 2 * priv->cpb_delay, - .dpb_output_delay = 2 * priv->dpb_delay, + .cpb_removal_delay = 2 * hpic->cpb_delay, + .dpb_output_delay = 2 * hpic->dpb_delay, }; priv->sei_needed |= SEI_TIMING; @@ -698,25 +703,25 @@ static int vaapi_encode_h264_init_picture_params(AVCodecContext *avctx, vpic->CurrPic = (VAPictureH264) { .picture_id = pic->recon_surface, - .frame_idx = priv->frame_num, + .frame_idx = hpic->frame_num, .flags = 0, - .TopFieldOrderCnt = priv->pic_order_cnt, - .BottomFieldOrderCnt = priv->pic_order_cnt, + .TopFieldOrderCnt = hpic->pic_order_cnt, + .BottomFieldOrderCnt = hpic->pic_order_cnt, }; for (i = 0; i < pic->nb_refs; i++) { - VAAPIEncodePicture *ref = pic->refs[i]; - unsigned int frame_num = (ref->encode_order - priv->last_idr_frame) & - ((1 << (4 + sps->log2_max_frame_num_minus4)) - 1); - unsigned int pic_order_cnt = ref->display_order - priv->last_idr_frame; + VAAPIEncodePicture *ref = pic->refs[i]; + VAAPIEncodeH264Picture *href; av_assert0(ref && ref->encode_order < pic->encode_order); + href = ref->priv_data; + vpic->ReferenceFrames[i] = (VAPictureH264) { .picture_id = ref->recon_surface, - .frame_idx = frame_num, + .frame_idx = href->frame_num, .flags = VA_PICTURE_H264_SHORT_TERM_REFERENCE, - .TopFieldOrderCnt = pic_order_cnt, - .BottomFieldOrderCnt = pic_order_cnt, + .TopFieldOrderCnt = href->pic_order_cnt, + .BottomFieldOrderCnt = href->pic_order_cnt, }; } for (; i < FF_ARRAY_ELEMS(vpic->ReferenceFrames); i++) { @@ -728,7 +733,7 @@ static int vaapi_encode_h264_init_picture_params(AVCodecContext *avctx, vpic->coded_buf = pic->output_buffer; - vpic->frame_num = priv->frame_num; + vpic->frame_num = hpic->frame_num; vpic->pic_fields.bits.idr_pic_flag = (pic->type == PICTURE_TYPE_IDR); vpic->pic_fields.bits.reference_pic_flag = (pic->type != PICTURE_TYPE_B); @@ -743,6 +748,8 @@ static int vaapi_encode_h264_init_slice_params(AVCodecContext *avctx, VAAPIEncodeSlice *slice) { VAAPIEncodeH264Context *priv = avctx->priv_data; + VAAPIEncodeH264Picture *hpic = pic->priv_data; + VAAPIEncodePicture *prev = pic->prev; H264RawSPS *sps = &priv->raw_sps; H264RawPPS *pps = &priv->raw_pps; H264RawSliceHeader *sh = &priv->raw_slice.header; @@ -755,19 +762,19 @@ static int vaapi_encode_h264_init_slice_params(AVCodecContext *avctx, sh->nal_unit_header.nal_ref_idc = 3; } else { sh->nal_unit_header.nal_unit_type = H264_NAL_SLICE; - sh->nal_unit_header.nal_ref_idc = pic->type != PICTURE_TYPE_B; + sh->nal_unit_header.nal_ref_idc = pic->is_reference; } // Only one slice per frame. sh->first_mb_in_slice = 0; - sh->slice_type = priv->slice_type; + sh->slice_type = hpic->slice_type; sh->pic_parameter_set_id = pps->pic_parameter_set_id; - sh->frame_num = priv->frame_num; - sh->idr_pic_id = priv->idr_pic_count; + sh->frame_num = hpic->frame_num; + sh->idr_pic_id = hpic->idr_pic_id; - sh->pic_order_cnt_lsb = priv->pic_order_cnt & + sh->pic_order_cnt_lsb = hpic->pic_order_cnt & ((1 << (4 + sps->log2_max_pic_order_cnt_lsb_minus4)) - 1); sh->direct_spatial_mv_pred_flag = 1; @@ -779,6 +786,56 @@ static int vaapi_encode_h264_init_slice_params(AVCodecContext *avctx, else sh->slice_qp_delta = priv->fixed_qp_idr - (pps->pic_init_qp_minus26 + 26); + if (pic->is_reference && pic->type != PICTURE_TYPE_IDR) { + VAAPIEncodeH264Picture *old; + int i, j, mmco = 0; + sh->adaptive_ref_pic_marking_mode_flag = 1; + // Discard everything which is in the DPB of the previous frame but + // not in the DPB of this one. + for (i = 0; i < prev->nb_dpb_pics; i++) { + int discard = 1; + for (j = 0; j < pic->nb_dpb_pics; j++) { + if (prev->dpb[i] == pic->dpb[j]) + discard = 0; + } + if (discard) { + old = prev->dpb[i]->priv_data; + av_assert0(old->frame_num < hpic->frame_num); + sh->mmco[mmco].memory_management_control_operation = 1; + sh->mmco[mmco].difference_of_pic_nums_minus1 = + hpic->frame_num - old->frame_num - 1; + ++mmco; + } + } + sh->mmco[mmco].memory_management_control_operation = 0; + } + + if (pic->type == PICTURE_TYPE_P || pic->type == PICTURE_TYPE_B) { + VAAPIEncodeH264Picture *href0 = pic->refs[0]->priv_data; + + // If the intended reference is not the first entry of RefPicListN + // by default, use ref-pic-list-modification to move it there. + + sh->ref_pic_list_modification_flag_l0 = 1; + + av_assert0(href0->frame_num < hpic->frame_num); + sh->rplm_l0[0].modification_of_pic_nums_idc = 0; + sh->rplm_l0[0].abs_diff_pic_num_minus1 = + hpic->frame_num - href0->frame_num - 1; + sh->rplm_l0[1].modification_of_pic_nums_idc = 3; + + if (pic->type == PICTURE_TYPE_B) { + VAAPIEncodeH264Picture *href1 = pic->refs[1]->priv_data; + + sh->ref_pic_list_modification_flag_l1 = 1; + + av_assert0(href1->frame_num < hpic->frame_num); + sh->rplm_l1[0].modification_of_pic_nums_idc = 0; + sh->rplm_l1[0].abs_diff_pic_num_minus1 = + hpic->frame_num - href1->frame_num - 1; + sh->rplm_l1[1].modification_of_pic_nums_idc = 3; + } + } vslice->macroblock_address = sh->first_mb_in_slice; vslice->num_macroblocks = priv->mb_width * priv->mb_height; @@ -904,10 +961,13 @@ static const VAAPIEncodeType vaapi_encode_type_h264 = { .profiles = vaapi_encode_h264_profiles, .flags = FLAG_B_PICTURES | + FLAG_B_PICTURE_REFERENCES | FLAG_NON_IDR_KEY_PICTURES, .configure = &vaapi_encode_h264_configure, + .picture_priv_data_size = sizeof(VAAPIEncodeH264Picture), + .sequence_params_size = sizeof(VAEncSequenceParameterBufferH264), .init_sequence_params = &vaapi_encode_h264_init_sequence_params,