From patchwork Sat Aug 14 11:36:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 29505 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp380539iov; Sat, 14 Aug 2021 04:37:13 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx0mSe5DE0ukdvmb5JHNnQP2WeewHPrqDVP9aIo65ug/sB2+AItGuNHV2N5SpNUbra/6gr1 X-Received: by 2002:aa7:d815:: with SMTP id v21mr8538353edq.262.1628941033504; Sat, 14 Aug 2021 04:37:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628941033; cv=none; d=google.com; s=arc-20160816; b=PRsJgcLAdKrWZmaqrxYdE0DPQcv2WZ6QYluynn3WuqYo6Oi6ZFiTPsGJFBJrnk5yfk XZjSisjEtazSJ59a2j/j020DBPtam555o/m7EjYVuIk8o1P7o7xpzQ0EoIIj6EiBJf5I oRcGKsPdoSbPixAlYsa0Ek/nHEO+NPydRO8pWMN1PaPb+DIgm4N718BCNYY0qqvN/oa+ Id68OsS4Zw7PRwwlfmJWL8h0nKJnH+cuVkNoJP2yII0Ne1TS6oIME852r6myYskqUl/I EhZbeSxgTHO9Id257ujOO+BcSXocKsZSOwz58jIRGg7MW2i8Y3m168LPTWWipFcIVHRG LEBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=X80rIxc9HTN/HRgBEJTVnChIyfJIwdLpLpmcUzt9q1A=; b=nM5CIU9v9cEnTwKwuejpSv6x6IG8lEQYx10QItPhRLASGSWQ8I3co2TBE84BqscmVG UHMrE4/qNJ/FO9/TNmTl7Nx+bdK3V7jLu2VEKzu00YAPy/aHTOjvTZSN6BqfNCzVL8LU swKCLQKIl5h9L1CFy/E90ThZ/jtBDrYQh/sbXSTBhfdrQqGQz7TymT4GXP6yYnNcatwP 1RnS1GkLsh7PGp2WPyTMaWLttIIc8KMz5scHpuODY7ziGu6SdfcjWBp24KNlZiaPAhrk 8PmiBsPhEdjp2+d2fZlIa4xuTNSHgGBuaz/ZrSe9y17xrOwnFE+EM1rMEx1ti4ht/dKT kX0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=JXI6bjAn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id rl13si4042087ejb.128.2021.08.14.04.37.13; Sat, 14 Aug 2021 04:37:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=JXI6bjAn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 25BC568A667; Sat, 14 Aug 2021 14:36:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DDC8168A613 for ; Sat, 14 Aug 2021 14:36:36 +0300 (EEST) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 9DDF24360C; Sat, 14 Aug 2021 13:36:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1628940996; bh=t99IpPkvtPSeh9gJKCJlH11AwGoT/U27/98dRcar1ng=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JXI6bjAnDJsWGHKCm4I8159O+tSGDmF2EtMGcg9y1g5ZSgdS1zXkLYLECxfWS+yN4 u9iquumHL0UzOkprHETZ+nI8sKKD16jm1LyReR7LOszyYC4gEI4nis8qU8kz5pzPg5 eI7/ybexaKSMSXnXTx8VId5cnKQ7dDAlTT+Gus4E= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Sat, 14 Aug 2021 13:36:20 +0200 Message-Id: <20210814113620.39290-4-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210814113620.39290-1-ffmpeg@haasn.xyz> References: <20210814113620.39290-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/4] avcodec/h264dec: apply H.274 film grain X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Hlps+f6kJ61p From: Niklas Haas Because we need access to ref frames without film grain applied, we have to add an extra AVFrame to H264Picture to avoid messing with the original. This requires some amount of overhead to make the reference moves work out, but it allows us to benefit from frame multithreading for film grain application "for free". Unfortunately, this approach requires twice as much RAM to be constantly allocated, due to the need for an extra buffer per H264Picture. In theory, we could get away with freeing up this memory as soon as it's no longer needed, but trying to call ff_thread_release_buffer() from output_frame() simply deadlocks the decoder and I haven't figured out why. Patches welcome(tm) Tested on all three cases of (no fg), (fg present but exported) and (fg present and not exported), with and without threading. --- libavcodec/h264_picture.c | 24 +++++++++++++++++++- libavcodec/h264_slice.c | 18 +++++++++++++-- libavcodec/h264dec.c | 48 +++++++++++++++++++++++++++++++-------- libavcodec/h264dec.h | 6 +++++ 4 files changed, 83 insertions(+), 13 deletions(-) diff --git a/libavcodec/h264_picture.c b/libavcodec/h264_picture.c index b79944b794..8284d12158 100644 --- a/libavcodec/h264_picture.c +++ b/libavcodec/h264_picture.c @@ -44,7 +44,7 @@ void ff_h264_unref_picture(H264Context *h, H264Picture *pic) { - int off = offsetof(H264Picture, tf) + sizeof(pic->tf); + int off = offsetof(H264Picture, tf_grain) + sizeof(pic->tf_grain); int i; if (!pic->f || !pic->f->buf[0]) @@ -61,6 +61,9 @@ void ff_h264_unref_picture(H264Context *h, H264Picture *pic) av_buffer_unref(&pic->ref_index_buf[i]); } + if (pic->needs_fg) + ff_thread_release_buffer(h->avctx, &pic->tf_grain); + memset((uint8_t*)pic + off, 0, sizeof(*pic) - off); } @@ -109,6 +112,15 @@ int ff_h264_ref_picture(H264Context *h, H264Picture *dst, H264Picture *src) if (ret < 0) goto fail; + if (src->needs_fg) { + av_assert0(src->tf_grain.f == src->f_grain); + dst->tf_grain.f = dst->f_grain; + ret = ff_thread_ref_frame(&dst->tf_grain, &src->tf_grain); + if (ret < 0) + goto fail; + dst->needs_fg = 1; + } + dst->qscale_table_buf = av_buffer_ref(src->qscale_table_buf); dst->mb_type_buf = av_buffer_ref(src->mb_type_buf); dst->pps_buf = av_buffer_ref(src->pps_buf); @@ -160,6 +172,16 @@ int ff_h264_replace_picture(H264Context *h, H264Picture *dst, const H264Picture if (ret < 0) goto fail; + if (src->needs_fg) { + av_assert0(src->tf_grain.f == src->f_grain); + dst->tf_grain.f = dst->f_grain; + ff_thread_release_buffer(h->avctx, &dst->tf_grain); + ret = ff_thread_ref_frame(&dst->tf_grain, &src->tf_grain); + if (ret < 0) + goto fail; + dst->needs_fg = 1; + } + ret = av_buffer_replace(&dst->qscale_table_buf, src->qscale_table_buf); ret |= av_buffer_replace(&dst->mb_type_buf, src->mb_type_buf); ret |= av_buffer_replace(&dst->pps_buf, src->pps_buf); diff --git a/libavcodec/h264_slice.c b/libavcodec/h264_slice.c index 65d0259a71..2b85f807df 100644 --- a/libavcodec/h264_slice.c +++ b/libavcodec/h264_slice.c @@ -197,6 +197,21 @@ static int alloc_picture(H264Context *h, H264Picture *pic) if (ret < 0) goto fail; + if (h->sei.film_grain_characteristics.present && + !(h->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN)) + { + // Extra buffer required for film grain synthesis in the decoder + pic->tf_grain.f = pic->f_grain; + pic->f_grain->format = pic->f->format; + pic->f_grain->width = pic->f->width; + pic->f_grain->height = pic->f->height; + ret = ff_thread_get_buffer(h->avctx, &pic->tf_grain, 0); + if (ret < 0) + goto fail; + pic->needs_fg = 1; + } + + if (h->avctx->hwaccel) { const AVHWAccel *hwaccel = h->avctx->hwaccel; av_assert0(!pic->hwaccel_picture_private); @@ -1329,8 +1344,7 @@ static int h264_export_frame_props(H264Context *h) } h->sei.unregistered.nb_buf_ref = 0; - if (h->sei.film_grain_characteristics.present && - (h->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN)) { + if (h->sei.film_grain_characteristics.present) { H264SEIFilmGrainCharacteristics *fgc = &h->sei.film_grain_characteristics; AVFilmGrainParams *fgp = av_film_grain_params_create_side_data(out); if (!fgp) diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c index 38f8967265..d5878b8a13 100644 --- a/libavcodec/h264dec.c +++ b/libavcodec/h264dec.c @@ -275,9 +275,22 @@ int ff_h264_slice_context_init(H264Context *h, H264SliceContext *sl) return 0; } +static int h264_init_pic(H264Picture *pic) +{ + pic->f = av_frame_alloc(); + if (!pic->f) + return AVERROR(ENOMEM); + + pic->f_grain = av_frame_alloc(); + if (!pic->f_grain) + return AVERROR(ENOMEM); + + return 0; +} + static int h264_init_context(AVCodecContext *avctx, H264Context *h) { - int i; + int i, ret; h->avctx = avctx; h->cur_chroma_format_idc = -1; @@ -308,18 +321,15 @@ static int h264_init_context(AVCodecContext *avctx, H264Context *h) } for (i = 0; i < H264_MAX_PICTURE_COUNT; i++) { - h->DPB[i].f = av_frame_alloc(); - if (!h->DPB[i].f) - return AVERROR(ENOMEM); + if ((ret = h264_init_pic(&h->DPB[i])) < 0) + return ret; } - h->cur_pic.f = av_frame_alloc(); - if (!h->cur_pic.f) - return AVERROR(ENOMEM); + if ((ret = h264_init_pic(&h->cur_pic)) < 0) + return ret; - h->last_pic_for_ec.f = av_frame_alloc(); - if (!h->last_pic_for_ec.f) - return AVERROR(ENOMEM); + if ((ret = h264_init_pic(&h->last_pic_for_ec)) < 0) + return ret; for (i = 0; i < h->nb_slice_ctx; i++) h->slice_ctx[i].h264 = h; @@ -826,6 +836,21 @@ static int output_frame(H264Context *h, AVFrame *dst, H264Picture *srcp) AVFrame *src = srcp->f; int ret; + if (srcp->needs_fg) { + AVFrameSideData *sd = av_frame_get_side_data(src, AV_FRAME_DATA_FILM_GRAIN_PARAMS); + av_assert0(sd); + ret = ff_h274_apply_film_grain(srcp->f_grain, src, &h->h274db, + (AVFilmGrainParams *) sd->data); + if (!ret) { + if ((ret = av_frame_copy_props(srcp->f_grain, src)) < 0) + return ret; + src = srcp->f_grain; + } else { + av_log(h->avctx, AV_LOG_WARNING, "Failed synthesizing film " + "grain, ignoring: %s\n", av_err2str(ret)); + } + } + ret = av_frame_ref(dst, src); if (ret < 0) return ret; @@ -841,6 +866,9 @@ static int output_frame(H264Context *h, AVFrame *dst, H264Picture *srcp) goto fail; } + if (!(h->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN)) + av_frame_remove_side_data(dst, AV_FRAME_DATA_FILM_GRAIN_PARAMS); + return 0; fail: av_frame_unref(dst); diff --git a/libavcodec/h264dec.h b/libavcodec/h264dec.h index 7c419de051..7d11d81f99 100644 --- a/libavcodec/h264dec.h +++ b/libavcodec/h264dec.h @@ -43,6 +43,7 @@ #include "h264dsp.h" #include "h264pred.h" #include "h264qpel.h" +#include "h274.h" #include "internal.h" #include "mpegutils.h" #include "parser.h" @@ -130,6 +131,9 @@ typedef struct H264Picture { AVFrame *f; ThreadFrame tf; + AVFrame *f_grain; ///< extra buffer for film grain synthesis + ThreadFrame tf_grain; + AVBufferRef *qscale_table_buf; int8_t *qscale_table; @@ -162,6 +166,7 @@ typedef struct H264Picture { int recovered; ///< picture at IDR or recovery point + recovery count int invalid_gap; int sei_recovery_frame_cnt; + int needs_fg; ///< whether picture needs film grain synthesis (see `f_grain`) AVBufferRef *pps_buf; const PPS *pps; @@ -349,6 +354,7 @@ typedef struct H264Context { H264DSPContext h264dsp; H264ChromaContext h264chroma; H264QpelContext h264qpel; + H274FilmGrainDatabase h274db; H264Picture DPB[H264_MAX_PICTURE_COUNT]; H264Picture *cur_pic_ptr;