From patchwork Tue Sep 28 23:06:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 30653 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp5693436iob; Tue, 28 Sep 2021 16:06:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz4r9cgiho+r0wYNu7ioZC4YUTUvmIuYqJyCFRHX5RLP1FgxSKvRWo0mztXSAk9zPCamSEs X-Received: by 2002:a05:6402:4d1:: with SMTP id n17mr10995244edw.337.1632870375842; Tue, 28 Sep 2021 16:06:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1632870375; cv=none; d=google.com; s=arc-20160816; b=jQty+6Ln+V+OW1brHc8I4Nt9Q0+SB0xbkeQGdqtkn99JEsSdb2gkt+8li/QN1IbOCI 7iwpKMKpJ+WmT8vdN9vqcsrwxHhVTG3G7q05XdDgc6GI/HHiYMMM7ToOK5S6gogMfu9P G1STbvovOwwVYF6Oiwf1qXtvebhOMAFuKLlNPHrfx+lc2HqqVsoAlDdQz+YXv74kd7Fi nVYbrLks1+uDKiOgv04Tnta5yjYaRG0I+QFq0StpBz8vPrWV1EicLDhPbYTKO/wN9BD1 5xPaJGxtyoxztajguXr6bevd6FzWQEeVIF0vKc/hQ7+zByzyfkYe3F/RFU7LmpudfI1T dhJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=wRIVRfEJDIXLOYoKTX4rY/O0XuaD7fZO++U2Bo2nqGM=; b=KXbudgXJKplnLpBvk93i1YNUO9meEBC6rfrGrbDC9spypWjnsq51orlqbK4vVKj6ln EVV+SdBKeCkLpB3PsoKk7m8q/poKMB/Xhz2a95NXvQbtmLvBQzRqv7zxOgIOLQvlSZI0 d+tXbS/s8LFXhnHsElukyMXE+WUchxWx+RsUO0cul84GsUr9eFUDz15XOMDkhFuSnqAB yPKoan1UB6TJpnkUxbEP38NzBoaJslTbD7F3Eml3T9Rf464xPofY7sp450gtov2GGv4s GAEW6gLlbfE6hCpJMrXbbRN5pHy6rKJzMZmThRW0q6TORRbM0+7wATyYmaOTySA3Ppuk fubw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b="jdrpE/kl"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t10si472163ejg.316.2021.09.28.16.06.15; Tue, 28 Sep 2021 16:06:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b="jdrpE/kl"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 122AA6808E6; Wed, 29 Sep 2021 02:06:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 03C76688156 for ; Wed, 29 Sep 2021 02:06:05 +0300 (EEST) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 475CB47E8E; Wed, 29 Sep 2021 01:06:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1632870365; bh=VJy/XXy0n4Y5HvmXznu688MmZ/qrfKl+9zcYmEz293I=; h=From:To:Cc:Subject:Date:From; b=jdrpE/klkRcp8RukO5xoo0DfAUQGLL15TPwTSWHUkw3lZ8EPPhk4ib17AzRCQYSZ7 zt2+qsWeh65fLlOqOs3mAZD2TmrWNBFmSvjosoWSjGpsr6upXAIyQFrbThfQqx5rkv x3TchayhtXyR2llRp2EBHN6DsgT+cYVxDEXRLlD8= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 29 Sep 2021 01:06:02 +0200 Message-Id: <20210928230602.93222-1-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2] avcodec/hevcdec: apply H.274 film grain X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1m7ZruyfLU84 From: Niklas Haas Similar in spirit and design to 66845cffc3bbb, but slightly simpler due to the lack of interlaced frames in HEVC. See that commit for more details. For the seed value, since no specification for this appears to exist, I semi-arbitrarily decided to base it off the POC id alone, since there's no analog of the idr_pic_id in HEVC's I-frames. This design is stable across remuxes and seeks, but changes for adjacent frames with a period that's typically long enough not to be noticeable, which makes it satisfy all of the requirements that a film grain seed should have. Tested with and without threading, using a patch to insert film grain metadata artificially (for lack of real files containing film grain). --- libavcodec/hevc_refs.c | 7 +++++- libavcodec/hevcdec.c | 57 ++++++++++++++++++++++++++++++++++++++++-- libavcodec/hevcdec.h | 5 ++++ 3 files changed, 66 insertions(+), 3 deletions(-) diff --git a/libavcodec/hevc_refs.c b/libavcodec/hevc_refs.c index 4f6d985ae6..a61b8b22b7 100644 --- a/libavcodec/hevc_refs.c +++ b/libavcodec/hevc_refs.c @@ -38,6 +38,8 @@ void ff_hevc_unref_frame(HEVCContext *s, HEVCFrame *frame, int flags) frame->flags &= ~flags; if (!frame->flags) { ff_thread_release_buffer(s->avctx, &frame->tf); + ff_thread_release_buffer(s->avctx, &frame->tf_grain); + frame->needs_fg = 0; av_buffer_unref(&frame->tab_mvf_buf); frame->tab_mvf = NULL; @@ -208,7 +210,7 @@ int ff_hevc_output_frame(HEVCContext *s, AVFrame *out, int flush) if (nb_output) { HEVCFrame *frame = &s->DPB[min_idx]; - ret = av_frame_ref(out, frame->frame); + ret = av_frame_ref(out, frame->needs_fg ? frame->frame_grain : frame->frame); if (frame->flags & HEVC_FRAME_FLAG_BUMPING) ff_hevc_unref_frame(s, frame, HEVC_FRAME_FLAG_OUTPUT | HEVC_FRAME_FLAG_BUMPING); else @@ -216,6 +218,9 @@ int ff_hevc_output_frame(HEVCContext *s, AVFrame *out, int flush) if (ret < 0) return ret; + if (!(s->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN)) + av_frame_remove_side_data(out, AV_FRAME_DATA_FILM_GRAIN_PARAMS); + av_log(s->avctx, AV_LOG_DEBUG, "Output frame with POC %d.\n", frame->poc); return 1; diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 57a61752a3..56811bee12 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -2881,14 +2881,14 @@ static int set_side_data(HEVCContext *s) s->sei.timecode.num_clock_ts = 0; } - if (s->sei.film_grain_characteristics.present && - (s->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN)) { + if (s->sei.film_grain_characteristics.present) { HEVCSEIFilmGrainCharacteristics *fgc = &s->sei.film_grain_characteristics; AVFilmGrainParams *fgp = av_film_grain_params_create_side_data(out); if (!fgp) return AVERROR(ENOMEM); fgp->type = AV_FILM_GRAIN_PARAMS_H274; + fgp->seed = s->ref->poc; /* no poc_offset in HEVC */ fgp->codec.h274.model_id = fgc->model_id; if (fgc->separate_colour_description_present_flag) { @@ -2983,6 +2983,18 @@ static int hevc_frame_start(HEVCContext *s) s->ref->frame->key_frame = IS_IRAP(s); + s->ref->needs_fg = s->sei.film_grain_characteristics.present && + !(s->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) && + !s->avctx->hwaccel; + + if (s->ref->needs_fg) { + s->ref->frame_grain->format = s->ref->frame->format; + s->ref->frame_grain->width = s->ref->frame->width; + s->ref->frame_grain->height = s->ref->frame->height; + if ((ret = ff_thread_get_buffer(s->avctx, &s->ref->tf_grain, 0)) < 0) + goto fail; + } + ret = set_side_data(s); if (ret < 0) goto fail; @@ -3009,6 +3021,31 @@ fail: return ret; } +static int hevc_frame_end(HEVCContext *s) +{ + HEVCFrame *out = s->ref; + const AVFrameSideData *sd; + int ret; + + if (out->needs_fg) { + sd = av_frame_get_side_data(out->frame, AV_FRAME_DATA_FILM_GRAIN_PARAMS); + av_assert0(out->frame_grain->buf[0] && sd); + ret = ff_h274_apply_film_grain(out->frame_grain, out->frame, &s->h274db, + (AVFilmGrainParams *) sd->data); + + if (ret < 0) { + av_log(s->avctx, AV_LOG_WARNING, "Failed synthesizing film " + "grain, ignoring: %s\n", av_err2str(ret)); + out->needs_fg = 0; + } else { + if ((ret = av_frame_copy_props(out->frame_grain, out->frame)) < 0) + return ret; + } + } + + return 0; +} + static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) { HEVCLocalContext *lc = s->HEVClc; @@ -3167,6 +3204,9 @@ static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) else ctb_addr_ts = hls_slice_data(s); if (ctb_addr_ts >= (s->ps.sps->ctb_width * s->ps.sps->ctb_height)) { + ret = hevc_frame_end(s); + if (ret < 0) + goto fail; s->is_decoded = 1; } @@ -3426,6 +3466,13 @@ static int hevc_ref_frame(HEVCContext *s, HEVCFrame *dst, HEVCFrame *src) if (ret < 0) return ret; + if (src->needs_fg) { + ret = ff_thread_ref_frame(&dst->tf_grain, &src->tf_grain); + if (ret < 0) + return ret; + dst->needs_fg = 1; + } + dst->tab_mvf_buf = av_buffer_ref(src->tab_mvf_buf); if (!dst->tab_mvf_buf) goto fail; @@ -3478,6 +3525,7 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { ff_hevc_unref_frame(s, &s->DPB[i], ~0); av_frame_free(&s->DPB[i].frame); + av_frame_free(&s->DPB[i].frame_grain); } ff_hevc_ps_uninit(&s->ps); @@ -3531,6 +3579,11 @@ static av_cold int hevc_init_context(AVCodecContext *avctx) if (!s->DPB[i].frame) goto fail; s->DPB[i].tf.f = s->DPB[i].frame; + + s->DPB[i].frame_grain = av_frame_alloc(); + if (!s->DPB[i].frame_grain) + goto fail; + s->DPB[i].tf_grain.f = s->DPB[i].frame_grain; } s->max_ra = INT_MAX; diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index 482638a8e5..8963aa83ae 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -39,6 +39,7 @@ #include "hevc_ps.h" #include "hevc_sei.h" #include "hevcdsp.h" +#include "h274.h" #include "internal.h" #include "thread.h" #include "videodsp.h" @@ -395,7 +396,10 @@ typedef struct DBParams { typedef struct HEVCFrame { AVFrame *frame; + AVFrame *frame_grain; ThreadFrame tf; + ThreadFrame tf_grain; + int needs_fg; /* 1 if grain needs to be applied by the decoder */ MvField *tab_mvf; RefPicList *refPicList; RefPicListTab **rpl_tab; @@ -525,6 +529,7 @@ typedef struct HEVCContext { HEVCDSPContext hevcdsp; VideoDSPContext vdsp; BswapDSPContext bdsp; + H274FilmGrainDatabase h274db; int8_t *qp_y_tab; uint8_t *horizontal_bs; uint8_t *vertical_bs;