From patchwork Fri Sep 24 09:55:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 30552 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp1329524iob; Fri, 24 Sep 2021 02:56:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz1mLDflcjSf6ySGyJ1J4VNwMzGn9Bzvzr+Vb8KQscN74Yxr3JQZ/9E3FzZ7VgU27YcacCM X-Received: by 2002:a17:906:ccca:: with SMTP id ot10mr10008975ejb.429.1632477368756; Fri, 24 Sep 2021 02:56:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1632477368; cv=none; d=google.com; s=arc-20160816; b=jm/tyfrDN0SgheG+JhF1cLI266Jzhu6I3n+HaD5RE9g+VKgl7uP9oJYwY+QzxFg+oZ 4UIZirlnhHTP159RCGZZXsp6ZZpWCvArkoTo+c2/dW/xUAtGe4yKyVOA4VhVJLN/akhp o7PVCXPgUaShmtSoUj4yqYxY/XA92zhdU3qrwvLDd8e9zgQTWMsXRere21LhmcmXeh57 YAt3nb4JPGtg6P1j/SsClwyIvVCO275FFJdL7TB3URWfjnzV0axOyg9M+zRI9LICrmAA ErZ30sbGulOrXyz1Ae4usaBMbOQ3iZeuE0Buep/6N1sam7Vof1Dp/VkZ9Xn6HKVifcg9 seWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=S4U7ZiZtHfgMvIIFjkPhvxSjKe0BEH1RsnPyZ11DhEE=; b=0XHqZBsBFZwuq7sad9E66d7ulKgK1vkmGH9GYUcWicBX5oK16kcJ4FDZ9X94obepSS 2XLVn8hRQAvYEkl7QhZe7TD6StV7Pf2u+fYq6NcUgQh27VUGsVgyupfMWwTt9ONbkJNF sqihiUreBEnyUOkzgKnkkrOzWWyOLOVRZ4GPhbzCThBj+gOByWjtNASAaEXA1N6nmYJJ ODR6pxa4KRIWVEXPE4nYLU1wnARa8tA7wxYbjqkgNumdIY+PONxKAKhHReDO4eOGzVvA JsFfaIUtwymFuYG8PawEew2mgFmvvM5LS9V2AezCSkuTCDSPchdztp9fFiOSTCe1aThU xmcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=n+vOIzpP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y20si8402966edi.311.2021.09.24.02.56.07; Fri, 24 Sep 2021 02:56:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=n+vOIzpP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0C79368A51F; Fri, 24 Sep 2021 12:56:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 045A5689E05 for ; Fri, 24 Sep 2021 12:55:56 +0300 (EEST) Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id E342C477EB; Fri, 24 Sep 2021 11:55:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1632477355; bh=pq+cbe7AynE0+miftFd893FEM5PQJQ23GX33zN92HoQ=; h=From:To:Cc:Subject:Date:From; b=n+vOIzpPmicOTxHCyPnxGgXl0qBliMRHPGjjR9+uf7XVY0fhVXrFXgnI7KQwHDI6G RB26xNJsGYc2riV7fztOK+ouskSTMoyHfURWYw5eaWBA4ecLX0m6l+AuioUeNtihUB z7ZNcKXxCE/Y6uMG7Mq8JxAQ+eUeCVcBKX93jA7k= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Fri, 24 Sep 2021 11:55:54 +0200 Message-Id: <20210924095554.116659-1-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/hevcdec: apply H.274 film grain X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 5olmzxj7mblG From: Niklas Haas Similar in spirit and design to 66845cffc3bbb, but slightly simpler due to the lack of interlaced frames in HEVC. See that commit for more details. For the seed value, since no specification for this appears to exist, I semi-arbitrarily decided to base it off the POC id alone, since there's no analog of the idr_pic_id in HEVC's I-frames. This design is stable across remuxes and seeks, but changes for adjacent frames with a period that's typically long enough not to be noticeable, which makes it satisfy all of the requirements that a film grain seed should have. Tested with and without threading, using a patch to insert film grain metadata artificially (for lack of real files containing film grain). --- libavcodec/hevc_refs.c | 4 ++- libavcodec/hevcdec.c | 57 ++++++++++++++++++++++++++++++++++++++++-- libavcodec/hevcdec.h | 5 ++++ 3 files changed, 63 insertions(+), 3 deletions(-) diff --git a/libavcodec/hevc_refs.c b/libavcodec/hevc_refs.c index 4f6d985ae6..0a532451d0 100644 --- a/libavcodec/hevc_refs.c +++ b/libavcodec/hevc_refs.c @@ -38,6 +38,8 @@ void ff_hevc_unref_frame(HEVCContext *s, HEVCFrame *frame, int flags) frame->flags &= ~flags; if (!frame->flags) { ff_thread_release_buffer(s->avctx, &frame->tf); + ff_thread_release_buffer(s->avctx, &frame->tf_grain); + frame->needs_fg = 0; av_buffer_unref(&frame->tab_mvf_buf); frame->tab_mvf = NULL; @@ -208,7 +210,7 @@ int ff_hevc_output_frame(HEVCContext *s, AVFrame *out, int flush) if (nb_output) { HEVCFrame *frame = &s->DPB[min_idx]; - ret = av_frame_ref(out, frame->frame); + ret = av_frame_ref(out, frame->needs_fg ? frame->frame_grain : frame->frame); if (frame->flags & HEVC_FRAME_FLAG_BUMPING) ff_hevc_unref_frame(s, frame, HEVC_FRAME_FLAG_OUTPUT | HEVC_FRAME_FLAG_BUMPING); else diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 57a61752a3..a86320a034 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -2881,14 +2881,14 @@ static int set_side_data(HEVCContext *s) s->sei.timecode.num_clock_ts = 0; } - if (s->sei.film_grain_characteristics.present && - (s->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN)) { + if (s->sei.film_grain_characteristics.present) { HEVCSEIFilmGrainCharacteristics *fgc = &s->sei.film_grain_characteristics; AVFilmGrainParams *fgp = av_film_grain_params_create_side_data(out); if (!fgp) return AVERROR(ENOMEM); fgp->type = AV_FILM_GRAIN_PARAMS_H274; + fgp->seed = s->ref->poc; /* no poc_offset in HEVC */ fgp->codec.h274.model_id = fgc->model_id; if (fgc->separate_colour_description_present_flag) { @@ -2983,6 +2983,17 @@ static int hevc_frame_start(HEVCContext *s) s->ref->frame->key_frame = IS_IRAP(s); + s->ref->needs_fg = s->sei.film_grain_characteristics.present && + !(s->avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN); + + if (s->ref->needs_fg) { + s->ref->frame_grain->format = s->ref->frame->format; + s->ref->frame_grain->width = s->ref->frame->width; + s->ref->frame_grain->height = s->ref->frame->height; + if ((ret = ff_thread_get_buffer(s->avctx, &s->ref->tf_grain, 0)) < 0) + goto fail; + } + ret = set_side_data(s); if (ret < 0) goto fail; @@ -3009,6 +3020,32 @@ fail: return ret; } +static int hevc_frame_end(HEVCContext *s) +{ + HEVCFrame *out = s->ref; + const AVFrameSideData *sd; + int ret; + + if (out->needs_fg) { + sd = av_frame_get_side_data(out->frame, AV_FRAME_DATA_FILM_GRAIN_PARAMS); + av_assert0(out->frame_grain->buf[0] && sd); + ret = ff_h274_apply_film_grain(out->frame_grain, out->frame, &s->h274db, + (AVFilmGrainParams *) sd->data); + av_frame_remove_side_data(out->frame, AV_FRAME_DATA_FILM_GRAIN_PARAMS); + + if (ret < 0) { + av_log(s->avctx, AV_LOG_WARNING, "Failed synthesizing film " + "grain, ignoring: %s\n", av_err2str(ret)); + out->needs_fg = 0; + } else { + if ((ret = av_frame_copy_props(out->frame_grain, out->frame)) < 0) + return ret; + } + } + + return 0; +} + static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) { HEVCLocalContext *lc = s->HEVClc; @@ -3167,6 +3204,9 @@ static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) else ctb_addr_ts = hls_slice_data(s); if (ctb_addr_ts >= (s->ps.sps->ctb_width * s->ps.sps->ctb_height)) { + ret = hevc_frame_end(s); + if (ret < 0) + goto fail; s->is_decoded = 1; } @@ -3426,6 +3466,13 @@ static int hevc_ref_frame(HEVCContext *s, HEVCFrame *dst, HEVCFrame *src) if (ret < 0) return ret; + if (src->needs_fg) { + ret = ff_thread_ref_frame(&dst->tf_grain, &src->tf_grain); + if (ret < 0) + return ret; + dst->needs_fg = 1; + } + dst->tab_mvf_buf = av_buffer_ref(src->tab_mvf_buf); if (!dst->tab_mvf_buf) goto fail; @@ -3478,6 +3525,7 @@ static av_cold int hevc_decode_free(AVCodecContext *avctx) for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { ff_hevc_unref_frame(s, &s->DPB[i], ~0); av_frame_free(&s->DPB[i].frame); + av_frame_free(&s->DPB[i].frame_grain); } ff_hevc_ps_uninit(&s->ps); @@ -3531,6 +3579,11 @@ static av_cold int hevc_init_context(AVCodecContext *avctx) if (!s->DPB[i].frame) goto fail; s->DPB[i].tf.f = s->DPB[i].frame; + + s->DPB[i].frame_grain = av_frame_alloc(); + if (!s->DPB[i].frame_grain) + goto fail; + s->DPB[i].tf_grain.f = s->DPB[i].frame_grain; } s->max_ra = INT_MAX; diff --git a/libavcodec/hevcdec.h b/libavcodec/hevcdec.h index 482638a8e5..8963aa83ae 100644 --- a/libavcodec/hevcdec.h +++ b/libavcodec/hevcdec.h @@ -39,6 +39,7 @@ #include "hevc_ps.h" #include "hevc_sei.h" #include "hevcdsp.h" +#include "h274.h" #include "internal.h" #include "thread.h" #include "videodsp.h" @@ -395,7 +396,10 @@ typedef struct DBParams { typedef struct HEVCFrame { AVFrame *frame; + AVFrame *frame_grain; ThreadFrame tf; + ThreadFrame tf_grain; + int needs_fg; /* 1 if grain needs to be applied by the decoder */ MvField *tab_mvf; RefPicList *refPicList; RefPicListTab **rpl_tab; @@ -525,6 +529,7 @@ typedef struct HEVCContext { HEVCDSPContext hevcdsp; VideoDSPContext vdsp; BswapDSPContext bdsp; + H274FilmGrainDatabase h274db; int8_t *qp_y_tab; uint8_t *horizontal_bs; uint8_t *vertical_bs;