From patchwork Wed Aug 21 07:56:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Poorva <2003gaikarpoorva@gmail.com> X-Patchwork-Id: 51096 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:612c:4062:b0:48e:c0f8:d0de with SMTP id kz34csp261526vqb; Wed, 21 Aug 2024 00:56:56 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUMuFjMtdFtd142c8MZvWqsnhzsNUdAV12eZifY/EG/+M0YeEZyQpPbagVyBtudo5stR+n457RGdNoGeegxJHVx@gmail.com X-Google-Smtp-Source: AGHT+IHE3s1u3rk46hVgC0+TPYuInYX57PRPyP9S084UOaIp6aRv3rUnPwflJW2pg84O1+lD8Gqu X-Received: by 2002:a2e:b04f:0:b0:2ef:3292:4bdf with SMTP id 38308e7fff4ca-2f3f88a7428mr8352631fa.17.1724227016571; Wed, 21 Aug 2024 00:56:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1724227016; cv=none; d=google.com; s=arc-20160816; b=SWEHPyHdGxvXnmJeA6xANoN0gaOtesK3ukrecRr7KzA6ySkOXXLDfQKmFwctXngx12 fOJ6jZDQ1Ys7ErGUnAJXUIoVwXnaqXN/vM52ftKP0k4fgSftfiRtblroQj/NHx1aAz/5 HxGL/ukDp7oqunSNDIyxtk2qvUPrqlMJy0yinNW9LndYLQzPONIngJ+/PaK6qrsjIDSP 3vO/OSzqvjjuWICMrm/FUy+0GzXR8lrEMNU+Fp7yF1tV+GI4a84UR4bSE9rFgweS339N lmm4oz3buCatnUMtl4Yx30uCbk/0xvlF8AsqErefIaB2Xf2TGsq4d5EzfJAHagioiKsu NpRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:to :message-id:date:from:mime-version:dkim-signature:delivered-to; bh=6qp3OlsoUnV6fSyAvIjnDhzWUJOU3jvhZhK9KvfGqX8=; fh=pyH7MndpbTeOtgCf5Kyqq1yxfR7bfA2oazyaJgqL0nI=; b=OT1eMEMMDEKhaCbZxeGYzUqkefo3zGOaZkizC4sBZS/3WB+e0M3AQMyPpIjaGwaZGE 9SYPAXM2ZRm+SvjVsFyI1USARbq7NsKXn85MTtGPx8Jac7dDCGnaCUerC/ucxDUBycc3 3qSh5j13h65+FM8MF3M7XDB7NFopiR5dcIP7n6gcDGkInEAV+zkTuXViHuEHsq9rF765 GBNQs1T5KQqozkVKqrtn5rT+Tevroqzbe+++HAAFmfLsUDgHxGXp52FGl3sxYKbIdg7k wfpw+uEdlaSpTquYo2spodRShym18zbWm4cH25jaR5BnPcSvty0kzARyRkz2BwkWle2H dQbQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=c6T+7yLH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 38308e7fff4ca-2f3faabdb3dsi2275871fa.657.2024.08.21.00.56.56; Wed, 21 Aug 2024 00:56:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=c6T+7yLH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8DBF268DA2D; Wed, 21 Aug 2024 10:56:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 49B9368DA2D for ; Wed, 21 Aug 2024 10:56:44 +0300 (EEST) Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2f3cb747fafso48466861fa.3 for ; Wed, 21 Aug 2024 00:56:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724227003; x=1724831803; darn=ffmpeg.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=5pZHoDG2ky5ML92vTe7nozBJPRkK6S6yTffsHp/9n/o=; b=c6T+7yLHzWwotXpNmF/n0BKDzRtZZkVHiECf5gBGxBH9aqKOzjAObp6oKIH6q139sf yq8ouYkJK/3KGeJ2cDh8kjwRUm0Jr9OOOs5P9r4gNvI1VEecocPQS32yWljnGFM7Nqh6 4v8wObnpGyKN/vzRIX1B8oDhJC/W+5kO7QeFZQtYK9Oh8CJQps6FeH3oy8LkV/RZ00sF e22C3FFF1aWJMpA2wzRHJjVn/J+MGrK1ZccykCj0xXYi2alxtSoSFS7xb5VNsWpujXl7 Xk+sJ12CZyN1babtPas+c3GLngMEtBQGgMToDlFv8oY4LU5rDlUIR7ubwHtHCKa2MpvT 9rTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724227003; x=1724831803; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=5pZHoDG2ky5ML92vTe7nozBJPRkK6S6yTffsHp/9n/o=; b=LnmOd0sPcH7jjCvR9i+OqXdrqr6wJwBh/FNqQvyIJBEMXFM/kAEUdgDvE1Eso6Mt37 5X1d65xgJnusLqOu8XGwnLK9PEB72uNYBQVhCpC+7JbbqzCZWcrtcHVtaj7v/Jy3Ulhy 4LgeQLZfTZcAhOAFoHV0IGjJBi+j1/DruA2GdGSAZxR3X3phfsqWVSAMyZji5hCOyHFh /o9guXsEx/uJpd6aq2yQYQd0DiaMFMgiff0lnXIXf5egO70T3Vz85IDMNewzg9DIkDji lkLhMbE3Yfk8R2iSeM5Rb9Mi+hBFTtsqApTSyafughviTF/+ZOoI0G4JlvL/HExnylgW GCLg== X-Gm-Message-State: AOJu0Yy6XSx2TXPBDGsyUNKnG1XSbBuPQu/5itpgMhKwZkHBuDX5s7Wj pSMzqA9fOPQCLYDj/Em4hUm0shiiIZn+guDn6UvC2qyqfScUTq/VxoHN+dT6+I3wNvdFt2YgUoZ tr2ZedSeyMZLjOraFvk0eC13aX3sQKA+C X-Received: by 2002:a2e:88d1:0:b0:2ef:1784:a20 with SMTP id 38308e7fff4ca-2f3f8b546ecmr8497141fa.38.1724227002462; Wed, 21 Aug 2024 00:56:42 -0700 (PDT) MIME-Version: 1.0 From: Poorva <2003gaikarpoorva@gmail.com> Date: Wed, 21 Aug 2024 13:26:29 +0530 Message-ID: To: FFmpeg development discussions and patches , Thilo Borgmann , Cosmin Stejerean X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [FFmpeg-devel] GSoC Project Submission: SiTi Filter Update for ITU-T P.910 Compliance and HDR Support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: YifT2HJ08Mju Hello FFmpeg Team, I am submitting the completed work from my Google Summer of Code project, which focuses on updating the SiTi filter for ITU-T P.910 07/22 compliance and adding HDR support. Attached is the patch for your review. I appreciate your feedback and suggestions. From e33fa1928164d986b5b62c02f00fab788a1f2f3b Mon Sep 17 00:00:00 2001 From: PoorvaGaikar <2003gaikarpoorva.com> Date: Wed, 17 Jul 2024 23:36:56 +0530 Subject: [PATCH] libavfilter/siti: Update SITI filter to align with ITU-T P.910 07/22 and add HDR support --- libavfilter/vf_siti.c | 217 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 208 insertions(+), 9 deletions(-) diff --git a/libavfilter/vf_siti.c b/libavfilter/vf_siti.c index 722e7cecc7..2bbcaa0afd 100644 --- a/libavfilter/vf_siti.c +++ b/libavfilter/vf_siti.c @@ -47,6 +47,19 @@ static const int Y_FILTER[9] = { -1, -2, -1 }; +static const float PU21_MATRICES[4][7] = { + // Reference: "PU21: A novel perceptually uniform encoding for adapting existing quality metrics for HDR" + // Rafał K. Mantiuk and M. Azimi, Picture Coding Symposium 2021 + // https://github.com/gfxdisp/pu21 + {1.070275272f, 0.4088273932f, 0.153224308f, 0.2520326168f, 1.063512885f, 1.14115047f, 521.4527484f}, // BANDING + {0.353487901f, 0.3734658629f, 8.277049286e-05f, 0.9062562627f, 0.09150303166f, 0.9099517204f, 596.3148142f}, // BANDING_GLARE + {1.043882782f, 0.6459495343f, 0.3194584211f, 0.374025247f, 1.114783422f, 1.095360363f, 384.9217577f}, // PEAKS + {816.885024f, 1479.463946f, 0.001253215609f, 0.9329636822f, 0.06746643971f, 1.573435413f, 419.6006374f} // PEAKS_GLARE +}; + +static const float PU21_MIN_VALUES[4] = {-1.5580e-07f, 5.4705e-10f, 1.3674e-07f, -9.7360e-08f}; +static const float PU21_MAX_VALUES[4] = {520.4673f, 595.3939f, 380.9853f, 407.5066f}; + typedef struct SiTiContext { const AVClass *class; int pixel_depth; @@ -63,8 +76,45 @@ typedef struct SiTiContext { float *motion_matrix; int full_range; int print_summary; + int hdr_mode; + int bit_depth; + int color_range; + int eotf_function; + int calculation_domain; + float l_max; + float l_min; + float gamma; + int pu21_mode; } SiTiContext; +enum HdrMode { + SDR, + HDR10, + HLG +}; + +enum ColorRange { + LIMITED, + FULL +}; + +enum EotfFunction { + BT1886, + INV_SRGB +}; + +enum CalculationDomain { + PQ, + PU21 +}; + +enum Pu21Mode { + BANDING, + BANDING_GLARE, + PEAKS, + PEAKS_GLARE +}; + static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUVJ420P, AV_PIX_FMT_YUVJ422P, @@ -78,6 +128,15 @@ static av_cold int init(AVFilterContext *ctx) SiTiContext *s = ctx->priv; s->max_si = 0; s->max_ti = 0; + s->hdr_mode = SDR; + s->bit_depth = 8; + s->color_range = LIMITED; + s->eotf_function = BT1886; + s->calculation_domain = PQ; + s->l_max = 300.0; + s->l_min = 0.1; + s->gamma = 2.4; + s->pu21_mode = BANDING; return 0; } @@ -166,6 +225,105 @@ static uint16_t convert_full_range(int factor, uint16_t y) return (full_upper * limit_y / limit_upper); } +// EOTF for BT.1886 +static float eotf_1886(float x, float gamma, float l_min, float l_max) +{ + // Reference: ITU-R BT.1886, Annex 1 + // https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1886-0-201103-I!!PDF-E.pdf + const float l_max_gamma = powf(l_max, 1.0f / gamma); + const float l_min_gamma = powf(l_min, 1.0f / gamma); + const float l_diff_gamma = l_max_gamma - l_min_gamma; + const float a = powf(l_diff_gamma, gamma); + const float b = l_min_gamma / l_diff_gamma; + const float adjusted_x = fmaxf(x + b, 0.0f); + + return a * powf(adjusted_x, gamma); +} + +static float eotf_inv_srgb(float x) +{ + // Inverse sRGB EOTF (Electro-Optical Transfer Function) according to IEC 61966-2-1:1999 + // Section G.2 (Encoding transformation) + // Reference: https://cdn.standards.iteh.ai/samples/10795/ae461684569b40bbbb2d9a22b1047f05/IEC-61966-2-1-1999-AMD1-2003.pdf + return (x <= 0.04045f) ? x / 12.92f : powf((x + 0.055f) / 1.055f, 2.4f); +} + +static float apply_display_model(SiTiContext *s, float x) +{ + // Apply either BT.1886 or inverse sRGB EOTF based on the context + if (s->eotf_function == BT1886) { + // BT.1886 EOTF + // Reference: ITU-R BT.1886-0, Annex 1 + // https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1886-0-201103-I!!PDF-E.pdf + return eotf_1886(x, s->gamma, 0.0f, 1.0f); + } else { + // Inverse sRGB EOTF + return eotf_inv_srgb(x); + } +} + +static float oetf_pq(float x) +{ + // PQ (Perceptual Quantizer) OETF according to ITU-R BT.2100-2 + // Reference: https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.2100-2-201807-I!!PDF-E.pdf + // See page 5, Table 4 + + const float m1 = 0.1593017578125f; + const float m2 = 78.84375f; + const float c1 = 0.8359375f; + const float c2 = 18.8515625f; + const float c3 = 18.6875f; + + float y = powf(x / 10000.0f, m1); + return powf((c1 + c2 * y) / (1.0f + c3 * y), m2); +} + +static float oetf_pu21(float x, int mode); + +static float oetf_pu21_wrapper(float x) { + int mode = 0; // Set the appropriate mode or pass it as a parameter + return oetf_pu21(x, mode); +} + +static float oetf_pu21(float x, int mode) +{ + // Reference: "PU21: A novel perceptually uniform encoding for adapting existing quality metrics for HDR" + // Rafał K. Mantiuk and M. Azimi, Picture Coding Symposium 2021 + // https://github.com/gfxdisp/pu21 + // Validate the mode + const float *p; + float p_min; + float p_max; + float numerator; + float denominator; + float base; + float exponentiated; + float scaled; + + if (mode < 0 || mode > 3) { + return x; // Invalid mode, return input unchanged + } + + // Get the parameters and min/max values for the given mode + + + p = PU21_MATRICES[mode]; + p_min = PU21_MIN_VALUES[mode]; + p_max = PU21_MAX_VALUES[mode]; + + // Declare all other variables at the beginning of the block + numerator = p[0] + p[1] * powf(x, p[3]); + denominator = 1.0f + p[2] * powf(x, p[3]); + base = numerator / denominator; + + // Calculate the exponentiated and scaled value + exponentiated = powf(base, p[4]); + scaled = p[6] * (exponentiated - p[5]); + + // Normalize the result to the range [0, 1] + return (scaled - p_min) / (p_max - p_min); +} + // Applies sobel convolution static void convolve_sobel(SiTiContext *s, const uint8_t *src, float *dst, int linesize) { @@ -186,20 +344,20 @@ static void convolve_sobel(SiTiContext *s, const uint8_t *src, float *dst, int l #define CONVOLVE(bps) \ { \ uint##bps##_t *vsrc = (uint##bps##_t*)src; \ - for (int j = 1; j < s->height - 1; j++) { \ - for (int i = 1; i < s->width - 1; i++) { \ + for (int img_j = 1; img_j < s->height - 1; img_j++) { \ + for (int img_i = 1; img_i < s->width - 1; img_i++) { \ x_conv_sum = 0.0; \ y_conv_sum = 0.0; \ for (int k = 0; k < filter_size; k++) { \ ki = k % filter_width - 1; \ kj = floor(k / filter_width) - 1; \ - index = (j + kj) * stride + (i + ki); \ + index = (img_j + kj) * stride + (img_i + ki); \ data = s->full_range ? vsrc[index] : convert_full_range(factor, vsrc[index]); \ x_conv_sum += data * X_FILTER[k]; \ y_conv_sum += data * Y_FILTER[k]; \ } \ gradient = sqrt(x_conv_sum * x_conv_sum + y_conv_sum * y_conv_sum); \ - dst[(j - 1) * (s->width - 2) + (i - 1)] = gradient; \ + dst[(img_j - 1) * (s->width - 2) + (img_i - 1)] = gradient; \ } \ } \ } @@ -254,15 +412,16 @@ static float std_deviation(float *img_metrics, int width, int height) int size = height * width; double mean = 0.0; double sqr_diff = 0; + int j, i; - for (int j = 0; j < height; j++) - for (int i = 0; i < width; i++) + for (j = 0; j < height; j++) + for (i = 0; i < width; i++) mean += img_metrics[j * width + i]; mean /= size; - for (int j = 0; j < height; j++) { - for (int i = 0; i < width; i++) { + for (j = 0; j < height; j++) { + for (i = 0; i < width; i++) { float mean_diff = img_metrics[j * width + i] - mean; sqr_diff += (mean_diff * mean_diff); } @@ -285,15 +444,46 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) float si; float ti; - s->full_range = is_full_range(frame); + s->full_range = is_full_range(frame); // Determine if full range is needed s->nb_frames++; + // Apply EOTF and OETF transformations + for (int idx = 0; idx < s->width * s->height; idx++) { + float pixel = ((uint8_t*)frame->data[0])[idx] / 255.0f; + float eotf = apply_display_model(s, pixel); + float oetf = oetf_pq(eotf * s->l_max); + ((uint8_t*)frame->data[0])[idx] = (uint8_t)(oetf * 255.0f); + } + // Calculate si and ti convolve_sobel(s, frame->data[0], s->gradient_matrix, frame->linesize[0]); calculate_motion(s, frame->data[0], s->motion_matrix, frame->linesize[0]); si = std_deviation(s->gradient_matrix, s->width - 2, s->height - 2); ti = std_deviation(s->motion_matrix, s->width, s->height); + // Apply HDR transformations if necessary + if (s->hdr_mode != SDR) { + float (*oetf_func)(float); + if (s->calculation_domain == PQ) { + oetf_func = oetf_pq; + } else { + oetf_func = oetf_pu21_wrapper; + } + + for (int i = 0; i < (s->width - 2) * (s->height - 2); i++) { + s->gradient_matrix[i] = oetf_func(apply_display_model(s, s->gradient_matrix[i] / 255.0f)) * 255.0f; + } + for (int i = 0; i < s->width * s->height; i++) { + s->motion_matrix[i] = oetf_func(apply_display_model(s, s->motion_matrix[i] / 255.0f)) * 255.0f; + } + + si = std_deviation(s->gradient_matrix, s->width - 2, s->height - 2); + ti = std_deviation(s->motion_matrix, s->width, s->height); + } + + // Print SI and TI values for each frame + av_log(ctx, AV_LOG_INFO, "Frame %"PRId64" - SI: %.6f, TI: %.6f\n", s->nb_frames, si, ti); + // Calculate statistics s->max_si = fmaxf(si, s->max_si); s->max_ti = fmaxf(ti, s->max_ti); @@ -314,6 +504,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) static const AVOption siti_options[] = { { "print_summary", "Print summary showing average values", OFFSET(print_summary), AV_OPT_TYPE_BOOL, { .i64=0 }, 0, 1, FLAGS }, + { "hdr_mode", "HDR mode (0: SDR, 1: HDR10, 2: HLG)", OFFSET(hdr_mode), AV_OPT_TYPE_INT, {.i64=0}, 0, 2, FLAGS }, + { "bit_depth", "Bit depth (8, 10, or 12)", OFFSET(bit_depth), AV_OPT_TYPE_INT, {.i64=8}, 8, 12, FLAGS }, + { "color_range", "Color range (0: limited, 1: full)", OFFSET(color_range), AV_OPT_TYPE_INT, {.i64=0}, 0, 1, FLAGS }, + { "eotf_function", "EOTF function (0: BT.1886, 1: Inverse sRGB)", OFFSET(eotf_function), AV_OPT_TYPE_INT, {.i64=0}, 0, 1, FLAGS }, + { "calculation_domain", "Calculation domain (0: PQ, 1: PU21)", OFFSET(calculation_domain), AV_OPT_TYPE_INT, {.i64=0}, 0, 1, FLAGS }, + { "l_max", "Maximum luminance", OFFSET(l_max), AV_OPT_TYPE_FLOAT, {.dbl=300.0}, 0, 10000, FLAGS }, + { "l_min", "Minimum luminance", OFFSET(l_min), AV_OPT_TYPE_FLOAT, {.dbl=0.1}, 0, 1, FLAGS }, + { "gamma", "Gamma value for BT.1886", OFFSET(gamma), AV_OPT_TYPE_FLOAT, {.dbl=2.4}, 1, 3, FLAGS }, + { "pu21_mode", "PU21 mode (0: BANDING, 1: BANDING_GLARE, 2: PEAKS, 3: PEAKS_GLARE)", OFFSET(pu21_mode), AV_OPT_TYPE_INT, {.i64=0}, 0, 3, FLAGS }, { NULL } }; -- 2.43.0