From patchwork Fri Mar 17 20:09:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devin Heitmueller X-Patchwork-Id: 40714 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:d046:b0:cd:afd7:272c with SMTP id hv6csp14826pzb; Fri, 17 Mar 2023 13:10:23 -0700 (PDT) X-Google-Smtp-Source: AK7set+45rgxlCM0hrp25XTNaEhwPmRnK+thd5kXuO/FgynqIHcO0+s5kmUu2X/bxITTsRcD/35X X-Received: by 2002:a17:906:b349:b0:878:6b39:6d2a with SMTP id cd9-20020a170906b34900b008786b396d2amr704291ejb.46.1679083822984; Fri, 17 Mar 2023 13:10:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679083822; cv=none; d=google.com; s=arc-20160816; b=pIZhcJp17vtaQ6+S+9igSV19PaI7Mwvy72FOBP9d+N8dr/CgRwrh/ZEE4bBED5OcLT OkfZcJjAy8En0i8+ViB85g5uy40HqpDPaKtNbKF52CHjNzbUCGJbTb/H1etjoGvkgUF/ WmpBPRW1Bs7e7GCTAzxxFp2uHD+8ly+xWx2w8Xop/W6+97I5NiRzsGYxLcaC2oWd4wbh pwjoK8Vx/kdMQEPhm1r0WR5+Bu7xbbS75CcCo10lmZ0Sa0ybrN8YINcKCCzzsbmcCdO5 lyRI0pzyAGDIXZmH3c6BohPZ9S4dLaM959jY0GqVUV9Fl4ey2WB+cKClY56OnG8JoRBm zsUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=dNpMxOsiXB37ff7q//nIpDU70bqWFQVgSreNwokrBfk=; b=FnnDuEnAKRkq7KjHOsqQZPTJGhtSv3LcXu5tJNDWXQQZ69gx3CyKTvMze59zb5snIN ndu7CplPr81o4tWEA0x44wP+WdLuR8ca6Fi9srG7Ea55gzEvIfVXNKDOSle/7tnfztip yXhhPE3tsJB9DhaNqhQC/2StgcrSGOdkUbatCoTzEv0oSxA+00issbyw7WRAb29Vi7oP /ULO9smT9enwXa9oyt+K2ZyPdsH+WhYURrwNBtqqhFphVSwzYQFx2oi48oWxbQvum6fp 4we4/QnllUJvM7Gi8qr2uRdCA1ddI9QzdBWTI6afK5J7z2q4hFsdOO5L8oK9ggbzYGMH /O5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20210112.gappssmtp.com header.s=20210112 header.b=KvQEQkf5; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e10-20020a170906844a00b0093065cf57absi2454274ejy.367.2023.03.17.13.10.21; Fri, 17 Mar 2023 13:10:22 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20210112.gappssmtp.com header.s=20210112 header.b=KvQEQkf5; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 43BC168C25D; Fri, 17 Mar 2023 22:09:56 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 714FA68C154 for ; Fri, 17 Mar 2023 22:09:48 +0200 (EET) Received: by mail-qt1-f170.google.com with SMTP id t9so6939831qtx.8 for ; Fri, 17 Mar 2023 13:09:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ltnglobal-com.20210112.gappssmtp.com; s=20210112; t=1679083787; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Bk+ORmg8byGfBpHsnXhu945d54blgWlky1zb/u6hCvU=; b=KvQEQkf5bhEcq/iaP4rGRMCRzOMnd7u6NY1Tq8d4SdVXVKT5Nz9Rp2SdAWy/1HgZ8z GRCgjIr5UJq9n3iHcyQyqfZgBvAPfIPDmQAJS7JzcAPIYJliWvmCm41n+lL5UD04aKDn kDhVJ5jBHqPdcnccZ6mxIpxYMtXHSIMqt3XuDL7mdU1h2Bmx7uhadCshfTWOOGpMYSwY /elB8KzjDRqTXdCUkvTd79GUx5yZ1vCER33f3o/LQHtTHzSUPMFG1ZmVEyjFIqzLhqTu eNeCUBoVdDUEqHc0+JmgqSgqMAOS2Pcglv26z1QV3hBgIbyJT6PjkAIRMwhcIJslkfXg 7BKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679083787; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Bk+ORmg8byGfBpHsnXhu945d54blgWlky1zb/u6hCvU=; b=LvAzVWAGXJ/y3NuU2R85v3/tHxo5wSBDWh+wKfjAyWCRChq2GOA3Qw0ZX2H7LEh8xK qazzlQEXQ6ZoLMLuuP2uiUmtb7SPkZ6sOXhOElUSZEkVaCzbqj+F0ITLDS6fjA+BYBM6 4LrhJ4KQOUIU/WBwZJFr4YDZe5EN1xo/hjFIw2yQ1OidiBaGmqsWNW9NPEyUWLZe7H3A h5s+zE6d7nUaQhRx2bXGkolYW1YqsWc70V9X2NTa1H8Nyv3PczQkNPXiUIRguh16+K1L nfL1jYyEA94z8jpxG+996ewCJZHp847QcZ/4uyEbDL6SYt+QTo5ne1+VZVNGbR0J++sp 1ypg== X-Gm-Message-State: AO0yUKVKs6x5zDHd002uxC9asuruYekOiNe++2hmMIe+ZHbcFumki2Jl nOCXebQCPXIUdXyipcgZhUbkJN+HCD4mFI6Ud4s= X-Received: by 2002:ac8:5c96:0:b0:3b9:b5c5:ebb1 with SMTP id r22-20020ac85c96000000b003b9b5c5ebb1mr14502629qta.9.1679083786991; Fri, 17 Mar 2023 13:09:46 -0700 (PDT) Received: from klab-nyc-kernellabs2.localdomain (pool-71-105-132-214.nycmny.fios.verizon.net. [71.105.132.214]) by smtp.gmail.com with ESMTPSA id j5-20020ac86645000000b003bd0e7ff466sm2065685qtp.7.2023.03.17.13.09.46 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Mar 2023 13:09:46 -0700 (PDT) From: Devin Heitmueller X-Google-Original-From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Fri, 17 Mar 2023 16:09:37 -0400 Message-Id: <20230317200941.3936-2-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 2.35.1.655.ga68dfadae5 In-Reply-To: <20230317200941.3936-1-dheitmueller@ltnglobal.com> References: <20230317200941.3936-1-dheitmueller@ltnglobal.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [RFC PATCH 1/5] ccfifo: Properly handle CEA-708 captions through framerate conversion X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Devin Heitmueller Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: NzH2oAExMXKE When transcoding video that contains 708 closed captions, the caption data is tied to the frames as side data. Simply dropping or adding frames to change the framerate will result in loss of data, so the caption data needs to be preserved and reformatted. For example, without this patch converting 720p59 to 1080i59 would result in loss of 50% of the caption bytes, resulting in garbled 608 captions and 708 probably wouldn't render at all. Further, the frames that are there will have an illegal cc_count for the target framerate, so some decoders may ignore the packets entirely. Extract the 608 and 708 tuples and insert them onto queues. Then after dropping/adding frames, re-write the tuples back into the resulting frames at the appropriate rate given the target framerate. This includes both having the correct cc_count as well as clocking out the 608 pairs at the appropriate rate. Signed-off-by: Devin Heitmueller --- libavutil/Makefile | 2 + libavutil/ccfifo.c | 192 +++++++++++++++++++++++++++++++++++++++++++++ libavutil/ccfifo.h | 85 ++++++++++++++++++++ 3 files changed, 279 insertions(+) create mode 100644 libavutil/ccfifo.c create mode 100644 libavutil/ccfifo.h diff --git a/libavutil/Makefile b/libavutil/Makefile index dc9012f9a8..b6115859fa 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -17,6 +17,7 @@ HEADERS = adler32.h \ buffer.h \ cast5.h \ camellia.h \ + ccfifo.h \ channel_layout.h \ common.h \ cpu.h \ @@ -113,6 +114,7 @@ OBJS = adler32.o \ buffer.o \ cast5.o \ camellia.o \ + ccfifo.o \ channel_layout.o \ cpu.o \ crc.o \ diff --git a/libavutil/ccfifo.c b/libavutil/ccfifo.c new file mode 100644 index 0000000000..98ee365eee --- /dev/null +++ b/libavutil/ccfifo.c @@ -0,0 +1,192 @@ +/* + * CEA-708 Closed Captioning FIFO + * Copyright (c) 2023 LTN Global Communications + * + * Author: Devin Heitmueller + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "ccfifo.h" + +struct AVCCFifo { + AVFifo *cc_608_fifo; + AVFifo *cc_708_fifo; + int expected_cc_count; + int expected_608; + int cc_detected; + void *log_ctx; +}; + +#define MAX_CC_ELEMENTS 128 +#define CC_BYTES_PER_ENTRY 3 + +struct cc_lookup { + int num; + int den; + int cc_count; + int num_608; +}; + +const static struct cc_lookup cc_lookup_vals[] = { + { 15, 1, 40, 4 }, + { 24, 1, 25, 3 }, + { 24000, 1001, 25, 3 }, + { 30, 1, 20, 2 }, + { 30000, 1001, 20, 2}, + { 60, 1, 10, 1 }, + { 60000, 1001, 10, 1}, +}; + +void av_ccfifo_freep(AVCCFifo **ccf) +{ + if (ccf && *ccf) { + AVCCFifo *tmp = *ccf; + if (tmp->cc_608_fifo) + av_fifo_freep2(&tmp->cc_608_fifo); + if (tmp->cc_708_fifo) + av_fifo_freep2(&tmp->cc_708_fifo); + av_freep(*ccf); + } +} + +AVCCFifo *av_ccfifo_alloc(AVRational *framerate, void *log_ctx) +{ + AVCCFifo *ccf; + int i; + + ccf = av_mallocz(sizeof(*ccf)); + if (!ccf) + return NULL; + + if (!(ccf->cc_708_fifo = av_fifo_alloc2(MAX_CC_ELEMENTS, CC_BYTES_PER_ENTRY, 0))) + goto error; + + if (!(ccf->cc_608_fifo = av_fifo_alloc2(MAX_CC_ELEMENTS, CC_BYTES_PER_ENTRY, 0))) + goto error; + + printf("cc_fifo framerate=%d/%d\n", framerate->num, framerate->den); + /* Based on the target FPS, figure out the expected cc_count and number of + 608 tuples per packet. See ANSI/CTA-708-E Sec 4.3.6.1. */ + for (i = 0; i < (sizeof(cc_lookup_vals) / sizeof(struct cc_lookup)); i++) { + if (framerate->num == cc_lookup_vals[i].num && + framerate->den == cc_lookup_vals[i].den) { + ccf->expected_cc_count = cc_lookup_vals[i].cc_count; + ccf->expected_608 = cc_lookup_vals[i].num_608; + break; + } + } + + if (ccf->expected_608 == 0) { + av_log(ccf->log_ctx, AV_LOG_WARNING, "cc_fifo cannot transcode captions fps=%d/%d\n", + framerate->num, framerate->den); + return NULL; + } + + return ccf; + +error: + av_ccfifo_freep(&ccf); + return NULL; +} + +int av_ccfifo_inject(AVCCFifo *ccf, AVFrame *frame) +{ + AVFrameSideData *sd; + int cc_filled = 0; + int i; + + if (!ccf) + return 0; + + if (ccf->cc_detected == 0 || ccf->expected_cc_count == 0) + return 0; + + sd = av_frame_new_side_data(frame, AV_FRAME_DATA_A53_CC, + ccf->expected_cc_count * CC_BYTES_PER_ENTRY); + if (!sd) + return 0; + + for (i = 0; i < ccf->expected_608; i++) { + if (av_fifo_can_read(ccf->cc_608_fifo) >= CC_BYTES_PER_ENTRY) { + av_fifo_read(ccf->cc_608_fifo, &sd->data[cc_filled * CC_BYTES_PER_ENTRY], + CC_BYTES_PER_ENTRY); + cc_filled++; + } else { + break; + } + } + + /* Insert any available data from the 708 FIFO */ + while (cc_filled < ccf->expected_cc_count) { + if (av_fifo_can_read(ccf->cc_708_fifo) >= CC_BYTES_PER_ENTRY) { + av_fifo_read(ccf->cc_708_fifo, &sd->data[cc_filled * CC_BYTES_PER_ENTRY], + CC_BYTES_PER_ENTRY); + cc_filled++; + } else { + break; + } + } + + /* Insert 708 padding into any remaining fields */ + while (cc_filled < ccf->expected_cc_count) { + sd->data[cc_filled * CC_BYTES_PER_ENTRY] = 0xfa; + sd->data[cc_filled * CC_BYTES_PER_ENTRY + 1] = 0x00; + sd->data[cc_filled * CC_BYTES_PER_ENTRY + 2] = 0x00; + cc_filled++; + } + printf("enqueue cc_filled=%d\n", cc_filled); + return 0; +} + +int av_ccfifo_extract(AVCCFifo *ccf, AVFrame *frame) +{ + int i; + + if (!ccf) + return 0; + + /* Read the A53 side data, discard padding, and put 608/708 into + queues so we can ensure they get into the output frames at + the correct rate... */ + if (ccf->expected_cc_count > 0) { + AVFrameSideData *side_data = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC); + if (side_data) { + uint8_t *cc_bytes = side_data->data; + int cc_count = side_data->size / CC_BYTES_PER_ENTRY; + ccf->cc_detected = 1; + + for (i = 0; i < cc_count; i++) { + /* See ANSI/CTA-708-E Sec 4.3, Table 3 */ + uint8_t cc_valid = (cc_bytes[CC_BYTES_PER_ENTRY*i] & 0x04) >> 2; + uint8_t cc_type = cc_bytes[CC_BYTES_PER_ENTRY*i] & 0x03; + if (cc_type == 0x00 || cc_type == 0x01) { + av_fifo_write(ccf->cc_608_fifo, &cc_bytes[CC_BYTES_PER_ENTRY*i], + CC_BYTES_PER_ENTRY); + } else if (cc_valid && (cc_type == 0x02 || cc_type == 0x03)) { + av_fifo_write(ccf->cc_708_fifo, &cc_bytes[CC_BYTES_PER_ENTRY*i], + CC_BYTES_PER_ENTRY); + } + } + printf("dequeue cc_filled=%d\n", cc_count); + /* Remove the side data, as we will re-create it on the + output as needed */ + av_frame_remove_side_data(frame, AV_FRAME_DATA_A53_CC); + } + } + return 0; +} diff --git a/libavutil/ccfifo.h b/libavutil/ccfifo.h new file mode 100644 index 0000000000..ad48c0a39c --- /dev/null +++ b/libavutil/ccfifo.h @@ -0,0 +1,85 @@ +/* + * CEA-708 Closed Captioning FIFO + * Copyright (c) 2023 LTN Global Communications + * + * Author: Devin Heitmueller + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * CC FIFO Buffer + */ + +#ifndef AVUTIL_CCFIFO_H +#define AVUTIL_CCFIFO_H + +#include "avutil.h" +#include "frame.h" +#include "fifo.h" + +typedef struct AVCCFifo AVCCFifo; + +/** + * Allocate an AVCCFifo. + * + * @param sample_fmt sample format + * @param channels number of channels + * @param nb_samples initial allocation size, in samples + * @return newly allocated AVCCFifo, or NULL on error + */ +AVCCFifo *av_ccfifo_alloc(AVRational *framerate, void *log_ctx); + +/** + * Free an AVCCFifo + * + * @param ccf Pointer to the pointer to the AVCCFifo which should be freed + * @note `*ptr = NULL` is safe and leads to no action. + */ +void av_ccfifo_freep(AVCCFifo **ccf); + + +/** + * Read a frame into a CC Fifo + * + * Extract CC bytes from the AVFrame, insert them into our queue, and + * remove the side data from the AVFrame. The side data is removed + * as it will be re-inserted at the appropriate rate later in the + * filter. + * + * @param af AVCCFifo to write to + * @param frame AVFrame with the video frame to operate on + * @return Zero on success, or negative AVERROR + * code on failure. + */ +int av_ccfifo_extract(AVCCFifo *af, AVFrame *frame); + +/** + * Insert CC data from the FIFO into an AVFrame (as side data) + * + * Dequeue the appropriate number of CC tuples based on the + * frame rate, and insert them into the AVFrame + * + * @param af AVCCFifo to read from + * @param frame AVFrame with the video frame to operate on + * @return Zero on success, or negative AVERROR + * code on failure. + */ +int av_ccfifo_inject(AVCCFifo *af, AVFrame *frame); + +#endif /* AVUTIL_CCFIFO_H */