From patchwork Fri Apr 28 16:37:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devin Heitmueller X-Patchwork-Id: 41401 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp1439164pzb; Fri, 28 Apr 2023 08:42:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5ZKoa8KRZLEPETCHxg9xY0wpBZoejUxFJeWqd4QrPYWpxMHTPvFkhJdfV6xSls15OlhheN X-Received: by 2002:a50:fe8f:0:b0:506:c1a6:2771 with SMTP id d15-20020a50fe8f000000b00506c1a62771mr4514275edt.39.1682696555850; Fri, 28 Apr 2023 08:42:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682696555; cv=none; d=google.com; s=arc-20160816; b=tSKA9Qatf3h/nkaPvMezTOuEuu7x0TE21JlNWBsdUxttLTfYivPYlaWPcTYYD/UB5y JXi+rJemMyBC/mKbM9b9JNP7eJkpVpyvrVyLNHDdtwAFeebCYeWIc4NIu+W5S9YOGl6X 5xPqRaXyMX8cOLeAuG9kcDdQ/HODkgrQzKjBfUWEwLmU5Ps3gvVCs21baaOd35iLQ8kf YYaU6ykrMrX8QNrL6RCoiRGnRt/3mGDO/y6QZ4r2y8NFT9x4qiSF3q3v4paOSZk8O53E PlwWwFH3fiFu0FiyLAOscQCQIiAq3X4nHtgUUjDIwZRUVcm39LDGSimBVnzUgkcT9wvX qsSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=niAqHOsP86ZlU19izeqJBgOYMinZq705WMG80Cd89TE=; b=EBsdhzxE0BW7Ym/WsgJDQogBtFEBLpvyKAd84q+Bnv1oZKu7r/QlC0t5JvSxhMzcC1 ab1CgA9OBm4YElHmXGkYPDLl44ockY9mcvfDspM5zz1QTkH5QLNets8vZC5uZtpLwmZD IdSD4qFI+kxeeJ87vymu++o6i/yETjoOOQjiMukXUvvdpYtjdQ8l+vi/FjhWxjyLbUGz UguJtwCaiOekXCdn2yd3xxcQllNv/Y7Eyltn79Z9BPUhzP4nOnmgmV2yLV/j+yzG4eLs AiHuHGl3lmvsfgyTAZ+xeO94kbZjbgAU6KNsei7kITtaaEnClGgphVwNfDYcFSOgn1iL wlZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=D3mIbSVn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p18-20020a05640210d200b00506a7d9c0fdsi16795497edu.133.2023.04.28.08.42.35; Fri, 28 Apr 2023 08:42:35 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=D3mIbSVn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5747968C007; Fri, 28 Apr 2023 18:42:19 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ua1-f45.google.com (mail-ua1-f45.google.com [209.85.222.45]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CBCE968BFE0 for ; Fri, 28 Apr 2023 18:42:12 +0300 (EEST) Received: by mail-ua1-f45.google.com with SMTP id a1e0cc1a2514c-77297c876aaso3163651241.0 for ; Fri, 28 Apr 2023 08:42:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ltnglobal-com.20221208.gappssmtp.com; s=20221208; t=1682696531; x=1685288531; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=rShEXe66M0nR2zqmVJ0/mtYAr8RUfnYX8Urz51c0c5U=; b=D3mIbSVn/07ad+u8TzwpzbJaxBGcgGO8Rlao+nY3z3ilkFOEH9AMXw9HpPIkoMTrx2 7AnL5ZINOIuFjZ/k++3/MO1EOnAUZxG15/xwlQDDYX47bH0t/CIFr2eaoOMEQp6/jsTk o7BTuxfnIx759F3kEmkPDjDfQ2sFEFKblJ+q7z3lTqe/fh4OFn3rlZk4a86q3ekPe5RQ Tdio3Ava6nlsHROTCC2IewSBGwXQ1myC3HADE6Qo1xbDbpCZEnraxWZhHAOQ1zuuycPC w5ssX63Idk8eMJ8PB+77Ww2LypXAz26IbcCp51LsVLanhBjrsWKzN/QlFE82sS77iBFb EX0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682696531; x=1685288531; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rShEXe66M0nR2zqmVJ0/mtYAr8RUfnYX8Urz51c0c5U=; b=eDFY5hEKpTXeZxqHlcZeQO7ooOZd/822LQiAYg8TEPISRs/Fsl/vxnQcAhiNKUaoPM aXFYXqodh7BNMI307Idz/lYsq2QV9A9qfsOnNtwS+xPA6gSnl0nXzRplqET5mdfJZ2Wx xYMBrOkt5CKZvc/KQcnJ9qrot5oxxpVEwrc4fekDqhflbUJdbXgfR72ppnsiy5jtEEST 2ukjpesCvRTAjukBkbazGzXwJfaBymjq04XIC5O19y5HbCrAK2v7YwCZzjloRc6xrWaR lWwhRN58IX/9pEJfmBBzhF6Ye5vexb5w4vnC+EDC00/guN3MtgKcNOp8NAMmmozMbOF2 7Mpg== X-Gm-Message-State: AC+VfDw2KRTPtJRpzfe6K1l+5Gpx+MR++W/NRzpFCSdYvBQ5IllVqqSa oDYH7ymcgZM4ScSvmJxrtfh3Ou+egSANZulkL20= X-Received: by 2002:a1f:c1d3:0:b0:43f:e858:e948 with SMTP id r202-20020a1fc1d3000000b0043fe858e948mr2300273vkf.6.1682696530821; Fri, 28 Apr 2023 08:42:10 -0700 (PDT) Received: from ltnt-nyc-580testdevin.livetimenet.com (pool-71-105-132-214.nycmny.fios.verizon.net. [71.105.132.214]) by smtp.gmail.com with ESMTPSA id do54-20020a05620a2b3600b0074e13ed6ee9sm6733873qkb.132.2023.04.28.08.42.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Apr 2023 08:42:10 -0700 (PDT) From: Devin Heitmueller X-Google-Original-From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 Apr 2023 12:37:46 -0400 Message-Id: <1682699871-22331-2-git-send-email-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682699871-22331-1-git-send-email-dheitmueller@ltnglobal.com> References: <1682699871-22331-1-git-send-email-dheitmueller@ltnglobal.com> Subject: [FFmpeg-devel] [PATCH v4 1/6] ccfifo: Properly handle CEA-708 captions through framerate conversion X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Devin Heitmueller MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8TJ1VfAWdymq When transcoding video that contains 708 closed captions, the caption data is tied to the frames as side data. Simply dropping or adding frames to change the framerate will result in loss of data, so the caption data needs to be preserved and reformatted. For example, without this patch converting 720p59 to 1080i59 would result in loss of 50% of the caption bytes, resulting in garbled 608 captions and 708 probably wouldn't render at all. Further, the frames that are there will have an illegal cc_count for the target framerate, so some decoders may ignore the packets entirely. Extract the 608 and 708 tuples and insert them onto queues. Then after dropping/adding frames, re-write the tuples back into the resulting frames at the appropriate rate given the target framerate. This includes both having the correct cc_count as well as clocking out the 608 pairs at the appropriate rate. Thanks for Lance Wang for providing review/feedback. Signed-off-by: Devin Heitmueller --- libavfilter/Makefile | 1 + libavfilter/ccfifo.c | 240 +++++++++++++++++++++++++++++++++++++++++++++++++++ libavfilter/ccfifo.h | 94 ++++++++++++++++++++ 3 files changed, 335 insertions(+) create mode 100644 libavfilter/ccfifo.c create mode 100644 libavfilter/ccfifo.h diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 71e198b..628ade8 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -14,6 +14,7 @@ OBJS = allfilters.o \ buffersink.o \ buffersrc.o \ colorspace.o \ + ccfifo.o \ drawutils.o \ fifo.o \ formats.o \ diff --git a/libavfilter/ccfifo.c b/libavfilter/ccfifo.c new file mode 100644 index 0000000..05a77dd --- /dev/null +++ b/libavfilter/ccfifo.c @@ -0,0 +1,240 @@ +/* + * CEA-708 Closed Captioning FIFO + * Copyright (c) 2023 LTN Global Communications + * + * Author: Devin Heitmueller + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "ccfifo.h" + +struct AVCCFifo { + AVFifo *cc_608_fifo; + AVFifo *cc_708_fifo; + int expected_cc_count; + int expected_608; + int cc_detected; + int passthrough; + void *log_ctx; +}; + +#define MAX_CC_ELEMENTS 128 +#define CC_BYTES_PER_ENTRY 3 + +struct cc_lookup { + int num; + int den; + int cc_count; + int num_608; +}; + +const static struct cc_lookup cc_lookup_vals[] = { + { 15, 1, 40, 4 }, + { 24, 1, 25, 3 }, + { 24000, 1001, 25, 3 }, + { 30, 1, 20, 2 }, + { 30000, 1001, 20, 2}, + { 60, 1, 10, 1 }, + { 60000, 1001, 10, 1}, +}; + +void ff_ccfifo_freep(AVCCFifo **ccf) +{ + if (ccf && *ccf) { + AVCCFifo *tmp = *ccf; + if (tmp->cc_608_fifo) + av_fifo_freep2(&tmp->cc_608_fifo); + if (tmp->cc_708_fifo) + av_fifo_freep2(&tmp->cc_708_fifo); + av_freep(*ccf); + } +} + +AVCCFifo *ff_ccfifo_alloc(AVRational *framerate, void *log_ctx) +{ + AVCCFifo *ccf; + int i; + + ccf = av_mallocz(sizeof(*ccf)); + if (!ccf) + return NULL; + + if (!(ccf->cc_708_fifo = av_fifo_alloc2(MAX_CC_ELEMENTS, CC_BYTES_PER_ENTRY, 0))) + goto error; + + if (!(ccf->cc_608_fifo = av_fifo_alloc2(MAX_CC_ELEMENTS, CC_BYTES_PER_ENTRY, 0))) + goto error; + + /* Based on the target FPS, figure out the expected cc_count and number of + 608 tuples per packet. See ANSI/CTA-708-E Sec 4.3.6.1. */ + for (i = 0; i < FF_ARRAY_ELEMS(cc_lookup_vals); i++) { + if (framerate->num == cc_lookup_vals[i].num && + framerate->den == cc_lookup_vals[i].den) { + ccf->expected_cc_count = cc_lookup_vals[i].cc_count; + ccf->expected_608 = cc_lookup_vals[i].num_608; + break; + } + } + + if (ccf->expected_608 == 0) { + /* We didn't find an output frame we support. We'll let the call succeed + and the FIFO to be allocated, but the extract/inject functions will simply + leave everything the way it is */ + av_log(ccf->log_ctx, AV_LOG_WARNING, "cc_fifo cannot transcode captions fps=%d/%d\n", + framerate->num, framerate->den); + ccf->passthrough = 1; + } + + return ccf; + +error: + ff_ccfifo_freep(&ccf); + return NULL; +} + +int ff_ccfifo_injectbytes(AVCCFifo *ccf, uint8_t **data, size_t *len) +{ + char *cc_data; + int cc_filled = 0; + int i; + + if (!ccf) + return AVERROR(EINVAL); + + if (ccf->passthrough) { + *data = NULL; + *len = 0; + return 0; + } + + cc_data = av_mallocz(ccf->expected_cc_count * CC_BYTES_PER_ENTRY); + if (!cc_data) { + return AVERROR(ENOMEM); + } + + for (i = 0; i < ccf->expected_608; i++) { + if (av_fifo_can_read(ccf->cc_608_fifo) >= CC_BYTES_PER_ENTRY) { + av_fifo_read(ccf->cc_608_fifo, &cc_data[cc_filled * CC_BYTES_PER_ENTRY], + CC_BYTES_PER_ENTRY); + cc_filled++; + } else { + break; + } + } + + /* Insert any available data from the 708 FIFO */ + while (cc_filled < ccf->expected_cc_count) { + if (av_fifo_can_read(ccf->cc_708_fifo) >= CC_BYTES_PER_ENTRY) { + av_fifo_read(ccf->cc_708_fifo, &cc_data[cc_filled * CC_BYTES_PER_ENTRY], + CC_BYTES_PER_ENTRY); + cc_filled++; + } else { + break; + } + } + + /* Insert 708 padding into any remaining fields */ + while (cc_filled < ccf->expected_cc_count) { + cc_data[cc_filled * CC_BYTES_PER_ENTRY] = 0xfa; + cc_data[cc_filled * CC_BYTES_PER_ENTRY + 1] = 0x00; + cc_data[cc_filled * CC_BYTES_PER_ENTRY + 2] = 0x00; + cc_filled++; + } + + *data = cc_data; + *len = ccf->expected_cc_count * CC_BYTES_PER_ENTRY; + return 0; +} + +int ff_ccfifo_inject(AVCCFifo *ccf, AVFrame *frame) +{ + AVFrameSideData *sd; + uint8_t *cc_data; + size_t cc_size; + int ret; + + if (!ccf) + return AVERROR(EINVAL); + + if (ccf->passthrough == 1 || ccf->cc_detected == 0 || ccf->expected_cc_count == 0) + return 0; + + ret = ff_ccfifo_injectbytes(ccf, &cc_data, &cc_size); + if (ret == 0) { + sd = av_frame_new_side_data(frame, AV_FRAME_DATA_A53_CC, cc_size); + if (!sd) { + av_freep(&cc_data); + return AVERROR(ENOMEM); + } + memcpy(sd->data, cc_data, cc_size); + av_freep(&cc_data); + } + + return 0; +} + +int ff_ccfifo_extractbytes(AVCCFifo *ccf, uint8_t *cc_bytes, size_t len) +{ + int cc_count = len / CC_BYTES_PER_ENTRY; + + if (!ccf) + return AVERROR(EINVAL); + + if (ccf->passthrough == 1) + return 0; + + ccf->cc_detected = 1; + + for (int i = 0; i < cc_count; i++) { + /* See ANSI/CTA-708-E Sec 4.3, Table 3 */ + uint8_t cc_valid = (cc_bytes[CC_BYTES_PER_ENTRY*i] & 0x04) >> 2; + uint8_t cc_type = cc_bytes[CC_BYTES_PER_ENTRY*i] & 0x03; + if (cc_type == 0x00 || cc_type == 0x01) { + av_fifo_write(ccf->cc_608_fifo, &cc_bytes[CC_BYTES_PER_ENTRY*i], + CC_BYTES_PER_ENTRY); + } else if (cc_valid && (cc_type == 0x02 || cc_type == 0x03)) { + av_fifo_write(ccf->cc_708_fifo, &cc_bytes[CC_BYTES_PER_ENTRY*i], + CC_BYTES_PER_ENTRY); + } + } + return 0; +} + +int ff_ccfifo_extract(AVCCFifo *ccf, AVFrame *frame) +{ + if (!ccf) + return AVERROR(EINVAL); + + if (ccf->passthrough == 1) + return 0; + + /* Read the A53 side data, discard padding, and put 608/708 into + queues so we can ensure they get into the output frames at + the correct rate... */ + if (ccf->expected_cc_count > 0) { + AVFrameSideData *side_data = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC); + if (side_data) { + ff_ccfifo_extractbytes(ccf, side_data->data, side_data->size); + + /* Remove the side data, as we will re-create it on the + output as needed */ + av_frame_remove_side_data(frame, AV_FRAME_DATA_A53_CC); + } + } + return 0; +} diff --git a/libavfilter/ccfifo.h b/libavfilter/ccfifo.h new file mode 100644 index 0000000..baa4ae1 --- /dev/null +++ b/libavfilter/ccfifo.h @@ -0,0 +1,94 @@ +/* + * CEA-708 Closed Captioning FIFO + * Copyright (c) 2023 LTN Global Communications + * + * Author: Devin Heitmueller + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * CC FIFO Buffer + */ + +#ifndef AVFILTER_CCFIFO_H +#define AVFILTER_CCFIFO_H + +#include "libavutil/avutil.h" +#include "libavutil/frame.h" +#include "libavutil/fifo.h" + +typedef struct AVCCFifo AVCCFifo; + +/** + * Allocate an AVCCFifo. + * + * @param framerate output framerate + * @param log_ctx used for any av_log() calls + * @return newly allocated AVCCFifo, or NULL on error + */ +AVCCFifo *ff_ccfifo_alloc(AVRational *framerate, void *log_ctx); + +/** + * Free an AVCCFifo + * + * @param ccf Pointer to the pointer to the AVCCFifo which should be freed + * @note `*ptr = NULL` is safe and leads to no action. + */ +void ff_ccfifo_freep(AVCCFifo **ccf); + + +/** + * Extract CC data from an AVFrame + * + * Extract CC bytes from the AVFrame, insert them into our queue, and + * remove the side data from the AVFrame. The side data is removed + * as it will be re-inserted at the appropriate rate later in the + * filter. + * + * @param af AVCCFifo to write to + * @param frame AVFrame with the video frame to operate on + * @return Zero on success, or negative AVERROR + * code on failure. + */ +int ff_ccfifo_extract(AVCCFifo *ccf, AVFrame *frame); + +/** + *Just like ff_ccfifo_extract(), but takes the raw bytes instead of an AVFrame + */ +int ff_ccfifo_extractbytes(AVCCFifo *ccf, uint8_t *data, size_t len); + +/** + * Insert CC data from the FIFO into an AVFrame (as side data) + * + * Dequeue the appropriate number of CC tuples based on the + * frame rate, and insert them into the AVFrame + * + * @param af AVCCFifo to read from + * @param frame AVFrame with the video frame to operate on + * @return Zero on success, or negative AVERROR + * code on failure. + */ +int ff_ccfifo_inject(AVCCFifo *ccf, AVFrame *frame); + +/* Just like ff_ccfifo_inject(), but takes the raw bytes instead of an AVFrame + * Note: the caller is responsible for calling av_freep() against the value + * returned in the "data" argument */ +int ff_ccfifo_injectbytes(AVCCFifo *ccf, uint8_t **data, size_t *len); + +#endif /* AVFILTER_CCFIFO_H */