From patchwork Tue Oct 25 09:13:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 38978 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4a86:b0:9d:28a3:170e with SMTP id fn6csp2684353pzb; Tue, 25 Oct 2022 02:14:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5Hgc5OcLo34HUxemDapTUG64GJeLvlSLFkE78+Wgz0YksrwKh0PrEHPTc1NwXt+3ZEs241 X-Received: by 2002:a17:907:763b:b0:7aa:1ae6:900b with SMTP id jy27-20020a170907763b00b007aa1ae6900bmr5883634ejc.416.1666689272362; Tue, 25 Oct 2022 02:14:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666689272; cv=none; d=google.com; s=arc-20160816; b=evjufe1hrU2ZYuE/jg2onFOhfDmGPRNjlmDpta97FAiWHnqj9e685rbi0NZ94/49+4 T4mx8BSvFAG2iwDMrQeJPtT/WCCDVmpqkuzdmKzZbBHPAWf/BTJ02L0tIL+/eUlnpOp8 DvAFw3E7EVA3ScalC9gR76J46cVfW9tJzH3DGAe4uLpJvYfsF/C9SgyQqTzHTX6D75N7 2OuXghsKXp5oG6mNUZfvImpuQ5ZAdrTeK6sqyRUrwVAoKQlm3HvXwL3gGJLeH1y4Z50P 2ynC1UCCHEA7ELJG7wztmNgp+qKPHhP3F57x0dE0BBk3ZAFjP27Qoo35o7qxcOhyB099 9kzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=MOSJipIte045KVLSeTczSu9DCQU0M7IbI/CWuknAPic=; b=D3IXVxpTF+FkssWJFo40LgcEr3FZuD/dpygFPsjSu25/SgsKUwt2AUFsxcqM4Oavoy IMcGWJglQl+k19ndUYQprq/82CDkrN450rpmwAsisvoAbIpl+QG7lKSm++Uoi6asvg9P LRxutJtHT/RQPjNhYAC9NahdT9g4oF2ZLz0YJQx0LEYafKYdlLfivVKf9/kiMRVxaWud Rody5gUivNc8DLd1S5k/pfk84JvyhPwT0ce15oS47Dk0IYjWiJ4DapFVNr13J/fOll1a eDJ4LfhHF6/+0+5yPSfqgPmVdcviTxRu3rXqlH5kqmz7zTuOw/XzlrPN7xOqWWwIGK7P naVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=IJ72gqPp; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d17-20020a170906305100b007317ad1f9a4si1922764ejd.310.2022.10.25.02.14.31; Tue, 25 Oct 2022 02:14:32 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=IJ72gqPp; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2F6F968BC88; Tue, 25 Oct 2022 12:14:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7E62968B9C0 for ; Tue, 25 Oct 2022 12:13:55 +0300 (EEST) Received: by mail-pl1-f179.google.com with SMTP id u6so10628479plq.12 for ; Tue, 25 Oct 2022 02:13:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date :references:in-reply-to:message-id:from:from:to:cc:subject:date :message-id:reply-to; bh=ZSKFZ8pHOGEGxjUj/zNVXaYGpi/CsnvD1MN5+m89hbM=; b=IJ72gqPpnCbNA05H0COogsI0RrcIaXp8v9TN8Dw7VZY12AK79adqmFlIXMBXy3IguO 13A7PI/4Sax0VIZiqq+RA8qYMKx5KgPgdo3SG0INpfT2Vx3P7yAJLjZ9cikaKLTJ/AYu M1zDjkYJG4SUKlgJcczae7x5ZXshN0uTEwiRyG1x+uFDBP93GSPp+tsO1+IvjQZ5M14O C9sAXi1dsPZRz9ZJdClPvTaZc3sT1ZbbppwK3CmTUfip+cV9pLCqSwmHchftgcr4DTrp pWtne95Rxcfr8IUzQHzZfvwUvtMXq1onqVOT+rXQ7l2kxmtMh5c9sTGRe89AUiKwZgof 1MqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date :references:in-reply-to:message-id:from:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=ZSKFZ8pHOGEGxjUj/zNVXaYGpi/CsnvD1MN5+m89hbM=; b=51TJlOVNETZbEs7b6VNUvP85pX/VzWXFSXEaceXRi30xNRQwXnV9BeDAzb9dXQ+tqt qnM38WLN1XtYFOzXQK/6J0bcqx7X/dPngJuAwjO87/r83IamJFQnvEVovqM2M+cb4Oyh LdDV5WHqR2CbN8TBdMsehR7UbBxbDVy0nRFexFV9UTQKyMU334swDJyi6rPcKt7/wUe6 KE7f+UHGPZjsZyGmggrHdkvZZ/puavXIQmOYpYSnkIB8vSL4sWeEaPBjolrSFTjzcRgw DWdprjv9JkODqi06D7e2b1v+qFckhwdT7Zndkj2ElDzypxuy8BbvFde8m4eQA9ooFrBz 1Ufw== X-Gm-Message-State: ACrzQf00Q/ME7R2mkTDfa2EgdHeGjHYm8VAFtstDCS3/w1uSnBVmOm13 9MFbIcs8+gfVbGjLdw9DkqdS2RSZJCc= X-Received: by 2002:a17:902:f54f:b0:186:a987:c733 with SMTP id h15-20020a170902f54f00b00186a987c733mr9492532plf.170.1666689233307; Tue, 25 Oct 2022 02:13:53 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id u13-20020a17090ae00d00b0020a11217682sm1046055pjy.27.2022.10.25.02.13.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Oct 2022 02:13:52 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <3598ec3b9aad8e1ce49b104faa9b91fea799cbbd.1666689226.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Tue, 25 Oct 2022 09:13:24 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v9 03/25] avcodec/subtitles: Introduce new frame-based subtitle decoding API X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: softworkz Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: sjAXIKaacL6c From: softworkz - Modify avcodec_send_packet() to support subtitles via the regular frame based decoding API - Add decode_subtitle_shim() which takes subtitle frames, and serves as a compatibility shim to the legacy subtitle decoding API until all subtitle decoders are migrated to the frame-based API - Add additional methods for conversion between old and new API Signed-off-by: softworkz --- libavcodec/avcodec.c | 8 ++ libavcodec/avcodec.h | 10 ++- libavcodec/decode.c | 60 ++++++++++++-- libavcodec/internal.h | 18 +++++ libavcodec/utils.c | 184 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 271 insertions(+), 9 deletions(-) diff --git a/libavcodec/avcodec.c b/libavcodec/avcodec.c index a85d3c2309..3518dd2185 100644 --- a/libavcodec/avcodec.c +++ b/libavcodec/avcodec.c @@ -350,6 +350,14 @@ FF_ENABLE_DEPRECATION_WARNINGS goto free_and_end; } + // Set the subtitle type from the codec descriptor in case the decoder hasn't done itself + if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE && avctx->subtitle_type == AV_SUBTITLE_FMT_UNKNOWN) { + if(avctx->codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB) + avctx->subtitle_type = AV_SUBTITLE_FMT_BITMAP; + if(avctx->codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB) + avctx->subtitle_type = AV_SUBTITLE_FMT_ASS; + } + #if FF_API_AVCTX_TIMEBASE if (avctx->framerate.num > 0 && avctx->framerate.den > 0) avctx->time_base = av_inv_q(av_mul_q(avctx->framerate, (AVRational){avctx->ticks_per_frame, 1})); diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h index bf06b01e22..68b588861f 100644 --- a/libavcodec/avcodec.h +++ b/libavcodec/avcodec.h @@ -1700,7 +1700,7 @@ typedef struct AVCodecContext { /** * Header containing style information for text subtitles. - * For SUBTITLE_ASS subtitle type, it should contain the whole ASS + * For AV_SUBTITLE_FMT_ASS subtitle type, it should contain the whole ASS * [Script Info] and [V4+ Styles] section, plus the [Events] line and * the Format line following. It shouldn't include any Dialogue line. * - encoding: Set/allocated/freed by user (before avcodec_open2()) @@ -2058,6 +2058,8 @@ typedef struct AVCodecContext { * The decoder can then override during decoding as needed. */ AVChannelLayout ch_layout; + + enum AVSubtitleType subtitle_type; } AVCodecContext; /** @@ -2434,7 +2436,10 @@ int avcodec_close(AVCodecContext *avctx); * Free all allocated data in the given subtitle struct. * * @param sub AVSubtitle to free. + * + * @deprecated Use the regular frame based encode and decode APIs instead. */ +attribute_deprecated void avsubtitle_free(AVSubtitle *sub); /** @@ -2533,7 +2538,10 @@ enum AVChromaLocation avcodec_chroma_pos_to_enum(int xpos, int ypos); * must be freed with avsubtitle_free if *got_sub_ptr is set. * @param[in,out] got_sub_ptr Zero if no subtitle could be decompressed, otherwise, it is nonzero. * @param[in] avpkt The input AVPacket containing the input buffer. + * + * @deprecated Use the new decode API (avcodec_send_packet, avcodec_receive_frame) instead. */ +attribute_deprecated int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, int *got_sub_ptr, AVPacket *avpkt); diff --git a/libavcodec/decode.c b/libavcodec/decode.c index 6be2d3d6ed..6a4702807a 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -634,6 +634,39 @@ FF_ENABLE_DEPRECATION_WARNINGS return ret; } +static int decode_subtitle2_priv(AVCodecContext *avctx, AVSubtitle *sub, + int *got_sub_ptr, AVPacket *avpkt); + +static int decode_subtitle_shim(AVCodecContext *avctx, AVFrame *frame, AVPacket *avpkt) +{ + int ret, got_sub_ptr = 0; + AVSubtitle subtitle = { 0 }; + + if (frame->buf[0]) + return AVERROR(EAGAIN); + + av_frame_unref(frame); + + ret = decode_subtitle2_priv(avctx, &subtitle, &got_sub_ptr, avpkt); + + if (ret >= 0 && got_sub_ptr) { + frame->type = AVMEDIA_TYPE_SUBTITLE; + frame->format = subtitle.format; + ret = av_frame_get_buffer2(frame, 0); + + if (ret >= 0) + ret = ff_frame_put_subtitle(frame, &subtitle); + + frame->width = avctx->width; + frame->height = avctx->height; + frame->pkt_dts = avpkt->dts; + } + + avsubtitle_free(&subtitle); + + return ret; +} + int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt) { AVCodecInternal *avci = avctx->internal; @@ -648,6 +681,13 @@ int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacke if (avpkt && !avpkt->size && avpkt->data) return AVERROR(EINVAL); + if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) + // this does not exactly implement the avcodec_send_packet/avcodec_receive_frame API + // but we know that no subtitle decoder produces multiple AVSubtitles per packet through + // the legacy API, and this will be changed when migrating the subtitle decoders + // to the frame based decoding api + return decode_subtitle_shim(avctx, avci->buffer_frame, avpkt); + av_packet_unref(avci->buffer_pkt); if (avpkt && (avpkt->data || avpkt->side_data_elems)) { ret = av_packet_ref(avci->buffer_pkt, avpkt); @@ -707,7 +747,9 @@ int ff_decode_receive_frame(AVCodecContext *avctx, AVFrame *frame) if (avci->buffer_frame->buf[0]) { av_frame_move_ref(frame, avci->buffer_frame); - } else { + } else if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) + return AVERROR(EAGAIN); + else { ret = decode_receive_frame_internal(avctx, frame); if (ret < 0) return ret; @@ -860,9 +902,8 @@ static int utf8_check(const uint8_t *str) return 1; } -int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, - int *got_sub_ptr, - AVPacket *avpkt) +static int decode_subtitle2_priv(AVCodecContext *avctx, AVSubtitle *sub, + int *got_sub_ptr, AVPacket *avpkt) { int ret = 0; @@ -908,10 +949,7 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, avctx->pkt_timebase, ms); } - if (avctx->codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB) - sub->format = 0; - else if (avctx->codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB) - sub->format = 1; + sub->format = (uint16_t)avctx->subtitle_type; for (unsigned i = 0; i < sub->num_rects; i++) { if (avctx->sub_charenc_mode != FF_SUB_CHARENC_MODE_IGNORE && @@ -932,6 +970,12 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, return ret; } +int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub, + int *got_sub_ptr, AVPacket *avpkt) +{ + return decode_subtitle2_priv(avctx, sub, got_sub_ptr, avpkt); +} + enum AVPixelFormat avcodec_default_get_format(struct AVCodecContext *avctx, const enum AVPixelFormat *fmt) { diff --git a/libavcodec/internal.h b/libavcodec/internal.h index 76a6ea6bc6..055c6735de 100644 --- a/libavcodec/internal.h +++ b/libavcodec/internal.h @@ -245,4 +245,22 @@ int64_t ff_guess_coded_bitrate(AVCodecContext *avctx); int ff_int_from_list_or_default(void *ctx, const char * val_name, int val, const int * array_valid_values, int default_value); +void ff_dvdsub_parse_palette(uint32_t *palette, const char *p); + +/** + * Copies subtitle data from AVSubtitle to AVFrame. + * + * @deprecated This is a compatibility method for interoperability with + * the legacy subtitle API. + */ +int ff_frame_put_subtitle(AVFrame* frame, const AVSubtitle* sub); + +/** + * Copies subtitle data from AVFrame to AVSubtitle. + * + * @deprecated This is a compatibility method for interoperability with + * the legacy subtitle API. + */ +int ff_frame_get_subtitle(AVSubtitle* sub, AVFrame* frame); + #endif /* AVCODEC_INTERNAL_H */ diff --git a/libavcodec/utils.c b/libavcodec/utils.c index 2b63a498b9..552cdff70f 100644 --- a/libavcodec/utils.c +++ b/libavcodec/utils.c @@ -823,6 +823,190 @@ FF_ENABLE_DEPRECATION_WARNINGS return FFMAX(0, duration); } +static int subtitle_area2rect(AVSubtitleRect *dst, const AVSubtitleArea *src) +{ + dst->x = src->x; + dst->y = src->y; + dst->w = src->w; + dst->h = src->h; + dst->nb_colors = src->nb_colors; + dst->type = src->type; + dst->flags = src->flags; + + switch (dst->type) { + case AV_SUBTITLE_FMT_BITMAP: + + if (src->h > 0 && src->w > 0 && src->buf[0]) { + uint32_t *pal; + AVBufferRef *buf = src->buf[0]; + dst->data[0] = av_mallocz(buf->size); + memcpy(dst->data[0], buf->data, buf->size); + dst->linesize[0] = src->linesize[0]; + + dst->data[1] = av_mallocz(256 * 4); + pal = (uint32_t *)dst->data[1]; + + for (unsigned i = 0; i < 256; i++) { + pal[i] = src->pal[i]; + } + } + + break; + case AV_SUBTITLE_FMT_TEXT: + + if (src->text) + dst->text = av_strdup(src->text); + else + dst->text = av_strdup(""); + + if (!dst->text) + return AVERROR(ENOMEM); + + break; + case AV_SUBTITLE_FMT_ASS: + + if (src->ass) + dst->ass = av_strdup(src->ass); + else + dst->ass = av_strdup(""); + + if (!dst->ass) + return AVERROR(ENOMEM); + + break; + default: + + av_log(NULL, AV_LOG_ERROR, "Subtitle rect has invalid format: %d", dst->type); + return AVERROR(EINVAL); + } + + return 0; +} + +static int subtitle_rect2area(AVSubtitleArea *dst, const AVSubtitleRect *src) +{ + dst->x = src->x; + dst->y = src->y; + dst->w = src->w; + dst->h = src->h; + dst->nb_colors = src->nb_colors; + dst->type = src->type; + dst->flags = src->flags; + + switch (dst->type) { + case AV_SUBTITLE_FMT_BITMAP: + + if (src->h > 0 && src->w > 0 && src->data[0]) { + AVBufferRef *buf = av_buffer_allocz(src->h * src->linesize[0]); + memcpy(buf->data, src->data[0], buf->size); + + dst->buf[0] = buf; + dst->linesize[0] = src->linesize[0]; + } + + if (src->data[1]) { + uint32_t *pal = (uint32_t *)src->data[1]; + + for (unsigned i = 0; i < 256; i++) { + dst->pal[i] = pal[i]; + } + } + + break; + case AV_SUBTITLE_FMT_TEXT: + + if (src->text) { + dst->text = av_strdup(src->text); + if (!dst->text) + return AVERROR(ENOMEM); + } + + break; + case AV_SUBTITLE_FMT_ASS: + + if (src->ass) { + dst->ass = av_strdup(src->ass); + if (!dst->ass) + return AVERROR(ENOMEM); + } + + break; + default: + + av_log(NULL, AV_LOG_ERROR, "Subtitle area has invalid format: %d", dst->type); + return AVERROR(EINVAL); + } + + return 0; +} + +/** + * Copies subtitle data from AVSubtitle (deprecated) to AVFrame + * + * @note This is a compatibility method for conversion to the legacy API + */ +int ff_frame_put_subtitle(AVFrame *frame, const AVSubtitle *sub) +{ + frame->format = sub->format; + frame->subtitle_timing.start_pts = sub->pts; + frame->subtitle_timing.start_pts += av_rescale_q(sub->start_display_time, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); + frame->subtitle_timing.duration = av_rescale_q(sub->end_display_time - sub->start_display_time, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q); + + if (sub->num_rects) { + frame->subtitle_areas = av_malloc_array(sub->num_rects, sizeof(AVSubtitleArea*)); + if (!frame->subtitle_areas) + return AVERROR(ENOMEM); + + for (unsigned i = 0; i < sub->num_rects; i++) { + int ret; + frame->subtitle_areas[i] = av_mallocz(sizeof(AVSubtitleArea)); + if (!frame->subtitle_areas[i]) + return AVERROR(ENOMEM); + ret = subtitle_rect2area(frame->subtitle_areas[i], sub->rects[i]); + if (ret < 0) { + frame->num_subtitle_areas = i; + return ret; + } + } + } + + frame->num_subtitle_areas = sub->num_rects; + return 0; +} + +/** + * Copies subtitle data from AVFrame to AVSubtitle (deprecated) + * + * @note This is a compatibility method for conversion to the legacy API + */ +int ff_frame_get_subtitle(AVSubtitle *sub, AVFrame *frame) +{ + const int64_t duration_ms = av_rescale_q(frame->subtitle_timing.duration, AV_TIME_BASE_Q, (AVRational){ 1, 1000 }); + + sub->start_display_time = 0; + sub->end_display_time = (int32_t)duration_ms; + sub->pts = frame->subtitle_timing.start_pts; + + if (frame->num_subtitle_areas) { + sub->rects = av_malloc_array(frame->num_subtitle_areas, sizeof(AVSubtitleRect*)); + if (!sub->rects) + return AVERROR(ENOMEM); + + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + int ret; + sub->rects[i] = av_mallocz(sizeof(AVSubtitleRect)); + ret = subtitle_area2rect(sub->rects[i], frame->subtitle_areas[i]); + if (ret < 0) { + sub->num_rects = i; + return ret; + } + } + } + + sub->num_rects = frame->num_subtitle_areas; + return 0; +} + int av_get_audio_frame_duration2(AVCodecParameters *par, int frame_bytes) { int channels = par->ch_layout.nb_channels;