From patchwork Sat Sep 11 08:12:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 30130 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp2234691iov; Sat, 11 Sep 2021 01:13:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwFpTscY+gQDYhTvhh5q617w7XY2DJzScI2gHdKmtXDLCp5BVp30Bo7qe0N1xXeKH5YnR0L X-Received: by 2002:a17:906:b46:: with SMTP id v6mr1801712ejg.262.1631347998962; Sat, 11 Sep 2021 01:13:18 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a14si1180728edf.400.2021.09.11.01.13.08; Sat, 11 Sep 2021 01:13:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b=U88JSljZ; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9AEFF68A5AF; Sat, 11 Sep 2021 11:13:06 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02olkn2087.outbound.protection.outlook.com [40.92.44.87]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C967C68A2DE for ; Sat, 11 Sep 2021 11:12:59 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aOO/VMrsyEQ8zQjGd/rcOpdPY7TkfyE20IhpbsZwhL0W5omMeCl/tGDmF87HH/jRePjkPvro1PtUguEs0TjWYfUM+k2YLun5DdTROBOBkjspoRiP6D3f4WEmXT0xZpCrHpZTmIx2i8HxfrXDstY1hV6fniPyX4QnarObPl+K4hQSu3yd63LTw7Jr+3FHr9Wb7SxCvBw2EKtDAt/u+GMojL8jAGEINmX0RLLdUJmjk0ij3SU3jNLlOAqC3ohdK8+/QR1CdO11JdDDDJjjLXwUmfuIfn+IGgwWZ7rVGyMUN5Qidu4n4QJBWb6OuhzaO32Dnx+gsYaHdJqTCKGYyuXIuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=KbBFX0HK1wIghhAZCiJ+nEuqusHQaio8COqqz800vbA=; b=Tv0Ui0JHjduYqD8w25aCjbHcgAkrj8+4D1Cjkzk6PcA1U/2mj9FDMkmBJg4wob9D9H8Joi2hZDISxmILl74ejqiVoljafhfGfWHstLnfCeCrVLvVqD6lGDad7GWQBs9Agpk6VyqMyKho15bZPWabI0A5nwoNbPsRVvCSsnr3boKDjPBfby4sjHb2b6vzTlgIF2T1BqdaA2PGVJmbGcXf0UKSXCK/DQPqWpCjz8k1cojs5pnRCek+7vEUeVcgzWWUmUxW3vrAm8tcEtbrfkXWxl3hhd28KA/hygvzd8uoeHEq1+z7f7ywUxe7sHgiK0Q783Wn0mwdca2CiyHYCbz6Lg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KbBFX0HK1wIghhAZCiJ+nEuqusHQaio8COqqz800vbA=; b=U88JSljZUB1lvZK030iSeu7zWm4mDO09uXWXk4tAuCIie6uSKqFFLgQ7rsrwspQbgvFqK+x2DtQ2tgm2vFA1kwXePRUeTTVtkdanReeG6SBALiOgtPkhzk0eTrb55h88TTvJjSbQuc5hnXblNzou9CEEsxgSNt29+qsV72BVkR3SjxhJDh+lp46qx5h91zrKJWvveZfjL//RR1hfSC7SM1ly5psuWIln4K26CTO/JhYnBfPTPRAIdFy1DA1qlUGxbhoKFSEqeCK6DJ9VAKzvmQJ3cGinQLuwhwJcfg9IQHj6O8MrXpv93GmxPlajyDZkqu6U9LIm7LSBSBygif5lRA== Received: from MN2PR04MB5981.namprd04.prod.outlook.com (2603:10b6:208:da::10) by MN2PR04MB6048.namprd04.prod.outlook.com (2603:10b6:208:e5::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.16; Sat, 11 Sep 2021 08:12:57 +0000 Received: from MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb]) by MN2PR04MB5981.namprd04.prod.outlook.com ([fe80::ecfe:2528:2012:22cb%5]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 08:12:56 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v4 01/18] avutil/frame: Subtitle Filtering - Add AVMediaType property to AVFrame Thread-Index: AdejcVrBX0YG5+jrTPmiEDLxuGTtkQ== Date: Sat, 11 Sep 2021 08:12:56 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [qeBBmhHvePfUFwNS/8+iUZicOofow3wlwA9EfJYQ2og=] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 2598177d-4a5e-4a50-7d52-08d974fbf671 x-ms-traffictypediagnostic: MN2PR04MB6048: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 0X6lbtJukhtZKqXoPSIBdf/fqO7rja89OJRQVYwz0rraCai2y3EpwiwTeeKbo1vUVfd5tls1ozB2/GHpqHCbBZRSt04uHz0throCWdA5up0DSipFcJbGJ0Uwgiq7h2gu6QfICS3d3qEN6KP8EjKFAL5sT16WdeNUk9a+79TompxvPHbSXLDlHxHNkAetek2by6y/5x/BR6V+d4q7d+S5CZr3w1IsGSWulwu+5SHLNwd0RcbUgK2gHfxHgXMmbgFcSgdNNlrGNun2g42EV0OPWHoFNfdeTqd7NEoY1G4Y0XHUtVokp9yEk4VaciVVmzYGdbAscreCD1UFM9HAJ9my2WsWrN/Yk/XdWZLAO7k6Zzq3CLLwJHjCai7RHc0UjUhc8eCFAuPVjn+lUVME6K3WyTTiuMtIwGK6qYpedkik1YtIxUVD/wlmtPJnm62GnILO x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: CQqvE3hlziVLNbnIDgAR0ciucajApnhgwlIpBOSzqBuVpkCM5DZPbfJTSihbrd1Ll19DlVPrRPSJfelI7LsZb8BJzLQlY3dSg6gZf1aosOX8oB8Bdw/0EJfeFRyFcM+4sZlVzsT8oqd1JNapuReGDQ== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-529c7.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR04MB5981.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 2598177d-4a5e-4a50-7d52-08d974fbf671 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2021 08:12:56.8492 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR04MB6048 Subject: [FFmpeg-devel] [PATCH v4 01/18] avutil/frame: Subtitle Filtering - Add AVMediaType property to AVFrame X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1T5SpquuEA3o This is the root commit for adding subtitle filtering capabilities. Adding the media type property to AVFrame replaces the previous way of distinction which was based on checking width and height to determine whether a frame is audio or video. Signed-off-by: softworkz --- libavutil/frame.c | 78 +++++++++++++++++++++++++++++++++++---------- libavutil/frame.h | 39 +++++++++++++++++++++-- libavutil/version.h | 2 +- 3 files changed, 99 insertions(+), 20 deletions(-) diff --git a/libavutil/frame.c b/libavutil/frame.c index b0ceaf7145..09149721c0 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -244,22 +244,39 @@ static int get_audio_buffer(AVFrame *frame, int align) } int av_frame_get_buffer(AVFrame *frame, int align) +{ + if (frame->width > 0 && frame->height > 0) + return av_frame_get_buffer2(frame, AVMEDIA_TYPE_VIDEO, align); + else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0)) + return av_frame_get_buffer2(frame, AVMEDIA_TYPE_AUDIO, align); + + return AVERROR(EINVAL); +} + +int av_frame_get_buffer2(AVFrame *frame, enum AVMediaType type, int align) { if (frame->format < 0) return AVERROR(EINVAL); - if (frame->width > 0 && frame->height > 0) + frame->type = type; + + switch(frame->type) { + case AVMEDIA_TYPE_VIDEO: return get_video_buffer(frame, align); - else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0)) + case AVMEDIA_TYPE_AUDIO: return get_audio_buffer(frame, align); - - return AVERROR(EINVAL); + case AVMEDIA_TYPE_SUBTITLE: + return 0; + default: + return AVERROR(EINVAL); + } } static int frame_copy_props(AVFrame *dst, const AVFrame *src, int force_copy) { int ret, i; + dst->type = src->type; dst->key_frame = src->key_frame; dst->pict_type = src->pict_type; dst->sample_aspect_ratio = src->sample_aspect_ratio; @@ -331,6 +348,7 @@ int av_frame_ref(AVFrame *dst, const AVFrame *src) av_assert1(dst->width == 0 && dst->height == 0); av_assert1(dst->channels == 0); + dst->type = src->type; dst->format = src->format; dst->width = src->width; dst->height = src->height; @@ -344,7 +362,7 @@ int av_frame_ref(AVFrame *dst, const AVFrame *src) /* duplicate the frame data if it's not refcounted */ if (!src->buf[0]) { - ret = av_frame_get_buffer(dst, 0); + ret = av_frame_get_buffer2(dst, dst->type, 0); if (ret < 0) goto fail; @@ -499,6 +517,7 @@ int av_frame_make_writable(AVFrame *frame) return 0; memset(&tmp, 0, sizeof(tmp)); + tmp.type = frame->type; tmp.format = frame->format; tmp.width = frame->width; tmp.height = frame->height; @@ -509,7 +528,7 @@ int av_frame_make_writable(AVFrame *frame) if (frame->hw_frames_ctx) ret = av_hwframe_get_buffer(frame->hw_frames_ctx, &tmp, 0); else - ret = av_frame_get_buffer(&tmp, 0); + ret = av_frame_get_buffer2(&tmp, tmp.type, 0); if (ret < 0) return ret; @@ -544,14 +563,22 @@ AVBufferRef *av_frame_get_plane_buffer(AVFrame *frame, int plane) uint8_t *data; int planes, i; - if (frame->nb_samples) { - int channels = frame->channels; - if (!channels) - return NULL; - CHECK_CHANNELS_CONSISTENCY(frame); - planes = av_sample_fmt_is_planar(frame->format) ? channels : 1; - } else + switch(frame->type) { + case AVMEDIA_TYPE_VIDEO: planes = 4; + break; + case AVMEDIA_TYPE_AUDIO: + { + int channels = frame->channels; + if (!channels) + return NULL; + CHECK_CHANNELS_CONSISTENCY(frame); + planes = av_sample_fmt_is_planar(frame->format) ? channels : 1; + break; + } + default: + return NULL; + } if (plane < 0 || plane >= planes || !frame->extended_data[plane]) return NULL; @@ -675,17 +702,34 @@ static int frame_copy_audio(AVFrame *dst, const AVFrame *src) return 0; } +static int frame_copy_subtitles(AVFrame *dst, const AVFrame *src) +{ + dst->type = AVMEDIA_TYPE_SUBTITLE; + dst->format = src->format; + + if (src->buf[0]) { + dst->buf[0] = av_buffer_ref(src->buf[0]); + dst->data[0] = src->data[0]; + } + + return 0; +} + int av_frame_copy(AVFrame *dst, const AVFrame *src) { if (dst->format != src->format || dst->format < 0) return AVERROR(EINVAL); - if (dst->width > 0 && dst->height > 0) + switch(dst->type) { + case AVMEDIA_TYPE_VIDEO: return frame_copy_video(dst, src); - else if (dst->nb_samples > 0 && dst->channels > 0) + case AVMEDIA_TYPE_AUDIO: return frame_copy_audio(dst, src); - - return AVERROR(EINVAL); + case AVMEDIA_TYPE_SUBTITLE: + return frame_copy_subtitles(dst, src); + default: + return AVERROR(EINVAL); + } } void av_frame_remove_side_data(AVFrame *frame, enum AVFrameSideDataType type) diff --git a/libavutil/frame.h b/libavutil/frame.h index ff2540a20f..c104815df9 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -271,7 +271,7 @@ typedef struct AVRegionOfInterest { } AVRegionOfInterest; /** - * This structure describes decoded (raw) audio or video data. + * This structure describes decoded (raw) audio, video or subtitle data. * * AVFrame must be allocated using av_frame_alloc(). Note that this only * allocates the AVFrame itself, the buffers for the data must be managed @@ -302,6 +302,13 @@ typedef struct AVRegionOfInterest { */ typedef struct AVFrame { #define AV_NUM_DATA_POINTERS 8 + /** + * Media type of the frame (audio, video, subtitles..) + * + * See AVMEDIA_TYPE_xxx + */ + enum AVMediaType type; + /** * pointer to the picture/channel planes. * This might be different from the first allocated byte @@ -371,7 +378,7 @@ typedef struct AVFrame { /** * format of the frame, -1 if unknown or unset * Values correspond to enum AVPixelFormat for video frames, - * enum AVSampleFormat for audio) + * enum AVSampleFormat for audio, AVSubtitleType for subtitles) */ int format; @@ -721,6 +728,8 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); /** * Allocate new buffer(s) for audio or video data. * + * Note: For subtitle data, use av_frame_get_buffer2 + * * The following fields must be set on frame before calling this function: * - format (pixel format for video, sample format for audio) * - width and height for video @@ -743,6 +752,32 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); */ int av_frame_get_buffer(AVFrame *frame, int align); +/** + * Allocate new buffer(s) for audio, video or subtitle data. + * + * The following fields must be set on frame before calling this function: + * - format (pixel format for video, sample format for audio) + * - width and height for video + * - nb_samples and channel_layout for audio + * + * This function will fill AVFrame.data and AVFrame.buf arrays and, if + * necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf. + * For planar formats, one buffer will be allocated for each plane. + * + * @warning: if frame already has been allocated, calling this function will + * leak memory. In addition, undefined behavior can occur in certain + * cases. + * + * @param frame frame in which to store the new buffers. + * @param type media type to set for the frame. + * @param align Required buffer size alignment. If equal to 0, alignment will be + * chosen automatically for the current CPU. It is highly + * recommended to pass 0 here unless you know what you are doing. + * + * @return 0 on success, a negative AVERROR on error. + */ +int av_frame_get_buffer2(AVFrame *frame, enum AVMediaType type, int align); + /** * Check if the frame data is writable. * diff --git a/libavutil/version.h b/libavutil/version.h index f220e192c8..2b5ac540a4 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 57 -#define LIBAVUTIL_VERSION_MINOR 5 +#define LIBAVUTIL_VERSION_MINOR 6 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \