From patchwork Fri Dec 3 19:31:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soft Works X-Patchwork-Id: 31925 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1096799iog; Fri, 3 Dec 2021 11:31:44 -0800 (PST) X-Google-Smtp-Source: ABdhPJxP9VgQLFYCNPka6yWG3R+zFk0Gd2atZyd/uJnX1xmnr7+XUnUrksD4eeqM5syzmzjIzy5c X-Received: by 2002:aa7:d652:: with SMTP id v18mr28710844edr.68.1638559904519; Fri, 03 Dec 2021 11:31:44 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d19si4611633edy.616.2021.12.03.11.31.42; Fri, 03 Dec 2021 11:31:44 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@hotmail.com header.s=selector1 header.b="YTmKdi/b"; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=hotmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CD4BA68AE42; Fri, 3 Dec 2021 21:31:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10olkn2068.outbound.protection.outlook.com [40.92.41.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6960B68AE13 for ; Fri, 3 Dec 2021 21:31:18 +0200 (EET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n/pGGrV0WfHaEriIIMQkJm+1d9SfLiOJP5pniokQ5hCCZAY2IDLcW3jrfDNIpsC5sqIoPO5aiNfbTdexxfvP7Hwbxgy273upJF6FwPSVAB3b48wTTFmZaUAiObhh3BbxxfB7Kbpf7XTwYQNTP4toIHmYtHGkWbfd54jeCdSdP+bLd3EXKf/CTp//SSUGsUOxuNBfVBxEEWb2bRqpZv42bdu3mUjGtpOYH+rnahTQU2qPbSz1nYlIhxHCEd7AXIftqBtZrr4aGmBJDlyzC3Sx/f/16aeMHTBqJmVRXl6cRlxFjOUsra+olHA719DN4iO+KporsKEHHhcc1DIaHNotEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fq/QWmREd8ZHrXv7KypmMwQ3DQRmwUxsNmr0gXKy1G8=; b=oXS31xUlzljj2e/QQLyUwMv+KyM7HkyaXfTDgwb+rjLc4yvTFH4yb8hrqdCKAWpwHR0EblWLAMyqGE5CTPqR8mWcpjupfFX9cG27dlUPZcg+lzHuuenW9lT4bwRkC0Ss/x+F6R8x0V6pMQLVMUT7bMQDamPv15LwaqPGECLCLoE4cQfnmk+vsNc1dRclc97U8UvCfzK0xVHP216N2GipTtM1jmyIyjM4JUHaC/WpemgYVdYL4gjmJm3bvddIvegQDYlYZjIRugRDgfxXh40nVMpPt29NCk3hsnIrnYZAUjdIjTJHO3wrTVgcHvLrXBxG/CIT52KMQKhDILOlZZBdlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fq/QWmREd8ZHrXv7KypmMwQ3DQRmwUxsNmr0gXKy1G8=; b=YTmKdi/bTx1yDOT5QWrdVV4HY/oMYX3sl9inHu6x85C51rtcd5EO+DDdAca+0eS3Yn/kvBvsOwLFCeUQOqLfhsWd3QBwlwWmtl4ETpoaMNkyu5IZe7rmEkMXVSJ7s88VOgaFi83AlH4f1HU77c2rUjos0JkEyo5NxGuPZcZyDnpTzKbjoCDn7GJuF2sb5bop5wN6OGxJ76yHqbaNgdNp4GVpAos3Rhsu1KKVMcDtz3NUS0qEQEYJJfZYWaqD+pOyTVE2SVRciQIx8WE/nFP1zu7Txrlkf6YoRNMVAwR8aQBZeda9MlMso0WSmotMaNFfRrB6wJIckqzXZCUHSs/hAA== Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM (2603:10b6:8:b::20) by DM8P223MB0333.NAMP223.PROD.OUTLOOK.COM (2603:10b6:8:8::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.23; Fri, 3 Dec 2021 19:31:08 +0000 Received: from DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::9c8d:fc63:9488:9775]) by DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM ([fe80::9c8d:fc63:9488:9775%8]) with mapi id 15.20.4755.016; Fri, 3 Dec 2021 19:31:08 +0000 From: Soft Works To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH v19 02/20] avutil/frame: Prepare AVFrame for subtitle handling Thread-Index: AQHX6HxSO3vCsaloGU+7TLPbM3UqiQ== Date: Fri, 3 Dec 2021 19:31:08 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [Lx/EgQPJmDnE5gX29ovihXfhFa4sj/Oa] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d34440a2-b787-4a0a-c0f7-08d9b69374fb x-ms-traffictypediagnostic: DM8P223MB0333: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Dva+3cOkpQg7L53rWu88fN6xPIHwqJICaXYY2xFGgttoZpJuItd1NHxSehw/uqOGi+gi3V5dCb0zX9q0wx7dnIMZ5z5NLCaLau6QAi9tyBDGF8AH224B5osM6rP9r20V/MzRSnsvlzzfb3zNTROd6dDDiJgMZiHN6WNngkbEd8NT6dzagwblYUrvsgB8CVAUMqLwpwlitgpxuQKddhDbYIZz+3pYtFMiFs7MaR0LyGMNpvYmkUgVo5Nvf9IHEAdhrpAvKT8O7799oErYkbKgC16NwczPPQESxZyDXDWTTuNI/SNxS3WL1sYzYlhQQ8tt9HfmE+1V9QlRXSmnqsdGgD6X3bbCTzctw3D0LwYQrpruonIYF1anvvWaK6aAwnLe+N2pUDZbstArwVRc3tx2Ob0zw4Im1oORMXLjKAiOpeWog92iFMc1baYmHOpwqqSosDMqJ9K8+14AUEh8GidEUrH8eQvISk55HgKqhuahLRC+9NIhqRUMiodAji4dTxPATsEVfCqZmUzLUKUQANSWzk+JYJ0yi4ahK+h64NaH+JTMOTCltMvYVQmW4XwfPros2qi8cf1M3dHpTa7Z/SVIgw== x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?j6XT7zvdNvCPVKNPEFLaI59LwkJL?= =?utf-8?q?CupPol/GYO0umK6YjzSVt08ANWi9eAHLQsSosLZermkQPVLx/5gei77Mp4y8vLPAt?= =?utf-8?q?Vbr7d9Yv1h8Nd8vZ0mLZibnnMUZb9sdVHwf3sCz8sZE8jvfP5GmmXrfbad4/BXUSD?= =?utf-8?q?U9uP9jLtQ9HzvnhV03L7cV+djT2Fevx0BJmmy3RiPXpSIIom+QoBQVeYdJsnVKdB/?= =?utf-8?q?gZ7UlimGZDA83sQ6aYkRHlypLF8G9mkREfzg2jCjTGc8ce9BHJk3ZDLdbXGRKoAnn?= =?utf-8?q?iLKTFg2pnyx4thMpOyJ7k7JtsKqfAIbC6AofD6GSRWOlwiVEr6WpKT9YS0rZlKNaS?= =?utf-8?q?zGNIDJXQVWPl/MQydjjoRv2pyvdcQY18y91HSGokkK5OMCRn+8dS7UIu1S7ouBUg3?= =?utf-8?q?3ZGGXl7IopBItoVGVAu0BQd1zk05Vi3qXK597vcWxNLgIrqbBRSHeFhwG8tS9HAkR?= =?utf-8?q?mmB6nPZNlEjD4X4LuiKzxGNbZY6oW/s9SGme5rU3Cxy/8oKEYUJCQSffO9wFei/Sx?= =?utf-8?q?W8tmMamhaJDUIAyqI4BEzcOHdL6MjTQtjIX4cFt+M/VsmgiOE9TogNZU8HmngYwsu?= =?utf-8?q?qpUrqoVhtLO7cwk6233mW9RKRa518pBdOw6ezMrRi95CmKhr0YowmRRiNir+q4zJT?= =?utf-8?q?Bb1HDrr4pfXKvfTu/Kvb5iWLbbbDBugvXHUyShMuFvr0DgwLwi7stv9pqpPjuTT7K?= =?utf-8?q?As6qkDbLcaunp0f1QekltaFerslH2WchUsEZJtkOeiLXGAIPER2y7T6eRDpxHN7dJ?= =?utf-8?q?ayq5darXCzm9UE1iCkIx9kheQz2D6Krbhxp3fHbB3/BVpX2/h7LFTi6rTr9de+MnM?= =?utf-8?q?57Ihg+kB+70xhntkwbHxZsf0VbBn3IiD+v3WKUBHe25dx/pcUXzxZmKu1adqq6bKT?= =?utf-8?q?vjhGN+SyKCgx3XasgRGj4ROqozBw0Z+k5Pf6WvXgw4xQ=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-1ff67.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: d34440a2-b787-4a0a-c0f7-08d9b69374fb X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2021 19:31:08.8072 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8P223MB0333 Subject: [FFmpeg-devel] [PATCH v19 02/20] avutil/frame: Prepare AVFrame for subtitle handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: X03MQZ3N0AGx Root commit for adding subtitle filtering capabilities. In detail: - Add type (AVMediaType) field to AVFrame Replaces previous way of distinction which was based on checking width and height to determine whether a frame is audio or video - Add subtitle fields to AVFrame - Add new struct AVSubtitleArea, similar to AVSubtitleRect, but different allocation logic. Cannot and must not be used interchangeably, hence the new struct Signed-off-by: softworkz --- doc/APIchanges | 10 +++ libavutil/Makefile | 1 + libavutil/frame.c | 211 ++++++++++++++++++++++++++++++++++++++++----- libavutil/frame.h | 77 ++++++++++++++++- libavutil/subfmt.c | 50 +++++++++++ libavutil/subfmt.h | 48 +++++++++++ 6 files changed, 373 insertions(+), 24 deletions(-) create mode 100644 libavutil/subfmt.c diff --git a/doc/APIchanges b/doc/APIchanges index c0e1fde8aa..e16672f8ca 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -14,6 +14,16 @@ libavutil: 2021-04-27 API changes, most recent first: +2021-12-03 - xxxxxxxxxx - lavu 58.1.100 - frame.h + Add AVMediaType field to AVFrame + Add Fields for carrying subtitle data to AVFrame + (subtitle_areas, subtitle_header, subtitle_pts, start/end time, etc.) + Add av_frame_get_buffer2() and deprecate av_frame_get_buffer() + +2021-12-03 - xxxxxxxxxx - lavu 58.1.100 - subfmt.h + Add struct AVSubtitleArea (replaces AVSubtitle) + Add av_get_subtitle_fmt_name() and av_get_subtitle_fmt() + 2021-12-03 - xxxxxxxxxx - lavu 58.1.100 - subfmt.h Add enum AVSubtitleType (moved from lavc), add new values, deprecate existing diff --git a/libavutil/Makefile b/libavutil/Makefile index c7843db1e4..7e79936876 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -160,6 +160,7 @@ OBJS = adler32.o \ slicethread.o \ spherical.o \ stereo3d.o \ + subfmt.o \ threadmessage.o \ time.o \ timecode.o \ diff --git a/libavutil/frame.c b/libavutil/frame.c index d4d3ad6988..f71ab35ebb 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -26,6 +26,7 @@ #include "imgutils.h" #include "mem.h" #include "samplefmt.h" +#include "subfmt.h" #include "hwcontext.h" #define CHECK_CHANNELS_CONSISTENCY(frame) \ @@ -50,6 +51,9 @@ const char *av_get_colorspace_name(enum AVColorSpace val) return name[val]; } #endif + +static int frame_copy_subtitles(AVFrame *dst, const AVFrame *src, int copy_data); + static void get_frame_defaults(AVFrame *frame) { if (frame->extended_data != frame->data) @@ -72,7 +76,12 @@ static void get_frame_defaults(AVFrame *frame) frame->colorspace = AVCOL_SPC_UNSPECIFIED; frame->color_range = AVCOL_RANGE_UNSPECIFIED; frame->chroma_location = AVCHROMA_LOC_UNSPECIFIED; - frame->flags = 0; + frame->subtitle_start_time = 0; + frame->subtitle_end_time = 0; + frame->num_subtitle_areas = 0; + frame->subtitle_areas = NULL; + frame->subtitle_pts = 0; + frame->subtitle_header = NULL; } static void free_side_data(AVFrameSideData **ptr_sd) @@ -243,23 +252,55 @@ static int get_audio_buffer(AVFrame *frame, int align) } +static int get_subtitle_buffer(AVFrame *frame) +{ + // Buffers in AVFrame->buf[] are not used in case of subtitle frames. + // To accomodate with existing code, checking ->buf[0] to determine + // whether a frame is ref-counted or has data, we're adding a 1-byte + // buffer here, which marks the subtitle frame to contain data. + frame->buf[0] = av_buffer_alloc(1); + if (!frame->buf[0]) { + av_frame_unref(frame); + return AVERROR(ENOMEM); + } + + frame->extended_data = frame->data; + + return 0; +} + int av_frame_get_buffer(AVFrame *frame, int align) +{ + if (frame->width > 0 && frame->height > 0) + frame->type = AVMEDIA_TYPE_VIDEO; + else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0)) + frame->type = AVMEDIA_TYPE_AUDIO; + + return av_frame_get_buffer2(frame, align); +} + +int av_frame_get_buffer2(AVFrame *frame, int align) { if (frame->format < 0) return AVERROR(EINVAL); - if (frame->width > 0 && frame->height > 0) + switch(frame->type) { + case AVMEDIA_TYPE_VIDEO: return get_video_buffer(frame, align); - else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0)) + case AVMEDIA_TYPE_AUDIO: return get_audio_buffer(frame, align); - - return AVERROR(EINVAL); + case AVMEDIA_TYPE_SUBTITLE: + return get_subtitle_buffer(frame); + default: + return AVERROR(EINVAL); + } } static int frame_copy_props(AVFrame *dst, const AVFrame *src, int force_copy) { int ret, i; + dst->type = src->type; dst->key_frame = src->key_frame; dst->pict_type = src->pict_type; dst->sample_aspect_ratio = src->sample_aspect_ratio; @@ -290,6 +331,12 @@ static int frame_copy_props(AVFrame *dst, const AVFrame *src, int force_copy) dst->colorspace = src->colorspace; dst->color_range = src->color_range; dst->chroma_location = src->chroma_location; + dst->subtitle_start_time = src->subtitle_start_time; + dst->subtitle_end_time = src->subtitle_end_time; + dst->subtitle_pts = src->subtitle_pts; + + if ((ret = av_buffer_replace(&dst->subtitle_header, src->subtitle_header)) < 0) + return ret; av_dict_copy(&dst->metadata, src->metadata, 0); @@ -331,6 +378,7 @@ int av_frame_ref(AVFrame *dst, const AVFrame *src) av_assert1(dst->width == 0 && dst->height == 0); av_assert1(dst->channels == 0); + dst->type = src->type; dst->format = src->format; dst->width = src->width; dst->height = src->height; @@ -344,7 +392,7 @@ int av_frame_ref(AVFrame *dst, const AVFrame *src) /* duplicate the frame data if it's not refcounted */ if (!src->buf[0]) { - ret = av_frame_get_buffer(dst, 0); + ret = av_frame_get_buffer2(dst, 0); if (ret < 0) goto fail; @@ -366,6 +414,10 @@ int av_frame_ref(AVFrame *dst, const AVFrame *src) } } + /* copy subtitle areas */ + if (src->type == AVMEDIA_TYPE_SUBTITLE) + frame_copy_subtitles(dst, src, 0); + if (src->extended_buf) { dst->extended_buf = av_calloc(src->nb_extended_buf, sizeof(*dst->extended_buf)); @@ -436,7 +488,7 @@ AVFrame *av_frame_clone(const AVFrame *src) void av_frame_unref(AVFrame *frame) { - int i; + unsigned i, n; if (!frame) return; @@ -455,6 +507,21 @@ void av_frame_unref(AVFrame *frame) av_buffer_unref(&frame->opaque_ref); av_buffer_unref(&frame->private_ref); + av_buffer_unref(&frame->subtitle_header); + + for (i = 0; i < frame->num_subtitle_areas; i++) { + AVSubtitleArea *area = frame->subtitle_areas[i]; + + for (n = 0; n < FF_ARRAY_ELEMS(area->buf); n++) + av_buffer_unref(&area->buf[n]); + + av_freep(&area->text); + av_freep(&area->ass); + av_free(area); + } + + av_freep(&frame->subtitle_areas); + get_frame_defaults(frame); } @@ -472,18 +539,28 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src) int av_frame_is_writable(AVFrame *frame) { - int i, ret = 1; + AVSubtitleArea *area; + int ret = 1; /* assume non-refcounted frames are not writable */ if (!frame->buf[0]) return 0; - for (i = 0; i < FF_ARRAY_ELEMS(frame->buf); i++) + for (unsigned i = 0; i < FF_ARRAY_ELEMS(frame->buf); i++) if (frame->buf[i]) ret &= !!av_buffer_is_writable(frame->buf[i]); - for (i = 0; i < frame->nb_extended_buf; i++) + for (unsigned i = 0; i < frame->nb_extended_buf; i++) ret &= !!av_buffer_is_writable(frame->extended_buf[i]); + for (unsigned i = 0; i < frame->num_subtitle_areas; i++) { + area = frame->subtitle_areas[i]; + if (area) { + for (unsigned n = 0; n < FF_ARRAY_ELEMS(area->buf); n++) + if (area->buf[n]) + ret &= !!av_buffer_is_writable(area->buf[n]); + } + } + return ret; } @@ -499,6 +576,7 @@ int av_frame_make_writable(AVFrame *frame) return 0; memset(&tmp, 0, sizeof(tmp)); + tmp.type = frame->type; tmp.format = frame->format; tmp.width = frame->width; tmp.height = frame->height; @@ -509,7 +587,7 @@ int av_frame_make_writable(AVFrame *frame) if (frame->hw_frames_ctx) ret = av_hwframe_get_buffer(frame->hw_frames_ctx, &tmp, 0); else - ret = av_frame_get_buffer(&tmp, 0); + ret = av_frame_get_buffer2(&tmp, 0); if (ret < 0) return ret; @@ -544,14 +622,22 @@ AVBufferRef *av_frame_get_plane_buffer(AVFrame *frame, int plane) uint8_t *data; int planes, i; - if (frame->nb_samples) { - int channels = frame->channels; - if (!channels) - return NULL; - CHECK_CHANNELS_CONSISTENCY(frame); - planes = av_sample_fmt_is_planar(frame->format) ? channels : 1; - } else + switch(frame->type) { + case AVMEDIA_TYPE_VIDEO: planes = 4; + break; + case AVMEDIA_TYPE_AUDIO: + { + int channels = frame->channels; + if (!channels) + return NULL; + CHECK_CHANNELS_CONSISTENCY(frame); + planes = av_sample_fmt_is_planar(frame->format) ? channels : 1; + break; + } + default: + return NULL; + } if (plane < 0 || plane >= planes || !frame->extended_data[plane]) return NULL; @@ -675,17 +761,98 @@ static int frame_copy_audio(AVFrame *dst, const AVFrame *src) return 0; } +static int av_subtitle_area2area(AVSubtitleArea *dst, const AVSubtitleArea *src, int copy_data) +{ + dst->x = src->x; + dst->y = src->y; + dst->w = src->w; + dst->h = src->h; + dst->nb_colors = src->nb_colors; + dst->type = src->type; + dst->flags = src->flags; + + switch (dst->type) { + case AV_SUBTITLE_FMT_BITMAP: + + for (unsigned i = 0; i < AV_NUM_BUFFER_POINTERS; i++) { + if (src->h > 0 && src->w > 0 && src->buf[i]) { + dst->buf[0] = av_buffer_ref(src->buf[i]); + if (!dst->buf[i]) + return AVERROR(ENOMEM); + + if (copy_data) { + const int ret = av_buffer_make_writable(&dst->buf[i]); + if (ret < 0) + return ret; + } + + dst->linesize[i] = src->linesize[i]; + } + } + + memcpy(&dst->pal[0], &src->pal[0], sizeof(src->pal[0]) * 256); + + break; + case AV_SUBTITLE_FMT_TEXT: + + if (src->text) { + dst->text = av_strdup(src->text); + if (!dst->text) + return AVERROR(ENOMEM); + } + + break; + case AV_SUBTITLE_FMT_ASS: + + if (src->ass) { + dst->ass = av_strdup(src->ass); + if (!dst->ass) + return AVERROR(ENOMEM); + } + + break; + default: + + av_log(NULL, AV_LOG_ERROR, "Subtitle area has invalid format: %d", dst->type); + return AVERROR(ENOMEM); + } + + return 0; +} + +static int frame_copy_subtitles(AVFrame *dst, const AVFrame *src, int copy_data) +{ + dst->format = src->format; + + dst->num_subtitle_areas = src->num_subtitle_areas; + + if (src->num_subtitle_areas) { + dst->subtitle_areas = av_malloc_array(src->num_subtitle_areas, sizeof(AVSubtitleArea *)); + + for (unsigned i = 0; i < src->num_subtitle_areas; i++) { + dst->subtitle_areas[i] = av_mallocz(sizeof(AVSubtitleArea)); + av_subtitle_area2area(dst->subtitle_areas[i], src->subtitle_areas[i], copy_data); + } + } + + return 0; +} + int av_frame_copy(AVFrame *dst, const AVFrame *src) { if (dst->format != src->format || dst->format < 0) return AVERROR(EINVAL); - if (dst->width > 0 && dst->height > 0) + switch(dst->type) { + case AVMEDIA_TYPE_VIDEO: return frame_copy_video(dst, src); - else if (dst->nb_samples > 0 && dst->channels > 0) + case AVMEDIA_TYPE_AUDIO: return frame_copy_audio(dst, src); - - return AVERROR(EINVAL); + case AVMEDIA_TYPE_SUBTITLE: + return frame_copy_subtitles(dst, src, 1); + default: + return AVERROR(EINVAL); + } } void av_frame_remove_side_data(AVFrame *frame, enum AVFrameSideDataType type) diff --git a/libavutil/frame.h b/libavutil/frame.h index 753234792e..9dfd5a886a 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -34,6 +34,7 @@ #include "rational.h" #include "samplefmt.h" #include "pixfmt.h" +#include "subfmt.h" #include "version.h" @@ -278,7 +279,7 @@ typedef struct AVRegionOfInterest { } AVRegionOfInterest; /** - * This structure describes decoded (raw) audio or video data. + * This structure describes decoded (raw) audio, video or subtitle data. * * AVFrame must be allocated using av_frame_alloc(). Note that this only * allocates the AVFrame itself, the buffers for the data must be managed @@ -309,6 +310,13 @@ typedef struct AVRegionOfInterest { */ typedef struct AVFrame { #define AV_NUM_DATA_POINTERS 8 + /** + * Media type of the frame (audio, video, subtitles..) + * + * See AVMEDIA_TYPE_xxx + */ + enum AVMediaType type; + /** * pointer to the picture/channel planes. * This might be different from the first allocated byte. For video, @@ -390,7 +398,7 @@ typedef struct AVFrame { /** * format of the frame, -1 if unknown or unset * Values correspond to enum AVPixelFormat for video frames, - * enum AVSampleFormat for audio) + * enum AVSampleFormat for audio, AVSubtitleType for subtitles) */ int format; @@ -481,6 +489,39 @@ typedef struct AVFrame { */ uint64_t channel_layout; + /** + * Display start time, relative to packet pts, in ms. + */ + uint32_t subtitle_start_time; + + /** + * Display end time, relative to packet pts, in ms. + */ + uint32_t subtitle_end_time; + + /** + * Number of items in the @ref subtitle_areas array. + */ + unsigned num_subtitle_areas; + + /** + * Array of subtitle areas, may be empty. + */ + AVSubtitleArea **subtitle_areas; + + /** + * Usually the same as packet pts, in AV_TIME_BASE. + * + * @deprecated This is kept for compatibility reasons and corresponds to + * AVSubtitle->pts. Might be removed in the future. + */ + int64_t subtitle_pts; + + /** + * Header containing style information for text subtitles. + */ + AVBufferRef *subtitle_header; + /** * AVBuffer references backing the data for this frame. If all elements of * this array are NULL, then this frame is not reference counted. This array @@ -740,6 +781,8 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); /** * Allocate new buffer(s) for audio or video data. * + * Note: For subtitle data, use av_frame_get_buffer2 + * * The following fields must be set on frame before calling this function: * - format (pixel format for video, sample format for audio) * - width and height for video @@ -759,9 +802,39 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src); * recommended to pass 0 here unless you know what you are doing. * * @return 0 on success, a negative AVERROR on error. + * + * @deprecated Use @ref av_frame_get_buffer2 instead and set @ref AVFrame.type + * before calling. */ +attribute_deprecated int av_frame_get_buffer(AVFrame *frame, int align); +/** + * Allocate new buffer(s) for audio, video or subtitle data. + * + * The following fields must be set on frame before calling this function: + * - format (pixel format for video, sample format for audio) + * - width and height for video + * - nb_samples and channel_layout for audio + * - type (AVMediaType) + * + * This function will fill AVFrame.data and AVFrame.buf arrays and, if + * necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf. + * For planar formats, one buffer will be allocated for each plane. + * + * @warning: if frame already has been allocated, calling this function will + * leak memory. In addition, undefined behavior can occur in certain + * cases. + * + * @param frame frame in which to store the new buffers. + * @param align Required buffer size alignment. If equal to 0, alignment will be + * chosen automatically for the current CPU. It is highly + * recommended to pass 0 here unless you know what you are doing. + * + * @return 0 on success, a negative AVERROR on error. + */ +int av_frame_get_buffer2(AVFrame *frame, int align); + /** * Check if the frame data is writable. * diff --git a/libavutil/subfmt.c b/libavutil/subfmt.c new file mode 100644 index 0000000000..3998a6d486 --- /dev/null +++ b/libavutil/subfmt.c @@ -0,0 +1,50 @@ +/* + * Copyright (c) 2021 softworkz + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include "common.h" +#include "subfmt.h" + +typedef struct SubtitleFmtInfo { + enum AVSubtitleType fmt; + const char* name; +} SubtitleFmtInfo; + +static const char sub_fmt_info[AV_SUBTITLE_FMT_NB][24] = { + [AV_SUBTITLE_FMT_UNKNOWN] = "Unknown subtitle format", + [AV_SUBTITLE_FMT_BITMAP] = "Graphical subtitles", + [AV_SUBTITLE_FMT_TEXT] = "Text subtitles (plain)", + [AV_SUBTITLE_FMT_ASS] = "Text subtitles (ass)", +}; + +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt) +{ + if (sub_fmt < 0 || sub_fmt >= AV_SUBTITLE_FMT_NB) + return NULL; + return sub_fmt_info[sub_fmt]; +} + +enum AVSubtitleType av_get_subtitle_fmt(const char *name) +{ + for (int i = 0; i < AV_SUBTITLE_FMT_NB; i++) + if (!strcmp(sub_fmt_info[i], name)) + return i; + return AV_SUBTITLE_FMT_NONE; +} diff --git a/libavutil/subfmt.h b/libavutil/subfmt.h index 88078a0d14..4e0250cd23 100644 --- a/libavutil/subfmt.h +++ b/libavutil/subfmt.h @@ -21,6 +21,10 @@ #ifndef AVUTIL_SUBFMT_H #define AVUTIL_SUBFMT_H +#include + +#include "buffer.h" + enum AVSubtitleType { /** @@ -63,4 +67,48 @@ enum AVSubtitleType { AV_SUBTITLE_FMT_NB, ///< number of subtitle formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions. }; +typedef struct AVSubtitleArea { +#define AV_NUM_BUFFER_POINTERS 1 + + enum AVSubtitleType type; + int flags; + + int x; ///< top left corner of area. + int y; ///< top left corner of area. + int w; ///< width of area. + int h; ///< height of area. + int nb_colors; ///< number of colors in bitmap palette (@ref pal). + + /** + * Buffers and line sizes for the bitmap of this subtitle. + * + * @{ + */ + AVBufferRef *buf[AV_NUM_BUFFER_POINTERS]; + int linesize[AV_NUM_BUFFER_POINTERS]; + /** + * @} + */ + + uint32_t pal[256]; ///< RGBA palette for the bitmap. + + char *text; ///< 0-terminated plain UTF-8 text + char *ass; ///< 0-terminated ASS/SSA compatible event line. + +} AVSubtitleArea; + +/** + * Return the name of sub_fmt, or NULL if sub_fmt is not + * recognized. + */ +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt); + +/** + * Return a subtitle format corresponding to name, or AV_SUBTITLE_FMT_NONE + * on error. + * + * @param name Subtitle format name. + */ +enum AVSubtitleType av_get_subtitle_fmt(const char *name); + #endif /* AVUTIL_SUBFMT_H */