From patchwork Fri Apr 28 18:40:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devin Heitmueller X-Patchwork-Id: 41412 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp1529560pzb; Fri, 28 Apr 2023 10:46:05 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ73xd9pmmwHAJfAHlP8NtNctUdeyR/4EvcCuyThGpGlQbZ9bAWMv1QA1GkmIJdaKpu1VpMj X-Received: by 2002:a17:907:7ea9:b0:931:df8d:113 with SMTP id qb41-20020a1709077ea900b00931df8d0113mr5821030ejc.26.1682703965098; Fri, 28 Apr 2023 10:46:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682703965; cv=none; d=google.com; s=arc-20160816; b=xNRBgz0o2yOWLHLNwFiyUeOOVtOq9296APIfen6G9Z9N9moxNcjphkGJXiLbjenZPK 5XU+1hZdtHTIUB3duJicCRN50YVbGES9WskgIKXSPEreosl9+w9UoLlaiA5yELiwKMW2 Q1t2BLf/HmEzAmP9tqLyBKLOMZNj4YvDEVlfzCwuqxdim/fP9Kt0Lc6ReI4g30cQQTUD jN33h+QBAk9o+bf7BRIjra224W3AmcCoyL1+9ZLflRSNQ+Zdx3JbLLoRCLCtl5n5m67L QbUtSW6rFAi3a/uLgKnSa+UI/H0tHA4gPax7zjSaVB2KdZMQ3UpuAoDNUmJGbe6/SRuN RrIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=6xw7Y0eCwWJGnjH+LxYmaEwu0H+pJE58joV2mdGZnzA=; b=GDYb+XEFSvF0ggkLS4fhHlCocPSZ0pv8jQMjjoVNIkufsi9cOvatX5UM4flCQ18jE8 puP1UlRDx9n9coaAn2fvCjGXZ9R9gwc9xX7o50WadNKE7j61sggtQ5dBf9D6E6TklqkF fgnMpJXZRLfmckkdeAPXCaFaVglhuVOJPvRWEIYvWw90dqxADTQBep9R+4n8ta3JE3OM T8JGQk61ce3Bbqku7P/JxhE6mnY2/v9JIbjw374fFbJXJmCzCA1FUS8+LhNr6/1AM74H MZRXQoLexQx/jpDL8t/DEcoOfQMh0pCJnnpDKu7KD2QCuMrQQxrlkn+RzLpeQSWhTinn Zjtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=LXYBZE7I; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c5-20020a170906340500b009307d609e83si12588771ejb.905.2023.04.28.10.46.04; Fri, 28 Apr 2023 10:46:05 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=LXYBZE7I; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6253568C058; Fri, 28 Apr 2023 20:45:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1F9C968C037 for ; Fri, 28 Apr 2023 20:45:18 +0300 (EEST) Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-5f16a6fe2b1so1456576d6.1 for ; Fri, 28 Apr 2023 10:45:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ltnglobal-com.20221208.gappssmtp.com; s=20221208; t=1682703917; x=1685295917; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=MOhIhhxwn25k3nxze5Q1JDoJTucc5llCwmvvqwMQZxo=; b=LXYBZE7II8rqfUWhWGvIXT603bd4BHbTzdD6j0Gm7C2jzkCPzWqfubKEuidudbXpIb x0+jXJssC+ictn6hobiqbRw93VZRGmFD7JPqa1O03hYPt5SRrNt5tOEMu5I13JvBx4Uh sTUTFEigyoArluLrgC9QVUIbwC+6RXvSLBDfsOhNTNttzzPE25G/Gi5ljseAOqF0hoe8 TC9ztiqHL+l0iwzSje+XneNqdDIxBdHS518mmYuPIejfDesKe9bUzmfVxMXkdbiCGwOW 8Fej7QsQEMjjPPqHY9ekn9/aOyRjbN9wGhR38aFYm9XSryPsgrIrPObQY5xwFQRUzhnI 2tDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682703917; x=1685295917; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MOhIhhxwn25k3nxze5Q1JDoJTucc5llCwmvvqwMQZxo=; b=aWutoMukS6/U7pq/D9gNBMPi9FwbHxUzM7tQ68Nu/iLsVa8vj4u4pPOVHryrA+z9VT NrA1VXDT45opsdEZhoLEAQWkagKET56f6ZqINThbEGaTitZkuEJZfAsny9g+IYtTjhga 5c/0U/VLVxa88UBYX4YbJOciCZM/GXZwFQZaiYzHUZTil3Qurr/KdBPXhhzwfb9M2dTs pxQm80Z+6A1bxkjmphLmJQklQLEQ7GdmmCqNCP10vKS2onwQvkZN5RLcOzY/XuGIWr1B ZR8153D1nBuFHvm2UcCNXWFOlCm3pCOXi/KVwP4HQWtx3tZxNENH2NP6dvpN5meRQGar 6qBw== X-Gm-Message-State: AC+VfDwQhCGn2cLLRQywbcZarOkYJRSOEARoXHNhZ2r3PnpTCCBUDN5F 3QGZj333IgKHXoo9A4/4flqxdwaoRHfXjBmsws0= X-Received: by 2002:ad4:5aa2:0:b0:5e8:3496:e641 with SMTP id u2-20020ad45aa2000000b005e83496e641mr11351920qvg.13.1682703916708; Fri, 28 Apr 2023 10:45:16 -0700 (PDT) Received: from ltnt-nyc-580testdevin.livetimenet.com (pool-71-105-132-214.nycmny.fios.verizon.net. [71.105.132.214]) by smtp.gmail.com with ESMTPSA id v14-20020a0ccd8e000000b006039f5a247esm5958794qvm.78.2023.04.28.10.45.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Apr 2023 10:45:16 -0700 (PDT) From: Devin Heitmueller X-Google-Original-From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 Apr 2023 14:40:54 -0400 Message-Id: <1682707254-27604-5-git-send-email-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682707254-27604-1-git-send-email-dheitmueller@ltnglobal.com> References: <1682707254-27604-1-git-send-email-dheitmueller@ltnglobal.com> Subject: [FFmpeg-devel] [PATCH v2 4/4] decklink_enc: add support for SMPTE 2038 VANC packet output X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Devin Heitmueller MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Rq3Z8BlI9W21 Support decoding and embedding VANC packets delivered via SMPTE 2038 into the SDI output. We leverage an intermediate queue because data packets are announced separately from video but we need to embed the data into the video frame when it is output. Note that this patch has some additional abstraction for data streams in general as opposed to just SMPTE 2038 packets. This is because subsequent patches will introduce support for other data codecs. Signed-off-by: Devin Heitmueller --- libavdevice/decklink_common.cpp | 16 +++++++ libavdevice/decklink_common.h | 4 ++ libavdevice/decklink_enc.cpp | 99 ++++++++++++++++++++++++++++++++++++++++- libavdevice/decklink_enc_c.c | 1 + 4 files changed, 119 insertions(+), 1 deletion(-) diff --git a/libavdevice/decklink_common.cpp b/libavdevice/decklink_common.cpp index af1b731..427751c 100644 --- a/libavdevice/decklink_common.cpp +++ b/libavdevice/decklink_common.cpp @@ -484,6 +484,22 @@ int ff_decklink_packet_queue_get(DecklinkPacketQueue *q, AVPacket *pkt, int bloc return ret; } +int64_t ff_decklink_packet_queue_peekpts(DecklinkPacketQueue *q) +{ + PacketListEntry *pkt1; + int64_t pts = -1; + + pthread_mutex_lock(&q->mutex); + pkt1 = q->pkt_list.head; + if (pkt1) { + pts = pkt1->pkt.pts; + } + pthread_mutex_unlock(&q->mutex); + + return pts; +} + + int ff_decklink_list_devices(AVFormatContext *avctx, struct AVDeviceInfoList *device_list, int show_inputs, int show_outputs) diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index 1cc6d9c..6762607 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -115,6 +115,9 @@ struct decklink_ctx { AVCCFifo *cc_fifo; ///< closed captions + /* Output VANC queue */ + DecklinkPacketQueue vanc_queue; + /* Streams present */ int audio; int video; @@ -241,5 +244,6 @@ void ff_decklink_packet_queue_end(DecklinkPacketQueue *q); unsigned long long ff_decklink_packet_queue_size(DecklinkPacketQueue *q); int ff_decklink_packet_queue_put(DecklinkPacketQueue *q, AVPacket *pkt); int ff_decklink_packet_queue_get(DecklinkPacketQueue *q, AVPacket *pkt, int block); +int64_t ff_decklink_packet_queue_peekpts(DecklinkPacketQueue *q); #endif /* AVDEVICE_DECKLINK_COMMON_H */ diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp index 616e9c7..82e5ed1 100644 --- a/libavdevice/decklink_enc.cpp +++ b/libavdevice/decklink_enc.cpp @@ -345,6 +345,25 @@ static int decklink_setup_subtitle(AVFormatContext *avctx, AVStream *st) return ret; } +static int decklink_setup_data(AVFormatContext *avctx, AVStream *st) +{ + int ret = -1; + + switch(st->codecpar->codec_id) { +#if CONFIG_LIBKLVANC + case AV_CODEC_ID_SMPTE_2038: + /* No specific setup required */ + ret = 0; + break; +#endif + default: + av_log(avctx, AV_LOG_ERROR, "Unsupported data codec specified\n"); + break; + } + + return ret; +} + av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) { struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; @@ -370,6 +389,7 @@ av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) #if CONFIG_LIBKLVANC klvanc_context_destroy(ctx->vanc_ctx); #endif + ff_decklink_packet_queue_end(&ctx->vanc_queue); ff_ccfifo_freep(&ctx->cc_fifo); av_freep(&cctx->ctx); @@ -554,6 +574,58 @@ static int decklink_construct_vanc(AVFormatContext *avctx, struct decklink_ctx * construct_cc(avctx, ctx, pkt, &vanc_lines); construct_afd(avctx, ctx, pkt, &vanc_lines, st); + /* See if there any pending data packets to process */ + while (ff_decklink_packet_queue_size(&ctx->vanc_queue) > 0) { + AVStream *vanc_st; + AVPacket vanc_pkt; + int64_t pts; + + pts = ff_decklink_packet_queue_peekpts(&ctx->vanc_queue); + if (pts > ctx->last_pts) { + /* We haven't gotten to the video frame we are supposed to inject + the oldest VANC packet into yet, so leave it on the queue... */ + break; + } + + ret = ff_decklink_packet_queue_get(&ctx->vanc_queue, &vanc_pkt, 1); + if (vanc_pkt.pts + 1 < ctx->last_pts) { + av_log(avctx, AV_LOG_WARNING, "VANC packet too old, throwing away\n"); + av_packet_unref(&vanc_pkt); + continue; + } + + vanc_st = avctx->streams[vanc_pkt.stream_index]; + if (vanc_st->codecpar->codec_id == AV_CODEC_ID_SMPTE_2038) { + struct klvanc_smpte2038_anc_data_packet_s *pkt_2038 = 0; + + klvanc_smpte2038_parse_pes_payload(vanc_pkt.data, vanc_pkt.size, &pkt_2038); + if (pkt_2038 == NULL) { + av_log(avctx, AV_LOG_ERROR, "failed to decode SMPTE 2038 PES packet"); + av_packet_unref(&vanc_pkt); + continue; + } + for (int i = 0; i < pkt_2038->lineCount; i++) { + struct klvanc_smpte2038_anc_data_line_s *l = &pkt_2038->lines[i]; + uint16_t *vancWords = NULL; + uint16_t vancWordCount; + + if (klvanc_smpte2038_convert_line_to_words(l, &vancWords, + &vancWordCount) < 0) + break; + + ret = klvanc_line_insert(ctx->vanc_ctx, &vanc_lines, vancWords, + vancWordCount, l->line_number, 0); + free(vancWords); + if (ret != 0) { + av_log(avctx, AV_LOG_ERROR, "VANC line insertion failed\n"); + break; + } + } + klvanc_smpte2038_anc_data_packet_free(pkt_2038); + } + av_packet_unref(&vanc_pkt); + } + IDeckLinkVideoFrameAncillary *vanc; int result = ctx->dlo->CreateAncillaryData(bmdFormat10BitYUV, &vanc); if (result != S_OK) { @@ -752,6 +824,23 @@ static int decklink_write_subtitle_packet(AVFormatContext *avctx, AVPacket *pkt) return 0; } +static int decklink_write_data_packet(AVFormatContext *avctx, AVPacket *pkt) +{ + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx; + + AVPacket *avpacket = av_packet_clone(pkt); + if (!avpacket) { + av_log(avctx, AV_LOG_ERROR, "Could not clone data packet.\n"); + return AVERROR(EIO); + } + if (ff_decklink_packet_queue_put(&ctx->vanc_queue, avpacket) < 0) { + av_log(avctx, AV_LOG_WARNING, "Failed to queue DATA packet\n"); + } + + return 0; +} + extern "C" { av_cold int ff_decklink_write_header(AVFormatContext *avctx) @@ -817,6 +906,9 @@ av_cold int ff_decklink_write_header(AVFormatContext *avctx) } else if (c->codec_type == AVMEDIA_TYPE_VIDEO) { if (decklink_setup_video(avctx, st)) goto error; + } else if (c->codec_type == AVMEDIA_TYPE_DATA) { + if (decklink_setup_data(avctx, st)) + goto error; } else if (c->codec_type == AVMEDIA_TYPE_SUBTITLE) { if (decklink_setup_subtitle(avctx, st)) goto error; @@ -826,13 +918,16 @@ av_cold int ff_decklink_write_header(AVFormatContext *avctx) } } + /* Reconfigure the data/subtitle stream clocks to match the video */ for (n = 0; n < avctx->nb_streams; n++) { AVStream *st = avctx->streams[n]; AVCodecParameters *c = st->codecpar; - if(c->codec_type == AVMEDIA_TYPE_SUBTITLE) + if(c->codec_type == AVMEDIA_TYPE_DATA || + c->codec_type == AVMEDIA_TYPE_SUBTITLE) avpriv_set_pts_info(st, 64, ctx->bmd_tb_num, ctx->bmd_tb_den); } + ff_decklink_packet_queue_init(avctx, &ctx->vanc_queue); frame_rate = av_make_q(ctx->bmd_tb_den, ctx->bmd_tb_num); if (!(ctx->cc_fifo = ff_ccfifo_alloc(&frame_rate, ctx))) @@ -853,6 +948,8 @@ int ff_decklink_write_packet(AVFormatContext *avctx, AVPacket *pkt) return decklink_write_video_packet(avctx, pkt); else if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) return decklink_write_audio_packet(avctx, pkt); + else if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) + return decklink_write_data_packet(avctx, pkt); else if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) return decklink_write_subtitle_packet(avctx, pkt); diff --git a/libavdevice/decklink_enc_c.c b/libavdevice/decklink_enc_c.c index 0a3984b..d593cca 100644 --- a/libavdevice/decklink_enc_c.c +++ b/libavdevice/decklink_enc_c.c @@ -32,6 +32,7 @@ static const AVOption options[] = { { "list_devices", "use ffmpeg -sinks decklink instead", OFFSET(list_devices), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, ENC | AV_OPT_FLAG_DEPRECATED}, { "list_formats", "list supported formats" , OFFSET(list_formats), AV_OPT_TYPE_INT , { .i64 = 0 }, 0, 1, ENC }, { "preroll" , "video preroll in seconds", OFFSET(preroll ), AV_OPT_TYPE_DOUBLE, { .dbl = 0.5 }, 0, 5, ENC }, + { "queue_size", "output queue buffer size", OFFSET(queue_size ), AV_OPT_TYPE_INT64, { .i64 = (1024 * 1024)}, 0, INT64_MAX, ENC }, #if BLACKMAGIC_DECKLINK_API_VERSION >= 0x0b000000 { "duplex_mode" , "duplex mode" , OFFSET(duplex_mode ), AV_OPT_TYPE_INT , { .i64 = 0 }, 0, 5, ENC, "duplex_mode"}, #else