From patchwork Thu Nov 16 18:34:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devin Heitmueller X-Patchwork-Id: 6123 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.94 with SMTP id m30csp6149504jah; Thu, 16 Nov 2017 10:34:54 -0800 (PST) X-Google-Smtp-Source: AGs4zMYnNegMMMiFKbUJ/wVwH/nduPj0rsR1kOky6E+gddrMmwUzECbRc5BKG0fFfupQh5+Wuap1 X-Received: by 10.223.200.6 with SMTP id d6mr97411wrh.118.1510857293925; Thu, 16 Nov 2017 10:34:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510857293; cv=none; d=google.com; s=arc-20160816; b=olcLN+Wx/QgBj4XvJavog85OrnR9FmWcimK/qZcbe2fn0IXdeB4Cc97FEXO4Ub5dZW TqXwVemcgNO0UBDdamdA8kaQFN2R3onodiHdSMuUULU4e44Vv11vHfAfaKQl7dZ7O5g4 FBSx0+Q21RhkOhBndzBUutOLZadZN1r5QLhX0oJNF8UMORUxJL5rMYsIQxSDOTPT48Nu 0ovJjoLO71YwuNPT+g5RXhtNqe5BMCnOFHDf1NjpndzBWdGsspumCwfXrHvQOHZqWesR l6qiSNkle85SS0MnyP3fdTlVgrYAR9J3fGmJXekxiQLDBIwSBoK2cUwx91yYRAldNFrZ sCUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:delivered-to:arc-authentication-results; bh=Su+pMWMCFDwyvUTSDHCg1lSwBt5osTmdqmwlb5lLTHs=; b=rNvPnSUSBr0lUtmIbDugVXmxMAkZ5Qn422Q1o2VQN2I1k74AyBnAj8Re2llHqih1GE rFvyHG1UX7AylbtYcKfXgZYvWgXrmPuJZ3N1tfVrSRtTmiysSAMSP5owyApxwppiNXSc m8kaF1K3PAwAq8AHvmtTeBmUpi2XdHS5ZfQ5g7TCggm+oXtJwcrKPlmxOoXQuCv/9wxa EYxXJEpgJyy7BoLiNTj61OcG+YN452mSSH37ra3tOP49Gl65jvbS2SL5MzrjtakpyhEC GNa+Q/PMN4P5gM9W6+v0pripzeF7k6gFmDVSm44W6H724N6uZSxO/zWEfG3m9lK3tYyu UVSQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4si72422wmz.132.2017.11.16.10.34.53; Thu, 16 Nov 2017 10:34:53 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0C0F3689F0F; Thu, 16 Nov 2017 20:34:21 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from was-smtp1.livetimenet.net (50-206-97-56-static.hfc.comcastbusiness.net [50.206.97.56]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 427416882E5 for ; Thu, 16 Nov 2017 20:34:13 +0200 (EET) Received: by was-smtp1.livetimenet.net with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.84_2) (envelope-from ) id 1eFOzb-0002o7-8P; Thu, 16 Nov 2017 13:34:28 -0500 From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Nov 2017 13:34:13 -0500 Message-Id: <20171116183417.93073-3-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 2.13.2 In-Reply-To: <20171116183417.93073-1-dheitmueller@ltnglobal.com> References: <20171116183417.93073-1-dheitmueller@ltnglobal.com> X-Spam-Score: -1.9 (-) Subject: [FFmpeg-devel] [PATCH 2/6] decklink: Introduce support for capture of multiple audio streams X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: jgreen@ltnglobal.com, Devin Heitmueller MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Add support for the ability to capture all audio pairs available to the capture hardware. Each pair is exposed as a different audio stream, which matches up with the most common use cases for the broadcast space (i.e. where there is one stereo pair per audio language). To support the existing use case where multi-channel audio can be captured (i.e. 7.1), we introduced a new configuration option, which defaults to the existing behavior. Signed-off-by: Devin Heitmueller --- libavdevice/decklink_common.cpp | 9 +++ libavdevice/decklink_common.h | 8 ++- libavdevice/decklink_common_c.h | 6 ++ libavdevice/decklink_dec.cpp | 134 +++++++++++++++++++++++++++++++--------- libavdevice/decklink_dec_c.c | 3 + 5 files changed, 130 insertions(+), 30 deletions(-) diff --git a/libavdevice/decklink_common.cpp b/libavdevice/decklink_common.cpp index c425f4a..050b839 100644 --- a/libavdevice/decklink_common.cpp +++ b/libavdevice/decklink_common.cpp @@ -473,5 +473,14 @@ int ff_decklink_init_device(AVFormatContext *avctx, const char* name) return AVERROR_EXTERNAL; } + if (ctx->attr->GetInt(BMDDeckLinkMaximumAudioChannels, &ctx->max_audio_channels) != S_OK) { + av_log(avctx, AV_LOG_WARNING, "Could not determine number of audio channels\n"); + ctx->max_audio_channels = 0; + } + if (ctx->max_audio_channels > DECKLINK_MAX_AUDIO_CHANNELS) { + av_log(avctx, AV_LOG_WARNING, "Decklink card reported support for more channels than ffmpeg supports\n"); + ctx->max_audio_channels = DECKLINK_MAX_AUDIO_CHANNELS; + } + return 0; } diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index e951b17..bbe4deb 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -31,6 +31,10 @@ #include "libklvanc/vanc.h" #endif +/* Maximum number of channels possible across variants of Blackmagic cards. + Actual number for any particular model of card may be lower */ +#define DECKLINK_MAX_AUDIO_CHANNELS 32 + class decklink_output_callback; class decklink_input_callback; @@ -65,6 +69,7 @@ struct decklink_ctx { int bmd_height; int bmd_field_dominance; int supports_vanc; + int64_t max_audio_channels; /* Capture buffer queue */ AVPacketQueue queue; @@ -79,7 +84,8 @@ struct decklink_ctx { int64_t last_pts; unsigned long frameCount; unsigned int dropped; - AVStream *audio_st; + AVStream *audio_st[DECKLINK_MAX_AUDIO_CHANNELS]; + int num_audio_streams; AVStream *video_st; AVStream *teletext_st; uint16_t cdp_sequence_num; diff --git a/libavdevice/decklink_common_c.h b/libavdevice/decklink_common_c.h index 368ac25..02011ed 100644 --- a/libavdevice/decklink_common_c.h +++ b/libavdevice/decklink_common_c.h @@ -30,6 +30,11 @@ typedef enum DecklinkPtsSource { PTS_SRC_WALLCLOCK = 4, } DecklinkPtsSource; +typedef enum DecklinkAudioMode { + AUDIO_MODE_DISCRETE = 0, + AUDIO_MODE_PAIRS = 1, +} DecklinkAudioMode; + struct decklink_cctx { const AVClass *cclass; @@ -42,6 +47,7 @@ struct decklink_cctx { double preroll; int v210; int audio_channels; + int audio_mode; int audio_depth; int duplex_mode; DecklinkPtsSource audio_pts_source; diff --git a/libavdevice/decklink_dec.cpp b/libavdevice/decklink_dec.cpp index e90b428..11b7e60 100644 --- a/libavdevice/decklink_dec.cpp +++ b/libavdevice/decklink_dec.cpp @@ -625,9 +625,54 @@ static int64_t get_pkt_pts(IDeckLinkVideoInputFrame *videoFrame, return pts; } +static int setup_audio(AVFormatContext *avctx) +{ + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx; + AVStream *st; + int ret = 0; + + if (cctx->audio_mode == AUDIO_MODE_DISCRETE) { + st = avformat_new_stream(avctx, NULL); + if (!st) { + av_log(avctx, AV_LOG_ERROR, "Cannot add stream\n"); + ret = AVERROR(ENOMEM); + goto error; + } + st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; + st->codecpar->codec_id = ctx->audio_depth == 32 ? AV_CODEC_ID_PCM_S32LE : AV_CODEC_ID_PCM_S16LE; + st->codecpar->sample_rate = bmdAudioSampleRate48kHz; + st->codecpar->channels = cctx->audio_channels; + avpriv_set_pts_info(st, 64, 1, 1000000); /* 64 bits pts in us */ + ctx->audio_st[0] = st; + ctx->num_audio_streams++; + } else { + for (int i = 0; i < ctx->max_audio_channels / 2; i++) { + st = avformat_new_stream(avctx, NULL); + if (!st) { + av_log(avctx, AV_LOG_ERROR, "Cannot add stream %d\n", i); + ret = AVERROR(ENOMEM); + goto error; + } + st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; + st->codecpar->codec_id = ctx->audio_depth == 32 ? AV_CODEC_ID_PCM_S32LE : AV_CODEC_ID_PCM_S16LE; + st->codecpar->sample_rate = bmdAudioSampleRate48kHz; + st->codecpar->channels = 2; + avpriv_set_pts_info(st, 64, 1, 1000000); /* 64 bits pts in us */ + ctx->audio_st[i] = st; + ctx->num_audio_streams++; + } + cctx->audio_channels = ctx->max_audio_channels; + } + +error: + return ret; +} + HRESULT decklink_input_callback::VideoInputFrameArrived( IDeckLinkVideoInputFrame *videoFrame, IDeckLinkAudioInputPacket *audioFrame) { + decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; void *frameBytes; void *audioFrameBytes; BMDTimeValue frameTime; @@ -766,24 +811,57 @@ HRESULT decklink_input_callback::VideoInputFrameArrived( // Handle Audio Frame if (audioFrame) { - AVPacket pkt; - BMDTimeValue audio_pts; - av_init_packet(&pkt); - - //hack among hacks - pkt.size = audioFrame->GetSampleFrameCount() * ctx->audio_st->codecpar->channels * (ctx->audio_depth / 8); audioFrame->GetBytes(&audioFrameBytes); - audioFrame->GetPacketTime(&audio_pts, ctx->audio_st->time_base.den); - pkt.pts = get_pkt_pts(videoFrame, audioFrame, wallclock, ctx->audio_pts_source, ctx->audio_st->time_base, &initial_audio_pts); - pkt.dts = pkt.pts; - //fprintf(stderr,"Audio Frame size %d ts %d\n", pkt.size, pkt.pts); - pkt.flags |= AV_PKT_FLAG_KEY; - pkt.stream_index = ctx->audio_st->index; - pkt.data = (uint8_t *)audioFrameBytes; + if (cctx->audio_mode == AUDIO_MODE_DISCRETE) { + AVPacket pkt; + BMDTimeValue audio_pts; + av_init_packet(&pkt); - if (avpacket_queue_put(&ctx->queue, &pkt) < 0) { - ++ctx->dropped; + //hack among hacks + pkt.size = audioFrame->GetSampleFrameCount() * ctx->audio_st[0]->codecpar->channels * (ctx->audio_depth / 8); + audioFrame->GetBytes(&audioFrameBytes); + audioFrame->GetPacketTime(&audio_pts, ctx->audio_st[0]->time_base.den); + pkt.pts = get_pkt_pts(videoFrame, audioFrame, wallclock, ctx->audio_pts_source, ctx->audio_st[0]->time_base, &initial_audio_pts); + pkt.dts = pkt.pts; + + pkt.flags |= AV_PKT_FLAG_KEY; + pkt.stream_index = ctx->audio_st[0]->index; + pkt.data = (uint8_t *)audioFrameBytes; + + if (avpacket_queue_put(&ctx->queue, &pkt) < 0) { + ++ctx->dropped; + } + } else { + /* Need to deinterleave audio */ + int audio_offset = 0; + int audio_stride = cctx->audio_channels * ctx->audio_depth / 8; + for (int i = 0; i < ctx->num_audio_streams; i++) { + int sample_size = ctx->audio_st[i]->codecpar->channels * + ctx->audio_st[i]->codecpar->bits_per_coded_sample / 8; + AVPacket pkt; + int ret = av_new_packet(&pkt, audioFrame->GetSampleFrameCount() * sample_size); + if (ret != 0) + continue; + + pkt.pts = get_pkt_pts(videoFrame, audioFrame, wallclock, ctx->audio_pts_source, + ctx->audio_st[i]->time_base, &initial_audio_pts); + pkt.dts = pkt.pts; + pkt.flags |= AV_PKT_FLAG_KEY; + pkt.stream_index = ctx->audio_st[i]->index; + + uint8_t *audio_in = ((uint8_t *) audioFrameBytes) + audio_offset; + for (int x = 0; x < pkt.size; x += sample_size) { + memcpy(&pkt.data[x], audio_in, sample_size); + audio_in += audio_stride; + } + + if (avpacket_queue_put(&ctx->queue, &pkt) < 0) + ++ctx->dropped; + + av_packet_unref(&pkt); + audio_offset += sample_size; + } } } @@ -934,18 +1012,7 @@ av_cold int ff_decklink_read_header(AVFormatContext *avctx) #endif /* Setup streams. */ - st = avformat_new_stream(avctx, NULL); - if (!st) { - av_log(avctx, AV_LOG_ERROR, "Cannot add stream\n"); - ret = AVERROR(ENOMEM); - goto error; - } - st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - st->codecpar->codec_id = cctx->audio_depth == 32 ? AV_CODEC_ID_PCM_S32LE : AV_CODEC_ID_PCM_S16LE; - st->codecpar->sample_rate = bmdAudioSampleRate48kHz; - st->codecpar->channels = cctx->audio_channels; - avpriv_set_pts_info(st, 64, 1, 1000000); /* 64 bits pts in us */ - ctx->audio_st=st; + setup_audio(avctx); st = avformat_new_stream(avctx, NULL); if (!st) { @@ -1031,8 +1098,17 @@ av_cold int ff_decklink_read_header(AVFormatContext *avctx) ctx->teletext_st = st; } - av_log(avctx, AV_LOG_VERBOSE, "Using %d input audio channels\n", ctx->audio_st->codecpar->channels); - result = ctx->dli->EnableAudioInput(bmdAudioSampleRate48kHz, cctx->audio_depth == 32 ? bmdAudioSampleType32bitInteger : bmdAudioSampleType16bitInteger, ctx->audio_st->codecpar->channels); + if (cctx->audio_mode == AUDIO_MODE_DISCRETE) { + av_log(avctx, AV_LOG_VERBOSE, "Using %d input audio channels\n", ctx->audio_st[0]->codecpar->channels); + result = ctx->dli->EnableAudioInput(bmdAudioSampleRate48kHz, + ctx->audio_depth == 32 ? bmdAudioSampleType32bitInteger : bmdAudioSampleType16bitInteger, + ctx->audio_st[0]->codecpar->channels); + } else { + av_log(avctx, AV_LOG_VERBOSE, "Using %d input audio channels\n", (int)ctx->max_audio_channels); + result = ctx->dli->EnableAudioInput(bmdAudioSampleRate48kHz, + ctx->audio_depth == 32 ? bmdAudioSampleType32bitInteger : bmdAudioSampleType16bitInteger, + ctx->max_audio_channels); + } if (result != S_OK) { av_log(avctx, AV_LOG_ERROR, "Cannot enable audio input\n"); diff --git a/libavdevice/decklink_dec_c.c b/libavdevice/decklink_dec_c.c index 1c6d826..d3d8c84 100644 --- a/libavdevice/decklink_dec_c.c +++ b/libavdevice/decklink_dec_c.c @@ -44,6 +44,9 @@ static const AVOption options[] = { { "standard", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0x7fff9fffeLL}, 0, 0, DEC, "teletext_lines"}, { "all", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0x7ffffffffLL}, 0, 0, DEC, "teletext_lines"}, { "channels", "number of audio channels", OFFSET(audio_channels), AV_OPT_TYPE_INT , { .i64 = 2 }, 2, 16, DEC }, + { "audio_mode", "audio mode", OFFSET(audio_mode), AV_OPT_TYPE_INT, { .i64 = AUDIO_MODE_DISCRETE}, 0, 1, DEC, "audio_mode"}, + { "discrete", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = AUDIO_MODE_DISCRETE}, 0, 0, DEC, "audio_mode"}, + { "pairs", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = AUDIO_MODE_PAIRS}, 0, 0, DEC, "audio_mode"}, { "duplex_mode", "duplex mode", OFFSET(duplex_mode), AV_OPT_TYPE_INT, { .i64 = 0}, 0, 2, DEC, "duplex_mode"}, { "unset", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0}, 0, 0, DEC, "duplex_mode"}, { "half", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 1}, 0, 0, DEC, "duplex_mode"},