From patchwork Mon Dec 13 16:39:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32445 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5667644iog; Mon, 13 Dec 2021 08:40:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJxYzdQiB+qMSEzuk8UabYJ1ixihVbYBPHhYWoMT+LB6UhE9A2S6sfMRwL8+vmHzKtjFfFK1 X-Received: by 2002:a05:6402:1395:: with SMTP id b21mr66748092edv.299.1639413604238; Mon, 13 Dec 2021 08:40:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639413604; cv=none; d=google.com; s=arc-20160816; b=KWY0eHSgAa/KGwH5DFGHyM75NdJaxE8rgtC+4K59ljwrnegwjT8T1gkK1yNzihqJw4 m+8CAlthJg9YanrBtBXhupuwVoX7cfiNQqCKBQcLp0GydIKhAmJahPhMfXJ0ZtTn/Kos oNQpLfHu/Pnc1z6kQ4BgW773utkZQ2nSKjuYqSh/KpQ4RM4e6HeTEQoUQWx/FgvzbKOg IFHXhmYIvPQdchGrb2+cnXbIteCoek54Ee7sOjFCJ1PBbZoBRY6wwkqvJZdwuD/Uu7td iSjOrJadpZzik9ucz4EwYe9sz7IGoVQKd9JUo3g25yHusmrrAFetPY6Pu4crbiH+ssyu ZUGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=212fBTCPGSZHL6cv5VcTWzS4zP8QsUQ8JUx2obz7UKg=; b=z0IaTKFIWzkGPCQdwvBVhgy1MjzdGREyo9NaHNWhJy0lfQeEWtSbzKzBX3/MEQvj5P d4TEySfgjMlq6GxAZ7q9PuDdqCTdVyQzOCXmT5SUcipOFEjfVujUi0uJX33GV+HFP5Z9 Kgj/rWy1Gw1q/Mbl+T7Hfk0jSByW45iZhOnR+sIGrSTxotRa9TliFBgL+QztXMouboHM 5xhrePfCQ1yH9k6kEvT5DUSjHYOvsmTMzfh1TfOzgLrF2YgOhIycmSAkikjB7ysCqkrz kX+qVkigxwmTpYD8aPsdZf5rBTEV+NbuIVwBSzYnf0G3IQYlvjQoisk+m80C3y99m4j1 f5Cg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i4si15708304edb.450.2021.12.13.08.40.01; Mon, 13 Dec 2021 08:40:04 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2120768AFC9; Mon, 13 Dec 2021 18:39:57 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D734F68AF10 for ; Mon, 13 Dec 2021 18:39:50 +0200 (EET) Received: by mail-qt1-f176.google.com with SMTP id f20so15770767qtb.4 for ; Mon, 13 Dec 2021 08:39:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=W8Yztku6y07jkscR6+TJOyQW5yS1t+JgE89RigFTFaM=; b=LRLUBFm9VslKKa+WtjL5Ixsk+jX9sRo4C22myoWDa5JGCo0j9U1mxy8OaVUgcLwBpB dn/b9c5xWY+KQS1HwcXByvnfNUUdGvdIM8pdc8gJpZhzD24WTJxJr/Tvc5GoEZJQiPgE j/+FuzLfYdQvheJRkQ0dyJnn0LTmM49kW1FleozTcH3ea83p3JYlo43Zi+1stsaEfjxP suRgjSli6FVqzQ2eZKtkfP3fWCWBeWB6mlkbNSZ4UyZCpfd0FClF719kzwzu7jSpaHPM RyjXbov60MoRnv8fRFC99z0ulJlLXgMaY/eHfM3P4ZlrCuFViWMgOInl9b2pDD0CxA2Y 1E1Q== X-Gm-Message-State: AOAM530Z7xtXbpcFZT98yh9DsDHxeb4r5UaZZSYu17hPkxsorLLn2zdb Qt1OpTaKBqdSPBYqIpMfgpdZAlYP774= X-Received: by 2002:ac8:45d2:: with SMTP id e18mr46342165qto.112.1639413588761; Mon, 13 Dec 2021 08:39:48 -0800 (PST) Received: from smtpclient.apple ([172.58.129.176]) by smtp.gmail.com with ESMTPSA id bm25sm6277781qkb.4.2021.12.13.08.39.46 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Dec 2021 08:39:48 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: Date: Mon, 13 Dec 2021 10:39:43 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3654.100.0.2.22) Subject: [FFmpeg-devel] [PATCH v2 01/03] libavdevice/avfoundation.m: use AudioConvert, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Hrh4RDW/77o5 This is the first patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. This patch: * Implements support for AudioConverter * Switches to AudioConverter's API to convert unsupported PCM formats (non-interleaved, non-packed) to supported formats * Minimizes data copy. This fixes: https://trac.ffmpeg.org/ticket/9502 API ref: https://developer.apple.com/documentation/audiotoolbox/audio_converter_services Signed-off-by: Romain Beauxis --- libavdevice/avfoundation.m | 250 +++++++++++++++++++++---------------- 1 file changed, 144 insertions(+), 106 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 0cd6e646d5..79c9207cfa 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -111,16 +111,10 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; + UInt32 audio_buffers; + UInt32 audio_channels; + UInt32 bytes_per_sample; + AudioConverterRef audio_converter; enum AVPixelFormat pixel_format; @@ -299,7 +293,10 @@ static void destroy_context(AVFContext* ctx) ctx->avf_delegate = NULL; ctx->avf_audio_delegate = NULL; - av_freep(&ctx->audio_buffer); + if (ctx->audio_converter) { + AudioConverterDispose(ctx->audio_converter); + ctx->audio_converter = NULL; + } pthread_mutex_destroy(&ctx->frame_lock); @@ -673,6 +670,10 @@ static int get_audio_config(AVFormatContext *s) AVFContext *ctx = (AVFContext*)s->priv_data; CMFormatDescriptionRef format_desc; AVStream* stream = avformat_new_stream(s, NULL); + AudioStreamBasicDescription output_format = {0}; + int audio_bits_per_sample, audio_float, audio_be; + int audio_signed_integer, audio_packed, audio_non_interleaved; + int must_convert = 0; if (!stream) { return 1; @@ -690,60 +691,95 @@ static int get_audio_config(AVFormatContext *s) avpriv_set_pts_info(stream, 64, 1, avf_time_base); format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); + const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - if (!basic_desc) { + if (!input_format) { unlock_frames(ctx); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } + if (input_format->mFormatID != kAudioFormatLinearPCM) { + unlock_frames(ctx); + av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); + return 1; + } + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; + stream->codecpar->sample_rate = input_format->mSampleRate; + stream->codecpar->channels = input_format->mChannelsPerFrame; stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + audio_bits_per_sample = input_format->mBitsPerChannel; + audio_float = input_format->mFormatFlags & kAudioFormatFlagIsFloat; + audio_be = input_format->mFormatFlags & kAudioFormatFlagIsBigEndian; + audio_signed_integer = input_format->mFormatFlags & kAudioFormatFlagIsSignedInteger; + audio_packed = input_format->mFormatFlags & kAudioFormatFlagIsPacked; + audio_non_interleaved = input_format->mFormatFlags & kAudioFormatFlagIsNonInterleaved; + + ctx->bytes_per_sample = input_format->mBitsPerChannel >> 3; + ctx->audio_channels = input_format->mChannelsPerFrame; + + if (audio_non_interleaved) { + ctx->audio_buffers = input_format->mChannelsPerFrame; } else { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + ctx->audio_buffers = 1; + } + + if (audio_non_interleaved || !audio_packed) { + must_convert = 1; + } + + output_format.mBitsPerChannel = input_format->mBitsPerChannel; + output_format.mChannelsPerFrame = ctx->audio_channels; + output_format.mFramesPerPacket = 1; + output_format.mBytesPerFrame = output_format.mChannelsPerFrame * ctx->bytes_per_sample; + output_format.mBytesPerPacket = output_format.mFramesPerPacket * output_format.mBytesPerFrame; + output_format.mFormatFlags = kAudioFormatFlagIsPacked | audio_be; + output_format.mFormatID = kAudioFormatLinearPCM; + output_format.mReserved = 0; + output_format.mSampleRate = input_format->mSampleRate; + + if (audio_float && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_float && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F64BE : AV_CODEC_ID_PCM_F64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_signed_integer && + audio_bits_per_sample == 8) { + stream->codecpar->codec_id = AV_CODEC_ID_PCM_S8; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 16) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 24) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S64BE : AV_CODEC_ID_PCM_S64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mBitsPerChannel = 32; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + must_convert = 1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { + if (must_convert) { + OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); + if (ret != noErr) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); + av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } @@ -1048,6 +1084,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1091,7 +1128,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1105,82 +1142,83 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t input_size = CMBlockBufferGetDataLength(block_buffer); + int buffer_size = input_size / ctx->audio_buffers; + int nb_samples = input_size / (ctx->audio_channels * ctx->bytes_per_sample); + int output_size = buffer_size; - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { + UInt32 size = sizeof(output_size); + ret = AudioConverterGetProperty(ctx->audio_converter, kAudioConverterPropertyCalculateOutputBufferSize, &size, &output_size); + if (ret != noErr) { unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + return AVERROR(EIO); } - if (av_new_packet(pkt, block_buffer_size) < 0) { + if (av_new_packet(pkt, output_size) < 0) { unlock_frames(ctx); return AVERROR(EIO); } - CMItemCount count; - CMSampleTimingInfo timing_info; + if (ctx->audio_converter) { + size_t input_buffer_size = offsetof(AudioBufferList, mBuffers[0]) + (sizeof(AudioBuffer) * ctx->audio_buffers); + AudioBufferList *input_buffer = av_malloc(input_buffer_size); - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { - AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); - pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); - } + input_buffer->mNumberBuffers = ctx->audio_buffers; - pkt->stream_index = ctx->audio_stream_index; - pkt->flags |= AV_PKT_FLAG_KEY; + for (int c = 0; c < ctx->audio_buffers; c++) { + input_buffer->mBuffers[c].mNumberChannels = 1; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; + ret = CMBlockBufferGetDataPointer(block_buffer, c * buffer_size, (size_t *)&input_buffer->mBuffers[c].mDataByteSize, NULL, (void *)&input_buffer->mBuffers[c].mData); - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); + if (ret != kCMBlockBufferNoErr) { + av_free(input_buffer); + unlock_frames(ctx); + return AVERROR(EIO); + } } - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } + AudioBufferList output_buffer = { + .mNumberBuffers = 1, + .mBuffers[0] = { + .mNumberChannels = ctx->audio_channels, + .mDataByteSize = pkt->size, + .mData = pkt->data + } + }; - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { + ret = AudioConverterConvertComplexBuffer(ctx->audio_converter, nb_samples, input_buffer, &output_buffer); + av_free(input_buffer); + + if (ret != noErr) { unlock_frames(ctx); return AVERROR(EIO); } + + pkt->size = output_buffer.mBuffers[0].mDataByteSize; + } else { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } } + CMItemCount count; + CMSampleTimingInfo timing_info; + + if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); + pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); + } + + pkt->stream_index = ctx->audio_stream_index; + pkt->flags |= AV_PKT_FLAG_KEY; + CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); From patchwork Mon Dec 13 16:39:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32446 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5667921iog; Mon, 13 Dec 2021 08:40:15 -0800 (PST) X-Google-Smtp-Source: ABdhPJzb2Cp4q0lvIjC3cR/rR+kXZjFDbZVRAYLAvFXCFkltRH8ipGA76Pr+VS8htfEkYveoM6qL X-Received: by 2002:a17:906:d550:: with SMTP id cr16mr45681593ejc.544.1639413615675; Mon, 13 Dec 2021 08:40:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639413615; cv=none; d=google.com; s=arc-20160816; b=wWOyMf5ntZjbz+Kivaj0xUc8rk39FzLHCOgzJe/6cHOIuaJsaveECtPxdK6wbYbL/4 6HQgQIo6T6KR3eS3IrhQA+GOzqp931wkKeSsUTAB9fLXEq3xV7XF7rfqN4NiBBxfoGXz /EXw3Rp+fTwE4skQzCTrIUCoc7Al/sP36noIPU+Vq71Oj0sHkvC3n1YuZm77SD+p3Hcb 6d1R/vFJqgKQn/NP8yZORqrzBoaCjUTNHcWUNQRzdHSqrqt9ayplPjX5YLV/nQE/63fH RdtC5kHwQ4GJh8q87awS9IKfwqcon4UO4XExz87ThklVo1FbWbtxGCMFYCIrb3AWgyZv 6Zng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=AdinDwZ454+1Yl5f83uekXbPlEtoj5ab8cT+4qJNdZA=; b=iMHxBruNjl31kibT4OrWLwfdo3tLMJEaK+zrDwiyks/D4mKUgFWW+9hRGciXkEV4k2 vBt2ZlY2eeGr+LLa2M49zTzZIVByBlPgXCpwh538JZ1JlfDgBmwXQ6DtKS3sH+inMYn5 q6kAVTvq2OHdNI7XFkMvKA5sM8kimT3RB3qTUVb8CdawKxHPfntu8vOSHt74LovFBIRY VRWa5MYo5QoQpvtiz6D0a3Pzjo12bjN58g/QBePC8tyCUnbQloVTaxQaUFam1ljA45Fk 2blKcVA9DI+Vse8qOkZYaeiGAW/VxO8tlVE7Q56g7GjGYhwfLnwMX7hye8ZjghHe2Nw2 LLpw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id de57si14009873ejc.156.2021.12.13.08.40.15; Mon, 13 Dec 2021 08:40:15 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5621D68AFEC; Mon, 13 Dec 2021 18:40:04 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 90A1C68AF13 for ; Mon, 13 Dec 2021 18:40:02 +0200 (EET) Received: by mail-qt1-f182.google.com with SMTP id z9so15727824qtj.9 for ; Mon, 13 Dec 2021 08:40:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=fzqlf1LB03t06XTdToSIBmP2iThOdFh+EGfV1tdEm00=; b=N3SSTA7R8gtP1Uw95nWZasmI0vu7+D0drGoyPbm9e2p81ee/PveMNiLadg5LVcP1Ay m1aBeypWnQQEQEZoW9t+B4tOlfgiPECRXN2Srdi/jQY1UPS7tR07QwxtoS8fA+BMoGXB x9pKipyCwnGftoLIjk3ob0shqjf5jV5XGAqOIVttDa5pQUr5EWvPmFBlxOI92PIrE+x7 4Ub/HkHgbgnfPHfuUZCk36Qk3dugnzRhUNBRpHlMVHD7ugPQQ++gj+hTpp9HWd4fiiyT SdvQpnPVxESxIx0BZfk5Uwj8WAUhI9bl9gDV+ZPmWEFFq3YdE1+i5VSzK3h3MWsXEUSW 83rA== X-Gm-Message-State: AOAM531703PbpcGuOX6IfA3OknidodlZ/9cYbcnwbZA1H9Sen/gAfuAI tv/7pAclgmI988LvIqaKK6MZ9FWqMEw= X-Received: by 2002:a05:622a:49:: with SMTP id y9mr46442054qtw.529.1639413600317; Mon, 13 Dec 2021 08:40:00 -0800 (PST) Received: from smtpclient.apple ([172.58.129.176]) by smtp.gmail.com with ESMTPSA id bm25sm6277781qkb.4.2021.12.13.08.39.59 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Dec 2021 08:39:59 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: <1299D3A3-E37C-4EE2-9F00-BCBA46FAD441@rastageeks.org> Date: Mon, 13 Dec 2021 10:39:58 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3654.100.0.2.22) Subject: [FFmpeg-devel] [PATCH v2 02/03] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: gY/T1Vj2MC3S This is the second patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. This patch fixes the concurrency model. Avfoundation runs its own producing thread to send produced frames and ffmpeg runs its own thread to consume them. The existing implementation stores the last transmitted frame and uses a mutex to avoid concurrent access. However, this leads to situations where upcoming frames can be dropped if the ffmpeg thread is acessing the latest frame. This happens even when the thread would otherwise catch up and process frames fast enought. This patches changes this implementation to use a buffer queue with a max queue length and encapsulated thread-safety. This greatly simplifies the logic of the calling code and gives the consuming thread a chance to process all frames concurrently to the producing thread while avoiding memory leaks. Signed-off-by: Romain Beauxis --- libavdevice/avfoundation.m | 220 +++++++++++++++++++++---------------- 1 file changed, 127 insertions(+), 93 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 79c9207cfa..95414fd16a 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,7 +26,6 @@ */ #import -#include #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -80,13 +79,97 @@ { AV_PIX_FMT_NONE, 0 } }; +#define MAX_QUEUED_OBJECTS 10 + +@interface AvdeviceAvfoundationBuffer : NSObject ++ (AvdeviceAvfoundationBuffer *) fromCMSampleBufferRef:(CMSampleBufferRef)sampleBuffer; +- (CMSampleBufferRef) getCMSampleBuffer; +@end + +@implementation AvdeviceAvfoundationBuffer { + CMSampleBufferRef sampleBuffer; +} + ++ (AvdeviceAvfoundationBuffer *) fromCMSampleBufferRef:(CMSampleBufferRef)sampleBuffer { + return [[AvdeviceAvfoundationBuffer alloc] init:sampleBuffer]; +} + +- (id) init:(CMSampleBufferRef)buffer { + sampleBuffer = buffer; + return self; +} + +- (CMSampleBufferRef) getCMSampleBuffer { + return sampleBuffer; +} +@end + +@interface AvdeviceAvfoundationBufferQueue : NSObject +- (CMSampleBufferRef) dequeue; +- (NSUInteger) count; +- (void) enqueue:(CMSampleBufferRef)obj; +@end + +@implementation AvdeviceAvfoundationBufferQueue { + NSLock *mutex; + NSMutableArray *queue; +} + +- (id) init { + mutex = [[[NSLock alloc] init] retain]; + queue = [[[NSMutableArray alloc] init] retain]; + return self; +} + +- (oneway void) release { + NSEnumerator *enumerator = [queue objectEnumerator]; + AvdeviceAvfoundationBuffer *buffer; + + while (buffer = [enumerator nextObject]) { + CFRelease([buffer getCMSampleBuffer]); + } + + [mutex release]; + [queue release]; +} + +- (NSUInteger) count { + [mutex lock]; + NSUInteger c = [queue count]; + [mutex unlock]; + return c; +} + +- (CMSampleBufferRef) dequeue { + [mutex lock]; + + if ([queue count] < 1) { + [mutex unlock]; + return nil; + } + + AvdeviceAvfoundationBuffer *buffer = [queue objectAtIndex:0]; + CMSampleBufferRef sampleBuffer = [buffer getCMSampleBuffer]; + [queue removeObjectAtIndex:0]; + [mutex unlock]; + + return sampleBuffer; +} + +- (void) enqueue:(CMSampleBufferRef)buffer { + [mutex lock]; + while (MAX_QUEUED_OBJECTS < [queue count]) { + [queue removeObjectAtIndex:0]; + } + [queue addObject:[AvdeviceAvfoundationBuffer fromCMSampleBufferRef:(CMSampleBufferRef)CFRetain(buffer)]]; + [mutex unlock]; +} +@end + typedef struct { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -121,8 +204,8 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + AvdeviceAvfoundationBufferQueue *audio_frames; + AvdeviceAvfoundationBufferQueue *video_frames; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -131,16 +214,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -218,17 +291,7 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); - } - - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + [_context->video_frames enqueue:videoFrame]; } @end @@ -262,17 +325,7 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); - } - - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + [_context->audio_frames enqueue:audioFrame]; } @end @@ -284,12 +337,16 @@ static void destroy_context(AVFContext* ctx) [ctx->capture_session release]; [ctx->video_output release]; [ctx->audio_output release]; + [ctx->video_frames release]; + [ctx->audio_frames release]; [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; + ctx->video_frames = NULL; + ctx->audio_frames = NULL; ctx->avf_delegate = NULL; ctx->avf_audio_delegate = NULL; @@ -297,12 +354,6 @@ static void destroy_context(AVFContext* ctx) AudioConverterDispose(ctx->audio_converter); ctx->audio_converter = NULL; } - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static void parse_device_name(AVFormatContext *s) @@ -630,18 +681,18 @@ static int get_video_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->frames_captured < 1) { + while ([ctx->video_frames count] < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = [ctx->video_frames dequeue]; ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer) { image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); @@ -657,10 +708,7 @@ static int get_video_config(AVFormatContext *s) stream->codecpar->format = ctx->pixel_format; } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -680,27 +728,27 @@ static int get_audio_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { + while ([ctx->audio_frames count] < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = [ctx->audio_frames dequeue]; ctx->audio_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); + format_desc = CMSampleBufferGetFormatDescription(frame); const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); if (!input_format) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } if (input_format->mFormatID != kAudioFormatLinearPCM) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); return 1; } @@ -778,16 +826,13 @@ static int get_audio_config(AVFormatContext *s) if (must_convert) { OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -805,8 +850,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1006,6 +1049,8 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + ctx->video_frames = [[AvdeviceAvfoundationBufferQueue alloc] init]; + ctx->audio_frames = [[AvdeviceAvfoundationBufferQueue alloc] init]; if (video_device && add_video_device(s, video_device)) { goto fail; @@ -1088,35 +1133,31 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) AVFContext* ctx = (AVFContext*)s->priv_data; do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); - - if (ctx->current_frame != nil) { + if (1 <= [ctx->video_frames count]) { int status; int length = 0; - - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + CMSampleBufferRef video_frame = [ctx->video_frames dequeue]; + CVImageBufferRef image_buffer = CMSampleBufferGetImageBuffer(video_frame);; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(video_frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EINVAL); } if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EIO); } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(video_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1133,15 +1174,14 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = AVERROR(EIO); } } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; + CFRelease(video_frame); if (status < 0) { - unlock_frames(ctx); return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + } else if (1 <= [ctx->audio_frames count]) { + CMSampleBufferRef audio_frame = [ctx->audio_frames dequeue]; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(audio_frame); size_t input_size = CMBlockBufferGetDataLength(block_buffer); int buffer_size = input_size / ctx->audio_buffers; @@ -1151,12 +1191,12 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) UInt32 size = sizeof(output_size); ret = AudioConverterGetProperty(ctx->audio_converter, kAudioConverterPropertyCalculateOutputBufferSize, &size, &output_size); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } if (av_new_packet(pkt, output_size) < 0) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1173,7 +1213,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) if (ret != kCMBlockBufferNoErr) { av_free(input_buffer); - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1191,7 +1231,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) av_free(input_buffer); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1199,7 +1239,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } else { ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1207,7 +1247,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(audio_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1215,21 +1255,15 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(audio_frame); } else { pkt->data = NULL; - unlock_frames(ctx); if (ctx->observed_quit) { return AVERROR_EOF; } else { return AVERROR(EAGAIN); } } - - unlock_frames(ctx); } while (!pkt->data); return 0; From patchwork Mon Dec 13 16:40:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32447 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp5668499iog; Mon, 13 Dec 2021 08:40:43 -0800 (PST) X-Google-Smtp-Source: ABdhPJxyZ+F0GaZlheomj6Qy630sjT7fECkWUuQQyGK2r8gOHrgHln1YIcXsJ1kBOdZ4Vgn6/PRv X-Received: by 2002:a17:907:3da6:: with SMTP id he38mr43961518ejc.151.1639413643360; Mon, 13 Dec 2021 08:40:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639413643; cv=none; d=google.com; s=arc-20160816; b=Z5kViBiBYrjf1dmKrSvdFUUZnUn7uqUOgQcA7ih19kroz1/CP3K6fHoxp43dfCKgSe qTqk2ibUUzwGM2vy/Hd4rhF9XZjKBw3u4j5jb9yflKTD4WzQ2vWwe8MVuk33MDmKzzR1 I887dF9NMRIToilSJ9zPsJPzNQXDSsYIaXYoxM3bJJdBhsWoCV7dV9U1gtg2YHZxDle/ B3hbtU2Iaje9b9X+UaVE7b/yQlYR2wkP98jP/8BhtgrRisyNzQjcwllGITvki8aGJX2C Zz+c6A5P3jr2zN5GgmJmX2Qj4RlTMQbAFnXrdfnyq/2PVeksVm1MebgwTiPBdza8Fvgq hsLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=S3Z1k9cezHKZmCO+1e3CZlH4rqUG8jIhmpQGUNQZDQc=; b=JgLJ8J+RLEF4ZhyINRluFHrkeXoYgzQkyQGbevvnTmNu13WahTQI+FYGpomlbWYonR qrCw7CS05NFNqxR2LFuHRTXB1ut3+WpLK8A2qiE5mxc8YeiuHSvXiGiioIEXVL48iPES nQZO1JvQQUaSHkBZjodWTpeg/mjD+huSepJUc378LX0Q5e0IKSdEG2ATb6j7XFvPqSpc 9anrNQUEhv2EsBuffK3xKeGJJP/V5zbp2WVm0MIJVFRFkNItMYQxi9J0JlhVUTLjjNto e7vPuaDNo/9hcABL2LL15LNKnMxbWenEhZmiYcRHSZ2jJogVSRRvbambmUvWXl+vTiza OI3A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x22si1719693edl.139.2021.12.13.08.40.43; Mon, 13 Dec 2021 08:40:43 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6BC6268B005; Mon, 13 Dec 2021 18:40:40 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0B09168AFFA for ; Mon, 13 Dec 2021 18:40:34 +0200 (EET) Received: by mail-qv1-f42.google.com with SMTP id gu12so14885749qvb.6 for ; Mon, 13 Dec 2021 08:40:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=c8z5m6kPYJ+9OUuPFO4Gi78UBtQgLUNxZQgNru6HupY=; b=Kh/fQs8rhSUvQ3tsuNoy1RVm9IZPhvSs1D0u0OU38IWPgHmAAWm8DQylUN1F1aZbMQ nKNNhm3curFdXMm51zO+gqEvFNwe8SqWcRYn0fDWKT3lYFXns23/bph5yHy1w+Oo1yMz E7ThEo9ErDJmF/u/vq25fodoqXS6oZIW74nnPGIO3cWkPAadELcPReLUVqlTr+VeNz7k eMu3Kp1UifsYIoxQCMoJ9LtO6WKdHTR+2/fb6CJ74jkJVCzrsL5UXoEaP6DsnVBiv43r 3Z8RZd7u+HrfPb0CXB+ovZrf3CkH3JQ/rbe4Z4SOK/Bn8ODwFeY7LYDdV7oibczYT/gt iYig== X-Gm-Message-State: AOAM533+Rf0zt3WmsluAAPgzico2GT2sQ4L55L8+hLHFVN1iFAOOM9JO 02EVvL2GKQKTygFgH0HW4OoJKpuppAk= X-Received: by 2002:a05:6214:e47:: with SMTP id o7mr44257841qvc.61.1639413632260; Mon, 13 Dec 2021 08:40:32 -0800 (PST) Received: from smtpclient.apple ([172.58.129.176]) by smtp.gmail.com with ESMTPSA id bm25sm6277781qkb.4.2021.12.13.08.40.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Dec 2021 08:40:31 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: <5835AAC9-4ECD-4007-906B-8A9497EF911B@rastageeks.org> Date: Mon, 13 Dec 2021 10:40:31 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3654.100.0.2.22) Subject: [FFmpeg-devel] [PATCH v2 03/03] libavdevice/avfoundation.m: Allow to select devices by digest. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: VOd5mA9yRxre This is the third patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. This patch adds a digest to avfoundation devices, when available. This is needed because device index can change while the machine is running when devices are plugged or unplugged and device names can be tricky to use with localization and etc. The only device type that are excluded are screen capture because the logic to select them seems a little different and I wanted to minimized the changes. Also, for these devices, the name is localized in english, quite straight forward and should not change. Signed-off-by: Romain Beauxis --- doc/indevs.texi | 6 ++-- libavdevice/avfoundation.m | 60 ++++++++++++++++++++++++++++++++++---- 2 files changed, 58 insertions(+), 8 deletions(-) diff --git a/doc/indevs.texi b/doc/indevs.texi index 5be647f70a..8345b64a28 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -114,7 +114,7 @@ The input filename has to be given in the following syntax: -i "[[VIDEO]:[AUDIO]]" @end example The first entry selects the video input while the latter selects the audio input. -The stream has to be specified by the device name or the device index as shown by the device list. +The stream has to be specified by the device name, index or digest as shown by the device list. Alternatively, the video and/or audio input device can be chosen by index using the @option{ -video_device_index @@ -127,7 +127,9 @@ and/or device name or index given in the input filename. All available devices can be enumerated by using @option{-list_devices true}, listing -all device names and corresponding indices. +all device names, corresponding indices and digests, when available. Device name can be +tricky to use when localized and device index can change when devices are plugged or unplugged. A device +hash, when available, uniquely identifies a device and should not change over time. There are two device name aliases: @table @code diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 95414fd16a..bede51bda0 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,6 +26,7 @@ */ #import +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -79,6 +80,28 @@ { AV_PIX_FMT_NONE, 0 } }; +#define DEVICES_DIGEST_LENGTH 8 + +@interface AvdeviceAvfoundationDigest : NSObject ++ (NSString *)fromString:(NSString *)input; +@end + +@implementation AvdeviceAvfoundationDigest : NSObject ++ (NSString *) fromString:(NSString *)input { + const char *cStr = [input UTF8String]; + unsigned char digest[CC_SHA256_DIGEST_LENGTH]; + CC_SHA256( cStr, strlen(cStr), digest ); + + NSMutableString *output = [NSMutableString stringWithCapacity:CC_SHA256_DIGEST_LENGTH * 2]; + + for(int i = 0; i < CC_SHA256_DIGEST_LENGTH; i++) + [output appendFormat:@"%02x", digest[i]]; + + // The "d" prefix makes sure that digest strings are never mistaken for numbers. + return [@"d" stringByAppendingString:[output substringToIndex:DEVICES_DIGEST_LENGTH]]; +} +@end + #define MAX_QUEUED_OBJECTS 10 @interface AvdeviceAvfoundationBuffer : NSObject @@ -860,13 +883,15 @@ static int avf_read_header(AVFormatContext *s) av_log(ctx, AV_LOG_INFO, "AVFoundation video devices:\n"); for (AVCaptureDevice *device in devices) { const char *name = [[device localizedName] UTF8String]; + NSString *digest = [AvdeviceAvfoundationDigest fromString:[[NSString alloc] initWithUTF8String:name]]; index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + av_log(ctx, AV_LOG_INFO, "[%d] %s (digest: %s)\n", index, name, [digest UTF8String]); } for (AVCaptureDevice *device in devices_muxed) { const char *name = [[device localizedName] UTF8String]; + NSString *digest = [AvdeviceAvfoundationDigest fromString:[[NSString alloc] initWithUTF8String:name]]; index = [devices count] + [devices_muxed indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + av_log(ctx, AV_LOG_INFO, "[%d] %s (digest: %s)\n", index, name, [digest UTF8String]); } #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 if (num_screens > 0) { @@ -882,8 +907,9 @@ static int avf_read_header(AVFormatContext *s) devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { const char *name = [[device localizedName] UTF8String]; + NSString *digest = [AvdeviceAvfoundationDigest fromString:[[NSString alloc] initWithUTF8String:name]]; int index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + av_log(ctx, AV_LOG_INFO, "[%d] %s (digest: %s)\n", index, name, [digest UTF8String]); } goto fail; } @@ -945,14 +971,29 @@ static int avf_read_header(AVFormatContext *s) } else { // looking for video inputs for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + break; + } + + NSString *digest = [AvdeviceAvfoundationDigest fromString:[[NSString alloc] initWithUTF8String:name]]; + if (!strncmp(ctx->video_filename, [digest UTF8String], strlen(ctx->video_filename))) { video_device = device; break; } } // looking for muxed inputs for (AVCaptureDevice *device in devices_muxed) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + ctx->video_is_muxed = 1; + break; + } + + NSString *digest = [AvdeviceAvfoundationDigest fromString:[[NSString alloc] initWithUTF8String:name]]; + if (!strncmp(ctx->video_filename, [digest UTF8String], strlen(ctx->video_filename))) { video_device = device; ctx->video_is_muxed = 1; break; @@ -1017,7 +1058,14 @@ static int avf_read_header(AVFormatContext *s) NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->audio_filename, [[device localizedName] UTF8String], strlen(ctx->audio_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->audio_filename, name, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } + + NSString *digest = [AvdeviceAvfoundationDigest fromString:[[NSString alloc] initWithUTF8String:name]]; + if (!strncmp(ctx->audio_filename, [digest UTF8String], strlen(ctx->audio_filename))) { audio_device = device; break; }