From patchwork Fri Dec 17 15:12:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32685 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1658871iog; Fri, 17 Dec 2021 07:12:47 -0800 (PST) X-Google-Smtp-Source: ABdhPJwFoPGCo4Xjlb4IAq0pSY/EicQBNtNXTPcqlJT7/iGJEId46JN/Xuh3fMVhTg1kSLm71+x+ X-Received: by 2002:a17:906:544f:: with SMTP id d15mr2919783ejp.373.1639753967776; Fri, 17 Dec 2021 07:12:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639753967; cv=none; d=google.com; s=arc-20160816; b=OgFlNWCSbFTjkJAqkn1yS6bDMDjOnMO0mFQ/ywueneIoegHotvvpB3QQFZ4qUGYHby GMry/azVv/DqcMED6h4V6hM7T8qaRrxW7Jl1K86qN7PDFORTrd7R+Ua4w+fvtcWG2NOf sWQOlfx/4Ja/fwA5Rzyqf/YStCsp2XVPmKwSwE4xYo+lIHcYzqKI+Sx7x7OVr7iB5j4T RaArr/AtTIafHlHl3fNRT3wYnDEbndOrtA6YhBsWFAFZQiorDwkwGTGvF09GrVPQEGPG NZcJWbWc1rKCV5LXJMMK94M3W/o5ed3jNSMcdnGww6w/Mnhnsnow6bF1Z7gInp5amZxv lBCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:content-language:to:from:user-agent:mime-version :date:message-id:delivered-to; bh=at34Nzw9x810mpCNGq+ml/u+MCQ187yTfdWcHidNw2o=; b=wbvmKtGzSyP26rG8eTpc63/ILEBuFUZbbBVmUe8Uusce1JuOh0pDhaBA5w2K0Nomir sHQkzW3F0yu2FWEoJUvMxzevgKQFRHCsJIEOHQs+BziVREhqUAnRj9kdksptQbUm2jxn HU3ZD6Z05Q8o0m9LZpHibWtnHB2uDrisBgtPqpk4C2KPHSi+WWYB4dL99pVy3S1nLeEF UQ/j+kkRCNbvgC8KQ5yEb0BYZ4pZs8M4wzok0kliS5gb1ePkaD7VZGD/fI/ZIYth3C6Q seUlambD/7p1RdwnAIAA3ydMfy/qL+Nry3TF279cbrKD4A+gMKbjxJkcvEPjIUvVQE6x efSw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id op26si4749430ejb.630.2021.12.17.07.12.46; Fri, 17 Dec 2021 07:12:47 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4780968AE7F; Fri, 17 Dec 2021 17:12:42 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4251368008E for ; Fri, 17 Dec 2021 17:12:36 +0200 (EET) Received: by mail-qt1-f180.google.com with SMTP id t34so2879233qtc.7 for ; Fri, 17 Dec 2021 07:12:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:from :subject:to:content-language:content-transfer-encoding; bh=L+UYsfhaFLrdtBa6DU+iuy+xgNQcI1AVSuAkufi5Z3A=; b=M4qMCzmAGJ6Ny/5C12DRELtHowmBQV7S7As1rsR4WiBCM7ori6PaBX+t7YMrn1aPOE 5dP/WjUu42os5x/7orwJLRX0OLRtUFJrRIEKeI5uj4pWNq1icuja/dP3ycI5/5Jglq8G kdLntOLyNQil1z6+4j3Tm2IS85oBPPSSjIJLMCzjitFqlXenRfFjeIN6Qn7Vekje7Hk/ Z0yBIme/5Dzo9vRDQrbFEkbHm/9CnG2QCt2eNWTIq7iPLUD2bhmrgOcxvGAHr5sPrY9k TdquXQmdR0EncWH5N4nlyMidJgNsSBOKdzSMf6+aKcfCOOwgJEJ5r4/tDrX2s9hXo34C NQHQ== X-Gm-Message-State: AOAM53085Mxg5G0sn6PENGs3FR+Eueo6EgWxw3+kUTpEtYkXDoL8Ddpp uGeOUFiX3UXuVPfPKVaaONrqVsKcB8fvpQ== X-Received: by 2002:a05:622a:1392:: with SMTP id o18mr2682108qtk.270.1639753954285; Fri, 17 Dec 2021 07:12:34 -0800 (PST) Received: from [192.168.86.89] ([172.58.6.224]) by smtp.gmail.com with ESMTPSA id u7sm4914081qkp.17.2021.12.17.07.12.32 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 17 Dec 2021 07:12:33 -0800 (PST) Message-ID: <6aba2cc8-8e6e-e01f-d17c-c919bf0121dc@rastageeks.org> Date: Fri, 17 Dec 2021 09:12:29 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 From: Romain Beauxis To: ffmpeg-devel@ffmpeg.org Content-Language: en-US Subject: [FFmpeg-devel] [PATCH v4 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: MZo8sfHblTlx This is the first patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes: * v2: None * v3: None * v4: None * Implement support for AudioConverter * Switch to AudioConverter's API to convert unsupported PCM formats (non-interleaved, non-packed) to supported formats * Minimize data copy. This fixes: https://trac.ffmpeg.org/ticket/9502 API ref: https://developer.apple.com/documentation/audiotoolbox/audio_converter_services Signed-off-by: Romain Beauxis --- libavdevice/avfoundation.m | 250 +++++++++++++++++++++---------------- 1 file changed, 144 insertions(+), 106 deletions(-) + if (ctx->audio_converter) { + AudioConverterDispose(ctx->audio_converter); + ctx->audio_converter = NULL; + } pthread_mutex_destroy(&ctx->frame_lock); @@ -673,6 +670,10 @@ static int get_audio_config(AVFormatContext *s) AVFContext *ctx = (AVFContext*)s->priv_data; CMFormatDescriptionRef format_desc; AVStream* stream = avformat_new_stream(s, NULL); + AudioStreamBasicDescription output_format = {0}; + int audio_bits_per_sample, audio_float, audio_be; + int audio_signed_integer, audio_packed, audio_non_interleaved; + int must_convert = 0; if (!stream) { return 1; @@ -690,60 +691,95 @@ static int get_audio_config(AVFormatContext *s) avpriv_set_pts_info(stream, 64, 1, avf_time_base); format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); + const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - if (!basic_desc) { + if (!input_format) { unlock_frames(ctx); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } + if (input_format->mFormatID != kAudioFormatLinearPCM) { + unlock_frames(ctx); + av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); + return 1; + } + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; + stream->codecpar->sample_rate = input_format->mSampleRate; + stream->codecpar->channels = input_format->mChannelsPerFrame; stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + audio_bits_per_sample = input_format->mBitsPerChannel; + audio_float = input_format->mFormatFlags & kAudioFormatFlagIsFloat; + audio_be = input_format->mFormatFlags & kAudioFormatFlagIsBigEndian; + audio_signed_integer = input_format->mFormatFlags & kAudioFormatFlagIsSignedInteger; + audio_packed = input_format->mFormatFlags & kAudioFormatFlagIsPacked; + audio_non_interleaved = input_format->mFormatFlags & kAudioFormatFlagIsNonInterleaved; + + ctx->bytes_per_sample = input_format->mBitsPerChannel >> 3; + ctx->audio_channels = input_format->mChannelsPerFrame; + + if (audio_non_interleaved) { + ctx->audio_buffers = input_format->mChannelsPerFrame; } else { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + ctx->audio_buffers = 1; + } + + if (audio_non_interleaved || !audio_packed) { + must_convert = 1; + } + + output_format.mBitsPerChannel = input_format->mBitsPerChannel; + output_format.mChannelsPerFrame = ctx->audio_channels; + output_format.mFramesPerPacket = 1; + output_format.mBytesPerFrame = output_format.mChannelsPerFrame * ctx->bytes_per_sample; + output_format.mBytesPerPacket = output_format.mFramesPerPacket * output_format.mBytesPerFrame; + output_format.mFormatFlags = kAudioFormatFlagIsPacked | audio_be; + output_format.mFormatID = kAudioFormatLinearPCM; + output_format.mReserved = 0; + output_format.mSampleRate = input_format->mSampleRate; + + if (audio_float && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_float && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F64BE : AV_CODEC_ID_PCM_F64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_signed_integer && + audio_bits_per_sample == 8) { + stream->codecpar->codec_id = AV_CODEC_ID_PCM_S8; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 16) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 24) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S64BE : AV_CODEC_ID_PCM_S64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mBitsPerChannel = 32; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + must_convert = 1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { + if (must_convert) { + OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); + if (ret != noErr) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); + av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } @@ -1048,6 +1084,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1091,7 +1128,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1105,82 +1142,83 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t input_size = CMBlockBufferGetDataLength(block_buffer); + int buffer_size = input_size / ctx->audio_buffers; + int nb_samples = input_size / (ctx->audio_channels * ctx->bytes_per_sample); + int output_size = buffer_size; - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { + UInt32 size = sizeof(output_size); + ret = AudioConverterGetProperty(ctx->audio_converter, kAudioConverterPropertyCalculateOutputBufferSize, &size, &output_size); + if (ret != noErr) { unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + return AVERROR(EIO); } - if (av_new_packet(pkt, block_buffer_size) < 0) { + if (av_new_packet(pkt, output_size) < 0) { unlock_frames(ctx); return AVERROR(EIO); } - CMItemCount count; - CMSampleTimingInfo timing_info; + if (ctx->audio_converter) { + size_t input_buffer_size = offsetof(AudioBufferList, mBuffers[0]) + (sizeof(AudioBuffer) * ctx->audio_buffers); + AudioBufferList *input_buffer = av_malloc(input_buffer_size); - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { - AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); - pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); - } + input_buffer->mNumberBuffers = ctx->audio_buffers; - pkt->stream_index = ctx->audio_stream_index; - pkt->flags |= AV_PKT_FLAG_KEY; + for (int c = 0; c < ctx->audio_buffers; c++) { + input_buffer->mBuffers[c].mNumberChannels = 1; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; + ret = CMBlockBufferGetDataPointer(block_buffer, c * buffer_size, (size_t *)&input_buffer->mBuffers[c].mDataByteSize, NULL, (void *)&input_buffer->mBuffers[c].mData); - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); + if (ret != kCMBlockBufferNoErr) { + av_free(input_buffer); + unlock_frames(ctx); + return AVERROR(EIO); + } } - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } + AudioBufferList output_buffer = { + .mNumberBuffers = 1, + .mBuffers[0] = { + .mNumberChannels = ctx->audio_channels, + .mDataByteSize = pkt->size, + .mData = pkt->data + } + }; - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { + ret = AudioConverterConvertComplexBuffer(ctx->audio_converter, nb_samples, input_buffer, &output_buffer); + av_free(input_buffer); + + if (ret != noErr) { unlock_frames(ctx); return AVERROR(EIO); } + + pkt->size = output_buffer.mBuffers[0].mDataByteSize; + } else { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } } + CMItemCount count; + CMSampleTimingInfo timing_info; + + if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); + pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); + } + + pkt->stream_index = ctx->audio_stream_index; + pkt->flags |= AV_PKT_FLAG_KEY; + CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 0cd6e646d5..79c9207cfa 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -111,16 +111,10 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; + UInt32 audio_buffers; + UInt32 audio_channels; + UInt32 bytes_per_sample; + AudioConverterRef audio_converter; enum AVPixelFormat pixel_format; @@ -299,7 +293,10 @@ static void destroy_context(AVFContext* ctx) ctx->avf_delegate = NULL; ctx->avf_audio_delegate = NULL; - av_freep(&ctx->audio_buffer); From patchwork Fri Dec 17 15:12:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32687 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1659380iog; Fri, 17 Dec 2021 07:13:16 -0800 (PST) X-Google-Smtp-Source: ABdhPJx71aKUgGrHsn1Bm++2By7fGtm8f+bqejQArqQPWvkX7c3tN7s9Xpt+A4pa021Z8osFvnFt X-Received: by 2002:a17:906:f8d9:: with SMTP id lh25mr2764305ejb.631.1639753996386; Fri, 17 Dec 2021 07:13:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639753996; cv=none; d=google.com; s=arc-20160816; b=ShG3lIkl5439TRbosgvUa431IdCdIfx+95LJ0egVuekBuQ7rBuAfoa2/D+1yBHImmO HCD+MRwH5CRolEerEaPwC5Y9E/AefVVTip+ZwZ/BVr9/E2Vo/HHG0wmiKmSe5vYqeW+S sf+ukyCdPZe5g3G9M9gx/uP4BwfAdHWit/QnKNxtWxaOr1PDo205/yQaA1vT1KbG7usC tnT4obh3FPnDgrthsXGb/6MQo8jfnhB9GBBmVHQPT6e7ucwTGxw8QNHFKZwUQmnZsYc8 K3XC0dJ4jJnZ/ZixvIVUjljs1tiQur7Aw6wLVHQCPdnvxvjIIMSEfDOdqnWokxxlGc4p rUMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:content-language:to:from:user-agent:mime-version :date:message-id:delivered-to; bh=UOKL7dT5Sf10eJNapwWTEJ+cNa1cpsY5Q3r123gm9Jw=; b=l3uJBclWgjcDo70gRwU25zVxCfSnYl1Ss/00+dZmjl8SZ68r4G4U1kCwktSF9lJcIL HrNSIn+i5N7aH6HPTGgnawLdzD0FTVIv1zo1fhdtrzJnrz0Z99iY2dmASmtc9L9eFozk 0M6P7OUc24Y/mG+eZXp5XJYE+d85tJjwTSW/MSaZj7RWeJ4o+Z1KJKq9UWY3vI+SjH3t ljFcug6+ZGm2eGois6wCpPr6+9zJzrP1vDArSZT69LhfeeaXfxKJygKl/H2mn5sQb7VB vsIG0KVG3i8qsoEc13zCybv7Kxj+NTkp9BH4EfleHOUtR8Y6wd9M1CgBp7DqpaEKXdka fXjw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d15si5276562edk.568.2021.12.17.07.12.59; Fri, 17 Dec 2021 07:13:16 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 93BA068A9D3; Fri, 17 Dec 2021 17:12:50 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8D55568AE6B for ; Fri, 17 Dec 2021 17:12:44 +0200 (EET) Received: by mail-qv1-f42.google.com with SMTP id fo11so2653144qvb.4 for ; Fri, 17 Dec 2021 07:12:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:from :subject:to:content-language:content-transfer-encoding; bh=aHOS1sa/ayxaUct7bY3FtChlEOg1GNeWvisSRaEWJXU=; b=iZtceNRyFNZF/qLe6ler+kk5jIT0++sdnTmmKESkPa7e+CasBr8FF0cPRZHNp07GtS lzthWRH1s+Bv5qjs+KoNJLzQ+JC/op9k/Phc97cO0IMnEsbzFwfTmJRtQ8UhnnjZhZ5j WkvnRH1iZFgRf+lJsXoJoZtftbAYoUq8rzqxDVKy22onA3ew3/OZI2YIwAEdYvEEoNls beVxhu5hA+MPI+XXG9T5rjy/PMwDE9paTzf9akBRmp46ip8IAhLYZz7L7yfAPIZ2nOT9 axzsJID88Rxf7SE0G8bREgDpE+vVr1xBMsXzKBkXshiqQkQZfSi1ZahM5UiF6dmemAak 6jKA== X-Gm-Message-State: AOAM533sPxuYSmz/j3IPtqM7cvVlT1u0yVu92AVVmPOP5NcSkZHjMWVf R8ZPmijF5FZQmmoRF1lSDkvRRPR4ujwsGQ== X-Received: by 2002:ad4:5bca:: with SMTP id t10mr2623829qvt.35.1639753962541; Fri, 17 Dec 2021 07:12:42 -0800 (PST) Received: from [192.168.86.89] ([172.58.6.224]) by smtp.gmail.com with ESMTPSA id bs16sm4864199qkb.45.2021.12.17.07.12.41 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 17 Dec 2021 07:12:42 -0800 (PST) Message-ID: Date: Fri, 17 Dec 2021 09:12:40 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 From: Romain Beauxis To: ffmpeg-devel@ffmpeg.org Content-Language: en-US Subject: [FFmpeg-devel] [PATCH v4 2/3] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: sPoBEiPPv29X This is the second patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes: v2: None v3: Switched queue implementation to CMSimpleQueue v4: None This patch fixes the concurrency model. Avfoundation runs its own producing thread to send produced frames and ffmpeg runs its own thread to consume them. The existing implementation stores the last transmitted frame and uses a mutex to avoid concurrent access. However, this leads to situations where upcoming frames can be dropped if the ffmpeg thread is acessing the latest frame. This happens even when the thread would otherwise catch up and process frames fast enought. This patches changes this implementation to use a buffer queue with a max queue length and encapsulated thread-safety. This greatly simplifies the logic of the calling code and gives the consuming thread a chance to process all frames concurrently to the producing thread while avoiding memory leaks. Signed-off-by: Romain Beauxis --- libavdevice/avfoundation.m | 169 +++++++++++++++++-------------------- 1 file changed, 76 insertions(+), 93 deletions(-) }; +#define MAX_QUEUED_FRAMES 10 + typedef struct { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -121,8 +120,8 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + CMSimpleQueueRef audio_frames_queue; + CMSimpleQueueRef video_frames_queue; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -131,16 +130,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -218,17 +207,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); - } - - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + CFRetain(videoFrame); + CMSimpleQueueEnqueue(_context->video_frames_queue, videoFrame); } @end @@ -262,17 +242,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); - } - - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + CFRetain(audioFrame); + CMSimpleQueueEnqueue(_context->audio_frames_queue, audioFrame); } @end @@ -287,6 +258,30 @@ static void destroy_context(AVFContext* ctx) [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; + CMSampleBufferRef frame; + + if (ctx->video_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + do { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + } while (frame); + + CFRelease(ctx->video_frames_queue); + ctx->video_frames_queue = NULL; + } + + if (ctx->audio_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + do { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + } while (frame); + + CFRelease(ctx->audio_frames_queue); + ctx->audio_frames_queue = NULL; + } + ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; @@ -297,12 +292,6 @@ static void destroy_context(AVFContext* ctx) AudioConverterDispose(ctx->audio_converter); ctx->audio_converter = NULL; } - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static void parse_device_name(AVFormatContext *s) @@ -630,18 +619,18 @@ static int get_video_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->video_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer) { image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); @@ -657,10 +646,7 @@ static int get_video_config(AVFormatContext *s) stream->codecpar->format = ctx->pixel_format; } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -680,27 +666,27 @@ static int get_audio_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->audio_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); ctx->audio_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); + format_desc = CMSampleBufferGetFormatDescription(frame); const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); if (!input_format) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } if (input_format->mFormatID != kAudioFormatLinearPCM) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); return 1; } @@ -778,16 +764,13 @@ static int get_audio_config(AVFormatContext *s) if (must_convert) { OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -805,8 +788,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1007,6 +988,19 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + OSStatus ret; + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->video_frames_queue); + + if (ret != noErr) { + goto fail; + } + + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->audio_frames_queue); + + if (ret != noErr) { + goto fail; + } + if (video_device && add_video_device(s, video_device)) { goto fail; } @@ -1088,35 +1082,31 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) AVFContext* ctx = (AVFContext*)s->priv_data; do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); - - if (ctx->current_frame != nil) { + if (1 <= CMSimpleQueueGetCount(ctx->video_frames_queue)) { int status; int length = 0; - - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + CMSampleBufferRef video_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + CVImageBufferRef image_buffer = CMSampleBufferGetImageBuffer(video_frame);; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(video_frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EINVAL); } if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EIO); } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(video_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1133,15 +1123,14 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = AVERROR(EIO); } } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; + CFRelease(video_frame); if (status < 0) { - unlock_frames(ctx); return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + } else if (1 <= CMSimpleQueueGetCount(ctx->audio_frames_queue)) { + CMSampleBufferRef audio_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(audio_frame); size_t input_size = CMBlockBufferGetDataLength(block_buffer); int buffer_size = input_size / ctx->audio_buffers; @@ -1151,12 +1140,12 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) UInt32 size = sizeof(output_size); ret = AudioConverterGetProperty(ctx->audio_converter, kAudioConverterPropertyCalculateOutputBufferSize, &size, &output_size); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } if (av_new_packet(pkt, output_size) < 0) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1173,7 +1162,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) if (ret != kCMBlockBufferNoErr) { av_free(input_buffer); - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1191,7 +1180,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) av_free(input_buffer); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1199,7 +1188,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } else { ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1207,7 +1196,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(audio_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1215,21 +1204,15 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(audio_frame); } else { pkt->data = NULL; - unlock_frames(ctx); if (ctx->observed_quit) { return AVERROR_EOF; } else { return AVERROR(EAGAIN); } } - - unlock_frames(ctx); } while (!pkt->data); return 0; diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 79c9207cfa..b602cfbe95 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,7 +26,7 @@ */ #import -#include +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -80,13 +80,12 @@ { AV_PIX_FMT_NONE, 0 } From patchwork Fri Dec 17 15:12:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32686 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1659289iog; Fri, 17 Dec 2021 07:13:12 -0800 (PST) X-Google-Smtp-Source: ABdhPJwizDXVPF2miTjp8hQeCPQgj+TNXTzGWq3mhAx00CGnjfd4RA5sIsYtP2qyLaQFKvEs3yBq X-Received: by 2002:a05:6402:11c9:: with SMTP id j9mr3218270edw.346.1639753992375; Fri, 17 Dec 2021 07:13:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639753992; cv=none; d=google.com; s=arc-20160816; b=ffDo2PUKam1xNl455uwMDrrFOjUxaxsyCl15uqNYkHCBl9JHZoMSv8L4DWwTarTg6m DncHRDclrkvXGRlSAla1p7ed8TNFsUBYFfisGU07UPVdGXH7awGEtPl9wMyw+dq0W4UN 0e2ZnXn2JtIi7PAlAfRIUMfqnYhwy27VHhjbTOQ3QfQe6KeKwRzEntlJcydVrq4X+nce zIQ7eBIGxyY6aZD1/8eXQVJL/RMNaD3hRTSr4UWHuQvU7HFIh8ujh070edNfGC4Zzqqb pvOXvJ3kdB/lHjEPe8KVOBk3ynnDX3h64ogSVMQNrAsRm9zmHwOVpeHv9SowDJy0xLSb WZmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:content-language:to:from:user-agent:mime-version :date:message-id:delivered-to; bh=ZorCf9Iv158j4nf2FhKs68Tr5jFcACxu4IvtuQVafmE=; b=IA5Ru5Q2/5VODn1bQKlCB6eEzDQ0sqRDBGyUkNixvUyliuWd7G2dV/Z+VTuQGlkxFX MUAO7yWn/360bbvLM5/06VFwKl5AaAtoJ7w7DLDzSej6JG2O/BfwSH0Xceun68+mqf5N amkvF7sJ4Lxt3mzJtlYOSDkB+jVHKmGNiJNvFqQnfqSMqKL13NUmDXlFLTInlA74Nu0M Ww9nd74XDNgtMNbErAKxB8H7cYDZkfam+A7visbjGWZ8cA1TueNPmxl6JghMxFhaL69q 1TbNGstOqJIoVjNkie6zDpCRhB/EDXNh5pIdBYFC2KivhpajzDVV0eZOM600NhxaSf7/ wNeA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id go35si3896058ejc.864.2021.12.17.07.13.12; Fri, 17 Dec 2021 07:13:12 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9893668AED0; Fri, 17 Dec 2021 17:12:59 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 98BA868AE9A for ; Fri, 17 Dec 2021 17:12:53 +0200 (EET) Received: by mail-qv1-f54.google.com with SMTP id eq6so2620614qvb.7 for ; Fri, 17 Dec 2021 07:12:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:from :subject:to:content-language:content-transfer-encoding; bh=jM4/SgGAoDgw1RjXOXyBlXR51eQ/i9PpU83Nc5pU+fY=; b=XRlTUU8oOkVH+FB79wmHKoDpZK7GaErQHIkmDib+QvN0ar20ZJPyxSBkCLZRBxDk29 JIl+XONsTVvZD7H+AzUpuhoBDtn4rmsW8hcIUycVlKB2z9p5Nfsvc73D3Vcq0ugwfW0w qeK9EweDNpnG+rFekmeCHo0wqsysbABZM4MRA+IWAubQRs/VNIa6LdFxkN266hASRd24 1VilBSQU+0a1UtOJva+H1zksRxg8GZqo0rb5qPoKkUnCUTgdOenfWrClnvPf96cViYVp BvXidoW2L7eX9m/ckK7dWf8WBn4GvCHiJ3PZsPxaQEi5uT3yMBS9Is13nEgIFt3mF2fd W4Gg== X-Gm-Message-State: AOAM531cXDEGW1QVUTCO5oIzcCibuPaOHBBFmeYJ29hWtVidgehuwjJ6 NZstSSWP6Xj25962gObapGcvUbO/7QUWtg== X-Received: by 2002:a05:6214:f29:: with SMTP id iw9mr2712800qvb.37.1639753971818; Fri, 17 Dec 2021 07:12:51 -0800 (PST) Received: from [192.168.86.89] ([172.58.6.224]) by smtp.gmail.com with ESMTPSA id b5sm4679688qka.51.2021.12.17.07.12.51 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 17 Dec 2021 07:12:51 -0800 (PST) Message-ID: Date: Fri, 17 Dec 2021 09:12:49 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 From: Romain Beauxis To: ffmpeg-devel@ffmpeg.org Content-Language: en-US Subject: [FFmpeg-devel] [PATCH v4 3/3] libavdevice/avfoundation.m: Allow to select devices by unique ID. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: m//TcYm1hgqL This is the third patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes: v2: None v3: * Switched unique ID to use system-prodvided unique ID * Implemented unique IDs for screen capture v4: Cleanup This patch adds a unique ID to avfoundation devices. This is needed because device index can change while the machine is running when devices are plugged or unplugged and device names can be tricky to use with localization and etc. Example of output: ./ffmpeg -f avfoundation -list_devices true -i "" [...] [AVFoundation indev @ 0x158705230] AVFoundation video devices: [AVFoundation indev @ 0x158705230] [0] FaceTime HD Camera (ID: 47B4B64B70674B9CAD2BAE273A71F4B5) [AVFoundation indev @ 0x158705230] [1] Capture screen 0 (ID: AvfilterAvfoundationCaptureScreen1) [AVFoundation indev @ 0x158705230] AVFoundation audio devices: [AVFoundation indev @ 0x158705230] [0] Loopback Audio (ID: com.rogueamoeba.Loopback.A5668B36-711E-4DF5-8A8D-7148508C735B) [AVFoundation indev @ 0x158705230] [1] MacBook Pro Microphone (ID: BuiltInMicrophoneDevice) Notes: * Unique names do not seem to follow any specific pattern. I have used one similar to the builtin microphone for screen capture * The : substitution is actually required. The loopback device above did have it in its name. Signed-off-by: Romain Beauxis --- doc/indevs.texi | 6 ++-- libavdevice/avfoundation.m | 72 +++++++++++++++++++++++++++++--------- 2 files changed, 60 insertions(+), 18 deletions(-) int index = 0; av_log(ctx, AV_LOG_INFO, "AVFoundation video devices:\n"); for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } for (AVCaptureDevice *device in devices_muxed) { - const char *name = [[device localizedName] UTF8String]; - index = [devices count] + [devices_muxed indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices count] + [devices_muxed indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 if (num_screens > 0) { CGDirectDisplayID screens[num_screens]; CGGetActiveDisplayList(num_screens, screens, &num_screens); for (int i = 0; i < num_screens; i++) { - av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d\n", ctx->num_video_devices + i, i); + av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d (ID: AvfilterAvfoundationCaptureScreen%d)\n", ctx->num_video_devices + i, i, screens[i]); } } #endif @@ -819,9 +823,10 @@ static int avf_read_header(AVFormatContext *s) av_log(ctx, AV_LOG_INFO, "AVFoundation audio devices:\n"); devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - int index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + int index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } goto fail; } @@ -883,14 +888,29 @@ static int avf_read_header(AVFormatContext *s) } else { // looking for video inputs for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; break; } } // looking for muxed inputs for (AVCaptureDevice *device in devices_muxed) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + ctx->video_is_muxed = 1; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; ctx->video_is_muxed = 1; break; @@ -901,10 +921,23 @@ static int avf_read_header(AVFormatContext *s) // looking for screen inputs if (!video_device) { int idx; + CGDirectDisplayID screens[num_screens]; + CGGetActiveDisplayList(num_screens, screens, &num_screens); + AVCaptureScreenInput* capture_screen_input = NULL; + if(sscanf(ctx->video_filename, "Capture screen %d", &idx) && idx < num_screens) { - CGDirectDisplayID screens[num_screens]; - CGGetActiveDisplayList(num_screens, screens, &num_screens); - AVCaptureScreenInput* capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + } + + if(sscanf(ctx->video_filename, "AvfilterAvfoundationCaptureScreen%d", &idx)) { + for (int i = 0; i < num_screens; i++) { + if (screens[i] == idx) { + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:idx] autorelease]; + } + } + } + + if (capture_screen_input) { video_device = (AVCaptureDevice*) capture_screen_input; ctx->video_device_index = ctx->num_video_devices + idx; ctx->video_is_screen = 1; @@ -955,7 +988,14 @@ static int avf_read_header(AVFormatContext *s) NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->audio_filename, [[device localizedName] UTF8String], strlen(ctx->audio_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->audio_filename, name, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->audio_filename, uniqueId, strlen(ctx->audio_filename))) { audio_device = device; break; } diff --git a/doc/indevs.texi b/doc/indevs.texi index 5be647f70a..2b55399c8c 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -114,7 +114,7 @@ The input filename has to be given in the following syntax: -i "[[VIDEO]:[AUDIO]]" @end example The first entry selects the video input while the latter selects the audio input. -The stream has to be specified by the device name or the device index as shown by the device list. +The stream has to be specified by the device name, index or ID as shown by the device list. Alternatively, the video and/or audio input device can be chosen by index using the @option{ -video_device_index @@ -127,7 +127,9 @@ and/or device name or index given in the input filename. All available devices can be enumerated by using @option{-list_devices true}, listing -all device names and corresponding indices. +all device names, corresponding indices and IDs, when available. Device name can be +tricky to use when localized and device index can change when devices are plugged or unplugged. A device +hash, when available, uniquely identifies a device and should not change over time. There are two device name aliases: @table @code diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index b602cfbe95..25286507d6 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -39,6 +39,8 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +#define CLEANUP_DEVICE_ID(s) [[s stringByReplacingOccurrencesOfString:@":" withString:@"."] UTF8String] + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -797,21 +799,23 @@ static int avf_read_header(AVFormatContext *s)