From patchwork Wed Dec 15 00:15:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32505 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp7677697iog; Tue, 14 Dec 2021 16:16:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJwim7V/srysDWOFXEIi8LibP9+6qDhX1890S19hpP7DwKOpslS9dNVfXsxp535ErmporGJj X-Received: by 2002:a17:907:86a6:: with SMTP id qa38mr8903456ejc.286.1639527363875; Tue, 14 Dec 2021 16:16:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639527363; cv=none; d=google.com; s=arc-20160816; b=T4cnH3RSVsrsPxsOi05WM6PechI6AMmWOTIyJuaxoaxRUR7p2dzK56gDqCh7qLqNR0 0orxIcQl1yLY0jSSlIRFjpiK6hgzc+BH4LqIWRhz9f+DX8jGYiVPceeIWcH/Ix4r4Wkd YWADXNbvcdWAoQ1Cm5MDh8FE/iM2A1+rd09BW0WPxGtx0GRUoJPLSaXNi2dgaALFqR6I GS4VrLrGMILPLUyswTJFnAmb2xmO2cHcyxPGmkA4Rkc/cqbyQFMhdLOS51LXZZjCvqFt 36nBh4ZEz43x7DNYIpJzBg/qMTZVT/qXhNHKrw5O1hVREE8/6Fzt97VLEu88RJppjjuM IBCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=68Q1ZQxFKtc9Ejv+CJyU1F1N89aLj0YJXFgnWXgkKf0=; b=uypFNqjCBDH9KZBe+0B9YaDvgEYbGCFbS0xGfoCTuA18Y8JWn5uSB1DUwXdgYPEnyE AzSSXwuk/uHzaYmYJbXS7rHdmvTDFz3Q92pxsJagbe76e31q3a2kailDqzY5k4MR8sF2 hLaLhppMtFfkDAwUgsyLpODCi2xOeFj6OhhqIKoMfoC42nmgM6G7/Og3mF5/tHhc+pmQ GQwiE32Xiq9DxIWSQxSBxfmO395YjcHn8je5ya/By8BCEaLFWlKFwgEmciv+0JWOWsIr sIPSyuNrI/++sf2OfvMPn8apPK65h6ye09GDeP4pHyz3Qp2965KvUL56IAnG7qTgZiEJ g6Mg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id sg38si160689ejc.17.2021.12.14.16.16.02; Tue, 14 Dec 2021 16:16:03 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4937E68AEAE; Wed, 15 Dec 2021 02:15:59 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 774B568AEAE for ; Wed, 15 Dec 2021 02:15:52 +0200 (EET) Received: by mail-qt1-f170.google.com with SMTP id f20so20187478qtb.4 for ; Tue, 14 Dec 2021 16:15:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=frfnS07ZQ0saHIiqtdm57xLlxgThxiC9ckGE9hXmOiU=; b=ULYdgS+JLc4yEv9whn+YFPE67apWkDDYoBwFVMJMMXd+/OoTrnSpeaXoonXahLLNb1 IVGCfx8xnFe3BdWlwVLULjMPYQkPfoW2xAegHkSAtS/OKfhN0H+DdKtt7+omFY8DQ/B6 s2U+QXyZa1XAsA//dbE1zoATruQ3NjhqVz6sAmg2ICpLFVX0TFh6zl7qZwGiJLfz98Yj I/wA1VzmAXhzW8+YwjJJpXbNt0FcsiO8uPauef0/emJvlC8A256y89iagcTXFrenB1kl l3y/cK6CXiP6bnFlj0/XFqzlrmhhfJjWUc/8ivHnUeMHovLu6MmiwHNhZKfml+064rJL 1ONg== X-Gm-Message-State: AOAM530BKPLfqE1hcS11HaJCnq9wEGcMaFUaWdwaq6KgWW5zrRLE110P fNVPD+uu5B8hKQpi7RsVdcdxxbrmhsA= X-Received: by 2002:ac8:4e8c:: with SMTP id 12mr9543142qtp.45.1639527350454; Tue, 14 Dec 2021 16:15:50 -0800 (PST) Received: from smtpclient.apple ([2607:fb90:25d0:a095:84ff:682b:30ce:d185]) by smtp.gmail.com with ESMTPSA id t9sm183259qkp.110.2021.12.14.16.15.48 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Dec 2021 16:15:49 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: <7017D463-6BC2-4222-AC8C-4F62F4F8E613@rastageeks.org> Date: Tue, 14 Dec 2021 18:15:46 -0600 To: FFmpeg development discussions and patches X-Mailer: Apple Mail (2.3654.100.0.2.22) Subject: [FFmpeg-devel] [PATCH v3 01/03] libavdevice/avfoundation.m: use AudioConvert, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: S94oYftU1oXy This is the first patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes since previous version: none This patch: * Implements support for AudioConverter * Switches to AudioConverter's API to convert unsupported PCM formats (non-interleaved, non-packed) to supported formats * Minimizes data copy. This fixes: https://trac.ffmpeg.org/ticket/9502 API ref: https://developer.apple.com/documentation/audiotoolbox/audio_converter_services Signed-off-by: Romain Beauxis < toots at rastageeks.org> --- libavdevice/avfoundation.m | 250 +++++++++++++++++++++---------------- 1 file changed, 144 insertions(+), 106 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 0cd6e646d5..79c9207cfa 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -111,16 +111,10 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; + UInt32 audio_buffers; + UInt32 audio_channels; + UInt32 bytes_per_sample; + AudioConverterRef audio_converter; enum AVPixelFormat pixel_format; @@ -299,7 +293,10 @@ static void destroy_context(AVFContext* ctx) ctx->avf_delegate = NULL; ctx->avf_audio_delegate = NULL; - av_freep(&ctx->audio_buffer); + if (ctx->audio_converter) { + AudioConverterDispose(ctx->audio_converter); + ctx->audio_converter = NULL; + } pthread_mutex_destroy(&ctx->frame_lock); @@ -673,6 +670,10 @@ static int get_audio_config(AVFormatContext *s) AVFContext *ctx = (AVFContext*)s->priv_data; CMFormatDescriptionRef format_desc; AVStream* stream = avformat_new_stream(s, NULL); + AudioStreamBasicDescription output_format = {0}; + int audio_bits_per_sample, audio_float, audio_be; + int audio_signed_integer, audio_packed, audio_non_interleaved; + int must_convert = 0; if (!stream) { return 1; @@ -690,60 +691,95 @@ static int get_audio_config(AVFormatContext *s) avpriv_set_pts_info(stream, 64, 1, avf_time_base); format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); + const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - if (!basic_desc) { + if (!input_format) { unlock_frames(ctx); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } + if (input_format->mFormatID != kAudioFormatLinearPCM) { + unlock_frames(ctx); + av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); + return 1; + } + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; + stream->codecpar->sample_rate = input_format->mSampleRate; + stream->codecpar->channels = input_format->mChannelsPerFrame; stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + audio_bits_per_sample = input_format->mBitsPerChannel; + audio_float = input_format->mFormatFlags & kAudioFormatFlagIsFloat; + audio_be = input_format->mFormatFlags & kAudioFormatFlagIsBigEndian; + audio_signed_integer = input_format->mFormatFlags & kAudioFormatFlagIsSignedInteger; + audio_packed = input_format->mFormatFlags & kAudioFormatFlagIsPacked; + audio_non_interleaved = input_format->mFormatFlags & kAudioFormatFlagIsNonInterleaved; + + ctx->bytes_per_sample = input_format->mBitsPerChannel >> 3; + ctx->audio_channels = input_format->mChannelsPerFrame; + + if (audio_non_interleaved) { + ctx->audio_buffers = input_format->mChannelsPerFrame; } else { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + ctx->audio_buffers = 1; + } + + if (audio_non_interleaved || !audio_packed) { + must_convert = 1; + } + + output_format.mBitsPerChannel = input_format->mBitsPerChannel; + output_format.mChannelsPerFrame = ctx->audio_channels; + output_format.mFramesPerPacket = 1; + output_format.mBytesPerFrame = output_format.mChannelsPerFrame * ctx->bytes_per_sample; + output_format.mBytesPerPacket = output_format.mFramesPerPacket * output_format.mBytesPerFrame; + output_format.mFormatFlags = kAudioFormatFlagIsPacked | audio_be; + output_format.mFormatID = kAudioFormatLinearPCM; + output_format.mReserved = 0; + output_format.mSampleRate = input_format->mSampleRate; + + if (audio_float && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_float && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F64BE : AV_CODEC_ID_PCM_F64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_signed_integer && + audio_bits_per_sample == 8) { + stream->codecpar->codec_id = AV_CODEC_ID_PCM_S8; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 16) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 24) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S64BE : AV_CODEC_ID_PCM_S64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mBitsPerChannel = 32; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + must_convert = 1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { + if (must_convert) { + OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); + if (ret != noErr) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); + av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } @@ -1048,6 +1084,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1091,7 +1128,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1105,82 +1142,83 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t input_size = CMBlockBufferGetDataLength(block_buffer); + int buffer_size = input_size / ctx->audio_buffers; + int nb_samples = input_size / (ctx->audio_channels * ctx->bytes_per_sample); + int output_size = buffer_size; - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { + UInt32 size = sizeof(output_size); + ret = AudioConverterGetProperty(ctx->audio_converter, kAudioConverterPropertyCalculateOutputBufferSize, &size, &output_size); + if (ret != noErr) { unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + return AVERROR(EIO); } - if (av_new_packet(pkt, block_buffer_size) < 0) { + if (av_new_packet(pkt, output_size) < 0) { unlock_frames(ctx); return AVERROR(EIO); } - CMItemCount count; - CMSampleTimingInfo timing_info; + if (ctx->audio_converter) { + size_t input_buffer_size = offsetof(AudioBufferList, mBuffers[0]) + (sizeof(AudioBuffer) * ctx->audio_buffers); + AudioBufferList *input_buffer = av_malloc(input_buffer_size); - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { - AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); - pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); - } + input_buffer->mNumberBuffers = ctx->audio_buffers; - pkt->stream_index = ctx->audio_stream_index; - pkt->flags |= AV_PKT_FLAG_KEY; + for (int c = 0; c < ctx->audio_buffers; c++) { + input_buffer->mBuffers[c].mNumberChannels = 1; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; + ret = CMBlockBufferGetDataPointer(block_buffer, c * buffer_size, (size_t *)&input_buffer->mBuffers[c].mDataByteSize, NULL, (void *)&input_buffer->mBuffers[c].mData); - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); + if (ret != kCMBlockBufferNoErr) { + av_free(input_buffer); + unlock_frames(ctx); + return AVERROR(EIO); + } } - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } + AudioBufferList output_buffer = { + .mNumberBuffers = 1, + .mBuffers[0] = { + .mNumberChannels = ctx->audio_channels, + .mDataByteSize = pkt->size, + .mData = pkt->data + } + }; - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { + ret = AudioConverterConvertComplexBuffer(ctx->audio_converter, nb_samples, input_buffer, &output_buffer); + av_free(input_buffer); + + if (ret != noErr) { unlock_frames(ctx); return AVERROR(EIO); } + + pkt->size = output_buffer.mBuffers[0].mDataByteSize; + } else { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } } + CMItemCount count; + CMSampleTimingInfo timing_info; + + if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); + pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); + } + + pkt->stream_index = ctx->audio_stream_index; + pkt->flags |= AV_PKT_FLAG_KEY; + CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); From patchwork Wed Dec 15 00:18:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32506 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp7679767iog; Tue, 14 Dec 2021 16:18:54 -0800 (PST) X-Google-Smtp-Source: ABdhPJxGWxq8IfzDW8mkAMX68xabyut1KoqaYQ6oNJwKePSupiEN/43Acal/pYhe2djUAO5E9CIi X-Received: by 2002:a05:6402:3596:: with SMTP id y22mr11956043edc.297.1639527533909; Tue, 14 Dec 2021 16:18:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639527533; cv=none; d=google.com; s=arc-20160816; b=voBnCdmDkOnkMUAvUGN+fgborr6AEV97Cf8vuXkgthl5RsFEHfQcroycablmA9wY3h c7rwkpKpqxucInEhC6GqWquCxEGu8BSaO1S0T2v+cCw1ij4w53fuKBZTtweJTppFr1l/ 6K1euxvoyua2Gxhn+rw455FevMsmYTGVc/j5T6wUhEODNw87bQP5FDK2vrLmUtS6kc5v OKmPjhA2w34h2A5JfdXVoYDLTMAW+WzZpx/d38vGuseWCu+yEs+Omg535+EutNNAMe/H vLHjj1rbDxvrnJBwMKTrEjF8gRFcZbwSgPPP5n0mcWwTs+XalC2DDDVhqh+S9sHXTobJ SMGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=Jl77sJdYQ/6nebLMc8esxKqaPJVGjyhWo77rFKizEEY=; b=su18s6tps675dS+emwAxPtyB7JPNUF1NSK716RBNf+uu8Jjc3GaWNWlNr0ve/tsxXl luChAP49U83Aem5BicV5MyCs+LTQPbVu9No8NVlfYQ8+r/f15LjBiwofYGe+nRC+/C8E mR997tUom3QmuSwskUlNfCDv1t/2wS9vgq2WTvzLHrPr2cgYgkUsyrrZjoXTx1C2tCYL mQVXVABSl6nz/bXeXOXNPF+8gLlvlIAmmVaBNRUo5qrZm3pxSxWQT1Gpz9tSQ5draQY+ MxjvCf87tz3AcBfOAqZ49uGLG/kcSERnA/OIDKLTIi0uVnTKppKEcUkJh9uEFUHnLnNU 9eOQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id sa10si150599ejc.689.2021.12.14.16.18.53; Tue, 14 Dec 2021 16:18:53 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 73DE968B007; Wed, 15 Dec 2021 02:18:50 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4A83268A03B for ; Wed, 15 Dec 2021 02:18:44 +0200 (EET) Received: by mail-qt1-f174.google.com with SMTP id j17so20194015qtx.2 for ; Tue, 14 Dec 2021 16:18:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=0vx2f2sLn8ebv80nAUPutMMQ7h9oVBR3uhPFkvgzjcI=; b=cSzUcuhclkza5h/O987sFe3x/1XEyowdl4kELJQBIH1PqrOT2kX1B5HOS5fS8h7YN7 EbJQRea6SmPtqKJpDjmNpv1RYv8Nj4kLMJvtQx3HvSj5ZhSfeiMXlvcnTOaCVJalQTkp SaO5yYQBxeCiaYdc7fvRVVSFS+wV0KZkX3YIc4ph3O5k8z8VB8ZK8fcyqSA6pnPyTnYE cTLqKgbxSzHJZhtuPz43raRAEEBCvHHPZ0kPSJOcz1REhdKWBpwhKidW1tmEbBIF25B7 j4v9tFNFPEr+Y3+hPBOENw7KEXLTT7F4pk2atYv9s/5uEHrLQK9mpMN0+iFuzUJ1JIwd lZlw== X-Gm-Message-State: AOAM5339hMKaqDif0SeWJwFj8uGh4VrdIHFqstWKgjPrGHNUl/zgcYoj NE/qOtV1Mkf8nFwKaPHWsq0GJGIDfkc= X-Received: by 2002:ac8:5b01:: with SMTP id m1mr4664285qtw.313.1639527522561; Tue, 14 Dec 2021 16:18:42 -0800 (PST) Received: from smtpclient.apple ([2607:fb90:25d0:a095:84ff:682b:30ce:d185]) by smtp.gmail.com with ESMTPSA id z8sm309733qta.50.2021.12.14.16.18.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Dec 2021 16:18:42 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: Date: Tue, 14 Dec 2021 18:18:39 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3654.100.0.2.22) Subject: [FFmpeg-devel] [PATCH v3 02/03] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ++ODRVitnMvL This is the second patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes since last version: * Switched queue implementation to CMSimpleQueue This patch fixes the concurrency model. Avfoundation runs its own producing thread to send produced frames and ffmpeg runs its own thread to consume them. The existing implementation stores the last transmitted frame and uses a mutex to avoid concurrent access. However, this leads to situations where upcoming frames can be dropped if the ffmpeg thread is acessing the latest frame. This happens even when the thread would otherwise catch up and process frames fast enought. This patches changes this implementation to use a buffer queue with a max queue length and encapsulated thread-safety. This greatly simplifies the logic of the calling code and gives the consuming thread a chance to process all frames concurrently to the producing thread while avoiding memory leaks. Signed-off-by: Romain Beauxis --- libavdevice/avfoundation.m | 169 +++++++++++++++++-------------------- 1 file changed, 76 insertions(+), 93 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 79c9207cfa..b602cfbe95 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,7 +26,7 @@ */ #import -#include +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -80,13 +80,12 @@ { AV_PIX_FMT_NONE, 0 } }; +#define MAX_QUEUED_FRAMES 10 + typedef struct { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -121,8 +120,8 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + CMSimpleQueueRef audio_frames_queue; + CMSimpleQueueRef video_frames_queue; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -131,16 +130,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -218,17 +207,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); - } - - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + CFRetain(videoFrame); + CMSimpleQueueEnqueue(_context->video_frames_queue, videoFrame); } @end @@ -262,17 +242,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); - } - - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + CFRetain(audioFrame); + CMSimpleQueueEnqueue(_context->audio_frames_queue, audioFrame); } @end @@ -287,6 +258,30 @@ static void destroy_context(AVFContext* ctx) [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; + CMSampleBufferRef frame; + + if (ctx->video_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + do { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + } while (frame); + + CFRelease(ctx->video_frames_queue); + ctx->video_frames_queue = NULL; + } + + if (ctx->audio_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + do { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + } while (frame); + + CFRelease(ctx->audio_frames_queue); + ctx->audio_frames_queue = NULL; + } + ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; @@ -297,12 +292,6 @@ static void destroy_context(AVFContext* ctx) AudioConverterDispose(ctx->audio_converter); ctx->audio_converter = NULL; } - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static void parse_device_name(AVFormatContext *s) @@ -630,18 +619,18 @@ static int get_video_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->video_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer) { image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); @@ -657,10 +646,7 @@ static int get_video_config(AVFormatContext *s) stream->codecpar->format = ctx->pixel_format; } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -680,27 +666,27 @@ static int get_audio_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->audio_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); ctx->audio_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); + format_desc = CMSampleBufferGetFormatDescription(frame); const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); if (!input_format) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } if (input_format->mFormatID != kAudioFormatLinearPCM) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); return 1; } @@ -778,16 +764,13 @@ static int get_audio_config(AVFormatContext *s) if (must_convert) { OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -805,8 +788,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1007,6 +988,19 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + OSStatus ret; + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->video_frames_queue); + + if (ret != noErr) { + goto fail; + } + + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->audio_frames_queue); + + if (ret != noErr) { + goto fail; + } + if (video_device && add_video_device(s, video_device)) { goto fail; } @@ -1088,35 +1082,31 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) AVFContext* ctx = (AVFContext*)s->priv_data; do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); - - if (ctx->current_frame != nil) { + if (1 <= CMSimpleQueueGetCount(ctx->video_frames_queue)) { int status; int length = 0; - - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + CMSampleBufferRef video_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + CVImageBufferRef image_buffer = CMSampleBufferGetImageBuffer(video_frame);; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(video_frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EINVAL); } if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EIO); } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(video_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1133,15 +1123,14 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = AVERROR(EIO); } } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; + CFRelease(video_frame); if (status < 0) { - unlock_frames(ctx); return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + } else if (1 <= CMSimpleQueueGetCount(ctx->audio_frames_queue)) { + CMSampleBufferRef audio_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(audio_frame); size_t input_size = CMBlockBufferGetDataLength(block_buffer); int buffer_size = input_size / ctx->audio_buffers; @@ -1151,12 +1140,12 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) UInt32 size = sizeof(output_size); ret = AudioConverterGetProperty(ctx->audio_converter, kAudioConverterPropertyCalculateOutputBufferSize, &size, &output_size); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } if (av_new_packet(pkt, output_size) < 0) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1173,7 +1162,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) if (ret != kCMBlockBufferNoErr) { av_free(input_buffer); - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1191,7 +1180,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) av_free(input_buffer); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1199,7 +1188,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } else { ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1207,7 +1196,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(audio_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1215,21 +1204,15 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(audio_frame); } else { pkt->data = NULL; - unlock_frames(ctx); if (ctx->observed_quit) { return AVERROR_EOF; } else { return AVERROR(EAGAIN); } } - - unlock_frames(ctx); } while (!pkt->data); return 0; From patchwork Wed Dec 15 00:20:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32507 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp7680985iog; Tue, 14 Dec 2021 16:20:33 -0800 (PST) X-Google-Smtp-Source: ABdhPJyGG2B4IMJKx0ZrYuSoI0FcvZ4iwBjm6wruRMtwXhCKK6lsUmUmsbG+BmtPxtR0atsNH6mM X-Received: by 2002:a17:907:2d10:: with SMTP id gs16mr8759909ejc.353.1639527633718; Tue, 14 Dec 2021 16:20:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639527633; cv=none; d=google.com; s=arc-20160816; b=McEXtt/Vcx1WDd3MXiMsszamEwLk5mjsZpAUmWX2IW1Tx4RC43rN6wGiK0+1BmD+wy DFzJ70RuoET/eViTKWvivD3AU7oDTEAAw9OO0tSZ7z7wqA5pVt9P4GXfB8qMjkha4eXu y7YrFYmmBnmGtis1wHmdVMnCce1eJcJOXgMfjOaKic1jE2nHvB9mNYxwQKJLwlX9kIpz 9hFiDbqjLvpIINwrtPSsz+fa67GJvgFh0obBG7JG18jkKg5/vi/Qn49SzSFz3F6mpoJX dHw63Skn2I6Mq4oDkJiTbcJ2R+Zfl+65wqtCVVgPrL3NOQmkOx4kTaOz4RcHfQpTxa0c U/hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=vkayG1eLuMUI8Mc/jvLXeVkiOFpdbp6C37D9LxHCryU=; b=xoT1x5Jj2ItxqENatgKInUhdD1Rw8YzJdz3/bSW7onUPPx1M63jszyaZDgcfm//lUR J0hK6ZS88E+w0p9uZTqOQZLYDYJ1KGvQMn8LUSPxOtdEIkRwtki44Muv8TMC5isWRdvZ vHEYGuaDW8/1Z8gjB3A54Pt/xgwIozUqKRiKSgGAxpj0bpKG7ZMUCMhQgXfJymniJXWI O94V0QTXsbK1gZkYBMCa49kBNvf84Fy0yZ1lXQVJ4FMKx3JK1eAWUDsRo2ebPZqBA4xV Ia7DNPXQSSSzm/osVinsw6y5eNDp0EtckcFYpptYTRWzmswAr9gruzQBSDSjXS6sp7WV LEJA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cs21si168539ejc.275.2021.12.14.16.20.32; Tue, 14 Dec 2021 16:20:33 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 993BC68AFFF; Wed, 15 Dec 2021 02:20:30 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4E4F768AF55 for ; Wed, 15 Dec 2021 02:20:24 +0200 (EET) Received: by mail-qk1-f178.google.com with SMTP id p4so18469003qkm.7 for ; Tue, 14 Dec 2021 16:20:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=1GCKWQzZuNfrbcVAocl5uAyL2RvpNRjYzsVw3A2sNbQ=; b=WkIL0GtDgu66URZH61sqEkZVtkS2ldGoiDOw+AfbqBegIr4iL1JOBVh8Zh30776VIL Q4JnLqDTE3c6B2Yc2FNj9858F1xavteGAefzro6uAxZKhAGPgyzbKoKgBFrv1+Ig4q/G TnmAqKJXc9pO8SUdL+gM3U3i7Fk46oMH9lHSwNwHv5Dc8WHtzPY6yeyqysntf8oOZkzx EpGxMzaipHq181oPzo1Chq2YDrj329CAqYb3CpX0PlXxTmCoWN9rilMX0URU0KSiKZ+H kBwtrCFDph97x9l3JTSFYhBRt0JGC0Jwqmvx5ySjPxoAnvwehZj8I2pl+IpbWL4f/Uq+ Fc+w== X-Gm-Message-State: AOAM530ZkD4EvDom0LUt4fd9JimSkB7BdZ6oK3kq0OfAsWvnNpze8rfy G63R/OfoCTvxcEytU3WKsaDdHlFibAE= X-Received: by 2002:a05:620a:2995:: with SMTP id r21mr7098371qkp.100.1639527622091; Tue, 14 Dec 2021 16:20:22 -0800 (PST) Received: from smtpclient.apple ([2607:fb90:25d0:a095:84ff:682b:30ce:d185]) by smtp.gmail.com with ESMTPSA id bq43sm215352qkb.32.2021.12.14.16.20.21 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Dec 2021 16:20:21 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: <93813544-8D82-4F98-B7AD-001BF88EBB75@rastageeks.org> Date: Tue, 14 Dec 2021 18:20:19 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3654.100.0.2.22) Subject: [FFmpeg-devel] [PATCH v3 03/03] libavdevice/avfoundation.m: Allow to select devices by unique ID. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: YDFe/vgIfVus This is the third patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes since last version: * Switched unique ID to use system-prodvided unique ID * Implemented unique IDs for screen capture This patch adds a unique ID to avfoundation devices. This is needed because device index can change while the machine is running when devices are plugged or unplugged and device names can be tricky to use with localization and etc. Signed-off-by: Romain Beauxis --- doc/indevs.texi | 6 ++-- libavdevice/avfoundation.m | 73 +++++++++++++++++++++++++++++--------- 2 files changed, 61 insertions(+), 18 deletions(-) diff --git a/doc/indevs.texi b/doc/indevs.texi index 5be647f70a..2b55399c8c 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -114,7 +114,7 @@ The input filename has to be given in the following syntax: -i "[[VIDEO]:[AUDIO]]" @end example The first entry selects the video input while the latter selects the audio input. -The stream has to be specified by the device name or the device index as shown by the device list. +The stream has to be specified by the device name, index or ID as shown by the device list. Alternatively, the video and/or audio input device can be chosen by index using the @option{ -video_device_index @@ -127,7 +127,9 @@ and/or device name or index given in the input filename. All available devices can be enumerated by using @option{-list_devices true}, listing -all device names and corresponding indices. +all device names, corresponding indices and IDs, when available. Device name can be +tricky to use when localized and device index can change when devices are plugged or unplugged. A device +hash, when available, uniquely identifies a device and should not change over time. There are two device name aliases: @table @code diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index b602cfbe95..036fb57568 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -39,6 +39,8 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +#define CLEANUP_DEVICE_ID(s) [[s stringByReplacingOccurrencesOfString:@":" withString:@"."] UTF8String] + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -797,21 +799,23 @@ static int avf_read_header(AVFormatContext *s) int index = 0; av_log(ctx, AV_LOG_INFO, "AVFoundation video devices:\n"); for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } for (AVCaptureDevice *device in devices_muxed) { - const char *name = [[device localizedName] UTF8String]; - index = [devices count] + [devices_muxed indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices count] + [devices_muxed indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 if (num_screens > 0) { CGDirectDisplayID screens[num_screens]; CGGetActiveDisplayList(num_screens, screens, &num_screens); for (int i = 0; i < num_screens; i++) { - av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d\n", ctx->num_video_devices + i, i); + av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d (ID: AvfilterAvfoundationCptureScreen%d)\n", ctx->num_video_devices + i, i, screens[i]); } } #endif @@ -819,9 +823,10 @@ static int avf_read_header(AVFormatContext *s) av_log(ctx, AV_LOG_INFO, "AVFoundation audio devices:\n"); devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - int index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + int index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } goto fail; } @@ -883,14 +888,30 @@ static int avf_read_header(AVFormatContext *s) } else { // looking for video inputs for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); +printf("uniqueID: %s, filename: %s\n", uniqueId, ctx->video_filename); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; break; } } // looking for muxed inputs for (AVCaptureDevice *device in devices_muxed) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + ctx->video_is_muxed = 1; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; ctx->video_is_muxed = 1; break; @@ -901,10 +922,23 @@ static int avf_read_header(AVFormatContext *s) // looking for screen inputs if (!video_device) { int idx; + CGDirectDisplayID screens[num_screens]; + CGGetActiveDisplayList(num_screens, screens, &num_screens); + AVCaptureScreenInput* capture_screen_input = NULL; + if(sscanf(ctx->video_filename, "Capture screen %d", &idx) && idx < num_screens) { - CGDirectDisplayID screens[num_screens]; - CGGetActiveDisplayList(num_screens, screens, &num_screens); - AVCaptureScreenInput* capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + } + + if(sscanf(ctx->video_filename, "AvfilterAvfoundationCptureScreen%d", &idx)) { + for (int i = 0; i < num_screens; i++) { + if (screens[i] == idx) { + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:idx] autorelease]; + } + } + } + + if (capture_screen_input) { video_device = (AVCaptureDevice*) capture_screen_input; ctx->video_device_index = ctx->num_video_devices + idx; ctx->video_is_screen = 1; @@ -955,7 +989,14 @@ static int avf_read_header(AVFormatContext *s) NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->audio_filename, [[device localizedName] UTF8String], strlen(ctx->audio_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->audio_filename, name, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->audio_filename, uniqueId, strlen(ctx->audio_filename))) { audio_device = device; break; }