From patchwork Fri Dec 31 15:58:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32966 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp14069773iog; Fri, 31 Dec 2021 07:58:27 -0800 (PST) X-Google-Smtp-Source: ABdhPJxgAcFMyHai260Eo+vsHJFfBOSqBkV6qMP4Ebt8e5dhI6MY/U5qWSCdFcGK1cOIjHFdLrSW X-Received: by 2002:a17:907:76f4:: with SMTP id kg20mr28683605ejc.54.1640966307253; Fri, 31 Dec 2021 07:58:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640966307; cv=none; d=google.com; s=arc-20160816; b=QwSxGWq1LYncvQeJp0Ls7zjf0wCz6NSUb3UoHfMjlNRRthvw8UFSYuixKkTH3rWMfT 9pg+rNwUn9PNvWunl0YHShZrnYk64Uwfs6+0g8FdBwDKT8pFfgJ/IWB1Ze79AW9Bbphe 4iycEQk4b+SqFCVGSjbKizWtqu1wmiF+AwifXkwro3Nx9sDLXh/TB6kNSI9nINp98q3X nfdEE8yloxIjcY2XEai21Yzm63p+vXkkA0s4D8SzcL1JqYijTF7YCCQ5obhMjb4w/yPr o23TLsHPdfVBMHqpkcL5zefYV6VbBQ8qT7r2iJ+C7NecvmqrJre+AF7+TEavOEamjNSS iLmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=yUfHVHLvD1xI7HPfGqngeO0rBTFfAomhivPACVEYSI0=; b=dG9nkwTNmGGggVlh/kD3OHy2kOfvjF2nS/qG268PdC2V75caiSqgrocJTvMrmwTutU QmFGdAdjNfrDdRz9EE1MbcCIDqL3P+rJwItalFtDEiI2paS8KoZbu2/AFveTxVD+H27X fZnkTuUKkJtVKOILWjdCN4sfA4nfdOfZvqt0ko6jM+Obpi8tckt63QOWq8i6vzMR6Gz/ /8DNASQpff0fFBfMRCsLIgv5J/VavUvoUtN40fLL1FckDw1QYdVTTYTY45E6FKpTxaV8 lOvHRcm3lj0Pjhb3ZjZroMgEK6ELBHaiyDLMsdLRxrCQ/9T/zMIo+DbHvPnK9k8mtlFE 8gSA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w14si13205127edq.514.2021.12.31.07.58.26; Fri, 31 Dec 2021 07:58:27 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EDB1668B030; Fri, 31 Dec 2021 17:58:23 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 981EC6805AE for ; Fri, 31 Dec 2021 17:58:17 +0200 (EET) Received: by mail-qk1-f173.google.com with SMTP id t66so13942621qkb.4 for ; Fri, 31 Dec 2021 07:58:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=0D0tygtNBuNGBicJchRKNfvBcTyWUd0F0XFsofbgWCg=; b=AV2vMYWpPqI3T1sAaJoH4FlSI9z5zPF7HUTYtjzAT1jN1d6EY167IFvG4cD8HL+V9h B3cUeDnoiO87aGhJhgg9c7zZen0L06OLhXXxVbtCzxjoTflllrRA/JgjYFtFtN/OwSdJ eLdskRwEZ3QecgIY6ude235trZ5EA3LODj9ung1NxZbW7cd2YRADYCfFfuxSSyHTrHYE Rrlw8zZpd7uANiedU1UyeB4qLGfAeB1gazUaAV3YWoGvLyqThL80f/iiKte8JgvB9Rrx oeJbzTtgwIFcq69dDb5rm9Cht2V38IpcpBfHccGxXYwYSPvtx2NzMU1VCmkACQvUUb2g uBLg== X-Gm-Message-State: AOAM5334v+oglJIDaZLMUHakMWlzqEP5UJ/y+XpCEUMRwXnAaBCOMYF1 E3N0ZKh01bR/1isGW+tnWMhsV0xwzJEOdQ== X-Received: by 2002:a05:620a:2806:: with SMTP id f6mr25907339qkp.87.1640966295927; Fri, 31 Dec 2021 07:58:15 -0800 (PST) Received: from smtpclient.apple ([172.58.171.222]) by smtp.gmail.com with ESMTPSA id bs16sm22722727qkb.45.2021.12.31.07.58.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Dec 2021 07:58:15 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: Date: Fri, 31 Dec 2021 09:58:13 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v7 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Thilo Borgmann , Aman Karmani Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 0ppPss8akcO8 * Implement support for AudioConverter * Switch to AudioConverter's API to convert unsupported PCM formats (non-interleaved, non-packed) to supported formats * Minimize data copy. This fixes: https://trac.ffmpeg.org/ticket/9502 API ref: https://developer.apple.com/documentation/audiotoolbox/audio_converter_services Signed-off-by: Romain Beauxis — This is the first patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. Changelog for this patch: * v2: None * v3: None * v4: None * v5: Fix indentation/wrapping * v6: None * v7: Removed use of kAudioConverterPropertyCalculateOutputBufferSize to calculate output buffer size. The calculation is trivial and this call was randomly failing for no reason libavdevice/avfoundation.m | 255 +++++++++++++++++++++---------------- 1 file changed, 145 insertions(+), 110 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 0cd6e646d5..738cd93375 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -111,16 +111,11 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; + UInt32 audio_buffers; + UInt32 audio_channels; + UInt32 input_bytes_per_sample; + UInt32 output_bytes_per_sample; + AudioConverterRef audio_converter; enum AVPixelFormat pixel_format; @@ -299,7 +294,10 @@ static void destroy_context(AVFContext* ctx) ctx->avf_delegate = NULL; ctx->avf_audio_delegate = NULL; - av_freep(&ctx->audio_buffer); + if (ctx->audio_converter) { + AudioConverterDispose(ctx->audio_converter); + ctx->audio_converter = NULL; + } pthread_mutex_destroy(&ctx->frame_lock); @@ -673,6 +671,10 @@ static int get_audio_config(AVFormatContext *s) AVFContext *ctx = (AVFContext*)s->priv_data; CMFormatDescriptionRef format_desc; AVStream* stream = avformat_new_stream(s, NULL); + AudioStreamBasicDescription output_format = {0}; + int audio_bits_per_sample, audio_float, audio_be; + int audio_signed_integer, audio_packed, audio_non_interleaved; + int must_convert = 0; if (!stream) { return 1; @@ -690,60 +692,97 @@ static int get_audio_config(AVFormatContext *s) avpriv_set_pts_info(stream, 64, 1, avf_time_base); format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); + const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - if (!basic_desc) { + if (!input_format) { unlock_frames(ctx); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } + if (input_format->mFormatID != kAudioFormatLinearPCM) { + unlock_frames(ctx); + av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); + return 1; + } + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; + stream->codecpar->sample_rate = input_format->mSampleRate; + stream->codecpar->channels = input_format->mChannelsPerFrame; stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + audio_bits_per_sample = input_format->mBitsPerChannel; + audio_float = input_format->mFormatFlags & kAudioFormatFlagIsFloat; + audio_be = input_format->mFormatFlags & kAudioFormatFlagIsBigEndian; + audio_signed_integer = input_format->mFormatFlags & kAudioFormatFlagIsSignedInteger; + audio_packed = input_format->mFormatFlags & kAudioFormatFlagIsPacked; + audio_non_interleaved = input_format->mFormatFlags & kAudioFormatFlagIsNonInterleaved; + + ctx->input_bytes_per_sample = input_format->mBitsPerChannel >> 3; + ctx->output_bytes_per_sample = ctx->input_bytes_per_sample; + ctx->audio_channels = input_format->mChannelsPerFrame; + + if (audio_non_interleaved) { + ctx->audio_buffers = input_format->mChannelsPerFrame; } else { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + ctx->audio_buffers = 1; + } + + if (audio_non_interleaved || !audio_packed) { + must_convert = 1; + } + + output_format.mBitsPerChannel = input_format->mBitsPerChannel; + output_format.mChannelsPerFrame = ctx->audio_channels; + output_format.mFramesPerPacket = 1; + output_format.mBytesPerFrame = output_format.mChannelsPerFrame * ctx->input_bytes_per_sample; + output_format.mBytesPerPacket = output_format.mFramesPerPacket * output_format.mBytesPerFrame; + output_format.mFormatFlags = kAudioFormatFlagIsPacked | audio_be; + output_format.mFormatID = kAudioFormatLinearPCM; + output_format.mReserved = 0; + output_format.mSampleRate = input_format->mSampleRate; + + if (audio_float && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_float && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F64BE : AV_CODEC_ID_PCM_F64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_signed_integer && + audio_bits_per_sample == 8) { + stream->codecpar->codec_id = AV_CODEC_ID_PCM_S8; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 16) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 24) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S64BE : AV_CODEC_ID_PCM_S64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + ctx->output_bytes_per_sample = 4; + output_format.mBitsPerChannel = 32; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + must_convert = 1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { + if (must_convert) { + OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); + if (ret != noErr) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); + av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } @@ -1048,6 +1087,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1091,7 +1131,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1105,21 +1145,60 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t input_size = CMBlockBufferGetDataLength(block_buffer); + int buffer_size = input_size / ctx->audio_buffers; + int nb_samples = input_size / (ctx->audio_channels * ctx->input_bytes_per_sample); + int output_size = nb_samples * ctx->output_bytes_per_sample * ctx->audio_channels; - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { - unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + status = av_new_packet(pkt, output_size); + if (status < 0) { + CFRelease(audio_frame); + return status; } - if (av_new_packet(pkt, block_buffer_size) < 0) { - unlock_frames(ctx); - return AVERROR(EIO); + if (ctx->audio_converter) { + size_t input_buffer_size = offsetof(AudioBufferList, mBuffers[0]) + (sizeof(AudioBuffer) * ctx->audio_buffers); + AudioBufferList *input_buffer = av_malloc(input_buffer_size); + + input_buffer->mNumberBuffers = ctx->audio_buffers; + + for (int c = 0; c < ctx->audio_buffers; c++) { + input_buffer->mBuffers[c].mNumberChannels = 1; + + ret = CMBlockBufferGetDataPointer(block_buffer, c * buffer_size, (size_t *)&input_buffer->mBuffers[c].mDataByteSize, NULL, (void *)&input_buffer->mBuffers[c].mData); + + if (ret != kCMBlockBufferNoErr) { + av_free(input_buffer); + unlock_frames(ctx); + return AVERROR(EIO); + } + } + + AudioBufferList output_buffer = { + .mNumberBuffers = 1, + .mBuffers[0] = { + .mNumberChannels = ctx->audio_channels, + .mDataByteSize = pkt->size, + .mData = pkt->data + } + }; + + ret = AudioConverterConvertComplexBuffer(ctx->audio_converter, nb_samples, input_buffer, &output_buffer); + av_free(input_buffer); + + if (ret != noErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } + + pkt->size = output_buffer.mBuffers[0].mDataByteSize; + } else { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } } CMItemCount count; @@ -1133,54 +1212,10 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; - - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } - - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - } - CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); From patchwork Fri Dec 31 16:00:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32967 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp14071534iog; Fri, 31 Dec 2021 08:00:56 -0800 (PST) X-Google-Smtp-Source: ABdhPJz9MBVuqPWCdIhAG06yD7HmC3kNWIMqZd8lqbNmQPx9g4OGkt6KdiJPYiqQEzSvEWPxMYMl X-Received: by 2002:a17:906:56c9:: with SMTP id an9mr14768670ejc.755.1640966456331; Fri, 31 Dec 2021 08:00:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640966456; cv=none; d=google.com; s=arc-20160816; b=lFFxLpI6tYyAeCsARh4jZmuX0eEieRijiF7mV3w0u6kkiPGeGXnBtQJ/tddTsu0z3j PYUItNFmnxSoICiYvR7uZ3wz+WcwFFTIUxHzZk66aA2WGVLZVcVnVlNS+jk1ljZIfYXX q1tQ1mobBj6J7gOQ1ICH01f6GnaIXcexmJNU2Y6cAuitkfHmRbc2A5LHGSL50D8u/VE3 JkcjegJElRAA4OdG8K8tlRYDoWdn8yaAjgLkdxccq+zJHg/e/stuyTfUHhtOfWZkqEka rL2y+kRs1LB2/Xiji//kichUYRveSKHB4oGb4hmuCqF37FWYpBKNlzClcLIRIb2jfVxC v1Tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=6h014sy7qWHyf2byNPyOoHkkHQ78rUVB3/MfWV2c6Bc=; b=R54xaaUVQJ7FABwvUEPasXayV4Jue3dmfhEdTg4EcoouSUeRA90xskfQEHeTHWKgTE l2tdnWSEkOI9oe0V6LbpC6jKk6mYC+u/TA7eFAN7RJIvG2UPOTIbpV0HHnFownNNqus2 KqXZ/rv3sbmFINdhdhM9kl/+1JxTjS2DHVmqBrp2PJEuv6yFNoya1AYvmjWyUM6zHRP1 YbCbzOLn/fLEr6gJiMDc6d+xmo6vMe8rhr4uFXLcNU0PH6r2dvwAIX3PAUy5ev2VMje/ 8G1ZurlG9FwbM902VMjVKet5pT1QGK2PijvnbSPi5uqnf1qCzynhTbQgNeTwbjhSQTb5 jaSw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w18si14976426edl.552.2021.12.31.08.00.56; Fri, 31 Dec 2021 08:00:56 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 23F9368B040; Fri, 31 Dec 2021 18:00:53 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5BBF46805AE for ; Fri, 31 Dec 2021 18:00:46 +0200 (EET) Received: by mail-qt1-f171.google.com with SMTP id v4so21887569qtk.0 for ; Fri, 31 Dec 2021 08:00:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=W4A4UJMwwG7lw6PJ+qxap6ZLdQ2zvkl369DeShDqJMI=; b=Po+xsz31OVGCuMZPKSSrnDKPQGU0Ns0taMXBJdzEclXtwrVsXn76KpvGzrDVRBQMkn wBPhWx7IG4CbgM/zmf0bTfmMN/NRRfK/y7NJ4nAwJHjbAI/lJL34ZV55ZuLtRrT49zBc ufTQULBh1w7gE3z0VuwMeeVEnc1B9qVfpHYNNXnQGgUwiv92zYcqxqGycs5CTviuvzso uyLPeWysKCgdlzd7VJa+21mYARTTtzzEK74g+e6o/Rv0jnWhXQoyYIk7HW29pSogEAX9 ktZTWtHZJlsK2QcQkvCPwXVFdIH75vaAYUL6IUgt1H0rouQ5b/cbzquK9mC8HvAAt0GF jfdg== X-Gm-Message-State: AOAM532vAt9dkQN4rsg9bLXdKwc2O3xlSzvfNkzaMDDkBPsh8wljubC1 OpJnhdZc/IPDu848uL7xD9nBbJOEYZE0zw== X-Received: by 2002:ac8:59c9:: with SMTP id f9mr30778276qtf.204.1640966444417; Fri, 31 Dec 2021 08:00:44 -0800 (PST) Received: from smtpclient.apple ([172.58.171.222]) by smtp.gmail.com with ESMTPSA id d4sm22132118qkn.79.2021.12.31.08.00.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Dec 2021 08:00:44 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: <01FD4E04-7D9C-4284-9AB4-FF8AA3E6C20F@rastageeks.org> Date: Fri, 31 Dec 2021 10:00:42 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v7 2/3] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Thilo Borgmann , Aman Karmani Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: s40XK1Rdwt2L * Use a CMSimpleQueueEnqueue with maximum length to queue and process incoming audio and video frames. * Log avfoundation errors. * Use AVERROR_EXTERNAL instead of AVERROR(EIO) in avfoundation errors. Signed-off-by: Romain Beauxis — This is the second patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. This patch fixes the concurrency model. Avfoundation runs its own producing thread to send produced frames and ffmpeg runs its own thread to consume them. The existing implementation stores the last transmitted frame and uses a mutex to avoid concurrent access. However, this leads to situations where upcoming frames can be dropped if the ffmpeg thread is acessing the latest frame. This happens even when the thread would otherwise catch up and process frames fast enought. This patches changes this implementation to use a buffer queue with a max queue length and encapsulated thread-safety. This greatly simplifies the logic of the calling code and gives the consuming thread a chance to process all frames concurrently to the producing thread while avoiding memory leaks. Changes: v2: None v3: Switched queue implementation to CMSimpleQueue v4: None v5: Fix indentation/wrapping v6: Fix audio/video frame queue cleanup logic v7: Enhance avfoundation error reporting: add human-readable description, use AVERROR_EXTERNAL instead of AVERROR(EIO) libavdevice/avfoundation.m | 196 ++++++++++++++++++------------------- 1 file changed, 96 insertions(+), 100 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 738cd93375..c740745fce 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,7 +26,7 @@ */ #import -#include +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -39,6 +39,11 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +#define av_log_avfoundation_error(str, err) \ + av_log(s, AV_LOG_ERROR, "Avfoundation: %s, %s\n", str, \ + [[[NSError errorWithDomain:NSOSStatusErrorDomain code:err userInfo:nil] localizedDescription] UTF8String] \ + ) + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -80,13 +85,12 @@ { AV_PIX_FMT_NONE, 0 } }; +#define MAX_QUEUED_FRAMES 10 + typedef struct { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -122,8 +126,8 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + CMSimpleQueueRef audio_frames_queue; + CMSimpleQueueRef video_frames_queue; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -132,16 +136,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -219,17 +213,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); - } - - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + CFRetain(videoFrame); + CMSimpleQueueEnqueue(_context->video_frames_queue, videoFrame); } @end @@ -263,17 +248,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); - } - - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + CFRetain(audioFrame); + CMSimpleQueueEnqueue(_context->audio_frames_queue, audioFrame); } @end @@ -288,6 +264,30 @@ static void destroy_context(AVFContext* ctx) [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; + CMSampleBufferRef frame; + + if (ctx->video_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + } + + CFRelease(ctx->video_frames_queue); + ctx->video_frames_queue = NULL; + } + + if (ctx->audio_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + } + + CFRelease(ctx->audio_frames_queue); + ctx->audio_frames_queue = NULL; + } + ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; @@ -298,12 +298,6 @@ static void destroy_context(AVFContext* ctx) AudioConverterDispose(ctx->audio_converter); ctx->audio_converter = NULL; } - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static void parse_device_name(AVFormatContext *s) @@ -631,18 +625,18 @@ static int get_video_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->video_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer) { image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); @@ -658,10 +652,7 @@ static int get_video_config(AVFormatContext *s) stream->codecpar->format = ctx->pixel_format; } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -681,27 +672,27 @@ static int get_audio_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->audio_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); ctx->audio_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); + format_desc = CMSampleBufferGetFormatDescription(frame); const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); if (!input_format) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } if (input_format->mFormatID != kAudioFormatLinearPCM) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); return 1; } @@ -781,16 +772,13 @@ static int get_audio_config(AVFormatContext *s) if (must_convert) { OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); if (ret != noErr) { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); + CFRelease(frame); + av_log_avfoundation_error("error while creating audio converter", ret); return 1; } } - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -808,8 +796,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1010,6 +996,21 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + OSStatus ret; + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->video_frames_queue); + + if (ret != noErr) { + av_log_avfoundation_error("error while creating frame queue", ret); + goto fail; + } + + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->audio_frames_queue); + + if (ret != noErr) { + av_log_avfoundation_error("error while creating frame queue", ret); + goto fail; + } + if (video_device && add_video_device(s, video_device)) { goto fail; } @@ -1039,7 +1040,8 @@ static int avf_read_header(AVFormatContext *s) fail: [pool release]; destroy_context(ctx); - return AVERROR(EIO); + av_log(s, AV_LOG_ERROR, "Error while opening AVfoundation capture session\n"); + return AVERROR_EXTERNAL; } static int copy_cvpixelbuffer(AVFormatContext *s, @@ -1088,38 +1090,35 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { OSStatus ret; + int status; AVFContext* ctx = (AVFContext*)s->priv_data; do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); - - if (ctx->current_frame != nil) { - int status; + if (1 <= CMSimpleQueueGetCount(ctx->video_frames_queue)) { int length = 0; - - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + CMSampleBufferRef video_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + CVImageBufferRef image_buffer = CMSampleBufferGetImageBuffer(video_frame);; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(video_frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EINVAL); } - if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); - return AVERROR(EIO); + status = av_new_packet(pkt, length); + if (status < 0) { + CFRelease(video_frame); + return status; } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(video_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1133,18 +1132,18 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = 0; ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - status = AVERROR(EIO); + av_log_avfoundation_error("error while copying buffer data", ret); + status = AVERROR_EXTERNAL; } } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; + CFRelease(video_frame); if (status < 0) { - unlock_frames(ctx); return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + } else if (1 <= CMSimpleQueueGetCount(ctx->audio_frames_queue)) { + CMSampleBufferRef audio_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(audio_frame); size_t input_size = CMBlockBufferGetDataLength(block_buffer); int buffer_size = input_size / ctx->audio_buffers; @@ -1170,8 +1169,9 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) if (ret != kCMBlockBufferNoErr) { av_free(input_buffer); - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation_error("error while accessing audio buffer data", ret); + return AVERROR_EXTERNAL; } } @@ -1188,23 +1188,25 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) av_free(input_buffer); if (ret != noErr) { - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation_error("error while converting audio data", ret); + return AVERROR_EXTERNAL; } pkt->size = output_buffer.mBuffers[0].mDataByteSize; } else { ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation_error("error while copying audio data", ret); + return AVERROR_EXTERNAL; } } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(audio_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1212,21 +1214,15 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(audio_frame); } else { pkt->data = NULL; - unlock_frames(ctx); if (ctx->observed_quit) { return AVERROR_EOF; } else { return AVERROR(EAGAIN); } } - - unlock_frames(ctx); } while (!pkt->data); return 0; From patchwork Fri Dec 31 16:03:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32968 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp14074101iog; Fri, 31 Dec 2021 08:03:43 -0800 (PST) X-Google-Smtp-Source: ABdhPJyAzKXPOCOn75WjIj77obGcB25yh/lpHF6gBvL5tD4WCVBdpMQc8CrfP97lGH59teUIwhyd X-Received: by 2002:a05:6402:1d4b:: with SMTP id dz11mr34486671edb.15.1640966623554; Fri, 31 Dec 2021 08:03:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640966623; cv=none; d=google.com; s=arc-20160816; b=Inc107MD1z+AuBNYYrhq3pRQZQ+szQxygYpKSwtDB0H2YSiOAakffMk/Wfbj6JkaFd 0X+lsEs3lndCJV0iU10vZWIDVb8xUiUGPWl7Uv9rLvlH/j5GCPHQPP3DoxOV6tg0O2sI z+Ep8Vjh6tvyuI9inPmt8YM3zUxQTqdklo2NoEImRJPnJX84aOYIj07jN6YTXRArTqM0 hQpUfOOBOFMIdx+YsNy+q9rItxXgK1GybPfLEHm5r5vAYgnnXGD4oZZNNCdVG4MpHlSK dFYWE9EuBoPoa6+MyTt1eC+NFpgnjdSPhOMgSahU5fWYquV8TI1zUDREg+p0lTpd278A AKMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=uskCgIzwoZ9opsXyg6tL581yAqo917sHwJ2KWcC1bb4=; b=I3pHByvp3y0GuLykFnVDyb2WxU0WPzTnyMyTZbrZYU8ZMGZGETVOLghdsrOg9K8iUQ qluy0XvTBTkjKarUnT8xSyaT5PzDrTpQOnb+7PN7pAGsqjfzVwNy0SpxjUfKF2v3Au3M loSnoA/+ixBczmzNWuq+l2BQlbMMGcSNmX4zlMmPpgiRAdLeSy42FkgF4VLz7VjmXsae xCmAPo0hwhXbGHJVLYTARdCGpV5CEKuDBJc6rdVIahZ/QEgTwbLSQWOKAge9XYeyqAcN VqrAnGBMIEPSXyl1CYGasOFffuX3D95X7wGZO9af+47ozYYMsJMd6KpNM1UvxDKNm05e 2wbQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l17si13953112edc.520.2021.12.31.08.03.43; Fri, 31 Dec 2021 08:03:43 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 653F468B03F; Fri, 31 Dec 2021 18:03:40 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3C3A368AED3 for ; Fri, 31 Dec 2021 18:03:34 +0200 (EET) Received: by mail-qv1-f48.google.com with SMTP id kk22so25047479qvb.0 for ; Fri, 31 Dec 2021 08:03:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=CzyHweL8Sw2AzqNvJeQZOwwpuyhXSSszKy/uXBdSVGY=; b=FwmfSKPA89QQ9jQhiXoK7VDrVomp1NECJgBARaRJpJT3mmP8PsdQpFi9rubZUzWNyA 3YDz196xwdGK7NRPWHOf+eiNoVWx0AykvQImQOPmMF+Oxm7CtNCwajGcFbgBhIfQHZVI Oaes5xBnQK41pCtILoTCmK0XMjQ/OcWy3WZFAyIjynMX9iDBAl2lR7eZmEROkjRERFTQ pw3ckuUml5aZtOaYLDbWuDW+Asm8O32UBk2bKq/4p+YGn4XhnsSnZE769qLDtK8LmTSj lJhJToOCciAdI7Rr+MVs6WYjQhFnj3R+rKLAK0DWqJ9R05b4HAyschhTBe9HvjP3Vlvh cMiA== X-Gm-Message-State: AOAM533meDhmC2D7zF/cujc2ydPLXmd+hU2CKqG1z9JktM0eUSGcMYZv 2ERKK3qhNxZwN26OxElQpnUjg8MTd0WNNQ== X-Received: by 2002:a05:6214:da3:: with SMTP id h3mr33076328qvh.112.1640966612607; Fri, 31 Dec 2021 08:03:32 -0800 (PST) Received: from smtpclient.apple ([172.58.171.222]) by smtp.gmail.com with ESMTPSA id d4sm22136028qkn.79.2021.12.31.08.03.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Dec 2021 08:03:32 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: Date: Fri, 31 Dec 2021 10:03:30 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v7 3/3] libavdevice/avfoundation.m: Allow to select devices by unique ID X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Thilo Borgmann , Aman Karmani Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: iDcGKm8j6bwO Signed-off-by: Romain Beauxis — This is the third patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes: v2: None v3: * Switched unique ID to use system-prodvided unique ID * Implemented unique IDs for screen capture v4: Cleanup v5: Fix indentation/wrapping v6: None v7: None This patch adds a unique ID to avfoundation devices. This is needed because device index can change while the machine is running when devices are plugged or unplugged and device names can be tricky to use with localization and etc. Example of output: ./ffmpeg -f avfoundation -list_devices true -i "" [...] [AVFoundation indev @ 0x158705230] AVFoundation video devices: [AVFoundation indev @ 0x158705230] [0] FaceTime HD Camera (ID: 47B4B64B70674B9CAD2BAE273A71F4B5) [AVFoundation indev @ 0x158705230] [1] Capture screen 0 (ID: AvfilterAvfoundationCaptureScreen1) [AVFoundation indev @ 0x158705230] AVFoundation audio devices: [AVFoundation indev @ 0x158705230] [0] Loopback Audio (ID: com.rogueamoeba.Loopback.A5668B36-711E-4DF5-8A8D-7148508C735B) [AVFoundation indev @ 0x158705230] [1] MacBook Pro Microphone (ID:BuiltInMicrophoneDevice) Notes: * Unique names do not seem to follow any specific pattern. I have used one similar to the builtin microphone for screen capture * The : substitution is actually required. The loopback device above did have it in its name. doc/indevs.texi | 6 ++-- libavdevice/avfoundation.m | 72 +++++++++++++++++++++++++++++--------- 2 files changed, 60 insertions(+), 18 deletions(-) diff --git a/doc/indevs.texi b/doc/indevs.texi index 9d8020311a..858c0fa4e4 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -114,7 +114,7 @@ The input filename has to be given in the following syntax: -i "[[VIDEO]:[AUDIO]]" @end example The first entry selects the video input while the latter selects the audio input. -The stream has to be specified by the device name or the device index as shown by the device list. +The stream has to be specified by the device name, index or ID as shown by the device list. Alternatively, the video and/or audio input device can be chosen by index using the @option{ -video_device_index @@ -127,7 +127,9 @@ and/or device name or index given in the input filename. All available devices can be enumerated by using @option{-list_devices true}, listing -all device names and corresponding indices. +all device names, corresponding indices and IDs, when available. Device name can be +tricky to use when localized and device index can change when devices are plugged or unplugged. A device +hash, when available, uniquely identifies a device and should not change over time. There are two device name aliases: @table @code diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index c740745fce..bad0cd7155 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -39,6 +39,8 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +#define CLEANUP_DEVICE_ID(s) [[s stringByReplacingOccurrencesOfString:@":" withString:@"."] UTF8String] + #define av_log_avfoundation_error(str, err) \ av_log(s, AV_LOG_ERROR, "Avfoundation: %s, %s\n", str, \ [[[NSError errorWithDomain:NSOSStatusErrorDomain code:err userInfo:nil] localizedDescription] UTF8String] \ @@ -805,21 +807,23 @@ static int avf_read_header(AVFormatContext *s) int index = 0; av_log(ctx, AV_LOG_INFO, "AVFoundation video devices:\n"); for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } for (AVCaptureDevice *device in devices_muxed) { - const char *name = [[device localizedName] UTF8String]; - index = [devices count] + [devices_muxed indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices count] + [devices_muxed indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 if (num_screens > 0) { CGDirectDisplayID screens[num_screens]; CGGetActiveDisplayList(num_screens, screens, &num_screens); for (int i = 0; i < num_screens; i++) { - av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d\n", ctx->num_video_devices + i, i); + av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d (ID: AvfilterAvfoundationCaptureScreen%d)\n", ctx->num_video_devices + i, i, screens[i]); } } #endif @@ -827,9 +831,10 @@ static int avf_read_header(AVFormatContext *s) av_log(ctx, AV_LOG_INFO, "AVFoundation audio devices:\n"); devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - int index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + int index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } goto fail; } @@ -891,14 +896,29 @@ static int avf_read_header(AVFormatContext *s) } else { // looking for video inputs for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; break; } } // looking for muxed inputs for (AVCaptureDevice *device in devices_muxed) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + ctx->video_is_muxed = 1; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; ctx->video_is_muxed = 1; break; @@ -909,10 +929,23 @@ static int avf_read_header(AVFormatContext *s) // looking for screen inputs if (!video_device) { int idx; + CGDirectDisplayID screens[num_screens]; + CGGetActiveDisplayList(num_screens, screens, &num_screens); + AVCaptureScreenInput* capture_screen_input = NULL; + if(sscanf(ctx->video_filename, "Capture screen %d", &idx) && idx < num_screens) { - CGDirectDisplayID screens[num_screens]; - CGGetActiveDisplayList(num_screens, screens, &num_screens); - AVCaptureScreenInput* capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + } + + if(sscanf(ctx->video_filename, "AvfilterAvfoundationCaptureScreen%d", &idx)) { + for (int i = 0; i < num_screens; i++) { + if (screens[i] == idx) { + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:idx] autorelease]; + } + } + } + + if (capture_screen_input) { video_device = (AVCaptureDevice*) capture_screen_input; ctx->video_device_index = ctx->num_video_devices + idx; ctx->video_is_screen = 1; @@ -963,7 +996,14 @@ static int avf_read_header(AVFormatContext *s) NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->audio_filename, [[device localizedName] UTF8String], strlen(ctx->audio_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->audio_filename, name, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->audio_filename, uniqueId, strlen(ctx->audio_filename))) { audio_device = device; break; }