From patchwork Fri Dec 31 17:42:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32970 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp14141255iog; Fri, 31 Dec 2021 09:42:53 -0800 (PST) X-Google-Smtp-Source: ABdhPJx3b1p3bLVK9QM1U2BH5JAEeztGD2KmvwVqtfBDdegjx6QWYpfyJul6jRM7iGo+wWWJb127 X-Received: by 2002:a17:906:b84a:: with SMTP id ga10mr28622057ejb.605.1640972573478; Fri, 31 Dec 2021 09:42:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640972573; cv=none; d=google.com; s=arc-20160816; b=xKAn187RZqUDetiZ72fBTI3p4q/zNzu1RitbumrGN8g9ss8JuL51mZIVXadY+wirsh bPMC8bbVVrbGs+GeTOkrdOrE0ibg86R6+6JBVeHhkIrBLoTfAQDgxnXOrHQdNKTmfc71 xjMkQFcu+0X7eonWtZ10Bi7YNNVFTonijc6rXV24/v55xTEfxPcVVYWXHBLjLJsAqoIQ 9riQTmV+SnNWlOvoCn5D9RbqKlUcR6xktSwcjDwpuelIATJpV/p8kfTtupYGEcIvGn1h VcRLnZT5rU8qYC5XcbWmOlID4/DOwFx6C7t4HT/e3sq2GMaoqiAiP1aunPBXTops9XoS 8V0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=8LozG3w1MaCte/MAS2Ac3l0eyvcATtSCZhGxrjj7qX0=; b=q5Ka7vXsYB+4jHcllRwkmV8Kn1eq+cyh8/H7Kls6/sEXGGgqzHDIsGiDHj+C8OJisV r5Mg6lZBlX7Si/9FyjsyBl9SUkbyPUDzTfux8+/gZUmFkKDA42pJB0qCfSTmTRF0TsF9 qMGODZLvAfS1zh4YGLklLyDijZSJ7ZcTiWmfI1CFDgQBQhaaCkOeoJJ/vB/yXnmSrbi1 ennjcbOAjuy1qmBYqJWlSKurg+ZRj0ekgAN3TwMGxGK6isvLI97QlBdSxIHyS3g6yp1D zjEmxGBwJthLBCYTIuIN641lZUDBvHMeTOlmkvZjiLbRqDRPBAasLtxVlZLVOAnTIYFV 1fog== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e2si16957169ejs.133.2021.12.31.09.42.52; Fri, 31 Dec 2021 09:42:53 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id ECDA668A92E; Fri, 31 Dec 2021 19:42:48 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CAB416808DF for ; Fri, 31 Dec 2021 19:42:42 +0200 (EET) Received: by mail-qk1-f181.google.com with SMTP id 131so25617015qkk.2 for ; Fri, 31 Dec 2021 09:42:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=zv92bKgQlpMDqmblfLlPXaCWRtNRrsZaQ6YBz1xM37Y=; b=DKGWuQXjvwrHXOGqqmTp/3sjNMb69BZ6o3LStv35Pib2uoe84n3Hnq2ygZysdS/Kvn p8uMrQEIO68RvmUpr6g2Exi3U4KrAnYhePJhk76/OJ7IeNWB27g8zq5WHOQ06KUwP7ZU np+7DLUg+y5HM1zAHwMBuTaT1LJW/IZuDg9a0Uqc+geWvvWwT3IO1l5c28fZDd6sWgYl 4fn4qPhA60KkDB0U6gLgdz9cH60PvllLK+XM922LH8eCIE4juB5FV9o96xDaQt1Vtqwk 3MSDG90TG7MhjkX38OFyw7wx/NK/JKZTAT7Ufnhp8cstWuWMv1QevW5BFuBy3aAghp24 YV9A== X-Gm-Message-State: AOAM531L2MN+aqU++UmD4c/2qynK4AvHHL1ZCoGVNODqwzxz6F0tFs1U UvaAFvx26KytThFEYUXYE9noolGcOjhTQg== X-Received: by 2002:a05:620a:2955:: with SMTP id n21mr25277137qkp.581.1640972561079; Fri, 31 Dec 2021 09:42:41 -0800 (PST) Received: from smtpclient.apple ([172.58.171.222]) by smtp.gmail.com with ESMTPSA id q6sm17034781qke.61.2021.12.31.09.42.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Dec 2021 09:42:40 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: <44D555B2-4959-47EA-8210-917ACB4652EE@rastageeks.org> Date: Fri, 31 Dec 2021 11:42:38 -0600 To: FFmpeg development discussions and patches X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v8 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Thilo Borgmann , Aman Karmani Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ZeacNrctbue8 * Implement support for AudioConverter * Switch to AudioConverter's API to convert unsupported PCM formats (non-interleaved, non-packed) to supported formats * Minimize data copy. This fixes: https://trac.ffmpeg.org/ticket/9502 API ref: https://developer.apple.com/documentation/audiotoolbox/audio_converter_services Signed-off-by: Romain Beauxis — [Sorry for the noise but an issue came up with the previous set] This is the first patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. Changelog for this patch: * v2: None * v3: None * v4: None * v5: Fix indentation/wrapping * v6: None * v7: Removed use of kAudioConverterPropertyCalculateOutputBufferSize to calculate output buffer size. The calculation is trivial and this call was randomly failing for no reason * v8: None libavdevice/avfoundation.m | 255 +++++++++++++++++++++---------------- 1 file changed, 145 insertions(+), 110 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 0cd6e646d5..738cd93375 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -111,16 +111,11 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; + UInt32 audio_buffers; + UInt32 audio_channels; + UInt32 input_bytes_per_sample; + UInt32 output_bytes_per_sample; + AudioConverterRef audio_converter; enum AVPixelFormat pixel_format; @@ -299,7 +294,10 @@ static void destroy_context(AVFContext* ctx) ctx->avf_delegate = NULL; ctx->avf_audio_delegate = NULL; - av_freep(&ctx->audio_buffer); + if (ctx->audio_converter) { + AudioConverterDispose(ctx->audio_converter); + ctx->audio_converter = NULL; + } pthread_mutex_destroy(&ctx->frame_lock); @@ -673,6 +671,10 @@ static int get_audio_config(AVFormatContext *s) AVFContext *ctx = (AVFContext*)s->priv_data; CMFormatDescriptionRef format_desc; AVStream* stream = avformat_new_stream(s, NULL); + AudioStreamBasicDescription output_format = {0}; + int audio_bits_per_sample, audio_float, audio_be; + int audio_signed_integer, audio_packed, audio_non_interleaved; + int must_convert = 0; if (!stream) { return 1; @@ -690,60 +692,97 @@ static int get_audio_config(AVFormatContext *s) avpriv_set_pts_info(stream, 64, 1, avf_time_base); format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); + const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - if (!basic_desc) { + if (!input_format) { unlock_frames(ctx); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } + if (input_format->mFormatID != kAudioFormatLinearPCM) { + unlock_frames(ctx); + av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); + return 1; + } + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; + stream->codecpar->sample_rate = input_format->mSampleRate; + stream->codecpar->channels = input_format->mChannelsPerFrame; stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + audio_bits_per_sample = input_format->mBitsPerChannel; + audio_float = input_format->mFormatFlags & kAudioFormatFlagIsFloat; + audio_be = input_format->mFormatFlags & kAudioFormatFlagIsBigEndian; + audio_signed_integer = input_format->mFormatFlags & kAudioFormatFlagIsSignedInteger; + audio_packed = input_format->mFormatFlags & kAudioFormatFlagIsPacked; + audio_non_interleaved = input_format->mFormatFlags & kAudioFormatFlagIsNonInterleaved; + + ctx->input_bytes_per_sample = input_format->mBitsPerChannel >> 3; + ctx->output_bytes_per_sample = ctx->input_bytes_per_sample; + ctx->audio_channels = input_format->mChannelsPerFrame; + + if (audio_non_interleaved) { + ctx->audio_buffers = input_format->mChannelsPerFrame; } else { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + ctx->audio_buffers = 1; + } + + if (audio_non_interleaved || !audio_packed) { + must_convert = 1; + } + + output_format.mBitsPerChannel = input_format->mBitsPerChannel; + output_format.mChannelsPerFrame = ctx->audio_channels; + output_format.mFramesPerPacket = 1; + output_format.mBytesPerFrame = output_format.mChannelsPerFrame * ctx->input_bytes_per_sample; + output_format.mBytesPerPacket = output_format.mFramesPerPacket * output_format.mBytesPerFrame; + output_format.mFormatFlags = kAudioFormatFlagIsPacked | audio_be; + output_format.mFormatID = kAudioFormatLinearPCM; + output_format.mReserved = 0; + output_format.mSampleRate = input_format->mSampleRate; + + if (audio_float && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_float && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F64BE : AV_CODEC_ID_PCM_F64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_signed_integer && + audio_bits_per_sample == 8) { + stream->codecpar->codec_id = AV_CODEC_ID_PCM_S8; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 16) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 24) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S64BE : AV_CODEC_ID_PCM_S64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + ctx->output_bytes_per_sample = 4; + output_format.mBitsPerChannel = 32; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + must_convert = 1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { + if (must_convert) { + OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); + if (ret != noErr) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); + av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } @@ -1048,6 +1087,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1091,7 +1131,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1105,21 +1145,60 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t input_size = CMBlockBufferGetDataLength(block_buffer); + int buffer_size = input_size / ctx->audio_buffers; + int nb_samples = input_size / (ctx->audio_channels * ctx->input_bytes_per_sample); + int output_size = nb_samples * ctx->output_bytes_per_sample * ctx->audio_channels; - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { - unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + status = av_new_packet(pkt, output_size); + if (status < 0) { + CFRelease(audio_frame); + return status; } - if (av_new_packet(pkt, block_buffer_size) < 0) { - unlock_frames(ctx); - return AVERROR(EIO); + if (ctx->audio_converter) { + size_t input_buffer_size = offsetof(AudioBufferList, mBuffers[0]) + (sizeof(AudioBuffer) * ctx->audio_buffers); + AudioBufferList *input_buffer = av_malloc(input_buffer_size); + + input_buffer->mNumberBuffers = ctx->audio_buffers; + + for (int c = 0; c < ctx->audio_buffers; c++) { + input_buffer->mBuffers[c].mNumberChannels = 1; + + ret = CMBlockBufferGetDataPointer(block_buffer, c * buffer_size, (size_t *)&input_buffer->mBuffers[c].mDataByteSize, NULL, (void *)&input_buffer->mBuffers[c].mData); + + if (ret != kCMBlockBufferNoErr) { + av_free(input_buffer); + unlock_frames(ctx); + return AVERROR(EIO); + } + } + + AudioBufferList output_buffer = { + .mNumberBuffers = 1, + .mBuffers[0] = { + .mNumberChannels = ctx->audio_channels, + .mDataByteSize = pkt->size, + .mData = pkt->data + } + }; + + ret = AudioConverterConvertComplexBuffer(ctx->audio_converter, nb_samples, input_buffer, &output_buffer); + av_free(input_buffer); + + if (ret != noErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } + + pkt->size = output_buffer.mBuffers[0].mDataByteSize; + } else { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } } CMItemCount count; @@ -1133,54 +1212,10 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; - - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } - - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - } - CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); From patchwork Fri Dec 31 17:43:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32971 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp14142089iog; Fri, 31 Dec 2021 09:44:11 -0800 (PST) X-Google-Smtp-Source: ABdhPJwjZD+Y+8ccOvvx2v6ostMn0tLOUPJnmSDylVfXzTHU3wOJbmOAyOvwFldfWOGs3UmXRFU9 X-Received: by 2002:a17:906:5ada:: with SMTP id x26mr29084530ejs.720.1640972651486; Fri, 31 Dec 2021 09:44:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640972651; cv=none; d=google.com; s=arc-20160816; b=ZMyzk7cKBWcrFS6TwhiQBOAwLUTUA0fAQUB1c+UgQ9juiF8u0/4ZsePr/s10NvEWV0 8kqQssnNPzoX7S61ZfmmLFzM1VN40DRfihO4NJRA0s8M44csDAdmOHLbFIGpk0HPZ6tf MbnMgKiZWUQMkJeud0AjVnM3gmeuKwLyqJizadgeUk39KT9mTYvu1TK66Ukz64A4nZg0 VhTXMJ4j9kY86BZJFIS1gy0C9ny6Lak3bKoN8q2G8E/+TSKNGnGthj5NlO/Greuucz38 8SZQhTp3OOgs5mzrU5McxlBxmbATN4LId4ARSB63b/grmOwfJjPFKUS3RRglTQTXB1Zp qhHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=mSsaCyEZ0H0GqYng/IYLr+jFhTommRHmDPgSpTE8Iz4=; b=TCTB425Qkk1pDfshVNNYNBYF98gsIdD++Kc/s4hdYM0uiuuYI8WfXa8J5WSAHwmyUY +lNHl045LraePUJd4r3X3+Z4qOx+A/7+rPFtibegjjwdCnHPzKTUqqXyPd2Eyqa55hdA vA+8/EdzSHwz74s6g7iDP3BLruUdDELj3O5sHEw+tlmx0fL0j3Z/qQKsRMrGattDJ+Pq qBBDNSc/KorXSAT1rGVsPuE82Owh4EFCiU7ZSRpuqBUmD8kk+7OkA5VXKpfryZ5jQ9qw 39hg5Pe2qxd8ygDJ2OzI1snYxFK6Gb3tTowJ+l8rTcrMhu3o+I5U1Ntp3NniQe6g0+Bu ZYSA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id go6si4590184ejc.603.2021.12.31.09.44.11; Fri, 31 Dec 2021 09:44:11 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 230F968B059; Fri, 31 Dec 2021 19:44:09 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 57FD06808DF for ; Fri, 31 Dec 2021 19:44:02 +0200 (EET) Received: by mail-qt1-f174.google.com with SMTP id bp39so24407163qtb.6 for ; Fri, 31 Dec 2021 09:44:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=6yqkIE47KrD3MErtJ+9lWMKB0rcDQ5CoKTvqvhcqfmY=; b=TaJlkwxkgjNcDbxBDpKhczPHx77DiyjwcjPEMNZ7zoJjLREjv8tAqgHIl2vr0h+aFL JKsPj+DkTDLj7c5mXNgNv1SqNrNSN2/VWNcZ7vVJFVpqG94O8e/9Gsb230osOhxiY0wN MEd7Ey4kD4clhprVvFYxzw1gDYTC8FrFfb6Rh5+G3x3Bq78cC19XX5o+nWapQO8569aq B8IryYDL3GSt9jssIN79XLRBwY0TXeReIA6CQPPYGPl3zNEns+jD6zMF3kGitaJKMGXG T9YMtDEh3UI/7w9zgvSXk4wdOLRXN9afNRGuXVTN82zBlcJQzZEikzPCfmbAq+d4veoA GQ0Q== X-Gm-Message-State: AOAM530A0PR/HriF6kPvtRXU1YRZyeh+0E98/nGsQd51F+2VOkSAzQh6 FyUhPO3yp8gM76xfnxClGQq0zeZ5UEp35w== X-Received: by 2002:ac8:5989:: with SMTP id e9mr31362889qte.55.1640972640575; Fri, 31 Dec 2021 09:44:00 -0800 (PST) Received: from smtpclient.apple ([172.58.171.222]) by smtp.gmail.com with ESMTPSA id 137sm20043035qkm.69.2021.12.31.09.43.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Dec 2021 09:44:00 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: Date: Fri, 31 Dec 2021 11:43:57 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v8 2/3] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Thilo Borgmann , Aman Karmani Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: iMItTX/tC04S * Use a CMSimpleQueueEnqueue with maximum length to queue and process incoming audio and video frames. * Log avfoundation errors. * Use AVERROR_EXTERNAL instead of AVERROR(EIO) in avfoundation errors. Signed-off-by: Romain Beauxis — [Sorry for the noise but an issue came up with the previous set] This is the second patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. Changelog for this patch: * v2: None * v3: None * v4: None * v5: Fix indentation/wrapping * v6: None * v7: Removed use of kAudioConverterPropertyCalculateOutputBufferSize to calculate output buffer size. The calculation is trivial and this call was randomly failing for no reason * v8: Fix memory leak when video or audio queue is full libavdevice/avfoundation.m | 194 +++++++++++++++++++------------------ 1 file changed, 100 insertions(+), 94 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 738cd93375..36f9fdc53d 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,7 +26,7 @@ */ #import -#include +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -39,6 +39,11 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +#define av_log_avfoundation_error(s, str, err) \ + av_log(s, AV_LOG_ERROR, "Avfoundation: %s, %s\n", str, \ + [[[NSError errorWithDomain:NSOSStatusErrorDomain code:err userInfo:nil] localizedDescription] UTF8String] \ + ) + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -80,13 +85,12 @@ { AV_PIX_FMT_NONE, 0 } }; +#define MAX_QUEUED_FRAMES 10 + typedef struct { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -122,8 +126,8 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + CMSimpleQueueRef audio_frames_queue; + CMSimpleQueueRef video_frames_queue; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -132,16 +136,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -219,17 +213,13 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); + OSStatus ret = CMSimpleQueueEnqueue(_context->video_frames_queue, videoFrame); - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); + if (ret != noErr) { + av_log_avfoundation_error(_context, "Error while queueing video frame", ret); } - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + CFRetain(videoFrame); } @end @@ -263,17 +253,13 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); + OSStatus ret = CMSimpleQueueEnqueue(_context->audio_frames_queue, audioFrame); - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); + if (ret != noErr) { + av_log_avfoundation_error(_context, "Error while queueing audio frame", ret); } - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + CFRetain(audioFrame); } @end @@ -288,6 +274,30 @@ static void destroy_context(AVFContext* ctx) [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; + CMSampleBufferRef frame; + + if (ctx->video_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + } + + CFRelease(ctx->video_frames_queue); + ctx->video_frames_queue = NULL; + } + + if (ctx->audio_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + } + + CFRelease(ctx->audio_frames_queue); + ctx->audio_frames_queue = NULL; + } + ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; @@ -298,12 +308,6 @@ static void destroy_context(AVFContext* ctx) AudioConverterDispose(ctx->audio_converter); ctx->audio_converter = NULL; } - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static void parse_device_name(AVFormatContext *s) @@ -631,18 +635,18 @@ static int get_video_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->video_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer) { image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); @@ -658,10 +662,7 @@ static int get_video_config(AVFormatContext *s) stream->codecpar->format = ctx->pixel_format; } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -681,27 +682,27 @@ static int get_audio_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->audio_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); ctx->audio_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); + format_desc = CMSampleBufferGetFormatDescription(frame); const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); if (!input_format) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } if (input_format->mFormatID != kAudioFormatLinearPCM) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); return 1; } @@ -781,16 +782,13 @@ static int get_audio_config(AVFormatContext *s) if (must_convert) { OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); if (ret != noErr) { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); + CFRelease(frame); + av_log_avfoundation_error(s, "error while creating audio converter", ret); return 1; } } - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -808,8 +806,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1010,6 +1006,21 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + OSStatus ret; + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->video_frames_queue); + + if (ret != noErr) { + av_log_avfoundation_error(s, "error while creating frame queue", ret); + goto fail; + } + + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->audio_frames_queue); + + if (ret != noErr) { + av_log_avfoundation_error(s, "error while creating frame queue", ret); + goto fail; + } + if (video_device && add_video_device(s, video_device)) { goto fail; } @@ -1039,7 +1050,8 @@ static int avf_read_header(AVFormatContext *s) fail: [pool release]; destroy_context(ctx); - return AVERROR(EIO); + av_log(s, AV_LOG_ERROR, "Error while opening AVfoundation capture session\n"); + return AVERROR_EXTERNAL; } static int copy_cvpixelbuffer(AVFormatContext *s, @@ -1088,38 +1100,35 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { OSStatus ret; + int status; AVFContext* ctx = (AVFContext*)s->priv_data; do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); - - if (ctx->current_frame != nil) { - int status; + if (1 <= CMSimpleQueueGetCount(ctx->video_frames_queue)) { int length = 0; - - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + CMSampleBufferRef video_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + CVImageBufferRef image_buffer = CMSampleBufferGetImageBuffer(video_frame);; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(video_frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EINVAL); } - if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); - return AVERROR(EIO); + status = av_new_packet(pkt, length); + if (status < 0) { + CFRelease(video_frame); + return status; } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(video_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1133,18 +1142,18 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = 0; ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - status = AVERROR(EIO); + av_log_avfoundation_error(s, "error while copying buffer data", ret); + status = AVERROR_EXTERNAL; } } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; + CFRelease(video_frame); if (status < 0) { - unlock_frames(ctx); return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + } else if (1 <= CMSimpleQueueGetCount(ctx->audio_frames_queue)) { + CMSampleBufferRef audio_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(audio_frame); size_t input_size = CMBlockBufferGetDataLength(block_buffer); int buffer_size = input_size / ctx->audio_buffers; @@ -1170,8 +1179,9 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) if (ret != kCMBlockBufferNoErr) { av_free(input_buffer); - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation_error(s, "error while accessing audio buffer data", ret); + return AVERROR_EXTERNAL; } } @@ -1188,23 +1198,25 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) av_free(input_buffer); if (ret != noErr) { - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation_error(s, "error while converting audio data", ret); + return AVERROR_EXTERNAL; } pkt->size = output_buffer.mBuffers[0].mDataByteSize; } else { ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation_error(s, "error while copying audio data", ret); + return AVERROR_EXTERNAL; } } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(audio_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1212,21 +1224,15 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(audio_frame); } else { pkt->data = NULL; - unlock_frames(ctx); if (ctx->observed_quit) { return AVERROR_EOF; } else { return AVERROR(EAGAIN); } } - - unlock_frames(ctx); } while (!pkt->data); return 0; From patchwork Fri Dec 31 17:45:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32972 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp14142861iog; Fri, 31 Dec 2021 09:45:31 -0800 (PST) X-Google-Smtp-Source: ABdhPJx6/khqbhpkRa47E3M3P9LUoRf7sGITm/elzHLv2Lhy66zH8+zziCIlNvM/NrzC8x8BANiS X-Received: by 2002:a17:906:cec5:: with SMTP id si5mr30110896ejb.17.1640972731710; Fri, 31 Dec 2021 09:45:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640972731; cv=none; d=google.com; s=arc-20160816; b=hZDYPGay6BC1qD9EzjgQg/jPc0srOZt6s1dr3p+hJdqp8kBaWoJG0BiXsE8p/m+BGX cDuQq6/rzpZU+i+GXj+0j+1UMfVd9rJpoPg6CTvJZwBNvUjnMXSsCX7Ogh/CoY8gTYVL KsKtyQbh7zZXkRFVN7MKjT9vDr85YfuBmU8rdr8oP9fdFrx4JpA1p15rmWGrIJ6ruy8O qlyL/2wO0UD8m28c7pYrmkh14oD3zHw6VczwJ2nngGAD0rUjt6oohsPqVufi5fv/A7eq mmHiMcPh838ykluJvBtf/iaIlLkKZGvQXBybpNGFK+QuWNJqVbqBiSfAPc05dbu4X2oZ nC6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=QWz0PuCYSgLDJVkDTqaL5NwUMBx0LFQKrgKmBpVPHRs=; b=mKgzclSFmsqWd4iTJJhoW5BNMoRtn9kMygM4tNVaQS0X0O9gJOQhxGhgd/huwiXzrG wz2MiQ1aLIR23kn3hxNA1hFT49csRWRsorco7hNPVduaIlq6uFmbmxDAOQW7vroDIRRR MWDdOYD6sDrUfNg4T16/5moiraRRbKJ1/064+quNX1Gxb4at3q9H0ESZfyeXjOVt4MVI KEp56pOPFgRcVdlPNTn3n7yJlWz9MW1OdK/H5LqLz0s75Ermdr3dwozU3/HHrapz8TmD iX1q5VrhHKlpSehhAJgi9tH3sDnN0u5m1qvhmuFhmGtH8J6DDekA1AS+W0N/87qq/E0N t6Wg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p11si13451656edy.216.2021.12.31.09.45.31; Fri, 31 Dec 2021 09:45:31 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 309ED68B059; Fri, 31 Dec 2021 19:45:29 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3DA896808DF for ; Fri, 31 Dec 2021 19:45:23 +0200 (EET) Received: by mail-qk1-f170.google.com with SMTP id b85so25616287qkc.1 for ; Fri, 31 Dec 2021 09:45:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=t1EXcYJ8mj+iTdy6fm2uO9pyP1bb6ET7emNuXTQDtXc=; b=KBEnlPsvhHMC+V+kvvcJ4VkwnmEG15zx4xPejtJGqv8sb28LFJmTNA8ZDfAXUDHFoC 90zCUhSZfWqzP6ZLXpCn7JltMWsiRaOkxnoJAiqp0fZqqetGuUw9ZaRFtDnjCTpl4zgD 83mkwmGxwZoNXLoSVCpMTfsqcmOYbbqDafAmotkMzgdHExmgdbaqw0bzDth8tC3nuNcC RWJQcXF1RK4PPAd5gFRltrnC1Y2NidT4t9c+qjUyQannQO7iHE98cfGrEF3/zU0Ej3FJ Lw2rW2BvoKh87WxOGk32ir09dWcvUFO/szo3UYZFek5IRc1Ltm4IqOyKK84UgbDT9NVm npXQ== X-Gm-Message-State: AOAM531EEBW00+GPNZQUf9gY5I0DT6HqvXrsi88QLpAoo2ktfYnm9SIL iuGfaRkPY/Xu9AzpThzb8RbPCq6oovnrQg== X-Received: by 2002:a37:a483:: with SMTP id n125mr25461326qke.305.1640972721574; Fri, 31 Dec 2021 09:45:21 -0800 (PST) Received: from smtpclient.apple ([172.58.171.222]) by smtp.gmail.com with ESMTPSA id c25sm22759161qkp.31.2021.12.31.09.45.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Dec 2021 09:45:21 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: <0A27B227-B693-4B6B-834F-ADC51464C4F6@rastageeks.org> Date: Fri, 31 Dec 2021 11:45:19 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v8 3/3] libavdevice/avfoundation.m: Allow to select devices by unique ID X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Thilo Borgmann , Aman Karmani Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: wqN2P7HtGhyI Signed-off-by: Romain Beauxis — [Sorry for the noise but an issue came up with the previous set] This is the third patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production Changes: v2: None v3: * Switched unique ID to use system-prodvided unique ID * Implemented unique IDs for screen capture v4: Cleanup v5: Fix indentation/wrapping v6: None v7: None V8: None This patch adds a unique ID to avfoundation devices. This is needed because device index can change while the machine is running when devices are plugged or unplugged and device names can be tricky to use with localization and etc. Example of output: ./ffmpeg -f avfoundation -list_devices true -i "" [...] [AVFoundation indev @ 0x158705230] AVFoundation video devices: [AVFoundation indev @ 0x158705230] [0] FaceTime HD Camera (ID: 47B4B64B70674B9CAD2BAE273A71F4B5) [AVFoundation indev @ 0x158705230] [1] Capture screen 0 (ID: AvfilterAvfoundationCaptureScreen1) [AVFoundation indev @ 0x158705230] AVFoundation audio devices: [AVFoundation indev @ 0x158705230] [0] Loopback Audio (ID: com.rogueamoeba.Loopback.A5668B36-711E-4DF5-8A8D-7148508C735B) [AVFoundation indev @ 0x158705230] [1] MacBook Pro Microphone (ID:BuiltInMicrophoneDevice) Notes: * Unique names do not seem to follow any specific pattern. I have used one similar to the builtin microphone for screen capture * The : substitution is actually required. The loopback device above did have it in its name. doc/indevs.texi | 6 ++-- libavdevice/avfoundation.m | 72 +++++++++++++++++++++++++++++--------- 2 files changed, 60 insertions(+), 18 deletions(-) diff --git a/doc/indevs.texi b/doc/indevs.texi index 9d8020311a..858c0fa4e4 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -114,7 +114,7 @@ The input filename has to be given in the following syntax: -i "[[VIDEO]:[AUDIO]]" @end example The first entry selects the video input while the latter selects the audio input. -The stream has to be specified by the device name or the device index as shown by the device list. +The stream has to be specified by the device name, index or ID as shown by the device list. Alternatively, the video and/or audio input device can be chosen by index using the @option{ -video_device_index @@ -127,7 +127,9 @@ and/or device name or index given in the input filename. All available devices can be enumerated by using @option{-list_devices true}, listing -all device names and corresponding indices. +all device names, corresponding indices and IDs, when available. Device name can be +tricky to use when localized and device index can change when devices are plugged or unplugged. A device +hash, when available, uniquely identifies a device and should not change over time. There are two device name aliases: @table @code diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 36f9fdc53d..d09a81cb3b 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -39,6 +39,8 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +#define CLEANUP_DEVICE_ID(s) [[s stringByReplacingOccurrencesOfString:@":" withString:@"."] UTF8String] + #define av_log_avfoundation_error(s, str, err) \ av_log(s, AV_LOG_ERROR, "Avfoundation: %s, %s\n", str, \ [[[NSError errorWithDomain:NSOSStatusErrorDomain code:err userInfo:nil] localizedDescription] UTF8String] \ @@ -815,21 +817,23 @@ static int avf_read_header(AVFormatContext *s) int index = 0; av_log(ctx, AV_LOG_INFO, "AVFoundation video devices:\n"); for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } for (AVCaptureDevice *device in devices_muxed) { - const char *name = [[device localizedName] UTF8String]; - index = [devices count] + [devices_muxed indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices count] + [devices_muxed indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 if (num_screens > 0) { CGDirectDisplayID screens[num_screens]; CGGetActiveDisplayList(num_screens, screens, &num_screens); for (int i = 0; i < num_screens; i++) { - av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d\n", ctx->num_video_devices + i, i); + av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d (ID: AvfilterAvfoundationCaptureScreen%d)\n", ctx->num_video_devices + i, i, screens[i]); } } #endif @@ -837,9 +841,10 @@ static int avf_read_header(AVFormatContext *s) av_log(ctx, AV_LOG_INFO, "AVFoundation audio devices:\n"); devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - int index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + int index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } goto fail; } @@ -901,14 +906,29 @@ static int avf_read_header(AVFormatContext *s) } else { // looking for video inputs for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; break; } } // looking for muxed inputs for (AVCaptureDevice *device in devices_muxed) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + ctx->video_is_muxed = 1; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; ctx->video_is_muxed = 1; break; @@ -919,10 +939,23 @@ static int avf_read_header(AVFormatContext *s) // looking for screen inputs if (!video_device) { int idx; + CGDirectDisplayID screens[num_screens]; + CGGetActiveDisplayList(num_screens, screens, &num_screens); + AVCaptureScreenInput* capture_screen_input = NULL; + if(sscanf(ctx->video_filename, "Capture screen %d", &idx) && idx < num_screens) { - CGDirectDisplayID screens[num_screens]; - CGGetActiveDisplayList(num_screens, screens, &num_screens); - AVCaptureScreenInput* capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + } + + if(sscanf(ctx->video_filename, "AvfilterAvfoundationCaptureScreen%d", &idx)) { + for (int i = 0; i < num_screens; i++) { + if (screens[i] == idx) { + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:idx] autorelease]; + } + } + } + + if (capture_screen_input) { video_device = (AVCaptureDevice*) capture_screen_input; ctx->video_device_index = ctx->num_video_devices + idx; ctx->video_is_screen = 1; @@ -973,7 +1006,14 @@ static int avf_read_header(AVFormatContext *s) NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->audio_filename, [[device localizedName] UTF8String], strlen(ctx->audio_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->audio_filename, name, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->audio_filename, uniqueId, strlen(ctx->audio_filename))) { audio_device = device; break; }