From patchwork Thu Jan 6 14:24:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 33124 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1695875iog; Thu, 6 Jan 2022 06:24:22 -0800 (PST) X-Google-Smtp-Source: ABdhPJzoIR/0vlMgYnAe1EfPFWUs4Eh8B9rElWnPajI3B0QPbxbdE7dLWGnyLKDPsdR2NU4p7rKj X-Received: by 2002:a50:ed06:: with SMTP id j6mr5782239eds.21.1641479061867; Thu, 06 Jan 2022 06:24:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1641479061; cv=none; d=google.com; s=arc-20160816; b=jXVZWaycrFo/XEvzFbCDw9ceaFurRdL4cG8GwnPALtRqd/aKt+l4F9odxsXjvX7cZF 2jKuKZO7ecN04B2Txj3atJkvcxt9F94m3THV3yoBKolF5O4iFtYaqUz74PEH6EXAeVNh 0Sn030ey9TB3VbSVA20jC9l1W2Ukl5L2PTNqHfSW98mPxKNIaQcJlevTjLXZmEdt4owP uwkHDKjMwiCdTohenoqj6grv9ty2Ezj3lccQo0842201BM0kw+2xvHZ2yymQYXGmcUno 3ZOUMfAXHauzJekxUTETcqyvMdxRmOaZiqcQbgDUln5VVg3p7/yj2vEI6IUYz3EO809D gMjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=h1Mu8/8DEjaO8+QZ4QUaxlMZtf5UkR2QAPrWrVYt/KA=; b=aFPlXDZOEpJ11E2U8DiFrq7ZzdmR/WfRPKAGvqcIpM3iIRU/as7C6SD/W+2KcvsPy6 6lMB4TPrDpOkG4zwjNPX/qZgW7vEBDMDK2OBmDkNzszp2w2euWT93fU3e16ZrnTbVAhC 0OGWOh/eDAPPS7G8JCmjlhuCrnpYTZ6FeuasKqqKv+VZU8BJa3wRRbOYiACkeSc6bExf 0Q2tozHFTHBOFcFBfsx/b9F+l8aqcMZY5KyequvZMu92nzSQVoL8RNbcKsojOfI8DxYw Y1sqXm4HeGYSHVAzbfYlaU5g0IyCVooye9K1JWeh7lwrzhWUV+/CYtt0uShZ9xtMDa5Q TJfA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w12si1228693edj.233.2022.01.06.06.24.20; Thu, 06 Jan 2022 06:24:21 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1798468A7FD; Thu, 6 Jan 2022 16:24:17 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2FEF06800E0 for ; Thu, 6 Jan 2022 16:24:11 +0200 (EET) Received: by mail-qk1-f178.google.com with SMTP id f138so2738069qke.10 for ; Thu, 06 Jan 2022 06:24:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=9IlDBr9ruBjEvdQbwYZiFCJfL2nazNewlJuxVde7G1U=; b=qgTZXV/d/kKrnMzjnYbIRWXqdTOlcGBAhQpnOh38VX/Aw7Yu2oLF0/g+8zEKICs3s6 VmIuPioEYmPsmmBsBeB6w7qhf/NopYWl5kv5KXXTSDhxXMf3JW82vBL4V+Nv5BmYWYiv 66lAzCsqPE2W91yKqahaaxc3UgrKTB/BnAzPz6s+O1CNYlYNOUgU+6VgIFhS3zgj80Bz 0P2xlFQJL/0GuwG7FvNG/wWyu8b5r2aSraFZVmeZwbY8NQ6+cBb66hzWhxIeI8jOszzx LwDe+z0FCE2h3nypc19OS4GfrrO/xiWjCjL3BbyzbQp4HJ5bxh3pEPCRCjTnaul846T4 MjiQ== X-Gm-Message-State: AOAM5315e0E0H/Kqs+z66Vj1IKU0Z7ojQ6d13VciEtSz6+PMsGdBUFy+ sQRXNzHTgNXMPrJdjQJ6PGTLirib7KEZOA== X-Received: by 2002:a37:b001:: with SMTP id z1mr41450404qke.252.1641479049435; Thu, 06 Jan 2022 06:24:09 -0800 (PST) Received: from smtpclient.apple ([172.58.129.153]) by smtp.gmail.com with ESMTPSA id br43sm1599883qkb.57.2022.01.06.06.24.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jan 2022 06:24:08 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: <03970EF9-6434-417E-B32D-70E37743B16E@rastageeks.org> Date: Thu, 6 Jan 2022 08:24:05 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v9 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: thilo.borgmann@mail.de, Aman Karmani , epirat07@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 7sU7X8dlrseT * Implement support for AudioConverter * Switch to AudioConverter's API to convert unsupported PCM formats (non-interleaved, non-packed) to supported formats * Minimize data copy. This fixes: https://trac.ffmpeg.org/ticket/9502 API ref: https://developer.apple.com/documentation/audiotoolbox/audio_converter_services Signed-off-by: Romain Beauxis --- This is the first patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. Changelog for this patch: * v2: None * v3: None * v4: None * v5: Fix indentation/wrapping * v6: None * v7: Removed use of kAudioConverterPropertyCalculateOutputBufferSize to calculate output buffer size. The calculation is trivial and this call was randomly failing for no reason * v8: None * v9: None libavdevice/avfoundation.m | 255 +++++++++++++++++++++---------------- 1 file changed, 145 insertions(+), 110 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 0cd6e646d5..738cd93375 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -111,16 +111,11 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; + UInt32 audio_buffers; + UInt32 audio_channels; + UInt32 input_bytes_per_sample; + UInt32 output_bytes_per_sample; + AudioConverterRef audio_converter; enum AVPixelFormat pixel_format; @@ -299,7 +294,10 @@ static void destroy_context(AVFContext* ctx) ctx->avf_delegate = NULL; ctx->avf_audio_delegate = NULL; - av_freep(&ctx->audio_buffer); + if (ctx->audio_converter) { + AudioConverterDispose(ctx->audio_converter); + ctx->audio_converter = NULL; + } pthread_mutex_destroy(&ctx->frame_lock); @@ -673,6 +671,10 @@ static int get_audio_config(AVFormatContext *s) AVFContext *ctx = (AVFContext*)s->priv_data; CMFormatDescriptionRef format_desc; AVStream* stream = avformat_new_stream(s, NULL); + AudioStreamBasicDescription output_format = {0}; + int audio_bits_per_sample, audio_float, audio_be; + int audio_signed_integer, audio_packed, audio_non_interleaved; + int must_convert = 0; if (!stream) { return 1; @@ -690,60 +692,97 @@ static int get_audio_config(AVFormatContext *s) avpriv_set_pts_info(stream, 64, 1, avf_time_base); format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); + const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - if (!basic_desc) { + if (!input_format) { unlock_frames(ctx); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } + if (input_format->mFormatID != kAudioFormatLinearPCM) { + unlock_frames(ctx); + av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); + return 1; + } + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; + stream->codecpar->sample_rate = input_format->mSampleRate; + stream->codecpar->channels = input_format->mChannelsPerFrame; stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + audio_bits_per_sample = input_format->mBitsPerChannel; + audio_float = input_format->mFormatFlags & kAudioFormatFlagIsFloat; + audio_be = input_format->mFormatFlags & kAudioFormatFlagIsBigEndian; + audio_signed_integer = input_format->mFormatFlags & kAudioFormatFlagIsSignedInteger; + audio_packed = input_format->mFormatFlags & kAudioFormatFlagIsPacked; + audio_non_interleaved = input_format->mFormatFlags & kAudioFormatFlagIsNonInterleaved; + + ctx->input_bytes_per_sample = input_format->mBitsPerChannel >> 3; + ctx->output_bytes_per_sample = ctx->input_bytes_per_sample; + ctx->audio_channels = input_format->mChannelsPerFrame; + + if (audio_non_interleaved) { + ctx->audio_buffers = input_format->mChannelsPerFrame; } else { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + ctx->audio_buffers = 1; + } + + if (audio_non_interleaved || !audio_packed) { + must_convert = 1; + } + + output_format.mBitsPerChannel = input_format->mBitsPerChannel; + output_format.mChannelsPerFrame = ctx->audio_channels; + output_format.mFramesPerPacket = 1; + output_format.mBytesPerFrame = output_format.mChannelsPerFrame * ctx->input_bytes_per_sample; + output_format.mBytesPerPacket = output_format.mFramesPerPacket * output_format.mBytesPerFrame; + output_format.mFormatFlags = kAudioFormatFlagIsPacked | audio_be; + output_format.mFormatID = kAudioFormatLinearPCM; + output_format.mReserved = 0; + output_format.mSampleRate = input_format->mSampleRate; + + if (audio_float && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_float && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_F64BE : AV_CODEC_ID_PCM_F64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsFloat; + } else if (audio_signed_integer && + audio_bits_per_sample == 8) { + stream->codecpar->codec_id = AV_CODEC_ID_PCM_S8; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 16) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 24) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 32) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else if (audio_signed_integer && + audio_bits_per_sample == 64) { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S64BE : AV_CODEC_ID_PCM_S64LE; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + } else { + stream->codecpar->codec_id = audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; + ctx->output_bytes_per_sample = 4; + output_format.mBitsPerChannel = 32; + output_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + must_convert = 1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { + if (must_convert) { + OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); + if (ret != noErr) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); + av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } @@ -1048,6 +1087,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1091,7 +1131,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1105,21 +1145,60 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t input_size = CMBlockBufferGetDataLength(block_buffer); + int buffer_size = input_size / ctx->audio_buffers; + int nb_samples = input_size / (ctx->audio_channels * ctx->input_bytes_per_sample); + int output_size = nb_samples * ctx->output_bytes_per_sample * ctx->audio_channels; - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { - unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + status = av_new_packet(pkt, output_size); + if (status < 0) { + CFRelease(audio_frame); + return status; } - if (av_new_packet(pkt, block_buffer_size) < 0) { - unlock_frames(ctx); - return AVERROR(EIO); + if (ctx->audio_converter) { + size_t input_buffer_size = offsetof(AudioBufferList, mBuffers[0]) + (sizeof(AudioBuffer) * ctx->audio_buffers); + AudioBufferList *input_buffer = av_malloc(input_buffer_size); + + input_buffer->mNumberBuffers = ctx->audio_buffers; + + for (int c = 0; c < ctx->audio_buffers; c++) { + input_buffer->mBuffers[c].mNumberChannels = 1; + + ret = CMBlockBufferGetDataPointer(block_buffer, c * buffer_size, (size_t *)&input_buffer->mBuffers[c].mDataByteSize, NULL, (void *)&input_buffer->mBuffers[c].mData); + + if (ret != kCMBlockBufferNoErr) { + av_free(input_buffer); + unlock_frames(ctx); + return AVERROR(EIO); + } + } + + AudioBufferList output_buffer = { + .mNumberBuffers = 1, + .mBuffers[0] = { + .mNumberChannels = ctx->audio_channels, + .mDataByteSize = pkt->size, + .mData = pkt->data + } + }; + + ret = AudioConverterConvertComplexBuffer(ctx->audio_converter, nb_samples, input_buffer, &output_buffer); + av_free(input_buffer); + + if (ret != noErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } + + pkt->size = output_buffer.mBuffers[0].mDataByteSize; + } else { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { + unlock_frames(ctx); + return AVERROR(EIO); + } } CMItemCount count; @@ -1133,54 +1212,10 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; - - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } - - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - } - CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); From patchwork Thu Jan 6 14:30:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 33125 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1700720iog; Thu, 6 Jan 2022 06:30:40 -0800 (PST) X-Google-Smtp-Source: ABdhPJzy2DhBjmnd7b4yV0nhk6gpDxePLfP0nMkKiNgWjQvIDFVdTVybBmggBy6y/Tj5kpWVqhTh X-Received: by 2002:a05:6402:2813:: with SMTP id h19mr58849174ede.365.1641479440116; Thu, 06 Jan 2022 06:30:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1641479440; cv=none; d=google.com; s=arc-20160816; b=MniufPY2KuiGWnfbVAMzxei64EW0OwgHpB6QWWB21oQePi8rEzh3RrtpYS7mkzUo46 84ITT+kUInfePhKF9q+qXD+y8Bmrx/a7whOU5h5cyMqO9iPRvcfi7eR4fbsg8Je9xf1l RNazSImKcw0wWz6nzAcUaAkXhVhYb6Zjkd2msl2DrSTrc1pdK5RxWFdB+SOM3Pu8yCFv YLigsrYQrMFImLov055a9Gw8QJQBqZoAo7nQfjRQWConM6hkatkxCwEEPCsXiO4g+CgD 8a41a8LQwinPgerMQkuD6IXlF8DQ8QXpWb9ELketIEUx5abJDdfS5+lVzpwX94EyXJeO +Iww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=iQRFhgebL2HbWO4I6Ttx5sADx5PIll7cJQVdAMIE5Ns=; b=ndnUfO4KkvOyrmasg7jYCgEhGKEPVzWrJbPgjTf7RRqeeZ3vGatuqvgrOTRT/ft647 2yGETB9Hsx890cy5pk3fzmfhIyUL0jtJzf9v1X93WnjOGj8by+GEuNi1LxzNA0z7HR3s Mcapf+LhfxhsAgtYdBlpBc22Zspol5i/R2Bhl+PmHEweZfQUbV4tLzD6RdqUS/6tFZUY z8VpLLEbpk7YTQShkoqL+YeTQq6RYP/i3v41/KTcQ1SktlXGMRSSTNz0a3Vc5tmGSCK+ h79+aylpgXnzjoErVZ8RRax6RJfoU7k446GO+hbtKRb8JIgzC3yiJFpejpscEMv1FwSk i2rg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id qf22si1164541ejc.560.2022.01.06.06.30.39; Thu, 06 Jan 2022 06:30:40 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B3B0868A8D9; Thu, 6 Jan 2022 16:30:36 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0B4AE68A7FD for ; Thu, 6 Jan 2022 16:30:30 +0200 (EET) Received: by mail-qv1-f46.google.com with SMTP id o10so2462094qvc.5 for ; Thu, 06 Jan 2022 06:30:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=5wYgXqFHZc8VO0JDmKlxIsQvoqxtEQrciNRl1urpSvY=; b=0c3NtJkdE2b48WW9f8FKZkm2FpWDlLyQIlfr1+n7Rw8b4qhQ6F1oql52y5PVK0+WL5 i2iL8E8ipLdhBWtwi7bfFrnvIwXThxCjVQ/fG7w4C9P+1VY+l3/HJvwvxqOekH8JN8t8 A1qo4migoVVy1aiAIPo3Vad/rSTrSd7Q/fKjIlqOhyRWvwh3AODjCcH4wMkLw/6tZts4 nIEZRIQbsYIv9UFlNgevz2t17P/qUdSsiIq02nBHBLJVUlMkuKIvjkxeIqKMpGE8gEbk 6pzpIQfoKSGq6Ob295+vqIEDOT89LJwVglkpxkRkDld3I94OJwWQVxm/2Ndrn12VoqZJ 6qPw== X-Gm-Message-State: AOAM532iEPXf6VZnd1WVi3SjwvOkiLWsasD/gM7XP+pWWeGUb7FQdulE DCw78SsomW38h8AeFCRBy2DpRcqPW5de7A== X-Received: by 2002:a05:6214:1d2b:: with SMTP id f11mr54272932qvd.56.1641479428454; Thu, 06 Jan 2022 06:30:28 -0800 (PST) Received: from smtpclient.apple ([172.58.129.153]) by smtp.gmail.com with ESMTPSA id t6sm1555066qkj.33.2022.01.06.06.30.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jan 2022 06:30:27 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: Date: Thu, 6 Jan 2022 08:30:25 -0600 To: ffmpeg-devel@ffmpeg.org X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v9 2/3] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: thilo.borgmann@mail.de, Aman Karmani , epirat07@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: mwzVI8CJbYoe * Use a CMSimpleQueueEnqueue with maximum length to queue and process incoming audio and video frames. * Log avfoundation errors. * Use AVERROR_EXTERNAL instead of AVERROR(EIO) in avfoundation errors. Signed-off-by: Romain Beauxis --- This is the first patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. Changes: * v2: None * v3: Switched queue implementation to CMSimpleQueue * v4: None * v5: Fix indentation/wrapping * v6: Fix audio/video frame queue cleanup logic * v7: Enhance avfoundation error reporting: add human-readable description, use AVERROR_EXTERNAL instead of AVERROR(EIO) * v8: Fix memory leak when video or audio queue is full * v9: Use auto-release pool for av_log_avfoundation, make it an inline function, drop log level to AV_LOG_DEBUG when frame queueing fails as it is quite likely to happen while setting up a processing pipeline. libavdevice/avfoundation.m | 196 +++++++++++++++++++------------------ 1 file changed, 102 insertions(+), 94 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 738cd93375..5ee19f4863 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,7 +26,7 @@ */ #import -#include +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -39,6 +39,13 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +static inline void av_log_avfoundation(void *s, int lvl, const char *str, OSStatus err) { + NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init]; + NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:err userInfo:nil]; + av_log(s, lvl, "AVFoundation: %s, %s\n", str, [[error localizedDescription] UTF8String]); + [pool release]; +} + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -80,13 +87,12 @@ { AV_PIX_FMT_NONE, 0 } }; +#define MAX_QUEUED_FRAMES 10 + typedef struct { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -122,8 +128,8 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + CMSimpleQueueRef audio_frames_queue; + CMSimpleQueueRef video_frames_queue; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -132,16 +138,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -219,17 +215,13 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); + OSStatus ret = CMSimpleQueueEnqueue(_context->video_frames_queue, videoFrame); - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); + if (ret != noErr) { + av_log_avfoundation(_context, AV_LOG_DEBUG, "Error while queueing video frame", ret); } - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + CFRetain(videoFrame); } @end @@ -263,17 +255,13 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); + OSStatus ret = CMSimpleQueueEnqueue(_context->audio_frames_queue, audioFrame); - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); + if (ret != noErr) { + av_log_avfoundation(_context, AV_LOG_DEBUG, "Error while queueing audio frame", ret); } - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + CFRetain(audioFrame); } @end @@ -288,6 +276,30 @@ static void destroy_context(AVFContext* ctx) [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; + CMSampleBufferRef frame; + + if (ctx->video_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + } + + CFRelease(ctx->video_frames_queue); + ctx->video_frames_queue = NULL; + } + + if (ctx->audio_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + } + + CFRelease(ctx->audio_frames_queue); + ctx->audio_frames_queue = NULL; + } + ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; @@ -298,12 +310,6 @@ static void destroy_context(AVFContext* ctx) AudioConverterDispose(ctx->audio_converter); ctx->audio_converter = NULL; } - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static void parse_device_name(AVFormatContext *s) @@ -631,18 +637,18 @@ static int get_video_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->video_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer) { image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); @@ -658,10 +664,7 @@ static int get_video_config(AVFormatContext *s) stream->codecpar->format = ctx->pixel_format; } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -681,27 +684,27 @@ static int get_audio_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->audio_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); ctx->audio_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); + format_desc = CMSampleBufferGetFormatDescription(frame); const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); if (!input_format) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } if (input_format->mFormatID != kAudioFormatLinearPCM) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); return 1; } @@ -781,16 +784,13 @@ static int get_audio_config(AVFormatContext *s) if (must_convert) { OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); if (ret != noErr) { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); + CFRelease(frame); + av_log_avfoundation(s, AV_LOG_ERROR, "error while creating audio converter", ret); return 1; } } - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -808,8 +808,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1010,6 +1008,21 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + OSStatus ret; + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->video_frames_queue); + + if (ret != noErr) { + av_log_avfoundation(s, AV_LOG_ERROR, "error while creating frame queue", ret); + goto fail; + } + + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->audio_frames_queue); + + if (ret != noErr) { + av_log_avfoundation(s, AV_LOG_ERROR, "error while creating frame queue", ret); + goto fail; + } + if (video_device && add_video_device(s, video_device)) { goto fail; } @@ -1039,7 +1052,8 @@ static int avf_read_header(AVFormatContext *s) fail: [pool release]; destroy_context(ctx); - return AVERROR(EIO); + av_log(s, AV_LOG_ERROR, "Error while opening AVfoundation capture session\n"); + return AVERROR_EXTERNAL; } static int copy_cvpixelbuffer(AVFormatContext *s, @@ -1088,38 +1102,35 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { OSStatus ret; + int status; AVFContext* ctx = (AVFContext*)s->priv_data; do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); - - if (ctx->current_frame != nil) { - int status; + if (1 <= CMSimpleQueueGetCount(ctx->video_frames_queue)) { int length = 0; - - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + CMSampleBufferRef video_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + CVImageBufferRef image_buffer = CMSampleBufferGetImageBuffer(video_frame);; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(video_frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EINVAL); } - if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); - return AVERROR(EIO); + status = av_new_packet(pkt, length); + if (status < 0) { + CFRelease(video_frame); + return status; } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(video_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1133,18 +1144,18 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = 0; ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - status = AVERROR(EIO); + av_log_avfoundation(s, AV_LOG_ERROR, "error while copying buffer data", ret); + status = AVERROR_EXTERNAL; } } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; + CFRelease(video_frame); if (status < 0) { - unlock_frames(ctx); return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + } else if (1 <= CMSimpleQueueGetCount(ctx->audio_frames_queue)) { + CMSampleBufferRef audio_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(audio_frame); size_t input_size = CMBlockBufferGetDataLength(block_buffer); int buffer_size = input_size / ctx->audio_buffers; @@ -1170,8 +1181,9 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) if (ret != kCMBlockBufferNoErr) { av_free(input_buffer); - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation(s, AV_LOG_ERROR, "error while accessing audio buffer data", ret); + return AVERROR_EXTERNAL; } } @@ -1188,23 +1200,25 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) av_free(input_buffer); if (ret != noErr) { - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation(s, AV_LOG_ERROR, "error while converting audio data", ret); + return AVERROR_EXTERNAL; } pkt->size = output_buffer.mBuffers[0].mDataByteSize; } else { ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(audio_frame); + av_log_avfoundation(s, AV_LOG_ERROR, "error while copying audio data", ret); + return AVERROR_EXTERNAL; } } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(audio_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1212,21 +1226,15 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(audio_frame); } else { pkt->data = NULL; - unlock_frames(ctx); if (ctx->observed_quit) { return AVERROR_EOF; } else { return AVERROR(EAGAIN); } } - - unlock_frames(ctx); } while (!pkt->data); return 0; From patchwork Thu Jan 6 14:31:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 33126 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1701906iog; Thu, 6 Jan 2022 06:31:56 -0800 (PST) X-Google-Smtp-Source: ABdhPJzfHEp/d+2g7MJnMqIZYh6IZ+DKo6cUDwVJcBjyI+c5EUHynlUv57VFk3VNWnIzwMhrRueK X-Received: by 2002:a17:906:bcd6:: with SMTP id lw22mr1587177ejb.114.1641479516279; Thu, 06 Jan 2022 06:31:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1641479516; cv=none; d=google.com; s=arc-20160816; b=pgqa7Lo3NOd5eO+T7goqHQtgt2LUuI4P12rodOdxY1E5Ua68Op/vaWKcBh5Sgy+CkK 4jOAIL6dkp4N8CUO4FoDIdZ5RsJQF52JermaA4j/KuVQe8yJLi/J0TI8SzdGUtI9dsNp JdhIt/huios5hbLVCdn52mPLxTypa+y0gOG7xyaODBGctZs2YBDp+eHnM7i/IF9pHAp1 K+qei+FkqL0WSiWtuRvuj9UNjF9MqFd41X4E6yFlpra8t7swBB3xIsUfDodPF7/FuFzq bW6mKbzdTa6KWZVK+p6OyvlFbQqv32CgpaAz5Smll/4zQzwGzRZty6p8NUU0yu6Uy+cW JBmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=txKzSfx4TQCaN9+oUF4LQOrHU3JhcZ9LiOWtvb75FcA=; b=brGNYq6pCUasdpeaRHrDN811RMK9N8uMGutbJQsBY7NW0GnoDz4HPekwVKapfWYben Zc6AkwILyVOw92IztQAkqonG5uQhrhpfWhmD4aWJe/gQN2nzqZvbWtGyM4FZHzMbVLFz nPzbQTVEs3gCtVPpmo5MFi6ILDqqrNkqDSx+N1/AR3AJldR758GLQKlq6FyxkD15+sH0 gWku4aqxGbP4TO9McX9XQ/t2I/hWbXCPy10UJVxgdaKN+ZUj/m2Za4sxt8UadRCG3Ti0 X09xz12oV1otS6I1eGiQ+Na2UL2lQ6gocP3MTKNjSczsooIbTBVnbdRYJ6PD3WPVvYoW Dvfw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d21si100254edr.60.2022.01.06.06.31.55; Thu, 06 Jan 2022 06:31:56 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B83EF68A8F5; Thu, 6 Jan 2022 16:31:53 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2DFDC68A7FD for ; Thu, 6 Jan 2022 16:31:47 +0200 (EET) Received: by mail-qv1-f52.google.com with SMTP id kk22so2519749qvb.0 for ; Thu, 06 Jan 2022 06:31:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:cc:to; bh=OoVBTQtBtKarobpm+NRx35CTx8FUeSrmvwba4r4qY0s=; b=2D94BTwmhp2GCrRkdiZEbv3yqCPOHHmliSFrGU0ixyl7TCAGLjSeZ+DiSllnVtvndH P/TZ/92MESfU3Awj1YLIUIGUgwN0xJ5ytY2Z9EMCjrOoxHBhr5+N7GPmWOEq/i2jHEEI puvXgJvpLa7e5gVflEthT5aUywRyLmCMhLuPjqIq9fidyv/YHLSMLBVCg+VoB8oceKgn 3qb6QrHg8ETvLJeSdWYu6ybuWY9wCF9qC+h1lk8nWiiANwZZZb6wcOXrcsHxaJafRDWe yRF3mSS2YFwzvRoFf/hnHE6YuVKlTjw2+1/t7iZChRXU/SOFWQkYVB3A1sr4AbS0KzeO a2NA== X-Gm-Message-State: AOAM5313Dh4ffNfkKlyKZmt2n4VwU89sgl2yczJJiMam7+xARh+Yz7FM 6hJkBTB264TqbndW7kFZy0atqIR0DhfkUg== X-Received: by 2002:a05:6214:29c1:: with SMTP id gh1mr53555270qvb.108.1641479505531; Thu, 06 Jan 2022 06:31:45 -0800 (PST) Received: from smtpclient.apple ([172.58.129.153]) by smtp.gmail.com with ESMTPSA id v5sm1673812qkp.126.2022.01.06.06.31.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jan 2022 06:31:45 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: Date: Thu, 6 Jan 2022 08:31:42 -0600 To: FFmpeg development discussions and patches X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v9 3/3] libavdevice/avfoundation.m: Allow to select devices by unique ID X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: thilo.borgmann@mail.de, Aman Karmani , epirat07@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: vJzdsddi/LyF Signed-off-by: Romain Beauxis --- This is the third patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. Changes: v2: None v3: * Switched unique ID to use system-prodvided unique ID * Implemented unique IDs for screen capture v4: Cleanup v5: Fix indentation/wrapping v6: None v7: None V8: None v9: None doc/indevs.texi | 6 ++-- libavdevice/avfoundation.m | 72 +++++++++++++++++++++++++++++--------- 2 files changed, 60 insertions(+), 18 deletions(-) diff --git a/doc/indevs.texi b/doc/indevs.texi index 9d8020311a..858c0fa4e4 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -114,7 +114,7 @@ The input filename has to be given in the following syntax: -i "[[VIDEO]:[AUDIO]]" @end example The first entry selects the video input while the latter selects the audio input. -The stream has to be specified by the device name or the device index as shown by the device list. +The stream has to be specified by the device name, index or ID as shown by the device list. Alternatively, the video and/or audio input device can be chosen by index using the @option{ -video_device_index @@ -127,7 +127,9 @@ and/or device name or index given in the input filename. All available devices can be enumerated by using @option{-list_devices true}, listing -all device names and corresponding indices. +all device names, corresponding indices and IDs, when available. Device name can be +tricky to use when localized and device index can change when devices are plugged or unplugged. A device +hash, when available, uniquely identifies a device and should not change over time. There are two device name aliases: @table @code diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 5ee19f4863..c4a4272965 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -46,6 +46,8 @@ static inline void av_log_avfoundation(void *s, int lvl, const char *str, OSStat [pool release]; } +#define CLEANUP_DEVICE_ID(s) [[s stringByReplacingOccurrencesOfString:@":" withString:@"."] UTF8String] + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -817,21 +819,23 @@ static int avf_read_header(AVFormatContext *s) int index = 0; av_log(ctx, AV_LOG_INFO, "AVFoundation video devices:\n"); for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } for (AVCaptureDevice *device in devices_muxed) { - const char *name = [[device localizedName] UTF8String]; - index = [devices count] + [devices_muxed indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices count] + [devices_muxed indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 if (num_screens > 0) { CGDirectDisplayID screens[num_screens]; CGGetActiveDisplayList(num_screens, screens, &num_screens); for (int i = 0; i < num_screens; i++) { - av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d\n", ctx->num_video_devices + i, i); + av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d (ID: AvfilterAvfoundationCaptureScreen%d)\n", ctx->num_video_devices + i, i, screens[i]); } } #endif @@ -839,9 +843,10 @@ static int avf_read_header(AVFormatContext *s) av_log(ctx, AV_LOG_INFO, "AVFoundation audio devices:\n"); devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - int index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + int index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } goto fail; } @@ -903,14 +908,29 @@ static int avf_read_header(AVFormatContext *s) } else { // looking for video inputs for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; break; } } // looking for muxed inputs for (AVCaptureDevice *device in devices_muxed) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + ctx->video_is_muxed = 1; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; ctx->video_is_muxed = 1; break; @@ -921,10 +941,23 @@ static int avf_read_header(AVFormatContext *s) // looking for screen inputs if (!video_device) { int idx; + CGDirectDisplayID screens[num_screens]; + CGGetActiveDisplayList(num_screens, screens, &num_screens); + AVCaptureScreenInput* capture_screen_input = NULL; + if(sscanf(ctx->video_filename, "Capture screen %d", &idx) && idx < num_screens) { - CGDirectDisplayID screens[num_screens]; - CGGetActiveDisplayList(num_screens, screens, &num_screens); - AVCaptureScreenInput* capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + } + + if(sscanf(ctx->video_filename, "AvfilterAvfoundationCaptureScreen%d", &idx)) { + for (int i = 0; i < num_screens; i++) { + if (screens[i] == idx) { + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:idx] autorelease]; + } + } + } + + if (capture_screen_input) { video_device = (AVCaptureDevice*) capture_screen_input; ctx->video_device_index = ctx->num_video_devices + idx; ctx->video_is_screen = 1; @@ -975,7 +1008,14 @@ static int avf_read_header(AVFormatContext *s) NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->audio_filename, [[device localizedName] UTF8String], strlen(ctx->audio_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->audio_filename, name, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->audio_filename, uniqueId, strlen(ctx->audio_filename))) { audio_device = device; break; }