From patchwork Tue Mar 22 13:39:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 34895 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:ab0:5fda:0:0:0:0:0 with SMTP id g26csp680150uaj; Tue, 22 Mar 2022 06:41:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzaCuUGtW3Ozv0q2B/0stkGi+XUf578ISN3xCJ2atOvxWV0O9CWszDuxwd6ZEhLtZxhsseX X-Received: by 2002:aa7:c704:0:b0:418:ee8f:3fd0 with SMTP id i4-20020aa7c704000000b00418ee8f3fd0mr28472225edq.248.1647956461714; Tue, 22 Mar 2022 06:41:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647956461; cv=none; d=google.com; s=arc-20160816; b=RAqkz7LibbtBIy23rK7nZQNhmsHDe9xGu/lOYDyctQ0Gj4F1pkcVe+UWgKFPL/8vnB g1iW9E1SvrDziik++VpNKhEjt6nYOsKCg0VZdfBx+udwgDTUxUBsFurEPWBHwXrH8V58 ile/v8iP7pObOZOLw9gLtzohPT85QBmCcF+7CAUJjbEsWpPen9RIa+g/61uP4G3n581F vMNzyypB8CNunTPd/X23Wr3jugJESRorMxL/SlMbjKkTnZo4N+qu4BEaiDGNwN53ZkRQ l79b3JVPMzfEodONm9kEzRv7gvJIHK67MejU2shQkAxmCQDhcS60LFS9kJw47XYifxLB 4diQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=LXmzMno9onS8ijIRsZB6o/RRRSD62Y4AUJ+tgH3zto4=; b=yZtIW9ps18RtH8rNjaKizgZMOvEr9klXmJoQJVKjTqlGBzn0hnJrenUe4hWpjtYrPe TsCeNlTnYYUsa45lsUHqkfwCV9bnBjhrXXfdRHzgvmdYYjlczqhyBWf3PNccVRU5wyMm lMuvgvYviOiF0A7NG9iWXUd6p+897hPeb3s4QsZKS9bdP/v4KuyFAB0UJFCib3Vo1A7Q QnnVfT//LhfqzhcS+s2c2fIZKj3CXyGZ58SJblEon+2vvvShcnr11K3+3eZF1TMKkdb/ dtEigMJtsq7lgfX7wnmx9nbL+xOykgEcpXXDzRGbGWelw/r5oXQaHZf9NRxCNnsu274J kUtw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q21-20020a170906389500b006df76385d85si8898989ejd.549.2022.03.22.06.41.01; Tue, 22 Mar 2022 06:41:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E595968B0F6; Tue, 22 Mar 2022 15:40:30 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BD40C68B177 for ; Tue, 22 Mar 2022 15:40:24 +0200 (EET) Received: by mail-ej1-f41.google.com with SMTP id yy13so36290251ejb.2 for ; Tue, 22 Mar 2022 06:40:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UkWiP+I+3JMZccj0pnDMzQTxPSZIm8rT97doMbMY0pw=; b=AdS52fv5K3q06ekbS5BJws6/Ls9a2ny1ANfeIDw527eCc1/4WI0p3/5D0xzOXbcWit 1/PCaIrYaapvH7SHW71cijL45/QiyWYRTRZwFMBX+bY4EK/UYab/GM3mjH765srcA2qO xxAQOhgPZKizn8bYYSRpi/aqwpf5A+Yn+l4spmB2+bnDIEftF+m+dmFTrQJrxLQs+F1g G0saNoYkQ3XCS2bnoUwr8fCGoTl+mPqOG2PtLe1LGvAMj1AtDdj1oOYILzkYCDjIO25d hTbagGfUbdNlxgC8wY0lfvXf+tjrNqTsuglWexHIcCP6aD5p4KFxm2msjMwzYtLHmx8p /FTQ== X-Gm-Message-State: AOAM530+AXT0QqKFCWEDSi/hUTlryYvb9u2LASDvFb9DeJKtr3tNvYbW 1SiwAyuVii2/vp58hxnxJGZwU0g2b6U9DtU1 X-Received: by 2002:a17:907:3e92:b0:6df:cb08:14f5 with SMTP id hs18-20020a1709073e9200b006dfcb0814f5mr17985877ejc.308.1647956423642; Tue, 22 Mar 2022 06:40:23 -0700 (PDT) Received: from localhost.localdomain (82-171-134-41.fixed.kpn.net. [82.171.134.41]) by smtp.gmail.com with ESMTPSA id z6-20020a056402274600b004194fc1b7casm2125829edd.48.2022.03.22.06.40.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Mar 2022 06:40:23 -0700 (PDT) From: toots@rastageeks.org To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Mar 2022 14:39:55 +0100 Message-Id: <20220322133957.81743-5-toots@rastageeks.org> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220322133957.81743-1-toots@rastageeks.org> References: <20220322133957.81743-1-toots@rastageeks.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/6] libavdevice/avfoundation.m: use setAudioSettings, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Romain Beauxis Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: UN08VoJKYlJr From: Romain Beauxis Previous version of these changes used the AudioConverter API to perform audio conversion explicitly however, it was found to be bug prone with issues seemingly coming from the underlying OS. This fixes: https://trac.ffmpeg.org/ticket/9502 --- libavdevice/avfoundation.m | 200 ++++++++++++------------------------- 1 file changed, 63 insertions(+), 137 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index af52246bf3..055e8f62e4 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -96,6 +96,11 @@ AVRational framerate; int width, height; + int channels; + int big_endian; + int sample_rate; + enum AVSampleFormat sample_format; + int capture_cursor; int capture_mouse_clicks; int capture_raw_data; @@ -115,17 +120,6 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; - enum AVPixelFormat pixel_format; AVCaptureSession *capture_session; @@ -304,7 +298,6 @@ static void destroy_context(AVFContext* ctx) ctx->avf_audio_delegate = NULL; av_freep(&ctx->url); - av_freep(&ctx->audio_buffer); pthread_mutex_destroy(&ctx->frame_lock); @@ -680,88 +673,61 @@ static int get_video_config(AVFormatContext *s) static int get_audio_config(AVFormatContext *s) { AVFContext *ctx = (AVFContext*)s->priv_data; - CMFormatDescriptionRef format_desc; - AVStream* stream = avformat_new_stream(s, NULL); + AVStream* stream; + int bits_per_sample, is_float; - if (!stream) { - return 1; - } + enum AVCodecID codec_id = av_get_pcm_codec(ctx->sample_format, ctx->big_endian); - // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { - CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); + if (codec_id == AV_CODEC_ID_NONE) { + av_log(ctx, AV_LOG_ERROR, "Error: invalid sample format!\n"); + return AVERROR(EINVAL); } - lock_frames(ctx); - - ctx->audio_stream_index = stream->index; - - avpriv_set_pts_info(stream, 64, 1, avf_time_base); - - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - - if (!basic_desc) { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format not available\n"); - return 1; + switch (ctx->sample_format) { + case AV_SAMPLE_FMT_S16: + bits_per_sample = 16; + is_float = 0; + break; + case AV_SAMPLE_FMT_S32: + bits_per_sample = 32; + is_float = 0; + break; + case AV_SAMPLE_FMT_FLT: + bits_per_sample = 32; + is_float = 1; + break; + default: + av_log(ctx, AV_LOG_ERROR, "Error: invalid sample format!\n"); + unlock_frames(ctx); + return AVERROR(EINVAL); } - stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; - stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; - } else { + [ctx->audio_output setAudioSettings:@{ + AVFormatIDKey: @(kAudioFormatLinearPCM), + AVLinearPCMBitDepthKey: @(bits_per_sample), + AVLinearPCMIsFloatKey: @(is_float), + AVLinearPCMIsBigEndianKey: @(ctx->big_endian), + AVNumberOfChannelsKey: @(ctx->channels), + AVLinearPCMIsNonInterleaved: @NO, + AVSampleRateKey: @(ctx->sample_rate) + }]; + + stream = avformat_new_stream(s, NULL); + if (!stream) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + return -1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); - return 1; - } - } + avpriv_set_pts_info(stream, 64, 1, avf_time_base); - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; + stream->codecpar->sample_rate = ctx->sample_rate; + av_channel_layout_default(&stream->codecpar->ch_layout, ctx->channels); + stream->codecpar->codec_id = codec_id; - unlock_frames(ctx); + ctx->audio_stream_index = stream->index; + unlock_frames(ctx); return 0; } @@ -1065,6 +1031,7 @@ static int avf_read_header(AVFormatContext *s) goto fail; } if (audio_device && add_audio_device(s, audio_device)) { + goto fail; } [ctx->capture_session startRunning]; @@ -1140,6 +1107,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1183,7 +1151,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1197,19 +1165,17 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { + int status = av_new_packet(pkt, buffer_size); + if (status < 0) { unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + return status; } - if (av_new_packet(pkt, block_buffer_size) < 0) { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { unlock_frames(ctx); return AVERROR(EIO); } @@ -1225,54 +1191,10 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; - - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } - - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - } - CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); @@ -1297,6 +1219,10 @@ static int avf_close(AVFormatContext *s) } static const AVOption options[] = { + { "channels", "number of audio channels", offsetof(AVFContext, channels), AV_OPT_TYPE_INT, {.i64=2}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { "sample_rate", "audio sample rate", offsetof(AVFContext, sample_rate), AV_OPT_TYPE_INT, {.i64=44100}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { "big_endian", "return big endian samples for audio data", offsetof(AVFContext, big_endian), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_ENCODING_PARAM }, + { "sample_format", "audio sample format", offsetof(AVFContext, sample_format), AV_OPT_TYPE_INT, {.i64=AV_SAMPLE_FMT_S16}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, { "list_devices", "list available devices", offsetof(AVFContext, list_devices), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_DECODING_PARAM }, { "video_device_index", "select video device by index for devices with same name (starts at 0)", offsetof(AVFContext, video_device_index), AV_OPT_TYPE_INT, {.i64 = -1}, -1, INT_MAX, AV_OPT_FLAG_DECODING_PARAM }, { "audio_device_index", "select audio device by index for devices with same name (starts at 0)", offsetof(AVFContext, audio_device_index), AV_OPT_TYPE_INT, {.i64 = -1}, -1, INT_MAX, AV_OPT_FLAG_DECODING_PARAM },