From patchwork Tue Dec 28 22:50:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 32937 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp11621017iog; Tue, 28 Dec 2021 14:51:15 -0800 (PST) X-Google-Smtp-Source: ABdhPJw7G8IU4+FUT1R2rWe9rytWtGR11v7nolkfe40GXZW9yKmkTZ5+/D4AeJr9PBqib+brqZqA X-Received: by 2002:a17:906:4e45:: with SMTP id g5mr18526773ejw.257.1640731875374; Tue, 28 Dec 2021 14:51:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640731875; cv=none; d=google.com; s=arc-20160816; b=ebcZaR2qcLzgINOcXoC3zbnsONfFWGo8SE+TWSXKzfutAQlV3XBu6ZaAEmFLNmJMsx qdAMXbEPqNelQ53ZxQqNm9mdNjE4NGAOYaSEWLzeY8NX4YrclZGByfoNcZ5X2QcS+dmb Fse3L3Vcg7ZGLEhAxEDZQ2lN5M8c+fhAVgjy2M2hA2qGTN12LCQI0VCXKK1qqj2ccsmS sb94vVP4T2DNxSWqNVFsDQBLT1uWsNcBZ87dVBuf3HDGIoPU1CmW5I+yOMDjPHbzbA6I RYyhwx4i8lStj0MgdE3YMZTaJuW4uB28FNjt2Yffvb8xhnigwhL7EvRTyA5SiuNZSqCS P1Uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:date:message-id:mime-version:from :delivered-to; bh=2s0mmslwmjFN6vn0BDJtbskxiJo/Cbem701VACrKdWY=; b=nr/425Fl2Db0hIIAwuNrQDWX046M1oB5q/CQHZV5c8oGsElXeZOHfx5rb9W+TKIHq2 fFfZ1xwCYp2FnT3IbyqEuc5qdK/WTc5ZWw2nCzE1Njig3FuvD1yvFaywUPltAHQUFyPm wiOnjUeuTctvy948g8yNW3mQIZidJ6t1NkCPhhXo1xVWIbAqBkZJmYCHyfXnGSZwnceK 3p2MpWngeg8iostCy54S1X95FdYYFj6cFVoeGPZBa7dvpcsR42yAIDgp8joiVuIWh8tL 7pw5osb/oOZBm2wj1V2Q+A8GIamrEyndtkklzMwYKdgBPX/nJUXAqmBBWrmaU/vB2dFH dpQg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s11si8324648ejx.544.2021.12.28.14.51.15; Tue, 28 Dec 2021 14:51:15 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A35A268B00B; Wed, 29 Dec 2021 00:51:12 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BCF6068AE46 for ; Wed, 29 Dec 2021 00:51:04 +0200 (EET) Received: by mail-qt1-f172.google.com with SMTP id bp39so17137355qtb.6 for ; Tue, 28 Dec 2021 14:51:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=RhahyZY96Qboenc9ny9aO1a4t+PYAlVRLfhP3atnuhE=; b=XCRj3qlPcXWi79zqWRgj5XEjwsr2axRFPr4DhlSEsqF+A4npEm2kH/qEvN+e8LfR8G 8dICEybiRQ7/H9qjf7Fd5HBLnF9nY4N8V6yWvt/c8dYHRg2MQRkvqBWitnKiUWzEMU9J nxyerlx+wcwEfjh1HIUO0mPHUwaDUVDUO/ZBeGiwcB8qeptzKlHpwJPCqALV4Ifyn8wk +ApTzGgbAg0Fwgkz1ZsSaarYR4dcANsWIDtcCO67uBE566QPVhGHjRF0yHZVEKT8p73C CHJ4lssZq6EQJX2yKXtbr/CXNbzvT4TZyUlWDKZuM/9wqa8XZRgTU00Du6Sse8XaETlS H3mg== X-Gm-Message-State: AOAM532hDeYF2e9YDQhVz4CFpsF9r7h+Nhh8t7NRKn4Tcv+uwszOb5Su CSWnN0OTvbUwsEBADVcE1Q+7aqCPoyhO4Q== X-Received: by 2002:ac8:7282:: with SMTP id v2mr20398899qto.318.1640731862972; Tue, 28 Dec 2021 14:51:02 -0800 (PST) Received: from smtpclient.apple ([172.58.159.253]) by smtp.gmail.com with ESMTPSA id t18sm16532359qkp.84.2021.12.28.14.51.01 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Dec 2021 14:51:02 -0800 (PST) From: Romain Beauxis Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Message-Id: <0A0624A8-43ED-4484-85FE-55D944BAF0E2@rastageeks.org> Date: Tue, 28 Dec 2021 16:50:39 -0600 To: FFmpeg development discussions and patches X-Mailer: Apple Mail (2.3693.40.0.1.81) Subject: [FFmpeg-devel] [PATCH v6 02/03] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 5f9DGEJuLiaD This is the second patch of a series of 3 that cleanup and enhance the avfoundation implementation for libavdevice. Changes: v2: None v3: Switched queue implementation to CMSimpleQueue v4: None v5: Fix indentation/wrapping V6: Fix audio/video frame queue cleanup logic This patch fixes the concurrency model. Avfoundation runs its own producing thread to send produced frames and ffmpeg runs its own thread to consume them. The existing implementation stores the last transmitted frame and uses a mutex to avoid concurrent access. However, this leads to situations where upcoming frames can be dropped if the ffmpeg thread is acessing the latest frame. This happens even when the thread would otherwise catch up and process frames fast enought. This patches changes this implementation to use a buffer queue with a max queue length and encapsulated thread-safety. This greatly simplifies the logic of the calling code and gives the consuming thread a chance to process all frames concurrently to the producing thread while avoiding memory leaks. Signed-off-by: Romain Beauxis --- libavdevice/avfoundation.m | 169 +++++++++++++++++-------------------- 1 file changed, 76 insertions(+), 93 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 79c9207cfa..f7bd5be404 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,7 +26,7 @@ */ #import -#include +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -80,13 +80,12 @@ { AV_PIX_FMT_NONE, 0 } }; +#define MAX_QUEUED_FRAMES 10 + typedef struct { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -121,8 +120,8 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + CMSimpleQueueRef audio_frames_queue; + CMSimpleQueueRef video_frames_queue; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -131,16 +130,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -218,17 +207,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); - } - - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + CFRetain(videoFrame); + CMSimpleQueueEnqueue(_context->video_frames_queue, videoFrame); } @end @@ -262,17 +242,8 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); - - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); - } - - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + CFRetain(audioFrame); + CMSimpleQueueEnqueue(_context->audio_frames_queue, audioFrame); } @end @@ -287,6 +258,30 @@ static void destroy_context(AVFContext* ctx) [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; + CMSampleBufferRef frame; + + if (ctx->video_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + } + + CFRelease(ctx->video_frames_queue); + ctx->video_frames_queue = NULL; + } + + if (ctx->audio_frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + } + + CFRelease(ctx->audio_frames_queue); + ctx->audio_frames_queue = NULL; + } + ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; @@ -297,12 +292,6 @@ static void destroy_context(AVFContext* ctx) AudioConverterDispose(ctx->audio_converter); ctx->audio_converter = NULL; } - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static void parse_device_name(AVFormatContext *s) @@ -630,18 +619,18 @@ static int get_video_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->video_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer) { image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); @@ -657,10 +646,7 @@ static int get_video_config(AVFormatContext *s) stream->codecpar->format = ctx->pixel_format; } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -680,27 +666,27 @@ static int get_audio_config(AVFormatContext *s) } // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { + while (CMSimpleQueueGetCount(ctx->audio_frames_queue) < 1) { CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); } - lock_frames(ctx); + CMSampleBufferRef frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); ctx->audio_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); + format_desc = CMSampleBufferGetFormatDescription(frame); const AudioStreamBasicDescription *input_format = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); if (!input_format) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "audio format not available\n"); return 1; } if (input_format->mFormatID != kAudioFormatLinearPCM) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the moment\n"); return 1; } @@ -778,16 +764,13 @@ static int get_audio_config(AVFormatContext *s) if (must_convert) { OSStatus ret = AudioConverterNew(input_format, &output_format, &ctx->audio_converter); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(frame); av_log(s, AV_LOG_ERROR, "Error while allocating audio converter\n"); return 1; } } - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(frame); return 0; } @@ -805,8 +788,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1007,6 +988,19 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + OSStatus ret; + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->video_frames_queue); + + if (ret != noErr) { + goto fail; + } + + ret = CMSimpleQueueCreate(kCFAllocatorDefault, MAX_QUEUED_FRAMES, &ctx->audio_frames_queue); + + if (ret != noErr) { + goto fail; + } + if (video_device && add_video_device(s, video_device)) { goto fail; } @@ -1088,35 +1082,31 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) AVFContext* ctx = (AVFContext*)s->priv_data; do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); - - if (ctx->current_frame != nil) { + if (1 <= CMSimpleQueueGetCount(ctx->video_frames_queue)) { int status; int length = 0; - - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + CMSampleBufferRef video_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->video_frames_queue); + CVImageBufferRef image_buffer = CMSampleBufferGetImageBuffer(video_frame);; + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(video_frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EINVAL); } if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); + CFRelease(video_frame); return AVERROR(EIO); } CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(video_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1133,15 +1123,14 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = AVERROR(EIO); } } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; + CFRelease(video_frame); if (status < 0) { - unlock_frames(ctx); return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + } else if (1 <= CMSimpleQueueGetCount(ctx->audio_frames_queue)) { + CMSampleBufferRef audio_frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->audio_frames_queue); + CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(audio_frame); size_t input_size = CMBlockBufferGetDataLength(block_buffer); int buffer_size = input_size / ctx->audio_buffers; @@ -1151,12 +1140,12 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) UInt32 size = sizeof(output_size); ret = AudioConverterGetProperty(ctx->audio_converter, kAudioConverterPropertyCalculateOutputBufferSize, &size, &output_size); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } if (av_new_packet(pkt, output_size) < 0) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1173,7 +1162,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) if (ret != kCMBlockBufferNoErr) { av_free(input_buffer); - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1191,7 +1180,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) av_free(input_buffer); if (ret != noErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } @@ -1199,7 +1188,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } else { ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); + CFRelease(audio_frame); return AVERROR(EIO); } } @@ -1207,7 +1196,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) CMItemCount count; CMSampleTimingInfo timing_info; - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { + if (CMSampleBufferGetOutputSampleTimingInfoArray(audio_frame, 1, &timing_info, &count) == noErr) { AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1215,21 +1204,15 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); + CFRelease(audio_frame); } else { pkt->data = NULL; - unlock_frames(ctx); if (ctx->observed_quit) { return AVERROR_EOF; } else { return AVERROR(EAGAIN); } } - - unlock_frames(ctx); } while (!pkt->data); return 0;