From patchwork Tue Mar 22 13:39:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 34892 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:ab0:5fda:0:0:0:0:0 with SMTP id g26csp680081uaj; Tue, 22 Mar 2022 06:40:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzW4WbhXZjF6XAUGhnQVjs0cpzzW/hCI4CBrL4Ve4bEo5XZKxqz3UVbw+pkEPHRRJacJCso X-Received: by 2002:a50:fb05:0:b0:419:2600:e439 with SMTP id d5-20020a50fb05000000b004192600e439mr16017965edq.71.1647956432212; Tue, 22 Mar 2022 06:40:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647956432; cv=none; d=google.com; s=arc-20160816; b=POBE7i/gdaoV4l3u0NnuN/Yyl62d7h/GcNFHH7QiBITRvwE3vYKmheB58P+ztjRy1L JlB7yEZgj+a2vCY6rX+567IFYsxxb2LIJNEGIOovMuyp+BOica/AQ2xYVwfWfmY8CYFd 2Hb3Yd+UxLRQnWrVaaolpHqUmbhQREmqtHoeEDn4oNb1afU4BW+z6kye+64cwGHywubE 6DsSpHxI57C0zJIzYWFgwSrzvQ1LagwTnJU+Dy5ravmAolYnoq/TY+Tp1BVP/N32KEai dvQ7i1ycmekid5gvLzT5fXbAYTwdQxCU3UxrK2jaOya9WtpmTjRmOLPrNeTLoQHGZ5UO BZEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=WOW2zxl915Yk3uxY8ppVmzMJ8JDlnhVgTJ0E18hmPcA=; b=XUJfvZSNXICvkFXJ+Xx36PyNl7IwdWxXoy9HpsBDdqF61n9+fQKI/u20MqdaObHnM8 EytoNr+by4ISd9d3t8XpZQqLakI3mFkgiJF3sSQawVJs2yPfbwq4XVpQTf+vZMDFTgfs Fa+INBqsVghjA1lAuIHkZUbO5P1D7QOfDvjvMY7luCjmrCgRsZzl6hm8jKoKiZe+yVnK 7fYFWnn1xTEhNn+XYVmPVqkF7uMhvq2o5g7P9xZf2BNtjIkBLHxIDwMXc/ygilAkMhAE wFoaoA9V6RQYVqhkGOSb4XgX1mCtZjXs9ny76TiZc1zm0WzFTDpktMJgVRjhLPCHb8+f pAxw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 13-20020a17090601cd00b006df76385bf4si9269064ejj.148.2022.03.22.06.40.31; Tue, 22 Mar 2022 06:40:32 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CFD1668B15A; Tue, 22 Mar 2022 15:40:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B15B468ADD4 for ; Tue, 22 Mar 2022 15:40:18 +0200 (EET) Received: by mail-ed1-f48.google.com with SMTP id h1so21694852edj.1 for ; Tue, 22 Mar 2022 06:40:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J9GOLLxh8UmQL8SowJd1iOHeER281UVgAHRfEg5H5dg=; b=I+T6lhjliqXomgNRVLODRqA7Le8TGIbpQMTIqzslEImYV9vqwT02LSbhAhNjJMPpbg bxS3F3zyWGxUTtnFUdNXuGCZHDxevzQ+n61FqziT/8PLdIBLnJXnBSWQ745fEVhWtaho zq3GZwlwVvyI5xZTn1RmfUjEDZU5J5iJqymCBr9cXdwcHcV7vGb7prwZxyObwWfdD/sm n7F0muYvEiz3xWyj0ijbepCiypBumZwH1YpWsc0XXitEMWq5uGhwNNfCJYPYLU98enyf KAXPm7WAwAjoOg061mSxVvTBKzYMbaFy12IE6lskR+v6kUk3Ica7Xyjeu4RmbWSbN/3X Ntpg== X-Gm-Message-State: AOAM532VNNiZmsyU0sDJ98XTkBrmMm9XUO4V3gzWDe4FCP2aFyRzPk3G BrMpDbv8gXMXoA/7CTtzoUvdjrrvLxQTcQil X-Received: by 2002:a05:6402:11d2:b0:419:4da5:ed71 with SMTP id j18-20020a05640211d200b004194da5ed71mr7335571edw.272.1647956417615; Tue, 22 Mar 2022 06:40:17 -0700 (PDT) Received: from localhost.localdomain (82-171-134-41.fixed.kpn.net. [82.171.134.41]) by smtp.gmail.com with ESMTPSA id z6-20020a056402274600b004194fc1b7casm2125829edd.48.2022.03.22.06.40.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Mar 2022 06:40:16 -0700 (PDT) From: toots@rastageeks.org To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Mar 2022 14:39:52 +0100 Message-Id: <20220322133957.81743-2-toots@rastageeks.org> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220322133957.81743-1-toots@rastageeks.org> References: <20220322133957.81743-1-toots@rastageeks.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/6] Fix dshow device name/description X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Romain Beauxis Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: RVvjOgdkJbFd From: Romain Beauxis diff --git a/libavdevice/dshow.c b/libavdevice/dshow.c index 6039578ff9..4ee3f6e194 100644 --- a/libavdevice/dshow.c +++ b/libavdevice/dshow.c @@ -552,8 +552,8 @@ dshow_cycle_devices(AVFormatContext *avctx, ICreateDevEnum *devenum, if (!device) goto fail; - device->device_name = av_strdup(friendly_name); - device->device_description = av_strdup(unique_name); + device->device_name = av_strdup(unique_name); + device->device_description = av_strdup(friendly_name); if (!device->device_name || !device->device_description) goto fail; From patchwork Tue Mar 22 13:39:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 34893 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:ab0:5fda:0:0:0:0:0 with SMTP id g26csp680103uaj; Tue, 22 Mar 2022 06:40:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzCS3AWRgpQnXOR9KOWt7tGcnJt6wF5prFjHBbyVUVXpaZENKYbm6TIVU39PcrKfK10bUVO X-Received: by 2002:a17:907:8d17:b0:6df:c072:2f66 with SMTP id tc23-20020a1709078d1700b006dfc0722f66mr18028111ejc.444.1647956441930; Tue, 22 Mar 2022 06:40:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647956441; cv=none; d=google.com; s=arc-20160816; b=e4SKovFajd6NlZWskD9B0BXdop2kDuEd27p8D7DPOFFwM4zK0kVmtbQK86x3l93uHL GCN/zvzAQ/bT1nbHHzg3rTbRyMzvgy4Fb0e/CSGNiLJ27jIQKFVGtqQqS8AuJPrqOV5O R2WCoW/bpoPQVB/c3HHzbte/pYdWo6tLP+z8i4zJ10358cpSV8UJDFpdBBTHWjltBoGV L3cQ/hboUR+ZHKdleDLMINGfB4o1WzVCPdwztCfiXBzpEdfHv4YpK2yLjJH3tjzn7Bgj ETj61QHOL4KkhK3LYj+zFe/2oupz01DVE0jpjV434oDPmQTuS/0H+Ut6i469qho7ttLh qgsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=QKoAAvXU8DfellwxFZrCQwqYrhY7CSVwIlmmyOxVKdc=; b=W+mPbAwXVclDnql6bapYNrt2x5ItAmVLmlN5uFzNPsX0qZtDWvWCkPbIXbdrkXshNi xFIDfgOLJs3I0rvjk5DdnAuTYTMIvje3qhxNLtfNLF4ZJz3P81mYqcBD6f730kHHS+wW KBa21PCCEaRxYg00nXzmWZuVNpo3fmkOtTytTTolvQnkM5S9G/64DzUXgYwEwsCHDG4T ZakLuiIodvt+wW3HoAYeV475UPzfX+dJXXmJQAcETB2Q5BFjxIcpDS8Y6pLuSGe/ZwRA WfVwTDIUB8f4Ns2bADGVlTnFF6+ii4G3fZRzP2uECVbll2K6cU5u2sIVTdEgsDzsSN6n AXlg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c1-20020a50e3c1000000b00418c2b5bf72si14657040edm.596.2022.03.22.06.40.41; Tue, 22 Mar 2022 06:40:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EBDDC68B00F; Tue, 22 Mar 2022 15:40:26 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BB85668B0F6 for ; Tue, 22 Mar 2022 15:40:20 +0200 (EET) Received: by mail-ej1-f51.google.com with SMTP id pv16so36343008ejb.0 for ; Tue, 22 Mar 2022 06:40:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/PO9ettZfVYrmzd6zHZ9yogWaJS/iD4eon3KI/ITwDk=; b=0X6FtOptc9qb8L7nJuUwRMcoo7z6c0aUajysloHzDKCoLTwWE4f8lU1SIGFNgRovGL TzXH33qMMnM9VRq2Hq4VqhGvtvQ0kLml9e3VMRddlYsTl7n9peoamgLET2RXdrT1R/Wk n2L1I+F2lr0Ddlw9Q87MDcOS7XZ1WOCG1WBVhglllZLZxmMUBhkw1PcNO2fH7W085/Tg rcu013/7CKpzKVnaAm8315Ud7kNEULASbh11rQcI6lJAzUmlbctjgbwG8w+wxotoarqW 7U0OT4xReeYvubGi9TgeO5vCXiWyx0rJQH16guKbDaTLRyzx1fI64ld4sb5Q5jlFCCCH 5a8A== X-Gm-Message-State: AOAM530z2yj6+0J6os0F5ZNr/F1laQs5cAY8CJWR7zZRFzbJqqiEunvN S3DNgvjcAKp2d5NsgCSxcMuuC0KQ61RZH4QA X-Received: by 2002:a17:907:7fa2:b0:6d8:2397:42 with SMTP id qk34-20020a1709077fa200b006d823970042mr25692212ejc.218.1647956419747; Tue, 22 Mar 2022 06:40:19 -0700 (PDT) Received: from localhost.localdomain (82-171-134-41.fixed.kpn.net. [82.171.134.41]) by smtp.gmail.com with ESMTPSA id z6-20020a056402274600b004194fc1b7casm2125829edd.48.2022.03.22.06.40.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Mar 2022 06:40:19 -0700 (PDT) From: toots@rastageeks.org To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Mar 2022 14:39:53 +0100 Message-Id: <20220322133957.81743-3-toots@rastageeks.org> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220322133957.81743-1-toots@rastageeks.org> References: <20220322133957.81743-1-toots@rastageeks.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/6] Use appropriate method for device discovery, fix crash with bogus device index. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Romain Beauxis Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: TnEr63s9YzzY From: Romain Beauxis diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index c9de93f774..719276cabf 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -27,6 +27,7 @@ #import #include +#include #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -771,8 +772,34 @@ static int avf_read_header(AVFormatContext *s) AVCaptureDevice *video_device = nil; AVCaptureDevice *audio_device = nil; // Find capture device - NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo]; - NSArray *devices_muxed = [AVCaptureDevice devicesWithMediaType:AVMediaTypeMuxed]; +#if defined(__MAC_10_15) || (TARGET_OS_IPHONE && defined(__IPHONE_10_0)) + AVCaptureDeviceDiscoverySession *discoverySession = + [AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:@[ +#if TARGET_OS_IPHONE + AVCaptureDeviceTypeBuiltInDualCamera, + AVCaptureDeviceTypeBuiltInDualWideCamera, + AVCaptureDeviceTypeBuiltInUltraWideCamera, + AVCaptureDeviceTypeBuiltInTrueDepthCamera, + AVCaptureDeviceTypeBuiltInTelephotoCamera, +#endif + AVCaptureDeviceTypeBuiltInWideAngleCamera, + AVCaptureDeviceTypeExternalUnknown + ] + mediaType:NULL + position:AVCaptureDevicePositionUnspecified]; + + NSMutableArray *devices = [NSMutableArray array]; + NSMutableArray *devices_muxed = [NSMutableArray array]; + for (AVCaptureDevice *device in [discoverySession devices]) { + if ([device hasMediaType:AVMediaTypeVideo]) + [devices addObject:device]; + else if ([device hasMediaType:AVMediaTypeMuxed]) + [devices_muxed addObject:device]; + } +#else + NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo]; + NSArray *devices_muxed = [AVCaptureDevice devicesWithMediaType:AVMediaTypeMuxed]; +#endif ctx->num_video_devices = [devices count] + [devices_muxed count]; @@ -782,6 +809,21 @@ static int avf_read_header(AVFormatContext *s) CGGetActiveDisplayList(0, NULL, &num_screens); #endif + NSArray *audio_devices; +#if defined(__MAC_10_15) || (TARGET_OS_IPHONE && defined(__IPHONE_10_0)) + discoverySession = + [AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:@[ + AVCaptureDeviceTypeBuiltInMicrophone, + AVCaptureDeviceTypeExternalUnknown + ] + mediaType:AVMediaTypeAudio + position:AVCaptureDevicePositionUnspecified]; + + audio_devices = [discoverySession devices]; +#else + audio_devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; +#endif + // List devices if requested if (ctx->list_devices) { int index = 0; @@ -807,8 +849,7 @@ static int avf_read_header(AVFormatContext *s) #endif av_log(ctx, AV_LOG_INFO, "AVFoundation audio devices:\n"); - devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; - for (AVCaptureDevice *device in devices) { + for (AVCaptureDevice *device in audio_devices) { const char *name = [[device localizedName] UTF8String]; int index = [devices indexOfObject:device]; av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); @@ -833,9 +874,12 @@ static int avf_read_header(AVFormatContext *s) if (ctx->video_device_index < ctx->num_video_devices) { if (ctx->video_device_index < [devices count]) { video_device = [devices objectAtIndex:ctx->video_device_index]; - } else { + } else if (ctx->video_device_index - [devices count] < [devices_muxed count]) { video_device = [devices_muxed objectAtIndex:(ctx->video_device_index - [devices count])]; ctx->video_is_muxed = 1; + } else { + av_log(ctx, AV_LOG_ERROR, "Invalid video device index\n"); + goto fail; } } else if (ctx->video_device_index < ctx->num_video_devices + num_screens) { #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -931,9 +975,7 @@ static int avf_read_header(AVFormatContext *s) // get audio device if (ctx->audio_device_index >= 0) { - NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; - - if (ctx->audio_device_index >= [devices count]) { + if (ctx->audio_device_index >= [audio_devices count]) { av_log(ctx, AV_LOG_ERROR, "Invalid audio device index\n"); goto fail; } @@ -944,15 +986,14 @@ static int avf_read_header(AVFormatContext *s) if (!strncmp(ctx->audio_filename, "default", 7)) { audio_device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeAudio]; } else { - NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]; - - for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->audio_filename, [[device localizedName] UTF8String], strlen(ctx->audio_filename))) { - audio_device = device; - break; + for (AVCaptureDevice *device in audio_devices) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->audio_filename, name, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } } } - } if (!audio_device) { av_log(ctx, AV_LOG_ERROR, "Audio device not found\n"); From patchwork Tue Mar 22 13:39:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 34894 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:ab0:5fda:0:0:0:0:0 with SMTP id g26csp680130uaj; Tue, 22 Mar 2022 06:40:52 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyP9vsE9e28bSsAVd1HMDe3KXLb/MIX/ThKFDF8P/Pzu7JbhCkl1e3RZLIE5S6Kxr/Mfa7X X-Received: by 2002:a05:6402:34d5:b0:416:2f46:48f with SMTP id w21-20020a05640234d500b004162f46048fmr28444517edc.88.1647956452061; Tue, 22 Mar 2022 06:40:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647956452; cv=none; d=google.com; s=arc-20160816; b=P31bQapgxOnFuFATvUv+3vt2icXF+ASpsZi0Go0N683qjvbkQXYgUReMP4tAAaiVz+ jrrOJm7oST6AzCWMFHX670g1P0O/sBj47m3ddF9sbiKHMGf03QLMt6jzGocFgX3RqXBO /KPDbbtZaqR+FHmhLHXNj6+TUfyvpX8U/UO7+AKaAISAvt3QgZXimfB6k38waajZg6Ie Vkd2LGsowr8Ac+gfwevNrZIJY5MC6tUb+jcr235GF2+E42+4BdAUGzk8bwgMBPWzRd7k bhVQqhpMebo46u93ugn2+WhlVhG4SDRf3bAFaKhoKgmI8xkigcxioBAAmKzuz970V6HG UjSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=Z0Jjek2G4DhBiVSPgp8nvYAYXbtgYTFlGFk/+84PB5g=; b=VUKPUMEyodTBOwSVUOuStEesZzmP3eVGYNhuMzWbLgiRTzXBlOZJLpsjuWLExKSp2r U4bAvDMFEro2MEP3WUQyxlF+ovD10jaDtDHloYBzQRY2UcAshIlDrHkhmmHEe+oGZ4Ug cNctVlhziLcrC+YqzeBXHLALwzDwEFHpS4gLKCB7i+7W06dYMmsDQUktq8uSba0jJLu4 Kla5pFtBofjd9Lrz6J4yQxqJk6dcNICVjEcSgmdjQwSFVuw8RATKNs4I/geED/OuV7Vp 9AadBeg2jb3PLWDTUP832/ypz+e1L8ElwexmIkIsXzLlpwB4atvNAY84eQ1kVKIz+yC9 ZNaw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hb42-20020a170907162a00b006df76385bd5si10576398ejc.117.2022.03.22.06.40.51; Tue, 22 Mar 2022 06:40:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C854E68B1A5; Tue, 22 Mar 2022 15:40:28 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9FD6E68B177 for ; Tue, 22 Mar 2022 15:40:22 +0200 (EET) Received: by mail-ed1-f51.google.com with SMTP id c62so2204328edf.5 for ; Tue, 22 Mar 2022 06:40:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nwxxubwggUA8z1BUIndEKlsCo6aXEfi0TH3pI1/KhrE=; b=pZmHbxtOZz/ZGc3GXdwubbvQnSZFnLNvH9sUgrZ7PpkPseUPuwYXPVoclWm/K+bnw3 UnOVs5si0LsJhNfe5mB11oMwnTsCZMVhjUZPz6WijVlD/gPXeg6hklrnvkzpn6rbW8Gm T81Pb83qRA+fP4jpUStDiEiINp9aEKOHM4Ddf3buKTVUSeYP4NjXENdyQCzLZXynJIHy zr6iy9Vb0SLYYKEPo8Qm5ExCNpxQeZ6+3fKlk53I4OaZ1+LN5kc95NC0AZQHMQr13zCK FhOw5S+axK1lODhvAwoAjVu5Y9ll12SnO24WdzLMy1k5QthhHlBf0HWKam+3f2gz41IF 87JA== X-Gm-Message-State: AOAM531fWuK8fiG658vxPCJYLH3gSovTnw0a2S53PMnZfAp5/XKCOWyn qA0oIeXoQ7FrpRqgl1AK2jSlLY2x5SUV3HZI X-Received: by 2002:a50:f19a:0:b0:418:f94c:fc24 with SMTP id x26-20020a50f19a000000b00418f94cfc24mr26763602edl.34.1647956421587; Tue, 22 Mar 2022 06:40:21 -0700 (PDT) Received: from localhost.localdomain (82-171-134-41.fixed.kpn.net. [82.171.134.41]) by smtp.gmail.com with ESMTPSA id z6-20020a056402274600b004194fc1b7casm2125829edd.48.2022.03.22.06.40.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Mar 2022 06:40:21 -0700 (PDT) From: toots@rastageeks.org To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Mar 2022 14:39:54 +0100 Message-Id: <20220322133957.81743-4-toots@rastageeks.org> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220322133957.81743-1-toots@rastageeks.org> References: <20220322133957.81743-1-toots@rastageeks.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/6] libavdevice/avfoundation.m: Allow to select devices by unique ID X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Romain Beauxis Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: eJEuccGJdc5A From: Romain Beauxis diff --git a/doc/indevs.texi b/doc/indevs.texi index 9d8020311a..858c0fa4e4 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -114,7 +114,7 @@ The input filename has to be given in the following syntax: -i "[[VIDEO]:[AUDIO]]" @end example The first entry selects the video input while the latter selects the audio input. -The stream has to be specified by the device name or the device index as shown by the device list. +The stream has to be specified by the device name, index or ID as shown by the device list. Alternatively, the video and/or audio input device can be chosen by index using the @option{ -video_device_index @@ -127,7 +127,9 @@ and/or device name or index given in the input filename. All available devices can be enumerated by using @option{-list_devices true}, listing -all device names and corresponding indices. +all device names, corresponding indices and IDs, when available. Device name can be +tricky to use when localized and device index can change when devices are plugged or unplugged. A device +hash, when available, uniquely identifies a device and should not change over time. There are two device name aliases: @table @code diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 719276cabf..af52246bf3 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -40,6 +40,8 @@ #include "libavutil/imgutils.h" #include "avdevice.h" +#define CLEANUP_DEVICE_ID(s) [[s stringByReplacingOccurrencesOfString:@":" withString:@"."] UTF8String] + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -829,21 +831,23 @@ static int avf_read_header(AVFormatContext *s) int index = 0; av_log(ctx, AV_LOG_INFO, "AVFoundation video devices:\n"); for (AVCaptureDevice *device in devices) { - const char *name = [[device localizedName] UTF8String]; - index = [devices indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } for (AVCaptureDevice *device in devices_muxed) { - const char *name = [[device localizedName] UTF8String]; - index = [devices count] + [devices_muxed indexOfObject:device]; - av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); + const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + index = [devices count] + [devices_muxed indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); } #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 if (num_screens > 0) { CGDirectDisplayID screens[num_screens]; CGGetActiveDisplayList(num_screens, screens, &num_screens); for (int i = 0; i < num_screens; i++) { - av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d\n", ctx->num_video_devices + i, i); + av_log(ctx, AV_LOG_INFO, "[%d] Capture screen %d (ID: AvfilterAvfoundationCaptureScreen%d)\n", ctx->num_video_devices + i, i, screens[i]); } } #endif @@ -851,7 +855,9 @@ static int avf_read_header(AVFormatContext *s) av_log(ctx, AV_LOG_INFO, "AVFoundation audio devices:\n"); for (AVCaptureDevice *device in audio_devices) { const char *name = [[device localizedName] UTF8String]; + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); int index = [devices indexOfObject:device]; + av_log(ctx, AV_LOG_INFO, "[%d] %s (ID: %s)\n", index, name, uniqueId); av_log(ctx, AV_LOG_INFO, "[%d] %s\n", index, name); } goto fail; @@ -919,14 +925,29 @@ static int avf_read_header(AVFormatContext *s) } else { // looking for video inputs for (AVCaptureDevice *device in devices) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; break; } } // looking for muxed inputs for (AVCaptureDevice *device in devices_muxed) { - if (!strncmp(ctx->video_filename, [[device localizedName] UTF8String], strlen(ctx->video_filename))) { + const char *name = [[device localizedName] UTF8String]; + if (!strncmp(ctx->video_filename, name, strlen(ctx->video_filename))) { + video_device = device; + ctx->video_is_muxed = 1; + break; + } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->video_filename, uniqueId, strlen(ctx->video_filename))) { video_device = device; ctx->video_is_muxed = 1; break; @@ -937,10 +958,23 @@ static int avf_read_header(AVFormatContext *s) // looking for screen inputs if (!video_device) { int idx; + CGDirectDisplayID screens[num_screens]; + CGGetActiveDisplayList(num_screens, screens, &num_screens); + AVCaptureScreenInput* capture_screen_input = NULL; + if(sscanf(ctx->video_filename, "Capture screen %d", &idx) && idx < num_screens) { - CGDirectDisplayID screens[num_screens]; - CGGetActiveDisplayList(num_screens, screens, &num_screens); - AVCaptureScreenInput* capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:screens[idx]] autorelease]; + } + + if(sscanf(ctx->video_filename, "AvfilterAvfoundationCaptureScreen%d", &idx)) { + for (int i = 0; i < num_screens; i++) { + if (screens[i] == idx) { + capture_screen_input = [[[AVCaptureScreenInput alloc] initWithDisplayID:idx] autorelease]; + } + } + } + + if (capture_screen_input) { video_device = (AVCaptureDevice*) capture_screen_input; ctx->video_device_index = ctx->num_video_devices + idx; ctx->video_is_screen = 1; @@ -992,6 +1026,12 @@ static int avf_read_header(AVFormatContext *s) audio_device = device; break; } + + const char *uniqueId = CLEANUP_DEVICE_ID([device uniqueID]); + if (!strncmp(ctx->audio_filename, uniqueId, strlen(ctx->audio_filename))) { + audio_device = device; + break; + } } } From patchwork Tue Mar 22 13:39:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 34895 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:ab0:5fda:0:0:0:0:0 with SMTP id g26csp680150uaj; Tue, 22 Mar 2022 06:41:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzaCuUGtW3Ozv0q2B/0stkGi+XUf578ISN3xCJ2atOvxWV0O9CWszDuxwd6ZEhLtZxhsseX X-Received: by 2002:aa7:c704:0:b0:418:ee8f:3fd0 with SMTP id i4-20020aa7c704000000b00418ee8f3fd0mr28472225edq.248.1647956461714; Tue, 22 Mar 2022 06:41:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647956461; cv=none; d=google.com; s=arc-20160816; b=RAqkz7LibbtBIy23rK7nZQNhmsHDe9xGu/lOYDyctQ0Gj4F1pkcVe+UWgKFPL/8vnB g1iW9E1SvrDziik++VpNKhEjt6nYOsKCg0VZdfBx+udwgDTUxUBsFurEPWBHwXrH8V58 ile/v8iP7pObOZOLw9gLtzohPT85QBmCcF+7CAUJjbEsWpPen9RIa+g/61uP4G3n581F vMNzyypB8CNunTPd/X23Wr3jugJESRorMxL/SlMbjKkTnZo4N+qu4BEaiDGNwN53ZkRQ l79b3JVPMzfEodONm9kEzRv7gvJIHK67MejU2shQkAxmCQDhcS60LFS9kJw47XYifxLB 4diQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=LXmzMno9onS8ijIRsZB6o/RRRSD62Y4AUJ+tgH3zto4=; b=yZtIW9ps18RtH8rNjaKizgZMOvEr9klXmJoQJVKjTqlGBzn0hnJrenUe4hWpjtYrPe TsCeNlTnYYUsa45lsUHqkfwCV9bnBjhrXXfdRHzgvmdYYjlczqhyBWf3PNccVRU5wyMm lMuvgvYviOiF0A7NG9iWXUd6p+897hPeb3s4QsZKS9bdP/v4KuyFAB0UJFCib3Vo1A7Q QnnVfT//LhfqzhcS+s2c2fIZKj3CXyGZ58SJblEon+2vvvShcnr11K3+3eZF1TMKkdb/ dtEigMJtsq7lgfX7wnmx9nbL+xOykgEcpXXDzRGbGWelw/r5oXQaHZf9NRxCNnsu274J kUtw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q21-20020a170906389500b006df76385d85si8898989ejd.549.2022.03.22.06.41.01; Tue, 22 Mar 2022 06:41:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E595968B0F6; Tue, 22 Mar 2022 15:40:30 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BD40C68B177 for ; Tue, 22 Mar 2022 15:40:24 +0200 (EET) Received: by mail-ej1-f41.google.com with SMTP id yy13so36290251ejb.2 for ; Tue, 22 Mar 2022 06:40:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UkWiP+I+3JMZccj0pnDMzQTxPSZIm8rT97doMbMY0pw=; b=AdS52fv5K3q06ekbS5BJws6/Ls9a2ny1ANfeIDw527eCc1/4WI0p3/5D0xzOXbcWit 1/PCaIrYaapvH7SHW71cijL45/QiyWYRTRZwFMBX+bY4EK/UYab/GM3mjH765srcA2qO xxAQOhgPZKizn8bYYSRpi/aqwpf5A+Yn+l4spmB2+bnDIEftF+m+dmFTrQJrxLQs+F1g G0saNoYkQ3XCS2bnoUwr8fCGoTl+mPqOG2PtLe1LGvAMj1AtDdj1oOYILzkYCDjIO25d hTbagGfUbdNlxgC8wY0lfvXf+tjrNqTsuglWexHIcCP6aD5p4KFxm2msjMwzYtLHmx8p /FTQ== X-Gm-Message-State: AOAM530+AXT0QqKFCWEDSi/hUTlryYvb9u2LASDvFb9DeJKtr3tNvYbW 1SiwAyuVii2/vp58hxnxJGZwU0g2b6U9DtU1 X-Received: by 2002:a17:907:3e92:b0:6df:cb08:14f5 with SMTP id hs18-20020a1709073e9200b006dfcb0814f5mr17985877ejc.308.1647956423642; Tue, 22 Mar 2022 06:40:23 -0700 (PDT) Received: from localhost.localdomain (82-171-134-41.fixed.kpn.net. [82.171.134.41]) by smtp.gmail.com with ESMTPSA id z6-20020a056402274600b004194fc1b7casm2125829edd.48.2022.03.22.06.40.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Mar 2022 06:40:23 -0700 (PDT) From: toots@rastageeks.org To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Mar 2022 14:39:55 +0100 Message-Id: <20220322133957.81743-5-toots@rastageeks.org> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220322133957.81743-1-toots@rastageeks.org> References: <20220322133957.81743-1-toots@rastageeks.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/6] libavdevice/avfoundation.m: use setAudioSettings, extend supported formats X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Romain Beauxis Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: UN08VoJKYlJr From: Romain Beauxis Previous version of these changes used the AudioConverter API to perform audio conversion explicitly however, it was found to be bug prone with issues seemingly coming from the underlying OS. This fixes: https://trac.ffmpeg.org/ticket/9502 --- libavdevice/avfoundation.m | 200 ++++++++++++------------------------- 1 file changed, 63 insertions(+), 137 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index af52246bf3..055e8f62e4 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -96,6 +96,11 @@ AVRational framerate; int width, height; + int channels; + int big_endian; + int sample_rate; + enum AVSampleFormat sample_format; + int capture_cursor; int capture_mouse_clicks; int capture_raw_data; @@ -115,17 +120,6 @@ int num_video_devices; - int audio_channels; - int audio_bits_per_sample; - int audio_float; - int audio_be; - int audio_signed_integer; - int audio_packed; - int audio_non_interleaved; - - int32_t *audio_buffer; - int audio_buffer_size; - enum AVPixelFormat pixel_format; AVCaptureSession *capture_session; @@ -304,7 +298,6 @@ static void destroy_context(AVFContext* ctx) ctx->avf_audio_delegate = NULL; av_freep(&ctx->url); - av_freep(&ctx->audio_buffer); pthread_mutex_destroy(&ctx->frame_lock); @@ -680,88 +673,61 @@ static int get_video_config(AVFormatContext *s) static int get_audio_config(AVFormatContext *s) { AVFContext *ctx = (AVFContext*)s->priv_data; - CMFormatDescriptionRef format_desc; - AVStream* stream = avformat_new_stream(s, NULL); + AVStream* stream; + int bits_per_sample, is_float; - if (!stream) { - return 1; - } + enum AVCodecID codec_id = av_get_pcm_codec(ctx->sample_format, ctx->big_endian); - // Take stream info from the first frame. - while (ctx->audio_frames_captured < 1) { - CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); + if (codec_id == AV_CODEC_ID_NONE) { + av_log(ctx, AV_LOG_ERROR, "Error: invalid sample format!\n"); + return AVERROR(EINVAL); } - lock_frames(ctx); - - ctx->audio_stream_index = stream->index; - - avpriv_set_pts_info(stream, 64, 1, avf_time_base); - - format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame); - const AudioStreamBasicDescription *basic_desc = CMAudioFormatDescriptionGetStreamBasicDescription(format_desc); - - if (!basic_desc) { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format not available\n"); - return 1; + switch (ctx->sample_format) { + case AV_SAMPLE_FMT_S16: + bits_per_sample = 16; + is_float = 0; + break; + case AV_SAMPLE_FMT_S32: + bits_per_sample = 32; + is_float = 0; + break; + case AV_SAMPLE_FMT_FLT: + bits_per_sample = 32; + is_float = 1; + break; + default: + av_log(ctx, AV_LOG_ERROR, "Error: invalid sample format!\n"); + unlock_frames(ctx); + return AVERROR(EINVAL); } - stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; - stream->codecpar->sample_rate = basic_desc->mSampleRate; - stream->codecpar->channels = basic_desc->mChannelsPerFrame; - stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); - - ctx->audio_channels = basic_desc->mChannelsPerFrame; - ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel; - ctx->audio_float = basic_desc->mFormatFlags & kAudioFormatFlagIsFloat; - ctx->audio_be = basic_desc->mFormatFlags & kAudioFormatFlagIsBigEndian; - ctx->audio_signed_integer = basic_desc->mFormatFlags & kAudioFormatFlagIsSignedInteger; - ctx->audio_packed = basic_desc->mFormatFlags & kAudioFormatFlagIsPacked; - ctx->audio_non_interleaved = basic_desc->mFormatFlags & kAudioFormatFlagIsNonInterleaved; - - if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_float && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_F32BE : AV_CODEC_ID_PCM_F32LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 16 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S16BE : AV_CODEC_ID_PCM_S16LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 24 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S24BE : AV_CODEC_ID_PCM_S24LE; - } else if (basic_desc->mFormatID == kAudioFormatLinearPCM && - ctx->audio_signed_integer && - ctx->audio_bits_per_sample == 32 && - ctx->audio_packed) { - stream->codecpar->codec_id = ctx->audio_be ? AV_CODEC_ID_PCM_S32BE : AV_CODEC_ID_PCM_S32LE; - } else { + [ctx->audio_output setAudioSettings:@{ + AVFormatIDKey: @(kAudioFormatLinearPCM), + AVLinearPCMBitDepthKey: @(bits_per_sample), + AVLinearPCMIsFloatKey: @(is_float), + AVLinearPCMIsBigEndianKey: @(ctx->big_endian), + AVNumberOfChannelsKey: @(ctx->channels), + AVLinearPCMIsNonInterleaved: @NO, + AVSampleRateKey: @(ctx->sample_rate) + }]; + + stream = avformat_new_stream(s, NULL); + if (!stream) { unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "audio format is not supported\n"); - return 1; + return -1; } - if (ctx->audio_non_interleaved) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - ctx->audio_buffer_size = CMBlockBufferGetDataLength(block_buffer); - ctx->audio_buffer = av_malloc(ctx->audio_buffer_size); - if (!ctx->audio_buffer) { - unlock_frames(ctx); - av_log(s, AV_LOG_ERROR, "error allocating audio buffer\n"); - return 1; - } - } + avpriv_set_pts_info(stream, 64, 1, avf_time_base); - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; + stream->codecpar->sample_rate = ctx->sample_rate; + av_channel_layout_default(&stream->codecpar->ch_layout, ctx->channels); + stream->codecpar->codec_id = codec_id; - unlock_frames(ctx); + ctx->audio_stream_index = stream->index; + unlock_frames(ctx); return 0; } @@ -1065,6 +1031,7 @@ static int avf_read_header(AVFormatContext *s) goto fail; } if (audio_device && add_audio_device(s, audio_device)) { + goto fail; } [ctx->capture_session startRunning]; @@ -1140,6 +1107,7 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { + OSStatus ret; AVFContext* ctx = (AVFContext*)s->priv_data; do { @@ -1183,7 +1151,7 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = copy_cvpixelbuffer(s, image_buffer, pkt); } else { status = 0; - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { status = AVERROR(EIO); } @@ -1197,19 +1165,17 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) } } else if (ctx->current_audio_frame != nil) { CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); - int block_buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (!block_buffer || !block_buffer_size) { - unlock_frames(ctx); - return AVERROR(EIO); - } + size_t buffer_size = CMBlockBufferGetDataLength(block_buffer); - if (ctx->audio_non_interleaved && block_buffer_size > ctx->audio_buffer_size) { + int status = av_new_packet(pkt, buffer_size); + if (status < 0) { unlock_frames(ctx); - return AVERROR_BUFFER_TOO_SMALL; + return status; } - if (av_new_packet(pkt, block_buffer_size) < 0) { + ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); + if (ret != kCMBlockBufferNoErr) { unlock_frames(ctx); return AVERROR(EIO); } @@ -1225,54 +1191,10 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - if (ctx->audio_non_interleaved) { - int sample, c, shift, num_samples; - - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, ctx->audio_buffer); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - - num_samples = pkt->size / (ctx->audio_channels * (ctx->audio_bits_per_sample >> 3)); - - // transform decoded frame into output format - #define INTERLEAVE_OUTPUT(bps) \ - { \ - int##bps##_t **src; \ - int##bps##_t *dest; \ - src = av_malloc(ctx->audio_channels * sizeof(int##bps##_t*)); \ - if (!src) { \ - unlock_frames(ctx); \ - return AVERROR(EIO); \ - } \ - \ - for (c = 0; c < ctx->audio_channels; c++) { \ - src[c] = ((int##bps##_t*)ctx->audio_buffer) + c * num_samples; \ - } \ - dest = (int##bps##_t*)pkt->data; \ - shift = bps - ctx->audio_bits_per_sample; \ - for (sample = 0; sample < num_samples; sample++) \ - for (c = 0; c < ctx->audio_channels; c++) \ - *dest++ = src[c][sample] << shift; \ - av_freep(&src); \ - } - - if (ctx->audio_bits_per_sample <= 16) { - INTERLEAVE_OUTPUT(16) - } else { - INTERLEAVE_OUTPUT(32) - } - } else { - OSStatus ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); - if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); - } - } - CFRelease(ctx->current_audio_frame); ctx->current_audio_frame = nil; + + unlock_frames(ctx); } else { pkt->data = NULL; unlock_frames(ctx); @@ -1297,6 +1219,10 @@ static int avf_close(AVFormatContext *s) } static const AVOption options[] = { + { "channels", "number of audio channels", offsetof(AVFContext, channels), AV_OPT_TYPE_INT, {.i64=2}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { "sample_rate", "audio sample rate", offsetof(AVFContext, sample_rate), AV_OPT_TYPE_INT, {.i64=44100}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { "big_endian", "return big endian samples for audio data", offsetof(AVFContext, big_endian), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_ENCODING_PARAM }, + { "sample_format", "audio sample format", offsetof(AVFContext, sample_format), AV_OPT_TYPE_INT, {.i64=AV_SAMPLE_FMT_S16}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, { "list_devices", "list available devices", offsetof(AVFContext, list_devices), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_DECODING_PARAM }, { "video_device_index", "select video device by index for devices with same name (starts at 0)", offsetof(AVFContext, video_device_index), AV_OPT_TYPE_INT, {.i64 = -1}, -1, INT_MAX, AV_OPT_FLAG_DECODING_PARAM }, { "audio_device_index", "select audio device by index for devices with same name (starts at 0)", offsetof(AVFContext, audio_device_index), AV_OPT_TYPE_INT, {.i64 = -1}, -1, INT_MAX, AV_OPT_FLAG_DECODING_PARAM }, From patchwork Tue Mar 22 13:39:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 34896 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:ab0:5fda:0:0:0:0:0 with SMTP id g26csp680172uaj; Tue, 22 Mar 2022 06:41:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy+ZTibvQv01+nuTvDFE5jVh6w5Zf2oFgBtu2oKSGkn4K75+WLeHhwVLPpI/sSiEPlQBtXO X-Received: by 2002:a17:907:1b0e:b0:6da:81ae:a798 with SMTP id mp14-20020a1709071b0e00b006da81aea798mr26007049ejc.699.1647956472346; Tue, 22 Mar 2022 06:41:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647956472; cv=none; d=google.com; s=arc-20160816; b=F2ZdAltUjhF0wu3T1Hef2Ee1dKhDcanrTsc8TrVdmDcaRLUHTernDQkUljyqeTuwTY 9yv4ScCJJX6sh/PPec/D8cliifs9O482MdbVqQ9YU9RmlHKXYCFBlB0B0t5EbqIpopK4 2yFx9JH5JxhoKDE5b07LOVIMZYImg4fIkoZEwkTySNzVfx/EjoXUIeZWBFnTRB9eSemD o5zWWaZV9IRMXuMb/tZntpijK14D9Z8p1xSSl8WFT+ZHnhkO+M7IEhuc+opKTNAjkJmC DdzGQ4LgFY+WwX+e797J+hTmHDxUYWqgd78qfg6QchOWHn1svjP4ixXbxYa7zx9Ae5yK dzAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=IatPzpJllNXNOSbTn3NQCopFIJQX7Zew4Ojuh4AwJ1Q=; b=bbcejlbQhbAGHUtNQGAa6Yr+5BLFWukvEPV5KOmeI15VGJqJApe95voRUp57ZyZzZT eo4DbAOWk+i1jfLYY2PW08H4M8wjDnoz8lhTsBS7rNeLzi+TF90D1k56UHHETaX29px2 d7WJF8dvYnaFqUhzeLNnLwnxF4aj9TOhQgeOaJguAlGahb09l0UZzHogKQhCOKfoGjKL MOYB8W3REio8emxUIW5inqZFOQrm2vm2i37Eb9TE+WvjfyCZR/wfLJ9p8vm1aS9520gY 3kaBrMLfvNHsdy/ifcTu3ruO6Vyt7TZwj7857f3xO75yOIFbENpJt1EUczc4XevHgDo/ 7AAA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t17-20020a508d51000000b00418eee96003si12284747edt.315.2022.03.22.06.41.11; Tue, 22 Mar 2022 06:41:12 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EB67468B1C7; Tue, 22 Mar 2022 15:40:34 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 84C1E68B177 for ; Tue, 22 Mar 2022 15:40:28 +0200 (EET) Received: by mail-ej1-f48.google.com with SMTP id dr20so35801718ejc.6 for ; Tue, 22 Mar 2022 06:40:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T5jphvpwPLH3vILuIEqge8O+8Wth8lSGvaaLPjIJGgM=; b=5CX15XxGPJkYvZ6XE3yOIFc96/wqvjR6V5ECjMq/sFJbYThwotFyj0w+Ik6jJLimZO iUU//Q3gkPIELW/vJ1ITJphvnJtJ0qneUudW1R3q5GBDspiErFiONi1N4Uv1HNTl6hJU CyIe0CncHb3k3z45Q1WeFhcI0WU7LELfK7n9v9JCg+PMLatSG88NCIoj4O1TENvgUKJN E4Y1srdR/IzZBQEMlhnKOCo4sRRQFIa1R81FlPItVvkMzImz6MawnxZgs9ZX1FylSqpa UxPu2XhsiGkaVesN9cD+Egl27OLDQVuGeaP7vyPPI5LYcgRkstyb9dlammYMGFgiSpLO T0lQ== X-Gm-Message-State: AOAM5309o6LsB1UWpanLldFEfTxsQ7tFsJsP+WXre7v3dOTV8DZ09TGi RzrKEA0BiECb0A1CQgw7k8cYrntzxFQRxZ4T X-Received: by 2002:a17:907:6d8f:b0:6e0:1512:913b with SMTP id sb15-20020a1709076d8f00b006e01512913bmr8906619ejc.491.1647956426180; Tue, 22 Mar 2022 06:40:26 -0700 (PDT) Received: from localhost.localdomain (82-171-134-41.fixed.kpn.net. [82.171.134.41]) by smtp.gmail.com with ESMTPSA id z6-20020a056402274600b004194fc1b7casm2125829edd.48.2022.03.22.06.40.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Mar 2022 06:40:25 -0700 (PDT) From: toots@rastageeks.org To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Mar 2022 14:39:56 +0100 Message-Id: <20220322133957.81743-6-toots@rastageeks.org> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220322133957.81743-1-toots@rastageeks.org> References: <20220322133957.81743-1-toots@rastageeks.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 5/6] libavdevice/avfoundation.m: Replace mutex-based concurrency handling in avfoundation.m by a thread-safe fifo queue with maximum length X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Romain Beauxis Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: N8Gr6exS859y From: Romain Beauxis This issue was particularly noticeable when working with audio input. --- libavdevice/avfoundation.m | 232 ++++++++++++++++--------------------- 1 file changed, 100 insertions(+), 132 deletions(-) diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m index 055e8f62e4..1825b9787a 100644 --- a/libavdevice/avfoundation.m +++ b/libavdevice/avfoundation.m @@ -26,8 +26,8 @@ */ #import -#include #include +#import #include "libavutil/channel_layout.h" #include "libavutil/pixdesc.h" @@ -42,6 +42,13 @@ #define CLEANUP_DEVICE_ID(s) [[s stringByReplacingOccurrencesOfString:@":" withString:@"."] UTF8String] +static void av_log_avfoundation(void *s, int lvl, const char *str, OSStatus err) { + NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init]; + av_log(s, lvl, "AVFoundation: %s, %s\n", str, + [[[NSError errorWithDomain:NSOSStatusErrorDomain code:err userInfo:nil] localizedDescription] UTF8String]); + [pool release]; +} + static const int avf_time_base = 1000000; static const AVRational avf_time_base_q = { @@ -87,9 +94,6 @@ { AVClass* class; - int frames_captured; - int audio_frames_captured; - pthread_mutex_t frame_lock; id avf_delegate; id avf_audio_delegate; @@ -125,8 +129,9 @@ AVCaptureSession *capture_session; AVCaptureVideoDataOutput *video_output; AVCaptureAudioDataOutput *audio_output; - CMSampleBufferRef current_frame; - CMSampleBufferRef current_audio_frame; + + CMSimpleQueueRef frames_queue; + int max_frames; AVCaptureDevice *observed_device; #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 @@ -135,16 +140,6 @@ int observed_quit; } AVFContext; -static void lock_frames(AVFContext* ctx) -{ - pthread_mutex_lock(&ctx->frame_lock); -} - -static void unlock_frames(AVFContext* ctx) -{ - pthread_mutex_unlock(&ctx->frame_lock); -} - /** FrameReciever class - delegate for AVCaptureSession */ @interface AVFFrameReceiver : NSObject @@ -222,17 +217,13 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)videoFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); + OSStatus ret = CMSimpleQueueEnqueue(_context->frames_queue, videoFrame); - if (_context->current_frame != nil) { - CFRelease(_context->current_frame); + if (ret != noErr) { + av_log_avfoundation(_context, AV_LOG_DEBUG, "Error while queueing video frame", ret); } - _context->current_frame = (CMSampleBufferRef)CFRetain(videoFrame); - - unlock_frames(_context); - - ++_context->frames_captured; + CFRetain(videoFrame); } @end @@ -266,17 +257,13 @@ - (void) captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)audioFrame fromConnection:(AVCaptureConnection *)connection { - lock_frames(_context); + OSStatus ret = CMSimpleQueueEnqueue(_context->frames_queue, audioFrame); - if (_context->current_audio_frame != nil) { - CFRelease(_context->current_audio_frame); + if (ret != noErr) { + av_log_avfoundation(_context, AV_LOG_DEBUG, "Error while queueing audio frame", ret); } - _context->current_audio_frame = (CMSampleBufferRef)CFRetain(audioFrame); - - unlock_frames(_context); - - ++_context->audio_frames_captured; + CFRetain(audioFrame); } @end @@ -291,6 +278,19 @@ static void destroy_context(AVFContext* ctx) [ctx->avf_delegate release]; [ctx->avf_audio_delegate release]; + CMSampleBufferRef frame; + + if (ctx->frames_queue) { + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->frames_queue); + while (frame) { + CFRelease(frame); + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->frames_queue); + } + + CFRelease(ctx->frames_queue); + ctx->frames_queue = NULL; + } + ctx->capture_session = NULL; ctx->video_output = NULL; ctx->audio_output = NULL; @@ -298,12 +298,6 @@ static void destroy_context(AVFContext* ctx) ctx->avf_audio_delegate = NULL; av_freep(&ctx->url); - - pthread_mutex_destroy(&ctx->frame_lock); - - if (ctx->current_frame) { - CFRelease(ctx->current_frame); - } } static int parse_device_name(AVFormatContext *s) @@ -343,15 +337,14 @@ static int configure_video_device(AVFormatContext *s, AVCaptureDevice *video_dev NSObject *format = nil; NSObject *selected_range = nil; NSObject *selected_format = nil; + CMFormatDescriptionRef formatDescription; + CMVideoDimensions dimensions; // try to configure format by formats list // might raise an exception if no format list is given // (then fallback to default, no configuration) @try { for (format in [video_device valueForKey:@"formats"]) { - CMFormatDescriptionRef formatDescription; - CMVideoDimensions dimensions; - formatDescription = (CMFormatDescriptionRef) [format performSelector:@selector(formatDescription)]; dimensions = CMVideoFormatDescriptionGetDimensions(formatDescription); @@ -378,6 +371,9 @@ static int configure_video_device(AVFormatContext *s, AVCaptureDevice *video_dev goto unsupported_format; } + ctx->width = dimensions.width; + ctx->height = dimensions.height; + if (!selected_range) { av_log(s, AV_LOG_ERROR, "Selected framerate (%f) is not supported by the device.\n", framerate); @@ -625,47 +621,21 @@ static int add_audio_device(AVFormatContext *s, AVCaptureDevice *audio_device) static int get_video_config(AVFormatContext *s) { AVFContext *ctx = (AVFContext*)s->priv_data; - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - CGSize image_buffer_size; AVStream* stream = avformat_new_stream(s, NULL); if (!stream) { return 1; } - // Take stream info from the first frame. - while (ctx->frames_captured < 1) { - CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES); - } - - lock_frames(ctx); - ctx->video_stream_index = stream->index; avpriv_set_pts_info(stream, 64, 1, avf_time_base); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); - - if (image_buffer) { - image_buffer_size = CVImageBufferGetEncodedSize(image_buffer); - - stream->codecpar->codec_id = AV_CODEC_ID_RAWVIDEO; - stream->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; - stream->codecpar->width = (int)image_buffer_size.width; - stream->codecpar->height = (int)image_buffer_size.height; - stream->codecpar->format = ctx->pixel_format; - } else { - stream->codecpar->codec_id = AV_CODEC_ID_DVVIDEO; - stream->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; - stream->codecpar->format = ctx->pixel_format; - } - - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - unlock_frames(ctx); + stream->codecpar->codec_id = AV_CODEC_ID_RAWVIDEO; + stream->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; + stream->codecpar->width = ctx->width; + stream->codecpar->height = ctx->height; + stream->codecpar->format = ctx->pixel_format; return 0; } @@ -698,7 +668,6 @@ static int get_audio_config(AVFormatContext *s) break; default: av_log(ctx, AV_LOG_ERROR, "Error: invalid sample format!\n"); - unlock_frames(ctx); return AVERROR(EINVAL); } @@ -713,10 +682,8 @@ static int get_audio_config(AVFormatContext *s) }]; stream = avformat_new_stream(s, NULL); - if (!stream) { - unlock_frames(ctx); + if (!stream) return -1; - } avpriv_set_pts_info(stream, 64, 1, avf_time_base); @@ -727,7 +694,6 @@ static int get_audio_config(AVFormatContext *s) ctx->audio_stream_index = stream->index; - unlock_frames(ctx); return 0; } @@ -771,8 +737,6 @@ static int avf_read_header(AVFormatContext *s) ctx->num_video_devices = [devices count] + [devices_muxed count]; - pthread_mutex_init(&ctx->frame_lock, NULL); - #if !TARGET_OS_IPHONE && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1070 CGGetActiveDisplayList(0, NULL, &num_screens); #endif @@ -1027,6 +991,14 @@ static int avf_read_header(AVFormatContext *s) // Initialize capture session ctx->capture_session = [[AVCaptureSession alloc] init]; + OSStatus err; + err = CMSimpleQueueCreate(kCFAllocatorDefault, ctx->max_frames, &ctx->frames_queue); + + if (err != noErr) { + av_log_avfoundation(s, AV_LOG_ERROR, "error while creating frame queue", err); + goto fail; + } + if (video_device && add_video_device(s, video_device)) { goto fail; } @@ -1059,7 +1031,7 @@ static int avf_read_header(AVFormatContext *s) destroy_context(ctx); if (ret) return ret; - return AVERROR(EIO); + return AVERROR_EXTERNAL; } static int copy_cvpixelbuffer(AVFormatContext *s, @@ -1108,39 +1080,46 @@ static int copy_cvpixelbuffer(AVFormatContext *s, static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) { OSStatus ret; + int status, length; + CMBlockBufferRef block_buffer; + CMSampleBufferRef frame; + CMFormatDescriptionRef format; AVFContext* ctx = (AVFContext*)s->priv_data; + CMItemCount count; + CMSampleTimingInfo timing_info; + CVImageBufferRef image_buffer; + size_t buffer_size; + AVRational timebase_q; - do { - CVImageBufferRef image_buffer; - CMBlockBufferRef block_buffer; - lock_frames(ctx); + if (CMSimpleQueueGetCount(ctx->frames_queue) < 1) + return AVERROR(EAGAIN); - if (ctx->current_frame != nil) { - int status; - int length = 0; + frame = (CMSampleBufferRef)CMSimpleQueueDequeue(ctx->frames_queue); + format = CMSampleBufferGetFormatDescription(frame); - image_buffer = CMSampleBufferGetImageBuffer(ctx->current_frame); - block_buffer = CMSampleBufferGetDataBuffer(ctx->current_frame); + switch (CMFormatDescriptionGetMediaType(format)) { + case kCMMediaType_Video: + length = 0; + image_buffer = CMSampleBufferGetImageBuffer(frame); + block_buffer = CMSampleBufferGetDataBuffer(frame); if (image_buffer != nil) { length = (int)CVPixelBufferGetDataSize(image_buffer); } else if (block_buffer != nil) { length = (int)CMBlockBufferGetDataLength(block_buffer); } else { - unlock_frames(ctx); + CFRelease(frame); return AVERROR(EINVAL); } - if (av_new_packet(pkt, length) < 0) { - unlock_frames(ctx); - return AVERROR(EIO); + status = av_new_packet(pkt, length); + if (status < 0) { + CFRelease(frame); + return status; } - CMItemCount count; - CMSampleTimingInfo timing_info; - - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_frame, 1, &timing_info, &count) == noErr) { - AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); + if (CMSampleBufferGetOutputSampleTimingInfoArray(frame, 1, &timing_info, &count) == noErr) { + timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } @@ -1153,62 +1132,50 @@ static int avf_read_packet(AVFormatContext *s, AVPacket *pkt) status = 0; ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - status = AVERROR(EIO); + av_log_avfoundation(s, AV_LOG_ERROR, "error while copying buffer data", ret); + status = AVERROR_EXTERNAL; } - } - CFRelease(ctx->current_frame); - ctx->current_frame = nil; - - if (status < 0) { - unlock_frames(ctx); - return status; } - } else if (ctx->current_audio_frame != nil) { - CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(ctx->current_audio_frame); + CFRelease(frame); + + return status; - size_t buffer_size = CMBlockBufferGetDataLength(block_buffer); + case kCMMediaType_Audio: + block_buffer = CMSampleBufferGetDataBuffer(frame); + buffer_size = CMBlockBufferGetDataLength(block_buffer); - int status = av_new_packet(pkt, buffer_size); + status = av_new_packet(pkt, buffer_size); if (status < 0) { - unlock_frames(ctx); + CFRelease(frame); return status; } ret = CMBlockBufferCopyDataBytes(block_buffer, 0, pkt->size, pkt->data); if (ret != kCMBlockBufferNoErr) { - unlock_frames(ctx); - return AVERROR(EIO); + CFRelease(frame); + av_log_avfoundation(s, AV_LOG_ERROR, "error while copying audio data", ret); + return AVERROR_EXTERNAL; } - CMItemCount count; - CMSampleTimingInfo timing_info; - - if (CMSampleBufferGetOutputSampleTimingInfoArray(ctx->current_audio_frame, 1, &timing_info, &count) == noErr) { - AVRational timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); + if (CMSampleBufferGetOutputSampleTimingInfoArray(frame, 1, &timing_info, &count) == noErr) { + timebase_q = av_make_q(1, timing_info.presentationTimeStamp.timescale); pkt->pts = pkt->dts = av_rescale_q(timing_info.presentationTimeStamp.value, timebase_q, avf_time_base_q); } pkt->stream_index = ctx->audio_stream_index; pkt->flags |= AV_PKT_FLAG_KEY; - CFRelease(ctx->current_audio_frame); - ctx->current_audio_frame = nil; - - unlock_frames(ctx); - } else { + CFRelease(frame); + return 0; + default: pkt->data = NULL; - unlock_frames(ctx); - if (ctx->observed_quit) { + if (ctx->observed_quit) return AVERROR_EOF; - } else { - return AVERROR(EAGAIN); - } - } - unlock_frames(ctx); - } while (!pkt->data); + return AVERROR(EAGAIN); + } - return 0; + return AVERROR_BUG; } static int avf_close(AVFormatContext *s) @@ -1233,6 +1200,7 @@ static int avf_close(AVFormatContext *s) { "capture_mouse_clicks", "capture the screen mouse clicks", offsetof(AVFContext, capture_mouse_clicks), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_DECODING_PARAM }, { "capture_raw_data", "capture the raw data from device connection", offsetof(AVFContext, capture_raw_data), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_DECODING_PARAM }, { "drop_late_frames", "drop frames that are available later than expected", offsetof(AVFContext, drop_late_frames), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, AV_OPT_FLAG_DECODING_PARAM }, + { "max_frames", "Maximun length of the queue of pending frames", offsetof(AVFContext, max_frames), AV_OPT_TYPE_INT, {.i64=10}, 0, INT_MAX, AV_OPT_FLAG_DECODING_PARAM }, { NULL }, }; From patchwork Tue Mar 22 13:39:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Romain Beauxis X-Patchwork-Id: 34897 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:ab0:5fda:0:0:0:0:0 with SMTP id g26csp680207uaj; Tue, 22 Mar 2022 06:41:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxPwfwNLpq/MwZmk11aTpMqt3SJU9bibt9sg9loc3twhkQBQ35siwfjZQ+PBuf2+skintkV X-Received: by 2002:a17:906:5d09:b0:6df:d52f:58f6 with SMTP id g9-20020a1709065d0900b006dfd52f58f6mr16402519ejt.721.1647956486662; Tue, 22 Mar 2022 06:41:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647956486; cv=none; d=google.com; s=arc-20160816; b=l6Ot28lgHn5yheSKg/v2q7IzhxS55U0WRr2IHh2vs6VjCqLnK0RuAo3HxjH5zoVyMc 5Tvft7Aa4ER9U/6yjmmHuFm37ozX38KOBuMF3ptHD+IePJflqx9nwqBOxC+WcCAhama2 SL+Zp+hBRY8bAtq9Ecpv43aJbNKSBj2Faeu7VVxjgP2jQnRkTFcgSenxr0zBYqhALydz N8q8wgOzm8muBurj6asdXxYY1n3sIRV6jmU062cmO5XAJwXMXYsBAg6LSN9dtXoyZ8Th y96yITce2n9wyEwH64F2O/i/MCHvFB6PqJcXWfsZsCwfgTpd6ZiNF6SFcCfQSHrPsiUb qnAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:delivered-to; bh=OaD8aNgWLxDbfKhUZq8OE+nG+6dx/hLJF0XTgjoVWvs=; b=dxzx1NyGefudFGLX0Efd1pBcat5Qfm9Qf4qCYGtY3TUoBiEVV7sXchbpO3vbWkobWB GIvRS0HZRsklxabUQdU4wpRPOHahR1IJSCdW8o4U45jh5PW4Ga3yyYrXgJoCmb6iugG+ zhoTN8HViSbhJ+CA/OYuRARjrMYOexTN+mp2238RXDhrkHtBGGyZT7d4Y3p1xUu4k14C mDiM/XMGMWL3i2ELuo7UMzZ7t2CEe1oxdbd3k4w3gcbvmW+sOPeAUnEnObuxxUS7F5Zu 5HIODjGl+w3UjzX5FU4qgkEPpOLT4ZEMZ0261Hd+ZkeL+55IKUrVLpKhkoYE9ErFluIu EBUQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id fy2-20020a170906b7c200b006df76385eebsi9651656ejb.907.2022.03.22.06.41.22; Tue, 22 Mar 2022 06:41:26 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 14F7468B1C5; Tue, 22 Mar 2022 15:40:36 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C05EF68B177 for ; Tue, 22 Mar 2022 15:40:28 +0200 (EET) Received: by mail-ej1-f52.google.com with SMTP id bg10so36265809ejb.4 for ; Tue, 22 Mar 2022 06:40:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S8v3rejqqFiXOR3auaJPl9qtuBQ2YQ4lqfcBMiPjzZI=; b=1Xf2PVi0eOX4Fn3OZdx3DEZ06HyH4VXAwsVIpEmj4AHIp8PZUEHYT838+B752pap6G 32frLEGIO/SDMg0XojdEgFDVogtnkOg2XP0Tc5rEHmSfmqJUyasrqMifVaYz6ySpTdpF m5U9UoRNqTSctD14vYuQS1rAq/2RnWe9Emhu+iPQQ/zYQLeWddWMPjR79t6TQ9V5MSay EPeQ5o8FNDOgrxTPUn6Xcfc5+HmuoFtS+BU80GGSsCDg2KvxAQbS/2A0TvPGyLdVUEqE EkN17sI44N69gpLlmYODlqXJ7eh+aNx3cB9oWYXxXcEYIuiTDiOgP2NseGWuh4DKszPG 9+Rw== X-Gm-Message-State: AOAM533JDrFNp8UZqRT0+MLm9gR7NA+8qeGmExaWEvnOaTi1Z9FQuZ9J Emoab0bJ49Y1t2awwAp7ZkGnOxJfxXGX5hFW X-Received: by 2002:a17:906:2695:b0:6ce:f9c:b476 with SMTP id t21-20020a170906269500b006ce0f9cb476mr25942308ejc.235.1647956427468; Tue, 22 Mar 2022 06:40:27 -0700 (PDT) Received: from localhost.localdomain (82-171-134-41.fixed.kpn.net. [82.171.134.41]) by smtp.gmail.com with ESMTPSA id z6-20020a056402274600b004194fc1b7casm2125829edd.48.2022.03.22.06.40.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Mar 2022 06:40:26 -0700 (PDT) From: toots@rastageeks.org To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Mar 2022 14:39:57 +0100 Message-Id: <20220322133957.81743-7-toots@rastageeks.org> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220322133957.81743-1-toots@rastageeks.org> References: <20220322133957.81743-1-toots@rastageeks.org> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 6/6] Add AudioToolbox audio input device. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Romain Beauxis Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 0dIOJ5WwufoA From: Romain Beauxis diff --git a/configure b/configure index a7953ffc16..37f9f7b80a 100755 --- a/configure +++ b/configure @@ -203,6 +203,7 @@ External library support: --disable-avfoundation disable Apple AVFoundation framework [autodetect] --enable-avisynth enable reading of AviSynth script files [no] --disable-bzlib disable bzlib [autodetect] + --disable-coremedia disable Apple CoreMedia framework [autodetect] --disable-coreimage disable Apple CoreImage framework [autodetect] --enable-chromaprint enable audio fingerprinting with chromaprint [no] --enable-frei0r enable frei0r video filtering [no] @@ -1751,6 +1752,7 @@ EXTERNAL_AUTODETECT_LIBRARY_LIST=" appkit avfoundation bzlib + coremedia coreimage iconv libxcb @@ -3488,6 +3490,8 @@ alsa_outdev_deps="alsa" avfoundation_indev_deps="avfoundation corevideo coremedia pthreads" avfoundation_indev_suggest="coregraphics applicationservices" avfoundation_indev_extralibs="-framework Foundation" +audiotoolbox_indev_deps="coremedia audiotoolbox" +audiotoolbox_indev_extralibs="-framework CoreMedia -framework AudioToolbox" audiotoolbox_outdev_deps="audiotoolbox pthreads" audiotoolbox_outdev_extralibs="-framework AudioToolbox -framework CoreAudio" bktr_indev_deps_any="dev_bktr_ioctl_bt848_h machine_ioctl_bt848_h dev_video_bktr_ioctl_bt848_h dev_ic_bt8xx_h" @@ -6330,6 +6334,7 @@ check_lib camera2ndk "stdbool.h stdint.h camera/NdkCameraManager.h" ACameraManag enabled appkit && check_apple_framework AppKit enabled audiotoolbox && check_apple_framework AudioToolbox enabled avfoundation && check_apple_framework AVFoundation +enabled coremedia && check_apple_framework CoreMedia enabled coreimage && check_apple_framework CoreImage enabled metal && check_apple_framework Metal enabled videotoolbox && check_apple_framework VideoToolbox diff --git a/doc/indevs.texi b/doc/indevs.texi index 858c0fa4e4..8d57a26a5f 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -103,6 +103,41 @@ Set the maximum number of frames to buffer. Default is 5. @end table +@section AudioToolbox + +AudioToolbox input device. + +Allows native input from CoreAudio devices on OSX. + +@subsection Options + +AudioToolbox supports the following options: + +@table @option + +@item channels +Set the number of channels. Default is device's default. + +@item frames_queue_length +Maximum of buffers in the input queue + +@item buffer_frame_size +Buffer frame size, gouverning internal latency + +@item big_endian +Return big endian samples + +@item sample_format +Sample format + +@end table + +@subsection Examples + +@itemize + +@end itemize + @section avfoundation AVFoundation input device. diff --git a/libavdevice/Makefile b/libavdevice/Makefile index 99fea7133a..78d4168521 100644 --- a/libavdevice/Makefile +++ b/libavdevice/Makefile @@ -15,6 +15,7 @@ OBJS-$(HAVE_LIBC_MSVCRT) += file_open.o OBJS-$(CONFIG_ALSA_INDEV) += alsa_dec.o alsa.o timefilter.o OBJS-$(CONFIG_ALSA_OUTDEV) += alsa_enc.o alsa.o OBJS-$(CONFIG_ANDROID_CAMERA_INDEV) += android_camera.o +OBJS-$(CONFIG_AUDIOTOOLBOX_INDEV) += audiotoolbox_dec.o OBJS-$(CONFIG_AUDIOTOOLBOX_OUTDEV) += audiotoolbox.o OBJS-$(CONFIG_AVFOUNDATION_INDEV) += avfoundation.o OBJS-$(CONFIG_BKTR_INDEV) += bktr.o diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c index 22323a0a44..fbecdbb0b2 100644 --- a/libavdevice/alldevices.c +++ b/libavdevice/alldevices.c @@ -26,6 +26,7 @@ extern const AVInputFormat ff_alsa_demuxer; extern const AVOutputFormat ff_alsa_muxer; extern const AVInputFormat ff_android_camera_demuxer; extern const AVOutputFormat ff_audiotoolbox_muxer; +extern const AVInputFormat ff_audiotoolbox_demuxer; extern const AVInputFormat ff_avfoundation_demuxer; extern const AVInputFormat ff_bktr_demuxer; extern const AVOutputFormat ff_caca_muxer; diff --git a/libavdevice/audiotoolbox.m b/libavdevice/audiotoolbox.m index 0cb97b5e46..3216f14607 100644 --- a/libavdevice/audiotoolbox.m +++ b/libavdevice/audiotoolbox.m @@ -84,7 +84,7 @@ static av_cold int at_write_header(AVFormatContext *avctx) AudioObjectPropertyAddress prop; prop.mSelector = kAudioHardwarePropertyDevices; prop.mScope = kAudioObjectPropertyScopeGlobal; - prop.mElement = kAudioObjectPropertyElementMaster; + prop.mElement = 0; err = AudioObjectGetPropertyDataSize(kAudioObjectSystemObject, &prop, 0, NULL, &data_size); if (check_status(avctx, &err, "AudioObjectGetPropertyDataSize devices")) return AVERROR(EINVAL); @@ -173,7 +173,7 @@ static av_cold int at_write_header(AVFormatContext *avctx) device_format.mFormatFlags |= (codecpar->codec_id == AV_CODEC_ID_PCM_S16BE) ? kAudioFormatFlagIsBigEndian : 0; device_format.mFormatFlags |= (codecpar->codec_id == AV_CODEC_ID_PCM_S24BE) ? kAudioFormatFlagIsBigEndian : 0; device_format.mFormatFlags |= (codecpar->codec_id == AV_CODEC_ID_PCM_S32BE) ? kAudioFormatFlagIsBigEndian : 0; - device_format.mChannelsPerFrame = codecpar->channels; + device_format.mChannelsPerFrame = codecpar->ch_layout.nb_channels; device_format.mBitsPerChannel = (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S24BE, AV_CODEC_ID_PCM_S24LE)) ? 24 : (av_get_bytes_per_sample(codecpar->format) << 3); device_format.mBytesPerFrame = (device_format.mBitsPerChannel >> 3) * device_format.mChannelsPerFrame; device_format.mFramesPerPacket = 1; @@ -193,9 +193,9 @@ static av_cold int at_write_header(AVFormatContext *avctx) av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags |= %s\n", (codecpar->codec_id == AV_CODEC_ID_PCM_S24BE) ? "kAudioFormatFlagIsBigEndian" : "0"); av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags |= %s\n", (codecpar->codec_id == AV_CODEC_ID_PCM_S32BE) ? "kAudioFormatFlagIsBigEndian" : "0"); av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags == %i\n", device_format.mFormatFlags); - av_log(ctx, AV_LOG_DEBUG, "device_format.mChannelsPerFrame = %i\n", codecpar->channels); + av_log(ctx, AV_LOG_DEBUG, "device_format.mChannelsPerFrame = %i\n", codecpar->ch_layout.nb_channels); av_log(ctx, AV_LOG_DEBUG, "device_format.mBitsPerChannel = %i\n", av_get_bytes_per_sample(codecpar->format) << 3); - av_log(ctx, AV_LOG_DEBUG, "device_format.mBytesPerFrame = %i\n", (device_format.mBitsPerChannel >> 3) * codecpar->channels); + av_log(ctx, AV_LOG_DEBUG, "device_format.mBytesPerFrame = %i\n", (device_format.mBitsPerChannel >> 3) * codecpar->ch_layout.nb_channels); av_log(ctx, AV_LOG_DEBUG, "device_format.mBytesPerPacket = %i\n", device_format.mBytesPerFrame); av_log(ctx, AV_LOG_DEBUG, "device_format.mFramesPerPacket = %i\n", 1); av_log(ctx, AV_LOG_DEBUG, "device_format.mReserved = %i\n", 0); diff --git a/libavdevice/audiotoolbox_dec.m b/libavdevice/audiotoolbox_dec.m new file mode 100644 index 0000000000..ae80c0e54a --- /dev/null +++ b/libavdevice/audiotoolbox_dec.m @@ -0,0 +1,530 @@ +/* + * AudioToolbox input device + * Copyright (c) 2022 Romain Beauxis + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * AudioToolbox input device + * @author Romain Beauxis + */ + +#import +#import + +#include "libavutil/channel_layout.h" +#include "libavformat/internal.h" +#include "avdevice.h" + +typedef struct { + void *data; + int size; +} buffer_t; + +typedef struct { + AVClass *class; + CMSimpleQueueRef frames_queue; + AudioUnit audio_unit; + AudioStreamBasicDescription record_format; + uint64_t position; + int frames_queue_length; + int buffer_frame_size; + int stream_index; + int big_endian; + enum AVSampleFormat sample_format; + int channels; +} ATContext; + +static int check_status(void *ctx, OSStatus status, const char *msg) { + if (status == noErr) { + av_log(ctx, AV_LOG_DEBUG, "OK: %s\n", msg); + return 0; + } + + NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init]; + NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:status userInfo:nil]; + av_log(ctx, AV_LOG_ERROR, "Error: %s (%s)\n", msg, [[error localizedDescription] UTF8String]); + [pool release]; + return 1; +} + +static OSStatus input_callback(void *priv, + AudioUnitRenderActionFlags *ioActionFlags, + const AudioTimeStamp *inTimeStamp, + UInt32 inBusNumber, + UInt32 inNumberFrames, + AudioBufferList *ioData) { + ATContext *ctx = (ATContext *)priv; + OSStatus err; + + AudioBuffer audio_buffer; + + audio_buffer.mNumberChannels = ctx->channels; + audio_buffer.mDataByteSize = inNumberFrames * ctx->record_format.mBytesPerFrame; + + audio_buffer.mData = av_malloc(audio_buffer.mDataByteSize); + memset(audio_buffer.mData, 0, audio_buffer.mDataByteSize); + + AudioBufferList bufferList; + bufferList.mNumberBuffers = 1; + bufferList.mBuffers[0] = audio_buffer; + + err = AudioUnitRender(ctx->audio_unit, + ioActionFlags, + inTimeStamp, + inBusNumber, + inNumberFrames, + &bufferList); + if (check_status(ctx, err, "AudioUnitRender")) { + av_freep(&audio_buffer.mData); + return err; + } + + buffer_t *buffer = av_malloc(sizeof(buffer_t)); + buffer->data = audio_buffer.mData; + buffer->size = audio_buffer.mDataByteSize; + err = CMSimpleQueueEnqueue(ctx->frames_queue, buffer); + + if (err != noErr) { + av_log(ctx, AV_LOG_DEBUG, "Could not enqueue audio frame!\n"); + return err; + } + + return noErr; +} + +static av_cold int audiotoolbox_read_header(AVFormatContext *avctx) { + ATContext *ctx = (ATContext*)avctx->priv_data; + OSStatus err = noErr; + AudioChannelLayout *channel_layout = NULL; + AudioDeviceID device = kAudioObjectUnknown; + AudioObjectPropertyAddress prop; + UInt32 data_size; + + enum AVCodecID codec_id = av_get_pcm_codec(ctx->sample_format, ctx->big_endian); + + if (codec_id == AV_CODEC_ID_NONE) { + av_log(ctx, AV_LOG_ERROR, "Error: invalid sample format!\n"); + return AVERROR(EINVAL); + } + + prop.mScope = kAudioObjectPropertyScopeGlobal; + prop.mElement = 0; + + if (!strcmp(avctx->url, "default")) { + prop.mSelector = kAudioHardwarePropertyDefaultInputDevice; + data_size = sizeof(AudioDeviceID); + err = AudioObjectGetPropertyData(kAudioObjectSystemObject, &prop, 0, NULL, &data_size, &device); + if (check_status(avctx, err, "AudioObjectGetPropertyData DefaultInputDevice")) + goto fail; + } else { + prop.mSelector = kAudioHardwarePropertyDeviceForUID; + CFStringRef deviceUID = CFStringCreateWithCStringNoCopy(NULL, avctx->url, kCFStringEncodingUTF8, kCFAllocatorNull); + AudioValueTranslation value; + value.mInputData = &deviceUID; + value.mInputDataSize = sizeof(deviceUID);; + value.mOutputData = &device; + value.mOutputDataSize = sizeof(device); + data_size = sizeof(value); + + err = AudioObjectGetPropertyData(kAudioObjectSystemObject, &prop, 0, NULL, &data_size, &value); + CFRelease(deviceUID); + + if (check_status(avctx, err, "AudioObjectGetPropertyData DeviceForUID")) + goto fail; + } + + Float64 sample_rate; + prop.mSelector = kAudioDevicePropertyNominalSampleRate; + prop.mScope = kAudioObjectPropertyScopeInput; + data_size = sizeof(sample_rate); + err = AudioObjectGetPropertyData(device, &prop, 0, NULL, &data_size, &sample_rate); + if (check_status(avctx, err, "AudioObjectGetPropertyData SampleRate")) + goto fail; + + if (!ctx->channels) { + prop.mSelector = kAudioDevicePropertyPreferredChannelLayout; + prop.mScope = kAudioObjectPropertyScopeInput; + UInt32 channel_layout_size; + + err = AudioObjectGetPropertyDataSize(device, &prop, 0, NULL, &channel_layout_size); + if (check_status(avctx, err, "AudioObjectGetPropertyDataSize PreferredChannelLayout")) + goto fail; + + channel_layout = av_malloc(channel_layout_size); + err = AudioObjectGetPropertyData(device, &prop, 0, NULL, &channel_layout_size, channel_layout); + if (check_status(avctx, err, "AudioObjectGetPropertyData PreferredChannelLayout")) + goto fail; + + data_size = sizeof(ctx->channels); + err = AudioFormatGetProperty(kAudioFormatProperty_NumberOfChannelsForLayout, channel_layout_size, channel_layout, &data_size, &ctx->channels); + if (check_status(avctx, err, "AudioFormatGetProperty NumberOfChannelsForLayout")) + goto fail; + } + + ctx->record_format.mFormatID = kAudioFormatLinearPCM; + ctx->record_format.mChannelsPerFrame = ctx->channels; + ctx->record_format.mFormatFlags = kAudioFormatFlagIsPacked; + ctx->record_format.mBitsPerChannel = av_get_bytes_per_sample(ctx->sample_format) << 3; + + if (ctx->big_endian) + ctx->record_format.mFormatFlags |= kAudioFormatFlagIsBigEndian; + + switch (ctx->sample_format) { + case AV_SAMPLE_FMT_S16: + case AV_SAMPLE_FMT_S32: + ctx->record_format.mFormatFlags |= kAudioFormatFlagIsSignedInteger; + break; + case AV_SAMPLE_FMT_FLT: + ctx->record_format.mFormatFlags |= kAudioFormatFlagIsFloat; + break; + default: + av_log(ctx, AV_LOG_ERROR, "Error: invalid sample format!\n"); + goto fail; + } + + av_log(ctx, AV_LOG_DEBUG, "Audio Input: %s\n", avctx->url); + av_log(ctx, AV_LOG_DEBUG, "samplerate: %d\n", (int)sample_rate); + av_log(ctx, AV_LOG_DEBUG, "channels: %d\n", ctx->channels); + av_log(ctx, AV_LOG_DEBUG, "Input format: %s\n", avcodec_get_name(codec_id)); + + data_size = sizeof(ctx->record_format); + err = AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &data_size, &ctx->record_format); + if (check_status(avctx, err, "AudioFormatGetProperty FormatInfo")) + goto fail; + + AudioComponentDescription desc; + AudioComponent comp; + + desc.componentType = kAudioUnitType_Output; + desc.componentSubType = kAudioUnitSubType_HALOutput; + desc.componentManufacturer = kAudioUnitManufacturer_Apple; + desc.componentFlags = 0; + desc.componentFlagsMask = 0; + + comp = AudioComponentFindNext(NULL, &desc); + if (comp == NULL) { + av_log(ctx, AV_LOG_ERROR, "Error: AudioComponentFindNext\n"); + goto fail; + } + + err = AudioComponentInstanceNew(comp, &ctx->audio_unit); + if (check_status(avctx, err, "AudioComponentInstanceNew")) + goto fail; + + UInt32 enableIO = 1; + err = AudioUnitSetProperty(ctx->audio_unit, + kAudioOutputUnitProperty_EnableIO, + kAudioUnitScope_Input, + 1, + &enableIO, + sizeof(enableIO)); + if (check_status(avctx, err, "AudioUnitSetProperty EnableIO")) + goto fail; + + enableIO = 0; + err = AudioUnitSetProperty(ctx->audio_unit, + kAudioOutputUnitProperty_EnableIO, + kAudioUnitScope_Output, + 0, + &enableIO, + sizeof(enableIO)); + if (check_status(avctx, err, "AudioUnitSetProperty EnableIO")) + goto fail; + + err = AudioUnitSetProperty(ctx->audio_unit, + kAudioOutputUnitProperty_CurrentDevice, + kAudioUnitScope_Global, + 0, + &device, + sizeof(device)); + if (check_status(avctx, err, "AudioUnitSetProperty CurrentDevice")) + goto fail; + + err = AudioUnitSetProperty(ctx->audio_unit, + kAudioUnitProperty_StreamFormat, + kAudioUnitScope_Input, + 0, + &ctx->record_format, + sizeof(ctx->record_format)); + if (check_status(avctx, err, "AudioUnitSetProperty StreamFormat")) + goto fail; + + err = AudioUnitSetProperty(ctx->audio_unit, + kAudioUnitProperty_StreamFormat, + kAudioUnitScope_Output, + 1, + &ctx->record_format, + sizeof(ctx->record_format)); + if (check_status(avctx, err, "AudioUnitSetProperty StreamFormat")) + goto fail; + + err = AudioUnitSetProperty(ctx->audio_unit, + kAudioDevicePropertyBufferFrameSize, + kAudioUnitScope_Global, + 0, + &ctx->buffer_frame_size, + sizeof(ctx->buffer_frame_size)); + if (check_status(avctx, err, "AudioUnitSetProperty BufferFrameSize")) + goto fail; + + AURenderCallbackStruct callback = {0}; + callback.inputProc = input_callback; + callback.inputProcRefCon = ctx; + err = AudioUnitSetProperty(ctx->audio_unit, + kAudioOutputUnitProperty_SetInputCallback, + kAudioUnitScope_Global, + 0, + &callback, + sizeof(callback)); + if (check_status(avctx, err, "AudioUnitSetProperty SetInputCallback")) + goto fail; + + err = AudioUnitInitialize(ctx->audio_unit); + if (check_status(avctx, err, "AudioUnitInitialize")) + goto fail; + + err = CMSimpleQueueCreate(kCFAllocatorDefault, ctx->frames_queue_length, &ctx->frames_queue); + if (check_status(avctx, err, "CMSimpleQueueCreate")) + goto fail; + + CFRetain(ctx->frames_queue); + + err = AudioUnitInitialize(ctx->audio_unit); + if (check_status(avctx, err, "AudioUnitInitialize")) + goto fail; + + err = AudioOutputUnitStart(ctx->audio_unit); + if (check_status(avctx, err, "AudioOutputUnitStart")) + goto fail; + + AVStream* stream = avformat_new_stream(avctx, NULL); + stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO; + stream->codecpar->sample_rate = sample_rate; + stream->codecpar->channels = ctx->channels; + stream->codecpar->channel_layout = av_get_default_channel_layout(stream->codecpar->channels); + stream->codecpar->codec_id = codec_id; + + avpriv_set_pts_info(stream, 64, 1, sample_rate); + + ctx->stream_index = stream->index; + ctx->position = 0; + + av_freep(&channel_layout); + return 0; + +fail: + av_freep(&channel_layout); + return AVERROR(EINVAL); +} + +static int audiotoolbox_read_packet(AVFormatContext *avctx, AVPacket *pkt) { + ATContext *ctx = (ATContext*)avctx->priv_data; + + if (CMSimpleQueueGetCount(ctx->frames_queue) < 1) + return AVERROR(EAGAIN); + + buffer_t *buffer = (buffer_t *)CMSimpleQueueDequeue(ctx->frames_queue); + + int status = av_packet_from_data(pkt, buffer->data, buffer->size); + if (status < 0) { + av_freep(&buffer->data); + av_freep(&buffer); + return status; + } + + pkt->stream_index = ctx->stream_index; + pkt->flags |= AV_PKT_FLAG_KEY; + pkt->pts = pkt->dts = ctx->position; + + ctx->position += pkt->size / (ctx->channels * av_get_bytes_per_sample(ctx->sample_format)); + + av_freep(&buffer); + return 0; +} + +static av_cold int audiotoolbox_close(AVFormatContext *avctx) { + ATContext *ctx = (ATContext*)avctx->priv_data; + + if (ctx->audio_unit) { + AudioOutputUnitStop(ctx->audio_unit); + AudioComponentInstanceDispose(ctx->audio_unit); + ctx->audio_unit = NULL; + } + + if (ctx->frames_queue) { + buffer_t *buffer = (buffer_t *)CMSimpleQueueDequeue(ctx->frames_queue); + + while (buffer) { + av_freep(&buffer->data); + av_freep(&buffer); + buffer = (buffer_t *)CMSimpleQueueDequeue(ctx->frames_queue); + } + + CFRelease(ctx->frames_queue); + ctx->frames_queue = NULL; + } + + return 0; +} + +static int audiotoolbox_get_device_list(AVFormatContext *avctx, AVDeviceInfoList *device_list) +{ + OSStatus err = noErr; + CFStringRef device_UID = NULL; + CFStringRef device_name = NULL; + AudioDeviceID *devices = NULL; + AVDeviceInfo *avdevice_info = NULL; + int num_devices, ret; + UInt32 i; + + avdevice_info = av_mallocz(sizeof(AVDeviceInfo)); + + if (!avdevice_info) { + ret = AVERROR(ENOMEM); + goto fail; + } + + avdevice_info->device_name = av_strdup("default"); + avdevice_info->device_description = av_strdup("Default audio input device"); + if (!avdevice_info->device_name || !avdevice_info->device_description) { + ret = AVERROR(ENOMEM); + goto fail; + } + avdevice_info->media_types = av_malloc_array(1, sizeof(enum AVMediaType)); + if (avdevice_info->media_types) { + avdevice_info->nb_media_types = 1; + avdevice_info->media_types[0] = AVMEDIA_TYPE_AUDIO; + } + + if ((ret = av_dynarray_add_nofree(&device_list->devices, + &device_list->nb_devices, avdevice_info)) < 0) + goto fail; + + // get devices + UInt32 data_size = 0; + AudioObjectPropertyAddress prop; + prop.mSelector = kAudioHardwarePropertyDevices; + prop.mScope = kAudioObjectPropertyScopeGlobal; + prop.mElement = 0; + err = AudioObjectGetPropertyDataSize(kAudioObjectSystemObject, &prop, 0, NULL, &data_size); + if (check_status(avctx, err, "AudioObjectGetPropertyDataSize devices")) + return AVERROR(EINVAL); + + num_devices = data_size / sizeof(AudioDeviceID); + devices = av_malloc(data_size); + + err = AudioObjectGetPropertyData(kAudioObjectSystemObject, &prop, 0, NULL, &data_size, devices); + if (check_status(avctx, err, "AudioObjectGetPropertyData devices")) { + ret = AVERROR(EINVAL); + goto fail; + } + + for(i = 0; i < num_devices; ++i) { + prop.mSelector = kAudioDevicePropertyStreams; + prop.mScope = kAudioDevicePropertyScopeInput; + data_size = 0; + + err = AudioObjectGetPropertyDataSize(devices[i], &prop, 0, NULL, &data_size); + if (check_status(avctx, err, "AudioObjectGetPropertyData Streams")) + continue; + + UInt32 streamCount = data_size / sizeof(AudioStreamID); + + if (streamCount <= 0) + continue; + + // UID + data_size = sizeof(device_UID); + prop.mSelector = kAudioDevicePropertyDeviceUID; + err = AudioObjectGetPropertyData(devices[i], &prop, 0, NULL, &data_size, &device_UID); + if (check_status(avctx, err, "AudioObjectGetPropertyData UID")) + continue; + + // name + data_size = sizeof(device_name); + prop.mSelector = kAudioDevicePropertyDeviceNameCFString; + err = AudioObjectGetPropertyData(devices[i], &prop, 0, NULL, &data_size, &device_name); + if (check_status(avctx, err, "AudioObjectGetPropertyData name")) + continue; + + avdevice_info = av_mallocz(sizeof(AVDeviceInfo)); + if (!avdevice_info) { + ret = AVERROR(ENOMEM); + goto fail; + } + avdevice_info->device_name = av_strdup(CFStringGetCStringPtr(device_UID, kCFStringEncodingUTF8)); + avdevice_info->device_description = av_strdup(CFStringGetCStringPtr(device_name, kCFStringEncodingUTF8)); + if (!avdevice_info->device_name || !avdevice_info->device_description) { + ret = AVERROR(ENOMEM); + goto fail; + } + avdevice_info->media_types = av_malloc_array(1, sizeof(enum AVMediaType)); + if (avdevice_info->media_types) { + avdevice_info->nb_media_types = 1; + avdevice_info->media_types[0] = AVMEDIA_TYPE_AUDIO; + } + + if ((ret = av_dynarray_add_nofree(&device_list->devices, + &device_list->nb_devices, avdevice_info)) < 0) + goto fail; + } + + av_freep(&devices); + return 0; + +fail: + av_freep(&devices); + if (avdevice_info) { + av_freep(&avdevice_info->device_name); + av_freep(&avdevice_info->device_description); + av_freep(&avdevice_info); + } + + return ret; +} + +static const AVOption options[] = { + { "channels", "number of audio channels", offsetof(ATContext, channels), AV_OPT_TYPE_INT, {.i64=0}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { "frames_queue_length", "maximum of buffers in the input queue", offsetof(ATContext, frames_queue_length), AV_OPT_TYPE_INT, {.i64=10}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { "buffer_frame_size", "buffer frame size, gouverning internal latency", offsetof(ATContext, buffer_frame_size), AV_OPT_TYPE_INT, {.i64=1024}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { "big_endian", "return big endian samples", offsetof(ATContext, big_endian), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_ENCODING_PARAM }, + { "sample_format", "sample format", offsetof(ATContext, sample_format), AV_OPT_TYPE_INT, {.i64=AV_SAMPLE_FMT_S16}, 0, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM }, + { NULL }, +}; + +static const AVClass audiotoolbox_class = { + .class_name = "AudioToolbox", + .item_name = av_default_item_name, + .option = options, + .version = LIBAVUTIL_VERSION_INT, + .category = AV_CLASS_CATEGORY_DEVICE_AUDIO_INPUT, +}; + +const AVInputFormat ff_audiotoolbox_demuxer = { + .name = "audiotoolbox", + .long_name = NULL_IF_CONFIG_SMALL("AudioToolbox input device"), + .priv_data_size = sizeof(ATContext), + .read_header = audiotoolbox_read_header, + .read_packet = audiotoolbox_read_packet, + .read_close = audiotoolbox_close, + .get_device_list = audiotoolbox_get_device_list, + .flags = AVFMT_NOFILE, + .priv_class = &audiotoolbox_class, +};