[FFmpeg-devel,v2] lavdevice: Add AudioToolbox output device.

Hi,

v2 due to Nicolas remarks. One device, several formats.

Thanks,
Thilo
From 64b100d7b4cfe780c273d6e7d8b940a8427cfdc9 Mon Sep 17 00:00:00 2001
From: Thilo Borgmann <thilo.borgmann@mail.de>
Date: Mon, 8 Jun 2020 13:04:11 +0200
Subject: [PATCH] lavdevice: Add AudioToolbox output device.

---
 configure                  |   3 +
 libavdevice/Makefile       |   1 +
 libavdevice/alldevices.c   |   1 +
 libavdevice/audiotoolbox.m | 299 +++++++++++++++++++++++++++++++++++++
 4 files changed, 304 insertions(+)
 create mode 100644 libavdevice/audiotoolbox.m

Thilo Borgmann (12020-06-08):
> v2 due to Nicolas remarks. One device, several formats.

I cannot comment on appleisms, but here are a few general remarks:

> From 64b100d7b4cfe780c273d6e7d8b940a8427cfdc9 Mon Sep 17 00:00:00 2001
> From: Thilo Borgmann <thilo.borgmann@mail.de>
> Date: Mon, 8 Jun 2020 13:04:11 +0200
> Subject: [PATCH] lavdevice: Add AudioToolbox output device.
> 
> ---

>  configure                  |   3 +
>  libavdevice/Makefile       |   1 +
>  libavdevice/alldevices.c   |   1 +
>  libavdevice/audiotoolbox.m | 299 +++++++++++++++++++++++++++++++++++++

Missing documentation.

>  4 files changed, 304 insertions(+)
>  create mode 100644 libavdevice/audiotoolbox.m
> 
> diff --git a/configure b/configure
> index f97cad0298..1aab998d8e 100755
> --- a/configure
> +++ b/configure
> @@ -3366,6 +3366,8 @@ alsa_outdev_deps="alsa"
>  avfoundation_indev_deps="avfoundation corevideo coremedia pthreads"
>  avfoundation_indev_suggest="coregraphics applicationservices"
>  avfoundation_indev_extralibs="-framework Foundation"
> +audiotoolbox_outdev_deps="audiotoolbox pthreads"
> +audiotoolbox_outdev_extralibs="-framework AudioToolbox -framework CoreAudio"
>  bktr_indev_deps_any="dev_bktr_ioctl_bt848_h machine_ioctl_bt848_h dev_video_bktr_ioctl_bt848_h dev_ic_bt8xx_h"
>  caca_outdev_deps="libcaca"
>  decklink_deps_any="libdl LoadLibrary"
> @@ -6151,6 +6153,7 @@ enabled videotoolbox && check_apple_framework VideoToolbox
>  check_apple_framework CoreFoundation
>  check_apple_framework CoreMedia
>  check_apple_framework CoreVideo
> +check_apple_framework CoreAudio
>  
>  enabled avfoundation && {
>      disable coregraphics applicationservices
> diff --git a/libavdevice/Makefile b/libavdevice/Makefile
> index 6ea62b914e..0dfe47a1f4 100644
> --- a/libavdevice/Makefile
> +++ b/libavdevice/Makefile
> @@ -15,6 +15,7 @@ OBJS-$(CONFIG_SHARED)                    += reverse.o
>  OBJS-$(CONFIG_ALSA_INDEV)                += alsa_dec.o alsa.o timefilter.o
>  OBJS-$(CONFIG_ALSA_OUTDEV)               += alsa_enc.o alsa.o
>  OBJS-$(CONFIG_ANDROID_CAMERA_INDEV)      += android_camera.o
> +OBJS-$(CONFIG_AUDIOTOOLBOX_OUTDEV)       += audiotoolbox.o
>  OBJS-$(CONFIG_AVFOUNDATION_INDEV)        += avfoundation.o
>  OBJS-$(CONFIG_BKTR_INDEV)                += bktr.o
>  OBJS-$(CONFIG_CACA_OUTDEV)               += caca.o
> diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c
> index 8633433254..a6f68dd3bb 100644
> --- a/libavdevice/alldevices.c
> +++ b/libavdevice/alldevices.c
> @@ -27,6 +27,7 @@
>  extern AVInputFormat  ff_alsa_demuxer;
>  extern AVOutputFormat ff_alsa_muxer;
>  extern AVInputFormat  ff_android_camera_demuxer;
> +extern AVOutputFormat ff_audiotoolbox_muxer;
>  extern AVInputFormat  ff_avfoundation_demuxer;
>  extern AVInputFormat  ff_bktr_demuxer;
>  extern AVOutputFormat ff_caca_muxer;
> diff --git a/libavdevice/audiotoolbox.m b/libavdevice/audiotoolbox.m
> new file mode 100644
> index 0000000000..c6626604ef
> --- /dev/null
> +++ b/libavdevice/audiotoolbox.m
> @@ -0,0 +1,299 @@
> +/*
> + * AudioToolbox output device
> + * Copyright (c) 2020 Thilo Borgmann <thilo.borgmann@mail.de>
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +/**
> + * @file
> + * AudioToolbox output device
> + * @author Thilo Borgmann <thilo.borgmann@mail.de>
> + */
> +
> +#import <AudioToolbox/AudioToolbox.h>
> +#include <pthread.h>
> +
> +#include "libavutil/opt.h"
> +#include "libavformat/internal.h"
> +#include "libavutil/internal.h"
> +#include "avdevice.h"
> +

> +typedef struct
> +{

Braces ont the same line.

> +    AVClass*            class;

Style: type *var, not type* var. Same below.

> +
> +    AudioDeviceID*      devices;
> +    int                 num_devices;

These are only used in write_header() (and freed later): they should be
local variables.

> +
> +    AudioQueueBufferRef buffer[2];
> +    pthread_mutex_t     buffer_lock[2];
> +    int                 cur_buf;
> +    AudioQueueRef       queue;
> +
> +    int                 list_devices;
> +    int                 audio_device_index;
> +
> +} ATContext;
> +

> +static int check_status(ATContext *ctx, OSStatus *status, const char *msg)

Unless there is a good reason otherwise, log for the AVFilterContext.

> +{
> +    if (*status != noErr) {
> +        av_log(ctx, AV_LOG_ERROR, "Error: %s (%i)\n", msg, *status);
> +        return 1;
> +    } else {
> +        av_log(ctx, AV_LOG_DEBUG, " OK  : %s\n", msg);
> +        return 0;
> +    }
> +}
> +
> +static void queue_callback(void* atctx, AudioQueueRef inAQ,
> +                           AudioQueueBufferRef inBuffer)
> +{
> +    // unlock the buffer that has just been consumed
> +    ATContext *ctx = (ATContext*)atctx;
> +    for (int i = 0; i < 2; i++) {
> +        if (inBuffer == ctx->buffer[i]) {
> +            pthread_mutex_unlock(&ctx->buffer_lock[i]);
> +        }
> +    }
> +}
> +

> +static av_cold int at_write_header(AVFormatContext *s)
> +{
> +    ATContext *ctx = (ATContext*)s->priv_data;

The convention is the opposite: ctx for the AVFormatContext, s for the
private context.

> +    OSStatus err = noErr;
> +    CFStringRef device_UID = NULL;
> +
> +    // get devices
> +    UInt32 data_size = 0;
> +    AudioObjectPropertyAddress prop;
> +    prop.mSelector = kAudioHardwarePropertyDevices;
> +    prop.mScope    = kAudioObjectPropertyScopeGlobal;
> +    prop.mElement  = kAudioObjectPropertyElementMaster;
> +    err = AudioObjectGetPropertyDataSize(kAudioObjectSystemObject, &prop, 0, NULL, &data_size);
> +    if (check_status(ctx, &err, "AudioObjectGetPropertyDataSize devices"))
> +        return AVERROR(EINVAL);
> +
> +    ctx->num_devices = data_size / sizeof(AudioDeviceID);
> +
> +    ctx->devices = (AudioDeviceID*)(av_malloc(data_size));
> +    err = AudioObjectGetPropertyData(kAudioObjectSystemObject, &prop, 0, NULL, &data_size, ctx->devices);
> +    if (check_status(ctx, &err, "AudioObjectGetPropertyData devices"))
> +        return AVERROR(EINVAL);
> +
> +    // list devices
> +    if (ctx->list_devices) {
> +        CFStringRef device_name = NULL;
> +        prop.mScope = kAudioDevicePropertyScopeInput;
> +
> +        av_log(ctx, AV_LOG_INFO, "CoreAudio devices:\n");
> +        for(UInt32 i = 0; i < ctx->num_devices; ++i) {
> +            // UID
> +            data_size = sizeof(device_UID);
> +            prop.mSelector = kAudioDevicePropertyDeviceUID;
> +            err = AudioObjectGetPropertyData(ctx->devices[i], &prop, 0, NULL, &data_size, &device_UID);
> +            if (check_status(ctx, &err, "AudioObjectGetPropertyData UID"))
> +                continue;
> +
> +            // name
> +            data_size = sizeof(device_name);
> +            prop.mSelector = kAudioDevicePropertyDeviceNameCFString;
> +            err = AudioObjectGetPropertyData(ctx->devices[i], &prop, 0, NULL, &data_size, &device_name);
> +            if (check_status(ctx, &err, "AudioObjecTGetPropertyData name"))
> +                continue;
> +
> +            av_log(ctx, AV_LOG_INFO, "[%d] %30s, %s\n", i,
> +                   CFStringGetCStringPtr(device_name, kCFStringEncodingMacRoman),
> +                   CFStringGetCStringPtr(device_UID, kCFStringEncodingMacRoman));
> +        }
> +    }
> +
> +    // get user-defined device UID or use default device
> +    // -audio_device_index overrides any URL given
> +    const char *stream_name = s->url;
> +    if (stream_name && ctx->audio_device_index == -1) {
> +        sscanf(stream_name, "%d", &ctx->audio_device_index);
> +    }
> +
> +    if (ctx->audio_device_index >= 0) {
> +        // get UID of selected device
> +        data_size = sizeof(device_UID);
> +        prop.mSelector = kAudioDevicePropertyDeviceUID;
> +        err = AudioObjectGetPropertyData(ctx->devices[ctx->audio_device_index], &prop, 0, NULL, &data_size, &device_UID);
> +        if (check_status(ctx, &err, "AudioObjecTGetPropertyData UID"))
> +            return AVERROR(EINVAL);
> +    } else {
> +        // use default device
> +        device_UID = NULL;
> +    }
> +
> +    av_log(ctx, AV_LOG_DEBUG, "stream_name:        %s\n", stream_name);
> +    av_log(ctx, AV_LOG_DEBUG, "audio_device_idnex: %i\n", ctx->audio_device_index);
> +    av_log(ctx, AV_LOG_DEBUG, "UID:                %s\n", CFStringGetCStringPtr(device_UID, kCFStringEncodingMacRoman));
> +
> +    // check input stream
> +    if (s->nb_streams != 1 || s->streams[0]->codecpar->codec_type != AVMEDIA_TYPE_AUDIO) {
> +        av_log(ctx, AV_LOG_ERROR, "Only a single audio stream is supported.\n");
> +        return AVERROR(EINVAL);
> +    }
> +
> +    AVCodecParameters *codecpar = s->streams[0]->codecpar;
> +
> +    // audio format
> +    AudioStreamBasicDescription device_format = {0};
> +    device_format.mSampleRate        = codecpar->sample_rate;
> +    device_format.mFormatID          = kAudioFormatLinearPCM;
> +    device_format.mFormatFlags      |= (codecpar->format == AV_SAMPLE_FMT_FLT) ? kLinearPCMFormatFlagIsFloat : 0;
> +    device_format.mFormatFlags      |= (codecpar->codec_id == AV_CODEC_ID_PCM_S8) ? kLinearPCMFormatFlagIsSignedInteger : 0;
> +    device_format.mFormatFlags      |= (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S16BE, AV_CODEC_ID_PCM_S16LE)) ? kLinearPCMFormatFlagIsSignedInteger : 0;
> +    device_format.mFormatFlags      |= (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S24BE, AV_CODEC_ID_PCM_S24LE)) ? kLinearPCMFormatFlagIsSignedInteger : 0;
> +    device_format.mFormatFlags      |= (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S32BE, AV_CODEC_ID_PCM_S32LE)) ? kLinearPCMFormatFlagIsSignedInteger : 0;
> +    device_format.mFormatFlags      |= (av_sample_fmt_is_planar(codecpar->format)) ? kAudioFormatFlagIsNonInterleaved : 0;
> +    device_format.mFormatFlags      |= AV_NE(kAudioFormatFlagIsBigEndian, 0);
> +    device_format.mChannelsPerFrame  = codecpar->channels;
> +    device_format.mBitsPerChannel    = (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S24BE, AV_CODEC_ID_PCM_S24LE)) ? 24 : (av_get_bytes_per_sample(codecpar->format) << 3);
> +    device_format.mBytesPerFrame     = (device_format.mBitsPerChannel >> 3) * device_format.mChannelsPerFrame;
> +    device_format.mFramesPerPacket   = 1;
> +    device_format.mBytesPerPacket    = device_format.mBytesPerFrame * device_format.mFramesPerPacket;
> +    device_format.mReserved          = 0;
> +
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mSampleRate        = %i\n", codecpar->sample_rate);
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatID          = %s\n", "kAudioFormatLinearPCM");
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      |= %s\n", (codecpar->format == AV_SAMPLE_FMT_FLT) ? "kLinearPCMFormatFlagIsFloat" : "0");
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      |= %s\n", (codecpar->codec_id == AV_CODEC_ID_PCM_S8) ? "kLinearPCMFormatFlagIsSignedInteger" : "0");
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      |= %s\n", (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S32BE, AV_CODEC_ID_PCM_S32LE)) ? "kLinearPCMFormatFlagIsSignedInteger" : "0");
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      |= %s\n", (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S16BE, AV_CODEC_ID_PCM_S16LE)) ? "kLinearPCMFormatFlagIsSignedInteger" : "0");
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      |= %s\n", (codecpar->codec_id == AV_NE(AV_CODEC_ID_PCM_S24BE, AV_CODEC_ID_PCM_S24LE)) ? "kLinearPCMFormatFlagIsSignedInteger" : "0");
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      |= %s\n", (av_sample_fmt_is_planar(codecpar->format)) ? "kAudioFormatFlagIsNonInterleaved" : "0");
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      |= %s\n", AV_NE("kAudioFormatFlagIsBigEndian", "0"));
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFormatFlags      == %i\n", device_format.mFormatFlags);
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mChannelsPerFrame  = %i\n", codecpar->channels);
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mBitsPerChannel    = %i\n", av_get_bytes_per_sample(codecpar->format) << 3);
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mBytesPerFrame     = %i\n", (device_format.mBitsPerChannel >> 3) * codecpar->channels);
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mBytesPerPacket    = %i\n", device_format.mBytesPerFrame);
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mFramesPerPacket   = %i\n", 1);
> +    av_log(ctx, AV_LOG_DEBUG, "device_format.mReserved          = %i\n", 0);
> +
> +    // create new output queue for the device
> +    err = AudioQueueNewOutput(&device_format, queue_callback, ctx,
> +                              NULL, kCFRunLoopCommonModes,
> +                              0, &ctx->queue);
> +    if (check_status(ctx, &err, "AudioQueueNewOutput")) {
> +        if (err == kAudioFormatUnsupportedDataFormatError)
> +            av_log(ctx, AV_LOG_ERROR, "Unsupported output format.\n");
> +        return AVERROR(EINVAL);
> +    }
> +
> +    // set user-defined device or leave untouched for default
> +    if (device_UID != NULL) {
> +        err = AudioQueueSetProperty(ctx->queue, kAudioQueueProperty_CurrentDevice, &device_UID, sizeof(device_UID));
> +        if (check_status(ctx, &err, "AudioQueueSetProperty output UID"))
> +            return AVERROR(EINVAL);
> +    }
> +
> +    // start the queue
> +    err = AudioQueueStart(ctx->queue, NULL);
> +    if (check_status(ctx, &err, "AudioQueueStart"))
> +        return AVERROR(EINVAL);
> +
> +    // init the mutexes for double-buffering
> +    pthread_mutex_init(&ctx->buffer_lock[0], NULL);
> +    pthread_mutex_init(&ctx->buffer_lock[1], NULL);
> +
> +    return 0;
> +}
> +
> +static int at_write_packet(AVFormatContext *s, AVPacket *pkt)
> +{
> +    ATContext *ctx = (ATContext*)s->priv_data;
> +    OSStatus err = noErr;
> +
> +    // use the other buffer
> +    ctx->cur_buf = !ctx->cur_buf;
> +
> +    // lock for writing or wait for the buffer to be available
> +    // will be unlocked by queue callback
> +    pthread_mutex_lock(&ctx->buffer_lock[ctx->cur_buf]);
> +
> +    // (re-)allocate the buffer if not existant or of different size
> +    if (!ctx->buffer[ctx->cur_buf] || ctx->buffer[ctx->cur_buf]->mAudioDataBytesCapacity != pkt->size) {
> +        err = AudioQueueAllocateBuffer(ctx->queue, pkt->size, &ctx->buffer[ctx->cur_buf]);
> +        if (check_status(ctx, &err, "AudioQueueAllocateBuffer")) {
> +            pthread_mutex_unlock(&ctx->buffer_lock[ctx->cur_buf]);
> +            return AVERROR(ENOMEM);
> +        }
> +    }
> +
> +    AudioQueueBufferRef buf = ctx->buffer[ctx->cur_buf];
> +
> +    // copy audio data into buffer and enqueue the buffer
> +    memcpy(buf->mAudioData, pkt->data, buf->mAudioDataBytesCapacity);
> +    buf->mAudioDataByteSize = buf->mAudioDataBytesCapacity;
> +    err = AudioQueueEnqueueBuffer(ctx->queue, buf, 0, NULL);
> +    if (check_status(ctx, &err, "AudioQueueEnqueueBuffer")) {
> +        pthread_mutex_unlock(&ctx->buffer_lock[ctx->cur_buf]);
> +        return AVERROR(EINVAL);
> +    }
> +
> +    return 0;
> +}
> +
> +static av_cold int at_write_trailer(AVFormatContext *s)
> +{
> +    ATContext *ctx = (ATContext*)s->priv_data;
> +    OSStatus err = noErr;
> +
> +    av_freep(&ctx->devices);
> +    pthread_mutex_destroy(&ctx->buffer_lock[0]);
> +    pthread_mutex_destroy(&ctx->buffer_lock[1]);
> +
> +    err = AudioQueueFlush(ctx->queue);
> +    check_status(ctx, &err, "AudioQueueFlush");
> +    err = AudioQueueDispose(ctx->queue, true);
> +    check_status(ctx, &err, "AudioQueueDispose");
> +
> +    return 0;
> +}
> +
> +static const AVOption options[] = {
> +    { "list_devices", "list available audio devices", offsetof(ATContext, list_devices), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, AV_OPT_FLAG_ENCODING_PARAM },
> +    { "audio_device_index", "select audio device by index (starts at 0)", offsetof(ATContext, audio_device_index), AV_OPT_TYPE_INT, {.i64 = -1}, -1, INT_MAX, AV_OPT_FLAG_ENCODING_PARAM },
> +    { NULL },
> +};
> +
> +static const AVClass at_class = {
> +    .class_name = "AudioToolbox",
> +    .item_name  = av_default_item_name,
> +    .option     = options,
> +    .version    = LIBAVUTIL_VERSION_INT,
> +    .category   = AV_CLASS_CATEGORY_DEVICE_AUDIO_OUTPUT,
> +};
> +
> +AVOutputFormat ff_audiotoolbox_muxer = {
> +    .name           = "audiotoolbox",
> +    .long_name      = NULL_IF_CONFIG_SMALL("AudioToolbox output device"),
> +    .priv_data_size = sizeof(ATContext),
> +    .audio_codec    = AV_NE(AV_CODEC_ID_PCM_S16BE, AV_CODEC_ID_PCM_S16LE),
> +    .video_codec    = AV_CODEC_ID_NONE,
> +    .write_header   = at_write_header,
> +    .write_packet   = at_write_packet,
> +    .write_trailer  = at_write_trailer,
> +    .flags          = AVFMT_NOFILE,
> +    .priv_class     = &at_class,
> +};
> +
> -- 
> 2.20.1 (Apple Git-117)

Regards,

Am 08.06.20 um 20:21 schrieb Nicolas George:
> Thilo Borgmann (12020-06-08):
>> v2 due to Nicolas remarks. One device, several formats.
> 
> I cannot comment on appleisms, but here are a few general remarks:
> 
>> From 64b100d7b4cfe780c273d6e7d8b940a8427cfdc9 Mon Sep 17 00:00:00 2001
>> From: Thilo Borgmann <thilo.borgmann@mail.de>
>> Date: Mon, 8 Jun 2020 13:04:11 +0200
>> Subject: [PATCH] lavdevice: Add AudioToolbox output device.
>>
>> ---
> 
>>  configure                  |   3 +
>>  libavdevice/Makefile       |   1 +
>>  libavdevice/alldevices.c   |   1 +
>>  libavdevice/audiotoolbox.m | 299 +++++++++++++++++++++++++++++++++++++
> 
[...]
> 
>> +static av_cold int at_write_header(AVFormatContext *s)
>> +{
>> +    ATContext *ctx = (ATContext*)s->priv_data;
> 
> The convention is the opposite: ctx for the AVFormatContext, s for the
> private context.

Hmm I never liked *s anyway, however a grep revealed _a lot_ of AVFormatContext *s.
And I always did c&p it from somewhere else...
Anyway, I did "AVFormatContext *avctx" in v3, which seems to be another common naming that fits in better IMHO.

All other remarks done in v3.

Thanks for review!
-Thilo