[FFmpeg-devel] Add android_capture indev

Hello,

I've written an indev for Android devices to allow capturing their 
builtin cameras.
What needs to be done to merge this?

Greetings,
Felix

Le duodi 12 brumaire, an CCXXVI, Felix Matouschek a écrit :
> I've written an indev for Android devices to allow capturing their builtin
> cameras.
> What needs to be done to merge this?

Thanks for the patch. It looks very interesting. Before considering
merging, there are a few technical point to address. See inline comments
below.

> From b21fc8729ef2e1d9867dd7652f2c6173378e4910 Mon Sep 17 00:00:00 2001
> From: Felix Matouschek <felix@matouschek.org>
> Date: Tue, 24 Oct 2017 13:11:23 +0200
> Subject: [PATCH] Add android_capture indev
> To: ffmpeg-devel@ffmpeg.org
> 
> This commit adds an indev for Android devices on API level 24+ which
> uses the Android NDK Camera2 API to capture video from builtin cameras
> 
> Signed-off-by: Felix Matouschek <felix@matouschek.org>
> ---
>  configure                     |   6 +
>  libavdevice/Makefile          |   1 +
>  libavdevice/alldevices.c      |   1 +

>  libavdevice/android_capture.c | 782 ++++++++++++++++++++++++++++++++++++++++++

When reading the subject of the mail, I first thought it would be about
screen capture. Furthermore, there is code for audio, but it is not
connected to anything, and it does not seem that the android API
connects audio and video together.

For all these reasons, I suggest you name this device maybe
android_camera, and keep android_mic for audio capture.

>  libavdevice/android_capture.h |  77 +++++

This header is only used once, it does not need to be separate.

>  5 files changed, 867 insertions(+)
>  create mode 100644 libavdevice/android_capture.c
>  create mode 100644 libavdevice/android_capture.h
> 
> diff --git a/configure b/configure
> index 7a53bc76c7..e2165f2ff9 100755
> --- a/configure
> +++ b/configure
> @@ -3068,6 +3068,8 @@ xmv_demuxer_select="riffdec"
>  xwma_demuxer_select="riffdec"
>  
>  # indevs / outdevs
> +android_capture_indev_deps="android mediandk camera2ndk pthreads"
> +android_capture_indev_extralibs="-landroid -lmediandk -lcamera2ndk"
>  alsa_indev_deps="alsa"
>  alsa_outdev_deps="alsa"
>  avfoundation_indev_deps="avfoundation corevideo coremedia pthreads"
> @@ -5836,6 +5838,10 @@ check_lib shell32  "windows.h shellapi.h" CommandLineToArgvW   -lshell32
>  check_lib wincrypt "windows.h wincrypt.h" CryptGenRandom       -ladvapi32
>  check_lib psapi    "windows.h psapi.h"    GetProcessMemoryInfo -lpsapi
>  
> +check_lib android android/native_window.h ANativeWindow_acquire -landroid
> +check_lib mediandk "stdint.h media/NdkImage.h" AImage_delete -lmediandk
> +check_lib camera2ndk "stdbool.h stdint.h camera/NdkCameraManager.h" ACameraManager_create -lcamera2ndk
> +
>  enabled appkit       && check_apple_framework AppKit
>  enabled audiotoolbox && check_apple_framework AudioToolbox
>  enabled avfoundation && check_apple_framework AVFoundation
> diff --git a/libavdevice/Makefile b/libavdevice/Makefile
> index 8228d62147..aa01dd7e24 100644
> --- a/libavdevice/Makefile
> +++ b/libavdevice/Makefile
> @@ -14,6 +14,7 @@ OBJS-$(CONFIG_SHARED)                    += reverse.o
>  # input/output devices
>  OBJS-$(CONFIG_ALSA_INDEV)                += alsa_dec.o alsa.o timefilter.o
>  OBJS-$(CONFIG_ALSA_OUTDEV)               += alsa_enc.o alsa.o
> +OBJS-$(CONFIG_ANDROID_CAPTURE_INDEV)     += android_capture.o
>  OBJS-$(CONFIG_AVFOUNDATION_INDEV)        += avfoundation.o
>  OBJS-$(CONFIG_BKTR_INDEV)                += bktr.o
>  OBJS-$(CONFIG_CACA_OUTDEV)               += caca.o
> diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c
> index b767b6a718..6cd57aa88a 100644
> --- a/libavdevice/alldevices.c
> +++ b/libavdevice/alldevices.c
> @@ -42,6 +42,7 @@ static void register_all(void)
>  {
>      /* devices */
>      REGISTER_INOUTDEV(ALSA,             alsa);
> +    REGISTER_INDEV   (ANDROID_CAPTURE,  android_capture);
>      REGISTER_INDEV   (AVFOUNDATION,     avfoundation);
>      REGISTER_INDEV   (BKTR,             bktr);
>      REGISTER_OUTDEV  (CACA,             caca);
> diff --git a/libavdevice/android_capture.c b/libavdevice/android_capture.c
> new file mode 100644
> index 0000000000..be0dee8f81
> --- /dev/null
> +++ b/libavdevice/android_capture.c
> @@ -0,0 +1,782 @@
> +/*
> + * Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)
> + *
> + * Copyright (C) 2017 Felix Matouschek
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <errno.h>
> +#include <stdbool.h>
> +#include <stdint.h>
> +
> +#include <camera/NdkCameraDevice.h>
> +#include <camera/NdkCameraManager.h>
> +#include <media/NdkImage.h>
> +
> +#include "libavformat/internal.h"
> +#include "libavutil/avstring.h"
> +#include "libavutil/display.h"
> +#include "libavutil/imgutils.h"
> +#include "libavutil/opt.h"
> +#include "libavutil/parseutils.h"
> +
> +#include "android_capture.h"
> +#include "version.h"
> +
> +/* This image format is available on all Android devices
> + * supporting the Camera2 API */
> +#define IMAGE_FORMAT_ANDROID AIMAGE_FORMAT_YUV_420_888
> +#define IMAGE_FORMAT_FFMPEG AV_PIX_FMT_YUV420P
> +#define IMAGE_NUM_PLANES 3
> +
> +#define MAX_BUF_COUNT 2
> +#define VIDEO_STREAM_ID 0
> +#define VIDEO_TIMEBASE 1000000000
> +#define AUDIO_STREAM_ID 1
> +

> +static int parse_token_to_int(char *token, int *out)
> +{
> +    long l;
> +    char *endptr;
> +    errno = 0;
> +
> +    l = strtol(token, &endptr, 0);
> +
> +    if (errno == ERANGE || *endptr != '\0' || token == endptr) {

> +        return AVERROR(ERANGE);

The second and third clauses should return EINVAL.

> +    }
> +
> +#if LONG_MIN < INT_MIN || LONG_MAX > INT_MAX
> +    if (l < INT_MIN || l > INT_MAX) {
> +        return AVERROR(ERANGE);
> +    }
> +#endif
> +
> +    *out = (int) l;
> +
> +    return 0;
> +}

This does not belong in an individual component. And I think it is not
needed at all, see below.

> +
> +static int parse_device_number(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;

> +    int *video_device_number = &ctx->video_device_number;
> +    int *audio_device_number = &ctx->audio_device_number;

These extra variable seem quite unnecessary.

> +    char *name = av_strdup(avctx->filename);
> +    char *tmp = name;
> +    char *type;
> +    char *save;
> +    int ret = 0;
> +
> +    *video_device_number = -1;
> +    *audio_device_number = -1;
> +
> +    while ((type = av_strtok(tmp, "=", &save))) {
> +        char *token = av_strtok(NULL, ":", &save);
> +        tmp = NULL;

You are parsing the "filename" of the device into a key=value syntax.
This is not a good idea, and I really would not like another key=value
parser in the code base.

How do cameraIds look? If the string values are friendly enough, they
can be used as filenames as is.

> +
> +        if (!strcmp(type, "video")) {
> +            ret = parse_token_to_int(token, video_device_number);
> +            if (ret < 0) {
> +                break;
> +            }
> +
> +        } else if (!strcmp(type, "audio")) {
> +            ret = parse_token_to_int(token, audio_device_number);
> +            if (ret < 0) {
> +                break;
> +            }
> +        } else {

> +            ret = AVERROR(ENXIO);

The correct return code would be EINVAL in this case.

> +            break;
> +        }
> +    }
> +
> +    if (ret >= 0 && *video_device_number < 0 && *audio_device_number < 0) {

> +        ret = AVERROR(ENXIO);

EINVAL here too.

> +    }
> +
> +    av_free(name);
> +    return ret;
> +}
> +
> +static void camera_dev_disconnected(void *context, ACameraDevice *device)
> +{

> +    AVFormatContext *avctx = (AVFormatContext *) context;

The cast is unnecessary, and as such harmful.

> +    android_capture_ctx *ctx = avctx->priv_data;

> +    ctx->read_closing = 1;

If I read the code correctly, Android is expected to call this from a
separate thread. Then this needs to be protected by a memory barrier,
mutex or other.

> +    av_log(avctx, AV_LOG_ERROR, "Camera with id %s disconnected.\n",
> +            ACameraDevice_getId(device));
> +}
> +
> +static void camera_dev_error(void *context, ACameraDevice *device, int error)
> +{

> +    AVFormatContext *avctx = (AVFormatContext *) context;

Same as above.

> +    android_capture_ctx *ctx = avctx->priv_data;

> +    ctx->read_closing = 1;

Same as above.

> +    av_log(avctx, AV_LOG_ERROR, "Error %d on camera with id %s.\n", error,
> +            ACameraDevice_getId(device));

Can the error be translated into a human-readable message?

> +}
> +
> +static int open_camera(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    camera_status_t ret;
> +    ACameraIdList *camera_ids;
> +
> +    ret = ACameraManager_getCameraIdList(ctx->camera_mgr, &camera_ids);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to get camera id list, camera_status_t: %d.\n", ret);

> +        return AVERROR(EIO);

Better translate the ACAMERA codes into AVERROR codes, or use
AVERROR_EXTERNAL. Same below.

> +    }
> +
> +    if (ctx->video_device_number < camera_ids->numCameras) {
> +        ctx->camera_id = av_strdup(
> +                camera_ids->cameraIds[ctx->video_device_number]);
> +    } else {
> +        av_log(avctx, AV_LOG_ERROR, "No camera with number %d available.\n",
> +                ctx->video_device_number);
> +        return AVERROR(ENXIO);
> +    }
> +
> +    ACameraManager_deleteCameraIdList(camera_ids);
> +
> +    ret = ACameraManager_getCameraCharacteristics(ctx->camera_mgr,
> +            ctx->camera_id, &ctx->camera_metadata);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to get metadata for camera with id %s, camera_status_t: %d.\n",
> +                ctx->camera_id, ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ctx->camera_state_callbacks.context = avctx;
> +    ctx->camera_state_callbacks.onDisconnected = camera_dev_disconnected;
> +    ctx->camera_state_callbacks.onError = camera_dev_error;
> +
> +    ret = ACameraManager_openCamera(ctx->camera_mgr, ctx->camera_id,
> +            &ctx->camera_state_callbacks, &ctx->camera_dev);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to open camera with id %s, camera_status_t: %d.\n",
> +                ctx->camera_id, ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    return 0;
> +}
> +
> +static void get_sensor_orientation(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    ACameraMetadata_const_entry lens_facing;
> +    ACameraMetadata_const_entry sensor_orientation;
> +
> +    ACameraMetadata_getConstEntry(ctx->camera_metadata, ACAMERA_LENS_FACING,
> +            &lens_facing);
> +    ACameraMetadata_getConstEntry(ctx->camera_metadata,
> +            ACAMERA_SENSOR_ORIENTATION, &sensor_orientation);
> +
> +    ctx->lens_facing = lens_facing.data.u8[0];
> +    ctx->sensor_orientation = sensor_orientation.data.i32[0];
> +}
> +
> +static int match_video_size(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    ACameraMetadata_const_entry available_configs;
> +    int ret = -1;
> +
> +    ACameraMetadata_getConstEntry(ctx->camera_metadata,
> +            ACAMERA_SCALER_AVAILABLE_STREAM_CONFIGURATIONS, &available_configs);
> +
> +    for (int i = 0; i < available_configs.count; i++) {

> +        int32_t input = available_configs.data.i32[i * 4 + 3];
> +        int32_t format = available_configs.data.i32[i * 4 + 0];

Is it really the official way of getting the resolution and format of
the video? If so, were the Android people drunk?

> +
> +        if (input) {
> +            continue;
> +        }
> +

> +        if (format == IMAGE_FORMAT_ANDROID) {

It would be better to support as many pixel formats as possible.

> +            int32_t width = available_configs.data.i32[i * 4 + 1];
> +            int32_t height = available_configs.data.i32[i * 4 + 2];
> +
> +            //Same ratio
> +            if (ctx->requested_width * ctx->requested_height
> +                    == width * height) {
> +                ctx->width = width;
> +                ctx->height = height;
> +                ret = 0;
> +                break;
> +            }
> +        }
> +    }
> +
> +    if (ret < 0) {
> +        ctx->width = 640;
> +        ctx->height = 480;
> +    }
> +
> +    return ret;
> +}
> +
> +static int match_framerate(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    ACameraMetadata_const_entry available_framerates;
> +    int ret = -1;
> +    int current_best_match = -1;
> +
> +    ACameraMetadata_getConstEntry(ctx->camera_metadata,
> +            ACAMERA_CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES,
> +            &available_framerates);
> +
> +    for (int i = 0; i < available_framerates.count; i++) {
> +        int32_t min = available_framerates.data.i32[i * 2 + 0];
> +        int32_t max = available_framerates.data.i32[i * 2 + 1];
> +        double requested_framerate = av_q2d(ctx->requested_framerate);
> +
> +        if (requested_framerate == max) {
> +            ret = 0;
> +
> +            if (min == max) {
> +                ctx->framerate_range[0] = min;
> +                ctx->framerate_range[1] = max;
> +                break;
> +            } else if (current_best_match >= 0) {
> +                int32_t current_best_match_min =
> +                        available_framerates.data.i32[current_best_match * 2 + 0];
> +                if (min > current_best_match_min) {
> +                    current_best_match = i;
> +                }
> +            } else {
> +                current_best_match = i;
> +            }
> +        }
> +    }
> +
> +    if (ret < 0) {
> +        ctx->framerate_range[0] = available_framerates.data.i32[0];
> +        ctx->framerate_range[1] = available_framerates.data.i32[1];
> +    }
> +
> +    return ret;
> +}
> +
> +static int shall_we_drop(AVFormatContext *avctx, int stream_index,
> +        int stream_id)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    static const uint8_t dropscore[] = { 62, 75, 87, 100 };
> +    const int ndropscores = FF_ARRAY_ELEMS(dropscore);
> +    const char *stream_name = stream_id ? "audio" : "video";
> +    unsigned int buffer_fullness = (ctx->curbufsize[stream_index] * 100)
> +            / avctx->max_picture_buffer;
> +
> +    if (dropscore[++ctx->video_frame_num % ndropscores] <= buffer_fullness) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "real-time buffer of [%s input] too full or near too full (%d%% of size: %d [rtbufsize parameter])! frame dropped!\n",
> +                stream_name, buffer_fullness, avctx->max_picture_buffer);
> +        return 1;
> +    }
> +
> +    return 0;
> +}
> +
> +static void image_available(void *context, AImageReader *reader)
> +{
> +    AVFormatContext *avctx = (AVFormatContext *) context;
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    media_status_t ret;
> +
> +    AImage *image;
> +    int64_t image_timestamp;
> +    uint8_t *image_plane_data[4];
> +    int32_t image_linestrides[4];
> +    int plane_data_length[4];
> +
> +    AVPacketList *pktl_next;
> +    AVPacketList **ppktl = &ctx->pktl;
> +    uint8_t *side_data;
> +    int pkt_buffer_size = av_image_get_buffer_size(IMAGE_FORMAT_FFMPEG,
> +            ctx->width, ctx->height, 1);
> +    int32_t display_matrix[9];
> +

> +    av_display_rotation_set(display_matrix, 360 - ctx->sensor_orientation);
> +
> +    if (ctx->lens_facing == ACAMERA_LENS_FACING_FRONT) {
> +        av_display_matrix_flip(display_matrix, 1, 0);
> +    }
> +
> +    ret = AImageReader_acquireLatestImage(reader, &image);
> +    if (ret != AMEDIA_OK) {

> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to acquire latest image from image reader, media_status_t: %d.\n",
> +                ret);

Strange indentation.

> +        goto error;
> +    }
> +
> +    // Silently drop frames when read_closing is set
> +    if (ctx->read_closing) {
> +        goto error;
> +    }
> +
> +    AImage_getTimestamp(image, &image_timestamp);
> +
> +    for (int i = 0; i < IMAGE_NUM_PLANES; i++) {
> +        AImage_getPlaneRowStride(image, i, &image_linestrides[i]);
> +        AImage_getPlaneData(image, i, &image_plane_data[i], &plane_data_length[i]);
> +    }
> +
> +    pthread_mutex_lock(&ctx->mutex);
> +
> +    if (shall_we_drop(avctx, ctx->video_stream_index, VIDEO_STREAM_ID)) {
> +        goto error;
> +    }
> +

> +    pktl_next = av_mallocz(sizeof(AVPacketList));

I think it would better to avoid malloc() and anything similar in a
critical section. Do all the heavy work first, then enter the critical
section.

> +    if (!pktl_next)
> +        goto error;
> +
> +    if (av_new_packet(&pktl_next->pkt, pkt_buffer_size) < 0) {
> +        av_free(pktl_next);
> +        goto error;
> +    }
> +

> +    side_data = av_packet_new_side_data(&pktl_next->pkt,
> +            AV_PKT_DATA_DISPLAYMATRIX, sizeof(display_matrix));

It would probably be best to avoid sending the side data repeatedly if
it does not change.

> +    if (!side_data) {
> +        av_packet_unref(&pktl_next->pkt);
> +        av_free(pktl_next);
> +        goto error;
> +    }
> +
> +    pktl_next->pkt.stream_index = ctx->video_stream_index;
> +    pktl_next->pkt.pts = image_timestamp;
> +    av_image_copy_to_buffer(pktl_next->pkt.data, pkt_buffer_size,
> +            (const uint8_t * const *) image_plane_data, image_linestrides,
> +            IMAGE_FORMAT_FFMPEG, ctx->width, ctx->height, 1);
> +    memcpy(side_data, display_matrix, sizeof(display_matrix));
> +
> +    while (*ppktl) {
> +        ppktl = &(*ppktl)->next;
> +    }
> +    *ppktl = pktl_next;
> +
> +    ctx->curbufsize[ctx->video_stream_index] += pkt_buffer_size;
> +

> +error:
> +    pthread_mutex_unlock(&ctx->mutex);
> +    AImage_delete(image);
> +
> +    return;

It seems error conditions are not taken into account. Is it on purpose?

> +}
> +
> +static int create_image_reader(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    media_status_t ret;
> +
> +    ret = AImageReader_new(ctx->width, ctx->height, IMAGE_FORMAT_ANDROID,
> +            MAX_BUF_COUNT, &ctx->image_reader);
> +    if (ret != AMEDIA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to create image reader, media_status_t: %d.\n", ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ctx->image_listener.context = avctx;
> +    ctx->image_listener.onImageAvailable = image_available;
> +
> +    ret = AImageReader_setImageListener(ctx->image_reader,
> +            &ctx->image_listener);
> +    if (ret != AMEDIA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to set image listener on image reader, media_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ret = AImageReader_getWindow(ctx->image_reader, &ctx->image_reader_window);
> +    if (ret != AMEDIA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Could not get image reader window, media_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    return 0;
> +}
> +
> +static void capture_session_closed(void *context,
> +        ACameraCaptureSession *session)
> +{

> +    av_log((AVFormatContext *) context, AV_LOG_INFO,

The cast is wrong. Same below.

> +            "Android camera capture session was closed.\n");
> +}
> +
> +static void capture_session_ready(void *context, ACameraCaptureSession *session)
> +{
> +    av_log((AVFormatContext *) context, AV_LOG_INFO,
> +            "Android camera capture session is ready.\n");
> +}
> +
> +static void capture_session_active(void *context,
> +        ACameraCaptureSession *session)
> +{
> +    av_log((AVFormatContext *) context, AV_LOG_INFO,
> +            "Android camera capture session is active.\n");
> +}
> +
> +static int create_capture_session(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    camera_status_t ret;
> +
> +    ret = ACaptureSessionOutputContainer_create(
> +            &ctx->capture_session_output_container);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to create capture session output container, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ANativeWindow_acquire(ctx->image_reader_window);
> +
> +    ret = ACaptureSessionOutput_create(ctx->image_reader_window,
> +            &ctx->capture_session_output);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to create capture session container, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ret = ACaptureSessionOutputContainer_add(
> +            ctx->capture_session_output_container, ctx->capture_session_output);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to add output to output container, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ret = ACameraOutputTarget_create(ctx->image_reader_window,
> +            &ctx->camera_output_target);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to create camera output target, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ret = ACameraDevice_createCaptureRequest(ctx->camera_dev, TEMPLATE_RECORD,
> +            &ctx->capture_request);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to create capture request, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ret = ACaptureRequest_setEntry_i32(ctx->capture_request,
> +            ACAMERA_CONTROL_AE_TARGET_FPS_RANGE, 2, ctx->framerate_range);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to set target fps range in capture request, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ret = ACaptureRequest_addTarget(ctx->capture_request,
> +            ctx->camera_output_target);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to add capture request capture request, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ctx->capture_session_state_callbacks.context = avctx;
> +    ctx->capture_session_state_callbacks.onClosed = capture_session_closed;
> +    ctx->capture_session_state_callbacks.onReady = capture_session_ready;
> +    ctx->capture_session_state_callbacks.onActive = capture_session_active;
> +
> +    ret = ACameraDevice_createCaptureSession(ctx->camera_dev,
> +            ctx->capture_session_output_container,
> +            &ctx->capture_session_state_callbacks, &ctx->capture_session);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to create capture session, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    ret = ACameraCaptureSession_setRepeatingRequest(ctx->capture_session, NULL,
> +            1, &ctx->capture_request, NULL);
> +    if (ret != ACAMERA_OK) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to set repeating request on capture session, camera_status_t: %d.\n",
> +                ret);
> +        return AVERROR(EIO);
> +    }
> +
> +    return 0;
> +}
> +
> +static int add_video_stream(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    AVStream *st;
> +    AVCodecParameters *codecpar;
> +
> +    st = avformat_new_stream(avctx, NULL);
> +    if (!st) {
> +        return AVERROR(ENOMEM);
> +    }
> +
> +    st->id = VIDEO_STREAM_ID;
> +    st->avg_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> +    st->r_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> +
> +    codecpar = st->codecpar;
> +    codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
> +    codecpar->codec_id = AV_CODEC_ID_RAWVIDEO;
> +    codecpar->format = IMAGE_FORMAT_FFMPEG;
> +    codecpar->width = ctx->width;
> +    codecpar->height = ctx->height;
> +
> +    avpriv_set_pts_info(st, 64, 1, VIDEO_TIMEBASE);
> +
> +    ctx->video_stream_index = st->index;
> +
> +    return 0;
> +}
> +
> +static int android_capture_read_close(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    AVPacketList *pktl;
> +
> +    ctx->read_closing = 1;
> +
> +    if (ctx->capture_session) {
> +        ACameraCaptureSession_stopRepeating(ctx->capture_session);
> +        // Following warning is emitted, after capture session closed callback is received:
> +        // ACameraCaptureSession: Device is closed but session 0 is not notified
> +        // Seems to be a bug in Android, we can ignore this
> +        ACameraCaptureSession_close(ctx->capture_session);
> +        ctx->capture_session = NULL;
> +    }
> +
> +    if (ctx->capture_request) {
> +        ACaptureRequest_removeTarget(ctx->capture_request,
> +                ctx->camera_output_target);
> +        ACaptureRequest_free(ctx->capture_request);
> +        ctx->capture_request = NULL;
> +    }
> +
> +    if (ctx->camera_output_target) {
> +        ACameraOutputTarget_free(ctx->camera_output_target);
> +        ctx->camera_output_target = NULL;
> +    }
> +
> +    if (ctx->capture_session_output) {
> +        ACaptureSessionOutputContainer_remove(
> +                ctx->capture_session_output_container,
> +                ctx->capture_session_output);
> +        ACaptureSessionOutput_free(ctx->capture_session_output);
> +        ctx->capture_session_output = NULL;
> +    }
> +
> +    ANativeWindow_release(ctx->image_reader_window);
> +
> +    if (ctx->capture_session_output_container) {
> +        ACaptureSessionOutputContainer_free(
> +                ctx->capture_session_output_container);
> +        ctx->capture_session_output_container = NULL;
> +    }
> +
> +    if (ctx->camera_dev) {
> +        ACameraDevice_close(ctx->camera_dev);
> +        ctx->camera_dev = NULL;
> +    }
> +
> +    if (ctx->image_reader) {
> +        AImageReader_delete(ctx->image_reader);
> +        ctx->image_reader = NULL;
> +        ctx->image_reader_window = NULL;
> +    }
> +
> +    if (ctx->camera_metadata) {
> +        ACameraMetadata_free(ctx->camera_metadata);
> +        ctx->camera_metadata = NULL;
> +    }
> +
> +    if (ctx->camera_id) {

> +        av_free(ctx->camera_id);
> +        ctx->camera_id = NULL;

av_freep(&ctx->camera_id)

> +    }
> +
> +    if (ctx->camera_mgr) {
> +        ACameraManager_delete(ctx->camera_mgr);
> +        ctx->camera_mgr = NULL;
> +    }
> +
> +    pktl = ctx->pktl;
> +    while (pktl) {
> +        AVPacketList *next = pktl->next;
> +        av_packet_unref(&pktl->pkt);
> +        av_free(pktl);
> +        pktl = next;
> +    }
> +
> +    pthread_mutex_destroy(&ctx->mutex);
> +
> +    return 0;
> +}
> +
> +static int android_capture_read_header(AVFormatContext *avctx)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    int ret;
> +
> +    pthread_mutex_init(&ctx->mutex, 0);
> +
> +    ret = parse_device_number(avctx);
> +    if (ret < 0) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Malformed android_capture input string.\n");
> +        goto error;
> +    }
> +
> +    if (ctx->framerate) {
> +        ret = av_parse_video_rate(&ctx->requested_framerate, ctx->framerate);
> +        if (ret < 0) {
> +            av_log(avctx, AV_LOG_ERROR, "Could not parse framerate '%s'.\n",
> +                    ctx->framerate);
> +            goto error;
> +        }
> +    }
> +
> +    ctx->camera_mgr = ACameraManager_create();
> +    if (!ctx->camera_mgr) {
> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to create Android camera manager.\n");
> +        ret = AVERROR(EIO);
> +        goto error;
> +    }
> +
> +    ret = open_camera(avctx);
> +    if (ret < 0) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to open Android camera.\n");
> +        goto error;
> +    }
> +
> +    get_sensor_orientation(avctx);
> +
> +    if (match_video_size(avctx) < 0) {
> +        av_log(avctx, AV_LOG_WARNING,
> +                "Requested video_size not available, falling back to 640x480\n");
> +    }
> +
> +    if (match_framerate(avctx) < 0) {
> +        av_log(avctx, AV_LOG_WARNING,
> +                "Requested framerate not available, falling back to min: %d and max: %d fps\n",
> +                ctx->framerate_range[0], ctx->framerate_range[1]);
> +    }
> +
> +    ret = create_image_reader(avctx);
> +    if (ret < 0) {
> +        goto error;
> +    }
> +
> +    ret = create_capture_session(avctx);
> +    if (ret < 0) {
> +        goto error;
> +    }
> +
> +    ret = add_video_stream(avctx);
> +
> +error:
> +    if (ret < 0) {
> +        android_capture_read_close(avctx);
> +        av_log(avctx, AV_LOG_ERROR, "Failed to open android_capture.\n");
> +    }
> +
> +    return ret;
> +}
> +
> +static int android_capture_read_packet(AVFormatContext *avctx, AVPacket *pkt)
> +{
> +    android_capture_ctx *ctx = avctx->priv_data;
> +    AVPacketList *pktl = NULL;
> +

> +    while (!ctx->read_closing && !pktl) {
> +        pthread_mutex_lock(&ctx->mutex);
> +        pktl = ctx->pktl;
> +        if (pktl) {
> +            *pkt = pktl->pkt;
> +            ctx->pktl = ctx->pktl->next;
> +            av_free(pktl);
> +            ctx->curbufsize[pkt->stream_index] -= pkt->size;
> +        }
> +        pthread_mutex_unlock(&ctx->mutex);
> +        if (!pktl) {
> +            if (avctx->flags & AVFMT_FLAG_NONBLOCK) {
> +                return AVERROR(EAGAIN);
> +            }
> +        }
> +    }
> +

I think this could be simplified using AVThreadMessageQueue.

> +    return ctx->read_closing ? AVERROR(EOF) : pkt->size;

AVERROR_EOF

> +}
> +
> +#define OFFSET(x) offsetof(android_capture_ctx, x)
> +#define DEC AV_OPT_FLAG_DECODING_PARAM
> +static const AVOption options[] = {
> +    { "video_size", "set video size given a string such as 640x480 or hd720.", OFFSET(requested_width), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, DEC },
> +    { "framerate", "set video frame rate", OFFSET(framerate), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, DEC },
> +    { "sample_rate", "set audio sample rate", OFFSET(sample_rate), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> +    { "sample_size", "set audio sample size", OFFSET(sample_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 16, DEC },
> +    { "channels", "set number of audio channels, such as 1 or 2", OFFSET(channels), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> +    { "audio_buffer_size", "set audio device buffer latency size in milliseconds (default is the device's default)", OFFSET(audio_buffer_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> +    { NULL },
> +};
> +
> +static const AVClass android_capture_class = {
> +    .class_name = "android_capture indev",
> +    .item_name  = av_default_item_name,
> +    .option     = options,
> +    .version    = LIBAVDEVICE_VERSION_INT,
> +    .category   = AV_CLASS_CATEGORY_DEVICE_VIDEO_INPUT,
> +};
> +
> +AVInputFormat ff_android_capture_demuxer = {
> +    .name           = "android_capture",
> +    .long_name      = NULL_IF_CONFIG_SMALL("Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)"),
> +    .priv_data_size = sizeof(android_capture_ctx),
> +    .read_header    = android_capture_read_header,
> +    .read_packet    = android_capture_read_packet,
> +    .read_close     = android_capture_read_close,
> +    .flags          = AVFMT_NOFILE,
> +    .priv_class     = &android_capture_class,
> +};
> diff --git a/libavdevice/android_capture.h b/libavdevice/android_capture.h
> new file mode 100644
> index 0000000000..44c888ea67
> --- /dev/null
> +++ b/libavdevice/android_capture.h
> @@ -0,0 +1,77 @@
> +/*
> + * Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)
> + *
> + * Copyright (C) 2017 Felix Matouschek
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#ifndef AVDEVICE_ANDROID_CAPTURE_H
> +#define AVDEVICE_ANDROID_CAPTURE_H
> +
> +#include <pthread.h>
> +
> +#include <camera/NdkCameraManager.h>
> +#include <media/NdkImageReader.h>
> +
> +#include "libavutil/log.h"
> +#include "libavformat/avformat.h"
> +

> +typedef struct android_capture_ctx {

As much as I dislike it, the preferred form for context types is
CamelCase.

> +    const AVClass *class;
> +
> +    int requested_width;
> +    int requested_height;
> +    char *framerate;
> +    int sample_rate;
> +    int sample_size;
> +    int channels;
> +    int audio_buffer_size;
> +
> +    int video_device_number;
> +    int audio_device_number;
> +    uint8_t lens_facing;
> +    int32_t sensor_orientation;
> +    int width;
> +    int height;
> +    AVRational requested_framerate;
> +    int32_t framerate_range[2];
> +    int video_stream_index;
> +
> +    ACameraManager *camera_mgr;
> +    char *camera_id;
> +    ACameraMetadata *camera_metadata;
> +    ACameraDevice *camera_dev;
> +    ACameraDevice_stateCallbacks camera_state_callbacks;
> +    AImageReader *image_reader;
> +    AImageReader_ImageListener image_listener;
> +    ANativeWindow *image_reader_window;
> +    ACaptureSessionOutputContainer *capture_session_output_container;
> +    ACaptureSessionOutput *capture_session_output;
> +    ACameraOutputTarget *camera_output_target;
> +    ACaptureRequest *capture_request;
> +    ACameraCaptureSession_stateCallbacks capture_session_state_callbacks;
> +    ACameraCaptureSession *capture_session;
> +
> +    AVPacketList *pktl;
> +    pthread_mutex_t mutex;
> +    int read_closing;
> +    int64_t curbufsize[2];
> +    int64_t video_frame_num;
> +} android_capture_ctx;
> +
> +#endif /* AVDEVICE_ANDROID_CAPTURE_H */

Regards,

In addition to Nicolas's, further (style) comments:

> Subject: [PATCH] Add android_capture indev
More like:
    avdevice: add android_capture [or android_camera] indev

>  configure                     |   6 +
>  libavdevice/Makefile          |   1 +
>  libavdevice/alldevices.c      |   1 +
>  libavdevice/android_capture.c | 782 ++++++++++++++++++++++++++++++++++++++++++
>  libavdevice/android_capture.h |  77 +++++
>  5 files changed, 867 insertions(+)
>  create mode 100644 libavdevice/android_capture.c
>  create mode 100644 libavdevice/android_capture.h

A Changelog entry, documentation in doc/indevs.texi, and a libavdevice
version bump need to be added.

>  # indevs / outdevs
> +android_capture_indev_deps="android mediandk camera2ndk pthreads"
> +android_capture_indev_extralibs="-landroid -lmediandk -lcamera2ndk"
>  alsa_indev_deps="alsa"
>  alsa_outdev_deps="alsa"

Alphabetical order.

> +    av_log(avctx, AV_LOG_ERROR, "Error %d on camera with id %s.\n", error,
> +            ACameraDevice_getId(device));

As Nicolas suggested, can you stringify the error codes?

> +        av_log(avctx, AV_LOG_ERROR,
> +                "Failed to get camera id list, camera_status_t: %d.\n", ret);

camera_status_t seems like an internal naming. (Also media_status_t and
the likes.) Not sure whether other words would be better. Without
stringification, it's only useful to the developer anyway.

> +        av_log(avctx, AV_LOG_ERROR,
> +                "real-time buffer of [%s input] too full or near too full (%d%% of size: %d [rtbufsize parameter])! frame dropped!\n",
> +                stream_name, buffer_fullness, avctx->max_picture_buffer);

I (personally) mislike exclamation marks in messages.

> +    { "sample_rate", "set audio sample rate", OFFSET(sample_rate), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> +    { "sample_size", "set audio sample size", OFFSET(sample_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 16, DEC },
> +    { "channels", "set number of audio channels, such as 1 or 2", OFFSET(channels), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> +    { "audio_buffer_size", "set audio device buffer latency size in milliseconds (default is the device's default)", OFFSET(audio_buffer_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },

While these aren't being used, don't expose them. (Will disappear
anyway if you split the functionality.)

> +    .long_name      = NULL_IF_CONFIG_SMALL("Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)"),

Perhaps a bit verbose, but if you split it up into video and audio as
suggested by Nicolas, you can shorten it. (I'm not sure "via Android
NDK APIs" is the correct to present.)

Moritz

2017-11-02 13:42 GMT+01:00 Felix Matouschek <felix@matouschek.org>:
> Hello,
>
> I've written an indev for Android devices to allow capturing their builtin
> cameras.
> What needs to be done to merge this?
>
> Greetings,
> Felix
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>

Looking forrward to this to be included. Just one note:

+    if (ctx->video_device_number < camera_ids->numCameras) {
+        ctx->camera_id = av_strdup(
+                camera_ids->cameraIds[ctx->video_device_number]);
+    } else {
+        av_log(avctx, AV_LOG_ERROR, "No camera with number %d available.\n",
+                ctx->video_device_number);
+        return AVERROR(ENXIO);
+    }

some devices have cameras which are unlisted - this condition makes
them unusable.

On 02.11.2017 16:20, Daniel Kučera wrote:
> 2017-11-02 13:42 GMT+01:00 Felix Matouschek <felix@matouschek.org>:
>> Hello,
>>
>> I've written an indev for Android devices to allow capturing their builtin
>> cameras.
>> What needs to be done to merge this?
>>
>> Greetings,
>> Felix
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
> Looking forrward to this to be included. 

+ 1

Best regards,
Thomas.

Hello,

Am 02.11.2017 um 14:40 schrieb Nicolas George:
> When reading the subject of the mail, I first thought it would be about
> screen capture. Furthermore, there is code for audio, but it is not
> connected to anything, and it does not seem that the android API
> connects audio and video together.
>
> For all these reasons, I suggest you name this device maybe
> android_camera, and keep android_mic for audio capture.
Will do so.

> You are parsing the "filename" of the device into a key=value syntax.
> This is not a good idea, and I really would not like another key=value
> parser in the code base.
>
> How do cameraIds look? If the string values are friendly enough, they
> can be used as filenames as is.
I removed parsing of the "filename" and replaced it with a 
"camera_index" parameter.

I do not know how cameraIds are formed on different devices, seems like 
internal cameras
just use numbers but external devices may use a string. I do not have 
multiple devices to test
this. In my opionion an index for the list of all cameras should be 
sufficient.

> Can the error be translated into a human-readable message?
Will try to do that, I need to add the according strings for it somehow.

> Is it really the official way of getting the resolution and format of
> the video? If so, were the Android people drunk?
Seem like it.

>> +        if (format == IMAGE_FORMAT_ANDROID) {
> It would be better to support as many pixel formats as possible.
For now I settled on YUV420P, as the API doc states all devices must 
support it. I do not
have the resources to test every format.

>> +    side_data = av_packet_new_side_data(&pktl_next->pkt,
>> +            AV_PKT_DATA_DISPLAYMATRIX, sizeof(display_matrix));
> It would probably be best to avoid sending the side data repeatedly if
> it does not change.
Is it sufficient to append the matrix just once? I thought every 
AVPacket could have a different matrix.

>> +error:
>> +    pthread_mutex_unlock(&ctx->mutex);
>> +    AImage_delete(image);
>> +
>> +    return;
> It seems error conditions are not taken into account. Is it on purpose?
You mean for example ENOMEM if allocating fails and aborting the whole 
"session"? Could do that.

> Regards, 

I working on addressing all issues.

Greetings,
Felix

2017-11-03 16:21 GMT+01:00 Felix Matouschek <felix@matouschek.org>:

>>> +        if (format == IMAGE_FORMAT_ANDROID) {
>>
>> It would be better to support as many pixel formats as possible.
>
> For now I settled on YUV420P, as the API doc states all
> devices must support it. I do not have the resources to
> test every format.

You could add those that you can test (but yuv420p is of
course a very useful start).

Carl Eugen

Am 02.11.2017 16:20, schrieb Daniel Kučera:

> some devices have cameras which are unlisted - this condition makes
> them unusable.

How does one use unlisted devices? As far as I understood all supported
devices are listed. If the Camera2 API does not support the camera
it cannot be used.

2017-11-07 15:25 GMT+01:00 Felix Matouschek <felix@matouschek.org>:
> Am 02.11.2017 16:20, schrieb Daniel Kučera:
>
>> some devices have cameras which are unlisted - this condition makes
>> them unusable.
>
>
> How does one use unlisted devices? As far as I understood all supported
> devices are listed. If the Camera2 API does not support the camera
> it cannot be used.
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

It's used like here:
https://github.com/danielkucera/ZidoStreamer/blob/master/app/src/main/java/eu/danman/zidostreamer/zidostreamer/StreamService.java#L62

If I open standard streaming or camera app, no camera is listed.

> Am 07.11.2017 um 16:17 schrieb Daniel Kučera <daniel.kucera@gmail.com>:
> 
> It's used like here:
> https://github.com/danielkucera/ZidoStreamer/blob/master/app/src/main/java/eu/danman/zidostreamer/zidostreamer/StreamService.java#L62
> 
> If I open standard streaming or camera app, no camera is listed.

Note that your application uses the Camera Java API, this indev uses the Camera2 NDK API.

I think this cannot be compared directly. Could you test it in your specific case, whether the camera is somehow available through the NDK? Please note that the Camera2 NDK API only works on devices with API Level 24+ (Android 7.0).

As far as I have seen the camera in your device must also specifically support the Camera2 API. Cameras that are designed for the old API are only available in the Java version of the Camera2 API. 

You can check this in Java with: https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics.html#INFO_SUPPORTED_HARDWARE_LEVEL

The NDK version only supports cameras that are on the hardware level LIMITED or above.

Greetings,
Felix

2017-11-07 18:47 GMT+01:00 Felix Matouschek <felix@matouschek.org>:
>
>> Am 07.11.2017 um 16:17 schrieb Daniel Kučera <daniel.kucera@gmail.com>:
>>
>> It's used like here:
>> https://github.com/danielkucera/ZidoStreamer/blob/master/app/src/main/java/eu/danman/zidostreamer/zidostreamer/StreamService.java#L62
>>
>> If I open standard streaming or camera app, no camera is listed.
>
> Note that your application uses the Camera Java API, this indev uses the Camera2 NDK API.
>

You are probably right. My device has Android 4.4.2 only without any
hope for update so it won't work anyway.

Hello,

here is a new version of the patch with further fixes.

Greetings,
Felix

Ping

No more interest?

Am 02.11.2017 13:42, schrieb Felix Matouschek:
> Hello,
> 
> I've written an indev for Android devices to allow capturing their
> builtin cameras.
> What needs to be done to merge this?
> 
> Greetings,
> Felix
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

On Tue, Nov 21, 2017 at 08:16:08AM +0100, Felix Matouschek wrote:
> No more interest?

If you hear no more comments for another week, then please ping this
with CC to me and ill take a look and apply unless i spot some major
issue

also you probably want to add yourself to the MAINTAINER file if you
intend to maintain the code

thanks

[...]

Am 22.11.2017 00:20, schrieb Michael Niedermayer:
> On Tue, Nov 21, 2017 at 08:16:08AM +0100, Felix Matouschek wrote:
>> No more interest?
> 
> If you hear no more comments for another week, then please ping this
> with CC to me and ill take a look and apply unless i spot some major
> issue
> 
> also you probably want to add yourself to the MAINTAINER file if you
> intend to maintain the code

Hello Michael,

as you suggested: Ping.

I added myself to the MAINTAINERS file.

One last note if you try to build it yourself:
Your NDK needs to have the patch in commit be9b8bc of this
issue https://github.com/android-ndk/ndk/issues/559 applied.

As of NDK R17 this patch will probably be included in the NDK release.

Greetings,
Felix

2017-11-28 8:50 GMT+01:00 Felix Matouschek <felix@matouschek.org>:

> One last note if you try to build it yourself:
> Your NDK needs to have the patch in commit be9b8bc of this
> issue https://github.com/android-ndk/ndk/issues/559 applied.

Sorry if I misunderstand:
The patch breaks default compilation for Android?

Carl Eugen

Am 28.11.2017 14:01, schrieb Carl Eugen Hoyos:

> The patch breaks default compilation for Android?

Actually... you are right, it does.
It's a bit unfortunate as the header file in the NDK is broken.

We can't work around that in ffmpeg, the NDK needs to be fixed.
As soon as you include NdkCameraManager.h even to official examples fail 
to build.

Sorry... tested it again, it does not break the default compilation.

The library check in the configure script just fails and the indev is 
not included when using the broken NDK.

On Fri, Nov 10, 2017 at 09:41:04PM +0100, Felix Matouschek wrote:
> Hello,
> 
> here is a new version of the patch with further fixes.
[...]

> +static const char *camera_status_string(camera_status_t val)
> +{
> +    switch(val) {
> +        case ACAMERA_OK:
> +            return "ACAMERA_OK";

if the identifer and the string always match you could do this
with a macro avoiding the neede to duplcate each string
see AV_STRINGIFY


[...]
> +static int read_closing(AndroidCameraCtx *ctx)
> +{
> +    int read_closing;
> +    pthread_mutex_lock(&ctx->read_closing_mutex);
> +    read_closing = ctx->read_closing;
> +    pthread_mutex_unlock(&ctx->read_closing_mutex);
> +    return read_closing;
> +}
> +
> +static void set_read_closing(AndroidCameraCtx *ctx, int read_closing)
> +{
> +    pthread_mutex_lock(&ctx->read_closing_mutex);
> +    ctx->read_closing = read_closing;
> +    pthread_mutex_unlock(&ctx->read_closing_mutex);
> +}

Thic can be simplified using C11 atomics


[...]
> +static void match_video_size(AVFormatContext *avctx)
> +{
> +    AndroidCameraCtx *ctx = avctx->priv_data;
> +    ACameraMetadata_const_entry available_configs;
> +    int ret = -1;
> +
> +    ACameraMetadata_getConstEntry(ctx->camera_metadata,
> +                                  ACAMERA_SCALER_AVAILABLE_STREAM_CONFIGURATIONS,
> +                                  &available_configs);
> +
> +    for (int i = 0; i < available_configs.count; i++) {
> +        int32_t input = available_configs.data.i32[i * 4 + 3];
> +        int32_t format = available_configs.data.i32[i * 4 + 0];
> +
> +        if (input) {
> +            continue;
> +        }
> +
> +        if (format == IMAGE_FORMAT_ANDROID) {
> +            int32_t width = available_configs.data.i32[i * 4 + 1];
> +            int32_t height = available_configs.data.i32[i * 4 + 2];
> +
> +            //Same ratio

> +            if (ctx->requested_width * ctx->requested_height == width * height) {

the product of these can maybe overflow i think


> +                ctx->width = width;
> +                ctx->height = height;
> +                ret = 0;
> +                break;
> +            }
> +        }
> +    }
> +

> +    if (ret < 0 || ctx->width == 0 || ctx->height == 0) {
> +        ctx->width = available_configs.data.i32[1];
> +        ctx->height = available_configs.data.i32[2];
> +        ret = -1;
> +
> +        av_log(avctx, AV_LOG_WARNING,
> +               "Requested video_size %dx%d not available, falling back to %dx%d\n",
> +               ctx->requested_width, ctx->requested_height, ctx->width, ctx->height);
> +    }

you set ret but nothing uses it

> +
> +    return;
> +}

[...]
> +static int add_video_stream(AVFormatContext *avctx)
> +{
> +    AndroidCameraCtx *ctx = avctx->priv_data;
> +    AVStream *st;
> +    AVCodecParameters *codecpar;
> +
> +    st = avformat_new_stream(avctx, NULL);
> +    if (!st) {
> +        return AVERROR(ENOMEM);
> +    }
> +
> +    st->id = VIDEO_STREAM_INDEX;

> +    st->avg_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> +    st->r_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };

Are these values always correct ?


[...]
> +    if (ctx->camera_id) {
> +        av_freep(&ctx->camera_id);
> +    }

the if() is unneeded


[...]

Am 29.11.2017 04:31, schrieb Michael Niedermayer:

> if the identifer and the string always match you could do this
> with a macro avoiding the neede to duplcate each string
> see AV_STRINGIFY

I changed it, is it ok like this?

> [...]
>> +static int add_video_stream(AVFormatContext *avctx)
>> +{
>> +    AndroidCameraCtx *ctx = avctx->priv_data;
>> +    AVStream *st;
>> +    AVCodecParameters *codecpar;
>> +
>> +    st = avformat_new_stream(avctx, NULL);
>> +    if (!st) {
>> +        return AVERROR(ENOMEM);
>> +    }
>> +
>> +    st->id = VIDEO_STREAM_INDEX;
> 
>> +    st->avg_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
>> +    st->r_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> 
> Are these values always correct ?

You mean avg_frame_rate and r_frame_rate?
The framerate can vary between the values in framerate_range[0] (min) 
and framerate_range[1] (max).
Ideally both values are the same, sometimes min can be lower but for the 
average the framerate should be what is in max.
Should I set r_frame_rate to min?

I fixed all other parts you mentioned.

Felix

Sorry, my mail client swallowed the attachment, sent it again.
> Am 30.11.2017 um 10:12 schrieb Felix Matouschek <felix@matouschek.org>:
> 
> Am 29.11.2017 04:31, schrieb Michael Niedermayer:
> 
>> if the identifer and the string always match you could do this
>> with a macro avoiding the neede to duplcate each string
>> see AV_STRINGIFY
> 
> I changed it, is it ok like this?
> 
>> [...]
>>> +static int add_video_stream(AVFormatContext *avctx)
>>> +{
>>> +    AndroidCameraCtx *ctx = avctx->priv_data;
>>> +    AVStream *st;
>>> +    AVCodecParameters *codecpar;
>>> +
>>> +    st = avformat_new_stream(avctx, NULL);
>>> +    if (!st) {
>>> +        return AVERROR(ENOMEM);
>>> +    }
>>> +
>>> +    st->id = VIDEO_STREAM_INDEX;
>>> +    st->avg_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
>>> +    st->r_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
>> Are these values always correct ?
> 
> You mean avg_frame_rate and r_frame_rate?
> The framerate can vary between the values in framerate_range[0] (min) and framerate_range[1] (max).
> Ideally both values are the same, sometimes min can be lower but for the average the framerate should be what is in max.
> Should I set r_frame_rate to min?
> 
> I fixed all other parts you mentioned.
> 
> Felix
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

On Thu, Nov 30, 2017 at 10:12:53AM +0100, Felix Matouschek wrote:
> Am 29.11.2017 04:31, schrieb Michael Niedermayer:
> 
> >if the identifer and the string always match you could do this
> >with a macro avoiding the neede to duplcate each string
> >see AV_STRINGIFY
> 
> I changed it, is it ok like this?
> 
> >[...]
> >>+static int add_video_stream(AVFormatContext *avctx)
> >>+{
> >>+    AndroidCameraCtx *ctx = avctx->priv_data;
> >>+    AVStream *st;
> >>+    AVCodecParameters *codecpar;
> >>+
> >>+    st = avformat_new_stream(avctx, NULL);
> >>+    if (!st) {
> >>+        return AVERROR(ENOMEM);
> >>+    }
> >>+
> >>+    st->id = VIDEO_STREAM_INDEX;
> >
> >>+    st->avg_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> >>+    st->r_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> >
> >Are these values always correct ?
> 
> You mean avg_frame_rate and r_frame_rate?
> The framerate can vary between the values in framerate_range[0]
> (min) and framerate_range[1] (max).
> Ideally both values are the same, sometimes min can be lower but for
> the average the framerate should be what is in max.
> Should I set r_frame_rate to min?

I think if you do not know the base or average frame rate you
should not set it.
Or does this lead to some unreasonable latency ?
or are the computed values worse ?


> 
> I fixed all other parts you mentioned.
> 
> Felix
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Am 30.11.2017 14:48, schrieb Michael Niedermayer:
>> You mean avg_frame_rate and r_frame_rate?
>> The framerate can vary between the values in framerate_range[0]
>> (min) and framerate_range[1] (max).
>> Ideally both values are the same, sometimes min can be lower but for
>> the average the framerate should be what is in max.
>> Should I set r_frame_rate to min?
> 
> I think if you do not know the base or average frame rate you
> should not set it.
> Or does this lead to some unreasonable latency ?
> or are the computed values worse ?

On some devices min and max are the same so the framerate is fixed and I 
know it.
On other devices the min framerate can be lower and so the actual 
framerate can vary between both values.

On average the framerate still should be around the maximum value.

Do you think I'm better off not setting it at all? Will it then get 
computed automatically?

On Thu, Nov 30, 2017 at 05:29:45PM +0100, Felix Matouschek wrote:
> Am 30.11.2017 14:48, schrieb Michael Niedermayer:
> >>You mean avg_frame_rate and r_frame_rate?
> >>The framerate can vary between the values in framerate_range[0]
> >>(min) and framerate_range[1] (max).
> >>Ideally both values are the same, sometimes min can be lower but for
> >>the average the framerate should be what is in max.
> >>Should I set r_frame_rate to min?
> >
> >I think if you do not know the base or average frame rate you
> >should not set it.
> >Or does this lead to some unreasonable latency ?
> >or are the computed values worse ?
> 
> On some devices min and max are the same so the framerate is fixed
> and I know it.
> On other devices the min framerate can be lower and so the actual
> framerate can vary between both values.
> 
> On average the framerate still should be around the maximum value.
> 

> Do you think I'm better off not setting it at all? Will it then get
> computed automatically?

yes, if you dont set it, it will be computet but it would give some
latency on startup as it needs a few frames first to base the computation
on. This latency may or may not be a bigger problem than slightly
incorrect values. I dont know, its up to you i think what you feel
is more important

[...]

On Thu, Nov 30, 2017 at 10:15:48AM +0100, Felix Matouschek wrote:
> Sorry, my mail client swallowed the attachment, sent it again.
> 

>  Changelog                    |    1 
>  MAINTAINERS                  |    1 
>  configure                    |    6 
>  doc/indevs.texi              |   40 ++
>  libavdevice/Makefile         |    1 
>  libavdevice/alldevices.c     |    1 
>  libavdevice/android_camera.c |  816 +++++++++++++++++++++++++++++++++++++++++++
>  libavdevice/version.h        |    2 
>  8 files changed, 867 insertions(+), 1 deletion(-)
> 2fea6cb3990aed83c3a3c492aa482f619a885ed7  0001-avdevice-add-android_camera-indev.patch
> From b70da28e33e07b4565daffc94c8ffe3c8df747ff Mon Sep 17 00:00:00 2001
> From: Felix Matouschek <felix@matouschek.org>
> Date: Thu, 30 Nov 2017 10:03:54 +0100
> Subject: [PATCH] avdevice: add android_camera indev
> To: ffmpeg-devel@ffmpeg.org
> 
> This commit adds an indev for Android devices on API level 24+ which
> uses the Android NDK Camera2 API to capture video from builtin cameras
> 
> Signed-off-by: Felix Matouschek <felix@matouschek.org>
> ---
>  Changelog                    |   1 +
>  MAINTAINERS                  |   1 +
>  configure                    |   6 +
>  doc/indevs.texi              |  40 +++
>  libavdevice/Makefile         |   1 +
>  libavdevice/alldevices.c     |   1 +
>  libavdevice/android_camera.c | 816 +++++++++++++++++++++++++++++++++++++++++++
>  libavdevice/version.h        |   2 +-
>  8 files changed, 867 insertions(+), 1 deletion(-)
>  create mode 100644 libavdevice/android_camera.c
> 
> diff --git a/Changelog b/Changelog
> index 6592d868da..f58cd810e0 100644
> --- a/Changelog
> +++ b/Changelog
> @@ -6,6 +6,7 @@ version <next>:
>  - Dropped support for OpenJPEG versions 2.0 and below. Using OpenJPEG now
>    requires 2.1 (or later) and pkg-config.
>  - VDA dropped (use VideoToolbox instead)
> +- Add android_camera indev
>  
>  
>  version 3.4:
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 1d2ff78b0e..d6cb135964 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -281,6 +281,7 @@ libavdevice
>  
>  
>    avfoundation.m                        Thilo Borgmann
> +  android_camera.c                      Felix Matouschek
>    decklink*                             Marton Balint
>    dshow.c                               Roger Pack (CC rogerdpack@gmail.com)
>    fbdev_enc.c                           Lukasz Marek
> diff --git a/configure b/configure
> index 7a53bc76c7..d52b18fab3 100755
> --- a/configure
> +++ b/configure
> @@ -3068,6 +3068,8 @@ xmv_demuxer_select="riffdec"
>  xwma_demuxer_select="riffdec"
>  
>  # indevs / outdevs
> +android_camera_indev_deps="android camera2ndk mediandk pthreads"
> +android_camera_indev_extralibs="-landroid -lcamera2ndk -lmediandk"
>  alsa_indev_deps="alsa"
>  alsa_outdev_deps="alsa"
>  avfoundation_indev_deps="avfoundation corevideo coremedia pthreads"
> @@ -5836,6 +5838,10 @@ check_lib shell32  "windows.h shellapi.h" CommandLineToArgvW   -lshell32
>  check_lib wincrypt "windows.h wincrypt.h" CryptGenRandom       -ladvapi32
>  check_lib psapi    "windows.h psapi.h"    GetProcessMemoryInfo -lpsapi
>  
> +check_lib android android/native_window.h ANativeWindow_acquire -landroid
> +check_lib mediandk "stdint.h media/NdkImage.h" AImage_delete -lmediandk
> +check_lib camera2ndk "stdbool.h stdint.h camera/NdkCameraManager.h" ACameraManager_create -lcamera2ndk
> +
>  enabled appkit       && check_apple_framework AppKit
>  enabled audiotoolbox && check_apple_framework AudioToolbox
>  enabled avfoundation && check_apple_framework AVFoundation
> diff --git a/doc/indevs.texi b/doc/indevs.texi
> index d308bbf7de..07056d762e 100644
> --- a/doc/indevs.texi
> +++ b/doc/indevs.texi
> @@ -63,6 +63,46 @@ Set the number of channels. Default is 2.
>  
>  @end table
>  
> +@section android_camera
> +
> +Android camera input device.
> +
> +This input devices uses the Android Camera2 NDK API which is
> +available on devices with API level 24+. The availability of
> +android_camera is autodetected during configuration.
> +
> +This device allows capturing from all cameras on an Android device,
> +which are integrated into the Camera2 NDK API.
> +
> +The available cameras are enumerated internally and can be selected
> +with the @var{camera_index} parameter. The input file string is
> +discarded.
> +
> +Generally the back facing camera has index 0 while the front facing
> +camera has index 1.
> +
> +@subsection Options
> +
> +@table @option
> +
> +@item video_size
> +Set the video size given as a string such as 640x480 or hd720.
> +Falls back to the first available configuration reported by
> +Android if requested video size is not available or by default.
> +
> +@item framerate
> +Set the video framerate.
> +Falls back to the first available configuration reported by
> +Android if requested framerate is not available or by default (-1).
> +
> +@item camera_index
> +Set the index of the camera to use. Default is 0.
> +
> +@item input_queue_size
> +Set the maximum number of frames to buffer. Default is 5.
> +
> +@end table
> +
>  @section avfoundation
>  
>  AVFoundation input device.
> diff --git a/libavdevice/Makefile b/libavdevice/Makefile
> index 8228d62147..f11a6f2a86 100644
> --- a/libavdevice/Makefile
> +++ b/libavdevice/Makefile
> @@ -14,6 +14,7 @@ OBJS-$(CONFIG_SHARED)                    += reverse.o
>  # input/output devices
>  OBJS-$(CONFIG_ALSA_INDEV)                += alsa_dec.o alsa.o timefilter.o
>  OBJS-$(CONFIG_ALSA_OUTDEV)               += alsa_enc.o alsa.o
> +OBJS-$(CONFIG_ANDROID_CAMERA_INDEV)      += android_camera.o
>  OBJS-$(CONFIG_AVFOUNDATION_INDEV)        += avfoundation.o
>  OBJS-$(CONFIG_BKTR_INDEV)                += bktr.o
>  OBJS-$(CONFIG_CACA_OUTDEV)               += caca.o
> diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c
> index b767b6a718..2c8d9035da 100644
> --- a/libavdevice/alldevices.c
> +++ b/libavdevice/alldevices.c
> @@ -42,6 +42,7 @@ static void register_all(void)
>  {
>      /* devices */
>      REGISTER_INOUTDEV(ALSA,             alsa);
> +    REGISTER_INDEV   (ANDROID_CAMERA,   android_camera);
>      REGISTER_INDEV   (AVFOUNDATION,     avfoundation);
>      REGISTER_INDEV   (BKTR,             bktr);
>      REGISTER_OUTDEV  (CACA,             caca);
> diff --git a/libavdevice/android_camera.c b/libavdevice/android_camera.c
> new file mode 100644
> index 0000000000..eaef032a4c
> --- /dev/null
> +++ b/libavdevice/android_camera.c
> @@ -0,0 +1,816 @@
> +/*
> + * Android camera input device
> + *
> + * Copyright (C) 2017 Felix Matouschek
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <errno.h>
> +#include <pthread.h>
> +#include <stdatomic.h>
> +#include <stdbool.h>
> +#include <stdint.h>
> +
> +#include <camera/NdkCameraDevice.h>
> +#include <camera/NdkCameraManager.h>
> +#include <media/NdkImage.h>
> +#include <media/NdkImageReader.h>
> +
> +#include "libavformat/avformat.h"
> +#include "libavformat/internal.h"
> +#include "libavutil/avstring.h"
> +#include "libavutil/display.h"
> +#include "libavutil/imgutils.h"
> +#include "libavutil/log.h"
> +#include "libavutil/opt.h"
> +#include "libavutil/parseutils.h"
> +#include "libavutil/threadmessage.h"
> +#include "libavutil/time.h"
> +
> +#include "version.h"
> +
> +/* This image format is available on all Android devices
> + * supporting the Camera2 API */
> +#define IMAGE_FORMAT_ANDROID AIMAGE_FORMAT_YUV_420_888
> +#define IMAGE_FORMAT_FFMPEG AV_PIX_FMT_YUV420P
> +#define IMAGE_NUM_PLANES 3
> +
> +#define MAX_BUF_COUNT 2
> +#define VIDEO_STREAM_INDEX 0
> +#define VIDEO_TIMEBASE 1000000000
> +
> +typedef struct AndroidCameraCtx {
> +    const AVClass *class;
> +
> +    int requested_width;
> +    int requested_height;
> +    AVRational framerate;
> +    int camera_index;
> +    int input_queue_size;
> +
> +    uint8_t lens_facing;
> +    int32_t sensor_orientation;
> +    int width;
> +    int height;
> +    int32_t framerate_range[2];
> +
> +    ACameraManager *camera_mgr;
> +    char *camera_id;
> +    ACameraMetadata *camera_metadata;
> +    ACameraDevice *camera_dev;
> +    ACameraDevice_StateCallbacks camera_state_callbacks;
> +    AImageReader *image_reader;
> +    AImageReader_ImageListener image_listener;
> +    ANativeWindow *image_reader_window;
> +    ACaptureSessionOutputContainer *capture_session_output_container;
> +    ACaptureSessionOutput *capture_session_output;
> +    ACameraOutputTarget *camera_output_target;
> +    ACaptureRequest *capture_request;
> +    ACameraCaptureSession_stateCallbacks capture_session_state_callbacks;
> +    ACameraCaptureSession *capture_session;
> +
> +    AVThreadMessageQueue *input_queue;
> +    atomic_int exit;
> +    int display_matrix_sent;
> +} AndroidCameraCtx;
> +
> +static const char *camera_status_string(camera_status_t val)
> +{
> +    switch(val) {
> +        case ACAMERA_OK:
> +            return AV_STRINGIFY(ACAMERA_OK);
> +        case ACAMERA_ERROR_UNKNOWN:
> +            return AV_STRINGIFY(ACAMERA_ERROR_UNKNOWN);
> +        case ACAMERA_ERROR_INVALID_PARAMETER:
> +            return AV_STRINGIFY(ACAMERA_ERROR_INVALID_PARAMETER);

what i meant was something like

#define RETURN_CASE(x) case x: return AV_STRINGIFY(x);
switch(val) {
    RETURN_CASE(ACAMERA_OK)
    RETURN_CASE(ACAMERA_ERROR_UNKNOWN)
    RETURN_CASE(ACAMERA_ERROR_INVALID_PARAMETER)


[...]

Hello Michael,

could you take a look at the patch?

Felix

> Am 30.11.2017 um 18:15 schrieb Michael Niedermayer <michael@niedermayer.cc>:
> 
> On Thu, Nov 30, 2017 at 10:15:48AM +0100, Felix Matouschek wrote:
>> Sorry, my mail client swallowed the attachment, sent it again.
>> 
> 
>> Changelog                    |    1 
>> MAINTAINERS                  |    1 
>> configure                    |    6 
>> doc/indevs.texi              |   40 ++
>> libavdevice/Makefile         |    1 
>> libavdevice/alldevices.c     |    1 
>> libavdevice/android_camera.c |  816 +++++++++++++++++++++++++++++++++++++++++++
>> libavdevice/version.h        |    2 
>> 8 files changed, 867 insertions(+), 1 deletion(-)
>> 2fea6cb3990aed83c3a3c492aa482f619a885ed7  0001-avdevice-add-android_camera-indev.patch
>> From b70da28e33e07b4565daffc94c8ffe3c8df747ff Mon Sep 17 00:00:00 2001
>> From: Felix Matouschek <felix@matouschek.org>
>> Date: Thu, 30 Nov 2017 10:03:54 +0100
>> Subject: [PATCH] avdevice: add android_camera indev
>> To: ffmpeg-devel@ffmpeg.org
>> 
>> This commit adds an indev for Android devices on API level 24+ which
>> uses the Android NDK Camera2 API to capture video from builtin cameras
>> 
>> Signed-off-by: Felix Matouschek <felix@matouschek.org>
>> ---
>> Changelog                    |   1 +
>> MAINTAINERS                  |   1 +
>> configure                    |   6 +
>> doc/indevs.texi              |  40 +++
>> libavdevice/Makefile         |   1 +
>> libavdevice/alldevices.c     |   1 +
>> libavdevice/android_camera.c | 816 +++++++++++++++++++++++++++++++++++++++++++
>> libavdevice/version.h        |   2 +-
>> 8 files changed, 867 insertions(+), 1 deletion(-)
>> create mode 100644 libavdevice/android_camera.c
>> 
>> diff --git a/Changelog b/Changelog
>> index 6592d868da..f58cd810e0 100644
>> --- a/Changelog
>> +++ b/Changelog
>> @@ -6,6 +6,7 @@ version <next>:
>> - Dropped support for OpenJPEG versions 2.0 and below. Using OpenJPEG now
>>   requires 2.1 (or later) and pkg-config.
>> - VDA dropped (use VideoToolbox instead)
>> +- Add android_camera indev
>> 
>> 
>> version 3.4:
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 1d2ff78b0e..d6cb135964 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -281,6 +281,7 @@ libavdevice
>> 
>> 
>>   avfoundation.m                        Thilo Borgmann
>> +  android_camera.c                      Felix Matouschek
>>   decklink*                             Marton Balint
>>   dshow.c                               Roger Pack (CC rogerdpack@gmail.com)
>>   fbdev_enc.c                           Lukasz Marek
>> diff --git a/configure b/configure
>> index 7a53bc76c7..d52b18fab3 100755
>> --- a/configure
>> +++ b/configure
>> @@ -3068,6 +3068,8 @@ xmv_demuxer_select="riffdec"
>> xwma_demuxer_select="riffdec"
>> 
>> # indevs / outdevs
>> +android_camera_indev_deps="android camera2ndk mediandk pthreads"
>> +android_camera_indev_extralibs="-landroid -lcamera2ndk -lmediandk"
>> alsa_indev_deps="alsa"
>> alsa_outdev_deps="alsa"
>> avfoundation_indev_deps="avfoundation corevideo coremedia pthreads"
>> @@ -5836,6 +5838,10 @@ check_lib shell32  "windows.h shellapi.h" CommandLineToArgvW   -lshell32
>> check_lib wincrypt "windows.h wincrypt.h" CryptGenRandom       -ladvapi32
>> check_lib psapi    "windows.h psapi.h"    GetProcessMemoryInfo -lpsapi
>> 
>> +check_lib android android/native_window.h ANativeWindow_acquire -landroid
>> +check_lib mediandk "stdint.h media/NdkImage.h" AImage_delete -lmediandk
>> +check_lib camera2ndk "stdbool.h stdint.h camera/NdkCameraManager.h" ACameraManager_create -lcamera2ndk
>> +
>> enabled appkit       && check_apple_framework AppKit
>> enabled audiotoolbox && check_apple_framework AudioToolbox
>> enabled avfoundation && check_apple_framework AVFoundation
>> diff --git a/doc/indevs.texi b/doc/indevs.texi
>> index d308bbf7de..07056d762e 100644
>> --- a/doc/indevs.texi
>> +++ b/doc/indevs.texi
>> @@ -63,6 +63,46 @@ Set the number of channels. Default is 2.
>> 
>> @end table
>> 
>> +@section android_camera
>> +
>> +Android camera input device.
>> +
>> +This input devices uses the Android Camera2 NDK API which is
>> +available on devices with API level 24+. The availability of
>> +android_camera is autodetected during configuration.
>> +
>> +This device allows capturing from all cameras on an Android device,
>> +which are integrated into the Camera2 NDK API.
>> +
>> +The available cameras are enumerated internally and can be selected
>> +with the @var{camera_index} parameter. The input file string is
>> +discarded.
>> +
>> +Generally the back facing camera has index 0 while the front facing
>> +camera has index 1.
>> +
>> +@subsection Options
>> +
>> +@table @option
>> +
>> +@item video_size
>> +Set the video size given as a string such as 640x480 or hd720.
>> +Falls back to the first available configuration reported by
>> +Android if requested video size is not available or by default.
>> +
>> +@item framerate
>> +Set the video framerate.
>> +Falls back to the first available configuration reported by
>> +Android if requested framerate is not available or by default (-1).
>> +
>> +@item camera_index
>> +Set the index of the camera to use. Default is 0.
>> +
>> +@item input_queue_size
>> +Set the maximum number of frames to buffer. Default is 5.
>> +
>> +@end table
>> +
>> @section avfoundation
>> 
>> AVFoundation input device.
>> diff --git a/libavdevice/Makefile b/libavdevice/Makefile
>> index 8228d62147..f11a6f2a86 100644
>> --- a/libavdevice/Makefile
>> +++ b/libavdevice/Makefile
>> @@ -14,6 +14,7 @@ OBJS-$(CONFIG_SHARED)                    += reverse.o
>> # input/output devices
>> OBJS-$(CONFIG_ALSA_INDEV)                += alsa_dec.o alsa.o timefilter.o
>> OBJS-$(CONFIG_ALSA_OUTDEV)               += alsa_enc.o alsa.o
>> +OBJS-$(CONFIG_ANDROID_CAMERA_INDEV)      += android_camera.o
>> OBJS-$(CONFIG_AVFOUNDATION_INDEV)        += avfoundation.o
>> OBJS-$(CONFIG_BKTR_INDEV)                += bktr.o
>> OBJS-$(CONFIG_CACA_OUTDEV)               += caca.o
>> diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c
>> index b767b6a718..2c8d9035da 100644
>> --- a/libavdevice/alldevices.c
>> +++ b/libavdevice/alldevices.c
>> @@ -42,6 +42,7 @@ static void register_all(void)
>> {
>>     /* devices */
>>     REGISTER_INOUTDEV(ALSA,             alsa);
>> +    REGISTER_INDEV   (ANDROID_CAMERA,   android_camera);
>>     REGISTER_INDEV   (AVFOUNDATION,     avfoundation);
>>     REGISTER_INDEV   (BKTR,             bktr);
>>     REGISTER_OUTDEV  (CACA,             caca);
>> diff --git a/libavdevice/android_camera.c b/libavdevice/android_camera.c
>> new file mode 100644
>> index 0000000000..eaef032a4c
>> --- /dev/null
>> +++ b/libavdevice/android_camera.c
>> @@ -0,0 +1,816 @@
>> +/*
>> + * Android camera input device
>> + *
>> + * Copyright (C) 2017 Felix Matouschek
>> + *
>> + * This file is part of FFmpeg.
>> + *
>> + * FFmpeg is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU Lesser General Public
>> + * License as published by the Free Software Foundation; either
>> + * version 2.1 of the License, or (at your option) any later version.
>> + *
>> + * FFmpeg is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + * Lesser General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU Lesser General Public
>> + * License along with FFmpeg; if not, write to the Free Software
>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
>> + */
>> +
>> +#include <errno.h>
>> +#include <pthread.h>
>> +#include <stdatomic.h>
>> +#include <stdbool.h>
>> +#include <stdint.h>
>> +
>> +#include <camera/NdkCameraDevice.h>
>> +#include <camera/NdkCameraManager.h>
>> +#include <media/NdkImage.h>
>> +#include <media/NdkImageReader.h>
>> +
>> +#include "libavformat/avformat.h"
>> +#include "libavformat/internal.h"
>> +#include "libavutil/avstring.h"
>> +#include "libavutil/display.h"
>> +#include "libavutil/imgutils.h"
>> +#include "libavutil/log.h"
>> +#include "libavutil/opt.h"
>> +#include "libavutil/parseutils.h"
>> +#include "libavutil/threadmessage.h"
>> +#include "libavutil/time.h"
>> +
>> +#include "version.h"
>> +
>> +/* This image format is available on all Android devices
>> + * supporting the Camera2 API */
>> +#define IMAGE_FORMAT_ANDROID AIMAGE_FORMAT_YUV_420_888
>> +#define IMAGE_FORMAT_FFMPEG AV_PIX_FMT_YUV420P
>> +#define IMAGE_NUM_PLANES 3
>> +
>> +#define MAX_BUF_COUNT 2
>> +#define VIDEO_STREAM_INDEX 0
>> +#define VIDEO_TIMEBASE 1000000000
>> +
>> +typedef struct AndroidCameraCtx {
>> +    const AVClass *class;
>> +
>> +    int requested_width;
>> +    int requested_height;
>> +    AVRational framerate;
>> +    int camera_index;
>> +    int input_queue_size;
>> +
>> +    uint8_t lens_facing;
>> +    int32_t sensor_orientation;
>> +    int width;
>> +    int height;
>> +    int32_t framerate_range[2];
>> +
>> +    ACameraManager *camera_mgr;
>> +    char *camera_id;
>> +    ACameraMetadata *camera_metadata;
>> +    ACameraDevice *camera_dev;
>> +    ACameraDevice_StateCallbacks camera_state_callbacks;
>> +    AImageReader *image_reader;
>> +    AImageReader_ImageListener image_listener;
>> +    ANativeWindow *image_reader_window;
>> +    ACaptureSessionOutputContainer *capture_session_output_container;
>> +    ACaptureSessionOutput *capture_session_output;
>> +    ACameraOutputTarget *camera_output_target;
>> +    ACaptureRequest *capture_request;
>> +    ACameraCaptureSession_stateCallbacks capture_session_state_callbacks;
>> +    ACameraCaptureSession *capture_session;
>> +
>> +    AVThreadMessageQueue *input_queue;
>> +    atomic_int exit;
>> +    int display_matrix_sent;
>> +} AndroidCameraCtx;
>> +
>> +static const char *camera_status_string(camera_status_t val)
>> +{
>> +    switch(val) {
>> +        case ACAMERA_OK:
>> +            return AV_STRINGIFY(ACAMERA_OK);
>> +        case ACAMERA_ERROR_UNKNOWN:
>> +            return AV_STRINGIFY(ACAMERA_ERROR_UNKNOWN);
>> +        case ACAMERA_ERROR_INVALID_PARAMETER:
>> +            return AV_STRINGIFY(ACAMERA_ERROR_INVALID_PARAMETER);
> 
> what i meant was something like
> 
> #define RETURN_CASE(x) case x: return AV_STRINGIFY(x);
> switch(val) {
>    RETURN_CASE(ACAMERA_OK)
>    RETURN_CASE(ACAMERA_ERROR_UNKNOWN)
>    RETURN_CASE(ACAMERA_ERROR_INVALID_PARAMETER)
> 
> 
> [...]
> 
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> If you fake or manipulate statistics in a paper in physics you will never
> get a job again.
> If you fake or manipulate statistics in a paper in medicin you will get
> a job for life at the pharma industry.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel