From patchwork Thu Nov 2 12:42:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Matouschek X-Patchwork-Id: 5831 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.90 with SMTP id m26csp2042419jah; Thu, 2 Nov 2017 05:42:23 -0700 (PDT) X-Google-Smtp-Source: ABhQp+Q82p8vjATRmKN5ms0FWLoMkH4ouH8PaymusKScHPyJ56xA81fr6sNYFsMDdhR6ECn74rTL X-Received: by 10.28.168.203 with SMTP id r194mr1553997wme.2.1509626543283; Thu, 02 Nov 2017 05:42:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1509626543; cv=none; d=google.com; s=arc-20160816; b=fD3qMLbJGPxOiPfha/DysbIfFqfx7JaG+xJ1FVpazRlLku9gWmt9zXhqsZFdgQs8F5 Kj3gzWE5LsT56jafa++FXFvXYbumRVXsFfX6eEw3RAZP0o6ZGGiqb/03jbXU7EWcbKIE WgAtj0QzCOykglGrieARx9bmthbzPxt5sE01MCdSuyEgYqQOfEBug6zdwSP5FrysdRR8 D+Arq4Jww4ww6W2jQDAgyBNRUIZp7XYVCEcmjgSCxqF7NNvvsgjsyrQ2F0mDFJCzK0/B pt5OBftEd2sBhkHV0dErHjqNlNRnym1WR5wJnJTJDzle9yxWTc+/tciJH5m5LUbkkTbl YZVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:message-id :to:from:date:mime-version:dkim-signature:delivered-to :arc-authentication-results; bh=EADlgkf+1rZwQo0ya0ASyZJcSXCIGaL7adteEIHwbFY=; b=Q/7v4oPa1jvheZgEvfw2VrbCxvuCx/JMQrKlCJ26CBCLboQkMIvfdq50jVPsfVHZT3 yZSeaTD068OXyy3xU7iNeb6sLa841o8iP9Ta4+/8696EdjtZU5CuF5WSBlKArWAI+pXk Q/IO/7LF9HmbI2URDEXVhnm/8sqELFMOY2URnOxKx0Euru9RKOYAEPBh0/7MxuX2oIqA 2GoifWWaBAfRrjYXGKMLBdo85xpCvOe434R6oqbj/8gFkbdvtLRdw8Rj+QOXnH0VXuAU LLGxIprg/v1ywz6nmmahq2+czPCjsyZoyGQYrj1w1UmDh/s+8w8XIpUZnDYBc3ijziEB vGkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@matouschek.org header.s=mail header.b=Aen7Gdkz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=matouschek.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id o3si2779572wrh.357.2017.11.02.05.42.22; Thu, 02 Nov 2017 05:42:23 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@matouschek.org header.s=mail header.b=Aen7Gdkz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=matouschek.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B470668A3B6; Thu, 2 Nov 2017 14:42:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.fmat.eu (mail.fmat.eu [37.120.191.63]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2813868A2CC for ; Thu, 2 Nov 2017 14:42:04 +0200 (EET) Received: from authenticated-user (mail.fmat.eu [37.120.191.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.fmat.eu (Postfix) with ESMTPSA id 328251A06C6 for ; Thu, 2 Nov 2017 13:42:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=matouschek.org; s=mail; t=1509626534; bh=DylcfFMlVymHxvRJ0LDOBDLg2OaTczPrW8g3A+xnAMs=; h=Date:From:To:Subject:From; b=Aen7GdkzARAv8a/ivCOEpiUjXvuPAxf6m4bg2fVXzpA7H+uHK3KJB82pXEoEYKNr/ /s3j7UmNc25LU8EzHowcoxcIGYpCdZO5qHdOxcL2Dsx3YWCPLEQNvxKWfPP+2QDKGC XcnSDyosRnyW9IE9G4gLEaB+u6boL8xfZtet4PEDUbAUYpq1Lu7C0vugb5dm7Efii+ 7WAgyv6fAoFISEAjCEVkuPugoGFXkbyl2k6Y21QCd/r6XTeNAEpZIAHhf4OS9tJhBS 7yI5i1OSWvRXdzTW9VxVO9MzxTo+7cPpK5NRrVu4IRcPiFA7TfqxyI5DJ+oXUQC9hi 7nNbWSn4tCkSg== MIME-Version: 1.0 Date: Thu, 02 Nov 2017 13:42:13 +0100 From: Felix Matouschek To: ffmpeg-devel@ffmpeg.org Message-ID: <2058953176dec69d815556013173bde2@matouschek.org> X-Sender: felix@matouschek.org Subject: [FFmpeg-devel] [PATCH] Add android_capture indev X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Hello, I've written an indev for Android devices to allow capturing their builtin cameras. What needs to be done to merge this? Greetings, Felix From b21fc8729ef2e1d9867dd7652f2c6173378e4910 Mon Sep 17 00:00:00 2001 From: Felix Matouschek Date: Tue, 24 Oct 2017 13:11:23 +0200 Subject: [PATCH] Add android_capture indev To: ffmpeg-devel@ffmpeg.org This commit adds an indev for Android devices on API level 24+ which uses the Android NDK Camera2 API to capture video from builtin cameras Signed-off-by: Felix Matouschek --- configure | 6 + libavdevice/Makefile | 1 + libavdevice/alldevices.c | 1 + libavdevice/android_capture.c | 782 ++++++++++++++++++++++++++++++++++++++++++ libavdevice/android_capture.h | 77 +++++ 5 files changed, 867 insertions(+) create mode 100644 libavdevice/android_capture.c create mode 100644 libavdevice/android_capture.h diff --git a/configure b/configure index 7a53bc76c7..e2165f2ff9 100755 --- a/configure +++ b/configure @@ -3068,6 +3068,8 @@ xmv_demuxer_select="riffdec" xwma_demuxer_select="riffdec" # indevs / outdevs +android_capture_indev_deps="android mediandk camera2ndk pthreads" +android_capture_indev_extralibs="-landroid -lmediandk -lcamera2ndk" alsa_indev_deps="alsa" alsa_outdev_deps="alsa" avfoundation_indev_deps="avfoundation corevideo coremedia pthreads" @@ -5836,6 +5838,10 @@ check_lib shell32 "windows.h shellapi.h" CommandLineToArgvW -lshell32 check_lib wincrypt "windows.h wincrypt.h" CryptGenRandom -ladvapi32 check_lib psapi "windows.h psapi.h" GetProcessMemoryInfo -lpsapi +check_lib android android/native_window.h ANativeWindow_acquire -landroid +check_lib mediandk "stdint.h media/NdkImage.h" AImage_delete -lmediandk +check_lib camera2ndk "stdbool.h stdint.h camera/NdkCameraManager.h" ACameraManager_create -lcamera2ndk + enabled appkit && check_apple_framework AppKit enabled audiotoolbox && check_apple_framework AudioToolbox enabled avfoundation && check_apple_framework AVFoundation diff --git a/libavdevice/Makefile b/libavdevice/Makefile index 8228d62147..aa01dd7e24 100644 --- a/libavdevice/Makefile +++ b/libavdevice/Makefile @@ -14,6 +14,7 @@ OBJS-$(CONFIG_SHARED) += reverse.o # input/output devices OBJS-$(CONFIG_ALSA_INDEV) += alsa_dec.o alsa.o timefilter.o OBJS-$(CONFIG_ALSA_OUTDEV) += alsa_enc.o alsa.o +OBJS-$(CONFIG_ANDROID_CAPTURE_INDEV) += android_capture.o OBJS-$(CONFIG_AVFOUNDATION_INDEV) += avfoundation.o OBJS-$(CONFIG_BKTR_INDEV) += bktr.o OBJS-$(CONFIG_CACA_OUTDEV) += caca.o diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c index b767b6a718..6cd57aa88a 100644 --- a/libavdevice/alldevices.c +++ b/libavdevice/alldevices.c @@ -42,6 +42,7 @@ static void register_all(void) { /* devices */ REGISTER_INOUTDEV(ALSA, alsa); + REGISTER_INDEV (ANDROID_CAPTURE, android_capture); REGISTER_INDEV (AVFOUNDATION, avfoundation); REGISTER_INDEV (BKTR, bktr); REGISTER_OUTDEV (CACA, caca); diff --git a/libavdevice/android_capture.c b/libavdevice/android_capture.c new file mode 100644 index 0000000000..be0dee8f81 --- /dev/null +++ b/libavdevice/android_capture.c @@ -0,0 +1,782 @@ +/* + * Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented) + * + * Copyright (C) 2017 Felix Matouschek + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include +#include +#include + +#include +#include +#include + +#include "libavformat/internal.h" +#include "libavutil/avstring.h" +#include "libavutil/display.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" + +#include "android_capture.h" +#include "version.h" + +/* This image format is available on all Android devices + * supporting the Camera2 API */ +#define IMAGE_FORMAT_ANDROID AIMAGE_FORMAT_YUV_420_888 +#define IMAGE_FORMAT_FFMPEG AV_PIX_FMT_YUV420P +#define IMAGE_NUM_PLANES 3 + +#define MAX_BUF_COUNT 2 +#define VIDEO_STREAM_ID 0 +#define VIDEO_TIMEBASE 1000000000 +#define AUDIO_STREAM_ID 1 + +static int parse_token_to_int(char *token, int *out) +{ + long l; + char *endptr; + errno = 0; + + l = strtol(token, &endptr, 0); + + if (errno == ERANGE || *endptr != '\0' || token == endptr) { + return AVERROR(ERANGE); + } + +#if LONG_MIN < INT_MIN || LONG_MAX > INT_MAX + if (l < INT_MIN || l > INT_MAX) { + return AVERROR(ERANGE); + } +#endif + + *out = (int) l; + + return 0; +} + +static int parse_device_number(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + int *video_device_number = &ctx->video_device_number; + int *audio_device_number = &ctx->audio_device_number; + char *name = av_strdup(avctx->filename); + char *tmp = name; + char *type; + char *save; + int ret = 0; + + *video_device_number = -1; + *audio_device_number = -1; + + while ((type = av_strtok(tmp, "=", &save))) { + char *token = av_strtok(NULL, ":", &save); + tmp = NULL; + + if (!strcmp(type, "video")) { + ret = parse_token_to_int(token, video_device_number); + if (ret < 0) { + break; + } + + } else if (!strcmp(type, "audio")) { + ret = parse_token_to_int(token, audio_device_number); + if (ret < 0) { + break; + } + } else { + ret = AVERROR(ENXIO); + break; + } + } + + if (ret >= 0 && *video_device_number < 0 && *audio_device_number < 0) { + ret = AVERROR(ENXIO); + } + + av_free(name); + return ret; +} + +static void camera_dev_disconnected(void *context, ACameraDevice *device) +{ + AVFormatContext *avctx = (AVFormatContext *) context; + android_capture_ctx *ctx = avctx->priv_data; + ctx->read_closing = 1; + av_log(avctx, AV_LOG_ERROR, "Camera with id %s disconnected.\n", + ACameraDevice_getId(device)); +} + +static void camera_dev_error(void *context, ACameraDevice *device, int error) +{ + AVFormatContext *avctx = (AVFormatContext *) context; + android_capture_ctx *ctx = avctx->priv_data; + ctx->read_closing = 1; + av_log(avctx, AV_LOG_ERROR, "Error %d on camera with id %s.\n", error, + ACameraDevice_getId(device)); +} + +static int open_camera(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + camera_status_t ret; + ACameraIdList *camera_ids; + + ret = ACameraManager_getCameraIdList(ctx->camera_mgr, &camera_ids); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to get camera id list, camera_status_t: %d.\n", ret); + return AVERROR(EIO); + } + + if (ctx->video_device_number < camera_ids->numCameras) { + ctx->camera_id = av_strdup( + camera_ids->cameraIds[ctx->video_device_number]); + } else { + av_log(avctx, AV_LOG_ERROR, "No camera with number %d available.\n", + ctx->video_device_number); + return AVERROR(ENXIO); + } + + ACameraManager_deleteCameraIdList(camera_ids); + + ret = ACameraManager_getCameraCharacteristics(ctx->camera_mgr, + ctx->camera_id, &ctx->camera_metadata); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to get metadata for camera with id %s, camera_status_t: %d.\n", + ctx->camera_id, ret); + return AVERROR(EIO); + } + + ctx->camera_state_callbacks.context = avctx; + ctx->camera_state_callbacks.onDisconnected = camera_dev_disconnected; + ctx->camera_state_callbacks.onError = camera_dev_error; + + ret = ACameraManager_openCamera(ctx->camera_mgr, ctx->camera_id, + &ctx->camera_state_callbacks, &ctx->camera_dev); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to open camera with id %s, camera_status_t: %d.\n", + ctx->camera_id, ret); + return AVERROR(EIO); + } + + return 0; +} + +static void get_sensor_orientation(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + ACameraMetadata_const_entry lens_facing; + ACameraMetadata_const_entry sensor_orientation; + + ACameraMetadata_getConstEntry(ctx->camera_metadata, ACAMERA_LENS_FACING, + &lens_facing); + ACameraMetadata_getConstEntry(ctx->camera_metadata, + ACAMERA_SENSOR_ORIENTATION, &sensor_orientation); + + ctx->lens_facing = lens_facing.data.u8[0]; + ctx->sensor_orientation = sensor_orientation.data.i32[0]; +} + +static int match_video_size(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + ACameraMetadata_const_entry available_configs; + int ret = -1; + + ACameraMetadata_getConstEntry(ctx->camera_metadata, + ACAMERA_SCALER_AVAILABLE_STREAM_CONFIGURATIONS, &available_configs); + + for (int i = 0; i < available_configs.count; i++) { + int32_t input = available_configs.data.i32[i * 4 + 3]; + int32_t format = available_configs.data.i32[i * 4 + 0]; + + if (input) { + continue; + } + + if (format == IMAGE_FORMAT_ANDROID) { + int32_t width = available_configs.data.i32[i * 4 + 1]; + int32_t height = available_configs.data.i32[i * 4 + 2]; + + //Same ratio + if (ctx->requested_width * ctx->requested_height + == width * height) { + ctx->width = width; + ctx->height = height; + ret = 0; + break; + } + } + } + + if (ret < 0) { + ctx->width = 640; + ctx->height = 480; + } + + return ret; +} + +static int match_framerate(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + ACameraMetadata_const_entry available_framerates; + int ret = -1; + int current_best_match = -1; + + ACameraMetadata_getConstEntry(ctx->camera_metadata, + ACAMERA_CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES, + &available_framerates); + + for (int i = 0; i < available_framerates.count; i++) { + int32_t min = available_framerates.data.i32[i * 2 + 0]; + int32_t max = available_framerates.data.i32[i * 2 + 1]; + double requested_framerate = av_q2d(ctx->requested_framerate); + + if (requested_framerate == max) { + ret = 0; + + if (min == max) { + ctx->framerate_range[0] = min; + ctx->framerate_range[1] = max; + break; + } else if (current_best_match >= 0) { + int32_t current_best_match_min = + available_framerates.data.i32[current_best_match * 2 + 0]; + if (min > current_best_match_min) { + current_best_match = i; + } + } else { + current_best_match = i; + } + } + } + + if (ret < 0) { + ctx->framerate_range[0] = available_framerates.data.i32[0]; + ctx->framerate_range[1] = available_framerates.data.i32[1]; + } + + return ret; +} + +static int shall_we_drop(AVFormatContext *avctx, int stream_index, + int stream_id) +{ + android_capture_ctx *ctx = avctx->priv_data; + static const uint8_t dropscore[] = { 62, 75, 87, 100 }; + const int ndropscores = FF_ARRAY_ELEMS(dropscore); + const char *stream_name = stream_id ? "audio" : "video"; + unsigned int buffer_fullness = (ctx->curbufsize[stream_index] * 100) + / avctx->max_picture_buffer; + + if (dropscore[++ctx->video_frame_num % ndropscores] <= buffer_fullness) { + av_log(avctx, AV_LOG_ERROR, + "real-time buffer of [%s input] too full or near too full (%d%% of size: %d [rtbufsize parameter])! frame dropped!\n", + stream_name, buffer_fullness, avctx->max_picture_buffer); + return 1; + } + + return 0; +} + +static void image_available(void *context, AImageReader *reader) +{ + AVFormatContext *avctx = (AVFormatContext *) context; + android_capture_ctx *ctx = avctx->priv_data; + media_status_t ret; + + AImage *image; + int64_t image_timestamp; + uint8_t *image_plane_data[4]; + int32_t image_linestrides[4]; + int plane_data_length[4]; + + AVPacketList *pktl_next; + AVPacketList **ppktl = &ctx->pktl; + uint8_t *side_data; + int pkt_buffer_size = av_image_get_buffer_size(IMAGE_FORMAT_FFMPEG, + ctx->width, ctx->height, 1); + int32_t display_matrix[9]; + + av_display_rotation_set(display_matrix, 360 - ctx->sensor_orientation); + + if (ctx->lens_facing == ACAMERA_LENS_FACING_FRONT) { + av_display_matrix_flip(display_matrix, 1, 0); + } + + ret = AImageReader_acquireLatestImage(reader, &image); + if (ret != AMEDIA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to acquire latest image from image reader, media_status_t: %d.\n", + ret); + goto error; + } + + // Silently drop frames when read_closing is set + if (ctx->read_closing) { + goto error; + } + + AImage_getTimestamp(image, &image_timestamp); + + for (int i = 0; i < IMAGE_NUM_PLANES; i++) { + AImage_getPlaneRowStride(image, i, &image_linestrides[i]); + AImage_getPlaneData(image, i, &image_plane_data[i], &plane_data_length[i]); + } + + pthread_mutex_lock(&ctx->mutex); + + if (shall_we_drop(avctx, ctx->video_stream_index, VIDEO_STREAM_ID)) { + goto error; + } + + pktl_next = av_mallocz(sizeof(AVPacketList)); + if (!pktl_next) + goto error; + + if (av_new_packet(&pktl_next->pkt, pkt_buffer_size) < 0) { + av_free(pktl_next); + goto error; + } + + side_data = av_packet_new_side_data(&pktl_next->pkt, + AV_PKT_DATA_DISPLAYMATRIX, sizeof(display_matrix)); + if (!side_data) { + av_packet_unref(&pktl_next->pkt); + av_free(pktl_next); + goto error; + } + + pktl_next->pkt.stream_index = ctx->video_stream_index; + pktl_next->pkt.pts = image_timestamp; + av_image_copy_to_buffer(pktl_next->pkt.data, pkt_buffer_size, + (const uint8_t * const *) image_plane_data, image_linestrides, + IMAGE_FORMAT_FFMPEG, ctx->width, ctx->height, 1); + memcpy(side_data, display_matrix, sizeof(display_matrix)); + + while (*ppktl) { + ppktl = &(*ppktl)->next; + } + *ppktl = pktl_next; + + ctx->curbufsize[ctx->video_stream_index] += pkt_buffer_size; + +error: + pthread_mutex_unlock(&ctx->mutex); + AImage_delete(image); + + return; +} + +static int create_image_reader(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + media_status_t ret; + + ret = AImageReader_new(ctx->width, ctx->height, IMAGE_FORMAT_ANDROID, + MAX_BUF_COUNT, &ctx->image_reader); + if (ret != AMEDIA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to create image reader, media_status_t: %d.\n", ret); + return AVERROR(EIO); + } + + ctx->image_listener.context = avctx; + ctx->image_listener.onImageAvailable = image_available; + + ret = AImageReader_setImageListener(ctx->image_reader, + &ctx->image_listener); + if (ret != AMEDIA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to set image listener on image reader, media_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ret = AImageReader_getWindow(ctx->image_reader, &ctx->image_reader_window); + if (ret != AMEDIA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Could not get image reader window, media_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + return 0; +} + +static void capture_session_closed(void *context, + ACameraCaptureSession *session) +{ + av_log((AVFormatContext *) context, AV_LOG_INFO, + "Android camera capture session was closed.\n"); +} + +static void capture_session_ready(void *context, ACameraCaptureSession *session) +{ + av_log((AVFormatContext *) context, AV_LOG_INFO, + "Android camera capture session is ready.\n"); +} + +static void capture_session_active(void *context, + ACameraCaptureSession *session) +{ + av_log((AVFormatContext *) context, AV_LOG_INFO, + "Android camera capture session is active.\n"); +} + +static int create_capture_session(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + camera_status_t ret; + + ret = ACaptureSessionOutputContainer_create( + &ctx->capture_session_output_container); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to create capture session output container, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ANativeWindow_acquire(ctx->image_reader_window); + + ret = ACaptureSessionOutput_create(ctx->image_reader_window, + &ctx->capture_session_output); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to create capture session container, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ret = ACaptureSessionOutputContainer_add( + ctx->capture_session_output_container, ctx->capture_session_output); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to add output to output container, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ret = ACameraOutputTarget_create(ctx->image_reader_window, + &ctx->camera_output_target); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to create camera output target, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ret = ACameraDevice_createCaptureRequest(ctx->camera_dev, TEMPLATE_RECORD, + &ctx->capture_request); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to create capture request, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ret = ACaptureRequest_setEntry_i32(ctx->capture_request, + ACAMERA_CONTROL_AE_TARGET_FPS_RANGE, 2, ctx->framerate_range); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to set target fps range in capture request, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ret = ACaptureRequest_addTarget(ctx->capture_request, + ctx->camera_output_target); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to add capture request capture request, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ctx->capture_session_state_callbacks.context = avctx; + ctx->capture_session_state_callbacks.onClosed = capture_session_closed; + ctx->capture_session_state_callbacks.onReady = capture_session_ready; + ctx->capture_session_state_callbacks.onActive = capture_session_active; + + ret = ACameraDevice_createCaptureSession(ctx->camera_dev, + ctx->capture_session_output_container, + &ctx->capture_session_state_callbacks, &ctx->capture_session); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to create capture session, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + ret = ACameraCaptureSession_setRepeatingRequest(ctx->capture_session, NULL, + 1, &ctx->capture_request, NULL); + if (ret != ACAMERA_OK) { + av_log(avctx, AV_LOG_ERROR, + "Failed to set repeating request on capture session, camera_status_t: %d.\n", + ret); + return AVERROR(EIO); + } + + return 0; +} + +static int add_video_stream(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + AVStream *st; + AVCodecParameters *codecpar; + + st = avformat_new_stream(avctx, NULL); + if (!st) { + return AVERROR(ENOMEM); + } + + st->id = VIDEO_STREAM_ID; + st->avg_frame_rate = (AVRational) { ctx->framerate_range[1], 1 }; + st->r_frame_rate = (AVRational) { ctx->framerate_range[1], 1 }; + + codecpar = st->codecpar; + codecpar->codec_type = AVMEDIA_TYPE_VIDEO; + codecpar->codec_id = AV_CODEC_ID_RAWVIDEO; + codecpar->format = IMAGE_FORMAT_FFMPEG; + codecpar->width = ctx->width; + codecpar->height = ctx->height; + + avpriv_set_pts_info(st, 64, 1, VIDEO_TIMEBASE); + + ctx->video_stream_index = st->index; + + return 0; +} + +static int android_capture_read_close(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + AVPacketList *pktl; + + ctx->read_closing = 1; + + if (ctx->capture_session) { + ACameraCaptureSession_stopRepeating(ctx->capture_session); + // Following warning is emitted, after capture session closed callback is received: + // ACameraCaptureSession: Device is closed but session 0 is not notified + // Seems to be a bug in Android, we can ignore this + ACameraCaptureSession_close(ctx->capture_session); + ctx->capture_session = NULL; + } + + if (ctx->capture_request) { + ACaptureRequest_removeTarget(ctx->capture_request, + ctx->camera_output_target); + ACaptureRequest_free(ctx->capture_request); + ctx->capture_request = NULL; + } + + if (ctx->camera_output_target) { + ACameraOutputTarget_free(ctx->camera_output_target); + ctx->camera_output_target = NULL; + } + + if (ctx->capture_session_output) { + ACaptureSessionOutputContainer_remove( + ctx->capture_session_output_container, + ctx->capture_session_output); + ACaptureSessionOutput_free(ctx->capture_session_output); + ctx->capture_session_output = NULL; + } + + ANativeWindow_release(ctx->image_reader_window); + + if (ctx->capture_session_output_container) { + ACaptureSessionOutputContainer_free( + ctx->capture_session_output_container); + ctx->capture_session_output_container = NULL; + } + + if (ctx->camera_dev) { + ACameraDevice_close(ctx->camera_dev); + ctx->camera_dev = NULL; + } + + if (ctx->image_reader) { + AImageReader_delete(ctx->image_reader); + ctx->image_reader = NULL; + ctx->image_reader_window = NULL; + } + + if (ctx->camera_metadata) { + ACameraMetadata_free(ctx->camera_metadata); + ctx->camera_metadata = NULL; + } + + if (ctx->camera_id) { + av_free(ctx->camera_id); + ctx->camera_id = NULL; + } + + if (ctx->camera_mgr) { + ACameraManager_delete(ctx->camera_mgr); + ctx->camera_mgr = NULL; + } + + pktl = ctx->pktl; + while (pktl) { + AVPacketList *next = pktl->next; + av_packet_unref(&pktl->pkt); + av_free(pktl); + pktl = next; + } + + pthread_mutex_destroy(&ctx->mutex); + + return 0; +} + +static int android_capture_read_header(AVFormatContext *avctx) +{ + android_capture_ctx *ctx = avctx->priv_data; + int ret; + + pthread_mutex_init(&ctx->mutex, 0); + + ret = parse_device_number(avctx); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, + "Malformed android_capture input string.\n"); + goto error; + } + + if (ctx->framerate) { + ret = av_parse_video_rate(&ctx->requested_framerate, ctx->framerate); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, "Could not parse framerate '%s'.\n", + ctx->framerate); + goto error; + } + } + + ctx->camera_mgr = ACameraManager_create(); + if (!ctx->camera_mgr) { + av_log(avctx, AV_LOG_ERROR, + "Failed to create Android camera manager.\n"); + ret = AVERROR(EIO); + goto error; + } + + ret = open_camera(avctx); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to open Android camera.\n"); + goto error; + } + + get_sensor_orientation(avctx); + + if (match_video_size(avctx) < 0) { + av_log(avctx, AV_LOG_WARNING, + "Requested video_size not available, falling back to 640x480\n"); + } + + if (match_framerate(avctx) < 0) { + av_log(avctx, AV_LOG_WARNING, + "Requested framerate not available, falling back to min: %d and max: %d fps\n", + ctx->framerate_range[0], ctx->framerate_range[1]); + } + + ret = create_image_reader(avctx); + if (ret < 0) { + goto error; + } + + ret = create_capture_session(avctx); + if (ret < 0) { + goto error; + } + + ret = add_video_stream(avctx); + +error: + if (ret < 0) { + android_capture_read_close(avctx); + av_log(avctx, AV_LOG_ERROR, "Failed to open android_capture.\n"); + } + + return ret; +} + +static int android_capture_read_packet(AVFormatContext *avctx, AVPacket *pkt) +{ + android_capture_ctx *ctx = avctx->priv_data; + AVPacketList *pktl = NULL; + + while (!ctx->read_closing && !pktl) { + pthread_mutex_lock(&ctx->mutex); + pktl = ctx->pktl; + if (pktl) { + *pkt = pktl->pkt; + ctx->pktl = ctx->pktl->next; + av_free(pktl); + ctx->curbufsize[pkt->stream_index] -= pkt->size; + } + pthread_mutex_unlock(&ctx->mutex); + if (!pktl) { + if (avctx->flags & AVFMT_FLAG_NONBLOCK) { + return AVERROR(EAGAIN); + } + } + } + + return ctx->read_closing ? AVERROR(EOF) : pkt->size; +} + +#define OFFSET(x) offsetof(android_capture_ctx, x) +#define DEC AV_OPT_FLAG_DECODING_PARAM +static const AVOption options[] = { + { "video_size", "set video size given a string such as 640x480 or hd720.", OFFSET(requested_width), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, DEC }, + { "framerate", "set video frame rate", OFFSET(framerate), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, DEC }, + { "sample_rate", "set audio sample rate", OFFSET(sample_rate), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC }, + { "sample_size", "set audio sample size", OFFSET(sample_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 16, DEC }, + { "channels", "set number of audio channels, such as 1 or 2", OFFSET(channels), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC }, + { "audio_buffer_size", "set audio device buffer latency size in milliseconds (default is the device's default)", OFFSET(audio_buffer_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC }, + { NULL }, +}; + +static const AVClass android_capture_class = { + .class_name = "android_capture indev", + .item_name = av_default_item_name, + .option = options, + .version = LIBAVDEVICE_VERSION_INT, + .category = AV_CLASS_CATEGORY_DEVICE_VIDEO_INPUT, +}; + +AVInputFormat ff_android_capture_demuxer = { + .name = "android_capture", + .long_name = NULL_IF_CONFIG_SMALL("Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)"), + .priv_data_size = sizeof(android_capture_ctx), + .read_header = android_capture_read_header, + .read_packet = android_capture_read_packet, + .read_close = android_capture_read_close, + .flags = AVFMT_NOFILE, + .priv_class = &android_capture_class, +}; diff --git a/libavdevice/android_capture.h b/libavdevice/android_capture.h new file mode 100644 index 0000000000..44c888ea67 --- /dev/null +++ b/libavdevice/android_capture.h @@ -0,0 +1,77 @@ +/* + * Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented) + * + * Copyright (C) 2017 Felix Matouschek + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVDEVICE_ANDROID_CAPTURE_H +#define AVDEVICE_ANDROID_CAPTURE_H + +#include + +#include +#include + +#include "libavutil/log.h" +#include "libavformat/avformat.h" + +typedef struct android_capture_ctx { + const AVClass *class; + + int requested_width; + int requested_height; + char *framerate; + int sample_rate; + int sample_size; + int channels; + int audio_buffer_size; + + int video_device_number; + int audio_device_number; + uint8_t lens_facing; + int32_t sensor_orientation; + int width; + int height; + AVRational requested_framerate; + int32_t framerate_range[2]; + int video_stream_index; + + ACameraManager *camera_mgr; + char *camera_id; + ACameraMetadata *camera_metadata; + ACameraDevice *camera_dev; + ACameraDevice_stateCallbacks camera_state_callbacks; + AImageReader *image_reader; + AImageReader_ImageListener image_listener; + ANativeWindow *image_reader_window; + ACaptureSessionOutputContainer *capture_session_output_container; + ACaptureSessionOutput *capture_session_output; + ACameraOutputTarget *camera_output_target; + ACaptureRequest *capture_request; + ACameraCaptureSession_stateCallbacks capture_session_state_callbacks; + ACameraCaptureSession *capture_session; + + AVPacketList *pktl; + pthread_mutex_t mutex; + int read_closing; + int64_t curbufsize[2]; + int64_t video_frame_num; +} android_capture_ctx; + +#endif /* AVDEVICE_ANDROID_CAPTURE_H */ -- 2.14.1.windows.1