From patchwork Tue Jun 11 02:46:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Michael Zucchi X-Patchwork-Id: 13492 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 6001C448ACE for ; Tue, 11 Jun 2019 06:16:17 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 34E9568A728; Tue, 11 Jun 2019 06:16:17 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 23392689AE0 for ; Tue, 11 Jun 2019 06:16:10 +0300 (EEST) Received: by mail-oi1-f175.google.com with SMTP id y6so7853247oix.2 for ; Mon, 10 Jun 2019 20:16:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-language; bh=e/o3cntLs8fOrTIMSrHKOZgXO0M6ELN2vq9muZVvDH4=; b=ZI2tiBUPhYwvvbM3w7sVEzLM5SFjK7YLpDn9J75vL+b/ZsMFo2xineZJMsOBj36wvw yUUO8mi9VHa1qaHIxolOQjbFbL7prAJKAr893R7axc+DJ7aMN/VGfwFcYXlSA1VVMDov vMc8Gzt/s4OrXT7PfybnK8RAm8hyqA3jzKo2UoC3MMCK8l+d5GbVt2sszT2WH5Zx8Ss5 +CLtWM54KTfSbxWTvoiFLrYT6dBYPDmnlphJxAHqWKOYN6ZoAQ+1xD0/2vWHmKEv0mYq sBSHaPmCwEiE6tdKlAAHLzc9CNBQrLH/YjSf25l36ecfOg6LRedCZzSUsdKjqJ9S25Eh j6yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-language; bh=e/o3cntLs8fOrTIMSrHKOZgXO0M6ELN2vq9muZVvDH4=; b=bi+rKYDtGjYeD6+TG55BvNTMK3wo/IVbx3QwwGmgv7TdX18YyMBBqi+VubICJDYok9 tX0ZxVSvJA8xE4EPj9abjjkUdq/CF9owDFzAOiFEuVYN76r1mHH4Nwx0NCrjKhQkWeHW 9Mll08TQJH/VNB+KEHNuuF679SvdY25sJChU5GaYMA+BLDE3d3gqTKINtaB512J5Gb20 dI0g2Bjy37DWMkIDBkCGcyxm1xAe5rlWhrxcHe2VU2FSWIDVjxTxSsQP4noYA/4/khvg Rf6XLP08LoVSfohTXdu+ZtS6JV6YLxxzmdnA0pDndSlLNYWWkiQOwJ8N2uU2MS8mQmP/ 6YCw== X-Gm-Message-State: APjAAAVqTonOqw+QJVa3PbgMmF62QkXMus2rtE602Q2XdUdEYbBYctLO NxDOL6wwDHHuPMvYwMLuvVnQe9Q0 X-Google-Smtp-Source: APXvYqyalSfjkR3qCDTW4Rjrk5F7T47TlEzBvKD5S4lG4a+NO08pnJpc6WHtJVyz8OPd/9YJDzxVMg== X-Received: by 2002:a63:d4c:: with SMTP id 12mr18847390pgn.30.1560221171891; Mon, 10 Jun 2019 19:46:11 -0700 (PDT) Received: from ?IPv6:2001:44b8:203:1200:be5f:f4ff:fef7:7bdc? (2001-44b8-0203-1200-be5f-f4ff-fef7-7bdc.static.ipv6.internode.on.net. [2001:44b8:203:1200:be5f:f4ff:fef7:7bdc]) by smtp.googlemail.com with ESMTPSA id m2sm12700472pgq.48.2019.06.10.19.46.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 19:46:11 -0700 (PDT) To: FFmpeg development discussions and patches From: Michael Zucchi Message-ID: <2038d999-9bfa-debd-55aa-6ed05875976b@gmail.com> Date: Tue, 11 Jun 2019 12:16:07 +0930 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Language: en-AU X-Content-Filtered-By: Mailman/MimeDel 2.1.20 Subject: [FFmpeg-devel] [PATCH] kinect for windows v2 indev X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Morning, Here's a patch to add a "Kinect for Windows v2" camera indev via libfreenect2[1].  It exports up to 3x streams from the device: raw jpeg from the camera, the libfreenect2 decoded infra-red as 16-bit greyscale, and the libfreenect2 decoded depth as 16-bit greyscale in 0.1mm units (this matches the matlab data i've seen). I have a few comments not addressed by the patch itself. libfreenect2 is unfortunately c++ so this patch is also partly c++, if this isn't acceptable for ffmpeg inclusion let me know and i'll just publish the patch somewhere else. libfreenect2 is apache 2.0 or gnu gpl 2.0 licensed, I'm not sure how that affects the licensing options/result. It requires a small patch to libfreenect2[2] which i've submitted to that project but am still waiting for it to be applied. I copied the way the decklink device integrates c++ into the build but some issues are outstanding although I imagine most apply equally to decklink.  The pkg-config check just checks against stdio.h as i couldn't work out how to do anything relevent with the libfreenect headers which only contain class declarations.  The makefile doesn't generate dependencies for .cpp files so incremental builds aren't absolutely consistent.  And finally the build spits out a bunch of ?harmless? warnings from the c-specific arguments passed to g++. The logging callback for libfreenect2 is global so the way i'm setting it might not be ideal.  Perhaps it should be a singleton and use a null or pseudo-class av log context? I'm not a c++ coder so some of the c++ might be a bit archaic and possibly incorrect.  I've used the bare minimum c++ to get it working and it's mostly just c++'s c-ish dialect. I've tested it on ubuntu 16.04 and successfully cross-compiled it to a microsoft windows target but haven't tested it there yet. I've done some limited valgrind testing but the noise from libfreenect2's dependencies is overwhelmingly voluminous. Regards,  Michael [1] [2] From 3a20c35229095f152980ba66c7ec2aa7712f1c4c Mon Sep 17 00:00:00 2001 From: Michael Zucchi Date: Sat, 8 Jun 2019 21:46:13 +0930 Subject: [PATCH] avdevice: Added Kinect for Windows V2 device via libfreenect2. --- Changelog | 1 + configure | 4 + doc/general.texi | 11 ++ doc/indevs.texi | 93 ++++++++++ libavdevice/Makefile | 1 + libavdevice/alldevices.c | 1 + libavdevice/freenect2_dec.cpp | 414 ++++++++++++++++++++++++++++++++++++++++++ libavdevice/freenect2_dec.h | 80 ++++++++ libavdevice/freenect2_dec_c.c | 75 ++++++++ libavdevice/version.h | 2 +- 10 files changed, 681 insertions(+), 1 deletion(-) create mode 100644 libavdevice/freenect2_dec.cpp create mode 100644 libavdevice/freenect2_dec.h create mode 100644 libavdevice/freenect2_dec_c.c diff --git a/Changelog b/Changelog index 1facc1f..632dd82 100644 --- a/Changelog +++ b/Changelog @@ -31,6 +31,7 @@ version : - xmedian filter - asr filter - showspatial multimedia filter +- libfreenect2 based Kinect for Windows V2 device version 4.1: diff --git a/configure b/configure index 32fc263..49cfc56 100755 --- a/configure +++ b/configure @@ -1744,6 +1744,7 @@ EXTERNAL_LIBRARY_LIST=" $EXTERNAL_LIBRARY_VERSION3_LIST $EXTERNAL_LIBRARY_GPLV3_LIST chromaprint + freenect2 gcrypt gnutls jni @@ -3309,6 +3310,8 @@ dshow_indev_deps="IBaseFilter" dshow_indev_extralibs="-lpsapi -lole32 -lstrmiids -luuid -loleaut32 -lshlwapi" fbdev_indev_deps="linux_fb_h" fbdev_outdev_deps="linux_fb_h" +freenect2_indev_deps="freenect2" +freenect2_indev_extralibs="-lstdc++" gdigrab_indev_deps="CreateDIBSection" gdigrab_indev_extralibs="-lgdi32" gdigrab_indev_select="bmp_decoder" @@ -6125,6 +6128,7 @@ enabled chromaprint && require chromaprint chromaprint.h chromaprint_get_v enabled decklink && { require_headers DeckLinkAPI.h && { test_cpp_condition DeckLinkAPIVersion.h "BLACKMAGIC_DECKLINK_API_VERSION >= 0x0a090500" || die "ERROR: Decklink API version must be >= 10.9.5."; } } enabled frei0r && require_headers "frei0r.h dlfcn.h" +enabled freenect2 && require_pkg_config freenect2 "freenect2 >= 0.3.0" stdio.h fopen enabled gmp && require gmp gmp.h mpz_export -lgmp enabled gnutls && require_pkg_config gnutls gnutls gnutls/gnutls.h gnutls_global_init enabled jni && { [ $target_os = "android" ] && check_headers jni.h && enabled pthreads || die "ERROR: jni not found"; } diff --git a/doc/general.texi b/doc/general.texi index ec43723..b87b401 100644 --- a/doc/general.texi +++ b/doc/general.texi @@ -151,6 +151,16 @@ Go to @url{http://lame.sourceforge.net/} and follow the instructions for installing the library. Then pass @code{--enable-libmp3lame} to configure to enable it. +@section libfreenect2 + +libfreenect2 is a reverse engineered library that can read video and +depth data from Kinect for Windows V2 cameras. FFmpeg can make use of +the libfreenect2 library to read these video streams. + +Go to @url{https://github.com/OpenKinect/libfreenect2} and follow the +instructions for installing the library. Then pass +@code{--enable-freenect2} to configure to enable the freenect2 device. + @section libilbc iLBC is a narrowband speech codec that has been made freely available @@ -1340,6 +1350,7 @@ performance on systems without hardware floating point support). @item BKTR @tab X @tab @item caca @tab @tab X @item DV1394 @tab X @tab +@item Kinect for Windows V2 @tab X @tab @item Lavfi virtual device @tab X @tab @item Linux framebuffer @tab X @tab X @item JACK @tab X @tab diff --git a/doc/indevs.texi b/doc/indevs.texi index 89ba4fb..ebf05c6 100644 --- a/doc/indevs.texi +++ b/doc/indevs.texi @@ -657,6 +657,99 @@ Set the frame rate. Default is 25. @end table +@section freenect2 + +Kinect for Windows V2 camera using libfreenect2. This is a +cross-platform driver and not restricted to Microsoft Windows. + +This driver supports colour, infra-red, and depth frames encoded as +separate streams. + +The colour signal is the data directly from the camera which is +individually encoded jpeg frames. The resolution is 1920x1080 at 30 +frames per second. + +Both infra-red and depth are received by the host in an encoded format +which is decoded by libfreenect2. These are presented as raw 16 +bit-per-pixel greyscale images within their respective streams. The +depth values are in 0.1mm units. The resolution of each of these +streams is 512x424 at 30 frames per second. + +See also @url{https://github.com/OpenKinect/libfreenect2}. + +@subsection Options + +The camera to open is supplied by the input filename. It is either a +single digit which is the index of the camera (as determined by +libfreenect2) or the serial number. By default the first camera is +opened. + +The @code{-sources} option of ffmpeg can be used to list the serial +numbers of connected cameras. + +@table @option + +@item capture +Selects which streams are captured. This is a bit-field where +@option{1}=rgb, @option{2}=infra-red, and @option{4}=depth. Symbolic +constants @option{rgb}, @option{ir}, and @option{depth} can be used +when only one channel is required. Decoding the infra-red and depth +streams requires significant additional processing which requires GPU +acceleration to achieve real-time performance. Default is 7 (all +streams). + +@item pipeline +Runtime selection for the the processing pipeline used to decode the +infra-red and depth data. The available pipelines depend on the +libfreenect2 build options. Possible values are @option{auto} for any +available pipeline, a gpu-accelerated pipeline will be used if +possible. @option{cpu} for only cpu, @option{opengl} to use OpenGL, +@option{opencl} to use OpenCL, @option{cuda} for CUDA. Default is +@option{auto}. + +@item device +Device identifier used for OpenCL or CUDA pipeline. This is passed +directly to libfreenect2. Default is 0. + +@item auto_exposure +Sets the camera colour sensor auto-exposure value. Default is 0.0. + +@item pseudo_exposure +Sets the camera colour sensor exposure time moderated by power-line +frequency. This is off by default. + +@item manual_exposure, manual_gain +Set the camera colour sensor to manual exposure and gain. This is off +by default. + +@end table + +@subsection Examples + +@itemize +@item +Capture only the colour stream and write to a motion jpeg file. +@example +ffmpeg -f freenect2 -capture rgb -i 0 -c:v copy output.mov +@end example + +@item +Capture all streams with minimal overhead. Infra-red and depth are +stored as raw 16-bit frames which has limited container support. +@example +ffmpeg -f freenect2 -i 0 -map 0:0 -c:v:0 copy -map 0:1 -pix_fmt:1 gray16be -c:v:1 rawvideo -map 0:2 -pix_fmt:2 gray16be -c:v:2 rawvideo output.mov +@end example + +@item +Capture all streams with compression. Only limited codecs support the +16-bit greyscale required, these include jpegls, jpeg2000, ffv1, and +png. +@example +ffmpeg -y -f freenect2 -i 0 -map 0:0 -c:v:0 copy -map 0:1 -c:v:1 jpegls -map 0:2 -c:v:2 jpegls output.mov +@end example + +@end itemize + @section gdigrab Win32 GDI-based screen capture device. diff --git a/libavdevice/Makefile b/libavdevice/Makefile index 6ea62b9..1b11a62 100644 --- a/libavdevice/Makefile +++ b/libavdevice/Makefile @@ -30,6 +30,7 @@ OBJS-$(CONFIG_FBDEV_OUTDEV) += fbdev_enc.o \ OBJS-$(CONFIG_GDIGRAB_INDEV) += gdigrab.o OBJS-$(CONFIG_IEC61883_INDEV) += iec61883.o OBJS-$(CONFIG_JACK_INDEV) += jack.o timefilter.o +OBJS-$(CONFIG_FREENECT2_INDEV) += freenect2_dec.o freenect2_dec_c.o OBJS-$(CONFIG_KMSGRAB_INDEV) += kmsgrab.o OBJS-$(CONFIG_LAVFI_INDEV) += lavfi.o OBJS-$(CONFIG_OPENAL_INDEV) += openal-dec.o diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c index 8633433..7393bf3 100644 --- a/libavdevice/alldevices.c +++ b/libavdevice/alldevices.c @@ -35,6 +35,7 @@ extern AVOutputFormat ff_decklink_muxer; extern AVInputFormat ff_dshow_demuxer; extern AVInputFormat ff_fbdev_demuxer; extern AVOutputFormat ff_fbdev_muxer; +extern AVInputFormat ff_freenect2_demuxer; extern AVInputFormat ff_gdigrab_demuxer; extern AVInputFormat ff_iec61883_demuxer; extern AVInputFormat ff_jack_demuxer; diff --git a/libavdevice/freenect2_dec.cpp b/libavdevice/freenect2_dec.cpp new file mode 100644 index 0000000..e3d5959 --- /dev/null +++ b/libavdevice/freenect2_dec.cpp @@ -0,0 +1,414 @@ +/* + * Copyright (c) 2019 Michael Zucchi + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Freenect2 input device. + * The c++ integration is based on decklink_dec + */ + +#include +#include +#include + +extern "C" { +#include "libavutil/log.h" +#include "libavutil/mem.h" +#include "libavutil/time.h" +#include "libavutil/pixdesc.h" +#include "libavutil/bswap.h" +#include "avdevice.h" +} +#include "freenect2_dec.h" + +typedef struct freenect2_ctx freenect2_ctx; + +/** + * Custom logger. + * libfreenect2's info is very noisy to it is mapped to debug level. + */ +class FN2Logger: public libfreenect2::Logger { +private: + void *ctx; + + int mapLevel(Level level) { + switch (level) { + case None: + return AV_LOG_QUIET; + case Error: // 1->16 + return AV_LOG_ERROR; + case Warning: // 2->24 + return AV_LOG_WARNING; + case Info: // 3->32 + case Debug: + default: + return AV_LOG_DEBUG; + } + } +public: + FN2Logger(void *ctx) { + this->ctx = ctx; + } + + virtual void log(Level level, const std::string &message) { + av_log(ctx, mapLevel(level), "%s\n", message.c_str()); + } +}; + +static int create_pipeline(AVFormatContext *avctx) { + freenect2_ctx *ktx = (freenect2_ctx *)avctx->priv_data; + + switch (ktx->pipelineid) { + case FN2_ANY: +#if defined(LIBFREENECT2_WITH_OPENGL_SUPPORT) + ktx->pipeline = new libfreenect2::OpenGLPacketPipeline(0, false, true); +#endif +#if defined(LIBFREENECT2_WITH_OPENCL_SUPPORT) + if (!ktx->pipeline) + ktx->pipeline = new libfreenect2::OpenCLPacketPipeline(ktx->deviceid, true); +#endif +#if defined(LIBFREENECT2_WITH_CUDA_SUPPORT) + if (!ktx->pipeline) + ktx->pipeline = new libfreenect2::CudaPacketPipeline(ktx->deviceid, true); +#endif + if (!ktx->pipeline) + ktx->pipeline = new libfreenect2::CpuPacketPipeline(true); + break; + + case FN2_CPU: + ktx->pipeline = new libfreenect2::CpuPacketPipeline(true); + break; + + case FN2_OPENGL: +#if defined(LIBFREENECT2_WITH_OPENGL_SUPPORT) + ktx->pipeline = new libfreenect2::OpenGLPacketPipeline(0, false, true); +#else + av_log(avctx, AV_LOG_ERROR, "OpenGL Pipeline not supported"); + return AVERROR(EINVAL); +#endif + + case FN2_OPENCL: +#if defined(LIBFREENECT2_WITH_OPENCL_SUPPORT) + ktx->pipeline = new libfreenect2::OpenCLPacketPipeline(ktx->deviceid, true); +#else + av_log(avctx, AV_LOG_ERROR, "OpenCL Pipeline not supported"); + return AVERROR(EINVAL); +#endif + + case FN2_CUDA: +#if defined(LIBFREENECT2_WITH_CUDA_SUPPORT) + ktx->pipeline = new libfreenect2::CudaPacketPipeline(ktx->deviceid, true); +#else + av_log(avctx, AV_LOG_ERROR, "CUDA Pipeline not supported"); + return AVERROR(EINVAL); +#endif + } + if (!ktx->pipeline) + return AVERROR(ENOMEM); + + return 0; +} + +int av_cold ff_freenect2_read_header(AVFormatContext *avctx) +{ + freenect2_ctx *ktx = (freenect2_ctx *)avctx->priv_data; + AVStream *st = NULL; + int res; + + libfreenect2::setGlobalLogger(new FN2Logger(avctx)); + + av_log(avctx, AV_LOG_DEBUG, "open device '%s' capture=%d pipeline=%d\n", avctx->url, ktx->capture, ktx->pipelineid); + + ktx->fn = new libfreenect2::Freenect2(); + if (!ktx->fn) + return AVERROR(ENOMEM); + + res = create_pipeline(avctx); + if (res != 0) + goto fail; + + // Open default, by index, or by serial no. + if (!avctx->url[0]) + ktx->dev = ktx->fn->openDevice(ktx->fn->getDefaultDeviceSerialNumber(), ktx->pipeline); + else if (strlen(avctx->url) < 4) + ktx->dev = ktx->fn->openDevice(atoi(avctx->url), ktx->pipeline); + else { + std::string ser(avctx->url); + ktx->dev = ktx->fn->openDevice(ser, ktx->pipeline); + } + if (!ktx->dev) { + res = AVERROR(ENOENT); + goto fail; + } + + // Ideally(?) could use an asynchronous listener, this makes it simple + ktx->listener = new libfreenect2::SyncMultiFrameListener(ktx->capture); + if (!ktx->listener) { + res = AVERROR(ENOMEM); + goto fail; + } + ktx->dev->setColorFrameListener(ktx->listener); + ktx->dev->setIrAndDepthFrameListener(ktx->listener); + + ktx->frames = new libfreenect2::FrameMap(); + if (!ktx->frames) { + res = AVERROR(ENOMEM); + goto fail; + } + + ktx->state = FN2_READY; + + // RGB video + if (ktx->capture & libfreenect2::Frame::Color) { + ktx->rgb_index = avctx->nb_streams; + st = avformat_new_stream(avctx, NULL); + if (!st) { + res = AVERROR(ENOMEM); + goto fail; + } + st->id = 0; + st->codecpar->format = AV_PIX_FMT_NONE; + st->codecpar->width = 1920; + st->codecpar->height = 1080; + st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; + st->codecpar->codec_id = AV_CODEC_ID_MJPEG; + st->codecpar->bit_rate = 8 * 1920 * 1080; + st->avg_frame_rate = (AVRational){30, 1}; + st->r_frame_rate = st->avg_frame_rate; + st->time_base = (AVRational){125, 1000000}; + } + + // IR video + if (ktx->capture & libfreenect2::Frame::Ir) { + ktx->ir_index = avctx->nb_streams; + st = avformat_new_stream(avctx, NULL); + if (!st) { + res = AVERROR(ENOMEM); + goto fail; + } + st->id = 1; + st->codecpar->format = AV_PIX_FMT_GRAY16LE; + st->codecpar->width = 512; + st->codecpar->height = 424; + st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; + st->codecpar->codec_id = AV_CODEC_ID_RAWVIDEO; + st->codecpar->codec_tag = avcodec_pix_fmt_to_codec_tag((AVPixelFormat)st->codecpar->format); + st->codecpar->bit_rate = 16 * 512 * 424; + st->avg_frame_rate = (AVRational){30, 1}; + st->r_frame_rate = st->avg_frame_rate; + st->time_base = (AVRational){125, 1000000}; + } + + // Depth video + if (ktx->capture & libfreenect2::Frame::Depth) { + ktx->depth_index = avctx->nb_streams; + st = avformat_new_stream(avctx, NULL); + if (!st) { + res = AVERROR(ENOMEM); + goto fail; + } + st->id = 2; + st->codecpar->format = AV_PIX_FMT_GRAY16LE; + st->codecpar->width = 512; + st->codecpar->height = 424; + st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO; + st->codecpar->codec_id = AV_CODEC_ID_RAWVIDEO; + st->codecpar->codec_tag = avcodec_pix_fmt_to_codec_tag((AVPixelFormat)st->codecpar->format); + st->codecpar->bit_rate = 16 * 512 * 424; + st->avg_frame_rate = (AVRational){30, 1}; + st->r_frame_rate = st->avg_frame_rate; + st->time_base = (AVRational){125, 1000000}; + } + + return 0; + + fail: + delete(ktx->frames); + delete(ktx->listener); + delete(ktx->dev); + delete(ktx->fn); + + return res; +} + +int ff_freenect2_read_packet(AVFormatContext *avctx, AVPacket *pkt) +{ + freenect2_ctx *ktx = (freenect2_ctx *)avctx->priv_data; + int res; + libfreenect2::Frame *frame; + float *fdata; + unsigned short *sdata; + + if (ktx->state < FN2_START) { + if (!ktx->dev->startStreams + ((ktx->capture & libfreenect2::Frame::Color) != 0, + (ktx->capture & (libfreenect2::Frame::Ir | libfreenect2::Frame::Depth)) != 0)) + return AVERROR(EIO); + ktx->state = FN2_START; + + // Settings must be applied after start + if (ktx->capture & libfreenect2::Frame::Color) { + if (ktx->auto_exposure != 0.0f) { + ktx->dev->setColorAutoExposure(ktx->auto_exposure); + } else if (ktx->pseudo_exposure != 0.0f) { + ktx->dev->setColorSemiAutoExposure(ktx->pseudo_exposure); + } else if (ktx->manual_gain != 1.0f || ktx->manual_exposure != 0.0f) { + ktx->dev->setColorManualExposure(ktx->manual_exposure, ktx->manual_gain); + } + } + } + + // Wait for all frames to be ready + if (ktx->state == FN2_START) { + if (!ktx->listener->waitForNewFrame(*ktx->frames, 3*1000)) + return 0; + + if (ktx->capture & libfreenect2::Frame::Color) + ktx->state = FN2_RGB; + else if (ktx->capture & libfreenect2::Frame::Ir) + ktx->state = FN2_IR; + else + ktx->state = FN2_DEPTH; + } + + switch (ktx->state) { + case FN2_RGB: + // RGB Packets are in Format::Raw and JPEG format. + frame = (*ktx->frames)[libfreenect2::Frame::Color]; + if ((res = av_new_packet(pkt, frame->bytes_per_pixel))) + return res; + + memcpy(pkt->data, frame->data, frame->bytes_per_pixel); + + pkt->stream_index = ktx->rgb_index; + pkt->pts = frame->timestamp; + + if (ktx->capture & libfreenect2::Frame::Ir) + ktx->state = FN2_IR; + else if (ktx->capture & libfreenect2::Frame::Depth) + ktx->state = FN2_DEPTH; + else { + ktx->listener->release(*ktx->frames); + ktx->state = FN2_START; + } + break; + case FN2_IR: + // IR Packets are Format::Float, 0.0 - 65535.0 + frame = (*ktx->frames)[libfreenect2::Frame::Ir]; + if ((res = av_new_packet(pkt, frame->width * frame->height * 2))) + return res; + + fdata = (float *)frame->data; + sdata = (unsigned short *)pkt->data; + for (unsigned int i=0;iwidth * frame->height;i++) + sdata[i] = (unsigned short)fdata[i]; + + pkt->stream_index = ktx->ir_index; + pkt->pts = frame->timestamp; + + if (ktx->capture & libfreenect2::Frame::Depth) + ktx->state = FN2_DEPTH; + else { + ktx->listener->release(*ktx->frames); + ktx->state = FN2_START; + } + break; + case FN2_DEPTH: + // Depth Packets are Format::Float, in mm. Convert to 0.1mm units. + frame = (*ktx->frames)[libfreenect2::Frame::Depth]; + if ((res = av_new_packet(pkt, frame->width * frame->height * 2))) + return res; + + fdata = (float *)frame->data; + sdata = (unsigned short *)pkt->data; + for (unsigned int i=0;iwidth * frame->height;i++) + sdata[i] = (unsigned short)(10.0f * fdata[i]); + + pkt->stream_index = ktx->depth_index; + pkt->pts = frame->timestamp; + + ktx->listener->release(*ktx->frames); + ktx->state = FN2_START; + break; + } + + return 0; +} + +int av_cold ff_freenect2_read_close(AVFormatContext *avctx) +{ + freenect2_ctx *ktx = (freenect2_ctx *)avctx->priv_data; + + ktx->dev->stop(); + ktx->dev->close(); + + if (ktx->state > FN2_START) + ktx->listener->release(*ktx->frames); + + delete(ktx->frames); + delete(ktx->listener); + delete(ktx->dev); + delete(ktx->fn); + + return 0; +} + +int ff_freenect2_get_device_list(AVFormatContext *s, AVDeviceInfoList *device_list) +{ + FN2Logger logger(s); + libfreenect2::setGlobalLogger(&logger); + + libfreenect2::Freenect2 fn; + int len = fn.enumerateDevices(); + std::string def = fn.getDefaultDeviceSerialNumber(); + int res = 0; + AVDeviceInfo *device = NULL; + + for (int i=0;idevice_name = av_strdup(ser.c_str()); + device->device_description = av_strdup("libfreenect2 Device"); + if (!device->device_name || !device->device_description) { + res = AVERROR(ENOMEM); + goto fail; + } + + if ((res = av_dynarray_add_nofree(&device_list->devices, &device_list->nb_devices, device)) < 0) + goto fail; + + if (ser == def) + device_list->default_device = device_list->nb_devices - 1; + } + device = NULL; + fail: + if (device) { + av_free(device->device_name); + av_free(device->device_description); + av_free(device); + } + return res; +} diff --git a/libavdevice/freenect2_dec.h b/libavdevice/freenect2_dec.h new file mode 100644 index 0000000..0cf91ee --- /dev/null +++ b/libavdevice/freenect2_dec.h @@ -0,0 +1,80 @@ +/* + * Copyright (c) 2019 Michael Zucchi + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVDEVICE_FREENECT2_DEC_H +#define AVDEVICE_FREENECT2_DEC_H + +#ifdef __cplusplus +extern "C" { +#define FNSTRUCT libfreenect2:: +#else +#define FNSTRUCT struct +#endif + +int ff_freenect2_read_header(AVFormatContext *avctx); +int ff_freenect2_read_packet(AVFormatContext *avctx, AVPacket *pkt); +int ff_freenect2_read_close(AVFormatContext *avctx); +int ff_freenect2_get_device_list(AVFormatContext *s, struct AVDeviceInfoList *device_list); + +enum freenect2_state { + FN2_READY = 1, + FN2_START, + FN2_RGB, + FN2_IR, + FN2_DEPTH +}; + +enum freenect2_pipeline { + FN2_ANY, + FN2_CPU, + FN2_OPENGL, + FN2_OPENCL, + FN2_CUDA +}; + +struct freenect2_ctx { + const AVClass *cclass; + + enum freenect2_state state; + int rgb_index, ir_index, depth_index; + + // libfrreenect2::Type bitset of frames to capture, 1, 2, 4 + int capture; + // Basic camera settings. + float auto_exposure; + float pseudo_exposure; + float manual_exposure, manual_gain; + // processing pipeline to use + enum freenect2_pipeline pipelineid; + // gpu id + int deviceid; + + FNSTRUCT Freenect2 *fn; + FNSTRUCT PacketPipeline *pipeline; + FNSTRUCT Freenect2Device *dev; + FNSTRUCT SyncMultiFrameListener *listener; + FNSTRUCT FrameMap *frames; +}; + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif diff --git a/libavdevice/freenect2_dec_c.c b/libavdevice/freenect2_dec_c.c new file mode 100644 index 0000000..d822bd9 --- /dev/null +++ b/libavdevice/freenect2_dec_c.c @@ -0,0 +1,75 @@ +/* + * Copyright (c) 2019 Michael Zucchi + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * kinect2 driver setup and config + */ + +#include "libavformat/avformat.h" +#include "libavutil/opt.h" +#include "freenect2_dec.h" + +#define OFF(x) offsetof(struct freenect2_ctx, x) +#define DEC AV_OPT_FLAG_DECODING_PARAM + +static const AVOption options[] = { + { "capture", "Streams to capture", OFF(capture), AV_OPT_TYPE_INT, {.i64 = 7 }, 1, 7, DEC, "capture" }, + { "pipeline", "Processing Pipeline", OFF(pipelineid), AV_OPT_TYPE_INT, {.i64 = FN2_ANY }, FN2_ANY, FN2_CUDA, DEC, "pipeline" }, + + { "device", "GPU Device ID", OFF(deviceid), AV_OPT_TYPE_INT, {.i64 = 0 }, 0.0, 128.0, DEC }, + + { "auto_exposure", "", OFF(auto_exposure), AV_OPT_TYPE_FLOAT, {.dbl = 0.0 }, -2.0, 2.0, DEC }, + { "pseudo_exposure", "", OFF(pseudo_exposure), AV_OPT_TYPE_FLOAT, {.dbl = 0.0 }, 0.0, 640.0, DEC }, + { "manual_exposure", "", OFF(manual_exposure), AV_OPT_TYPE_FLOAT, {.dbl = 0.0 }, 0.0, 66.0, DEC }, + { "manual_gain", "", OFF(manual_gain), AV_OPT_TYPE_FLOAT, {.dbl = 1.0 }, 1.0, 4.0, DEC }, + + { "rgb", "", 0, AV_OPT_TYPE_CONST, {.i64 = 1}, 0, 0, DEC, "capture" }, + { "ir", "", 0, AV_OPT_TYPE_CONST, {.i64 = 2}, 0, 0, DEC, "capture" }, + { "depth", "", 0, AV_OPT_TYPE_CONST, {.i64 = 4}, 0, 0, DEC, "capture" }, + + { "auto", "Automatic", 0, AV_OPT_TYPE_CONST, {.i64 = FN2_ANY}, 0, 0, DEC, "pipeline" }, + { "cpu", "CPU", 0, AV_OPT_TYPE_CONST, {.i64 = FN2_CPU}, 0, 0, DEC, "pipeline" }, + { "opengl", "OpenGL", 0, AV_OPT_TYPE_CONST, {.i64 = FN2_OPENGL}, 0, 0, DEC, "pipeline" }, + { "opencl", "OpenCL", 0, AV_OPT_TYPE_CONST, {.i64 = FN2_OPENCL}, 0, 0, DEC, "pipeline" }, + { "cuda", "CUDA", 0, AV_OPT_TYPE_CONST, {.i64 = FN2_CUDA}, 0, 0, DEC, "pipeline" }, + + { NULL }, +}; + +static const AVClass freenect2_class = { + .class_name = "freenect2 indev", + .item_name = av_default_item_name, + .option = options, + .version = LIBAVUTIL_VERSION_INT, + .category = AV_CLASS_CATEGORY_DEVICE_VIDEO_INPUT, +}; + +AVInputFormat ff_freenect2_demuxer = { + .name = "freenect2", + .long_name = NULL_IF_CONFIG_SMALL("Kinect V2 with libfreenect2"), + .priv_data_size = sizeof(struct freenect2_ctx), + .read_header = ff_freenect2_read_header, + .read_packet = ff_freenect2_read_packet, + .read_close = ff_freenect2_read_close, + .get_device_list = ff_freenect2_get_device_list, + .flags = AVFMT_NOFILE, + .priv_class = &freenect2_class, +}; diff --git a/libavdevice/version.h b/libavdevice/version.h index dc82519..840cefa 100644 --- a/libavdevice/version.h +++ b/libavdevice/version.h @@ -28,7 +28,7 @@ #include "libavutil/version.h" #define LIBAVDEVICE_VERSION_MAJOR 58 -#define LIBAVDEVICE_VERSION_MINOR 7 +#define LIBAVDEVICE_VERSION_MINOR 8 #define LIBAVDEVICE_VERSION_MICRO 100 #define LIBAVDEVICE_VERSION_INT AV_VERSION_INT(LIBAVDEVICE_VERSION_MAJOR, \ -- 2.7.4