From patchwork Sun Aug 15 08:04:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Playfair Cal X-Patchwork-Id: 29517 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp1046424iov; Sun, 15 Aug 2021 01:05:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzcl4tLRIcJt403B4HIa880DBniusTROFtVhNDAYiOr3/cifKi7AfdSnnrhJWHG2RSjlSEu X-Received: by 2002:aa7:cc0d:: with SMTP id q13mr4058472edt.109.1629014717510; Sun, 15 Aug 2021 01:05:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629014717; cv=none; d=google.com; s=arc-20160816; b=YzuJxZ0yC+T3hWfqaJta4O8PeGfl5ME6FTGLZG+iKX1NxVYIWi3GbY1Igbq3qLjjHN yoyFPbCNSHcKSuBlW/tN6iuTqBmbRoPn3y80599EzIcwV0a4iODOUObMWXf+QKM/3lyT YroZZb+Y7sYkOOQOlo4B6prJMbK0Wti3I55uVIbJFxVaBTJSe10lGeMnSm1rTZNmM9U1 vOPMvbIkEg0LTwAr4rMJyU2DZMfumwO4Q4pishwo8C8WFqWqYlyw6iVwQXHvc+B9gf41 VcsT1wRKofdwV+iWK71dHbB8yPgikaK7/1l3O02yenlr2ZFfSRo2wINaFz10O9EEwuRa idxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=U5QckUWWrRIMfQ8xCvBjic4hKfV6LMvBbrz/L5FFblM=; b=IcOpnhIp/cqobk1TLd671HVU2Q518O9F95gaC7fSKEOvvCujldjUA+s20GGBCiUDP5 3hLpusqyes2WPnx5+E7FPE8dJiAR/1gujUmOCPUY+rEkiJZ0IRni763X/txmdXfikIUS +nUWO8EZ+PbuLyhcfUY4bSddkkZbNpOampke796KxTj5fS3DKdz5XVwIxNotovjrPaBq pdCZ2Gu9Uq1DRQaZTt+Wpku/xGkl6z/OT5qa+WIF9eCAoMgfIIejJ698uaBtGtVaqtkv SbzGgzgcEYzApP9iVeTPTofnFARVSlavFvgOrYrVj8jjpH1Gv2kWufK6ECqN3BO/Tq88 tO+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=oz0QqNHB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bn3si2901140ejb.458.2021.08.15.01.05.16; Sun, 15 Aug 2021 01:05:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=oz0QqNHB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C2723689B0B; Sun, 15 Aug 2021 11:05:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7754D6808CD for ; Sun, 15 Aug 2021 11:05:06 +0300 (EEST) Received: by mail-pl1-f173.google.com with SMTP id q2so17242690plr.11 for ; Sun, 15 Aug 2021 01:05:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=VulGL9kW1rCal5P28uuoeWoioVqYIC8EDr9OFEaDa7Y=; b=oz0QqNHBJAkIYjFbkrsawkeHsQWRMIujpBO/a0VncTNvWcuWSxaZuZ9qrM4QRzNcPM LHJFzZkzvGwrO4JJQNut8Wjcq4EXYyTuBTr0WDuGXPfO0TqgruTrdGBiGgHGiusNpkWf t7w69jQpHr2UflSUgORyAHfPOWqJkECWvoLaeWgVv1uS6HgECLTfz/9O4Ar/AvalV+6P dqOigmhUfO423OoiviVV16iXHBpm/3pr9pKHqjypB2J7xsoAskYdTPGL9Plmv/ziNNrS tqTXcjCrUy/mJbdAR9hn492L8t5zr2aYYEJGkav1dezymxoJ1URIsja2WR1g50K/jf2R eh9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=VulGL9kW1rCal5P28uuoeWoioVqYIC8EDr9OFEaDa7Y=; b=RPmHeaJMHrlt8d0WAL+JKF6/baMmsx83ufLOGVN2THbktOmDPaz5CaN0bY7n70lmD2 uH97Aw1787tXHoB61I1r3Y0bFGTgG0jJ2dzHbp2a5FZQvGsTxAqPb1ytGexybWvcVAoC 3UQhOqElRxQoQloHAvWmJSI00F52tpWkoYE+bNjiKMTnKgAj/bVk7AYXoQ0gfRJH490n PE2mJsOgQc2CaL2Ub/5ZEFtaLr36kWozjReNalCdPGr4Gwp9wtcScAUzlglNvQKrge2J oO/4EWwR1r4EmsrfDq9jc69G/GufMYOYfvPoDM3XjezVd8NOTOupog9I6CINKLRUT5Pr 4j6w== X-Gm-Message-State: AOAM532LRPePVgz1/P8c1qn1aDaKeA8FY+5yZb+C/VvMb1EASRasMN1H M+elv1A1XBAEoOdOB4UsICFLvNHvcGCvHw== X-Received: by 2002:a05:6a00:140e:b029:38b:c129:9f2f with SMTP id l14-20020a056a00140eb029038bc1299f2fmr10271537pfu.75.1629014704114; Sun, 15 Aug 2021 01:05:04 -0700 (PDT) Received: from localhost.localdomain ([61.69.132.45]) by smtp.gmail.com with ESMTPSA id u24sm7902027pfm.27.2021.08.15.01.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Aug 2021 01:05:03 -0700 (PDT) From: Daniel Playfair Cal To: ffmpeg-devel@ffmpeg.org Date: Sun, 15 Aug 2021 18:04:47 +1000 Message-Id: <20210815080447.342326-1-daniel.playfair.cal@gmail.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avfilter: add dewobble_opencl filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: daniel.playfair.cal@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: WAxJaxYZslor All of the existing filters for video stabilization use an affine model (or a limited version of it) to represent the movement of the camera. When used with cameras with a very wide field of view and/or where the camera shaking is severe, the corrections result in significant geometric distortion ("wobbling"). Dewobble (https://git.sr.ht/~hedgepigdaniel/dewobble) is a library built to solve this problem. It requires knowledge of the projection used by the input camera, and it performs stabilization using a homography model, which is limited to include only changes in camera orientation. Additionally, it can perform projection change by specifying a different projection for the output camera. This is more efficient and results in less loss of information than using separate filters to perform stabilization and projection change. The dewobble_opencl filter is a wrapper for Dewobble. Dewobble supports input and output in OpenCL buffers containing NV12 frames. Hence, the filter is named dewobble_opencl and has the same limitations. Currently all of the options of Dewobble are supported. Of the two types of filter available in Dewobble (FilterSync and FilterThreaded), FilterThreaded is used. The API is synchronous, but the transformations are done in a separate thread. The purpose of this is to isolate the global per thread OpenCL context used by OpenCV, which Dewobble uses internally. This prevents dewobble_opencl from interfering with any other usage of OpenCV from within FFmpeg. Signed-off-by: Daniel Playfair Cal --- Changelog | 1 + LICENSE.md | 2 +- configure | 4 + doc/filters.texi | 149 ++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/version.h | 4 +- libavfilter/vf_dewobble_opencl.c | 1256 ++++++++++++++++++++++++++++++ 8 files changed, 1415 insertions(+), 3 deletions(-) create mode 100644 libavfilter/vf_dewobble_opencl.c diff --git a/Changelog b/Changelog index 1037688682..31f7dbdfe9 100644 --- a/Changelog +++ b/Changelog @@ -9,6 +9,7 @@ version : - Argonaut Games CVG muxer - Concatf protocol - afwtdn audio filter +- Dewobble filter version 4.4: diff --git a/LICENSE.md b/LICENSE.md index 613070e1b6..dfdf010d8e 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -112,7 +112,7 @@ The VMAF, mbedTLS, RK MPI, OpenCORE and VisualOn libraries are under the Apache version 3 of those licenses. So to combine these libraries with FFmpeg, the license version needs to be upgraded by passing `--enable-version3` to configure. -The smbclient library is under the GPL v3, to combine it with FFmpeg, +The dewobble and smbclient libraries are under the GPL v3, to combine them with FFmpeg, the options `--enable-gpl` and `--enable-version3` have to be passed to configure to upgrade FFmpeg to the GPL v3. diff --git a/configure b/configure index 82639ce057..aa136b16e5 100755 --- a/configure +++ b/configure @@ -230,6 +230,7 @@ External library support: --enable-libdavs2 enable AVS2 decoding via libdavs2 [no] --enable-libdc1394 enable IIDC-1394 grabbing using libdc1394 and libraw1394 [no] + --enable-libdewobble enable video stabilization via libdewobble [no] --enable-libfdk-aac enable AAC de/encoding via libfdk-aac [no] --enable-libflite enable flite (voice synthesis) support via libflite [no] --enable-libfontconfig enable libfontconfig, useful for drawtext filter [no] @@ -1781,6 +1782,7 @@ EXTERNAL_LIBRARY_VERSION3_LIST=" " EXTERNAL_LIBRARY_GPLV3_LIST=" + libdewobble libsmbclient " @@ -3582,6 +3584,7 @@ denoise_vaapi_filter_deps="vaapi" derain_filter_select="dnn" deshake_filter_select="pixelutils" deshake_opencl_filter_deps="opencl" +dewobble_opencl_filter_deps="libdewobble opencl" dilation_opencl_filter_deps="opencl" dnn_classify_filter_select="dnn" dnn_detect_filter_select="dnn" @@ -6410,6 +6413,7 @@ enabled libcodec2 && require libcodec2 codec2/codec2.h codec2_create -lc enabled libdav1d && require_pkg_config libdav1d "dav1d >= 0.5.0" "dav1d/dav1d.h" dav1d_version enabled libdavs2 && require_pkg_config libdavs2 "davs2 >= 1.6.0" davs2.h davs2_decoder_open enabled libdc1394 && require_pkg_config libdc1394 libdc1394-2 dc1394/dc1394.h dc1394_new +enabled libdewobble && require_pkg_config libdewobble dewobble dewobble/filter.h dewobble_filter_create_threaded enabled libdrm && require_pkg_config libdrm libdrm xf86drm.h drmGetVersion enabled libfdk_aac && { check_pkg_config libfdk_aac fdk-aac "fdk-aac/aacenc_lib.h" aacEncOpen || { require libfdk_aac fdk-aac/aacenc_lib.h aacEncOpen -lfdk-aac && diff --git a/doc/filters.texi b/doc/filters.texi index bdeb3fedfd..9bc3a869ef 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -10113,6 +10113,155 @@ A number representing position of the first frame with respect to the telecine pattern. This is to be used if the stream is cut. The default value is @code{0}. @end table +@section dewobble_opencl + +Apply motion stabilization with awareness of lens projection and/or lens projection change using libdewobble (@url{https://git.sr.ht/~hedgepigdaniel/dewobble}). + +To enable compilation of this filter you need to configure FFmpeg with +@code{--enable-libdewobble}. + +This filter accepts the following options: + +@table @option +@item in_p +@item out_p +Set the lens projection model for the input and output. + +Available values are: +@table @samp +@item rect +Rectilinear projection. + +@item fish +Equidistant fisheye projection. + +@end table + +@item in_dfov +@item out_dfov +Diagonal field of view in degrees for the input and output. + +@item in_fx +@item in_fy +@item out_fx +@item out_fy +Location of the focal point in the input and output image. +Default value is the image centre in both cases. + +@item out_w +@item out_h +Dimensions of the output image. +Default value is the same as in input image. + +@item stab +Motion stabilization algorithm. + +Available values are: +@table @samp +@item fixed +Fix the camera orientation after the first frame. + +@item none +No not apply stabilization. + +@item sg +Smooth the camera motion using a Savitzky-Golay filter. + +@end table + +Default value is @samp{sg}. + +@item stab_r +For Savitzky-Golay smoothing: the number of frames to look ahead and behind. +Higher values result in a smoother output camera path. + +Default value is 15. + +Higher values increase (OpenCL) memory usage. + +@item stab_h +For stabilization: the number of frames to look ahead to interpolate input camera rotation in frames where it cannot be detected. + +Default value is 30. + +Higher values increase (OpenCL) memory usage. + +@item interp +Pixel interpolation algorithm. + +Available values are: +@table @samp +@item nearest +Nearest neighbour interpolation (fast OpenCL implementation). + +@item linear +Bilinear interpolation (fast OpenCL implementation). + +@item cubic +Bicubic interpolation (CPU implementation). + +@item lanczos +Lanczos4 interpolation in an 8x8 neighbourhood (CPU implementation). + +@end table + +Default value is @samp{linear}. + +@item border +Border extrapolation algorithm (determines how to color pixels in the output that do not map to the input). + +Available values are: +@table @samp +@item constant +Constant color. + +@item reflect +Reflection of the input horizontally or vertically about the edge. + +@item reflect101 +Reflection of the input horizontally or vertically about the point half a pixel from the edge. + +@item replicate +Replicate the pixel on the edge in a vertical or horizontal direction. + +@item wrap +Wrap around to the opposite side of the source image. + +@end table + +Default value is @samp{constant}. + +@item border_r +@item border_g +@item border_b +For @samp{constant} border, the color to fill with (red, green, blue components). + +Default value is black. + +@item debug +Include a suite of debugging information in the output. + +Default value is disabled. + +@end table + +@subsection Examples + +@itemize +@item +Apply motion stabilization to video from a popular action cam in a certain capture mode: +@example +ffmpeg -i INPUT -vf dewobble_opencl=in_p=fish:in_dfov=145.8:out_p=fish:out_dfov=145.8:stab=sg OUTPUT +@end example + +@item +Apply stabilization and lens projection change: +@example +ffmpeg -i INPUT -vf dewobble_opencl=in_p=fish:in_dfov=145.8:out_p=rect:out_dfov=145.8:stab=sg OUTPUT +@end example + +@end itemize + @section dilation Apply dilation effect to the video. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 49c0c8342b..9a5f9ccf71 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -240,6 +240,7 @@ OBJS-$(CONFIG_DESHAKE_OPENCL_FILTER) += vf_deshake_opencl.o opencl.o \ OBJS-$(CONFIG_DESHAKE_FILTER) += vf_deshake.o transform.o OBJS-$(CONFIG_DESPILL_FILTER) += vf_despill.o OBJS-$(CONFIG_DETELECINE_FILTER) += vf_detelecine.o +OBJS-$(CONFIG_DEWOBBLE_OPENCL_FILTER) += vf_dewobble_opencl.o opencl.o OBJS-$(CONFIG_DILATION_FILTER) += vf_neighbor.o OBJS-$(CONFIG_DILATION_OPENCL_FILTER) += vf_neighbor_opencl.o opencl.o \ opencl/neighbor.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index ae74f9c891..e46131a009 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -227,6 +227,7 @@ extern const AVFilter ff_vf_deshake; extern const AVFilter ff_vf_deshake_opencl; extern const AVFilter ff_vf_despill; extern const AVFilter ff_vf_detelecine; +extern const AVFilter ff_vf_dewobble_opencl; extern const AVFilter ff_vf_dilation; extern const AVFilter ff_vf_dilation_opencl; extern const AVFilter ff_vf_displace; diff --git a/libavfilter/version.h b/libavfilter/version.h index 75cd10dccd..67f2a5883c 100644 --- a/libavfilter/version.h +++ b/libavfilter/version.h @@ -30,8 +30,8 @@ #include "libavutil/version.h" #define LIBAVFILTER_VERSION_MAJOR 8 -#define LIBAVFILTER_VERSION_MINOR 1 -#define LIBAVFILTER_VERSION_MICRO 103 +#define LIBAVFILTER_VERSION_MINOR 2 +#define LIBAVFILTER_VERSION_MICRO 100 #define LIBAVFILTER_VERSION_INT AV_VERSION_INT(LIBAVFILTER_VERSION_MAJOR, \ diff --git a/libavfilter/vf_dewobble_opencl.c b/libavfilter/vf_dewobble_opencl.c new file mode 100644 index 0000000000..2054965f98 --- /dev/null +++ b/libavfilter/vf_dewobble_opencl.c @@ -0,0 +1,1256 @@ +/* + * Copyright (c) 2021 Daniel Playfair Cal + * + * This file is part of FFmpeg. + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ +#include +#include +#include +#include +#include +#include + +#include "libavutil/avassert.h" +#include "libavutil/common.h" +#include "libavutil/imgutils.h" +#include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" +#include "libavutil/thread.h" + +#include "avfilter.h" +#include "filters.h" +#include "internal.h" +#include "opencl.h" +#include "opencl_source.h" +#include "transpose.h" +#include "video.h" + +/** + * @file + * Apply motion stabilization with awareness of lens projection and/or change + * camera projection. + * + * This filter is essentially a wrapper around dewobble + * (https://git.sr.ht/~hedgepigdaniel/dewobble). + * + * @par Queued frames + * + * libdewobble requires a queue of frames before it can provide output because + * it looks ahead to calculate a smooth camera path and to interpolate camera + * positions from frames where it fails to detect motion. The number of queued + * frames required is determined by libdewobble. + * + * @par Hardware frame allocation + * + * Input OpenCL hardware frames contain `cl_image`s but these must be converted + * to `cl_buffer`s for libdewobble. Although the filter keeps a reference to + * the input frame until the output frame is sent, it unreferences the original + * hardware buffers immediately after copying them to a `cl_buffer` in + * `consume_input_frame`. This avoids OOM issues for example when using input + * frames mapped from VA-API hardware frames where there is a low limit for how + * many can be allocated at once. The filter only owns a single input/output + * hardware frame buffer at any time, although internally it allocates OpenCL + * buffers to store the contents of a queue of frames. + */ + +/// Camera properties, mirroring those present in libdewobble's camera object. +typedef struct Camera { + /// Camera projection model, e.g. `DEWOBBLE_PROJECTION_RECTILINEAR` + int model; + + /// Camera diagonal field of view in degrees + double diagonal_fov; + + /// Width in pixels + int width; + + /// Height in pixels + int height; + + /// Horizonal coordinate of focal point in pixels + double focal_point_x; + + /// Vertical coordinate of focal point in pixels + double focal_point_y; +} Camera; + +/// Motion stabilization algorithm, mirroring those available in libdewobble. +typedef enum StabilizationAlgorithm { + + /// Do not apply stabilization + STABILIZATION_ALGORITHM_ORIGINAL, + + /// Keep the camera orientation fixed at its orientation in the first frame + STABILIZATION_ALGORITHM_FIXED, + + /// Smooth camera orientation with a Savitsky-Golay filter + STABILIZATION_ALGORITHM_SMOOTH, + + /// Number of stabilization algorithms + NB_STABILIZATION_ALGORITHMS, + +} StabilizationAlgorithm; + +/** + * dewobble_opencl filter context + */ +typedef struct DewobbleOpenCLContext { + + /// Generic OpenCL filter context + OpenCLFilterContext ocf; + + /// OpenCL command queue + cl_command_queue command_queue; + + /// Input camera (projection, focal length, etc) + Camera input_camera; + + /// Output camera (projection, focal length, etc) + Camera output_camera; + + /** + * Stabilization algorithm applied by the filter + * (@ref StabilizationAlgorithm) + */ + int stabilization_algorithm; + + /** + * The number of frames to look ahead and behind for the purpose of + * stabilizing each frame + */ + int stabilization_radius; + + /** + * The number of frames to look ahead for the purpose of interpolating + * frame rotation for frames where detection fails + */ + int stabilization_horizon; + + /** + * The algorithm to interpolate the value between source image pixels + * (e.g.\ `DEWOBBLE_INTERPOLATION_LINEAR`) + */ + int interpolation_algorithm; + + /** + * The algorithm used to fill in unmapped areas of the output (e.g.\ + * `DEWOBBLE_BORDER_CONSTANT`) + */ + int border_type; + + /** + * The color used to fill unmapped areas of the output when + * @ref border_type is `DEWOBBLE_BORDER_CONSTANT` + */ + double border_color[4]; + + /// Whether to include debugging information in the output + int debug; + + /// Whether the filter has been initialized + int initialized; + + /// The status of the input link + int input_status; + + /// The time that the input status was reached + int64_t input_status_pts; + + /** + * Number of frame jobs currently in progress (read from inlink but not + * yet sent to outlink) + */ + int nb_frames_in_progress; + + /// Number of frames consumed so far + long nb_frames_consumed; + + /// The instance of libdewobble's filter + DewobbleFilter dewobble_filter; + +} DewobbleOpenCLContext; + +/** + * Convert degrees to radians. + * @param degrees the number of degrees + * @return the equivalent number of radians + */ +static double degrees_to_radians(double degrees) +{ + return degrees * M_PI / 180; +} + +/** + * Initialize the libdewobble filter instance. + * @param avctx the filter context + * @return 0 on success, otherwise a negative error code + */ +static int init_libdewobble_filter(AVFilterContext * avctx) +{ + DewobbleOpenCLContext *ctx = avctx->priv; + DewobbleStabilizer stabilizer = NULL; + DewobbleCamera input_camera = NULL, output_camera = NULL; + DewobbleFilterConfig config = NULL; + + input_camera = dewobble_camera_create(ctx->input_camera.model, + degrees_to_radians + (ctx->input_camera.diagonal_fov), + ctx->input_camera.width, + ctx->input_camera.height, + ctx->input_camera.focal_point_x, + ctx->input_camera.focal_point_y); + if (input_camera == NULL) { + goto fail; + } + output_camera = dewobble_camera_create(ctx->output_camera.model, + degrees_to_radians + (ctx-> + output_camera.diagonal_fov), + ctx->output_camera.width, + ctx->output_camera.height, + ctx-> + output_camera.focal_point_x, + ctx-> + output_camera.focal_point_y); + if (output_camera == NULL) { + goto fail; + } + switch (ctx->stabilization_algorithm) { + case STABILIZATION_ALGORITHM_ORIGINAL: + stabilizer = dewobble_stabilizer_create_none(); + break; + case STABILIZATION_ALGORITHM_FIXED: + stabilizer = + dewobble_stabilizer_create_fixed(input_camera, + ctx->stabilization_horizon); + break; + case STABILIZATION_ALGORITHM_SMOOTH: + stabilizer = + dewobble_stabilizer_create_savitzky_golay(input_camera, + ctx->stabilization_radius, + ctx->stabilization_horizon); + break; + } + if (stabilizer == NULL) { + goto fail; + } + config = dewobble_filter_config_create(input_camera, + output_camera, stabilizer); + dewobble_filter_config_set_opencl_context(config, + ctx->ocf.hwctx->context); + dewobble_filter_config_set_opencl_device(config, + ctx->ocf.hwctx->device_id); + dewobble_filter_config_set_interpolation(config, + ctx->interpolation_algorithm); + dewobble_filter_config_set_border_type(config, ctx->border_type); + dewobble_filter_config_set_border_color(config, ctx->border_color); + dewobble_filter_config_set_debug(config, ctx->debug); + + ctx->dewobble_filter = dewobble_filter_create_threaded(config); + + dewobble_filter_config_destroy(&config); + if (ctx->dewobble_filter == NULL) { + goto fail; + } + dewobble_stabilizer_destroy(&stabilizer); + return 0; + + fail: + dewobble_stabilizer_destroy(&stabilizer); + dewobble_camera_destroy(&input_camera); + dewobble_camera_destroy(&output_camera); + return AVERROR(ENOMEM); +} + +/** + * Initialize the filter based on the options + * @param avctx the filter context + * @return 0 on success, otherwise a negative error code + */ +static int dewobble_opencl_init(AVFilterContext * avctx) +{ + DewobbleOpenCLContext *ctx = avctx->priv; + av_log(avctx, AV_LOG_VERBOSE, "Init\n"); + if (ctx->input_camera.model == DEWOBBLE_NB_PROJECTIONS + || ctx->output_camera.model == DEWOBBLE_NB_PROJECTIONS) { + av_log(avctx, AV_LOG_ERROR, "both in_p and out_p must be set\n"); + return AVERROR(EINVAL); + } + if (ctx->input_camera.diagonal_fov == 0 || + ctx->output_camera.diagonal_fov == 0) { + av_log(avctx, AV_LOG_ERROR, + "both in_dfov and out_dfov must be set\n"); + return AVERROR(EINVAL); + } + if (ctx->stabilization_algorithm == STABILIZATION_ALGORITHM_ORIGINAL) { + ctx->stabilization_horizon = 0; + } + return ff_opencl_filter_init(avctx); +} + +/** + * Clean up the filter on destruction. + * @param avctx the filter context + */ +static void dewobble_opencl_uninit(AVFilterContext * avctx) +{ + cl_int cle; + DewobbleOpenCLContext *ctx = avctx->priv; + av_log(avctx, AV_LOG_VERBOSE, "Uninit\n"); + if (ctx->command_queue) { + cle = clReleaseCommandQueue(ctx->command_queue); + if (cle != CL_SUCCESS) { + av_log(avctx, + AV_LOG_ERROR, + "Failed to release command queue: %d.\n", cle); + } + } + dewobble_filter_destroy(&ctx->dewobble_filter); + ff_opencl_filter_uninit(avctx); +} + +/** + * Perform further initialization of the filter when the first input frame is + * available. + * @param avctx the filter context + * @param first_frame the first input frame + * @return 0 on success, otherwise a negative error code + */ +static int dewobble_opencl_frames_init(AVFilterContext * avctx, + AVFrame * first_frame) +{ + DewobbleOpenCLContext *ctx = avctx->priv; + AVFilterLink *inlink = avctx->inputs[0]; + cl_int cle; + int err; + + if (first_frame->crop_top % 2 == 1 || first_frame->crop_bottom % 2 == 1 + || first_frame->crop_left % 2 == 1 + || first_frame->crop_right % 2 == 1) { + av_log(avctx, AV_LOG_ERROR, + "Cropping by an odd number of pixels is not supported!\n"); + return AVERROR(EINVAL); + } + + if ((first_frame->crop_top || first_frame->crop_bottom) && + (ctx->output_camera.height == 0 || + ctx->output_camera.focal_point_y == DBL_MAX)) { + av_log(avctx, + AV_LOG_WARNING, + "Input is vertically cropped, but output height or vertical " + "focal point is not set. The default values are based on the " + "uncropped input!\n"); + } + + if ((first_frame->crop_left || first_frame->crop_right) && + (ctx->output_camera.width == 0 || + ctx->output_camera.focal_point_x == DBL_MAX)) { + av_log(avctx, + AV_LOG_WARNING, + "Input is horizontally cropped, but output width or horizontal " + "focal point is not set. The default values are based on the " + "uncropped input!\n"); + } + + ctx->input_camera.width = inlink->w - first_frame->crop_left - + first_frame->crop_right; + ctx->input_camera.height = inlink->h - first_frame->crop_top - + first_frame->crop_bottom; + + // Output camera width must match the filter output + ctx->output_camera.width = ctx->ocf.output_width; + ctx->output_camera.height = ctx->ocf.output_height; + + // Focal points default to the image center (disregarding cropping) + if (ctx->input_camera.focal_point_x == DBL_MAX) { + ctx->input_camera.focal_point_x = (inlink->w - 1) / 2.0 - + first_frame->crop_left; + } + if (ctx->input_camera.focal_point_y == DBL_MAX) { + ctx->input_camera.focal_point_y = (inlink->h - 1) / 2.0 - + first_frame->crop_top; + } + if (ctx->output_camera.focal_point_x == DBL_MAX) { + ctx->output_camera.focal_point_x = + (ctx->output_camera.width - 1) / 2.0; + } + if (ctx->output_camera.focal_point_y == DBL_MAX) { + ctx->output_camera.focal_point_y = + (ctx->output_camera.height - 1) / 2.0; + } + + ctx->command_queue = clCreateCommandQueue(ctx->ocf.hwctx->context, + ctx->ocf.hwctx->device_id, + 0, &cle); + if (cle) { + av_log(avctx, + AV_LOG_ERROR, + "Failed to create OpenCL command queue %d.\n", cle); + return AVERROR(EIO); + } + + err = init_libdewobble_filter(avctx); + if (err) { + av_log(avctx, + AV_LOG_ERROR, + "Failed to initialise libdewobble filter %d.\n", err); + return AVERROR(EIO); + } + + ctx->initialized = 1; + return 0; +} + +/** + * Perform initialization based on the input filter link. + * @param inlink the input filter link + * @return 0 on success, otherwise a negative error code + */ +static int dewobble_opencl_config_input(AVFilterLink * inlink) +{ + AVFilterContext *avctx = inlink->dst; + DewobbleOpenCLContext *ctx = avctx->priv; + int ret; + + ret = ff_opencl_filter_config_input(inlink); + if (ret < 0) { + return ret; + } + + if (ctx->ocf.output_format != AV_PIX_FMT_NV12) { + av_log(avctx, AV_LOG_ERROR, "Only NV12 input is supported!\n"); + return AVERROR(ENOSYS); + } + + if (inlink->w % 2 == 1 || inlink->h % 2 == 1) { + av_log(avctx, + AV_LOG_ERROR, + "Input with odd dimensions is not supported!\n"); + return AVERROR(EINVAL); + } + + if (ctx->output_camera.width % 2 == 1 || + ctx->output_camera.height % 2 == 1) { + av_log(avctx, + AV_LOG_ERROR, "Output camera must have even dimensions!\n"); + return AVERROR(EINVAL); + } + // Output dimensions default to the input dimensions (disregarding cropping) + ctx->ocf.output_width = ctx->output_camera.width + ? ctx->output_camera.width : inlink->w; + ctx->ocf.output_height = ctx->output_camera.height + ? ctx->output_camera.height : inlink->h; + + return 0; +} + +/** + * Copy the contents of an input frame to an OpenCL buffer. + * @param avctx the filter context + * @param context the OpenCL context to use + * @param command_queue the OpenCL command queue to use + * @param frame the input @ref AVFrame + * @param input_buffer the OpenCL buffer to copy the frame into + * @return 0 on success, otherwise a negative error code + */ +static cl_int copy_frame_to_buffer(AVFilterContext * avctx, + cl_context context, + cl_command_queue command_queue, + AVFrame * frame, cl_mem input_buffer) +{ + int err; + DewobbleOpenCLContext *ctx = avctx->priv; + cl_mem luma = (cl_mem) frame->data[0]; + cl_mem chroma = (cl_mem) frame->data[1]; + cl_int cle = 0; + size_t src_luma_origin[3] = { frame->crop_left, frame->crop_top, 0 }; + size_t src_chroma_origin[3] = { + frame->crop_left / 2, + frame->crop_top / 2, + 0, + }; + size_t luma_region[3] = { + ctx->input_camera.width, + ctx->input_camera.height, + 1, + }; + size_t chroma_region[3] = { + ctx->input_camera.width / 2, + ctx->input_camera.height / 2, + 1, + }; + cl_event copy_finished[2]; + + cle = clEnqueueCopyImageToBuffer(command_queue, + luma, + input_buffer, + src_luma_origin, + luma_region, + 0, 0, NULL, ©_finished[0] + ); + CL_FAIL_ON_ERROR(AVERROR(EINVAL), + "Failed to enqueue copy luma image to buffer: %d\n", + cle); + + cle = clEnqueueCopyImageToBuffer(command_queue, + chroma, + input_buffer, + src_chroma_origin, + chroma_region, + ctx->input_camera.width * + ctx->input_camera.height * 1, 0, NULL, + ©_finished[1] + ); + CL_FAIL_ON_ERROR(AVERROR(EINVAL), + "Failed to enqueue copy chroma image to buffer: %d\n", + cle); + + cle = clWaitForEvents(2, copy_finished); + CL_FAIL_ON_ERROR(AVERROR(EINVAL), + "Failed to copy images to buffer: %d\n", cle); + + return 0; + + fail: + return err; +} + +/** + * Copy the contents of an OpenCL buffer to an output frame. + * @param avctx the filter context + * @param buffer the OpenCL buffer + * @param output_frame the output frame + * @return 0 on success, otherwise a negative error code + */ +static int copy_buffer_to_frame(AVFilterContext * avctx, + cl_mem buffer, AVFrame * output_frame) +{ + int err; + DewobbleOpenCLContext *ctx = avctx->priv; + cl_mem luma = (cl_mem) output_frame->data[0]; + cl_mem chroma = (cl_mem) output_frame->data[1]; + cl_int cle = 0; + size_t dst_origin[3] = { 0, 0, 0 }; + size_t luma_region[3] = + { output_frame->width, output_frame->height, 1 }; + size_t chroma_region[3] = { + output_frame->width / 2, + output_frame->height / 2, + 1, + }; + cl_event copy_finished[2]; + + cle = clEnqueueCopyBufferToImage(ctx->command_queue, + buffer, + luma, + 0, + dst_origin, + luma_region, + 0, NULL, ©_finished[0] + ); + CL_FAIL_ON_ERROR(AVERROR(EINVAL), + "Failed to enqueue copy buffer to luma image: %d\n", + cle); + cle = clEnqueueCopyBufferToImage(ctx->command_queue, + buffer, + chroma, + output_frame->width * + output_frame->height * 1, dst_origin, + chroma_region, 0, NULL, + ©_finished[1] + ); + CL_FAIL_ON_ERROR(AVERROR(EINVAL), + "Failed to enqueue copy buffer to luma image: %d\n", + cle); + + cle = clWaitForEvents(2, copy_finished); + CL_FAIL_ON_ERROR(AVERROR(EINVAL), + "Failed to copy buffer to images: %d\n", cle); + + return 0; + + fail: + return err; +} + +/** + * Consume an input frame and push it to the libdewobble filter. + * @param avctx the filter context + * @param input_frame the input frame + * @return 0 on success, otherwise a negative error code + */ +static int consume_input_frame(AVFilterContext * avctx, + AVFrame * input_frame) +{ + DewobbleOpenCLContext *ctx = avctx->priv; + cl_mem input_buffer; + int err = 0; + cl_int cle; + int i; + + if (!input_frame->hw_frames_ctx) { + return AVERROR(EINVAL); + } + + if (!ctx->initialized) { + av_log(avctx, AV_LOG_VERBOSE, "Initializing\n"); + err = dewobble_opencl_frames_init(avctx, input_frame); + if (err < 0) { + return err; + } + } + + input_buffer = + dewobble_filter_get_input_frame_buffer(ctx->dewobble_filter, &cle); + + CL_FAIL_ON_ERROR(AVERROR(ENOMEM), "Failed to create buffer: %d\n", + cle); + + err = copy_frame_to_buffer(avctx, + ctx->ocf.hwctx->context, + ctx->command_queue, + input_frame, input_buffer); + if (err) { + goto fail; + } + // Free original input frame buffers + for (i = 0; input_frame->buf[i] != NULL; i++) { + av_buffer_unref(&input_frame->buf[i]); + } + + dewobble_filter_push_frame(ctx->dewobble_filter, input_buffer, + (void **) input_frame); + + ctx->nb_frames_in_progress += 1; + ctx->nb_frames_consumed += 1; + + return 0; + + fail: + return err; +} + +/** + * Create and send on an output frame using an output buffer pulled from the + * libdewobble filter. + * @param avctx the filter context + * @return 0 on success, otherwise a negative error code. + */ +static int send_output_frame(AVFilterContext * avctx) +{ + AVFilterLink *inlink = avctx->inputs[0]; + AVFilterLink *outlink = avctx->outputs[0]; + DewobbleOpenCLContext *ctx = avctx->priv; + AVFrame *input_frame; + AVFrame *output_frame = NULL; + cl_mem output_buffer = NULL, input_buffer; + int err; + + dewobble_filter_pull_frame(ctx->dewobble_filter, + &output_buffer, &input_buffer, + (void **) &input_frame); + dewobble_filter_release_input_frame_buffer(ctx->dewobble_filter, + &input_buffer); + + output_frame = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (output_frame == NULL) { + err = AVERROR(ENOMEM); + goto fail; + } + + err = av_frame_copy_props(output_frame, input_frame); + if (err) { + goto fail; + } + + output_frame->crop_top = 0; + output_frame->crop_bottom = 0; + output_frame->crop_left = 0; + output_frame->crop_right = 0; + + err = copy_buffer_to_frame(avctx, output_buffer, output_frame); + if (err) { + goto fail; + } + + dewobble_filter_release_output_frame_buffer(ctx->dewobble_filter, + &output_buffer); + + av_log(avctx, + AV_LOG_VERBOSE, + "Sending output frame %ld (%d in progress)\n", + ctx->nb_frames_consumed - ctx->nb_frames_in_progress, + ctx->nb_frames_in_progress); + ctx->nb_frames_in_progress -= 1; + + err = ff_filter_frame(outlink, output_frame); + if (err < 0) { + goto fail; + } + + if (!dewobble_filter_frame_ready(ctx->dewobble_filter)) { + ff_inlink_request_frame(inlink); + } + + if (ctx->input_status && ctx->nb_frames_in_progress == 0) { + av_log(avctx, AV_LOG_VERBOSE, "Output reached EOF\n"); + ff_outlink_set_status(outlink, + ctx->input_status, ctx->input_status_pts); + } + + av_frame_free(&input_frame); + return 0; + + fail: + av_frame_free(&input_frame); + av_log(avctx, AV_LOG_ERROR, "Failed to send output frame: %d\n", err); + av_frame_free(&output_frame); + return err; +} + +/** + * Attempt to consume an input frame, and push it to the libdewobble filter + * if one is available. + * @param avctx the filter context + * @return 0 on success, otherwise a negative error code + */ +static int try_consume_input_frame(AVFilterContext * avctx) +{ + AVFilterLink *inlink = avctx->inputs[0]; + DewobbleOpenCLContext *ctx = avctx->priv; + int err = 0; + AVFrame *input_frame; + + // If necessary, attempt to consume a frame from the input + if (!ctx->initialized || + !dewobble_filter_frame_ready(ctx->dewobble_filter) + ) { + err = ff_inlink_consume_frame(inlink, &input_frame); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to read input frame\n"); + return err; + } else if (err > 0) { + av_log(avctx, + AV_LOG_VERBOSE, + "Consuming input frame %ld (%d in progress)\n", + ctx->nb_frames_consumed, ctx->nb_frames_in_progress); + err = consume_input_frame(avctx, input_frame); + + if (err) { + av_log(avctx, + AV_LOG_ERROR, + "Failed to consume input frame: %d\n", err); + return err; + } + } + } + return err; +} + +/** + * Read the input status and update the filter state and output status as + * appropriate. + * @param avctx the filter context + */ +static void check_input_status(AVFilterContext * avctx) +{ + AVFilterLink *inlink = avctx->inputs[0]; + AVFilterLink *outlink = avctx->outputs[0]; + DewobbleOpenCLContext *ctx = avctx->priv; + + // Check for end of input + if (!ctx->input_status && ff_inlink_acknowledge_status(inlink, + &ctx->input_status, + &ctx->input_status_pts)) + { + if (ctx->input_status == AVERROR_EOF) { + av_log(avctx, AV_LOG_VERBOSE, "Reached input EOF\n"); + dewobble_filter_end_input(ctx->dewobble_filter); + } else { + av_log(avctx, + AV_LOG_ERROR, "Input status: %d\n", ctx->input_status); + } + + if (ctx->nb_frames_in_progress == 0) { + av_log(avctx, AV_LOG_VERBOSE, "Sending output EOF\n"); + ff_outlink_set_status(outlink, + ctx->input_status, + ctx->input_status_pts); + } + } +} + +/** + * Perform some work to advance the filtering process + * @param avctx the filter context + * @return 0 if progress was made, otherwise a negative error code + */ +static int activate(AVFilterContext * avctx) +{ + DewobbleOpenCLContext *ctx = avctx->priv; + AVFilterLink *inlink = avctx->inputs[0]; + AVFilterLink *outlink = avctx->outputs[0]; + int err = 0; + + // Forward any output status to input + err = ff_outlink_get_status(outlink); + if (err) { + av_log(avctx, + AV_LOG_VERBOSE, "forwarding status to inlink: %d\n", err); + ff_inlink_set_status(inlink, err); + return 0; + } + // Consume an input frame if possible + err = try_consume_input_frame(avctx); + if (err) { + av_log(avctx, + AV_LOG_ERROR, "try_consume_input_frame failed: %d\n", err); + return 0; + } + // Check input status, including detecting EOF + check_input_status(avctx); + + // If possible, send an output frame + if (dewobble_filter_frame_ready(ctx->dewobble_filter)) { + err = send_output_frame(avctx); + if (err < 0) { + av_log(avctx, + AV_LOG_ERROR, "send_output_frame failed: %d\n", err); + goto fail; + } + } + // Schedule the next activation + if (ff_inlink_check_available_frame(inlink)) { + // Immediately, if input frames are still queued + ff_filter_set_ready(avctx, 1); + } else if (dewobble_filter_frame_ready(ctx->dewobble_filter)) { + // Immediately, if output frames are ready + ff_filter_set_ready(avctx, 1); + } else { + // Otherwise when more input frames are ready + ff_inlink_request_frame(inlink); + } + + return FFERROR_NOT_READY; + + fail: + ff_outlink_set_status(outlink, AVERROR_UNKNOWN, 0); + return err; +} + +/// Get the offset of a member in @ref DewobbleOpenCLContext +#define OFFSET(x) offsetof(DewobbleOpenCLContext, x) + +/// Get the offset of a member in @ref Camera +#define OFFSET_CAMERA(x) offsetof(Camera, x) + +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) + +static const AVOption dewobble_opencl_options[] = { + // Input camera options + { + "in_p", + "input camera projection model", + OFFSET(input_camera) + OFFSET_CAMERA(model), + AV_OPT_TYPE_INT, + {.i64 = DEWOBBLE_PROJECTION_EQUIDISTANT_FISHEYE}, + 0, + DEWOBBLE_NB_PROJECTIONS - 1, + FLAGS, + "model", + }, + { + "in_dfov", + "input camera diagonal field of view in degrees", + OFFSET(input_camera) + OFFSET_CAMERA(diagonal_fov), + AV_OPT_TYPE_DOUBLE, + {.dbl = 0}, + 0, + DBL_MAX, + .flags = FLAGS, + }, + { + "in_fx", + "horizontal coordinate of focal point in input camera (default: center)", + OFFSET(input_camera) + OFFSET_CAMERA(focal_point_x), + AV_OPT_TYPE_DOUBLE, + {.dbl = DBL_MAX}, + -DBL_MAX, + DBL_MAX, + .flags = FLAGS, + }, + { + "in_fy", + "vertical coordinate of focal point in input camera (default: center)", + OFFSET(input_camera) + OFFSET_CAMERA(focal_point_y), + AV_OPT_TYPE_DOUBLE, + {.dbl = DBL_MAX}, + -DBL_MAX, + DBL_MAX, + .flags = FLAGS, + }, + + // Output camera options + { + "out_p", + "output camera projection model", + OFFSET(output_camera) + OFFSET_CAMERA(model), + AV_OPT_TYPE_INT, + {.i64 = DEWOBBLE_PROJECTION_RECTILINEAR}, + 0, + DEWOBBLE_NB_PROJECTIONS - 1, + FLAGS, + "model", + }, + { + "out_dfov", + "output camera diagonal field of view in degrees", + OFFSET(output_camera) + OFFSET_CAMERA(diagonal_fov), + AV_OPT_TYPE_DOUBLE, + {.dbl = 0}, + 0, + DBL_MAX, + .flags = FLAGS, + }, + { + "out_w", + "output camera width in pixels (default: same as input)", + OFFSET(output_camera) + OFFSET_CAMERA(width), + AV_OPT_TYPE_INT, + {.i64 = 0}, + 0, + SHRT_MAX, + .flags = FLAGS, + }, + { + "out_h", + "output camera height in pixels (default: same as input)", + OFFSET(output_camera) + OFFSET_CAMERA(height), + AV_OPT_TYPE_INT, + {.i64 = 0}, + 0, + SHRT_MAX, + .flags = FLAGS, + }, + { + "out_fx", + "horizontal coordinate of focal point in output camera " + "(default: center)", + OFFSET(output_camera) + OFFSET_CAMERA(focal_point_x), + AV_OPT_TYPE_DOUBLE, + {.dbl = DBL_MAX}, + -DBL_MAX, + DBL_MAX, + .flags = FLAGS, + }, + { + "out_fy", + "vertical coordinate of focal point in output camera " + "(default: center)", + OFFSET(output_camera) + OFFSET_CAMERA(focal_point_y), + AV_OPT_TYPE_DOUBLE, + {.dbl = DBL_MAX}, + -DBL_MAX, + DBL_MAX, + .flags = FLAGS, + }, + + // Stabilization options + { + "stab", + "camera orientation stabilization algorithm", + OFFSET(stabilization_algorithm), + AV_OPT_TYPE_INT, + {.i64 = STABILIZATION_ALGORITHM_SMOOTH}, + 0, + NB_STABILIZATION_ALGORITHMS - 1, + FLAGS, + "stab", + }, + { + "stab_r", + "for Savitzky-Golay smoothing: the number of frames " + "to look ahead and behind", + OFFSET(stabilization_radius), + AV_OPT_TYPE_INT, + {.i64 = 15}, + 1, + INT_MAX, + FLAGS, + }, + { + "stab_h", + "for stabilization: the number of frames to look " + "ahead to interpolate rotation in frames where it cannot be detected", + OFFSET(stabilization_horizon), + AV_OPT_TYPE_INT, + {.i64 = 30}, + 0, + INT_MAX, + FLAGS, + }, + + // General options + { + "interp", + "interpolation algorithm", + OFFSET(interpolation_algorithm), + AV_OPT_TYPE_INT, + {.i64 = DEWOBBLE_INTERPOLATION_LINEAR}, + 0, + DEWOBBLE_NB_INTERPOLATIONS - 1, + FLAGS, + "interpolation", + }, + { + "border", + "border fill mode", + OFFSET(border_type), + AV_OPT_TYPE_INT, + {.i64 = DEWOBBLE_BORDER_CONSTANT}, + 0, + DEWOBBLE_NB_BORDER_TYPES - 1, + FLAGS, + "border_type", + }, + { + "border_r", + "border fill color (red component)", + OFFSET(border_color) + sizeof(double) * 2, + AV_OPT_TYPE_DOUBLE, + {.i64 = 0}, + 0, + 255, + FLAGS, + }, + { + "border_g", + "border fill color (green component)", + OFFSET(border_color) + sizeof(double) * 1, + AV_OPT_TYPE_DOUBLE, + {.i64 = 0}, + 0, + 255, + FLAGS, + }, + { + "border_b", + "border fill color (blue component)", + OFFSET(border_color) + sizeof(double) * 0, + AV_OPT_TYPE_DOUBLE, + {.i64 = 0}, + 0, + 255, + FLAGS, + }, + { + "debug", + "whether to include debugging information in the output", + OFFSET(debug), + AV_OPT_TYPE_BOOL, + {.i64 = 0}, + 0, + 1, + FLAGS, + }, + + // Camera models + { + "rect", + "rectilinear projection", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_PROJECTION_RECTILINEAR}, + INT_MIN, + INT_MAX, + FLAGS, + "model" }, + { + "fish", + "equidistant fisheye projection", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_PROJECTION_EQUIDISTANT_FISHEYE}, + INT_MIN, + INT_MAX, + FLAGS, + "model" }, + + // Stabilization algorithms + { + "fixed", + "fix the camera orientation after the first frame", + 0, + AV_OPT_TYPE_CONST, + {.i64 = STABILIZATION_ALGORITHM_FIXED}, + INT_MIN, + INT_MAX, + FLAGS, + "stab" }, + { + "none", + "do not apply stabilization", + 0, + AV_OPT_TYPE_CONST, + {.i64 = STABILIZATION_ALGORITHM_ORIGINAL}, + INT_MIN, + INT_MAX, + FLAGS, + "stab" }, + { + "sg", + "smooth the camera orientation using a Savitzky-Golay filter", + 0, + AV_OPT_TYPE_CONST, + {.i64 = STABILIZATION_ALGORITHM_SMOOTH}, + INT_MIN, + INT_MAX, + FLAGS, + "stab" }, + + // Interpolation algorithms + { + "nearest", + "nearest neighbour interpolation (fast)", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_INTERPOLATION_NEAREST}, + INT_MIN, + INT_MAX, + FLAGS, + "interpolation" }, + { + "linear", + "bilinear interpolation (fast)", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_INTERPOLATION_LINEAR}, + INT_MIN, + INT_MAX, + FLAGS, + "interpolation" }, + { + "cubic", + "bicubic interpolation (medium)", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_INTERPOLATION_CUBIC}, + INT_MIN, + INT_MAX, + FLAGS, + "interpolation" }, + { + "lanczos", + "Lanczos4, in an 8x8 neighbourhood (slow)", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_INTERPOLATION_LANCZOS4}, + INT_MIN, + INT_MAX, + FLAGS, + "interpolation" }, + + // Border fill algorithms + { + "constant", + "constant color (default black)", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_BORDER_CONSTANT}, + INT_MIN, + INT_MAX, + FLAGS, + "border_type" }, + { + "reflect", + "reflection of the input about the edge", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_BORDER_REFLECT}, + INT_MIN, + INT_MAX, + FLAGS, + "border_type" }, + { + "reflect101", + "reflection of the input about the middle of the pixel on the edge", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_BORDER_REFLECT_101}, + INT_MIN, + INT_MAX, + FLAGS, + "border_type" }, + { + "replicate", + "replicate the pixel on the edge", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_BORDER_REPLICATE}, + INT_MIN, + INT_MAX, + FLAGS, + "border_type" }, + { + "wrap", + "wrap around to the opposite side of the source image", + 0, + AV_OPT_TYPE_CONST, + {.i64 = DEWOBBLE_BORDER_WRAP}, + INT_MIN, + INT_MAX, + FLAGS, + "border_type" }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(dewobble_opencl); + +static const AVFilterPad dewobble_opencl_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = &dewobble_opencl_config_input, + }, + { NULL } +}; + +static const AVFilterPad dewobble_opencl_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = &ff_opencl_filter_config_output, + }, + { NULL } +}; + +const AVFilter ff_vf_dewobble_opencl = { + .name = "dewobble_opencl", + .description = + NULL_IF_CONFIG_SMALL + ("apply motion stabilization with awareness of camera projection " + "and/or change camera projection"), + .priv_size = sizeof(DewobbleOpenCLContext), + .priv_class = &dewobble_opencl_class, + .init = &dewobble_opencl_init, + .uninit = &dewobble_opencl_uninit, + .query_formats = &ff_opencl_filter_query_formats, + .inputs = dewobble_opencl_inputs, + .outputs = dewobble_opencl_outputs, + .activate = activate, + .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, +};