From patchwork Tue Jun 8 10:45:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Liu X-Patchwork-Id: 28167 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp4277940iof; Tue, 8 Jun 2021 03:45:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyiCCkrrXyMTQ95kOU9JNfm+UW8+C6VZ13w5/9o6W2ZCvMmt0r1UL4nMlDQfoLx2LaRLfiK X-Received: by 2002:a50:fe81:: with SMTP id d1mr24986035edt.219.1623149126219; Tue, 08 Jun 2021 03:45:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1623149126; cv=none; d=google.com; s=arc-20160816; b=CUp37V4GuW7TxZPiFL7UJPYFV6Sxn14ZosF/BOfSXfkIkassbNTGJu8oYnsfBgTSMQ 9y115w3YDhwAz5h0cDpsZawYUChMJCiGFCIvXs/w3k8sxBnH6pLSGJWdT+Jez4B+Iuw/ +1UIViyLrmOi3DRfH9B6IVwo9xOKKVGBngFtKkNIESAmwK2eO+zTbn8y1FWHQ9gYz0NT BX7W/dHLx3ZSrh1VD0gIl1yf/Md77VMksdAKanjuLAE1krQ/HrMlEUOUpjwdWsHAiiIO 7UJS/LYJAiRUbons+5tMvL/bl/9D+ZtE0Mb4nBBWp2mOALFloUobe5KoaJ7e/BcpndQ8 wrhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:feedback-id:mime-version:references :in-reply-to:message-id:date:to:from:delivered-to; bh=C2TbLuxLk2+g09MMn6qtVL8dvaZhJmZcztPJARWb82M=; b=RtX2qtaq1XNyctIplLPgNq71wg9xC0UdU+4gYoFQvPjRUTuttyrPHrgDPN9Ye8Zths WB1G2u9pG4VD6pS7DT+0ffyey9i5FFC74cAWwC8sn3m35Cz5mQIaSxCfHrHhQhk83JJH c1bmuY/agI4HOZVNPotXbgnahmDgX1DZ7Ms0Zc6dqbdp2h1UzUbmKT1OEYVAqUkcFj0A zXM4XMmlTBtvdPeq/zNKA3V3DVBDK4VVzezMnG0R/uKJiymRdzcQrCHpBFIqOSr/5Q6e qABv4+dYiDOtL/SOx1BMpTcWqAY6Ne0aD4UB7+bdZHme7CsunnlnxQxvz1kkQU03KqmT cgCQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kuaishou.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j1si11884538ejj.226.2021.06.08.03.45.25; Tue, 08 Jun 2021 03:45:26 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kuaishou.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 584B36808BA; Tue, 8 Jun 2021 13:45:21 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtpbgeu1.qq.com (smtpbgeu1.qq.com [52.59.177.22]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 084FB6802B3 for ; Tue, 8 Jun 2021 13:45:13 +0300 (EEST) X-QQ-mid: bizesmtp37t1623149107tz11prwu Received: from localhost (unknown [103.107.216.225]) by esmtp6.qq.com (ESMTP) with id ; Tue, 08 Jun 2021 18:45:05 +0800 (CST) X-QQ-SSF: 01100000002000Z0Z000B00A0020000 X-QQ-FEAT: TU3YmX8YeZlbWkPpKoz4s+dtNgAyj/AHIvxtI/jBybDnMc7aXu9whH8eZ+spZ mGv2ZtLjk8oQHk82okXfrYmsfvKS+5jt4ouy9bdF0an/nFB8Oj3H0amQ+Q3UDEpSTOpMZdk Jag0+sDv3iIwidsa65wF29HlFkyJrcvCtoVvjnCKw+QY7WB8FvpR0XqTjEmWQcFftOIWbjH 1rU/yop64q14VnXzQ8mHvCIgLO2Vz5bmoNRiV07ODdc8QzrGkb1fPSg0Ozra43rcYZCWoaV qLGWbdgHPGSSxGz66LmCX+kMCdXbZc0TDKFI6oS0DWTxTHgLKrUlmV416fEO17Sk6vt0tvE r7mG7Za X-QQ-GoodBg: 0 X-QQ-CSender: lq@chinaffmpeg.org From: Steven Liu To: ffmpeg-devel@ffmpeg.org Date: Tue, 8 Jun 2021 18:45:03 +0800 Message-Id: <20210608104503.25677-1-liuqi05@kuaishou.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <8d392a1f-3301-7b73-3a1a-2df5f0e28cd4@rothenpieler.org> References: <8d392a1f-3301-7b73-3a1a-2df5f0e28cd4@rothenpieler.org> MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:chinaffmpeg.org:qybgforeign:qybgforeign5 X-QQ-Bgrelay: 1 Subject: [FFmpeg-devel] [PATCH v2] avfilter/vf_overlay_cuda: support expression of x y position X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Steven Liu Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: LI4GmMavnK8B and add per-frame / init mode for it. Signed-off-by: Steven Liu --- libavfilter/vf_overlay_cuda.c | 153 ++++++++++++++++++++++++++++++++-- 1 file changed, 144 insertions(+), 9 deletions(-) diff --git a/libavfilter/vf_overlay_cuda.c b/libavfilter/vf_overlay_cuda.c index 8a4d2c4312..b2ed8de24e 100644 --- a/libavfilter/vf_overlay_cuda.c +++ b/libavfilter/vf_overlay_cuda.c @@ -30,6 +30,7 @@ #include "libavutil/hwcontext.h" #include "libavutil/hwcontext_cuda_internal.h" #include "libavutil/cuda_check.h" +#include "libavutil/eval.h" #include "avfilter.h" #include "framesync.h" @@ -41,6 +42,9 @@ #define BLOCK_X 32 #define BLOCK_Y 16 +#define MAIN 0 +#define OVERLAY 1 + static const enum AVPixelFormat supported_main_formats[] = { AV_PIX_FMT_NV12, AV_PIX_FMT_YUV420P, @@ -54,6 +58,38 @@ static const enum AVPixelFormat supported_overlay_formats[] = { AV_PIX_FMT_NONE, }; +enum var_name { + VAR_MAIN_W, VAR_MW, + VAR_MAIN_H, VAR_MH, + VAR_OVERLAY_W, VAR_OW, + VAR_OVERLAY_H, VAR_OH, + VAR_X, + VAR_Y, + VAR_N, + VAR_POS, + VAR_T, + VAR_VARS_NB +}; + +enum EvalMode { + EVAL_MODE_INIT, + EVAL_MODE_FRAME, + EVAL_MODE_NB +}; + +static const char *const var_names[] = { + "main_w", "W", ///< width of the main video + "main_h", "H", ///< height of the main video + "overlay_w", "w", ///< width of the overlay video + "overlay_h", "h", ///< height of the overlay video + "x", + "y", + "n", ///< number of frame + "pos", ///< position in the file + "t", ///< timestamp expressed in seconds + NULL +}; + /** * OverlayCUDAContext */ @@ -73,9 +109,14 @@ typedef struct OverlayCUDAContext { FFFrameSync fs; + int eval_mode; int x_position; int y_position; + double var_values[VAR_VARS_NB]; + char *x_expr, *y_expr; + + AVExpr *x_pexpr, *y_pexpr; } OverlayCUDAContext; /** @@ -89,6 +130,49 @@ static int format_is_supported(const enum AVPixelFormat formats[], enum AVPixelF return 0; } +static inline int normalize_xy(double d, int chroma_sub) +{ + if (isnan(d)) + return INT_MAX; + return (int)d & ~((1 << chroma_sub) - 1); +} + +static void eval_expr(AVFilterContext *ctx) +{ + OverlayCUDAContext *s = ctx->priv; + + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->var_values[VAR_Y] = av_expr_eval(s->y_pexpr, s->var_values, NULL); + /* It is necessary if x is expressed from y */ + s->var_values[VAR_X] = av_expr_eval(s->x_pexpr, s->var_values, NULL); + + s->x_position = normalize_xy(s->var_values[VAR_X], 1); + + /* the cuda pixel format is using hwaccel, y_position unnecessary normalize y */ + s->y_position = s->var_values[VAR_Y]; +} + +static int set_expr(AVExpr **pexpr, const char *expr, const char *option, void *log_ctx) +{ + int ret; + AVExpr *old = NULL; + + if (*pexpr) + old = *pexpr; + ret = av_expr_parse(pexpr, expr, var_names, + NULL, NULL, NULL, NULL, 0, log_ctx); + if (ret < 0) { + av_log(log_ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s' for %s\n", + expr, option); + *pexpr = old; + return ret; + } + + av_expr_free(old); + return 0; +} + /** * Helper checks if we can process main and overlay pixel formats */ @@ -151,10 +235,8 @@ static int overlay_cuda_blend(FFFrameSync *fs) CUcontext dummy, cuda_ctx = ctx->hwctx->cuda_ctx; AVFrame *input_main, *input_overlay; - const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); - int hsub = pix_desc->log2_chroma_w; - int vsub = pix_desc->log2_chroma_h; + int pos = 0; ctx->cu_ctx = cuda_ctx; @@ -183,8 +265,24 @@ static int overlay_cuda_blend(FFFrameSync *fs) return ret; } - ctx->x_position &= (1 << hsub) - 1; - ctx->y_position &= (1 << vsub) - 1; + if (ctx->eval_mode == EVAL_MODE_FRAME) { + pos = input_main->pkt_pos; + ctx->var_values[VAR_N] = inlink->frame_count_out; + ctx->var_values[VAR_T] = input_main->pts == AV_NOPTS_VALUE ? + NAN : input_main->pts * av_q2d(inlink->time_base); + ctx->var_values[VAR_POS] = pos == -1 ? NAN : pos; + ctx->var_values[VAR_OVERLAY_W] = ctx->var_values[VAR_OW] = input_overlay->width; + ctx->var_values[VAR_OVERLAY_H] = ctx->var_values[VAR_OH] = input_overlay->height; + ctx->var_values[VAR_MAIN_W ] = ctx->var_values[VAR_MW] = input_main->width; + ctx->var_values[VAR_MAIN_H ] = ctx->var_values[VAR_MH] = input_main->height; + + eval_expr(avctx); + + av_log(avctx, AV_LOG_DEBUG, "n:%f t:%f pos:%f x:%f xi:%d y:%f yi:%d\n", + ctx->var_values[VAR_N], ctx->var_values[VAR_T], ctx->var_values[VAR_POS], + ctx->var_values[VAR_X], ctx->x_position, + ctx->var_values[VAR_Y], ctx->y_position); + } // overlay first plane @@ -238,6 +336,39 @@ static int overlay_cuda_blend(FFFrameSync *fs) return ff_filter_frame(outlink, input_main); } +static int config_input_overlay(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + OverlayCUDAContext *s = inlink->dst->priv; + int ret; + + + /* Finish the configuration by evaluating the expressions + now when both inputs are configured. */ + s->var_values[VAR_MAIN_W ] = s->var_values[VAR_MW] = ctx->inputs[MAIN ]->w; + s->var_values[VAR_MAIN_H ] = s->var_values[VAR_MH] = ctx->inputs[MAIN ]->h; + s->var_values[VAR_OVERLAY_W] = s->var_values[VAR_OW] = ctx->inputs[OVERLAY]->w; + s->var_values[VAR_OVERLAY_H] = s->var_values[VAR_OH] = ctx->inputs[OVERLAY]->h; + s->var_values[VAR_X] = NAN; + s->var_values[VAR_Y] = NAN; + s->var_values[VAR_N] = 0; + s->var_values[VAR_T] = NAN; + s->var_values[VAR_POS] = NAN; + + if ((ret = set_expr(&s->x_pexpr, s->x_expr, "x", ctx)) < 0 || + (ret = set_expr(&s->y_pexpr, s->y_expr, "y", ctx)) < 0) + return ret; + + if (s->eval_mode == EVAL_MODE_INIT) { + eval_expr(ctx); + av_log(ctx, AV_LOG_VERBOSE, "x:%f xi:%d y:%f yi:%d\n", + s->var_values[VAR_X], s->x_position, + s->var_values[VAR_Y], s->y_position); + } + + return 0; +} + /** * Initialize overlay_cuda */ @@ -266,6 +397,8 @@ static av_cold void overlay_cuda_uninit(AVFilterContext *avctx) CHECK_CU(cu->cuCtxPopCurrent(&dummy)); } + av_expr_free(ctx->x_pexpr); ctx->x_pexpr = NULL; + av_expr_free(ctx->y_pexpr); ctx->y_pexpr = NULL; av_buffer_unref(&ctx->hw_device_ctx); ctx->hwctx = NULL; } @@ -405,16 +538,17 @@ static int overlay_cuda_config_output(AVFilterLink *outlink) #define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) static const AVOption overlay_cuda_options[] = { - { "x", "Overlay x position", - OFFSET(x_position), AV_OPT_TYPE_INT, { .i64 = 0 }, INT_MIN, INT_MAX, .flags = FLAGS }, - { "y", "Overlay y position", - OFFSET(y_position), AV_OPT_TYPE_INT, { .i64 = 0 }, INT_MIN, INT_MAX, .flags = FLAGS }, + { "x", "set the x expression of overlay", OFFSET(x_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, + { "y", "set the y expression of overlay", OFFSET(y_expr), AV_OPT_TYPE_STRING, {.str = "0"}, 0, 0, FLAGS }, { "eof_action", "Action to take when encountering EOF from secondary input ", OFFSET(fs.opt_eof_action), AV_OPT_TYPE_INT, { .i64 = EOF_ACTION_REPEAT }, EOF_ACTION_REPEAT, EOF_ACTION_PASS, .flags = FLAGS, "eof_action" }, { "repeat", "Repeat the previous frame.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_REPEAT }, .flags = FLAGS, "eof_action" }, { "endall", "End both streams.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_ENDALL }, .flags = FLAGS, "eof_action" }, { "pass", "Pass through the main input.", 0, AV_OPT_TYPE_CONST, { .i64 = EOF_ACTION_PASS }, .flags = FLAGS, "eof_action" }, + { "eval", "specify when to evaluate expressions", OFFSET(eval_mode), AV_OPT_TYPE_INT, {.i64 = EVAL_MODE_FRAME}, 0, EVAL_MODE_NB-1, FLAGS, "eval" }, + { "init", "eval expressions once during initialization", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_INIT}, .flags = FLAGS, .unit = "eval" }, + { "frame", "eval expressions per-frame", 0, AV_OPT_TYPE_CONST, {.i64=EVAL_MODE_FRAME}, .flags = FLAGS, .unit = "eval" }, { "shortest", "force termination when the shortest input terminates", OFFSET(fs.opt_shortest), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { "repeatlast", "repeat overlay of the last overlay frame", OFFSET(fs.opt_repeatlast), AV_OPT_TYPE_BOOL, {.i64=1}, 0, 1, FLAGS }, { NULL }, @@ -430,6 +564,7 @@ static const AVFilterPad overlay_cuda_inputs[] = { { .name = "overlay", .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input_overlay, }, { NULL } };