From patchwork Mon Oct 24 03:16:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 38947 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4a86:b0:9d:28a3:170e with SMTP id fn6csp1998089pzb; Sun, 23 Oct 2022 20:18:55 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7voeIXuwfdxkz7GCG1zAGQqKk/4lM1iNSh9I0Ic5pg//fgjJ/JsA6huYISmG2nW32mxHEI X-Received: by 2002:a05:6402:5107:b0:45d:409:1337 with SMTP id m7-20020a056402510700b0045d04091337mr29573511edd.185.1666581535235; Sun, 23 Oct 2022 20:18:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666581535; cv=none; d=google.com; s=arc-20160816; b=GLMQP+gr0tjKMV4IbuCScFEoH7NzWgff/cedjYqyVzrPYyC8XsOCeHa+tGqkUTv8e9 b2HyBe5t5aLl9elBIR8qfQgZeLBBoCwXQ6OylkS1yIRmIBibufDe7/AZtIlmdtLwsvs+ jXqnGeH98zWGZSUPeJ0sS/LUdM55bocSdDanoT8I9VEcK0kL0N5KSFKpvEktsUnWbfbZ rEUu4UF3hx9dxgb80CliRXwFi0jL6wN+Ls26J20IHrDRdMtxH6u6dobHFL2yq+YlLLDT +625DcDC3dMMF9fxWyDvMNVhcCYNmEthukCVnRVa7Y6m2o5wBb1+6CHEoXCx7KjtZBve 7tqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=C9JEYdJVz+lcQ7GjHNHB/wBO2ttb9BfcOKpvmT04GUQ=; b=OmfxcZJygAdCwt1Cu5tyEQk3nnMymiMarCb6VON01OtJ/ClYUFFfCWuppyydIpAc5y Y00h7UwBt7eL6Q5FZrbxLxdpSkkhR4EfejiDWGULn3QyRatbL3zuq+NJffSsjHEOzPnA fFTOFSAKPwgQtL6RzcwLcYMgrNWqfMKqwzqfO+HtolIIYmtUTTh1NE5DwiTAZowTtoDX kVhKzjJ4Su4EdeyLfkYgHOG63r9DLjAxJvgce+m0e5G6oVSQPFtGsuk0npZ/5XeTn2xZ CYHwsUxNH99EgStPkSthG2GSVBNLvg4Anl54v3WzFPq9Cr6ekKZsVsqaQnOH+PmSAeRr T+/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="TuDt0r/Z"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hd43-20020a17090796ab00b0079fe11e92ddsi6945731ejc.906.2022.10.23.20.18.54; Sun, 23 Oct 2022 20:18:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="TuDt0r/Z"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0AE8268BDAC; Mon, 24 Oct 2022 06:18:42 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-58-216.mail.qq.com (out162-62-58-216.mail.qq.com [162.62.58.216]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A8E1E68BDA4 for ; Mon, 24 Oct 2022 06:18:35 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1666581512; bh=kRsGIYX5HznuLWEGDPSIMmGJQO4i46DklQ/meCPAY2U=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=TuDt0r/Z4FSFLJ/cwxOgzDJHKCUohQx5BROnN+Za1NjSEtHZuLbPWwK9OhGFXDHRi ZhOPIWZqY2hpaPQw0RDCfX0Dc2ucfyIicmciaWOQt1ZsKoVps/Dhkc+UmbsOSk43vJ wFVy1Fp8rRucG8sXqyF72eNbbWnqlt0Iz8MTiS4Y= Received: from localhost.localdomain ([59.37.125.102]) by newxmesmtplogicsvrsza31.qq.com (NewEsmtp) with SMTP id 44A14270; Mon, 24 Oct 2022 11:17:10 +0800 X-QQ-mid: xmsmtpt1666581441tflbousg8 Message-ID: X-QQ-XMAILINFO: NQR8mRxMnur9/ThzoXtse/E97HdkXTsm+svZQwNHqrKg+NFuKa2O3LXgCstF0f RmKrwzZfgwB8oK0iYioKCdXNQSraEYTF5VzKPxpvS/vhqlu8vaNq1Oa9hMjA7iTkTR8OxLCiLcbp pr6qsCWExkauB2CDyNHb12n7xXeXdmW03q00BmkNF69XKpNuZ+2UGfqRwZmBggJu77ETv/Po90fx znO7s2ZqVa66zX8Gy02aSwQcKtyas18Z8apFrD2+crySfXUy+IFKqqxSp/OspydbyyuVy9ZfOURf yeVcqEX0i2nI65FacNzXLAHrRCYqvJEa5SQTFRLo7VkJDUJdU/YcCMJG0+XWPKlTsJHF1Bt93vhV YMzFguknd9QSOgnW85G8zD6ltQdz+ncCxLi3gLE2fdSuC92qAya2ipwhpYUrCoirpJmu/BOE6scA 1/HCFAxeHT5z21vcLObL5MPyXe0dO3Q8PyocJAUeqyGeybMkPuvo+MlCmVssxmcfU134mgnEQWir 7N7Divb0+iquDIFtA2OFsDXJtZVS+7/lY5rakzf2zjqxGnyuo1G5/BXavXH+fkgU2n5BnaZXWBEN J5Ga5Uj/8GWRO4HkFLGFh2buHtIZHEQHR5VFxZ0XaWAFhzktyzwAoQfobpJ05b7jDV+bO8ZkSyI/ fB4K8Wl1A9F4gVMkftybFEy09TZWm9eFGeZ/+I7+c7LDA6X47sPR0io3teFoaXY3fna4EaybpjkB BN6EMV0GcZijp+P89KCQaJPbG5UQ9bEUSYkqI1IlMu8032wF5yP7Z+U4vmS7U3lcprtfkyk9zC1T Tk5HI09grpaDozue2MBJ2fps6j8Hr5SeFDim1gw1KpqBQqI+bPo9FLuyMNL2payikGuNQ34Cutqp g6UGSG2QtsXAfx5N1fTVbheqMKaaq0RBnU/xrHjKOBXwT7C84IbrXDpb6R5snkCBIfd2OdGX+czr zsXq8GZYFo1mgpFf0T4ZSCMbo4DeIUAjFE5zredULqt4mzZGkrIAmAU66DPqoJ From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Mon, 24 Oct 2022 11:16:55 +0800 X-OQ-MSGID: <20221024031658.1026118-10-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221024031658.1026118-1-quinkblack@foxmail.com> References: <20221024031658.1026118-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 09/12] avcodec/mediacodec: add ANativeWindow support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: matthieu.bouron@stupeflix.com, Zhao Zhili , aman@tmm1.net Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: zbX9pxQXtONr From: Zhao Zhili ANativeWindow can be used without JVM. --- libavcodec/mediacodec_surface.c | 46 ++++++++++++++++++++++++------- libavcodec/mediacodec_surface.h | 8 ++++-- libavcodec/mediacodec_wrapper.c | 30 +++++++++++++++----- libavcodec/mediacodec_wrapper.h | 8 ++++-- libavcodec/mediacodecdec_common.c | 4 +-- 5 files changed, 72 insertions(+), 24 deletions(-) diff --git a/libavcodec/mediacodec_surface.c b/libavcodec/mediacodec_surface.c index 09a42295d2..ef41cdafa7 100644 --- a/libavcodec/mediacodec_surface.c +++ b/libavcodec/mediacodec_surface.c @@ -20,33 +20,59 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include #include +#include "libavutil/mem.h" #include "ffjni.h" #include "mediacodec_surface.h" -FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *log_ctx) +FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *native_window, void *log_ctx) { - JNIEnv *env = NULL; + FFANativeWindow *ret; - env = ff_jni_get_env(log_ctx); - if (!env) { + ret = av_mallocz(sizeof(*ret)); + if (!ret) return NULL; + + if (surface) { + JNIEnv *env = NULL; + + env = ff_jni_get_env(log_ctx); + if (env) + ret->surface = (*env)->NewGlobalRef(env, surface); + } + + if (native_window) { + ANativeWindow_acquire(native_window); + ret->native_window = native_window; } - return (*env)->NewGlobalRef(env, surface); + if (!ret->surface && !ret->native_window) { + av_log(log_ctx, AV_LOG_ERROR, "Both surface and native_window are NULL\n"); + av_freep(&ret); + } + + return ret; } int ff_mediacodec_surface_unref(FFANativeWindow *window, void *log_ctx) { - JNIEnv *env = NULL; + if (!window) + return 0; - env = ff_jni_get_env(log_ctx); - if (!env) { - return AVERROR_EXTERNAL; + if (window->surface) { + JNIEnv *env = NULL; + + env = ff_jni_get_env(log_ctx); + if (env) + (*env)->DeleteGlobalRef(env, window->surface); } - (*env)->DeleteGlobalRef(env, window); + if (window->native_window) + ANativeWindow_release(window->native_window); + + av_free(window); return 0; } diff --git a/libavcodec/mediacodec_surface.h b/libavcodec/mediacodec_surface.h index 933dc2bf51..e2ac1c9057 100644 --- a/libavcodec/mediacodec_surface.h +++ b/libavcodec/mediacodec_surface.h @@ -25,10 +25,12 @@ #include "libavcodec/avcodec.h" -struct FFANativeWindow; -typedef struct FFANativeWindow FFANativeWindow; +typedef struct FFANativeWindow { + void *surface; + void *native_window; +} FFANativeWindow; -FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *log_ctx); +FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *native_window, void *log_ctx); int ff_mediacodec_surface_unref(FFANativeWindow *window, void *log_ctx); #endif /* AVCODEC_MEDIACODEC_SURFACE_H */ diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c index b12aced711..284d615980 100644 --- a/libavcodec/mediacodec_wrapper.c +++ b/libavcodec/mediacodec_wrapper.c @@ -1371,12 +1371,17 @@ fail: return ret; } -static int mediacodec_jni_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags) +static int mediacodec_jni_configure(FFAMediaCodec *ctx, + const FFAMediaFormat* format_ctx, + FFANativeWindow* window, + void *crypto, + uint32_t flags) { int ret = 0; JNIEnv *env = NULL; FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx; const FFAMediaFormatJni *format = (FFAMediaFormatJni *)format_ctx; + jobject *surface = window ? window->surface : NULL; JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL); @@ -2151,16 +2156,27 @@ static int mediacodec_ndk_delete(FFAMediaCodec* ctx) return ret; } -static int mediacodec_ndk_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags) +static int mediacodec_ndk_configure(FFAMediaCodec* ctx, + const FFAMediaFormat* format_ctx, + FFANativeWindow* window, + void *crypto, + uint32_t flags) { FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx; FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)format_ctx; media_status_t status; + ANativeWindow *native_window = NULL; - if (surface) { - JNIEnv *env = NULL; - JNI_GET_ENV_OR_RETURN(env, ctx, -1); - codec->window = ANativeWindow_fromSurface(env, surface); + if (window) { + if (window->surface) { + JNIEnv *env = NULL; + JNI_GET_ENV_OR_RETURN(env, ctx, -1); + native_window = ANativeWindow_fromSurface(env, window->surface); + // Save for release + codec->window = native_window; + } else if (window->native_window) { + native_window = window->native_window; + } } if (format_ctx->class != &amediaformat_ndk_class) { @@ -2168,7 +2184,7 @@ static int mediacodec_ndk_configure(FFAMediaCodec* ctx, const FFAMediaFormat* fo return AVERROR(EINVAL); } - status = codec->configure(codec->impl, format->impl, codec->window, NULL, flags); + status = codec->configure(codec->impl, format->impl, native_window, NULL, flags); if (status != AMEDIA_OK) { av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status); return AVERROR_EXTERNAL; diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h index b9b882f243..7cf3f4aecd 100644 --- a/libavcodec/mediacodec_wrapper.h +++ b/libavcodec/mediacodec_wrapper.h @@ -27,6 +27,7 @@ #include #include "avcodec.h" +#include "mediacodec_surface.h" /** * The following API around MediaCodec and MediaFormat is based on the @@ -163,7 +164,7 @@ struct FFAMediaCodec { FFAMediaCodec* (*createEncoderByType)(const char *mime_type); int (*delete)(FFAMediaCodec* codec); - int (*configure)(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags); + int (*configure)(FFAMediaCodec* codec, const FFAMediaFormat* format, FFANativeWindow* surface, void *crypto, uint32_t flags); int (*start)(FFAMediaCodec* codec); int (*stop)(FFAMediaCodec* codec); int (*flush)(FFAMediaCodec* codec); @@ -202,7 +203,10 @@ FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name, int ndk); FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type, int ndk); FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type, int ndk); -static inline int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags) +static inline int ff_AMediaCodec_configure(FFAMediaCodec *codec, + const FFAMediaFormat *format, + FFANativeWindow *surface, + void *crypto, uint32_t flags) { return codec->configure(codec, format, surface, crypto, flags); } diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c index c3d5988063..d56412aa6d 100644 --- a/libavcodec/mediacodecdec_common.c +++ b/libavcodec/mediacodecdec_common.c @@ -582,14 +582,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, if (device_ctx->type == AV_HWDEVICE_TYPE_MEDIACODEC) { if (device_ctx->hwctx) { AVMediaCodecDeviceContext *mediacodec_ctx = (AVMediaCodecDeviceContext *)device_ctx->hwctx; - s->surface = ff_mediacodec_surface_ref(mediacodec_ctx->surface, avctx); + s->surface = ff_mediacodec_surface_ref(mediacodec_ctx->surface, mediacodec_ctx->native_window, avctx); av_log(avctx, AV_LOG_INFO, "Using surface %p\n", s->surface); } } } if (!s->surface && user_ctx && user_ctx->surface) { - s->surface = ff_mediacodec_surface_ref(user_ctx->surface, avctx); + s->surface = ff_mediacodec_surface_ref(user_ctx->surface, NULL, avctx); av_log(avctx, AV_LOG_INFO, "Using surface %p\n", s->surface); } }