From patchwork Sun Nov 20 06:49:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 39357 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp1406913pzb; Sat, 19 Nov 2022 22:52:23 -0800 (PST) X-Google-Smtp-Source: AA0mqf5+l+/4yECZKmuTb/DPa/raSueeoofDce+Rur23y4aB8UMBoQAO02PjveE/ZE6SST4F6bY/ X-Received: by 2002:a05:6402:3893:b0:461:b033:90ac with SMTP id fd19-20020a056402389300b00461b03390acmr9972797edb.257.1668927143252; Sat, 19 Nov 2022 22:52:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668927143; cv=none; d=google.com; s=arc-20160816; b=asXtVylZrYp7zEHjjv2u2yxFzOyDCDqbvL8L//sksXZPbxu5SjRliF9kBuifgy8yvG k6xFU2/TEbM7I4YefnlMDPXHkThFsKdl2hu8ZWcEYUPq6MhUk4sH+Sc6inZWy0DmLYJQ EPeHgat1X/fO/kW1yu4MiQwn4HvcdurnywE2+xTqw9BxIE21QYcCcLI5uhWAmVcclLFy yBvhQCX3MDLkYtoYPckkW9MU163XFbmReZxMqx4tfcELVf1N66eulbcTUmQqekasWEun xQYZrnrzVBr/nB0UnPNLq5VS9x4wCCdSnY0a6mG4SWhrzu+KTp0Xym1t9MV8lkcqv4dH sNYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=FHyWQQgaIslgJ5AgkEOKh8JCBScWHiZ5ciMYe4D+31E=; b=DEprn5RUeSHlroDBBF6afCWCUg4FnFPBMnIWMuGzJKWo4YTAnyY33gFMaTZ3b836/C gaMIJmc5Ipqn39FtNge49dqyjRTt9+65V82ly+Nmmppn4SBSNFxD0fGFHGZM0Hlpjd5N ZWtU9orX9ybN4Lj5B7DDUgHRjHypx0W68GcmRgdVZZ0P56nN5sB/3tvFbllnlwtIECZ5 dNrZ0aDnDm8/ERg/h33XNnlIZQsIiGcBPsQOy6adEmpbcnOxBMfQLGIi6mE6WFU2Ot+D XitNceH8McolkvsBwusYKv++f4DfoZG5o1JkL74QF6jpVfBhfY/9bE+yajK9POrUQEpu E+BA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=iy9VAu1w; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d5-20020a056402400500b00456b734ceebsi6871442eda.436.2022.11.19.22.52.23; Sat, 19 Nov 2022 22:52:23 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=iy9VAu1w; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E64FC68BC82; Sun, 20 Nov 2022 08:50:56 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-58-211.mail.qq.com (out162-62-58-211.mail.qq.com [162.62.58.211]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5F06C68BA72 for ; Sun, 20 Nov 2022 08:50:42 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1668927040; bh=dnNzz9q6YqLn9b3O4NcQ4oHCejaoBxJeWn/2F+epSIg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=iy9VAu1wndQgo3b0UJSHcfAyJS2ZB8A6epiJyfkdNMC+ZNVcK4OZff8Pp3b9gLIJ1 vCPQ0o77lQOrYMplAtpOvyJqFM7JKiCIRfNIEzlO2saEGTlPWETBlpXptRpC/GI81f YQJEyEShVXu3dRTyWINZaiJ3kxI02MQJNap8iXL8= Received: from localhost.localdomain ([14.153.77.200]) by newxmesmtplogicsvrszc2-1.qq.com (NewEsmtp) with SMTP id C8F0626E; Sun, 20 Nov 2022 14:50:15 +0800 X-QQ-mid: xmsmtpt1668927039t32w5mggb Message-ID: X-QQ-XMAILINFO: NDgMZBR9sMma/4cFPIyQeoX2C+ERwwISY4oUXrkVyuOcWMk3uTG0Ara1TeCkPt BxUGMk6q5d4bpiP0ajnuXf092fwB/hRHE2dqmaW82rqVpi8QgIYjnRHzL0Jeq0Izbf7DIsXoU7qk 07VioJjAqYMfL/OBdY3SaAScxoxXtQdn+IOoIxzts/5Vi0mx8kNvGiq3hQVhUIh045igdQN/jDVu uWYEq+3lG+/YLrwiCFRXYR2X3+GyrUFCqVD/8DChy0GIZGCMCkwrw4p/MG9OBqvDjYooaOKWcn6c XPD+mPxLA1FfYW3bc3y5LLZr+vJucDVdBYPtE4zMAG5bq7tLvXS/0iabQetEEwBmFLcGLuG7TnKk eOeYjDzR71xbvfOkvKO/VJVhtap2SskUnZGdk64FyLUkGaT/Zqjp/LIRmsYRwT+IJ70+D7jEtv4m 0zewisXlOrZaHrziegWghpIZDBzyik+1qnJyLqsvRwRFQrxAHzqUy/XTBQ/SZEwBwQ40FqulWK5d yCHsRnSx/R9bZQL1G7VDeDyAf18UVOUIMKORWzPgy1tdJosDG10zkcR59tofPkM8/VqJ5lMP7Lkk YWIZL3sNPfzwU0yD2b9tjX9kt5hK/vS+KIdjrDKVyBkLwnRyjrkeNt7RFqPqcHQaxbDw7qQXtOGq nATnkJXdwg9sb8prRe1I28opdimV9dPDASzzujWiHyy87di1mV48il05cTA+2Kv+KpTSeThh6WQz /xJTeA1wY9PCuYijeGnv4PlGPRdqE1sQK/hEQj9+dzlvEKQDJUZyoMyskQhWUTUE2h3G3pclT0c4 5UtzwQZGfzg1lCLHe0jGO1Ce3bgflJNn4tvr8Cm73jNXMJvIEdYwqKoC+7iLJO86YSEpXxHlog/c OFcjtSMqqn3oX2QsHXFms8t7NJlhvZHOXz3sZdaPsTVVfpg7MOdj/AHP6BDZXssmcMPSj+9ijj+f ljPh4ewWxtR6vP3BAQBYNViYtYw8XrnBpS+OswaVXzM+ARbScry4jLUVrobll/UKNfTJxo1j0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 20 Nov 2022 14:49:44 +0800 X-OQ-MSGID: <20221120064947.94540-10-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221120064947.94540-1-quinkblack@foxmail.com> References: <20221120064947.94540-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 09/12] avcodec/mediacodec: add ANativeWindow support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: matthieu.bouron@stupeflix.com, Zhao Zhili , aman@tmm1.net Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 3NQxDeU+C2sl From: Zhao Zhili ANativeWindow can be used without JVM. --- libavcodec/mediacodec_surface.c | 46 ++++++++++++++++++++++++------- libavcodec/mediacodec_surface.h | 8 ++++-- libavcodec/mediacodec_wrapper.c | 30 +++++++++++++++----- libavcodec/mediacodec_wrapper.h | 8 ++++-- libavcodec/mediacodecdec_common.c | 4 +-- 5 files changed, 72 insertions(+), 24 deletions(-) diff --git a/libavcodec/mediacodec_surface.c b/libavcodec/mediacodec_surface.c index 09a42295d2..ef41cdafa7 100644 --- a/libavcodec/mediacodec_surface.c +++ b/libavcodec/mediacodec_surface.c @@ -20,33 +20,59 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include #include +#include "libavutil/mem.h" #include "ffjni.h" #include "mediacodec_surface.h" -FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *log_ctx) +FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *native_window, void *log_ctx) { - JNIEnv *env = NULL; + FFANativeWindow *ret; - env = ff_jni_get_env(log_ctx); - if (!env) { + ret = av_mallocz(sizeof(*ret)); + if (!ret) return NULL; + + if (surface) { + JNIEnv *env = NULL; + + env = ff_jni_get_env(log_ctx); + if (env) + ret->surface = (*env)->NewGlobalRef(env, surface); + } + + if (native_window) { + ANativeWindow_acquire(native_window); + ret->native_window = native_window; } - return (*env)->NewGlobalRef(env, surface); + if (!ret->surface && !ret->native_window) { + av_log(log_ctx, AV_LOG_ERROR, "Both surface and native_window are NULL\n"); + av_freep(&ret); + } + + return ret; } int ff_mediacodec_surface_unref(FFANativeWindow *window, void *log_ctx) { - JNIEnv *env = NULL; + if (!window) + return 0; - env = ff_jni_get_env(log_ctx); - if (!env) { - return AVERROR_EXTERNAL; + if (window->surface) { + JNIEnv *env = NULL; + + env = ff_jni_get_env(log_ctx); + if (env) + (*env)->DeleteGlobalRef(env, window->surface); } - (*env)->DeleteGlobalRef(env, window); + if (window->native_window) + ANativeWindow_release(window->native_window); + + av_free(window); return 0; } diff --git a/libavcodec/mediacodec_surface.h b/libavcodec/mediacodec_surface.h index 933dc2bf51..e2ac1c9057 100644 --- a/libavcodec/mediacodec_surface.h +++ b/libavcodec/mediacodec_surface.h @@ -25,10 +25,12 @@ #include "libavcodec/avcodec.h" -struct FFANativeWindow; -typedef struct FFANativeWindow FFANativeWindow; +typedef struct FFANativeWindow { + void *surface; + void *native_window; +} FFANativeWindow; -FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *log_ctx); +FFANativeWindow *ff_mediacodec_surface_ref(void *surface, void *native_window, void *log_ctx); int ff_mediacodec_surface_unref(FFANativeWindow *window, void *log_ctx); #endif /* AVCODEC_MEDIACODEC_SURFACE_H */ diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c index b12aced711..284d615980 100644 --- a/libavcodec/mediacodec_wrapper.c +++ b/libavcodec/mediacodec_wrapper.c @@ -1371,12 +1371,17 @@ fail: return ret; } -static int mediacodec_jni_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags) +static int mediacodec_jni_configure(FFAMediaCodec *ctx, + const FFAMediaFormat* format_ctx, + FFANativeWindow* window, + void *crypto, + uint32_t flags) { int ret = 0; JNIEnv *env = NULL; FFAMediaCodecJni *codec = (FFAMediaCodecJni *)ctx; const FFAMediaFormatJni *format = (FFAMediaFormatJni *)format_ctx; + jobject *surface = window ? window->surface : NULL; JNI_GET_ENV_OR_RETURN(env, codec, AVERROR_EXTERNAL); @@ -2151,16 +2156,27 @@ static int mediacodec_ndk_delete(FFAMediaCodec* ctx) return ret; } -static int mediacodec_ndk_configure(FFAMediaCodec* ctx, const FFAMediaFormat* format_ctx, void* surface, void *crypto, uint32_t flags) +static int mediacodec_ndk_configure(FFAMediaCodec* ctx, + const FFAMediaFormat* format_ctx, + FFANativeWindow* window, + void *crypto, + uint32_t flags) { FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx; FFAMediaFormatNdk *format = (FFAMediaFormatNdk *)format_ctx; media_status_t status; + ANativeWindow *native_window = NULL; - if (surface) { - JNIEnv *env = NULL; - JNI_GET_ENV_OR_RETURN(env, ctx, -1); - codec->window = ANativeWindow_fromSurface(env, surface); + if (window) { + if (window->surface) { + JNIEnv *env = NULL; + JNI_GET_ENV_OR_RETURN(env, ctx, -1); + native_window = ANativeWindow_fromSurface(env, window->surface); + // Save for release + codec->window = native_window; + } else if (window->native_window) { + native_window = window->native_window; + } } if (format_ctx->class != &amediaformat_ndk_class) { @@ -2168,7 +2184,7 @@ static int mediacodec_ndk_configure(FFAMediaCodec* ctx, const FFAMediaFormat* fo return AVERROR(EINVAL); } - status = codec->configure(codec->impl, format->impl, codec->window, NULL, flags); + status = codec->configure(codec->impl, format->impl, native_window, NULL, flags); if (status != AMEDIA_OK) { av_log(codec, AV_LOG_ERROR, "configure failed, %d\n", status); return AVERROR_EXTERNAL; diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h index b9b882f243..7cf3f4aecd 100644 --- a/libavcodec/mediacodec_wrapper.h +++ b/libavcodec/mediacodec_wrapper.h @@ -27,6 +27,7 @@ #include #include "avcodec.h" +#include "mediacodec_surface.h" /** * The following API around MediaCodec and MediaFormat is based on the @@ -163,7 +164,7 @@ struct FFAMediaCodec { FFAMediaCodec* (*createEncoderByType)(const char *mime_type); int (*delete)(FFAMediaCodec* codec); - int (*configure)(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags); + int (*configure)(FFAMediaCodec* codec, const FFAMediaFormat* format, FFANativeWindow* surface, void *crypto, uint32_t flags); int (*start)(FFAMediaCodec* codec); int (*stop)(FFAMediaCodec* codec); int (*flush)(FFAMediaCodec* codec); @@ -202,7 +203,10 @@ FFAMediaCodec* ff_AMediaCodec_createCodecByName(const char *name, int ndk); FFAMediaCodec* ff_AMediaCodec_createDecoderByType(const char *mime_type, int ndk); FFAMediaCodec* ff_AMediaCodec_createEncoderByType(const char *mime_type, int ndk); -static inline int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format, void* surface, void *crypto, uint32_t flags) +static inline int ff_AMediaCodec_configure(FFAMediaCodec *codec, + const FFAMediaFormat *format, + FFANativeWindow *surface, + void *crypto, uint32_t flags) { return codec->configure(codec, format, surface, crypto, flags); } diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c index ae82efaa0a..80089439ea 100644 --- a/libavcodec/mediacodecdec_common.c +++ b/libavcodec/mediacodecdec_common.c @@ -589,14 +589,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, if (device_ctx->type == AV_HWDEVICE_TYPE_MEDIACODEC) { if (device_ctx->hwctx) { AVMediaCodecDeviceContext *mediacodec_ctx = (AVMediaCodecDeviceContext *)device_ctx->hwctx; - s->surface = ff_mediacodec_surface_ref(mediacodec_ctx->surface, avctx); + s->surface = ff_mediacodec_surface_ref(mediacodec_ctx->surface, mediacodec_ctx->native_window, avctx); av_log(avctx, AV_LOG_INFO, "Using surface %p\n", s->surface); } } } if (!s->surface && user_ctx && user_ctx->surface) { - s->surface = ff_mediacodec_surface_ref(user_ctx->surface, avctx); + s->surface = ff_mediacodec_surface_ref(user_ctx->surface, NULL, avctx); av_log(avctx, AV_LOG_INFO, "Using surface %p\n", s->surface); } }