From patchwork Wed Nov 6 12:31:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 52615 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:612c:2fe5:b0:4a6:1c7:11b7 with SMTP id kw5csp617819vqb; Wed, 6 Nov 2024 04:39:09 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCVvPr2FuP3k8Eme7/bgmc2Vh1ECbKRlvW9NQycI3fqzdCwUz41sJOmZKPe2TS7WS7+tDK7/FN6PpH1HoXSAUYmp@gmail.com X-Google-Smtp-Source: AGHT+IEKwppRZ6z6OCdMrhRGbQ86Y6OzzPzhy/yBaG8Unc0J1A23/XZ2HCUXstsjsxmEMFlhyiMv X-Received: by 2002:a05:6512:12c5:b0:539:e88f:23a1 with SMTP id 2adb3069b0e04-53b3491c80emr20968640e87.44.1730896749396; Wed, 06 Nov 2024 04:39:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1730896749; cv=none; d=google.com; s=arc-20240605; b=hbhUl/XlGC3a+1fbAIYnoVX7k6or1rhW0CA+Pk8BUSzoadhc01EvEZlIBnE/H0vhZk ihMWxR+Z4LkCYIDL5u86pSZcGOJRZ7K1/8WU1Qsk4+H/Vfo9DpTzWLKMkRfmzgOwWLD+ p9kGzGnyElhYynrJtQ8vJRzONPnnNZSTbCA1Z6zdy6IOFElNzVfXisF8XZS033Bw1PcP YfwYZqJm52Ly58UQQ4dC6WYswwwg/mqzfMOjn13Yg04ertwj2pg26QQfAq4UWqFrrUzO fH9iODJYUW9wR1U+vBKkaAIxvJSLvacP1gGwmnaLLb8QrkHPt+fvdjqJmKazJQYC130q NZbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:from:message-id :dkim-signature:delivered-to; bh=bpWfqsoVFCs+V7f6kNAY5X3exiJx4IWX1wop4bmKNFA=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=dr3Tj2CO6miDp36TD0pRMMu6T3YT2J3Zsfn084iO8wNZUko5Xpua5zr+UWF7rO5W+W TIeSKIJWb7wqy/U7oyJqWgdwJMmD4t40P9Ouo8a0YcHs0q7+fyW3o3+6EEMBLSWhbwaN AjJC2mTsba4Eoq2spBFfWK6Qry5khJ0P2x+D844yqp3P4NJv5OmKPUW0cQuBySZuJM4c xTem1euQ70Zwi9tU1OSd9zguno53yrJw69drWVOmZ0w3RBS/bKf+EgdolE14V5IuGl3G B/fjVi++oPTSZu3G4g4KcgNkUG5EyXl7LnQBn2tnDh98L97L0Od+6lctah9L88LONMCD 3AFw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=w12tpI62; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-53c7bdcb8d4si5156910e87.374.2024.11.06.04.39.09; Wed, 06 Nov 2024 04:39:09 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=w12tpI62; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 46F3668DDF0; Wed, 6 Nov 2024 14:31:55 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from xmbghk7.mail.qq.com (xmbghk7.mail.qq.com [43.163.128.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1561868DDAE for ; Wed, 6 Nov 2024 14:31:41 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1730896291; bh=VVtXP6IVO3RbFdhKIPSsJqCzP3oH3vycc/RmQsR+r1U=; h=From:To:Cc:Subject:Date; b=w12tpI62CWmrLgED3vAirsh3TP57fAcUu0Nujp8ALqr0y4fJvrloysS0yCcQUV20F TRsB3Qhfl6dpDFAhksVGiUVmhL0WMRU42QDBxWRp669dc/cDnhMkTG+MhSqtD2OWOZ 7POuS3LNssyAMLX01+Z82nH+FIrXYcCsFEsp3WQ0= Received: from ZHILIZHAO-MB1.tencent.com ([59.37.125.101]) by newxmesmtplogicsvrsza36-0.qq.com (NewEsmtp) with SMTP id 7DDA0022; Wed, 06 Nov 2024 20:31:29 +0800 X-QQ-mid: xmsmtpt1730896289th3wuoq9s Message-ID: X-QQ-XMAILINFO: MmpliBmRb3iC3IwL82etZq2OnGLTd3cZddjL9C0zpWDII904DlV8XKkkgjj4VN GZpGZmui2Qmo7vMHqUgqPzVhdPg/LW7rpghUXjSfFspwOon+W4vIvLtazrpXB/HAdP2RsgBiSLOk g42VRIiPNqlrKUgui3kEX0hWxsYVyfny3cNfpxw1KflOUSmo/k9hMJWMKaEgmBJEtZFf08CJDx5S Dz382E0mdW7GDQvMTjHeXz0BOxKxFVG/g4HeuKdk8WvWMIhj/Tp+lXikDoL9qqDwqyPPLn5GrUrb n5IxEvIZMeBMTrG7GO5yJhluwCax4CY8WAgBEYpq1jiAF4DZkRn3GTTpdEdUcAc5fqVv0aaz84U0 pg+gld5iGj2f/ZCWzWBO+E3cnzYagObLYbs3RFBJmL80t14CskR+Ntowkd1/4GOX2uYHaFgYTXcC xRVszgjWoBE1hEPhdY+7IO6cUioUW3/nEfEq9XZcG1CVLPetm+tE3Ne0p1nQus3t5P3n5KSGfNS+ NksO8WBhA2jHkPRsIJHpdKimSrmlXtdYsj0dRk8nv6AfLzn3YuOXKQWlxbvF2KXFhYr8uhlh5VY9 TPy0IolLWQbTd740HyZHvX8d09OSWzzXjFM3dW9A0MIvlzyhoya5lxD1ClAYzI/wkCBPui94bg6T eCf9pfj9WFOKoGekg7U2NRxD6fp28jRyEyzoT6cghdR4Mas4NQRrAg5naJ9OsODLd8AOY8r0DrkM e3TtCs419gLRsoOTwNsM4n50MpVFr1l+a0YOa9hcv/ps+yRdbqkK6oUfhrFfnjXOlFQajcf+q9vY U53YxEXFHUUSqc/Mmyw+ixHefQ10zAHjFY1143KHJC103A1Pi8hQtzEEEHad1JDZoi0bep7/yn7y aT88iGoUCWBg15QY3UE4083I/S3urt7SZGT2LUh2BRQY3faeLb7gKBB6B+94/mgXcBF01uRFuv08 YdQyiLmNuzLgi/Owyuq3+cWOlT4OQVOoIzCRVE9l58f158TmMYVw== X-QQ-XMRINFO: NI4Ajvh11aEj8Xl/2s1/T8w= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Wed, 6 Nov 2024 20:31:28 +0800 X-OQ-MSGID: <20241106123128.17352-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.46.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/4] avcodec/mediacodec_wrapper: add async mode support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OkoOa5sOI0C8 From: Zhao Zhili --- libavcodec/mediacodec_wrapper.c | 130 ++++++++++++++++++++++++++++++++ libavcodec/mediacodec_wrapper.h | 28 +++++++ 2 files changed, 158 insertions(+) diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c index 96c886666a..283bbe72d6 100644 --- a/libavcodec/mediacodec_wrapper.c +++ b/libavcodec/mediacodec_wrapper.c @@ -1762,6 +1762,14 @@ static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx) return 0; } +static int mediacodec_jni_setAsyncNotifyCallback(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + av_log(codec, AV_LOG_ERROR, "Doesn't support aync mode with JNI, please try ndk_codec=1\n"); + return AVERROR(ENOSYS); +} + static const FFAMediaFormat media_format_jni = { .class = &amediaformat_class, @@ -1821,6 +1829,7 @@ static const FFAMediaCodec media_codec_jni = { .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode, .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers, .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream, + .setAsyncNotifyCallback = mediacodec_jni_setAsyncNotifyCallback, }; typedef struct FFAMediaFormatNdk { @@ -1842,6 +1851,9 @@ typedef struct FFAMediaCodecNdk { AMediaCodec *impl; ANativeWindow *window; + FFAMediaCodecOnAsyncNotifyCallback async_cb; + void *async_userdata; + // Available since API level 28. media_status_t (*getName)(AMediaCodec*, char** out_name); void (*releaseName)(AMediaCodec*, char* name); @@ -1849,6 +1861,8 @@ typedef struct FFAMediaCodecNdk { // Available since API level 26. media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *); media_status_t (*signalEndOfInputStream)(AMediaCodec *); + media_status_t (*setAsyncNotifyCallback)(AMediaCodec *, + struct AMediaCodecOnAsyncNotifyCallback callback, void *userdata); } FFAMediaCodecNdk; static const FFAMediaFormat media_format_ndk; @@ -1866,6 +1880,32 @@ static const AVClass amediacodec_ndk_class = { .version = LIBAVUTIL_VERSION_INT, }; +static int media_status_to_error(media_status_t status) +{ + switch (status) { + case AMEDIA_OK: + return 0; + case AMEDIACODEC_ERROR_INSUFFICIENT_RESOURCE: + return AVERROR(ENOMEM); + case AMEDIA_ERROR_MALFORMED: + return AVERROR_INVALIDDATA; + case AMEDIA_ERROR_UNSUPPORTED: + return AVERROR(ENOTSUP); + case AMEDIA_ERROR_INVALID_PARAMETER: + return AVERROR(EINVAL); + case AMEDIA_ERROR_INVALID_OPERATION: + return AVERROR(EOPNOTSUPP); + case AMEDIA_ERROR_END_OF_STREAM: + return AVERROR_EOF; + case AMEDIA_ERROR_IO: + return AVERROR(EIO); + case AMEDIA_ERROR_WOULD_BLOCK: + return AVERROR(EWOULDBLOCK); + default: + return AVERROR_EXTERNAL; + } +} + static FFAMediaFormat *mediaformat_ndk_create(AMediaFormat *impl) { FFAMediaFormatNdk *format = av_mallocz(sizeof(*format)); @@ -2060,6 +2100,7 @@ static inline FFAMediaCodec *ndk_codec_create(int method, const char *arg) { GET_SYMBOL(setInputSurface) GET_SYMBOL(signalEndOfInputStream) + GET_SYMBOL(setAsyncNotifyCallback) #undef GET_SYMBOL @@ -2335,6 +2376,94 @@ static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx) return 0; } +static void mediacodec_ndk_onInputAvailable(AMediaCodec *impl, void *userdata, + int32_t index) +{ + FFAMediaCodecNdk *codec = userdata; + codec->async_cb.onAsyncInputAvailable((FFAMediaCodec *) codec, + codec->async_userdata, index); +} + +static void mediacodec_ndk_onOutputAvailable(AMediaCodec *impl, + void *userdata, + int32_t index, + AMediaCodecBufferInfo *buffer_info) +{ + FFAMediaCodecNdk *codec = userdata; + FFAMediaCodecBufferInfo info = { + .offset = buffer_info->offset, + .size = buffer_info->size, + .presentationTimeUs = buffer_info->presentationTimeUs, + .flags = buffer_info->flags, + }; + + codec->async_cb.onAsyncOutputAvailable(&codec->api, codec->async_userdata, + index, &info); +} + +static void mediacodec_ndk_onFormatChanged(AMediaCodec *impl, void *userdata, + AMediaFormat *format) +{ + FFAMediaCodecNdk *codec = userdata; + FFAMediaFormat *media_format = mediaformat_ndk_create(format); + if (!media_format) + return; + + codec->async_cb.onAsyncFormatChanged(&codec->api, codec->async_userdata, + media_format); + ff_AMediaFormat_delete(media_format); +} + +static void mediacodec_ndk_onError(AMediaCodec *impl, void *userdata, + media_status_t status, + int32_t actionCode, + const char *detail) +{ + FFAMediaCodecNdk *codec = userdata; + int error = media_status_to_error(status); + + codec->async_cb.onAsyncError(&codec->api, codec->async_userdata, error, + detail); +} + +static int mediacodec_ndk_setAsyncNotifyCallback(FFAMediaCodec *ctx, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx; + struct AMediaCodecOnAsyncNotifyCallback cb = { + .onAsyncInputAvailable = mediacodec_ndk_onInputAvailable, + .onAsyncOutputAvailable = mediacodec_ndk_onOutputAvailable, + .onAsyncFormatChanged = mediacodec_ndk_onFormatChanged, + .onAsyncError = mediacodec_ndk_onError, + }; + media_status_t status; + + if (!codec->setAsyncNotifyCallback) { + av_log(codec, AV_LOG_ERROR, "setAsyncNotifyCallback unavailable\n"); + return AVERROR(ENOSYS); + } + + if (!callback || + !callback->onAsyncInputAvailable || + !callback->onAsyncOutputAvailable || + !callback->onAsyncFormatChanged || + !callback->onAsyncError) + return AVERROR(EINVAL); + + codec->async_cb = *callback; + codec->async_userdata = userdata; + + status = codec->setAsyncNotifyCallback(codec->impl, cb, codec); + if (status != AMEDIA_OK) { + av_log(codec, AV_LOG_ERROR, "setAsyncNotifyCallback failed, %d\n", + status); + return AVERROR_EXTERNAL; + } + + return 0; +} + static const FFAMediaFormat media_format_ndk = { .class = &amediaformat_ndk_class, @@ -2396,6 +2525,7 @@ static const FFAMediaCodec media_codec_ndk = { .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode, .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers, .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream, + .setAsyncNotifyCallback = mediacodec_ndk_setAsyncNotifyCallback, }; FFAMediaFormat *ff_AMediaFormat_new(int ndk) diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h index 11a4260497..18d0796445 100644 --- a/libavcodec/mediacodec_wrapper.h +++ b/libavcodec/mediacodec_wrapper.h @@ -178,6 +178,22 @@ struct FFAMediaCodecBufferInfo { typedef struct FFAMediaCodecBufferInfo FFAMediaCodecBufferInfo; typedef struct FFAMediaCodec FFAMediaCodec; + +typedef struct FFAMediaCodecOnAsyncNotifyCallback { + void (*onAsyncInputAvailable)(FFAMediaCodec *codec, void *userdata, + int32_t index); + + void (*onAsyncOutputAvailable)(FFAMediaCodec *codec, void *userdata, + int32_t index, + FFAMediaCodecBufferInfo *buffer_info); + + void (*onAsyncFormatChanged)(FFAMediaCodec *codec, void *userdata, + FFAMediaFormat *format); + + void (*onAsyncError)(FFAMediaCodec *codec, void *userdata, int error, + const char *detail); +} FFAMediaCodecOnAsyncNotifyCallback; + struct FFAMediaCodec { const AVClass *class; @@ -219,6 +235,11 @@ struct FFAMediaCodec { // For encoder with FFANativeWindow as input. int (*signalEndOfInputStream)(FFAMediaCodec *); + + // Introduced in Android API 28 + int (*setAsyncNotifyCallback)(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata); }; static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec) @@ -343,6 +364,13 @@ static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec *codec) return codec->signalEndOfInputStream(codec); } +static inline int ff_AMediaCodec_setAsyncNotifyCallback(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + return codec->setAsyncNotifyCallback(codec, callback, userdata); +} + int ff_Build_SDK_INT(AVCodecContext *avctx); enum FFAMediaFormatColorRange {