From patchwork Thu Jan 4 16:52:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 45480 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6623:b0:194:e134:edd4 with SMTP id n35csp7718270pzh; Thu, 4 Jan 2024 00:52:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IE922DyVx3A9eY0Yzf3HaDGvC8Jk8t/Pqnt+IXFCvxWLhnVFlqXWyK66Jjz3r1VBX4D+IHd X-Received: by 2002:a05:6512:969:b0:50e:9dbe:e563 with SMTP id v9-20020a056512096900b0050e9dbee563mr122072lft.77.1704358361139; Thu, 04 Jan 2024 00:52:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704358361; cv=none; d=google.com; s=arc-20160816; b=Hf8QPLpi/lKuWnLxXLmmkUXnivC1Iryqyh3JlMW6D1P/Gk3GY6sERksgAidlXPG0kI zcgkk2pBtcSPP3Ft4YieCPCME6fCLizcjoUF+XAjj2//dw0aPTpRt0gEX1COT2Y2FddY ZtRTfg/P7i/aSh6wkrC0biH4q4sDkZueGZnokZ324S3aBN8f1JIYGVYtSg5ArMOJ13yz 6/NjhzL7ZTJoIE35GcwQa7Ca3jdCFhsoRwzZ9Whyo4eDG2HJ/lv0UMhW54CTY/Frvwkm m6JM5qfZ1Hjwo4vY3kdKefpLSVNMHFHulOSh8US2TH57A+ImtG1Fl9lCLD7I9ByNt5eG 77/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:from:message-id :dkim-signature:delivered-to; bh=n536Mrh1EYWTdKLHbkRI84+NobMswJ1oGJefuL8j6t8=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=EJqoCujsRpX+ZHRtiTOPKRRcrzPwDxXMO9Dsr15Aa6iau9zNG6W5rgF6gaN0NMVdz1 VyyTDYxlwqKz7/WwOeYtmJgvg1BFVmz96c+h79ojA5EfQZa1laX6QLHpZvASE09laDAx aTiZzfkgMKsrPZhzwI9WYV8htj1tkB9x2gdqunLCmRCHSpoK623mVZ1poK26UPaN4lKL Fj8fYbPakdBv9H0BaeWOo46eDfHaQ2h5yF/h80h/ZZQ32vQ2mV9kaVthBG6CxlciY3FO N6TB79NUdTQy1exVNhITKaxRFyW8AR/dtbcpHi4b/5dfsWGkmkxgnPo8yHBRA5b4WbFa Mc4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=yD7STY4P; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y17-20020a1709060bd100b00a271c6df7c5si7213372ejg.245.2024.01.04.00.52.40; Thu, 04 Jan 2024 00:52:41 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=yD7STY4P; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AC5E968CDE9; Thu, 4 Jan 2024 10:52:26 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-192.mail.qq.com (out203-205-221-192.mail.qq.com [203.205.221.192]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 157AD68CD4E for ; Thu, 4 Jan 2024 10:52:17 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1704358327; bh=9qJxYwEh0AZua0QXR0Rrv7WlKbBlWz8BrZTvJOez99k=; h=From:To:Cc:Subject:Date; b=yD7STY4PDlbdsw7fsyiWXEZ5muakxKRK35+Q2bCLLIabwbEFb0CuvDm6NP4/9WKDE I/VavESsMDDbm9odM4gRFUc/5tdVub7ln5fTRmlPRaQxrt2uByOUOWx5Kw+As0yvcc SQNOvfLuodgBndmDbTtyRof+4Kr4ndrx2/d+63ck= Received: from localhost.localdomain ([113.108.77.63]) by newxmesmtplogicsvrszc5-2.qq.com (NewEsmtp) with SMTP id D062DED4; Thu, 04 Jan 2024 16:52:06 +0800 X-QQ-mid: xmsmtpt1704358326tjd7c1l78 Message-ID: X-QQ-XMAILINFO: OZsapEVPoiO6I7yq676vBMMcqgC59Q6Zj3XLr1ouDrZyn6OuUONoQ8kJvs2rJc 2k5+Wgd7pQ/wq/xXQIusEYvxUgKlbMFAi0YxIboSbeLMtwCZ9hOQ1N3A+fEn1vsGjinOs5DNGSX9 JcVWPv4CSI5fmxEy9COkSnHP+fk+VKmRkbeow1g5gsEGIFkixTrliFEeRjpU0I7/vZUB7ioc1e9w bzcm7DXwjMTx83n1UnAZeYAXLDQLkLgyBO07eUoaUqjWj2WEHyF9f1pFM16Vk2/TuyoV51GxlW0K FBwWmXkYX0ZToTtbbkMOUF8A81YLhrxe+MfruzxFMdzjN8WIWiZOq6vsuJtAqsqHPi9yjqW8NOkl 8pyEAZlbQhpgX98rxJ+9QCaydmxdq6uVweGIZ5ixphgpLoxAnq14JPV52kQ6/0xWAr7LBA8q0IJU pIaiLvlf62Y4ZOlGGw4H5bbb5ZRqKcJpC3V5iIETGPa89AyF+0GsgoT4zKwZBsqNe9TxpJ9FcArG IAPwiezhHMzLeM7LeeK3rNADo6bynBru8QZ/F/PLeoiAnnpMnTS8uI1VS/ShK0wQUDwvbAJu9XmZ OFTEbiFbqX+5lc5xVD4QhegnwfonMpj69SLmvqpnhSISJ+1cCxaGvGMFNkzXfJEmgymwopAgUybN o/x3TsTY7+BttX9HJ/RSYcYYLqhqA9eurfyck21QV8NqETWZ1PHtECkG82EeeJ2LWwHQxeRRqF5H zCR28C55a6fDuk8bFXeib0ZM3fbELcRn5fe9MfAjt5MefpPv8sKgcK0Hl996Yu0Nh5pCs09eNptB fFneYZmo/svQ3lLy2+8Fuz64Et3KNtnZoMN/7VLQGtqbE8S+ItDDy4l215Ms+vdwBozuw4mxw9p2 BPVqzCSzF8KzQGDfyEFYQnuFQHPrRRav0vN50ACHaJTDfySu6rHg3OiZVkCj958DR/UzfbkC3Jvq LRB2NppRGB+QPgsadQhwxT75EbR/vngTDTED+WCbw= X-QQ-XMRINFO: Nq+8W0+stu50PRdwbJxPCL0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Fri, 5 Jan 2024 00:52:08 +0800 X-OQ-MSGID: <20240104165209.995565-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] avcodec/mediacodec_wrapper: add async mode support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tbTQ2mUBOYgp From: Zhao Zhili Only support NDK mediacodec. --- libavcodec/mediacodec_wrapper.c | 127 ++++++++++++++++++++++++++++++++ libavcodec/mediacodec_wrapper.h | 25 +++++++ 2 files changed, 152 insertions(+) diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c index f2fb392329..2f4a3855e4 100644 --- a/libavcodec/mediacodec_wrapper.c +++ b/libavcodec/mediacodec_wrapper.c @@ -1848,6 +1848,14 @@ static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx) return 0; } +static int mediacodec_jni_setAsyncNotifyCallback(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + av_log(codec, AV_LOG_ERROR, "Doesn't support aync mode with JNI, please try ndk_codec=1\n"); + return AVERROR(ENOSYS); +} + static const FFAMediaFormat media_format_jni = { .class = &amediaformat_class, @@ -1907,6 +1915,8 @@ static const FFAMediaCodec media_codec_jni = { .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode, .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers, .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream, + + .setAsyncNotifyCallback = mediacodec_jni_setAsyncNotifyCallback, }; typedef struct FFAMediaFormatNdk { @@ -1945,6 +1955,9 @@ typedef struct FFAMediaCodecNdk { AMediaCodec *impl; ANativeWindow *window; + FFAMediaCodecOnAsyncNotifyCallback async_cb; + void *async_userdata; + AMediaCodec* (*createCodecByName)(const char *name); AMediaCodec* (*createDecoderByType)(const char *mime_type); AMediaCodec* (*createEncoderByType)(const char *mime_type); @@ -1980,6 +1993,11 @@ typedef struct FFAMediaCodecNdk { // Available since API level 26. media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *); media_status_t (*signalEndOfInputStream)(AMediaCodec *); + + // Available since API level 28 + media_status_t (*setAsyncNotifyCallback)(AMediaCodec *, + struct AMediaCodecOnAsyncNotifyCallback callback, + void *userdata); } FFAMediaCodecNdk; static const FFAMediaFormat media_format_ndk; @@ -1995,6 +2013,32 @@ static const AVClass amediacodec_ndk_class = { .version = LIBAVUTIL_VERSION_INT, }; +static int media_status_to_error(media_status_t status) +{ + switch (status) { + case AMEDIA_OK: + return 0; + case AMEDIACODEC_ERROR_INSUFFICIENT_RESOURCE: + return AVERROR(ENOMEM); + case AMEDIA_ERROR_MALFORMED: + return AVERROR_INVALIDDATA; + case AMEDIA_ERROR_UNSUPPORTED: + return AVERROR(ENOTSUP); + case AMEDIA_ERROR_INVALID_PARAMETER: + return AVERROR(EINVAL); + case AMEDIA_ERROR_INVALID_OPERATION: + return AVERROR(EOPNOTSUPP); + case AMEDIA_ERROR_END_OF_STREAM: + return AVERROR_EOF; + case AMEDIA_ERROR_IO: + return AVERROR(EIO); + case AMEDIA_ERROR_WOULD_BLOCK: + return AVERROR(EWOULDBLOCK); + default: + return AVERROR_EXTERNAL; + } +} + static FFAMediaFormat *mediaformat_ndk_create(AMediaFormat *impl) { FFAMediaFormatNdk *format = av_mallocz(sizeof(*format)); @@ -2239,6 +2283,8 @@ static inline FFAMediaCodec *ndk_codec_create(int method, const char *arg) { GET_SYMBOL(setInputSurface, 0) GET_SYMBOL(signalEndOfInputStream, 0) + GET_SYMBOL(setAsyncNotifyCallback, 0) + #undef GET_SYMBOL switch (method) { @@ -2513,6 +2559,85 @@ static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx) return 0; } +static void mediacodec_ndk_onInputAvailable(AMediaCodec *impl, void *userdata, int32_t index) +{ + FFAMediaCodecNdk *codec = userdata; + codec->async_cb.onAsyncInputAvailable((FFAMediaCodec *) codec, codec->async_userdata, index); +} + +static void mediacodec_ndk_onOutputAvailable(AMediaCodec *impl, + void *userdata, + int32_t index, + AMediaCodecBufferInfo *buffer_info) +{ + FFAMediaCodecNdk *codec = userdata; + FFAMediaCodecBufferInfo info = { + .offset = buffer_info->offset, + .size = buffer_info->size, + .presentationTimeUs = buffer_info->presentationTimeUs, + .flags = buffer_info->flags, + }; + + codec->async_cb.onAsyncOutputAvailable(&codec->api, codec->async_userdata, index, &info); +} + +static void mediacodec_ndk_onFormatChanged(AMediaCodec *impl, void *userdata, AMediaFormat *format) +{ + FFAMediaCodecNdk *codec = userdata; + FFAMediaFormat *media_format = mediaformat_ndk_create(format); + if (!media_format) + return; + + codec->async_cb.onAsyncFormatChanged(&codec->api, codec->async_userdata, media_format); + ff_AMediaFormat_delete(media_format); +} + +static void mediacodec_ndk_onError(AMediaCodec *impl, void *userdata, + media_status_t status, + int32_t actionCode, + const char *detail) +{ + FFAMediaCodecNdk *codec = userdata; + int error = media_status_to_error(status); + + codec->async_cb.onAsyncError(&codec->api, codec->async_userdata, error, detail); +} + +static int mediacodec_ndk_setAsyncNotifyCallback(FFAMediaCodec *ctx, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx; + struct AMediaCodecOnAsyncNotifyCallback cb = { + .onAsyncInputAvailable = mediacodec_ndk_onInputAvailable, + .onAsyncOutputAvailable = mediacodec_ndk_onOutputAvailable, + .onAsyncFormatChanged = mediacodec_ndk_onFormatChanged, + .onAsyncError = mediacodec_ndk_onError, + }; + media_status_t status; + + if (!codec->setAsyncNotifyCallback) { + av_log(codec, AV_LOG_ERROR, "setAsyncNotifyCallback unavailable\n"); + return AVERROR(ENOSYS); + } + + if (!callback || + !callback->onAsyncInputAvailable || !callback->onAsyncOutputAvailable || + !callback->onAsyncFormatChanged || !callback->onAsyncError) + return AVERROR(EINVAL); + + codec->async_cb = *callback; + codec->async_userdata = userdata; + + status = codec->setAsyncNotifyCallback(codec->impl, cb, codec); + if (status != AMEDIA_OK) { + av_log(codec, AV_LOG_ERROR, "setAsyncNotifyCallback failed, %d\n", status); + return AVERROR_EXTERNAL; + } + + return 0; +} + static const FFAMediaFormat media_format_ndk = { .class = &amediaformat_ndk_class, @@ -2574,6 +2699,8 @@ static const FFAMediaCodec media_codec_ndk = { .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode, .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers, .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream, + + .setAsyncNotifyCallback = mediacodec_ndk_setAsyncNotifyCallback, }; FFAMediaFormat *ff_AMediaFormat_new(int ndk) diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h index 11a4260497..084e9b3638 100644 --- a/libavcodec/mediacodec_wrapper.h +++ b/libavcodec/mediacodec_wrapper.h @@ -178,6 +178,19 @@ struct FFAMediaCodecBufferInfo { typedef struct FFAMediaCodecBufferInfo FFAMediaCodecBufferInfo; typedef struct FFAMediaCodec FFAMediaCodec; + +typedef struct FFAMediaCodecOnAsyncNotifyCallback { + void (*onAsyncInputAvailable)(FFAMediaCodec *codec, void *userdata, int32_t index); + + void (*onAsyncOutputAvailable)(FFAMediaCodec *codec, void *userdata, + int32_t index, + FFAMediaCodecBufferInfo *buffer_info); + + void (*onAsyncFormatChanged)(FFAMediaCodec *codec, void *userdata, FFAMediaFormat *format); + + void (*onAsyncError)(FFAMediaCodec *codec, void *userdata, int error, const char *detail); +} FFAMediaCodecOnAsyncNotifyCallback; + struct FFAMediaCodec { const AVClass *class; @@ -219,6 +232,11 @@ struct FFAMediaCodec { // For encoder with FFANativeWindow as input. int (*signalEndOfInputStream)(FFAMediaCodec *); + + // Introduced in Android API 28 + int (*setAsyncNotifyCallback)(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata); }; static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec) @@ -343,6 +361,13 @@ static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec *codec) return codec->signalEndOfInputStream(codec); } +static inline int ff_AMediaCodec_setAsyncNotifyCallback(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + return codec->setAsyncNotifyCallback(codec, callback, userdata); +} + int ff_Build_SDK_INT(AVCodecContext *avctx); enum FFAMediaFormatColorRange {