From patchwork Thu Jan 4 16:52:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 45480 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6623:b0:194:e134:edd4 with SMTP id n35csp7718270pzh; Thu, 4 Jan 2024 00:52:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IE922DyVx3A9eY0Yzf3HaDGvC8Jk8t/Pqnt+IXFCvxWLhnVFlqXWyK66Jjz3r1VBX4D+IHd X-Received: by 2002:a05:6512:969:b0:50e:9dbe:e563 with SMTP id v9-20020a056512096900b0050e9dbee563mr122072lft.77.1704358361139; Thu, 04 Jan 2024 00:52:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704358361; cv=none; d=google.com; s=arc-20160816; b=Hf8QPLpi/lKuWnLxXLmmkUXnivC1Iryqyh3JlMW6D1P/Gk3GY6sERksgAidlXPG0kI zcgkk2pBtcSPP3Ft4YieCPCME6fCLizcjoUF+XAjj2//dw0aPTpRt0gEX1COT2Y2FddY ZtRTfg/P7i/aSh6wkrC0biH4q4sDkZueGZnokZ324S3aBN8f1JIYGVYtSg5ArMOJ13yz 6/NjhzL7ZTJoIE35GcwQa7Ca3jdCFhsoRwzZ9Whyo4eDG2HJ/lv0UMhW54CTY/Frvwkm m6JM5qfZ1Hjwo4vY3kdKefpLSVNMHFHulOSh8US2TH57A+ImtG1Fl9lCLD7I9ByNt5eG 77/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:from:message-id :dkim-signature:delivered-to; bh=n536Mrh1EYWTdKLHbkRI84+NobMswJ1oGJefuL8j6t8=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=EJqoCujsRpX+ZHRtiTOPKRRcrzPwDxXMO9Dsr15Aa6iau9zNG6W5rgF6gaN0NMVdz1 VyyTDYxlwqKz7/WwOeYtmJgvg1BFVmz96c+h79ojA5EfQZa1laX6QLHpZvASE09laDAx aTiZzfkgMKsrPZhzwI9WYV8htj1tkB9x2gdqunLCmRCHSpoK623mVZ1poK26UPaN4lKL Fj8fYbPakdBv9H0BaeWOo46eDfHaQ2h5yF/h80h/ZZQ32vQ2mV9kaVthBG6CxlciY3FO N6TB79NUdTQy1exVNhITKaxRFyW8AR/dtbcpHi4b/5dfsWGkmkxgnPo8yHBRA5b4WbFa Mc4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=yD7STY4P; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y17-20020a1709060bd100b00a271c6df7c5si7213372ejg.245.2024.01.04.00.52.40; Thu, 04 Jan 2024 00:52:41 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=yD7STY4P; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AC5E968CDE9; Thu, 4 Jan 2024 10:52:26 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-192.mail.qq.com (out203-205-221-192.mail.qq.com [203.205.221.192]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 157AD68CD4E for ; Thu, 4 Jan 2024 10:52:17 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1704358327; bh=9qJxYwEh0AZua0QXR0Rrv7WlKbBlWz8BrZTvJOez99k=; h=From:To:Cc:Subject:Date; b=yD7STY4PDlbdsw7fsyiWXEZ5muakxKRK35+Q2bCLLIabwbEFb0CuvDm6NP4/9WKDE I/VavESsMDDbm9odM4gRFUc/5tdVub7ln5fTRmlPRaQxrt2uByOUOWx5Kw+As0yvcc SQNOvfLuodgBndmDbTtyRof+4Kr4ndrx2/d+63ck= Received: from localhost.localdomain ([113.108.77.63]) by newxmesmtplogicsvrszc5-2.qq.com (NewEsmtp) with SMTP id D062DED4; Thu, 04 Jan 2024 16:52:06 +0800 X-QQ-mid: xmsmtpt1704358326tjd7c1l78 Message-ID: X-QQ-XMAILINFO: OZsapEVPoiO6I7yq676vBMMcqgC59Q6Zj3XLr1ouDrZyn6OuUONoQ8kJvs2rJc 2k5+Wgd7pQ/wq/xXQIusEYvxUgKlbMFAi0YxIboSbeLMtwCZ9hOQ1N3A+fEn1vsGjinOs5DNGSX9 JcVWPv4CSI5fmxEy9COkSnHP+fk+VKmRkbeow1g5gsEGIFkixTrliFEeRjpU0I7/vZUB7ioc1e9w bzcm7DXwjMTx83n1UnAZeYAXLDQLkLgyBO07eUoaUqjWj2WEHyF9f1pFM16Vk2/TuyoV51GxlW0K FBwWmXkYX0ZToTtbbkMOUF8A81YLhrxe+MfruzxFMdzjN8WIWiZOq6vsuJtAqsqHPi9yjqW8NOkl 8pyEAZlbQhpgX98rxJ+9QCaydmxdq6uVweGIZ5ixphgpLoxAnq14JPV52kQ6/0xWAr7LBA8q0IJU pIaiLvlf62Y4ZOlGGw4H5bbb5ZRqKcJpC3V5iIETGPa89AyF+0GsgoT4zKwZBsqNe9TxpJ9FcArG IAPwiezhHMzLeM7LeeK3rNADo6bynBru8QZ/F/PLeoiAnnpMnTS8uI1VS/ShK0wQUDwvbAJu9XmZ OFTEbiFbqX+5lc5xVD4QhegnwfonMpj69SLmvqpnhSISJ+1cCxaGvGMFNkzXfJEmgymwopAgUybN o/x3TsTY7+BttX9HJ/RSYcYYLqhqA9eurfyck21QV8NqETWZ1PHtECkG82EeeJ2LWwHQxeRRqF5H zCR28C55a6fDuk8bFXeib0ZM3fbELcRn5fe9MfAjt5MefpPv8sKgcK0Hl996Yu0Nh5pCs09eNptB fFneYZmo/svQ3lLy2+8Fuz64Et3KNtnZoMN/7VLQGtqbE8S+ItDDy4l215Ms+vdwBozuw4mxw9p2 BPVqzCSzF8KzQGDfyEFYQnuFQHPrRRav0vN50ACHaJTDfySu6rHg3OiZVkCj958DR/UzfbkC3Jvq LRB2NppRGB+QPgsadQhwxT75EbR/vngTDTED+WCbw= X-QQ-XMRINFO: Nq+8W0+stu50PRdwbJxPCL0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Fri, 5 Jan 2024 00:52:08 +0800 X-OQ-MSGID: <20240104165209.995565-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] avcodec/mediacodec_wrapper: add async mode support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tbTQ2mUBOYgp From: Zhao Zhili Only support NDK mediacodec. --- libavcodec/mediacodec_wrapper.c | 127 ++++++++++++++++++++++++++++++++ libavcodec/mediacodec_wrapper.h | 25 +++++++ 2 files changed, 152 insertions(+) diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c index f2fb392329..2f4a3855e4 100644 --- a/libavcodec/mediacodec_wrapper.c +++ b/libavcodec/mediacodec_wrapper.c @@ -1848,6 +1848,14 @@ static int mediacodec_jni_signalEndOfInputStream(FFAMediaCodec *ctx) return 0; } +static int mediacodec_jni_setAsyncNotifyCallback(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + av_log(codec, AV_LOG_ERROR, "Doesn't support aync mode with JNI, please try ndk_codec=1\n"); + return AVERROR(ENOSYS); +} + static const FFAMediaFormat media_format_jni = { .class = &amediaformat_class, @@ -1907,6 +1915,8 @@ static const FFAMediaCodec media_codec_jni = { .getConfigureFlagEncode = mediacodec_jni_getConfigureFlagEncode, .cleanOutputBuffers = mediacodec_jni_cleanOutputBuffers, .signalEndOfInputStream = mediacodec_jni_signalEndOfInputStream, + + .setAsyncNotifyCallback = mediacodec_jni_setAsyncNotifyCallback, }; typedef struct FFAMediaFormatNdk { @@ -1945,6 +1955,9 @@ typedef struct FFAMediaCodecNdk { AMediaCodec *impl; ANativeWindow *window; + FFAMediaCodecOnAsyncNotifyCallback async_cb; + void *async_userdata; + AMediaCodec* (*createCodecByName)(const char *name); AMediaCodec* (*createDecoderByType)(const char *mime_type); AMediaCodec* (*createEncoderByType)(const char *mime_type); @@ -1980,6 +1993,11 @@ typedef struct FFAMediaCodecNdk { // Available since API level 26. media_status_t (*setInputSurface)(AMediaCodec*, ANativeWindow *); media_status_t (*signalEndOfInputStream)(AMediaCodec *); + + // Available since API level 28 + media_status_t (*setAsyncNotifyCallback)(AMediaCodec *, + struct AMediaCodecOnAsyncNotifyCallback callback, + void *userdata); } FFAMediaCodecNdk; static const FFAMediaFormat media_format_ndk; @@ -1995,6 +2013,32 @@ static const AVClass amediacodec_ndk_class = { .version = LIBAVUTIL_VERSION_INT, }; +static int media_status_to_error(media_status_t status) +{ + switch (status) { + case AMEDIA_OK: + return 0; + case AMEDIACODEC_ERROR_INSUFFICIENT_RESOURCE: + return AVERROR(ENOMEM); + case AMEDIA_ERROR_MALFORMED: + return AVERROR_INVALIDDATA; + case AMEDIA_ERROR_UNSUPPORTED: + return AVERROR(ENOTSUP); + case AMEDIA_ERROR_INVALID_PARAMETER: + return AVERROR(EINVAL); + case AMEDIA_ERROR_INVALID_OPERATION: + return AVERROR(EOPNOTSUPP); + case AMEDIA_ERROR_END_OF_STREAM: + return AVERROR_EOF; + case AMEDIA_ERROR_IO: + return AVERROR(EIO); + case AMEDIA_ERROR_WOULD_BLOCK: + return AVERROR(EWOULDBLOCK); + default: + return AVERROR_EXTERNAL; + } +} + static FFAMediaFormat *mediaformat_ndk_create(AMediaFormat *impl) { FFAMediaFormatNdk *format = av_mallocz(sizeof(*format)); @@ -2239,6 +2283,8 @@ static inline FFAMediaCodec *ndk_codec_create(int method, const char *arg) { GET_SYMBOL(setInputSurface, 0) GET_SYMBOL(signalEndOfInputStream, 0) + GET_SYMBOL(setAsyncNotifyCallback, 0) + #undef GET_SYMBOL switch (method) { @@ -2513,6 +2559,85 @@ static int mediacodec_ndk_signalEndOfInputStream(FFAMediaCodec *ctx) return 0; } +static void mediacodec_ndk_onInputAvailable(AMediaCodec *impl, void *userdata, int32_t index) +{ + FFAMediaCodecNdk *codec = userdata; + codec->async_cb.onAsyncInputAvailable((FFAMediaCodec *) codec, codec->async_userdata, index); +} + +static void mediacodec_ndk_onOutputAvailable(AMediaCodec *impl, + void *userdata, + int32_t index, + AMediaCodecBufferInfo *buffer_info) +{ + FFAMediaCodecNdk *codec = userdata; + FFAMediaCodecBufferInfo info = { + .offset = buffer_info->offset, + .size = buffer_info->size, + .presentationTimeUs = buffer_info->presentationTimeUs, + .flags = buffer_info->flags, + }; + + codec->async_cb.onAsyncOutputAvailable(&codec->api, codec->async_userdata, index, &info); +} + +static void mediacodec_ndk_onFormatChanged(AMediaCodec *impl, void *userdata, AMediaFormat *format) +{ + FFAMediaCodecNdk *codec = userdata; + FFAMediaFormat *media_format = mediaformat_ndk_create(format); + if (!media_format) + return; + + codec->async_cb.onAsyncFormatChanged(&codec->api, codec->async_userdata, media_format); + ff_AMediaFormat_delete(media_format); +} + +static void mediacodec_ndk_onError(AMediaCodec *impl, void *userdata, + media_status_t status, + int32_t actionCode, + const char *detail) +{ + FFAMediaCodecNdk *codec = userdata; + int error = media_status_to_error(status); + + codec->async_cb.onAsyncError(&codec->api, codec->async_userdata, error, detail); +} + +static int mediacodec_ndk_setAsyncNotifyCallback(FFAMediaCodec *ctx, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + FFAMediaCodecNdk *codec = (FFAMediaCodecNdk *)ctx; + struct AMediaCodecOnAsyncNotifyCallback cb = { + .onAsyncInputAvailable = mediacodec_ndk_onInputAvailable, + .onAsyncOutputAvailable = mediacodec_ndk_onOutputAvailable, + .onAsyncFormatChanged = mediacodec_ndk_onFormatChanged, + .onAsyncError = mediacodec_ndk_onError, + }; + media_status_t status; + + if (!codec->setAsyncNotifyCallback) { + av_log(codec, AV_LOG_ERROR, "setAsyncNotifyCallback unavailable\n"); + return AVERROR(ENOSYS); + } + + if (!callback || + !callback->onAsyncInputAvailable || !callback->onAsyncOutputAvailable || + !callback->onAsyncFormatChanged || !callback->onAsyncError) + return AVERROR(EINVAL); + + codec->async_cb = *callback; + codec->async_userdata = userdata; + + status = codec->setAsyncNotifyCallback(codec->impl, cb, codec); + if (status != AMEDIA_OK) { + av_log(codec, AV_LOG_ERROR, "setAsyncNotifyCallback failed, %d\n", status); + return AVERROR_EXTERNAL; + } + + return 0; +} + static const FFAMediaFormat media_format_ndk = { .class = &amediaformat_ndk_class, @@ -2574,6 +2699,8 @@ static const FFAMediaCodec media_codec_ndk = { .getConfigureFlagEncode = mediacodec_ndk_getConfigureFlagEncode, .cleanOutputBuffers = mediacodec_ndk_cleanOutputBuffers, .signalEndOfInputStream = mediacodec_ndk_signalEndOfInputStream, + + .setAsyncNotifyCallback = mediacodec_ndk_setAsyncNotifyCallback, }; FFAMediaFormat *ff_AMediaFormat_new(int ndk) diff --git a/libavcodec/mediacodec_wrapper.h b/libavcodec/mediacodec_wrapper.h index 11a4260497..084e9b3638 100644 --- a/libavcodec/mediacodec_wrapper.h +++ b/libavcodec/mediacodec_wrapper.h @@ -178,6 +178,19 @@ struct FFAMediaCodecBufferInfo { typedef struct FFAMediaCodecBufferInfo FFAMediaCodecBufferInfo; typedef struct FFAMediaCodec FFAMediaCodec; + +typedef struct FFAMediaCodecOnAsyncNotifyCallback { + void (*onAsyncInputAvailable)(FFAMediaCodec *codec, void *userdata, int32_t index); + + void (*onAsyncOutputAvailable)(FFAMediaCodec *codec, void *userdata, + int32_t index, + FFAMediaCodecBufferInfo *buffer_info); + + void (*onAsyncFormatChanged)(FFAMediaCodec *codec, void *userdata, FFAMediaFormat *format); + + void (*onAsyncError)(FFAMediaCodec *codec, void *userdata, int error, const char *detail); +} FFAMediaCodecOnAsyncNotifyCallback; + struct FFAMediaCodec { const AVClass *class; @@ -219,6 +232,11 @@ struct FFAMediaCodec { // For encoder with FFANativeWindow as input. int (*signalEndOfInputStream)(FFAMediaCodec *); + + // Introduced in Android API 28 + int (*setAsyncNotifyCallback)(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata); }; static inline char *ff_AMediaCodec_getName(FFAMediaCodec *codec) @@ -343,6 +361,13 @@ static inline int ff_AMediaCodec_signalEndOfInputStream(FFAMediaCodec *codec) return codec->signalEndOfInputStream(codec); } +static inline int ff_AMediaCodec_setAsyncNotifyCallback(FFAMediaCodec *codec, + const FFAMediaCodecOnAsyncNotifyCallback *callback, + void *userdata) +{ + return codec->setAsyncNotifyCallback(codec, callback, userdata); +} + int ff_Build_SDK_INT(AVCodecContext *avctx); enum FFAMediaFormatColorRange { From patchwork Thu Jan 4 16:52:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 45479 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6623:b0:194:e134:edd4 with SMTP id n35csp7718218pzh; Thu, 4 Jan 2024 00:52:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IGGRUSi3nWSd8z+u5fp65J3HDpdEINq4iqbY20I/JZHj+LSkELkB8SI123VwiS/ShI+jC0F X-Received: by 2002:a17:907:1de7:b0:a27:914d:4b76 with SMTP id og39-20020a1709071de700b00a27914d4b76mr72968ejc.12.1704358350456; Thu, 04 Jan 2024 00:52:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704358350; cv=none; d=google.com; s=arc-20160816; b=oIAxjqN4pEUEEPLeM9yWrfYvPRkYSlcCa+JIIuZIBHVP6vpeCKdEEW50/llNerse5h wdE9lDGPPkWhwQ+N7t7rqtH4Sw1BS/+d1WP2poUnchFw02Tj2hQEdVoobU2XkPPKqElC 9KKd3gh3rmHiAC94ljuyFCe4C1cdcpXop5lb/XXNHSXZnqVuff6+xAwS4czLmFMVAeYP nWLNIGMockm43Jjmuq9ESv5uUTvPIkPjgM2jMFh2o6KZ/tHPfYLBQk/W2wfsfvSCQwK1 BXp7oMdxBiN4VtpmRvW51aA8CEog3NhTwN4u1i5KSqNMtk+CWGoDPAMbtrwXvw/aUrEL MqHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=Q0JmPNqbmeEhG/z171PoxqLZlJAIHmUX3uZArJfLzuU=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=FOfpTDdmm/s4t9x1Ha+WAxVFAz/QbR1BK2sp4HZ+IzlcbGhGenyfLGPtvWq/Khwfh4 eNR3nMcrFTcHWmuoiUAWv/4Yjgy8yJnj3kRxmEXSlKgYf6D5qpH9TUk3ikP2Qsgc/e9u 31TpPB1b5WnO7P1/RItiAXcyqkLjV3nBSfwDOuXt6rjNqYomUQ7RLDfU50jjWp9jdk3i I77wS5SKY3itSCiVo2nqymluJXdv5uwsobj+j9M9UlNr1IFKT0RZHKD0wKH4085iHYoF i/zsNUXZtggSm4XeS2ymi1ZWXMCmIwISDKZp9Acrxgpm22HslJ8pdjVQoFGfs9sSPVnX luVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="E/jK2xR4"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a12-20020a170906368c00b00a2333846e72si12207086ejc.739.2024.01.04.00.52.29; Thu, 04 Jan 2024 00:52:30 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="E/jK2xR4"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B14BB68CD4E; Thu, 4 Jan 2024 10:52:25 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-164.mail.qq.com (out203-205-221-164.mail.qq.com [203.205.221.164]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 125A268C48A for ; Thu, 4 Jan 2024 10:52:17 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1704358327; bh=2PeWntEuuCg66o2gzWK4l9k+eQg4yrbv/XYFSUW4k1w=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=E/jK2xR4XoBU8mkAyb/lzGs919A/6Rm1ShjSCI1KTed9bQgKErqe4iNAqOkDJzmP6 NbovqD2p+ZpsxbP4Fz4E8vJfeLHjr257vUh+PjvEaCjaaxR/ptngwUBKgTQb4J5jVv HtSWI4piSVhemiUhrCSLsFIVONW4GxEOMD821Qis= Received: from localhost.localdomain ([113.108.77.63]) by newxmesmtplogicsvrszc5-2.qq.com (NewEsmtp) with SMTP id D062DED4; Thu, 04 Jan 2024 16:52:06 +0800 X-QQ-mid: xmsmtpt1704358327tg81cedos Message-ID: X-QQ-XMAILINFO: OW6mPzc+Kmdzgnn+zB+p04pu7AAIwDMNPvqEI1DBd57wothMiIHl2cjGfwIUJa BDN5kzNNTn4dnMOKBYmAgYlPR5P/v7lyTGA399OHjFTTATp/3rYssfWgVFu3LQYu7U3zki5FcOT9 uKmQKGIlUkS4Z6gUrxLPV5d8Rm6ymLoaiqaLjKOw3Tu4twwruHEZkG/plnop23U85N3QI3c8xTs3 9Z0CRPKYNYK7+BhBxHh62OQwsDquMSC5Mad5OARyofK0j5DA+kGQ9hNGLCoDwkrOFeZ6aqosPSqn izlH3KRFqKzV9gHwKxtX2nrP3R6WLCvBwTe6EnD+/lhVRF781++e4nvynBhNw3BOEeUK54L+wJDi hNboa/nWNn3WQ3UCuotv7JvHiNjjbna/96qUavNQtGwPYO8CEBRYjTEPJfkCXBnxt8qvUVPIfb3U J10duTlGbu6+lt+S360Vakd9/2eA1nzY3Eb+0DiJIbic9KhiPZvZ47uTewo62G0k+ssF/grCOBbb p6B7/jo4M9FDaGMk8NTroS9Vdop55wKzf+iShpRF+mywR39h2jjxQ3w0QEVSAriv7mpuUlhU6uxd XWKRwVWqj9FXeTYKzKHQmqL3pBssgMWcY6ek2OPwaqO6wLm2To1HsPO+oPv7P/uJ+mkMVn3P3fDx uf34xxZDDuhnjFkFLEvIskHnXzN58EEmm6Gj6XMeXlVnur9VDdi36mBGzt+Hf4bqwOCE8hhEyBTn N5AL5+odsWw+a1NcZPkhrAwL/ZQ4c9L5JEJ+h8WIhUluRdLSFZBLHt5na4Lr07+GmMheVIX2Ge2a Ez/tDHvJtY4U6kzGdH+2KDLhk2iYVBNrxS7Tx95WEe7AVywieCHDT/WRhm+PS4xVTqZ8mK5k4qpG Qjs3vlf8csM2jpJ04qH3VhWdHe3dzKxyn9J8CpeRn31d/to0XPqIWQ4cV9SKsbVlCDcvkCDZVMj3 LeFE5oDNZEKre3FWb3B7sdToQiquV0pDgB36yAmV4= X-QQ-XMRINFO: MPJ6Tf5t3I/ycC2BItcBVIA= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Fri, 5 Jan 2024 00:52:09 +0800 X-OQ-MSGID: <20240104165209.995565-2-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240104165209.995565-1-quinkblack@foxmail.com> References: <20240104165209.995565-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] avcodec/mediacodecenc: add async mode support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: im+xlckW/ID8 From: Zhao Zhili --- configure | 2 +- libavcodec/mediacodecenc.c | 289 +++++++++++++++++++++++++++++++++---- 2 files changed, 258 insertions(+), 33 deletions(-) diff --git a/configure b/configure index d15cfa4703..4f0bdd8c7b 100755 --- a/configure +++ b/configure @@ -3062,7 +3062,7 @@ d3d11va_deps="dxva_h ID3D11VideoDecoder ID3D11VideoContext" d3d12va_deps="dxva_h ID3D12Device ID3D12VideoDecoder" dxva2_deps="dxva2api_h DXVA2_ConfigPictureDecode ole32 user32" ffnvcodec_deps_any="libdl LoadLibrary" -mediacodec_deps="android" +mediacodec_deps="android pthreads" nvdec_deps="ffnvcodec" vaapi_x11_deps="xlib_x11" videotoolbox_hwaccel_deps="videotoolbox pthreads" diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c index cab0189a9f..4729860d9c 100644 --- a/libavcodec/mediacodecenc.c +++ b/libavcodec/mediacodecenc.c @@ -20,9 +20,11 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include #include "config_components.h" #include "libavutil/avassert.h" +#include "libavutil/fifo.h" #include "libavutil/hwcontext_mediacodec.h" #include "libavutil/imgutils.h" #include "libavutil/opt.h" @@ -56,6 +58,7 @@ typedef struct MediaCodecEncContext { AVClass *avclass; FFAMediaCodec *codec; int use_ndk_codec; + int async_mode; const char *name; FFANativeWindow *window; @@ -73,6 +76,16 @@ typedef struct MediaCodecEncContext { int bitrate_mode; int level; int pts_as_dts; + + pthread_mutex_t input_mutex; + pthread_cond_t input_cond; + AVFifo *input_index; + + pthread_mutex_t output_mutex; + pthread_cond_t output_cond; + int encode_status; + AVFifo *output_index; + AVFifo *output_buf_info; } MediaCodecEncContext; enum { @@ -97,17 +110,25 @@ static const enum AVPixelFormat avc_pix_fmts[] = { AV_PIX_FMT_NONE }; -static void mediacodec_output_format(AVCodecContext *avctx) +static void mediacodec_dump_format(AVCodecContext *avctx, FFAMediaFormat *out_format) { MediaCodecEncContext *s = avctx->priv_data; - char *name = ff_AMediaCodec_getName(s->codec); - FFAMediaFormat *out_format = ff_AMediaCodec_getOutputFormat(s->codec); + const char *name = s->name; char *str = ff_AMediaFormat_toString(out_format); av_log(avctx, AV_LOG_DEBUG, "MediaCodec encoder %s output format %s\n", name ? name : "unknown", str); - av_free(name); av_free(str); +} + +static void mediacodec_output_format(AVCodecContext *avctx) +{ + MediaCodecEncContext *s = avctx->priv_data; + FFAMediaFormat *out_format = ff_AMediaCodec_getOutputFormat(s->codec); + + if (!s->name) + s->name = ff_AMediaCodec_getName(s->codec); + mediacodec_dump_format(avctx, out_format); ff_AMediaFormat_delete(out_format); } @@ -147,6 +168,131 @@ static int mediacodec_init_bsf(AVCodecContext *avctx) return ret; } +static void copy_frame_to_buffer(AVCodecContext *avctx, const AVFrame *frame, uint8_t *dst, size_t size) +{ + MediaCodecEncContext *s = avctx->priv_data; + uint8_t *dst_data[4] = {}; + int dst_linesize[4] = {}; + + if (avctx->pix_fmt == AV_PIX_FMT_YUV420P) { + dst_data[0] = dst; + dst_data[1] = dst + s->width * s->height; + dst_data[2] = dst_data[1] + s->width * s->height / 4; + + dst_linesize[0] = s->width; + dst_linesize[1] = dst_linesize[2] = s->width / 2; + } else if (avctx->pix_fmt == AV_PIX_FMT_NV12) { + dst_data[0] = dst; + dst_data[1] = dst + s->width * s->height; + + dst_linesize[0] = s->width; + dst_linesize[1] = s->width; + } else { + av_assert0(0); + } + + av_image_copy2(dst_data, dst_linesize, frame->data, frame->linesize, + avctx->pix_fmt, avctx->width, avctx->height); +} + +static void on_input_available(FFAMediaCodec *codec, void *userdata, int32_t index) +{ + AVCodecContext *avctx = userdata; + MediaCodecEncContext *s = avctx->priv_data; + + pthread_mutex_lock(&s->input_mutex); + + av_fifo_write(s->input_index, &index, 1); + + pthread_mutex_unlock(&s->input_mutex); + pthread_cond_signal(&s->input_cond); +} + +static void on_output_available(FFAMediaCodec *codec, void *userdata, + int32_t index, + FFAMediaCodecBufferInfo *out_info) +{ + AVCodecContext *avctx = userdata; + MediaCodecEncContext *s = avctx->priv_data; + + pthread_mutex_lock(&s->output_mutex); + + av_fifo_write(s->output_index, &index, 1); + av_fifo_write(s->output_buf_info, out_info, 1); + + pthread_mutex_unlock(&s->output_mutex); + pthread_cond_signal(&s->output_cond); +} + +static void on_format_changed(FFAMediaCodec *codec, void *userdata, FFAMediaFormat *format) +{ + mediacodec_dump_format(userdata, format); +} + +static void on_error(FFAMediaCodec *codec, void *userdata, int error, const char *detail) +{ + AVCodecContext *avctx = userdata; + MediaCodecEncContext *s = avctx->priv_data; + + if (error == AVERROR(EAGAIN)) + return; + + av_log(avctx, AV_LOG_ERROR, "On error, %s, %s\n", av_err2str(error), detail); + + pthread_mutex_lock(&s->input_mutex); + pthread_mutex_lock(&s->output_mutex); + s->encode_status = error; + pthread_mutex_unlock(&s->output_mutex); + pthread_mutex_unlock(&s->input_mutex); + + pthread_cond_signal(&s->output_cond); + pthread_cond_signal(&s->input_cond); +} + +static int mediacodec_init_async_state(AVCodecContext *avctx) +{ + MediaCodecEncContext *s = avctx->priv_data; + size_t fifo_size = 16; + + if (!s->async_mode) + return 0; + + pthread_mutex_init(&s->input_mutex, NULL); + pthread_cond_init(&s->input_cond, NULL); + + pthread_mutex_init(&s->output_mutex, NULL); + pthread_cond_init(&s->output_cond, NULL); + + s->input_index = av_fifo_alloc2(fifo_size, sizeof(int32_t), AV_FIFO_FLAG_AUTO_GROW); + s->output_index = av_fifo_alloc2(fifo_size, sizeof(int32_t), AV_FIFO_FLAG_AUTO_GROW); + s->output_buf_info = av_fifo_alloc2(fifo_size, sizeof(FFAMediaCodecBufferInfo), AV_FIFO_FLAG_AUTO_GROW); + + if (!s->input_index || !s->output_index || !s->output_buf_info) + return AVERROR(ENOMEM); + + return 0; +} + +static void mediacodec_uninit_async_state(AVCodecContext *avctx) +{ + MediaCodecEncContext *s = avctx->priv_data; + + if (!s->async_mode) + return; + + pthread_mutex_destroy(&s->input_mutex); + pthread_cond_destroy(&s->input_cond); + + pthread_mutex_destroy(&s->output_mutex); + pthread_cond_destroy(&s->output_cond); + + av_fifo_freep2(&s->input_index); + av_fifo_freep2(&s->output_index); + av_fifo_freep2(&s->output_buf_info); + + s->async_mode = 0; +} + static av_cold int mediacodec_init(AVCodecContext *avctx) { const char *codec_mime = NULL; @@ -155,6 +301,11 @@ static av_cold int mediacodec_init(AVCodecContext *avctx) int ret; int gop; + // Init async state first, so we can do cleanup safely on error path. + ret = mediacodec_init_async_state(avctx); + if (ret < 0) + return ret; + if (s->use_ndk_codec < 0) s->use_ndk_codec = !av_jni_get_java_vm(avctx); @@ -322,10 +473,21 @@ static av_cold int mediacodec_init(AVCodecContext *avctx) goto bailout; } - ret = ff_AMediaCodec_start(s->codec); - if (ret) { - av_log(avctx, AV_LOG_ERROR, "MediaCodec failed to start, %s\n", av_err2str(ret)); - goto bailout; + if (s->async_mode) { + FFAMediaCodecOnAsyncNotifyCallback cb = { + .onAsyncInputAvailable = on_input_available, + .onAsyncOutputAvailable = on_output_available, + .onAsyncFormatChanged = on_format_changed, + .onAsyncError = on_error, + }; + + ret = ff_AMediaCodec_setAsyncNotifyCallback(s->codec, &cb, avctx); + if (ret < 0) { + av_log(avctx, AV_LOG_WARNING, + "Try MediaCodec async mode failed, %s, switch to sync mode\n", + av_err2str(ret)); + mediacodec_uninit_async_state(avctx); + } } ret = mediacodec_init_bsf(avctx); @@ -339,8 +501,16 @@ static av_cold int mediacodec_init(AVCodecContext *avctx) "Use extract_extradata bsf when necessary.\n"); s->frame = av_frame_alloc(); - if (!s->frame) + if (!s->frame) { ret = AVERROR(ENOMEM); + goto bailout; + } + + ret = ff_AMediaCodec_start(s->codec); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "MediaCodec failed to start, %s\n", av_err2str(ret)); + goto bailout; + } bailout: if (format) @@ -348,19 +518,63 @@ bailout: return ret; } +static int mediacodec_get_output_index(AVCodecContext *avctx, ssize_t *index, + FFAMediaCodecBufferInfo *out_info) +{ + MediaCodecEncContext *s = avctx->priv_data; + FFAMediaCodec *codec = s->codec; + int64_t timeout_us = s->eof_sent ? OUTPUT_DEQUEUE_TIMEOUT_US : 0; + int n; + int ret; + + if (!s->async_mode) { + *index = ff_AMediaCodec_dequeueOutputBuffer(codec, out_info, timeout_us); + return 0; + } + + pthread_mutex_lock(&s->output_mutex); + + n = -1; + while (n < 0 && !s->encode_status) { + if (av_fifo_can_read(s->output_index)) { + av_fifo_read(s->output_index, &n, 1); + av_fifo_read(s->output_buf_info, out_info, 1); + break; + } + + if (n < 0 && s->eof_sent && !s->encode_status) + pthread_cond_wait(&s->output_cond, &s->output_mutex); + else + break; + } + + ret = s->encode_status; + *index = n; + pthread_mutex_unlock(&s->output_mutex); + + // Get output index success + if (*index >= 0) + return 0; + + return ret ? ret : AVERROR(EAGAIN); +} + static int mediacodec_receive(AVCodecContext *avctx, AVPacket *pkt, int *got_packet) { MediaCodecEncContext *s = avctx->priv_data; FFAMediaCodec *codec = s->codec; + ssize_t index; FFAMediaCodecBufferInfo out_info = {0}; uint8_t *out_buf; size_t out_size = 0; int ret; int extradata_size = 0; - int64_t timeout_us = s->eof_sent ? OUTPUT_DEQUEUE_TIMEOUT_US : 0; - ssize_t index = ff_AMediaCodec_dequeueOutputBuffer(codec, &out_info, timeout_us); + + ret = mediacodec_get_output_index(avctx, &index, &out_info); + if (ret < 0) + return ret; if (ff_AMediaCodec_infoTryAgainLater(codec, index)) return AVERROR(EAGAIN); @@ -426,31 +640,36 @@ bailout: return ret; } -static void copy_frame_to_buffer(AVCodecContext *avctx, const AVFrame *frame, uint8_t *dst, size_t size) +static int mediacodec_get_input_index(AVCodecContext *avctx, ssize_t *index) { MediaCodecEncContext *s = avctx->priv_data; - uint8_t *dst_data[4] = {}; - int dst_linesize[4] = {}; + FFAMediaCodec *codec = s->codec; + int ret = 0; + int32_t n; - if (avctx->pix_fmt == AV_PIX_FMT_YUV420P) { - dst_data[0] = dst; - dst_data[1] = dst + s->width * s->height; - dst_data[2] = dst_data[1] + s->width * s->height / 4; + if (!s->async_mode) { + *index = ff_AMediaCodec_dequeueInputBuffer(codec, INPUT_DEQUEUE_TIMEOUT_US); + return 0; + } - dst_linesize[0] = s->width; - dst_linesize[1] = dst_linesize[2] = s->width / 2; - } else if (avctx->pix_fmt == AV_PIX_FMT_NV12) { - dst_data[0] = dst; - dst_data[1] = dst + s->width * s->height; + pthread_mutex_lock(&s->input_mutex); - dst_linesize[0] = s->width; - dst_linesize[1] = s->width; - } else { - av_assert0(0); + n = -1; + while (n < 0 && !s->encode_status) { + if (av_fifo_can_read(s->input_index) > 0) { + av_fifo_read(s->input_index, &n, 1); + break; + } + + if (n < 0 && !s->encode_status) + pthread_cond_wait(&s->input_cond, &s->input_mutex); } - av_image_copy2(dst_data, dst_linesize, frame->data, frame->linesize, - avctx->pix_fmt, avctx->width, avctx->height); + ret = s->encode_status; + *index = n; + pthread_mutex_unlock(&s->input_mutex); + + return ret; } static int mediacodec_send(AVCodecContext *avctx, @@ -462,7 +681,7 @@ static int mediacodec_send(AVCodecContext *avctx, size_t input_size = 0; int64_t pts = 0; uint32_t flags = 0; - int64_t timeout_us; + int ret; if (s->eof_sent) return 0; @@ -478,8 +697,10 @@ static int mediacodec_send(AVCodecContext *avctx, return 0; } - timeout_us = INPUT_DEQUEUE_TIMEOUT_US; - index = ff_AMediaCodec_dequeueInputBuffer(codec, timeout_us); + ret = mediacodec_get_input_index(avctx, &index); + if (ret < 0) + return ret; + if (ff_AMediaCodec_infoTryAgainLater(codec, index)) return AVERROR(EAGAIN); @@ -566,6 +787,8 @@ static av_cold int mediacodec_close(AVCodecContext *avctx) av_bsf_free(&s->bsf); av_frame_free(&s->frame); + mediacodec_uninit_async_state(avctx); + return 0; } @@ -587,6 +810,8 @@ static const AVCodecHWConfigInternal *const mediacodec_hw_configs[] = { #define COMMON_OPTION \ { "ndk_codec", "Use MediaCodec from NDK", \ OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1}, -1, 1, VE }, \ + { "ndk_async", "Try NDK MediaCodec in async mode", \ + OFFSET(async_mode), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, VE }, \ { "codec_name", "Select codec by name", \ OFFSET(name), AV_OPT_TYPE_STRING, {0}, 0, 0, VE }, \ { "bitrate_mode", "Bitrate control method", \