From patchwork Mon Aug 26 12:54:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthieu Bouron X-Patchwork-Id: 51146 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:b3c7:0:b0:48e:c0f8:d0de with SMTP id g7csp1745048vqt; Mon, 26 Aug 2024 05:55:33 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUNpPA/us0StS1zG9SY6XBQq9S5w+J1bT7Jonet5xLLxd6M5EJjpgsYkZngPgkGASYZSONzEhcGjwLSt4QeVAU1@gmail.com X-Google-Smtp-Source: AGHT+IG0l+RIjWpOodlAX2mxlOYWTt1YcH/vC7R4xoh52t44tB25r3ij2qQVU3S+gTqG9++w/WdA X-Received: by 2002:a05:6512:1089:b0:52f:c337:4c1f with SMTP id 2adb3069b0e04-5343870bfe1mr4105447e87.0.1724676933323; Mon, 26 Aug 2024 05:55:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1724676933; cv=none; d=google.com; s=arc-20240605; b=VHIPzp7M+0iVMuvBSfX6oM1B1TY/noS5UxBfJpnLWns04x5YnQ/sCnzXM8eQhqhLXb bKz8c13GlU39fEr/s4705gK19apC6ocuOBk+tMw4RoYVaaMvjGhq1L/vYG6+zHkU8z/v yG3LW+CodFA3aLCFxqYQalTSKXETWnGpMZdwDwSSdrj3USPYWgab/MtKEhlJYIU+alxj x+KCRod457jopsRZJyFHUQ6mu710CKaqxeSaJNmdltnELeOkl44reTjgr4QcD9+FlmPi zFfCRpOp5ghHdSlk1L2fOZr6hMku2+9vB5pxXsZBLB9/iuGI2A/VxrlENXvhC0vgBuMk i6uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=VhQNZ9X3m47RUM4q2IhOqr/s8cCRu0y8GRbSXKO2sQ4=; fh=bsOL+ub9JEnfNazRgCdgjxJAChubFVskZxBzYcxeDLU=; b=iNjkdI4CqLdJ9Fur6s+hW3XjeWWf2pQzwyho752B9yoH/9Tn0dDwCQqu2f+JwjmxH2 xD9c4swh/d+ZapTu4J3rFghY8/Pu90xfwedwCKtFMIjyplY3VF1sH6JWdnEVD+gJB6hK QCA0yegUdChkJ+X2naQLW6/YvYyASJfqWrwE9r/t3yqpCjuwxR96MyulnqiPv/BeZjXf wso/MdDOgAlIu5NLWrNXWBwj6uic413J/awZVJRDqlmYiw7mxxdzwgdLy7uMLddlF3Ps gdg2D7DFDwWymjq0QUVUEqk2LqlDhenUZ8ue/UtwNUWmEmaqyKF22FlKda3uNsIgSl7k UNZQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=eqXaC3Ri; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-5334ea2b5c0si3332489e87.62.2024.08.26.05.55.32; Mon, 26 Aug 2024 05:55:33 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=eqXaC3Ri; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0DDEE68DBF2; Mon, 26 Aug 2024 15:55:28 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AB52B68DA05 for ; Mon, 26 Aug 2024 15:55:21 +0300 (EEST) Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-42809d6e719so37460645e9.3 for ; Mon, 26 Aug 2024 05:55:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724676921; x=1725281721; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=klTDsg8gGEGhrwtWhaxV+Tg8yquQc6i4Pb/1RwdavAI=; b=eqXaC3Ri3izs8hm0RuGdnPj7Cvc4rj0rP4wlVzRMQ0dPsmuZVH5ZnxBVg1Lb9YMwzA YsPIP1pAiq2JdR7k0nVTls9WMyHzfHTJ9ymdunKSjDgwh74lEHx5K7PB6v/L5kjzdk6R EX6FZuwGGyT6RxpG1KHsGPZsNf3XqQoOHJiR+r1IKW2CjxUVkQWFSIlzVX6iSe+kroM9 4bcZY+zc9LETCeWYl9p7ShwJQPT90aOCN3N1EDlHX6oWqLFlIoOuCg6MpltMxsfM3lQY LfwcMOoUDw6w0XMEDOXrTWcipMsjQk62O/utvf7Yw132suUp/btnZkhrkCUWaPzSZnuV YsPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724676921; x=1725281721; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=klTDsg8gGEGhrwtWhaxV+Tg8yquQc6i4Pb/1RwdavAI=; b=Lljhpoct42Kw9mMGqw3qEOG7Hx5pFkVqHbk9pOSMqXXVH7fn0fm0akGdgU5cOjZRhO Gj1VHGs4WR0hypvdcTZc8oonK7cU4ZLdS/ImMbtbm/G049Obp1XiihqgFIFmqspRA1CZ T726KVqk9Msc1idHM17+J+z8/uej5BoRdeJfhfnP2xELZTMUY0F/yQwQ/XnX/696+n9T C6tEPmT8SCHvOPuqOk3BNzSSSfJQYlTxLFo8TRMbemrK3HNT3nnTlLU+RZ3Vil3l+ojI EM0luUj//uSLIrmIH/Gy8MFfU5b7tI/Lho2VTbTwJn6ONJmLudKsNddehuhLKZvPw6YR dZqw== X-Gm-Message-State: AOJu0YwfW6jYjKCDTekyCG/GAE9Umzo+SrWJGW1ntG6EYHChmX7CKF6L WqBp2kN529zG0l7FAzLc/x1BiZb5XGYGYtmhO6MZmvqSldvSamR1ypYkeg== X-Received: by 2002:a05:600c:150a:b0:42b:898a:1ce4 with SMTP id 5b1f17b1804b1-42b898a1e1cmr58218825e9.34.1724676920129; Mon, 26 Aug 2024 05:55:20 -0700 (PDT) Received: from localhost.localdomain (2a01cb040b6872000000000000000afa.ipv6.abo.wanadoo.fr. [2a01:cb04:b68:7200::afa]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42abefc627asm192206655e9.34.2024.08.26.05.55.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 05:55:19 -0700 (PDT) From: Matthieu Bouron To: ffmpeg-devel@ffmpeg.org Date: Mon, 26 Aug 2024 14:54:35 +0200 Message-ID: <20240826125510.9115-1-matthieu.bouron@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <172467357225.21344.1953328875328052119@lain.khirnov.net> References: <172467357225.21344.1953328875328052119@lain.khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3] avcodec: add Mediacodec audio decoders support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Matthieu Bouron Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: mXvnqtDJUoeL --- Diff with v2: - Dropped flac/vorbis/opus support --- configure | 8 + libavcodec/Makefile | 4 + libavcodec/allcodecs.c | 4 + libavcodec/mediacodecdec.c | 102 ++++++++- libavcodec/mediacodecdec_common.c | 333 +++++++++++++++++++++++++++--- 5 files changed, 420 insertions(+), 31 deletions(-) diff --git a/configure b/configure index 0fd7901581..e454b44655 100755 --- a/configure +++ b/configure @@ -3324,8 +3324,14 @@ amf_deps_any="libdl LoadLibrary" nvenc_deps="ffnvcodec" nvenc_deps_any="libdl LoadLibrary" +aac_mediacodec_decoder_deps="mediacodec" +aac_mediacodec_decoder_select="aac_adtstoasc_bsf aac_parser" aac_mf_encoder_deps="mediafoundation" ac3_mf_encoder_deps="mediafoundation" +amrnb_mediacodec_decoder_deps="mediacodec" +amrnb_mediacodec_decoder_select="amr_parser" +amrwb_mediacodec_decoder_deps="mediacodec" +amrwb_mediacodec_decoder_select="amr_parser" av1_amf_encoder_deps="amf" av1_cuvid_decoder_deps="cuvid CUVIDAV1PICPARAMS" av1_mediacodec_decoder_deps="mediacodec" @@ -3387,6 +3393,8 @@ mjpeg_qsv_encoder_select="qsvenc" mjpeg_vaapi_encoder_deps="VAEncPictureParameterBufferJPEG" mjpeg_vaapi_encoder_select="cbs_jpeg jpegtables vaapi_encode" mp3_mf_encoder_deps="mediafoundation" +mp3_mediacodec_decoder_deps="mediacodec" +mp3_mediacodec_decoder_select="mpegaudioheader" mpeg1_cuvid_decoder_deps="cuvid" mpeg1_v4l2m2m_decoder_deps="v4l2_m2m mpeg1_v4l2_m2m" mpeg2_cuvid_decoder_deps="cuvid" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 262d0a3d3e..eab8bfac01 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -197,6 +197,7 @@ OBJS-$(CONFIG_AAC_ENCODER) += aacenc.o aaccoder.o aacenctab.o \ aacenc_pred.o \ psymodel.o kbdwin.o \ mpeg4audio_sample_rates.o +OBJS-$(CONFIG_AAC_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_AAC_MF_ENCODER) += mfenc.o mf_utils.o OBJS-$(CONFIG_AASC_DECODER) += aasc.o msrledec.o OBJS-$(CONFIG_AC3_DECODER) += ac3dec_float.o ac3dec_data.o ac3.o \ @@ -223,6 +224,8 @@ OBJS-$(CONFIG_AMRWB_DECODER) += amrwbdec.o celp_filters.o \ celp_math.o acelp_filters.o \ acelp_vectors.o \ acelp_pitch_delay.o +OBJS-$(CONFIG_AMRNB_MEDIACODEC_DECODER) += mediacodecdec.o +OBJS-$(CONFIG_AMRWB_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_AMV_ENCODER) += mjpegenc.o mjpegenc_common.o OBJS-$(CONFIG_ANM_DECODER) += anm.o OBJS-$(CONFIG_ANULL_DECODER) += null.o @@ -521,6 +524,7 @@ OBJS-$(CONFIG_MP2FIXED_ENCODER) += mpegaudioenc_fixed.o mpegaudio.o \ mpegaudiotabs.o OBJS-$(CONFIG_MP2FLOAT_DECODER) += mpegaudiodec_float.o OBJS-$(CONFIG_MP3_DECODER) += mpegaudiodec_fixed.o +OBJS-$(CONFIG_MP3_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_MP3_MF_ENCODER) += mfenc.o mf_utils.o OBJS-$(CONFIG_MP3ADU_DECODER) += mpegaudiodec_fixed.o OBJS-$(CONFIG_MP3ADUFLOAT_DECODER) += mpegaudiodec_float.o diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c index 09385be4ee..563afde355 100644 --- a/libavcodec/allcodecs.c +++ b/libavcodec/allcodecs.c @@ -822,8 +822,11 @@ extern const FFCodec ff_idf_decoder; /* external libraries, that shouldn't be used by default if one of the * above is available */ +extern const FFCodec ff_aac_mediacodec_decoder; extern const FFCodec ff_aac_mf_encoder; extern const FFCodec ff_ac3_mf_encoder; +extern const FFCodec ff_amrnb_mediacodec_decoder; +extern const FFCodec ff_amrwb_mediacodec_decoder; extern const FFCodec ff_h263_v4l2m2m_encoder; extern const FFCodec ff_libaom_av1_decoder; /* hwaccel hooks only, so prefer external decoders */ @@ -863,6 +866,7 @@ extern const FFCodec ff_mjpeg_cuvid_decoder; extern const FFCodec ff_mjpeg_qsv_encoder; extern const FFCodec ff_mjpeg_qsv_decoder; extern const FFCodec ff_mjpeg_vaapi_encoder; +extern const FFCodec ff_mp3_mediacodec_decoder; extern const FFCodec ff_mp3_mf_encoder; extern const FFCodec ff_mpeg1_cuvid_decoder; extern const FFCodec ff_mpeg2_cuvid_decoder; diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c index 6d8dc600fe..6e1e7034ca 100644 --- a/libavcodec/mediacodecdec.c +++ b/libavcodec/mediacodecdec.c @@ -291,7 +291,11 @@ done: CONFIG_MPEG4_MEDIACODEC_DECODER || \ CONFIG_VP8_MEDIACODEC_DECODER || \ CONFIG_VP9_MEDIACODEC_DECODER || \ - CONFIG_AV1_MEDIACODEC_DECODER + CONFIG_AV1_MEDIACODEC_DECODER || \ + CONFIG_AAC_MEDIACODEC_DECODER || \ + CONFIG_AMRNB_MEDIACODEC_DECODER || \ + CONFIG_AMRWB_MEDIACODEC_DECODER || \ + CONFIG_MP3_MEDIACODEC_DECODER static int common_set_extradata(AVCodecContext *avctx, FFAMediaFormat *format) { int ret = 0; @@ -388,13 +392,55 @@ static av_cold int mediacodec_decode_init(AVCodecContext *avctx) goto done; break; #endif +#if CONFIG_AAC_MEDIACODEC_DECODER + case AV_CODEC_ID_AAC: + codec_mime = "audio/mp4a-latm"; + + ret = common_set_extradata(avctx, format); + if (ret < 0) + goto done; + break; +#endif +#if CONFIG_AMRNB_MEDIACODEC_DECODER + case AV_CODEC_ID_AMR_NB: + codec_mime = "audio/3gpp"; + + ret = common_set_extradata(avctx, format); + if (ret < 0) + goto done; + break; +#endif +#if CONFIG_AMRWB_MEDIACODEC_DECODER + case AV_CODEC_ID_AMR_WB: + codec_mime = "audio/amr-wb"; + + ret = common_set_extradata(avctx, format); + if (ret < 0) + goto done; + break; +#endif +#if CONFIG_MP3_MEDIACODEC_DECODER + case AV_CODEC_ID_MP3: + codec_mime = "audio/mpeg"; + + ret = common_set_extradata(avctx, format); + if (ret < 0) + goto done; + break; +#endif default: av_assert0(0); } ff_AMediaFormat_setString(format, "mime", codec_mime); - ff_AMediaFormat_setInt32(format, "width", avctx->width); - ff_AMediaFormat_setInt32(format, "height", avctx->height); + + if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) { + ff_AMediaFormat_setInt32(format, "width", avctx->width); + ff_AMediaFormat_setInt32(format, "height", avctx->height); + } else { + ff_AMediaFormat_setInt32(format, "channel-count", avctx->ch_layout.nb_channels); + ff_AMediaFormat_setInt32(format, "sample-rate", avctx->sample_rate); + } s->ctx = av_mallocz(sizeof(*s->ctx)); if (!s->ctx) { @@ -611,3 +657,53 @@ DECLARE_MEDIACODEC_VDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL) #if CONFIG_AV1_MEDIACODEC_DECODER DECLARE_MEDIACODEC_VDEC(av1, "AV1", AV_CODEC_ID_AV1, NULL) #endif + +#define AD AV_OPT_FLAG_AUDIO_PARAM | AV_OPT_FLAG_DECODING_PARAM +static const AVOption ff_mediacodec_adec_options[] = { + { "ndk_codec", "Use MediaCodec from NDK", + OFFSET(use_ndk_codec), AV_OPT_TYPE_BOOL, {.i64 = -1}, -1, 1, AD }, + { NULL } +}; + +#define DECLARE_MEDIACODEC_ACLASS(short_name) \ +static const AVClass ff_##short_name##_mediacodec_dec_class = { \ + .class_name = #short_name "_mediacodec", \ + .item_name = av_default_item_name, \ + .option = ff_mediacodec_adec_options, \ + .version = LIBAVUTIL_VERSION_INT, \ +}; + +#define DECLARE_MEDIACODEC_ADEC(short_name, full_name, codec_id, bsf) \ +DECLARE_MEDIACODEC_VCLASS(short_name) \ +const FFCodec ff_ ## short_name ## _mediacodec_decoder = { \ + .p.name = #short_name "_mediacodec", \ + CODEC_LONG_NAME(full_name " Android MediaCodec decoder"), \ + .p.type = AVMEDIA_TYPE_AUDIO, \ + .p.id = codec_id, \ + .p.priv_class = &ff_##short_name##_mediacodec_dec_class, \ + .priv_data_size = sizeof(MediaCodecH264DecContext), \ + .init = mediacodec_decode_init, \ + FF_CODEC_RECEIVE_FRAME_CB(mediacodec_receive_frame), \ + .flush = mediacodec_decode_flush, \ + .close = mediacodec_decode_close, \ + .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE, \ + .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE, \ + .bsfs = bsf, \ + .p.wrapper_name = "mediacodec", \ +}; \ + +#if CONFIG_AAC_MEDIACODEC_DECODER +DECLARE_MEDIACODEC_ADEC(aac, "AAC", AV_CODEC_ID_AAC, "aac_adtstoasc") +#endif + +#if CONFIG_AMRNB_MEDIACODEC_DECODER +DECLARE_MEDIACODEC_ADEC(amrnb, "AMR-NB", AV_CODEC_ID_AMR_NB, NULL) +#endif + +#if CONFIG_AMRWB_MEDIACODEC_DECODER +DECLARE_MEDIACODEC_ADEC(amrwb, "AMR-WB", AV_CODEC_ID_AMR_WB, NULL) +#endif + +#if CONFIG_MP3_MEDIACODEC_DECODER +DECLARE_MEDIACODEC_ADEC(mp3, "MP3", AV_CODEC_ID_MP3, NULL) +#endif diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c index c888dea8cf..b1ee8b609a 100644 --- a/libavcodec/mediacodecdec_common.c +++ b/libavcodec/mediacodecdec_common.c @@ -23,6 +23,7 @@ #include #include +#include "libavutil/avassert.h" #include "libavutil/common.h" #include "libavutil/hwcontext_mediacodec.h" #include "libavutil/mem.h" @@ -30,6 +31,7 @@ #include "libavutil/pixfmt.h" #include "libavutil/time.h" #include "libavutil/timestamp.h" +#include "libavutil/channel_layout.h" #include "avcodec.h" #include "decode.h" @@ -85,6 +87,107 @@ #define OUTPUT_DEQUEUE_TIMEOUT_US 8000 #define OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US 1000000 +enum { + ENCODING_PCM_16BIT = 0x00000002, + ENCODING_PCM_8BIT = 0x00000003, + ENCODING_PCM_FLOAT = 0x00000004, + ENCODING_PCM_24BIT_PACKED = 0x00000015, + ENCODING_PCM_32BIT = 0x00000016, +}; + +static const struct { + + int pcm_format; + enum AVSampleFormat sample_format; + +} sample_formats[] = { + + { ENCODING_PCM_16BIT, AV_SAMPLE_FMT_S16 }, + { ENCODING_PCM_8BIT, AV_SAMPLE_FMT_U8 }, + { ENCODING_PCM_FLOAT, AV_SAMPLE_FMT_FLT }, + { ENCODING_PCM_32BIT, AV_SAMPLE_FMT_S32 }, + { 0 } +}; + +static enum AVSampleFormat mcdec_map_pcm_format(AVCodecContext *avctx, + MediaCodecDecContext *s, + int pcm_format) +{ + enum AVSampleFormat ret = AV_SAMPLE_FMT_NONE; + + for (int i = 0; i < FF_ARRAY_ELEMS(sample_formats); i++) { + if (sample_formats[i].pcm_format == pcm_format) { + return sample_formats[i].sample_format; + } + } + + av_log(avctx, AV_LOG_ERROR, "Output sample format 0x%x (value=%d) is not supported\n", + pcm_format, pcm_format); + + return ret; +} + +enum +{ + CHANNEL_OUT_FRONT_LEFT = 0x4, + CHANNEL_OUT_FRONT_RIGHT = 0x8, + CHANNEL_OUT_FRONT_CENTER = 0x10, + CHANNEL_OUT_LOW_FREQUENCY = 0x20, + CHANNEL_OUT_BACK_LEFT = 0x40, + CHANNEL_OUT_BACK_RIGHT = 0x80, + CHANNEL_OUT_FRONT_LEFT_OF_CENTER = 0x100, + CHANNEL_OUT_FRONT_RIGHT_OF_CENTER = 0x200, + CHANNEL_OUT_BACK_CENTER = 0x400, + CHANNEL_OUT_SIDE_LEFT = 0x800, + CHANNEL_OUT_SIDE_RIGHT = 0x1000, + CHANNEL_OUT_TOP_CENTER = 0x2000, + CHANNEL_OUT_TOP_FRONT_LEFT = 0x4000, + CHANNEL_OUT_TOP_FRONT_CENTER = 0x8000, + CHANNEL_OUT_TOP_FRONT_RIGHT = 0x10000, + CHANNEL_OUT_TOP_BACK_LEFT = 0x20000, + CHANNEL_OUT_TOP_BACK_CENTER = 0x40000, + CHANNEL_OUT_TOP_BACK_RIGHT = 0x80000, +}; + +static const struct { + + int mask; + uint64_t layout; + +} channel_masks[] = { + { CHANNEL_OUT_FRONT_LEFT, AV_CH_FRONT_LEFT }, + { CHANNEL_OUT_FRONT_RIGHT, AV_CH_FRONT_RIGHT }, + { CHANNEL_OUT_FRONT_CENTER, AV_CH_FRONT_CENTER }, + { CHANNEL_OUT_LOW_FREQUENCY, AV_CH_LOW_FREQUENCY }, + { CHANNEL_OUT_BACK_LEFT, AV_CH_BACK_LEFT }, + { CHANNEL_OUT_BACK_RIGHT, AV_CH_BACK_RIGHT }, + { CHANNEL_OUT_FRONT_LEFT_OF_CENTER, AV_CH_FRONT_LEFT_OF_CENTER }, + { CHANNEL_OUT_FRONT_RIGHT_OF_CENTER, AV_CH_FRONT_RIGHT_OF_CENTER }, + { CHANNEL_OUT_BACK_CENTER, AV_CH_BACK_CENTER }, + { CHANNEL_OUT_SIDE_LEFT, AV_CH_SIDE_LEFT }, + { CHANNEL_OUT_SIDE_RIGHT, AV_CH_SIDE_RIGHT }, + { CHANNEL_OUT_TOP_CENTER, AV_CH_TOP_CENTER }, + { CHANNEL_OUT_TOP_FRONT_LEFT, AV_CH_TOP_FRONT_LEFT }, + { CHANNEL_OUT_TOP_FRONT_CENTER, AV_CH_TOP_FRONT_CENTER }, + { CHANNEL_OUT_TOP_FRONT_RIGHT, AV_CH_TOP_FRONT_RIGHT }, + { CHANNEL_OUT_TOP_BACK_LEFT, AV_CH_TOP_BACK_LEFT }, + { CHANNEL_OUT_TOP_BACK_CENTER, AV_CH_TOP_BACK_CENTER }, + { CHANNEL_OUT_TOP_BACK_RIGHT, AV_CH_TOP_BACK_RIGHT }, +}; + +static uint64_t mcdec_map_channel_mask(AVCodecContext *avctx, + int channel_mask) +{ + uint64_t channel_layout = 0; + + for (int i = 0; i < FF_ARRAY_ELEMS(channel_masks); i++) { + if (channel_mask & channel_masks[i].mask) + channel_layout |= channel_masks[i].layout; + } + + return channel_layout; +} + enum { COLOR_FormatYUV420Planar = 0x13, COLOR_FormatYUV420SemiPlanar = 0x15, @@ -265,13 +368,79 @@ fail: return ret; } -static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx, - MediaCodecDecContext *s, - uint8_t *data, - size_t size, - ssize_t index, - FFAMediaCodecBufferInfo *info, - AVFrame *frame) +static int mediacodec_wrap_sw_audio_buffer(AVCodecContext *avctx, + MediaCodecDecContext *s, + uint8_t *data, + size_t size, + ssize_t index, + FFAMediaCodecBufferInfo *info, + AVFrame *frame) +{ + int ret = 0; + int status = 0; + const int sample_size = av_get_bytes_per_sample(avctx->sample_fmt); + if (!sample_size) { + av_log(avctx, AV_LOG_ERROR, "Could not get bytes per sample\n"); + ret = AVERROR(ENOSYS); + goto done; + } + + frame->format = avctx->sample_fmt; + frame->sample_rate = avctx->sample_rate; + frame->nb_samples = info->size / (sample_size * avctx->ch_layout.nb_channels); + + ret = av_channel_layout_copy(&frame->ch_layout, &avctx->ch_layout); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, "Could not copy channel layout\n"); + goto done; + } + + /* MediaCodec buffers needs to be copied to our own refcounted buffers + * because the flush command invalidates all input and output buffers. + */ + ret = ff_get_buffer(avctx, frame, 0); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, "Could not allocate buffer\n"); + goto done; + } + + /* Override frame->pts as ff_get_buffer will override its value based + * on the last avpacket received which is not in sync with the frame: + * * N avpackets can be pushed before 1 frame is actually returned + * * 0-sized avpackets are pushed to flush remaining frames at EOS */ + if (avctx->pkt_timebase.num && avctx->pkt_timebase.den) { + frame->pts = av_rescale_q(info->presentationTimeUs, + AV_TIME_BASE_Q, + avctx->pkt_timebase); + } else { + frame->pts = info->presentationTimeUs; + } + frame->pkt_dts = AV_NOPTS_VALUE; + + av_log(avctx, AV_LOG_TRACE, + "Frame: format=%d channels=%d sample_rate=%d nb_samples=%d", + avctx->sample_fmt, avctx->ch_layout.nb_channels, avctx->sample_rate, frame->nb_samples); + + memcpy(frame->data[0], data, info->size); + + ret = 0; +done: + status = ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0); + if (status < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to release output buffer\n"); + ret = AVERROR_EXTERNAL; + } + + return ret; +} + +static int mediacodec_wrap_sw_video_buffer(AVCodecContext *avctx, + MediaCodecDecContext *s, + uint8_t *data, + size_t size, + ssize_t index, + FFAMediaCodecBufferInfo *info, + AVFrame *frame) { int ret = 0; int status = 0; @@ -343,6 +512,22 @@ done: return ret; } +static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx, + MediaCodecDecContext *s, + uint8_t *data, + size_t size, + ssize_t index, + FFAMediaCodecBufferInfo *info, + AVFrame *frame) +{ + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) + return mediacodec_wrap_sw_audio_buffer(avctx, s, data, size, index, info, frame); + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) + return mediacodec_wrap_sw_video_buffer(avctx, s, data, size, index, info, frame); + else + av_assert0(0); +} + #define AMEDIAFORMAT_GET_INT32(name, key, mandatory) do { \ int32_t value = 0; \ if (ff_AMediaFormat_getInt32(s->format, key, &value)) { \ @@ -354,7 +539,7 @@ done: } \ } while (0) \ -static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecContext *s) +static int mediacodec_dec_parse_video_format(AVCodecContext *avctx, MediaCodecDecContext *s) { int ret = 0; int width = 0; @@ -463,6 +648,63 @@ fail: return ret; } +static int mediacodec_dec_parse_audio_format(AVCodecContext *avctx, MediaCodecDecContext *s) +{ + int ret = 0; + int sample_rate = 0; + int channel_count = 0; + int channel_mask = 0; + int pcm_encoding = 0; + char *format = NULL; + + if (!s->format) { + av_log(avctx, AV_LOG_ERROR, "Output MediaFormat is not set\n"); + return AVERROR(EINVAL); + } + + format = ff_AMediaFormat_toString(s->format); + if (!format) { + return AVERROR_EXTERNAL; + } + av_log(avctx, AV_LOG_DEBUG, "Parsing MediaFormat %s\n", format); + + /* Mandatory fields */ + AMEDIAFORMAT_GET_INT32(channel_count, "channel-count", 1); + AMEDIAFORMAT_GET_INT32(sample_rate, "sample-rate", 1); + + AMEDIAFORMAT_GET_INT32(pcm_encoding, "pcm-encoding", 0); + if (pcm_encoding) + avctx->sample_fmt = mcdec_map_pcm_format(avctx, s, pcm_encoding); + else + avctx->sample_fmt = AV_SAMPLE_FMT_S16; + + avctx->sample_rate = sample_rate; + + AMEDIAFORMAT_GET_INT32(channel_mask, "channel-mask", 0); + if (channel_mask) + av_channel_layout_from_mask(&avctx->ch_layout, mcdec_map_channel_mask(avctx, channel_mask)); + else + av_channel_layout_default(&avctx->ch_layout, channel_count); + + av_log(avctx, AV_LOG_INFO, + "Output parameters channel-count=%d channel-layout=%x sample-rate=%d\n", + channel_count, channel_mask, sample_rate); + +fail: + av_freep(&format); + return ret; +} + +static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecContext *s) +{ + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) + return mediacodec_dec_parse_audio_format(avctx, s); + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) + return mediacodec_dec_parse_video_format(avctx, s); + else + av_assert0(0); +} + static int mediacodec_dec_flush_codec(AVCodecContext *avctx, MediaCodecDecContext *s) { FFAMediaCodec *codec = s->codec; @@ -486,11 +728,9 @@ static int mediacodec_dec_flush_codec(AVCodecContext *avctx, MediaCodecDecContex return 0; } -int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, - const char *mime, FFAMediaFormat *format) +static int mediacodec_dec_get_video_codec(AVCodecContext *avctx, MediaCodecDecContext *s, + const char *mime, FFAMediaFormat *format) { - int ret = 0; - int status; int profile; enum AVPixelFormat pix_fmt; @@ -499,12 +739,6 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, AV_PIX_FMT_NONE, }; - s->avctx = avctx; - atomic_init(&s->refcount, 1); - atomic_init(&s->hw_buffer_count, 0); - atomic_init(&s->serial, 1); - s->current_input_buffer = -1; - pix_fmt = ff_get_format(avctx, pix_fmts); if (pix_fmt == AV_PIX_FMT_MEDIACODEC) { AVMediaCodecContext *user_ctx = avctx->hwaccel_context; @@ -536,8 +770,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, // getCodecNameByType() can fail due to missing JVM, while NDK // mediacodec can be used without JVM. if (!s->use_ndk_codec) { - ret = AVERROR_EXTERNAL; - goto fail; + return AVERROR_EXTERNAL; } av_log(avctx, AV_LOG_INFO, "Failed to getCodecNameByType\n"); } else { @@ -556,10 +789,52 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, } if (!s->codec) { av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for type %s and name %s\n", mime, s->codec_name); - ret = AVERROR_EXTERNAL; - goto fail; + return AVERROR_EXTERNAL; + } + + return 0; +} + +static int mediacodec_dec_get_audio_codec(AVCodecContext *avctx, MediaCodecDecContext *s, + const char *mime, FFAMediaFormat *format) +{ + s->codec = ff_AMediaCodec_createDecoderByType(mime, s->use_ndk_codec); + if (!s->codec) { + av_log(avctx, AV_LOG_ERROR, "Failed to create media decoder for mime %s\n", mime); + return AVERROR_EXTERNAL; + } + + s->codec_name = ff_AMediaCodec_getName(s->codec); + if (!s->codec_name) { + s->codec_name = av_strdup(mime); + if (!s->codec_name) + return AVERROR(ENOMEM); } + return 0; +} + +int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, + const char *mime, FFAMediaFormat *format) +{ + int ret; + int status; + + s->avctx = avctx; + atomic_init(&s->refcount, 1); + atomic_init(&s->hw_buffer_count, 0); + atomic_init(&s->serial, 1); + s->current_input_buffer = -1; + + if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) + ret = mediacodec_dec_get_audio_codec(avctx, s, mime, format); + else if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) + ret = mediacodec_dec_get_video_codec(avctx, s, mime, format); + else + av_assert0(0); + if (ret < 0) + goto fail; + status = ff_AMediaCodec_configure(s->codec, format, s->surface, NULL, 0); if (status < 0) { char *desc = ff_AMediaFormat_toString(format); @@ -583,12 +858,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s, goto fail; } - s->format = ff_AMediaCodec_getOutputFormat(s->codec); - if (s->format) { - if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) { - av_log(avctx, AV_LOG_ERROR, - "Failed to configure context\n"); - goto fail; + if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) { + s->format = ff_AMediaCodec_getOutputFormat(s->codec); + if (s->format) { + if ((ret = mediacodec_dec_parse_format(avctx, s)) < 0) { + av_log(avctx, AV_LOG_ERROR, + "Failed to configure context\n"); + goto fail; + } } }