From patchwork Wed Apr 17 04:37:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48097 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:ce4e:b0:1a9:af23:56c1 with SMTP id id14csp776553pzb; Tue, 16 Apr 2024 21:38:53 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXicakZ5LSV2cFAvTo9dLHCUPJG6ttFuCRAk4nDlCMc9YNmoyw0Apnn/3itEhbfTo5o/LzQDNsAbrQIv4b/dqeZ0C6D+fGDyVbdLA== X-Google-Smtp-Source: AGHT+IHmHSp2dYB4DxuI4OnZBdbEElrot/NXtAUfP+MdoQiNU4qkF6rVUOKCGLA5LGieoD+UzMic X-Received: by 2002:a50:9e4e:0:b0:56f:e715:d1c7 with SMTP id z72-20020a509e4e000000b0056fe715d1c7mr10701657ede.1.1713328732839; Tue, 16 Apr 2024 21:38:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1713328732; cv=none; d=google.com; s=arc-20160816; b=RFvDsxKN+/4Jq1LiScBnE1jqcUijtzfFrbyFkRjQO9yKoIS9z6EKTYzmXzE5kLxfOi 8N+zA55vLVV2IhiIhKs6QzO5uwEfUDD/ElwwfSJhbZMIQNgu2hL+NzbmexoWaJU7arYO NtwWFXccdLG+AQHiY/Zlq89YXsjyG1/lf9I89WkbDElFqmpUe2Sf6fiGcVolxB6Q4d4u QVwqrtsUvApG1RWULwnwGfgd/keFEAdKL6I65X7DTb0BrfOxcehX0MYkPZDZ7KCJ7n8N UdVYsH3hIKYztLAAj11C1vVat7DkjV5NT2kQTKkSKP2ZwlYT64SLRzw631O9vExRAcO/ xkQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=qLuUXIdL3QATsEx2mOtJS0wjIl7NHFSJbWvB278PD/o=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=roIeVyS8+m+fS6ddTqKWoBu3OsooITq3RTwjRe2d4ANCBz1aKP6Nt4o/WPXvFDbu41 AboCCV06t4/hsCNUt2q9S8ncGZtt0SE/tBqgE1br4kn7n3OCWCWelOQ/djwr4aTTLgP3 E9S+tqoSCgdK2SDtw1pTO6YciXmIha8Fpko4b1JyZNuDi7m/zLzZQhTQgGkisbHmyVn9 M/QBMN9bj9CPdDmVqudZ9dUyOIFY5nUHo+oKnOKXQQ1zPZg9YrN8ID9PgXy3WZWW7EXd vuFxxLM3Ia/WGLUcS+ztDBtIfA+56C63xsIvUg09908jF8MsxabXoq1jbtSgHjEGjiHH duXw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=WvaGGEip; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f21-20020a50a6d5000000b0056c06e3883dsi6471835edc.311.2024.04.16.21.38.52; Tue, 16 Apr 2024 21:38:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=WvaGGEip; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 804AB68D22E; Wed, 17 Apr 2024 07:38:06 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-192.mail.qq.com (out203-205-221-192.mail.qq.com [203.205.221.192]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4983568CF59 for ; Wed, 17 Apr 2024 07:37:54 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1713328666; bh=3RqoSg7r9WYFeKd5gKwryIuSypD8sKKkt5AOh7l/Wbg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=WvaGGEipBCxLuszyevM5caoIIQJ/NGAiIMpAhqvmupx849DNQtKbssDTXEay4hPlh 9NUraA0lmEerqEIJgq01uXLXFRn2vD9Ylp6WE/9bDRpegNB+VmQCDMbRlMk59L+ut7 BC1cc9CjwbFZryHNb6kBral/W34/YTqD/3tU+hhc= Received: from localhost.localdomain ([119.147.10.204]) by newxmesmtplogicsvrsza30-0.qq.com (NewEsmtp) with SMTP id 96A0DC1B; Wed, 17 Apr 2024 12:37:42 +0800 X-QQ-mid: xmsmtpt1713328665tu09feff4 Message-ID: X-QQ-XMAILINFO: NssJ/YX50roakP4HDC5TB9nfdVx8Abtm8GUkTvyLrK8R81buocxw2Xqooww36D wZB8DnUdVc/iZ2bGZER3vvSU+ugjC3UnuOz2xPRH9V0Fy3lzsFXGudgbgwTLRK4kYbbNP81E8FVd qrjgKT0txqLS4L1A92LVA6yA9XjmLkPYwc8MTz4epSyMy1SZ12rtyiYU6ehYE7JDcUsdc6Hv1pfp aTBjwAtCr7QmBLf/UMRdOKzRMe6QiMNY/SCMKpQmZPnon6h3Gi6vxUxUYGE5+ir/6gEXN58Z2Ah0 3cGWuYdo1gBkFUsZVgR0nHbRaiY3SRFEGLCGm+83rsa2giVZvCTaX31ysNMUU4Q4eCKjf4y174dD bI2axuO/jG78E7nQ0uUVH9WTpUfxUuQTK0HD9LX/1wpdP6qJ3beoa1HrzwYVVfMC59Vloc8t6DFM VI1Sbq2AFLvco68o+AwurwTbZEOi1Ht8s32+BCfBOXj2Fvnw+6qx6/8Cz3LQxRfLzDugMUulq7/g nBukp6sy9DUhYza9XedPxPp4ezTnI6vXmF37tSjk+LVbMVv6vgAPBU2ASN+e0CNLUyeCxLZsOcMq 1PidBmx1NXWEKeoSWdu/bey9eDk1GmXTpEdbH2fTsOyI4i+MMYZi3bpDmal4GqrNPq8SsQ6tSEAu Kn08bVvcM3emBeRNJQbnldVbu64nhU6P90Vq/EalSomgBrN321rIaV5Lf9Dfo0BODSOmD/nxfZQq 4vQ+74soQbiTOmKHD86AUB+lvqWBbHZEPyaYlbhIrQN4S3HyL+hhNaKvPGhBf/ievReFZd3bs4DN mbIHZywjmpjsYIElDAOAFxGMDCdFh21YeTxa0ms2UOidzd8j9YHh9kOLOO53hF+uwAxgBfCCcBCU mgu7gLawuj5Jw1nwFwZs3/QSAgxI+7UtTJTkIef5Y8rupoM9iDJjP4UDryBGCxIsg2+n4QgIXvXz YsXIE/ib4uLYOmLjHke9EabJYWMSDi/dvPwOWx2oYXLwzNS9qL9dKMCE7vQOb2uHrzy5ppTmqi3Y TLgbtieQ== X-QQ-XMRINFO: M/715EihBoGSf6IYSX1iLFg= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Wed, 17 Apr 2024 12:37:40 +0800 X-OQ-MSGID: <20240417043742.505853-6-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240417043742.505853-1-quinkblack@foxmail.com> References: <20240417043742.505853-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 6/8] avcodec/mediacodecenc: Add global header support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: slYTt6KBejHq From: Zhao Zhili The extradata is generated by encoding a dummy frame, then reset the encoder state by mediacodec flush(). It only works for pixel format other than AV_PIX_FMT_MEDIACODEC, since I'm not sure how to create a dummy frame safely with AV_PIX_FMT_MEDIACODEC. Signed-off-by: Zhao Zhili --- configure | 6 +- libavcodec/mediacodecenc.c | 166 +++++++++++++++++++++++++++++++++---- 2 files changed, 155 insertions(+), 17 deletions(-) diff --git a/configure b/configure index d0d0e8430b..4da1c0be35 100755 --- a/configure +++ b/configure @@ -3313,6 +3313,7 @@ ac3_mf_encoder_deps="mediafoundation" av1_cuvid_decoder_deps="cuvid CUVIDAV1PICPARAMS" av1_mediacodec_decoder_deps="mediacodec" av1_mediacodec_encoder_deps="mediacodec" +av1_mediacodec_encoder_select="extract_extradata_bsf" av1_nvenc_encoder_deps="nvenc NV_ENC_PIC_PARAMS_AV1" av1_nvenc_encoder_select="atsc_a53" h263_v4l2m2m_decoder_deps="v4l2_m2m h263_v4l2_m2m" @@ -3323,7 +3324,7 @@ h264_cuvid_decoder_select="h264_mp4toannexb_bsf" h264_mediacodec_decoder_deps="mediacodec" h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser" h264_mediacodec_encoder_deps="mediacodec" -h264_mediacodec_encoder_select="h264_metadata" +h264_mediacodec_encoder_select="extract_extradata_bsf h264_metadata" h264_mf_encoder_deps="mediafoundation" h264_mmal_decoder_deps="mmal" h264_nvenc_encoder_deps="nvenc" @@ -3343,7 +3344,7 @@ hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf" hevc_mediacodec_decoder_deps="mediacodec" hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser" hevc_mediacodec_encoder_deps="mediacodec" -hevc_mediacodec_encoder_select="hevc_metadata" +hevc_mediacodec_encoder_select="extract_extradata_bsf hevc_metadata" hevc_mf_encoder_deps="mediafoundation" hevc_nvenc_encoder_deps="nvenc" hevc_nvenc_encoder_select="atsc_a53" @@ -3375,6 +3376,7 @@ mpeg2_v4l2m2m_decoder_deps="v4l2_m2m mpeg2_v4l2_m2m" mpeg4_cuvid_decoder_deps="cuvid" mpeg4_mediacodec_decoder_deps="mediacodec" mpeg4_mediacodec_encoder_deps="mediacodec" +mpeg4_mediacodec_encoder_select="extract_extradata_bsf" mpeg4_mmal_decoder_deps="mmal" mpeg4_omx_encoder_deps="omx" mpeg4_v4l2m2m_decoder_deps="v4l2_m2m mpeg4_v4l2_m2m" diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c index 8caaad729a..64816ccf0a 100644 --- a/libavcodec/mediacodecenc.c +++ b/libavcodec/mediacodecenc.c @@ -23,6 +23,7 @@ #include "config_components.h" #include "libavutil/avassert.h" +#include "libavutil/avstring.h" #include "libavutil/hwcontext_mediacodec.h" #include "libavutil/imgutils.h" #include "libavutil/mem.h" @@ -74,6 +75,7 @@ typedef struct MediaCodecEncContext { int bitrate_mode; int level; int pts_as_dts; + int extract_extradata; } MediaCodecEncContext; enum { @@ -112,6 +114,23 @@ static void mediacodec_output_format(AVCodecContext *avctx) ff_AMediaFormat_delete(out_format); } +static int extract_extradata_support(AVCodecContext *avctx) +{ + const AVBitStreamFilter *bsf = av_bsf_get_by_name("extract_extradata"); + + if (!bsf) { + av_log(avctx, AV_LOG_WARNING, "extract_extradata bsf not found\n"); + return 0; + } + + for (int i = 0; bsf->codec_ids[i] != AV_CODEC_ID_NONE; i++) { + if (bsf->codec_ids[i] == avctx->codec_id) + return 1; + } + + return 0; +} + static int mediacodec_init_bsf(AVCodecContext *avctx) { MediaCodecEncContext *s = avctx->priv_data; @@ -120,20 +139,32 @@ static int mediacodec_init_bsf(AVCodecContext *avctx) int crop_right = s->width - avctx->width; int crop_bottom = s->height - avctx->height; - if (!crop_right && !crop_bottom) + /* Nothing can be done for this format now */ + if (avctx->pix_fmt == AV_PIX_FMT_MEDIACODEC) return 0; - if (avctx->codec_id == AV_CODEC_ID_H264) - ret = snprintf(str, sizeof(str), "h264_metadata=crop_right=%d:crop_bottom=%d", - crop_right, crop_bottom); - else if (avctx->codec_id == AV_CODEC_ID_HEVC) - ret = snprintf(str, sizeof(str), "hevc_metadata=crop_right=%d:crop_bottom=%d", - crop_right, crop_bottom); - else + s->extract_extradata = (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER) && + extract_extradata_support(avctx); + if (!crop_right && !crop_bottom && !s->extract_extradata) return 0; - if (ret >= sizeof(str)) - return AVERROR_BUFFER_TOO_SMALL; + ret = 0; + if (crop_right || crop_bottom) { + if (avctx->codec_id == AV_CODEC_ID_H264) + ret = snprintf(str, sizeof(str), "h264_metadata=crop_right=%d:crop_bottom=%d", + crop_right, crop_bottom); + else if (avctx->codec_id == AV_CODEC_ID_HEVC) + ret = snprintf(str, sizeof(str), "hevc_metadata=crop_right=%d:crop_bottom=%d", + crop_right, crop_bottom); + if (ret >= sizeof(str)) + return AVERROR_BUFFER_TOO_SMALL; + } + + if (s->extract_extradata) { + ret = av_strlcatf(str, sizeof(str), "%sextract_extradata", ret ? "," : ""); + if (ret >= sizeof(str)) + return AVERROR_BUFFER_TOO_SMALL; + } ret = av_bsf_list_parse_str(str, &s->bsf); if (ret < 0) @@ -148,6 +179,8 @@ static int mediacodec_init_bsf(AVCodecContext *avctx) return ret; } +static int mediacodec_generate_extradata(AVCodecContext *avctx); + static av_cold int mediacodec_init(AVCodecContext *avctx) { const char *codec_mime = NULL; @@ -337,14 +370,14 @@ static av_cold int mediacodec_init(AVCodecContext *avctx) goto bailout; mediacodec_output_format(avctx); - if (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER) - av_log(avctx, AV_LOG_WARNING, - "Mediacodec encoder doesn't support AV_CODEC_FLAG_GLOBAL_HEADER. " - "Use extract_extradata bsf when necessary.\n"); s->frame = av_frame_alloc(); - if (!s->frame) + if (!s->frame) { ret = AVERROR(ENOMEM); + goto bailout; + } + + ret = mediacodec_generate_extradata(avctx); bailout: if (format) @@ -549,6 +582,109 @@ static int mediacodec_encode(AVCodecContext *avctx, AVPacket *pkt) return 0; } +static int mediacodec_send_dummy_frame(AVCodecContext *avctx) +{ + MediaCodecEncContext *s = avctx->priv_data; + int ret; + + s->frame->width = avctx->width; + s->frame->height = avctx->height; + s->frame->format = avctx->pix_fmt; + s->frame->pts = 0; + + ret = av_frame_get_buffer(s->frame, 0); + if (ret < 0) + return ret; + + do { + ret = mediacodec_send(avctx, s->frame); + } while (ret == AVERROR(EAGAIN)); + av_frame_unref(s->frame); + + if (ret < 0) + return ret; + + ret = mediacodec_send(avctx, NULL); + if (ret < 0) { + av_log(avctx, AV_LOG_ERROR, "Flush failed: %s\n", av_err2str(ret)); + return ret; + } + + return 0; +} + +static int mediacodec_receive_dummy_pkt(AVCodecContext *avctx, AVPacket *pkt) +{ + MediaCodecEncContext *s = avctx->priv_data; + int ret; + + do { + ret = mediacodec_receive(avctx, pkt); + } while (ret == AVERROR(EAGAIN)); + + if (ret < 0) + return ret; + + do { + ret = av_bsf_send_packet(s->bsf, pkt); + if (ret < 0) + return ret; + ret = av_bsf_receive_packet(s->bsf, pkt); + } while (ret == AVERROR(EAGAIN)); + + return ret; +} + +static int mediacodec_generate_extradata(AVCodecContext *avctx) +{ + MediaCodecEncContext *s = avctx->priv_data; + AVPacket *pkt = NULL; + int ret; + size_t side_size; + uint8_t *side; + + if (!(avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER)) + return 0; + + if (!s->extract_extradata) { + av_log(avctx, AV_LOG_WARNING, + "Mediacodec encoder doesn't support AV_CODEC_FLAG_GLOBAL_HEADER. " + "Use extract_extradata bsf when necessary.\n"); + return 0; + } + + pkt = av_packet_alloc(); + if (!pkt) + return AVERROR(ENOMEM); + + ret = mediacodec_send_dummy_frame(avctx); + if (ret < 0) + goto bailout; + ret = mediacodec_receive_dummy_pkt(avctx, pkt); + if (ret < 0) + goto bailout; + + side = av_packet_get_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, &side_size); + if (side && side_size > 0) { + avctx->extradata = av_mallocz(side_size + AV_INPUT_BUFFER_PADDING_SIZE); + if (!avctx->extradata) { + ret = AVERROR(ENOMEM); + goto bailout; + } + + memcpy(avctx->extradata, side, side_size); + avctx->extradata_size = side_size; + } + +bailout: + if (s->eof_sent) { + s->eof_sent = 0; + ff_AMediaCodec_flush(s->codec); + } + av_packet_free(&pkt); + return ret; +} + static av_cold int mediacodec_close(AVCodecContext *avctx) { MediaCodecEncContext *s = avctx->priv_data;