From patchwork Fri Dec 9 17:22:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 39663 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp1157075pzb; Fri, 9 Dec 2022 01:22:59 -0800 (PST) X-Google-Smtp-Source: AA0mqf4QqYg66bK7wrbSfSKvFh77QSW8GuG3Mxk9iPsihFu2cbfhceRYSjm9IoFDbLGp0yOvBj1n X-Received: by 2002:a17:906:c20c:b0:7ac:a2f5:cd0a with SMTP id d12-20020a170906c20c00b007aca2f5cd0amr4824820ejz.44.1670577779325; Fri, 09 Dec 2022 01:22:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670577779; cv=none; d=google.com; s=arc-20160816; b=Z+91bDFcz2O8w34gRXygXhYbDFKwk141E2k7W4c86aY4Uf0UPw/obwi6ye/Z/Ebggp W0JHziKu1YnvrSGIsKoADiuuAQ+5mJFUYohU4I2fgmd72Oij0rEKBYZuqa81ZJe2NaL+ x32pxF4Bm9bFSFB2/NrFWyARiDB0dHE7A/2RfwCOGO/Aas/wAt5zJaciqMW4w3y07cDV GrTGN2MantVEtrlyMVJcxkNM8iVJH/rSvpxryNr2fAK+OooL2jM9ndf0NFcemIwDltTw BJoQ/qkn5CD62ag2t0u4nXcJd+5om9SRVHkysVNibCSfZI57YETwpjQK6zz/+5Xuxmwp LUDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=Va4+y74uJ8plnfho6VDGS2UlZpLRMAb0MbvactKvuhI=; b=ZJdhiA3DhhX+stZcVsjcAhu1pVU2OaPyNXbz7+YML9ZiZ/vsToQy7WjZDoC8yPnYGJ Vc6uSxhwSlzu+mdJ3mCH7M6v1EgUUjqo9XChgfNG345YdJjfBwz7ueu0awV5W87aCPPl 0S831Wv+/81QuRmu2WSLlwaG2GuZx6l31SxgCsilvJR/9jWuBnD+IbSL13FcEs4oUt7w iv5yKkSnuPflzA5BAY9WcpX+6ZcIIpy37O8pejFkprgHSWvK1zVgPkDHM5QLiLsb9dlG uMY+PhiwjSffC9BJG1MyqjbcHECJNTZR6NSMHrTTxrFZEpFc5AFdF2aLR9Zdmdr3R2u/ cCog== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="UGtKx/4J"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id eb7-20020a0564020d0700b0046ae9e1bacesi1076904edb.604.2022.12.09.01.22.58; Fri, 09 Dec 2022 01:22:59 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="UGtKx/4J"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B7C1168BC0E; Fri, 9 Dec 2022 11:22:54 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-137.mail.qq.com (out162-62-57-137.mail.qq.com [162.62.57.137]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1AF4868BA81 for ; Fri, 9 Dec 2022 11:22:47 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1670577764; bh=XMRfNPuYTmNu6N4EGu9JYCdXZHf4nvJPRFb/KC1QT/U=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=UGtKx/4J+rNUnGyZ/DA06LfPYgwF2HnDt4ju70myQS7FL4RWsi1mpxgnbK7bit4u/ hcx4sclC2KK3x3S+kvhRlpjwDljSGroIGSDTbwTGiv3vvLvwljpYbACYw13k/4No3m 91CWv2CIXE0+m7+fZK2jy9yRpW7IAkzfexsgtMrk= Received: from localhost.localdomain ([59.37.125.102]) by newxmesmtplogicsvrszb9-0.qq.com (NewEsmtp) with SMTP id 5AB1F482; Fri, 09 Dec 2022 17:22:43 +0800 X-QQ-mid: xmsmtpt1670577763t6xmon7bq Message-ID: X-QQ-XMAILINFO: OaubouGXmhNzxlT2eaEGKqaxDozo0lCdGNURhmj0jZZrRqZWJ+vser7XZKNxtd fWKIZX43O3kTc8xEcLge/22qwYjlvhyEk/+pat8p/j6WoegDvQ3Jy1Y0TZisr/4gbdLDad7CSZfS L8ILj+nPcys+aAE/LnbGPg43M1Xhna9blC8zFIRFu6suZiwoen3keeyHDO1bXP9Nffsi+i5BQIKq UYWYb1IbixFHAU+z1IecDNVaerJhRzV/0lsI7V7pRGhNlaW4tCXmTPt2n24sGAbq+pICRGGNAg5s 3PQ4NeAeYGx2L4v99UVAYVQsCGjKiAsvak8DZi1SRoNqwIcr7WiUB3ix5vffuzkbLqbJJmpiKsdq LoO6OXNN+omjpvUBFCdeUbutAsGPIP5bCA3dUdzEfmeDJ7Rg4YJqvzcVdyuw6svx0425u7qyvlQA fGIVmsg9I2jdAohBzpGUXfQU5Qy5F2HEk+7FRiCw9UZjhJ7aAfuLHEV2E+SPKoXofJr3+sUCxTvq 5ppomox+r0+iVUcbnJZ2O40ip+FPjjfPYikKApTN4JNuzLBLrK6qZ8pJ3Eqq3ZTOzQ29gClTMjS5 zGN02M9J8XVdyytmQiWe5DNsANX4tXfb4Zvuyf3eUY3WI5LSkkGGtPW5XiiIptoeH+/5apuVxY+L nDZhT3/oDsVxMOe7MdqiLJSJxUAqeLrAePyC425DUP0VojGbX4+moxOjZl8Mw/snAZ/aWpNZfOS5 DzlLYe/HW/uIEthBmilLuuE5LudCVwtU8nfZe4bQXJAW/4u/78AuyKynw5o5fRt7Xs1VU04HTt2j XAfqOg3TsBcmQkM5BRw2i1gcuJPyAqnY7CBTjac2yoZad/IIY1RpOJKrvIZ4byUsk8G4+cLGdBjN CbXNmyLk6g1MszRAnHpQWzAassaETmAQCdY1oN8DEJV9hPTEBNvX+oiK+d/xHbWbTyOCJXo88Bxw 1TIBAO8Owy2HyAD5ePA/IJs0DKfTrR From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sat, 10 Dec 2022 01:22:40 +0800 X-OQ-MSGID: <20221209172240.68113-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 3/7] avcodec/mediacodecenc: use bsf to handle crop X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6M1KBW+3tXyJ From: Zhao Zhili It's well known that mediacodec encoder requires 16x16 alignment. Use our bsf to fix the crop info. --- v3: don't change the dimension for AV_PIX_FMT_MEDIACODEC. It can have side effect. configure | 2 + libavcodec/mediacodecenc.c | 78 +++++++++++++++++++++++++++++++++++--- 2 files changed, 75 insertions(+), 5 deletions(-) diff --git a/configure b/configure index f4eedfc207..2180ebb4f1 100755 --- a/configure +++ b/configure @@ -3169,6 +3169,7 @@ h264_mediacodec_decoder_extralibs="-landroid" h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser" h264_mediacodec_encoder_deps="mediacodec" h264_mediacodec_encoder_extralibs="-landroid" +h264_mediacodec_encoder_select="h264_metadata" h264_mf_encoder_deps="mediafoundation" h264_mmal_decoder_deps="mmal" h264_nvenc_encoder_deps="nvenc" @@ -3190,6 +3191,7 @@ hevc_mediacodec_decoder_extralibs="-landroid" hevc_mediacodec_decoder_select="hevc_mp4toannexb_bsf hevc_parser" hevc_mediacodec_encoder_deps="mediacodec" hevc_mediacodec_encoder_extralibs="-landroid" +hevc_mediacodec_encoder_select="hevc_metadata" hevc_mf_encoder_deps="mediafoundation" hevc_nvenc_encoder_deps="nvenc" hevc_nvenc_encoder_select="atsc_a53" diff --git a/libavcodec/mediacodecenc.c b/libavcodec/mediacodecenc.c index 2f78567451..4e8716e3a5 100644 --- a/libavcodec/mediacodecenc.c +++ b/libavcodec/mediacodecenc.c @@ -28,6 +28,7 @@ #include "libavutil/opt.h" #include "avcodec.h" +#include "bsf.h" #include "codec_internal.h" #include "encode.h" #include "hwconfig.h" @@ -78,6 +79,7 @@ typedef struct MediaCodecEncContext { int eof_sent; AVFrame *frame; + AVBSFContext *bsf; int bitrate_mode; int level; @@ -119,6 +121,42 @@ static void mediacodec_output_format(AVCodecContext *avctx) ff_AMediaFormat_delete(out_format); } +static int mediacodec_init_bsf(AVCodecContext *avctx) +{ + MediaCodecEncContext *s = avctx->priv_data; + char str[128]; + int ret; + int crop_right = s->width - avctx->width; + int crop_bottom = s->height - avctx->height; + + if (!crop_right && !crop_bottom) + return 0; + + if (avctx->codec_id == AV_CODEC_ID_H264) + ret = snprintf(str, sizeof(str), "h264_metadata=crop_right=%d:crop_bottom=%d", + crop_right, crop_bottom); + else if (avctx->codec_id == AV_CODEC_ID_HEVC) + ret = snprintf(str, sizeof(str), "hevc_metadata=crop_right=%d:crop_bottom=%d", + crop_right, crop_bottom); + else + return 0; + + if (ret >= sizeof(str)) + return AVERROR_BUFFER_TOO_SMALL; + + ret = av_bsf_list_parse_str(str, &s->bsf); + if (ret < 0) + return ret; + + ret = avcodec_parameters_from_context(s->bsf->par_in, avctx); + if (ret < 0) + return ret; + s->bsf->time_base_in = avctx->time_base; + ret = av_bsf_init(s->bsf); + + return ret; +} + static av_cold int mediacodec_init(AVCodecContext *avctx) { const char *codec_mime = NULL; @@ -158,8 +196,19 @@ static av_cold int mediacodec_init(AVCodecContext *avctx) } ff_AMediaFormat_setString(format, "mime", codec_mime); - s->width = FFALIGN(avctx->width, 16); - s->height = avctx->height; + // Workaround the alignment requirement of mediacodec. We can't do it + // silently for AV_PIX_FMT_MEDIACODEC. + if (avctx->pix_fmt != AV_PIX_FMT_MEDIACODEC) { + s->width = FFALIGN(avctx->width, 16); + s->height = FFALIGN(avctx->height, 16); + } else { + s->width = avctx->width; + s->height = avctx->height; + if (s->width % 16 || s->height % 16) + av_log(avctx, AV_LOG_WARNING, + "Video size %dx%d isn't align to 16, it may have device compatibility issue\n", + s->width, s->height); + } ff_AMediaFormat_setInt32(format, "width", s->width); ff_AMediaFormat_setInt32(format, "height", s->height); @@ -252,6 +301,10 @@ static av_cold int mediacodec_init(AVCodecContext *avctx) goto bailout; } + ret = mediacodec_init_bsf(avctx); + if (ret) + goto bailout; + mediacodec_output_format(avctx); s->frame = av_frame_alloc(); @@ -444,10 +497,24 @@ static int mediacodec_encode(AVCodecContext *avctx, AVPacket *pkt) // 2. Got a packet success // 3. No AVFrame is available yet (don't return if get_frame return EOF) while (1) { + if (s->bsf) { + ret = av_bsf_receive_packet(s->bsf, pkt); + if (!ret) + return 0; + if (ret != AVERROR(EAGAIN)) + return ret; + } + ret = mediacodec_receive(avctx, pkt, &got_packet); - if (!ret) - return 0; - else if (ret != AVERROR(EAGAIN)) + if (s->bsf) { + if (!ret || ret == AVERROR_EOF) + ret = av_bsf_send_packet(s->bsf, pkt); + } else { + if (!ret) + return 0; + } + + if (ret != AVERROR(EAGAIN)) return ret; if (!s->frame->buf[0]) { @@ -480,6 +547,7 @@ static av_cold int mediacodec_close(AVCodecContext *avctx) s->window = NULL; } + av_bsf_free(&s->bsf); av_frame_free(&s->frame); return 0;