From patchwork Mon Oct 24 03:16:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 34841 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4a86:b0:9d:28a3:170e with SMTP id fn6csp1997594pzb; Sun, 23 Oct 2022 20:17:29 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7lzjfVg1i9V3WJ0jDIwU19j0W41dCxEG67Vf1EmP3tZztzldi3tx/4nSHSg57TjwKeuKGb X-Received: by 2002:a05:6402:538f:b0:444:c17b:1665 with SMTP id ew15-20020a056402538f00b00444c17b1665mr29221278edb.98.1666581448858; Sun, 23 Oct 2022 20:17:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666581448; cv=none; d=google.com; s=arc-20160816; b=XJ3l79bv3qMGhXwARIaIyEj8RssQPK7y4e/Z/lZw9EgeqYrmhJz/8z9RfzianFEgRu /Iintwxr3LrMqAgCml4DMwuo526SM7+xf3qoTU+YQ5uAz/JF797QTvCXzm7E/KHYCrBB p3C+27HVpHcLs4KldtKRwSi6P4OtFXAMcEOTLLgc2CwW9aHDaNt1PagbpfEc/a91ZNy1 zISikfQzy9DyvFMvg6ziaGbDffkAq4fEoOf049Z5mXJxfCN7I3dPC23+eAYXhMIO7zeD jTqg92I5W4DIhsa5sOCPcbMoIj82OpNhBxuJfCiAf/njVqqOcZaiuvhsehiqA0llVqGT bCkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:from:message-id :dkim-signature:delivered-to; bh=O7BkdTwhz+9e88oC59UZjVOxHauOk4JryTvr6rZQ4yU=; b=OFXopLjI4YSYGIXjwT+BRcH6DeGIYWYVkOfxu/+ckyby+5YbnMP9GBYA1C/CNm09G8 uvv3HS4MvNTChJdGwBIquDQ5F7tH/Rhi9+zjX52+gifzOKbHsXgCwear4hKc7zkDbe7i F1HDZm5jR41NVrCK28tkocR/8na8q9h1V0PJV9bJSAlG0XtEvIaRn0am+lNSOwdzI4Vh yI00fKmI2yugk2jmUGSRiLUoXtobUhCXjbT3/OfT90Ag8+9SOzLgx62xwvXlZ5JAr+i7 Oog6MaM0sani3M+LED7td5mX2KpTjtae/G1J0Fag3LW4YdM++7lDiOrGuhdfjRS7vztY QLuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=kMi1UnfH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cb19-20020a0564020b7300b00461b84a010fsi2366908edb.295.2022.10.23.20.17.27; Sun, 23 Oct 2022 20:17:28 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=kMi1UnfH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D382A68BD05; Mon, 24 Oct 2022 06:17:23 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-191.mail.qq.com (out203-205-221-191.mail.qq.com [203.205.221.191]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6000968BBAF for ; Mon, 24 Oct 2022 06:17:15 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1666581432; bh=nD4q7KrXATJztRF0qkTb0jaq0gwTtGMbghz6axP8EtI=; h=From:To:Cc:Subject:Date; b=kMi1UnfH/437lVi9dCJ8LWeI+9nXA3FS6fUpLWJvS2EvNylOmKYe5wXeSCOTLyQ32 nogpgB6r1FMDZxaD7DBy2g8Y9C5DrNaPOou7Dzq7JBvV3Pg2iCkjB0SDKTnuFIP+rV oCMNMISd/im6MYx1EREpkne27aqDyGAh0VUVTWIU= Received: from localhost.localdomain ([59.37.125.102]) by newxmesmtplogicsvrsza31.qq.com (NewEsmtp) with SMTP id 44A14270; Mon, 24 Oct 2022 11:17:10 +0800 X-QQ-mid: xmsmtpt1666581430ty36qju0r Message-ID: X-QQ-XMAILINFO: NiAdzfE16ND4AnNFgO1bgtGqnDbdjpw0cRsihDcAMkgfhaP4hdMfFxSan8Ei6t H2xrx5BxntrVT5owReCwRs2+lL3m58dN4MuYGL+wPtMCnEdahBCqML6/M9gZphnUoEE3QgNwHvB2 TGcWP3rIOV7B4o8uaYyonla/+0JERGtBko+/Pz0wTjmDNkUnRrt2uwd1fh+RjlUD9CwoR6C9aSUJ aLfeZcmEYsqA2ujM1erfeivtJeZEIgEkgU5sxAChEXKmnETazDfRi0Jr3R/q4yDK4fK6uQB+0lY5 8kXcQTuorgOTdx7aPb+59FrfqwHdKmowki9aD91GtAgwLAonBhmP68us784q4ye1rUc3x4oVKBYt TY+iKs2goXieinwerqDREhYJxKCkVckqthla6BbKI3eT+3gfJzBxxxRcV0SutFdfGZpmbexGxvda K7Zc4gzTXIRPT6yquIlA0q9NGWdwzIPjt8qv3VH+il/RzAT8R5FVN0y4DUaiF0NKLQB06tFBrKrV h/i4+A8f7dFLIDfqwwhC9VqNwn8+KmjvqoZvGgyJuO7SAX56dAngkKUbJNhSK4ZIOQaQFkLnDNYU Jn94DF8ngb503Bmr2uQo719cgfKDQALd2zIIk0VaoIx/z2zmjqPJHyArXKTmoZCy+ujaPrJY7TzZ g/bE7Tmryz7IKg1t8ZfZ3sgn2WXyRNJDdAQL9JtnEc+8ZOjORvW3n4XjQIoM9aRExTCAqXTwmZ4m Nc1xhLtOYYDPM2zBUXCBBQ1Pk8X4G3dfrWCtIs/ci/T8oS8ApSHiyYHmr6Mi9iHHeYHXVym5xT0q H6f7ZTTeATmR1/l3iAiPYkhazsqGVWCOLiF+PMTqWNo4DTvSaGcguViKZ2lVX4gcbjdpmk/wmkma n77RHpdcWNc+Uzq6jbvIY3mtsBOp5fPRCTnlX+VFVuyg5Q6qAhMOB4VYIV0iQQ8B/Jdom0YLPOVO GjMe6TKmuG6wNeM+bj1g== From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Mon, 24 Oct 2022 11:16:46 +0800 X-OQ-MSGID: <20221024031658.1026118-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: matthieu.bouron@stupeflix.com, Zhao Zhili , aman@tmm1.net Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: p/1LiAFD3ykN From: Zhao Zhili Firstly, some bugs were fixed (patch 1-4). Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM, for example, run FFmpeg inside of termux (an Android terminal emulator). It's well known that NDK MediaCodec missing some important functions, like get the list of codecs, but still useable. Patch 7 add NDK MediaCodec decoder support. It can be enabled via options, and enabled automatically if no JVM is available. Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by user, and can be created via AMediaCodec_createPersistentInputSurface automatically. This is a preparation for encoder. Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth to note that AVMediaCodecContext has only surface. Although we provided av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to AVMediaCodecContext. Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and the frames come from our MediaCodec decoder wrapper, we can control it's 'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS generation strategy works in this case. However, if frames comes from other sources, like a camera, there is no way to control the 'render' yet, so DTS is missing in this case. Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android. More importantly, we can do MediaCodec decoder to encoder without copy frames, although it's very limited since most of avfilters doesn't work. For example: ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4 Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec. there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't work. So if -hwaccel_output_format isn't being specified like: ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4 It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such case. Zhao Zhili (12): avcodec/mediacodec: fix incorrect crop info avcodec/mediacodecdec: don't break out if both input and output port return try again avcodec/mediacodecdec_common: fix misuse av_free/av_freep avcodec/mediacodecdec_common: fix useless av_buffer_unref avcodec/mediacodec_wrapper: separate implementation from interface avcodec/mediacodec: add NDK media codec wrapper avcodec/mediacodecdec: enable NDK mediacodec avutil/hwcontext_mediacodec: add ANativeWindow support avcodec/mediacodec: add ANativeWindow support avcodec: add MediaCodec encoder avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec Changelog | 2 + configure | 6 + fftools/ffmpeg_opt.c | 4 + libavcodec/Makefile | 2 + libavcodec/allcodecs.c | 2 + libavcodec/mediacodec_surface.c | 46 +- libavcodec/mediacodec_surface.h | 8 +- libavcodec/mediacodec_wrapper.c | 942 +++++++++++++++++++++++++++--- libavcodec/mediacodec_wrapper.h | 275 +++++++-- libavcodec/mediacodecdec.c | 21 +- libavcodec/mediacodecdec_common.c | 35 +- libavcodec/mediacodecdec_common.h | 1 + libavcodec/mediacodecenc.c | 495 ++++++++++++++++ libavcodec/version.h | 2 +- libavutil/hwcontext.c | 6 +- libavutil/hwcontext_mediacodec.c | 56 +- libavutil/hwcontext_mediacodec.h | 11 + libavutil/version.h | 4 +- 18 files changed, 1780 insertions(+), 138 deletions(-) create mode 100644 libavcodec/mediacodecenc.c