From patchwork Sun Nov 20 06:49:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 34861 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:999a:b0:a4:2148:650a with SMTP id ve26csp1406399pzb; Sat, 19 Nov 2022 22:50:33 -0800 (PST) X-Google-Smtp-Source: AA0mqf7G4NrqPVkpggdN7qziHS4I7l/LwTfdPhExjdnwEXuTgRj6via4Xjh9WEj6teoQnie9vJYE X-Received: by 2002:a17:906:844c:b0:781:aff3:c68d with SMTP id e12-20020a170906844c00b00781aff3c68dmr11708056ejy.36.1668927033351; Sat, 19 Nov 2022 22:50:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668927033; cv=none; d=google.com; s=arc-20160816; b=kAtAIYKWX4juzjwzyPiiH6zri5QIdxgVEQzqGd6+Wh9ztQ9CmaRE9Pz6V8nBndOOZE pa97yO1DeHXAPR0S3P99NYRUGqt3ZCcEuEhZOcMUn6fy31oG5iVMltrgB434NcxG4+6u ZSmvVMK04s5cn5shWIjPvHq5UEaZtYPejbtyh4M6UfpQ/l06G4vxPrbrKrNhBAo7UZzs M20Pg7lq47caovUth5qzvEukj9YtSr/3Nt9eYfE/8Tv3ohb+1vNxH012JAm33K+cLuSM rxeJtMfJRBzVevBvwMJs8YoQQh2I+zq3nR8nKWT3/B+pd1+6Q8o2gR3TUw5ANxcHK2pk pnFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:from:message-id :dkim-signature:delivered-to; bh=Pk9aqb73yNJNUWckIY4skMK0Zh5++ruYIje57fznjpA=; b=gDiJwLOfQcUleXy/p2eTS0qMdwKXxxFKv9dRtIUNwA8bN3qgry8V0OZmhxkb80t1pA fNfWMkEWUsXaDwM13eTwgUhekoTxJWsnkm13CklfPy/nNLqTalLHNiTBW98WTcHiY+qt HJDlSfk8lX9PCBxYeHG1Hq5iniPxIs+4nIu8bBjCOpxLbJgeUbRT1cUBqEI1YxIk6yr3 WQBjUyvVhD82Wk44IE8qqQuhJS4JOKGiMzYeKvSM3eSAU5T8A2LIergTb1akjonie/ut FfyAgV4908SCMi5prXQFT0mEUI9zor+eN3V9h2cO+xQP3EozybGVtitEqvBNMSamS/Ub M36Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=sjTep8Sd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j26-20020a170906255a00b0078e1d213831si4909970ejb.122.2022.11.19.22.50.32; Sat, 19 Nov 2022 22:50:33 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=sjTep8Sd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AC7D268B8D7; Sun, 20 Nov 2022 08:50:28 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-252.mail.qq.com (out162-62-57-252.mail.qq.com [162.62.57.252]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 94A10680529 for ; Sun, 20 Nov 2022 08:50:20 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1668927016; bh=/2xDbEBtdT/ojcwtoLRU7r+pcYJKuNW3cYsnDK6E3WQ=; h=From:To:Cc:Subject:Date; b=sjTep8Sdt1riSADfGIuQtjq21Gd9ZKh/8JRueJUWr9PG+M3u0RCreZue89SfOkzcn L2LfGMz/a1HVBdlrglyeWIXGLQslygJ2eN1yxjGtNXWMfuXUl5hiKiKcYB/djPiiM/ l3lDzIERjJ0BIlwGgtysMK2cvpBws58M8E66F1NM= Received: from localhost.localdomain ([14.153.77.200]) by newxmesmtplogicsvrszc2-1.qq.com (NewEsmtp) with SMTP id C8F0626E; Sun, 20 Nov 2022 14:50:15 +0800 X-QQ-mid: xmsmtpt1668927015tsjj69u1d Message-ID: X-QQ-XMAILINFO: No84gL0ijbqcrGHHmIeZOYCHf21VT4kIyhneNzte5bmwDPFXk5D11Oxf9UKpx5 9gEqThzbTdIJH6eIf0Xa/Kzr+Jo7CZzONbit58oIoD49g9ZzRbhADS8XP+VNMOpxQ9nA5YTNw4N9 b7A3MINOftQliyrWHQN6Az+Zsr3oFBwtuHqEgeAYHG3YSCP1+dnW+vgle2145XSi5G36PBBFRl0A D5vF02eA4v5Lz2OjRRQZrobrlnPItQYkpWtZrBzLvJlyOZ2Dt3W1FhFu8nk0R3ScNd+5Ba9id9OR dz+nJ/ETqObsdlZUWQNPya48LHru0C98sTGOdh8QK1BllSN+XwlemqnW9R+BN0Qz3ReAbcqTLN1Z nJzoQzZpeILW4sSEJKSwNlv8hHF5wqTDOxayI79XuxY03+WZa8B8AdHxwxhkmmP+OXMn/HigWNC4 l2lpOmIPW8rWVYWiS63JQhxKCeQk4s6l9Jz8NBvGm9jJBkJu2G12glEI3ZYGrWt4mZeq0DuNxV5n xj+HGNOwxbXBQVQio2AHmThgpaLFGkiUa0WMbpavOqOZv2Qm4/v2h8+8kYYPQEOhxKcCCmAwa2rH vu0cJP4sg9ozB3EdxvzFy1bYfhPYSz64C96Jd+dbZqUTPqLEOPyvNheNWa6N6hAGFqdm4QzjYsX9 ZUkoC+JBX75Xl5B03aE8hfHHEs/DzJnQQzgTvqDThDtJVoGAoJZ3NPX0EQW7tJY/TMWh/44N2nOA UxygbFVv/YGI8CAvHu3wijX14yH4lgYJ6V6q2QrHc5d4R13QrYIfV3L5TWHVGFgn342NaGLw7jmW Ba2YUH0Yu9btUYImGEnClUKob47wG8qdLC7wQslo284Z5ihci4WEJhQ7fZ1517PUxtpjvXAh33tP o2EC9ejxbBFpR3jO5a/3Kze3D3W8sNefNZLWa18crZ1rWl4XDHhPTNP/IEF3+ZiDqJ3Lp5Q7TcxK 6B4IFfUOrnMDTUFyWk93zZ1OXLyHKY From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 20 Nov 2022 14:49:35 +0800 X-OQ-MSGID: <20221120064947.94540-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 00/12] Add MediaCodec encoder and NDK MediaCodec support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: matthieu.bouron@stupeflix.com, Zhao Zhili , aman@tmm1.net Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QPx0d7diOG1e From: Zhao Zhili v2: Rebased on master. 01/12: Use crop-width/crop-height as fallback and add TODO. Co-authored-by: Aman Karmani 07/12: Fix libavcodec/version.h conflict 10/12: Change default i-frame-interval to 1 and add log message; bump minor version 12/12: Fix rebase conflict v1: Firstly, some bugs were fixed (patch 1-4). Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM, for example, run FFmpeg inside of termux (an Android terminal emulator). It's well known that NDK MediaCodec missing some important functions, like get the list of codecs, but still useable. Patch 7 add NDK MediaCodec decoder support. It can be enabled via options, and enabled automatically if no JVM is available. Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by user, and can be created via AMediaCodec_createPersistentInputSurface automatically. This is a preparation for encoder. Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth to note that AVMediaCodecContext has only surface. Although we provided av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to AVMediaCodecContext. Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and the frames come from our MediaCodec decoder wrapper, we can control it's 'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS generation strategy works in this case. However, if frames comes from other sources, like a camera, there is no way to control the 'render' yet, so DTS is missing in this case. Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android. More importantly, we can do MediaCodec decoder to encoder without copy frames, although it's very limited since most of avfilters doesn't work. For example: ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4 Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec. there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't work. So if -hwaccel_output_format isn't being specified like: ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4 It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such case. Zhao Zhili (12): avcodec/mediacodec: fix incorrect crop info avcodec/mediacodecdec: don't break out if both input and output port return try again avcodec/mediacodecdec_common: fix misuse av_free/av_freep avcodec/mediacodecdec_common: fix useless av_buffer_unref avcodec/mediacodec_wrapper: separate implementation from interface avcodec/mediacodec: add NDK media codec wrapper avcodec/mediacodecdec: enable NDK mediacodec avutil/hwcontext_mediacodec: add ANativeWindow support avcodec/mediacodec: add ANativeWindow support avcodec: add MediaCodec encoder avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec Changelog | 2 + configure | 6 + fftools/ffmpeg_demux.c | 4 + libavcodec/Makefile | 2 + libavcodec/allcodecs.c | 2 + libavcodec/mediacodec_surface.c | 46 +- libavcodec/mediacodec_surface.h | 8 +- libavcodec/mediacodec_wrapper.c | 941 +++++++++++++++++++++++++++--- libavcodec/mediacodec_wrapper.h | 275 +++++++-- libavcodec/mediacodecdec.c | 21 +- libavcodec/mediacodecdec_common.c | 46 +- libavcodec/mediacodecdec_common.h | 1 + libavcodec/mediacodecenc.c | 504 ++++++++++++++++ libavcodec/version.h | 4 +- libavutil/hwcontext.c | 6 +- libavutil/hwcontext_mediacodec.c | 56 +- libavutil/hwcontext_mediacodec.h | 11 + libavutil/version.h | 2 +- 18 files changed, 1797 insertions(+), 140 deletions(-) create mode 100644 libavcodec/mediacodecenc.c