mbox series

[FFmpeg-devel,00/12] Add MediaCodec encoder and NDK MediaCodec support

Message ID tencent_936E75C3890A8EAA2553B3060840F3EA7305@qq.com
Headers show
Series Add MediaCodec encoder and NDK MediaCodec support | expand

Message

Zhao Zhili Oct. 24, 2022, 3:16 a.m. UTC 
From: Zhao Zhili <zhilizhao@tencent.com>

Firstly, some bugs were fixed (patch 1-4).

Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM,
for example, run FFmpeg inside of termux (an Android terminal emulator). It's
well known that NDK MediaCodec missing some important functions, like get the
list of codecs, but still useable.

Patch 7 add NDK MediaCodec decoder support. It can be enabled via options,
and enabled automatically if no JVM is available.

Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by
user, and can be created via AMediaCodec_createPersistentInputSurface
automatically. This is a preparation for encoder.

Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth
to note that AVMediaCodecContext has only surface. Although we provided
av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate
AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to
AVMediaCodecContext.

Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via
buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and
the frames come from our MediaCodec decoder wrapper, we can control it's
'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS
generation strategy works in this case. However, if frames comes from other
sources, like a camera, there is no way to control the 'render' yet, so DTS is
missing in this case.

Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android.
More importantly, we can do MediaCodec decoder to encoder without copy frames,
although it's very limited since most of avfilters doesn't work. For example:

./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4

Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec.
there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't
work. So if -hwaccel_output_format isn't being specified like:

./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4

It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on
hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such
case.


Zhao Zhili (12):
  avcodec/mediacodec: fix incorrect crop info
  avcodec/mediacodecdec: don't break out if both input and output port
    return try again
  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
  avcodec/mediacodecdec_common: fix useless av_buffer_unref
  avcodec/mediacodec_wrapper: separate implementation from interface
  avcodec/mediacodec: add NDK media codec wrapper
  avcodec/mediacodecdec: enable NDK mediacodec
  avutil/hwcontext_mediacodec: add ANativeWindow support
  avcodec/mediacodec: add ANativeWindow support
  avcodec: add MediaCodec encoder
  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec

 Changelog                         |   2 +
 configure                         |   6 +
 fftools/ffmpeg_opt.c              |   4 +
 libavcodec/Makefile               |   2 +
 libavcodec/allcodecs.c            |   2 +
 libavcodec/mediacodec_surface.c   |  46 +-
 libavcodec/mediacodec_surface.h   |   8 +-
 libavcodec/mediacodec_wrapper.c   | 942 +++++++++++++++++++++++++++---
 libavcodec/mediacodec_wrapper.h   | 275 +++++++--
 libavcodec/mediacodecdec.c        |  21 +-
 libavcodec/mediacodecdec_common.c |  35 +-
 libavcodec/mediacodecdec_common.h |   1 +
 libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
 libavcodec/version.h              |   2 +-
 libavutil/hwcontext.c             |   6 +-
 libavutil/hwcontext_mediacodec.c  |  56 +-
 libavutil/hwcontext_mediacodec.h  |  11 +
 libavutil/version.h               |   4 +-
 18 files changed, 1780 insertions(+), 138 deletions(-)
 create mode 100644 libavcodec/mediacodecenc.c