diff mbox

[FFmpeg-devel] Add support for RockChip Media Process Platform

Message ID MWHPR17MB161407CAB5A59699084EEFA5B46A0@MWHPR17MB1614.namprd17.prod.outlook.com
State Superseded
Headers show

Commit Message

LongChair . Sept. 9, 2017, 7 a.m. UTC 
From: LongChair <LongChair@hotmail.com>

This adds hardware decoding for h264 / HEVC / VP8 using MPP Rockchip API.
Will return frames holding an AVDRMFrameDescriptor struct in buf[0] that allows drm / dmabuf usage.
Was tested on RK3288 (TinkerBoard) and RK3328.

Changes from Previous patch :
- Frame colorspace info is now filled.
- Frame interlacing is now filled (Note : currently had a bug in mpp which will be fixed soon by rockchip, will set the frame to progressive).
- hw_frame_context returns none as format for the rockchip specific 10 bits.
- Added support for MPEG4 codec
- Added support for VP9 codec.
- the ION format has been kept for MPP init (rather than DRM) as this is the only one working right, using DRM format will cause assertions upon close.
- Other minor comments have been taken into account
---
 Changelog              |   1 +
 configure              |  16 ++
 libavcodec/Makefile    |   5 +
 libavcodec/allcodecs.c |   5 +
 libavcodec/rkmppdec.c  | 584 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 611 insertions(+)
 create mode 100644 libavcodec/rkmppdec.c

Comments

Mark Thompson Sept. 9, 2017, 12:59 p.m. UTC | #1 
On 09/09/17 08:00, LongChair . wrote:
> From: LongChair <LongChair@hotmail.com>
> 
> This adds hardware decoding for h264 / HEVC / VP8 using MPP Rockchip API.
> Will return frames holding an AVDRMFrameDescriptor struct in buf[0] that allows drm / dmabuf usage.
> Was tested on RK3288 (TinkerBoard) and RK3328.
> 
> Changes from Previous patch :
> - Frame colorspace info is now filled.
> - Frame interlacing is now filled (Note : currently had a bug in mpp which will be fixed soon by rockchip, will set the frame to progressive).
> - hw_frame_context returns none as format for the rockchip specific 10 bits.
> - Added support for MPEG4 codec
> - Added support for VP9 codec.
> - the ION format has been kept for MPP init (rather than DRM) as this is the only one working right, using DRM format will cause assertions upon close.
> - Other minor comments have been taken into account
> ---

Patch looks pretty good now (some minor issues below); mainly I've been testing it.


Is VP9 supported by the RK3288?  Trying to use it (both 8- and 10-bit) I get:

[ 5110.660270] rk-vcodec ff9a0000.vpu-service: can not find -22 buffer in list
[ 5110.666094] rk-vcodec ff9a0000.vpu-service: reg 28 fd 4 ion map iommu failed
[ 5110.673203] rk-vcodec ff9a0000.vpu-service: can not find -22 buffer in list
[ 5110.679026] rk_vcodec: reg_init:1242: error: translate reg address failed, dumping regs
[ 5110.683991] Unhandled fault: page domain fault (0x01b) at 0x81817360
[ 5110.688851] pgd = ec7e4000
[ 5110.693709] [81817360] *pgd=2a81c831, *pte=3a4f675f, *ppte=3a4f6c7f
... lots of dump ...
[ 5111.257947] Process mpp_dec_parser (pid: 14424, stack limit = 0xdd85a218)
... lots of dump ...
[ 5111.362078] [<c053509c>] (reg_init) from [<c0537298>] (vpu_service_ioctl+0x298/0x744)
[ 5111.367023] [<c0537298>] (vpu_service_ioctl) from [<c02388ac>] (do_vfs_ioctl+0x5b0/0x690)
[ 5111.371974] [<c02388ac>] (do_vfs_ioctl) from [<c02389e8>] (SyS_ioctl+0x5c/0x84)
[ 5111.376956] [<c02389e8>] (SyS_ioctl) from [<c01072c0>] (ret_fast_syscall+0x0/0x3c)
[ 5111.381917] Code: e51b3108 e1a02006 e51b00e0 e1a01007 (e7933106) 
[ 5111.386917] ---[ end trace 866a2bbc379f58ce ]---
[ 5115.649373] rk-vcodec ff9a0000.vpu-service: closed

The ffmpeg process then has to be killed.

If VP9 isn't supported, can we determine that somehow and not try to decode at all?

I can't get MPEG-4 part 2 to work either.  I get a lot of timeouts, then maybe a lot of green frames?  (Tried a few files.)

H.264, H.265 and VP8 still work for me.


The output for H.264 (didn't look at others carefully here) seems to be missing some frames.  For example, if I decode <h264-conformance/NL1_Sony_D.jsv>, there should be 17 frames in the output, but h264_rkmpp only gives me 4.

Running:

./ffmpeg_g -y -c:v h264_rkmpp -i ~/test/NL1_Sony_D.jsv -an -vf hwdownload,format=nv12,format=yuv420p out_hw.yuv
./ffmpeg_g -y -i ~/test/NL1_Sony_D.jsv -an out_sw.yuv

$ ls -l out_*
-rw-r--r-- 1 linaro linaro 152064 Sep  9 12:45 out_hw.yuv
-rw-r--r-- 1 linaro linaro 646272 Sep  9 12:44 out_sw.yuv

The missing frames seem to be the ones at the beginning of the stream:

$ cmp out_sw.yuv out_hw.yuv 
out_sw.yuv out_hw.yuv differ: byte 1, line 1
$ cmp out_sw.yuv out_hw.yuv 494208 0
$ 

>  Changelog              |   1 +
>  configure              |  16 ++
>  libavcodec/Makefile    |   5 +
>  libavcodec/allcodecs.c |   5 +
>  libavcodec/rkmppdec.c  | 584 +++++++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 611 insertions(+)
>  create mode 100644 libavcodec/rkmppdec.c
> 
> diff --git a/Changelog b/Changelog
> index 189a803..ccf78b2 100644
> --- a/Changelog
> +++ b/Changelog
> @@ -44,6 +44,7 @@ version <next>:
>  - drop deprecated qtkit input device (use avfoundation instead)
>  - despill video filter
>  - haas audio filter
> +- Rockchip MPP hardware decoding
>  
>  version 3.3:
>  - CrystalHD decoder moved to new decode API
> diff --git a/configure b/configure
> index 3ba27c5..cbff908 100755
> --- a/configure
> +++ b/configure
> @@ -314,6 +314,7 @@ External library support:
>    --disable-nvenc          disable Nvidia video encoding code [autodetect]
>    --enable-omx             enable OpenMAX IL code [no]
>    --enable-omx-rpi         enable OpenMAX IL code for Raspberry Pi [no]
> +  --enable-rkmpp           enable Rockchip Media Process Platform code [no]
>    --disable-vaapi          disable Video Acceleration API (mainly Unix/Intel) code [autodetect]
>    --disable-vda            disable Apple Video Decode Acceleration code [autodetect]
>    --disable-vdpau          disable Nvidia Video Decode and Presentation API for Unix code [autodetect]
> @@ -1545,6 +1546,7 @@ EXTERNAL_LIBRARY_VERSION3_LIST="
>      libopencore_amrnb
>      libopencore_amrwb
>      libvo_amrwbenc
> +    rkmpp
>  "
>  
>  EXTERNAL_LIBRARY_GPLV3_LIST="
> @@ -2774,6 +2776,8 @@ h264_qsv_decoder_deps="libmfx"
>  h264_qsv_decoder_select="h264_mp4toannexb_bsf h264_parser qsvdec h264_qsv_hwaccel"
>  h264_qsv_encoder_deps="libmfx"
>  h264_qsv_encoder_select="qsvenc"
> +h264_rkmpp_decoder_deps="rkmpp"
> +h264_rkmpp_decoder_select="h264_mp4toannexb_bsf"
>  h264_vaapi_encoder_deps="VAEncPictureParameterBufferH264"
>  h264_vaapi_encoder_select="vaapi_encode golomb"
>  h264_vda_decoder_deps="vda"
> @@ -2789,6 +2793,8 @@ hevc_qsv_decoder_deps="libmfx"
>  hevc_qsv_decoder_select="hevc_mp4toannexb_bsf hevc_parser qsvdec hevc_qsv_hwaccel"
>  hevc_qsv_encoder_deps="libmfx"
>  hevc_qsv_encoder_select="hevcparse qsvenc"
> +hevc_rkmpp_decoder_deps="rkmpp"
> +hevc_rkmpp_decoder_select="hevc_mp4toannexb_bsf"
>  hevc_vaapi_encoder_deps="VAEncPictureParameterBufferHEVC"
>  hevc_vaapi_encoder_select="vaapi_encode golomb"
>  mjpeg_cuvid_decoder_deps="cuda cuvid"
> @@ -2812,6 +2818,7 @@ mpeg4_cuvid_decoder_deps="cuda cuvid"
>  mpeg4_mediacodec_decoder_deps="mediacodec"
>  mpeg4_mmal_decoder_deps="mmal"
>  mpeg4_omx_encoder_deps="omx"
> +mpeg4_rkmpp_decoder_deps="rkmpp"
>  mpeg4_vdpau_decoder_deps="vdpau"
>  mpeg4_vdpau_decoder_select="mpeg4_decoder"
>  mpeg_vdpau_decoder_deps="vdpau"
> @@ -2828,10 +2835,12 @@ vp8_cuvid_decoder_deps="cuda cuvid"
>  vp8_mediacodec_decoder_deps="mediacodec"
>  vp8_qsv_decoder_deps="libmfx"
>  vp8_qsv_decoder_select="qsvdec vp8_qsv_hwaccel vp8_parser"
> +vp8_rkmpp_decoder_deps="rkmpp"
>  vp8_vaapi_encoder_deps="VAEncPictureParameterBufferVP8"
>  vp8_vaapi_encoder_select="vaapi_encode"
>  vp9_cuvid_decoder_deps="cuda cuvid"
>  vp9_mediacodec_decoder_deps="mediacodec"
> +vp9_rkmpp_decoder_deps="rkmpp"
>  vp9_vaapi_encoder_deps="VAEncPictureParameterBufferVP9"
>  vp9_vaapi_encoder_select="vaapi_encode"
>  wmv3_crystalhd_decoder_select="crystalhd"
> @@ -6032,6 +6041,13 @@ enabled openssl           && { use_pkg_config openssl openssl/ssl.h OPENSSL_init
>                                 check_lib openssl openssl/ssl.h SSL_library_init -lssl32 -leay32 ||
>                                 check_lib openssl openssl/ssl.h SSL_library_init -lssl -lcrypto -lws2_32 -lgdi32 ||
>                                 die "ERROR: openssl not found"; }
> +enabled rkmpp             && { { require_pkg_config rockchip_mpp rockchip/rk_mpi.h mpp_create ||
> +                                 die "ERROR : Rockchip MPP was not found."; } &&
> +                               { check_func_headers rockchip/rk_mpi_cmd.h "MPP_DEC_GET_FREE_PACKET_SLOT_COUNT" ||
> +                                 die "ERROR: Rockchip MPP is outdated, please get a more recent one."; } &&
> +                               { enabled libdrm ||
> +                                 die "ERROR: rkmpp requires --enable-libdrm"; }
> +                             }
>  
>  if enabled gcrypt; then
>      GCRYPT_CONFIG="${cross_prefix}libgcrypt-config"
> diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> index 999632c..3a86996 100644
> --- a/libavcodec/Makefile
> +++ b/libavcodec/Makefile
> @@ -338,6 +338,7 @@ OBJS-$(CONFIG_H264_VDA_DECODER)        += vda_h264_dec.o
>  OBJS-$(CONFIG_H264_OMX_ENCODER)        += omx.o
>  OBJS-$(CONFIG_H264_QSV_DECODER)        += qsvdec_h2645.o
>  OBJS-$(CONFIG_H264_QSV_ENCODER)        += qsvenc_h264.o
> +OBJS-$(CONFIG_H264_RKMPP_DECODER)      += rkmppdec.o
>  OBJS-$(CONFIG_H264_VAAPI_ENCODER)      += vaapi_encode_h264.o vaapi_encode_h26x.o
>  OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o
>  OBJS-$(CONFIG_HAP_DECODER)             += hapdec.o hap.o
> @@ -352,6 +353,7 @@ OBJS-$(CONFIG_NVENC_HEVC_ENCODER)      += nvenc_hevc.o
>  OBJS-$(CONFIG_HEVC_QSV_DECODER)        += qsvdec_h2645.o
>  OBJS-$(CONFIG_HEVC_QSV_ENCODER)        += qsvenc_hevc.o hevc_ps_enc.o       \
>                                            hevc_data.o
> +OBJS-$(CONFIG_HEVC_RKMPP_DECODER)      += rkmppdec.o
>  OBJS-$(CONFIG_HEVC_VAAPI_ENCODER)      += vaapi_encode_h265.o vaapi_encode_h26x.o
>  OBJS-$(CONFIG_HNM4_VIDEO_DECODER)      += hnm4video.o
>  OBJS-$(CONFIG_HQ_HQA_DECODER)          += hq_hqa.o hq_hqadata.o hq_hqadsp.o \
> @@ -431,6 +433,7 @@ OBJS-$(CONFIG_MPEG2_MEDIACODEC_DECODER) += mediacodecdec.o
>  OBJS-$(CONFIG_MPEG2_VAAPI_ENCODER)     += vaapi_encode_mpeg2.o
>  OBJS-$(CONFIG_MPEG4_DECODER)           += xvididct.o
>  OBJS-$(CONFIG_MPEG4_MEDIACODEC_DECODER) += mediacodecdec.o
> +OBJS-$(CONFIG_MPEG4_RKMPP_DECODER)      += rkmppdec.o
>  OBJS-$(CONFIG_MPEG4_OMX_ENCODER)       += omx.o
>  OBJS-$(CONFIG_MPL2_DECODER)            += mpl2dec.o ass.o
>  OBJS-$(CONFIG_MSA1_DECODER)            += mss3.o
> @@ -623,12 +626,14 @@ OBJS-$(CONFIG_VP8_DECODER)             += vp8.o vp56rac.o
>  OBJS-$(CONFIG_VP8_CUVID_DECODER)       += cuvid.o
>  OBJS-$(CONFIG_VP8_MEDIACODEC_DECODER)  += mediacodecdec.o
>  OBJS-$(CONFIG_VP8_QSV_DECODER)         += qsvdec_other.o
> +OBJS-$(CONFIG_VP8_RKMPP_DECODER)       += rkmppdec.o
>  OBJS-$(CONFIG_VP8_VAAPI_ENCODER)       += vaapi_encode_vp8.o
>  OBJS-$(CONFIG_VP9_DECODER)             += vp9.o vp9data.o vp9dsp.o vp9lpf.o vp9recon.o \
>                                            vp9block.o vp9prob.o vp9mvs.o vp56rac.o \
>                                            vp9dsp_8bpp.o vp9dsp_10bpp.o vp9dsp_12bpp.o
>  OBJS-$(CONFIG_VP9_CUVID_DECODER)       += cuvid.o
>  OBJS-$(CONFIG_VP9_MEDIACODEC_DECODER)  += mediacodecdec.o
> +OBJS-$(CONFIG_VP9_RKMPP_DECODER)       += rkmppdec.o
>  OBJS-$(CONFIG_VP9_VAAPI_ENCODER)       += vaapi_encode_vp9.o
>  OBJS-$(CONFIG_VPLAYER_DECODER)         += textdec.o ass.o
>  OBJS-$(CONFIG_VQA_DECODER)             += vqavideo.o
> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> index ce0bc7e..3335c11 100644
> --- a/libavcodec/allcodecs.c
> +++ b/libavcodec/allcodecs.c
> @@ -213,6 +213,7 @@ static void register_all(void)
>      REGISTER_DECODER(H264_MEDIACODEC,   h264_mediacodec);
>      REGISTER_DECODER(H264_MMAL,         h264_mmal);
>      REGISTER_DECODER(H264_QSV,          h264_qsv);
> +    REGISTER_DECODER(H264_RKMPP,        h264_rkmpp);
>      REGISTER_DECODER(H264_VDA,          h264_vda);
>  #if FF_API_VDPAU
>      REGISTER_DECODER(H264_VDPAU,        h264_vdpau);
> @@ -220,6 +221,7 @@ static void register_all(void)
>      REGISTER_ENCDEC (HAP,               hap);
>      REGISTER_DECODER(HEVC,              hevc);
>      REGISTER_DECODER(HEVC_QSV,          hevc_qsv);
> +    REGISTER_DECODER(HEVC_RKMPP,        hevc_rkmpp);
>      REGISTER_DECODER(HNM4_VIDEO,        hnm4_video);
>      REGISTER_DECODER(HQ_HQA,            hq_hqa);
>      REGISTER_DECODER(HQX,               hqx);
> @@ -255,6 +257,7 @@ static void register_all(void)
>      REGISTER_ENCDEC (MPEG4,             mpeg4);
>      REGISTER_DECODER(MPEG4_CRYSTALHD,   mpeg4_crystalhd);
>      REGISTER_DECODER(MPEG4_MMAL,        mpeg4_mmal);
> +    REGISTER_DECODER(MPEG4_RKMPP,       mpeg4_rkmpp);
>  #if FF_API_VDPAU
>      REGISTER_DECODER(MPEG4_VDPAU,       mpeg4_vdpau);
>  #endif
> @@ -373,7 +376,9 @@ static void register_all(void)
>      REGISTER_DECODER(VP6F,              vp6f);
>      REGISTER_DECODER(VP7,               vp7);
>      REGISTER_DECODER(VP8,               vp8);
> +    REGISTER_DECODER(VP8_RKMPP,         vp8_rkmpp);
>      REGISTER_DECODER(VP9,               vp9);
> +    REGISTER_DECODER(VP9_RKMPP,         vp9_rkmpp);
>      REGISTER_DECODER(VQA,               vqa);
>      REGISTER_DECODER(BITPACKED,         bitpacked);
>      REGISTER_DECODER(WEBP,              webp);
> diff --git a/libavcodec/rkmppdec.c b/libavcodec/rkmppdec.c
> new file mode 100644
> index 0000000..33f7f5d
> --- /dev/null
> +++ b/libavcodec/rkmppdec.c
> @@ -0,0 +1,584 @@
> +/*
> + * RockChip MPP Video Decoder
> + * Copyright (c) 2017 Lionel CHAZALLON
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <drm_fourcc.h>
> +#include <pthread.h>
> +#include <rockchip/mpp_buffer.h>
> +#include <rockchip/rk_mpi.h>
> +#include <time.h>
> +#include <unistd.h>
> +
> +#include "avcodec.h"
> +#include "decode.h"
> +#include "internal.h"
> +#include "libavutil/buffer.h"
> +#include "libavutil/common.h"
> +#include "libavutil/frame.h"
> +#include "libavutil/hwcontext.h"
> +#include "libavutil/hwcontext_drm.h"
> +#include "libavutil/imgutils.h"
> +#include "libavutil/log.h"
> +
> +#define RECEIVE_FRAME_TIMEOUT   100
> +#define FRAMEGROUP_MAX_FRAMES   16
> +
> +typedef struct {
> +    MppCtx ctx;
> +    MppApi *mpi;
> +    MppBufferGroup frame_group;
> +
> +    char first_frame;
> +    char first_packet;
> +    char eos_reached;
> +
> +    AVBufferRef *frames_ref;
> +    AVBufferRef *device_ref;
> +} RKMPPDecoder;
> +
> +typedef struct {
> +    AVClass *av_class;
> +    AVBufferRef *decoder_ref;
> +} RKMPPDecodeContext;
> +
> +typedef struct {
> +    MppFrame frame;
> +    AVBufferRef *decoder_ref;
> +} RKMPPFrameContext;
> +
> +static MppCodingType rkmpp_get_codingtype(AVCodecContext *avctx)
> +{
> +    switch (avctx->codec_id) {
> +    case AV_CODEC_ID_H264:          return MPP_VIDEO_CodingAVC;
> +    case AV_CODEC_ID_HEVC:          return MPP_VIDEO_CodingHEVC;
> +    case AV_CODEC_ID_MPEG4:         return MPP_VIDEO_CodingMPEG4;
> +    case AV_CODEC_ID_VP8:           return MPP_VIDEO_CodingVP8;
> +    case AV_CODEC_ID_VP9:           return MPP_VIDEO_CodingVP9;
> +    default:                        return MPP_VIDEO_CodingUnused;
> +    }
> +}
> +
> +static int rkmpp_get_frameformat(MppFrameFormat mppformat)
> +{
> +    switch (mppformat) {
> +    case MPP_FMT_YUV420SP:          return DRM_FORMAT_NV12;
> +#ifdef DRM_FORMAT_NV12_10
> +    case MPP_FMT_YUV420SP_10BIT:    return DRM_FORMAT_NV12_10;
> +#endif
> +    default:                        return 0;
> +    }
> +}
> +
> +static int rkmpp_write_data(AVCodecContext *avctx, uint8_t *buffer, int size, int64_t pts)
> +{
> +    RKMPPDecodeContext *rk_context = avctx->priv_data;
> +    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
> +    int ret = MPP_NOK;
> +    MppPacket packet;
> +
> +    // create the MPP packet
> +    ret = mpp_packet_init(&packet, buffer, size);
> +    if (ret != MPP_OK) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to init MPP packet (code = %d)\n", ret);
> +        return AVERROR_UNKNOWN;
> +    }
> +
> +    mpp_packet_set_pts(packet, pts);
> +
> +    if (!buffer)
> +        mpp_packet_set_eos(packet);
> +
> +    ret = decoder->mpi->decode_put_packet(decoder->ctx, packet);
> +    if (ret != MPP_OK) {
> +        if (ret == MPP_ERR_BUFFER_FULL) {
> +            av_log(avctx, AV_LOG_DEBUG, "Buffer full writing %d bytes to decoder\n", size);
> +            ret = AVERROR(EAGAIN);
> +        } else
> +            ret = AVERROR_UNKNOWN;
> +    }
> +    else
> +        av_log(avctx, AV_LOG_DEBUG, "Wrote %d bytes to decoder\n", size);
> +
> +    mpp_packet_deinit(&packet);
> +
> +    return ret;
> +}
> +
> +static int rkmpp_close_decoder(AVCodecContext *avctx)
> +{
> +    RKMPPDecodeContext *rk_context = avctx->priv_data;
> +    av_buffer_unref(&rk_context->decoder_ref);
> +    return 0;
> +}
> +
> +static void rkmpp_release_decoder(void *opaque, uint8_t *data)
> +{
> +    RKMPPDecoder *decoder = (RKMPPDecoder *)data;
> +
> +    decoder->mpi->reset(decoder->ctx);
> +    mpp_destroy(decoder->ctx);
> +    decoder->ctx = NULL;
> +
> +    if (decoder->frame_group) {
> +        mpp_buffer_group_put(decoder->frame_group);
> +        decoder->frame_group = NULL;
> +    }
> +
> +    av_buffer_unref(&decoder->frames_ref);
> +    av_buffer_unref(&decoder->device_ref);
> +
> +    av_free(decoder);
> +}
> +
> +static int rkmpp_init_decoder(AVCodecContext *avctx)
> +{
> +    RKMPPDecodeContext *rk_context = avctx->priv_data;
> +    RKMPPDecoder *decoder = NULL;
> +    MppCodingType codectype = MPP_VIDEO_CodingUnused;
> +    int ret = MPP_NOK;
> +    RK_S64 paramS64;
> +    RK_S32 paramS32;
> +
> +    avctx->pix_fmt = AV_PIX_FMT_DRM_PRIME;
> +
> +    // create a decoder and a ref to it
> +    decoder = av_mallocz(sizeof(RKMPPDecoder));
> +    if (!decoder) {
> +        ret = AVERROR(ENOMEM);
> +        goto fail;
> +    }
> +
> +    rk_context->decoder_ref = av_buffer_create((uint8_t *)decoder, sizeof(*decoder), rkmpp_release_decoder,
> +                                               NULL, AV_BUFFER_FLAG_READONLY);
> +    if (!rk_context->decoder_ref) {
> +        av_free(decoder);
> +        ret = AVERROR(ENOMEM);
> +        goto fail;
> +    }
> +
> +    av_log(avctx, AV_LOG_DEBUG, "Initializing RKMPP decoder.\n");
> +
> +    codectype = rkmpp_get_codingtype(avctx);
> +    if (codectype == MPP_VIDEO_CodingUnused) {
> +        av_log(avctx, AV_LOG_ERROR, "Unknown codec type (%d).\n", avctx->codec_id);
> +        ret = AVERROR_UNKNOWN;
> +        goto fail;
> +    }
> +
> +    // Create the MPP context
> +    ret = mpp_create(&decoder->ctx, &decoder->mpi);
> +    if (ret != MPP_OK) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to create MPP context (code = %d).\n", ret);
> +        ret = AVERROR_UNKNOWN;
> +        goto fail;
> +    }
> +
> +    // initialize mpp
> +    ret = mpp_init(decoder->ctx, MPP_CTX_DEC, codectype);
> +    if (ret != MPP_OK) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to initialize MPP context (code = %d).\n", ret);
> +        ret = AVERROR_UNKNOWN;
> +        goto fail;
> +    }
> +
> +    // make decode calls blocking with a timeout
> +    paramS32 = MPP_POLL_BLOCK;
> +    ret = decoder->mpi->control(decoder->ctx, MPP_SET_OUTPUT_BLOCK, &paramS32);
> +    if (ret != MPP_OK) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to set blocking mode on MPI (code = %d).\n", ret);
> +        ret = AVERROR_UNKNOWN;
> +        goto fail;
> +    }
> +
> +    paramS64 = RECEIVE_FRAME_TIMEOUT;
> +    ret = decoder->mpi->control(decoder->ctx, MPP_SET_OUTPUT_BLOCK_TIMEOUT, &paramS64);
> +    if (ret != MPP_OK) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to set block timeout on MPI (code = %d).\n", ret);
> +        ret = AVERROR_UNKNOWN;
> +        goto fail;
> +    }
> +
> +    ret = mpp_buffer_group_get_internal(&decoder->frame_group, MPP_BUFFER_TYPE_ION);

Can you add a comment for this change?  It is a bit surprising.

> +    if (ret) {
> +       av_log(avctx, AV_LOG_ERROR, "Failed to retrieve buffer group (code = %d)\n", ret);
> +       ret = AVERROR_UNKNOWN;
> +       goto fail;
> +    }
> +
> +    ret = decoder->mpi->control(decoder->ctx, MPP_DEC_SET_EXT_BUF_GROUP, decoder->frame_group);
> +    if (ret) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to assign buffer group (code = %d)\n", ret);
> +        ret = AVERROR_UNKNOWN;
> +        goto fail;
> +    }
> +
> +    ret = mpp_buffer_group_limit_config(decoder->frame_group, 0, FRAMEGROUP_MAX_FRAMES);
> +    if (ret) {
> +        av_log(avctx, AV_LOG_ERROR, "Failed to set buffer group limit (code = %d)\n", ret);
> +        ret = AVERROR_UNKNOWN;
> +        goto fail;
> +    }
> +
> +    decoder->first_packet = 1;
> +
> +    av_log(avctx, AV_LOG_DEBUG, "RKMPP decoder initialized successfully.\n");
> +
> +    decoder->device_ref = av_hwdevice_ctx_alloc(AV_HWDEVICE_TYPE_DRM);
> +    if (!decoder->device_ref) {
> +        ret = AVERROR(ENOMEM);
> +        goto fail;
> +    }
> +    ret = av_hwdevice_ctx_init(decoder->device_ref);
> +    if (ret < 0)
> +        goto fail;
> +
> +    return 0;
> +
> +fail:
> +    av_log(avctx, AV_LOG_ERROR, "Failed to initialize RKMPP decoder.\n");
> +    rkmpp_close_decoder(avctx);
> +    return ret;
> +}
> +
> +static int rkmpp_send_packet(AVCodecContext *avctx, const AVPacket *avpkt)
> +{
> +    RKMPPDecodeContext *rk_context = avctx->priv_data;
> +    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
> +    int ret = MPP_NOK;
> +
> +    // handle EOF
> +    if (!avpkt->size) {
> +        av_log(avctx, AV_LOG_DEBUG, "End of stream.\n");
> +        decoder->eos_reached = 1;
> +        ret = rkmpp_write_data(avctx, NULL, 0, 0);
> +        if (ret)
> +            av_log(avctx, AV_LOG_ERROR, "Failed to send EOS to decoder (code = %d)\n", ret);
> +        return ret;
> +    }
> +
> +    // on first packet, send extradata
> +    if (decoder->first_packet) {
> +        if (avctx->extradata_size) {
> +            ret = rkmpp_write_data(avctx, avctx->extradata,
> +                                            avctx->extradata_size,
> +                                            avpkt->pts);
> +            if (ret) {
> +                av_log(avctx, AV_LOG_ERROR, "Failed to write extradata to decoder (code = %d)\n", ret);
> +                return ret;
> +            }
> +        }
> +        decoder->first_packet = 0;
> +    }
> +
> +    // now send packet
> +    ret = rkmpp_write_data(avctx, avpkt->data, avpkt->size, avpkt->pts);
> +    if (ret && ret!=AVERROR(EAGAIN))
> +        av_log(avctx, AV_LOG_ERROR, "Failed to write data to decoder (code = %d)\n", ret);
> +
> +    return ret;
> +}
> +
> +static void rkmpp_release_frame(void *opaque, uint8_t *data)
> +{
> +    AVDRMFrameDescriptor *desc = (AVDRMFrameDescriptor *)data;
> +    AVBufferRef *framecontextref = (AVBufferRef *)opaque;
> +    RKMPPFrameContext *framecontext = (RKMPPFrameContext *)framecontextref->data;
> +
> +    mpp_frame_deinit(&framecontext->frame);
> +    av_buffer_unref(&framecontext->decoder_ref);
> +    av_buffer_unref(&framecontextref);
> +
> +    av_free(desc);
> +}
> +
> +static int rkmpp_retrieve_frame(AVCodecContext *avctx, AVFrame *frame)
> +{
> +    RKMPPDecodeContext *rk_context = avctx->priv_data;
> +    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
> +    RKMPPFrameContext *framecontext = NULL;
> +    AVBufferRef *framecontextref = NULL;
> +    int ret = MPP_NOK;
> +    MppFrame mppframe = NULL;
> +    MppBuffer buffer = NULL;
> +    AVDRMFrameDescriptor *desc = NULL;
> +    AVDRMLayerDescriptor *layer = NULL;
> +    int retrycount = 0;
> +    int format;
> +
> +    // on start of decoding, MPP can return -1, which is supposed to be expected
> +    // this is due to some internal MPP init which is not completed, that will
> +    // only happen in the first few frames queries, but should not be interpreted
> +    // as an error, Therefore we need to retry a couple times when we get -1
> +    // in order to let it time to complete it's init, then we sleep a bit between retries.
> +retry_get_frame:
> +    ret = decoder->mpi->decode_get_frame(decoder->ctx, &mppframe);
> +    if (ret != MPP_OK && ret != MPP_ERR_TIMEOUT && !decoder->first_frame) {
> +        if (retrycount < 5) {
> +            av_log(avctx, AV_LOG_DEBUG, "Failed to get a frame, retrying (code = %d, retrycount = %d)\n", ret, retrycount);
> +            usleep(10000);
> +            retrycount++;
> +            goto retry_get_frame;
> +        } else {
> +            av_log(avctx, AV_LOG_ERROR, "Failed to get a frame from MPP (code = %d)\n", ret);
> +            goto fail;
> +        }
> +    }
> +
> +    if (mppframe) {
> +        // Check wether we have a special frame or not
> +        if (mpp_frame_get_info_change(mppframe)) {
> +            AVHWFramesContext *hwframes;
> +
> +            av_log(avctx, AV_LOG_INFO, "Decoder noticed an info change (%dx%d), format=%d\n",
> +                                        (int)mpp_frame_get_width(mppframe), (int)mpp_frame_get_height(mppframe),
> +                                        (int)mpp_frame_get_fmt(mppframe));
> +
> +            avctx->width = mpp_frame_get_width(mppframe);
> +            avctx->height = mpp_frame_get_height(mppframe);
> +
> +            decoder->mpi->control(decoder->ctx, MPP_DEC_SET_INFO_CHANGE_READY, NULL);
> +            decoder->first_frame = 1;
> +
> +            av_buffer_unref(&decoder->frames_ref);
> +
> +            decoder->frames_ref = av_hwframe_ctx_alloc(decoder->device_ref);
> +            if (!decoder->frames_ref) {
> +                ret = AVERROR(ENOMEM);
> +                goto fail;
> +            }
> +
> +            format = (int)mpp_frame_get_fmt(mppframe);
> +
> +            hwframes = (AVHWFramesContext*)decoder->frames_ref->data;
> +            hwframes->format    = AV_PIX_FMT_DRM_PRIME;
> +            hwframes->sw_format = rkmpp_get_frameformat(format) == DRM_FORMAT_NV12 ? AV_PIX_FMT_NV12 : AV_PIX_FMT_NONE;
> +            hwframes->width     = avctx->width;
> +            hwframes->height    = avctx->height;
> +            ret = av_hwframe_ctx_init(decoder->frames_ref);
> +            if (ret < 0)
> +                goto fail;
> +
> +            // here decoder is fully initialized, we need to feed it again with data
> +            ret = AVERROR(EAGAIN);
> +            goto fail;
> +        } else if (mpp_frame_get_eos(mppframe)) {
> +            av_log(avctx, AV_LOG_DEBUG, "Received a EOS frame.\n");
> +            decoder->eos_reached = 1;
> +            ret = AVERROR_EOF;
> +            goto fail;
> +        } else if (mpp_frame_get_discard(mppframe)) {
> +            av_log(avctx, AV_LOG_DEBUG, "Received a discard frame.\n");
> +            ret = AVERROR(EAGAIN);
> +            goto fail;
> +        } else if (mpp_frame_get_errinfo(mppframe)) {
> +            av_log(avctx, AV_LOG_ERROR, "Received a errinfo frame.\n");
> +            ret = AVERROR_UNKNOWN;
> +            goto fail;
> +        }
> +
> +        // here we should have a valid frame
> +        av_log(avctx, AV_LOG_DEBUG, "Received a frame.\n");
> +
> +        // setup general frame fields
> +        frame->format           = AV_PIX_FMT_DRM_PRIME;
> +        frame->width            = mpp_frame_get_width(mppframe);
> +        frame->height           = mpp_frame_get_height(mppframe);
> +        frame->pts              = mpp_frame_get_pts(mppframe);
> +        frame->color_range      = mpp_frame_get_color_range(mppframe);
> +        frame->color_primaries  = mpp_frame_get_color_primaries(mppframe);
> +        frame->color_trc        = mpp_frame_get_color_trc(mppframe);
> +        frame->colorspace       = mpp_frame_get_colorspace(mppframe);
> +
> +        int mode = mpp_frame_get_mode(mppframe);

No mixed declarations and statements.

> +        frame->interlaced_frame = ((mode & MPP_FRAME_FLAG_FIELD_ORDER_MASK) == MPP_FRAME_FLAG_DEINTERLACED);
> +        frame->top_field_first  = ((mode & MPP_FRAME_FLAG_FIELD_ORDER_MASK) == MPP_FRAME_FLAG_TOP_FIRST);

This seems a bit dubious.  It's an interlaced frame if the deinterlaced flag is set?  Also the interlaced output can only be bottom-field-first (since you've got the same mask and equality comparison)?

> +
> +
> +        // now setup the frame buffer info
> +        buffer = mpp_frame_get_buffer(mppframe);
> +        if (buffer) {
> +            desc = av_mallocz(sizeof(AVDRMFrameDescriptor));
> +            if (!desc) {
> +                ret = AVERROR(ENOMEM);
> +                goto fail;
> +            }
> +
> +            desc->nb_objects = 1;
> +            desc->objects[0].fd = mpp_buffer_get_fd(buffer);
> +            desc->objects[0].size = mpp_buffer_get_size(buffer);
> +
> +            desc->nb_layers = 1;
> +            layer = &desc->layers[0];
> +            layer->format = rkmpp_get_frameformat(mpp_frame_get_fmt(mppframe));
> +            layer->nb_planes = 2;
> +
> +            layer->planes[0].object_index = 0;
> +            layer->planes[0].offset = 0;
> +            layer->planes[0].pitch = mpp_frame_get_hor_stride(mppframe);
> +
> +            layer->planes[1].object_index = 0;
> +            layer->planes[1].offset = layer->planes[0].pitch * mpp_frame_get_ver_stride(mppframe);
> +            layer->planes[1].pitch = layer->planes[0].pitch;
> +
> +            // we also allocate a struct in buf[0] that will allow to hold additionnal information
> +            // for releasing properly MPP frames and decoder
> +            framecontextref = av_buffer_allocz(sizeof(*framecontext));
> +            if (!framecontextref) {
> +                ret = AVERROR(ENOMEM);
> +                goto fail;
> +            }
> +
> +            // MPP decoder needs to be closed only when all frames have been released.
> +            framecontext = (RKMPPFrameContext *)framecontextref->data;
> +            framecontext->decoder_ref = av_buffer_ref(rk_context->decoder_ref);
> +            framecontext->frame = mppframe;
> +
> +            frame->data[0]  = (uint8_t *)desc;
> +            frame->buf[0]   = av_buffer_create((uint8_t *)desc, sizeof(*desc), rkmpp_release_frame,
> +                                               framecontextref, AV_BUFFER_FLAG_READONLY);
> +
> +            if (!frame->buf[0]) {
> +                ret = AVERROR(ENOMEM);
> +                goto fail;
> +            }
> +
> +            frame->hw_frames_ctx = av_buffer_ref(decoder->frames_ref);
> +            if (!frame->hw_frames_ctx) {
> +                ret = AVERROR(ENOMEM);
> +                goto fail;
> +            }
> +
> +            decoder->first_frame = 0;
> +            return 0;
> +        } else {
> +            av_log(avctx, AV_LOG_ERROR, "Failed to retrieve the frame buffer, frame is dropped (code = %d)\n", ret);
> +            mpp_frame_deinit(&mppframe);
> +        }
> +    } else if (decoder->eos_reached) {
> +        return AVERROR_EOF;
> +    } else if (ret == MPP_ERR_TIMEOUT) {
> +        av_log(avctx, AV_LOG_DEBUG, "Timeout when trying to get a frame from MPP\n");
> +    }
> +
> +    return AVERROR(EAGAIN);
> +
> +fail:
> +    if (mppframe)
> +        mpp_frame_deinit(&mppframe);
> +
> +    if (framecontext)
> +        av_buffer_unref(&framecontext->decoder_ref);
> +
> +    if (framecontextref)
> +        av_buffer_unref(&framecontextref);

Check is redundant (can be called with NULL).

> +
> +    if (desc)
> +        av_free(desc);

Same.

> +
> +    return ret;
> +}
> +
> +static int rkmpp_receive_frame(AVCodecContext *avctx, AVFrame *frame)
> +{
> +    RKMPPDecodeContext *rk_context = avctx->priv_data;
> +    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
> +    int ret = MPP_NOK;
> +    AVPacket pkt = {0};
> +    RK_S32 freeslots;
> +
> +    if (!decoder->eos_reached) {
> +        // we get the available slots in decoder
> +        ret = decoder->mpi->control(decoder->ctx, MPP_DEC_GET_FREE_PACKET_SLOT_COUNT, &freeslots);
> +        if (ret != MPP_OK) {
> +            av_log(avctx, AV_LOG_ERROR, "Failed to get decoder free slots (code = %d).\n", ret);
> +            return ret;
> +        }
> +
> +        if (freeslots > 0) {
> +            ret = ff_decode_get_packet(avctx, &pkt);
> +            if (ret < 0 && ret != AVERROR_EOF) {
> +                return ret;
> +            }
> +
> +            ret = rkmpp_send_packet(avctx, &pkt);
> +            av_packet_unref(&pkt);
> +
> +            if (ret < 0) {
> +                av_log(avctx, AV_LOG_ERROR, "Failed to send packet to decoder (code = %d)\n", ret);
> +                return ret;
> +            }
> +        }
> +
> +        // make sure we keep decoder full
> +        if (freeslots > 1 && decoder->first_frame)
> +            return AVERROR(EAGAIN);
> +    }
> +
> +    return rkmpp_retrieve_frame(avctx, frame);
> +}
> +
> +static void rkmpp_flush(AVCodecContext *avctx)
> +{
> +    RKMPPDecodeContext *rk_context = avctx->priv_data;
> +    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
> +    int ret = MPP_NOK;
> +
> +    av_log(avctx, AV_LOG_DEBUG, "Flush.\n");
> +
> +    ret = decoder->mpi->reset(decoder->ctx);
> +    if (ret == MPP_OK) {
> +        decoder->first_frame = 1;
> +        decoder->first_packet = 1;
> +    } else
> +        av_log(avctx, AV_LOG_ERROR, "Failed to reset MPI (code = %d)\n", ret);
> +}
> +
> +
> +#define RKMPP_DEC_CLASS(NAME) \
> +    static const AVClass rkmpp_##NAME##_dec_class = { \
> +        .class_name = "rkmpp_" #NAME "_dec", \
> +        .version    = LIBAVUTIL_VERSION_INT, \
> +    };
> +
> +#define RKMPP_DEC(NAME, ID, BSFS) \
> +    RKMPP_DEC_CLASS(NAME) \
> +    AVCodec ff_##NAME##_rkmpp_decoder = { \
> +        .name           = #NAME "_rkmpp", \
> +        .long_name      = NULL_IF_CONFIG_SMALL(#NAME " (rkmpp)"), \
> +        .type           = AVMEDIA_TYPE_VIDEO, \
> +        .id             = ID, \
> +        .priv_data_size = sizeof(RKMPPDecodeContext), \
> +        .init           = rkmpp_init_decoder, \
> +        .close          = rkmpp_close_decoder, \
> +        .receive_frame  = rkmpp_receive_frame, \
> +        .flush          = rkmpp_flush, \
> +        .priv_class     = &rkmpp_##NAME##_dec_class, \
> +        .capabilities   = AV_CODEC_CAP_DELAY, \
> +        .caps_internal  = FF_CODEC_CAP_SETS_PKT_DTS, \
> +        .pix_fmts       = (const enum AVPixelFormat[]) { AV_PIX_FMT_DRM_PRIME, \
> +                                                         AV_PIX_FMT_NONE}, \
> +        .bsfs           = BSFS, \
> +    };
> +
> +RKMPP_DEC(h264,  AV_CODEC_ID_H264,          "h264_mp4toannexb")
> +RKMPP_DEC(hevc,  AV_CODEC_ID_HEVC,          "hevc_mp4toannexb")
> +RKMPP_DEC(mpeg4, AV_CODEC_ID_MPEG4,         NULL)
> +RKMPP_DEC(vp8,   AV_CODEC_ID_VP8,           NULL)
> +RKMPP_DEC(vp9,   AV_CODEC_ID_VP9,           NULL)
James Almer Sept. 10, 2017, 5:13 p.m. UTC | #2 
On 9/9/2017 4:00 AM, LongChair . wrote:
> From: LongChair <LongChair@hotmail.com>
> 
> This adds hardware decoding for h264 / HEVC / VP8 using MPP Rockchip API.
> Will return frames holding an AVDRMFrameDescriptor struct in buf[0] that allows drm / dmabuf usage.
> Was tested on RK3288 (TinkerBoard) and RK3328.
> 
> Changes from Previous patch :
> - Frame colorspace info is now filled.
> - Frame interlacing is now filled (Note : currently had a bug in mpp which will be fixed soon by rockchip, will set the frame to progressive).
> - hw_frame_context returns none as format for the rockchip specific 10 bits.
> - Added support for MPEG4 codec
> - Added support for VP9 codec.
> - the ION format has been kept for MPP init (rather than DRM) as this is the only one working right, using DRM format will cause assertions upon close.
> - Other minor comments have been taken into account
> ---
>  Changelog              |   1 +
>  configure              |  16 ++
>  libavcodec/Makefile    |   5 +
>  libavcodec/allcodecs.c |   5 +
>  libavcodec/rkmppdec.c  | 584 +++++++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 611 insertions(+)
>  create mode 100644 libavcodec/rkmppdec.c

> +#define RKMPP_DEC(NAME, ID, BSFS) \
> +    RKMPP_DEC_CLASS(NAME) \
> +    AVCodec ff_##NAME##_rkmpp_decoder = { \
> +        .name           = #NAME "_rkmpp", \
> +        .long_name      = NULL_IF_CONFIG_SMALL(#NAME " (rkmpp)"), \
> +        .type           = AVMEDIA_TYPE_VIDEO, \
> +        .id             = ID, \
> +        .priv_data_size = sizeof(RKMPPDecodeContext), \
> +        .init           = rkmpp_init_decoder, \
> +        .close          = rkmpp_close_decoder, \
> +        .receive_frame  = rkmpp_receive_frame, \
> +        .flush          = rkmpp_flush, \
> +        .priv_class     = &rkmpp_##NAME##_dec_class, \
> +        .capabilities   = AV_CODEC_CAP_DELAY, \

Add AV_CODEC_CAP_AVOID_PROBING as well.

> +        .caps_internal  = FF_CODEC_CAP_SETS_PKT_DTS, \

It doesn't look like you're setting pkt_dts anywhere?

> +        .pix_fmts       = (const enum AVPixelFormat[]) { AV_PIX_FMT_DRM_PRIME, \
> +                                                         AV_PIX_FMT_NONE}, \
> +        .bsfs           = BSFS, \
> +    };
> +
> +RKMPP_DEC(h264,  AV_CODEC_ID_H264,          "h264_mp4toannexb")
> +RKMPP_DEC(hevc,  AV_CODEC_ID_HEVC,          "hevc_mp4toannexb")
> +RKMPP_DEC(mpeg4, AV_CODEC_ID_MPEG4,         NULL)
> +RKMPP_DEC(vp8,   AV_CODEC_ID_VP8,           NULL)
> +RKMPP_DEC(vp9,   AV_CODEC_ID_VP9,           NULL)
>
LongChair . Sept. 11, 2017, 8:21 a.m. UTC | #3 
Le 10/09/2017 à 19:13, James Almer a écrit :

On 9/9/2017 4:00 AM, LongChair . wrote:


From: LongChair <LongChair@hotmail.com><mailto:LongChair@hotmail.com>


This adds hardware decoding for h264 / HEVC / VP8 using MPP Rockchip API.
Will return frames holding an AVDRMFrameDescriptor struct in buf[0] that allows drm / dmabuf usage.
Was tested on RK3288 (TinkerBoard) and RK3328.

Changes from Previous patch :
- Frame colorspace info is now filled.
- Frame interlacing is now filled (Note : currently had a bug in mpp which will be fixed soon by rockchip, will set the frame to progressive).
- hw_frame_context returns none as format for the rockchip specific 10 bits.
- Added support for MPEG4 codec
- Added support for VP9 codec.
- the ION format has been kept for MPP init (rather than DRM) as this is the only one working right, using DRM format will cause assertions upon close.
- Other minor comments have been taken into account
---
 Changelog              |   1 +
 configure              |  16 ++
 libavcodec/Makefile    |   5 +
 libavcodec/allcodecs.c |   5 +
 libavcodec/rkmppdec.c  | 584 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 611 insertions(+)
 create mode 100644 libavcodec/rkmppdec.c





+#define RKMPP_DEC(NAME, ID, BSFS) \
+    RKMPP_DEC_CLASS(NAME) \
+    AVCodec ff_##NAME##_rkmpp_decoder = { \
+        .name           = #NAME "_rkmpp", \
+        .long_name      = NULL_IF_CONFIG_SMALL(#NAME " (rkmpp)"), \
+        .type           = AVMEDIA_TYPE_VIDEO, \
+        .id             = ID, \
+        .priv_data_size = sizeof(RKMPPDecodeContext), \
+        .init           = rkmpp_init_decoder, \
+        .close          = rkmpp_close_decoder, \
+        .receive_frame  = rkmpp_receive_frame, \
+        .flush          = rkmpp_flush, \
+        .priv_class     = &rkmpp_##NAME##_dec_class, \
+        .capabilities   = AV_CODEC_CAP_DELAY, \



Add AV_CODEC_CAP_AVOID_PROBING as well.

What would that flag change ?





+        .caps_internal  = FF_CODEC_CAP_SETS_PKT_DTS, \



It doesn't look like you're setting pkt_dts anywhere?

True, I can remove that one.





+        .pix_fmts       = (const enum AVPixelFormat[]) { AV_PIX_FMT_DRM_PRIME, \
+                                                         AV_PIX_FMT_NONE}, \
+        .bsfs           = BSFS, \
+    };
+
+RKMPP_DEC(h264,  AV_CODEC_ID_H264,          "h264_mp4toannexb")
+RKMPP_DEC(hevc,  AV_CODEC_ID_HEVC,          "hevc_mp4toannexb")
+RKMPP_DEC(mpeg4, AV_CODEC_ID_MPEG4,         NULL)
+RKMPP_DEC(vp8,   AV_CODEC_ID_VP8,           NULL)
+RKMPP_DEC(vp9,   AV_CODEC_ID_VP9,           NULL)




_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org<mailto:ffmpeg-devel@ffmpeg.org>
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
James Almer Sept. 13, 2017, 9:06 p.m. UTC | #4 
On 9/11/2017 5:21 AM, LongChair . wrote:
> 
> 
> Le 10/09/2017 à 19:13, James Almer a écrit :
>> On 9/9/2017 4:00 AM, LongChair . wrote:
>>> From: LongChair <LongChair@hotmail.com>
>>>
>>> This adds hardware decoding for h264 / HEVC / VP8 using MPP Rockchip API.
>>> Will return frames holding an AVDRMFrameDescriptor struct in buf[0] that allows drm / dmabuf usage.
>>> Was tested on RK3288 (TinkerBoard) and RK3328.
>>>
>>> Changes from Previous patch :
>>> - Frame colorspace info is now filled.
>>> - Frame interlacing is now filled (Note : currently had a bug in mpp which will be fixed soon by rockchip, will set the frame to progressive).
>>> - hw_frame_context returns none as format for the rockchip specific 10 bits.
>>> - Added support for MPEG4 codec
>>> - Added support for VP9 codec.
>>> - the ION format has been kept for MPP init (rather than DRM) as this is the only one working right, using DRM format will cause assertions upon close.
>>> - Other minor comments have been taken into account
>>> ---
>>>  Changelog              |   1 +
>>>  configure              |  16 ++
>>>  libavcodec/Makefile    |   5 +
>>>  libavcodec/allcodecs.c |   5 +
>>>  libavcodec/rkmppdec.c  | 584 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>  5 files changed, 611 insertions(+)
>>>  create mode 100644 libavcodec/rkmppdec.c
>>> +#define RKMPP_DEC(NAME, ID, BSFS) \
>>> +    RKMPP_DEC_CLASS(NAME) \
>>> +    AVCodec ff_##NAME##_rkmpp_decoder = { \
>>> +        .name           = #NAME "_rkmpp", \
>>> +        .long_name      = NULL_IF_CONFIG_SMALL(#NAME " (rkmpp)"), \
>>> +        .type           = AVMEDIA_TYPE_VIDEO, \
>>> +        .id             = ID, \
>>> +        .priv_data_size = sizeof(RKMPPDecodeContext), \
>>> +        .init           = rkmpp_init_decoder, \
>>> +        .close          = rkmpp_close_decoder, \
>>> +        .receive_frame  = rkmpp_receive_frame, \
>>> +        .flush          = rkmpp_flush, \
>>> +        .priv_class     = &rkmpp_##NAME##_dec_class, \
>>> +        .capabilities   = AV_CODEC_CAP_DELAY, \
>> Add AV_CODEC_CAP_AVOID_PROBING as well.
> What would that flag change ?

Avoids using this decoder for probing. Read the flag's doxy in avcodec.h
for the rationale.

>>
>>> +        .caps_internal  = FF_CODEC_CAP_SETS_PKT_DTS, \
>> It doesn't look like you're setting pkt_dts anywhere?
> True, I can remove that one.
>>
>>> +        .pix_fmts       = (const enum AVPixelFormat[]) { AV_PIX_FMT_DRM_PRIME, \
>>> +                                                         AV_PIX_FMT_NONE}, \
>>> +        .bsfs           = BSFS, \
>>> +    };
>>> +
>>> +RKMPP_DEC(h264,  AV_CODEC_ID_H264,          "h264_mp4toannexb")
>>> +RKMPP_DEC(hevc,  AV_CODEC_ID_HEVC,          "hevc_mp4toannexb")
>>> +RKMPP_DEC(mpeg4, AV_CODEC_ID_MPEG4,         NULL)
>>> +RKMPP_DEC(vp8,   AV_CODEC_ID_VP8,           NULL)
>>> +RKMPP_DEC(vp9,   AV_CODEC_ID_VP9,           NULL)
>>>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
diff mbox

Patch

diff --git a/Changelog b/Changelog
index 189a803..ccf78b2 100644
--- a/Changelog
+++ b/Changelog
@@ -44,6 +44,7 @@  version <next>:
 - drop deprecated qtkit input device (use avfoundation instead)
 - despill video filter
 - haas audio filter
+- Rockchip MPP hardware decoding
 
 version 3.3:
 - CrystalHD decoder moved to new decode API
diff --git a/configure b/configure
index 3ba27c5..cbff908 100755
--- a/configure
+++ b/configure
@@ -314,6 +314,7 @@  External library support:
   --disable-nvenc          disable Nvidia video encoding code [autodetect]
   --enable-omx             enable OpenMAX IL code [no]
   --enable-omx-rpi         enable OpenMAX IL code for Raspberry Pi [no]
+  --enable-rkmpp           enable Rockchip Media Process Platform code [no]
   --disable-vaapi          disable Video Acceleration API (mainly Unix/Intel) code [autodetect]
   --disable-vda            disable Apple Video Decode Acceleration code [autodetect]
   --disable-vdpau          disable Nvidia Video Decode and Presentation API for Unix code [autodetect]
@@ -1545,6 +1546,7 @@  EXTERNAL_LIBRARY_VERSION3_LIST="
     libopencore_amrnb
     libopencore_amrwb
     libvo_amrwbenc
+    rkmpp
 "
 
 EXTERNAL_LIBRARY_GPLV3_LIST="
@@ -2774,6 +2776,8 @@  h264_qsv_decoder_deps="libmfx"
 h264_qsv_decoder_select="h264_mp4toannexb_bsf h264_parser qsvdec h264_qsv_hwaccel"
 h264_qsv_encoder_deps="libmfx"
 h264_qsv_encoder_select="qsvenc"
+h264_rkmpp_decoder_deps="rkmpp"
+h264_rkmpp_decoder_select="h264_mp4toannexb_bsf"
 h264_vaapi_encoder_deps="VAEncPictureParameterBufferH264"
 h264_vaapi_encoder_select="vaapi_encode golomb"
 h264_vda_decoder_deps="vda"
@@ -2789,6 +2793,8 @@  hevc_qsv_decoder_deps="libmfx"
 hevc_qsv_decoder_select="hevc_mp4toannexb_bsf hevc_parser qsvdec hevc_qsv_hwaccel"
 hevc_qsv_encoder_deps="libmfx"
 hevc_qsv_encoder_select="hevcparse qsvenc"
+hevc_rkmpp_decoder_deps="rkmpp"
+hevc_rkmpp_decoder_select="hevc_mp4toannexb_bsf"
 hevc_vaapi_encoder_deps="VAEncPictureParameterBufferHEVC"
 hevc_vaapi_encoder_select="vaapi_encode golomb"
 mjpeg_cuvid_decoder_deps="cuda cuvid"
@@ -2812,6 +2818,7 @@  mpeg4_cuvid_decoder_deps="cuda cuvid"
 mpeg4_mediacodec_decoder_deps="mediacodec"
 mpeg4_mmal_decoder_deps="mmal"
 mpeg4_omx_encoder_deps="omx"
+mpeg4_rkmpp_decoder_deps="rkmpp"
 mpeg4_vdpau_decoder_deps="vdpau"
 mpeg4_vdpau_decoder_select="mpeg4_decoder"
 mpeg_vdpau_decoder_deps="vdpau"
@@ -2828,10 +2835,12 @@  vp8_cuvid_decoder_deps="cuda cuvid"
 vp8_mediacodec_decoder_deps="mediacodec"
 vp8_qsv_decoder_deps="libmfx"
 vp8_qsv_decoder_select="qsvdec vp8_qsv_hwaccel vp8_parser"
+vp8_rkmpp_decoder_deps="rkmpp"
 vp8_vaapi_encoder_deps="VAEncPictureParameterBufferVP8"
 vp8_vaapi_encoder_select="vaapi_encode"
 vp9_cuvid_decoder_deps="cuda cuvid"
 vp9_mediacodec_decoder_deps="mediacodec"
+vp9_rkmpp_decoder_deps="rkmpp"
 vp9_vaapi_encoder_deps="VAEncPictureParameterBufferVP9"
 vp9_vaapi_encoder_select="vaapi_encode"
 wmv3_crystalhd_decoder_select="crystalhd"
@@ -6032,6 +6041,13 @@  enabled openssl           && { use_pkg_config openssl openssl/ssl.h OPENSSL_init
                                check_lib openssl openssl/ssl.h SSL_library_init -lssl32 -leay32 ||
                                check_lib openssl openssl/ssl.h SSL_library_init -lssl -lcrypto -lws2_32 -lgdi32 ||
                                die "ERROR: openssl not found"; }
+enabled rkmpp             && { { require_pkg_config rockchip_mpp rockchip/rk_mpi.h mpp_create ||
+                                 die "ERROR : Rockchip MPP was not found."; } &&
+                               { check_func_headers rockchip/rk_mpi_cmd.h "MPP_DEC_GET_FREE_PACKET_SLOT_COUNT" ||
+                                 die "ERROR: Rockchip MPP is outdated, please get a more recent one."; } &&
+                               { enabled libdrm ||
+                                 die "ERROR: rkmpp requires --enable-libdrm"; }
+                             }
 
 if enabled gcrypt; then
     GCRYPT_CONFIG="${cross_prefix}libgcrypt-config"
diff --git a/libavcodec/Makefile b/libavcodec/Makefile
index 999632c..3a86996 100644
--- a/libavcodec/Makefile
+++ b/libavcodec/Makefile
@@ -338,6 +338,7 @@  OBJS-$(CONFIG_H264_VDA_DECODER)        += vda_h264_dec.o
 OBJS-$(CONFIG_H264_OMX_ENCODER)        += omx.o
 OBJS-$(CONFIG_H264_QSV_DECODER)        += qsvdec_h2645.o
 OBJS-$(CONFIG_H264_QSV_ENCODER)        += qsvenc_h264.o
+OBJS-$(CONFIG_H264_RKMPP_DECODER)      += rkmppdec.o
 OBJS-$(CONFIG_H264_VAAPI_ENCODER)      += vaapi_encode_h264.o vaapi_encode_h26x.o
 OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o
 OBJS-$(CONFIG_HAP_DECODER)             += hapdec.o hap.o
@@ -352,6 +353,7 @@  OBJS-$(CONFIG_NVENC_HEVC_ENCODER)      += nvenc_hevc.o
 OBJS-$(CONFIG_HEVC_QSV_DECODER)        += qsvdec_h2645.o
 OBJS-$(CONFIG_HEVC_QSV_ENCODER)        += qsvenc_hevc.o hevc_ps_enc.o       \
                                           hevc_data.o
+OBJS-$(CONFIG_HEVC_RKMPP_DECODER)      += rkmppdec.o
 OBJS-$(CONFIG_HEVC_VAAPI_ENCODER)      += vaapi_encode_h265.o vaapi_encode_h26x.o
 OBJS-$(CONFIG_HNM4_VIDEO_DECODER)      += hnm4video.o
 OBJS-$(CONFIG_HQ_HQA_DECODER)          += hq_hqa.o hq_hqadata.o hq_hqadsp.o \
@@ -431,6 +433,7 @@  OBJS-$(CONFIG_MPEG2_MEDIACODEC_DECODER) += mediacodecdec.o
 OBJS-$(CONFIG_MPEG2_VAAPI_ENCODER)     += vaapi_encode_mpeg2.o
 OBJS-$(CONFIG_MPEG4_DECODER)           += xvididct.o
 OBJS-$(CONFIG_MPEG4_MEDIACODEC_DECODER) += mediacodecdec.o
+OBJS-$(CONFIG_MPEG4_RKMPP_DECODER)      += rkmppdec.o
 OBJS-$(CONFIG_MPEG4_OMX_ENCODER)       += omx.o
 OBJS-$(CONFIG_MPL2_DECODER)            += mpl2dec.o ass.o
 OBJS-$(CONFIG_MSA1_DECODER)            += mss3.o
@@ -623,12 +626,14 @@  OBJS-$(CONFIG_VP8_DECODER)             += vp8.o vp56rac.o
 OBJS-$(CONFIG_VP8_CUVID_DECODER)       += cuvid.o
 OBJS-$(CONFIG_VP8_MEDIACODEC_DECODER)  += mediacodecdec.o
 OBJS-$(CONFIG_VP8_QSV_DECODER)         += qsvdec_other.o
+OBJS-$(CONFIG_VP8_RKMPP_DECODER)       += rkmppdec.o
 OBJS-$(CONFIG_VP8_VAAPI_ENCODER)       += vaapi_encode_vp8.o
 OBJS-$(CONFIG_VP9_DECODER)             += vp9.o vp9data.o vp9dsp.o vp9lpf.o vp9recon.o \
                                           vp9block.o vp9prob.o vp9mvs.o vp56rac.o \
                                           vp9dsp_8bpp.o vp9dsp_10bpp.o vp9dsp_12bpp.o
 OBJS-$(CONFIG_VP9_CUVID_DECODER)       += cuvid.o
 OBJS-$(CONFIG_VP9_MEDIACODEC_DECODER)  += mediacodecdec.o
+OBJS-$(CONFIG_VP9_RKMPP_DECODER)       += rkmppdec.o
 OBJS-$(CONFIG_VP9_VAAPI_ENCODER)       += vaapi_encode_vp9.o
 OBJS-$(CONFIG_VPLAYER_DECODER)         += textdec.o ass.o
 OBJS-$(CONFIG_VQA_DECODER)             += vqavideo.o
diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
index ce0bc7e..3335c11 100644
--- a/libavcodec/allcodecs.c
+++ b/libavcodec/allcodecs.c
@@ -213,6 +213,7 @@  static void register_all(void)
     REGISTER_DECODER(H264_MEDIACODEC,   h264_mediacodec);
     REGISTER_DECODER(H264_MMAL,         h264_mmal);
     REGISTER_DECODER(H264_QSV,          h264_qsv);
+    REGISTER_DECODER(H264_RKMPP,        h264_rkmpp);
     REGISTER_DECODER(H264_VDA,          h264_vda);
 #if FF_API_VDPAU
     REGISTER_DECODER(H264_VDPAU,        h264_vdpau);
@@ -220,6 +221,7 @@  static void register_all(void)
     REGISTER_ENCDEC (HAP,               hap);
     REGISTER_DECODER(HEVC,              hevc);
     REGISTER_DECODER(HEVC_QSV,          hevc_qsv);
+    REGISTER_DECODER(HEVC_RKMPP,        hevc_rkmpp);
     REGISTER_DECODER(HNM4_VIDEO,        hnm4_video);
     REGISTER_DECODER(HQ_HQA,            hq_hqa);
     REGISTER_DECODER(HQX,               hqx);
@@ -255,6 +257,7 @@  static void register_all(void)
     REGISTER_ENCDEC (MPEG4,             mpeg4);
     REGISTER_DECODER(MPEG4_CRYSTALHD,   mpeg4_crystalhd);
     REGISTER_DECODER(MPEG4_MMAL,        mpeg4_mmal);
+    REGISTER_DECODER(MPEG4_RKMPP,       mpeg4_rkmpp);
 #if FF_API_VDPAU
     REGISTER_DECODER(MPEG4_VDPAU,       mpeg4_vdpau);
 #endif
@@ -373,7 +376,9 @@  static void register_all(void)
     REGISTER_DECODER(VP6F,              vp6f);
     REGISTER_DECODER(VP7,               vp7);
     REGISTER_DECODER(VP8,               vp8);
+    REGISTER_DECODER(VP8_RKMPP,         vp8_rkmpp);
     REGISTER_DECODER(VP9,               vp9);
+    REGISTER_DECODER(VP9_RKMPP,         vp9_rkmpp);
     REGISTER_DECODER(VQA,               vqa);
     REGISTER_DECODER(BITPACKED,         bitpacked);
     REGISTER_DECODER(WEBP,              webp);
diff --git a/libavcodec/rkmppdec.c b/libavcodec/rkmppdec.c
new file mode 100644
index 0000000..33f7f5d
--- /dev/null
+++ b/libavcodec/rkmppdec.c
@@ -0,0 +1,584 @@ 
+/*
+ * RockChip MPP Video Decoder
+ * Copyright (c) 2017 Lionel CHAZALLON
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include <drm_fourcc.h>
+#include <pthread.h>
+#include <rockchip/mpp_buffer.h>
+#include <rockchip/rk_mpi.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "avcodec.h"
+#include "decode.h"
+#include "internal.h"
+#include "libavutil/buffer.h"
+#include "libavutil/common.h"
+#include "libavutil/frame.h"
+#include "libavutil/hwcontext.h"
+#include "libavutil/hwcontext_drm.h"
+#include "libavutil/imgutils.h"
+#include "libavutil/log.h"
+
+#define RECEIVE_FRAME_TIMEOUT   100
+#define FRAMEGROUP_MAX_FRAMES   16
+
+typedef struct {
+    MppCtx ctx;
+    MppApi *mpi;
+    MppBufferGroup frame_group;
+
+    char first_frame;
+    char first_packet;
+    char eos_reached;
+
+    AVBufferRef *frames_ref;
+    AVBufferRef *device_ref;
+} RKMPPDecoder;
+
+typedef struct {
+    AVClass *av_class;
+    AVBufferRef *decoder_ref;
+} RKMPPDecodeContext;
+
+typedef struct {
+    MppFrame frame;
+    AVBufferRef *decoder_ref;
+} RKMPPFrameContext;
+
+static MppCodingType rkmpp_get_codingtype(AVCodecContext *avctx)
+{
+    switch (avctx->codec_id) {
+    case AV_CODEC_ID_H264:          return MPP_VIDEO_CodingAVC;
+    case AV_CODEC_ID_HEVC:          return MPP_VIDEO_CodingHEVC;
+    case AV_CODEC_ID_MPEG4:         return MPP_VIDEO_CodingMPEG4;
+    case AV_CODEC_ID_VP8:           return MPP_VIDEO_CodingVP8;
+    case AV_CODEC_ID_VP9:           return MPP_VIDEO_CodingVP9;
+    default:                        return MPP_VIDEO_CodingUnused;
+    }
+}
+
+static int rkmpp_get_frameformat(MppFrameFormat mppformat)
+{
+    switch (mppformat) {
+    case MPP_FMT_YUV420SP:          return DRM_FORMAT_NV12;
+#ifdef DRM_FORMAT_NV12_10
+    case MPP_FMT_YUV420SP_10BIT:    return DRM_FORMAT_NV12_10;
+#endif
+    default:                        return 0;
+    }
+}
+
+static int rkmpp_write_data(AVCodecContext *avctx, uint8_t *buffer, int size, int64_t pts)
+{
+    RKMPPDecodeContext *rk_context = avctx->priv_data;
+    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
+    int ret = MPP_NOK;
+    MppPacket packet;
+
+    // create the MPP packet
+    ret = mpp_packet_init(&packet, buffer, size);
+    if (ret != MPP_OK) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to init MPP packet (code = %d)\n", ret);
+        return AVERROR_UNKNOWN;
+    }
+
+    mpp_packet_set_pts(packet, pts);
+
+    if (!buffer)
+        mpp_packet_set_eos(packet);
+
+    ret = decoder->mpi->decode_put_packet(decoder->ctx, packet);
+    if (ret != MPP_OK) {
+        if (ret == MPP_ERR_BUFFER_FULL) {
+            av_log(avctx, AV_LOG_DEBUG, "Buffer full writing %d bytes to decoder\n", size);
+            ret = AVERROR(EAGAIN);
+        } else
+            ret = AVERROR_UNKNOWN;
+    }
+    else
+        av_log(avctx, AV_LOG_DEBUG, "Wrote %d bytes to decoder\n", size);
+
+    mpp_packet_deinit(&packet);
+
+    return ret;
+}
+
+static int rkmpp_close_decoder(AVCodecContext *avctx)
+{
+    RKMPPDecodeContext *rk_context = avctx->priv_data;
+    av_buffer_unref(&rk_context->decoder_ref);
+    return 0;
+}
+
+static void rkmpp_release_decoder(void *opaque, uint8_t *data)
+{
+    RKMPPDecoder *decoder = (RKMPPDecoder *)data;
+
+    decoder->mpi->reset(decoder->ctx);
+    mpp_destroy(decoder->ctx);
+    decoder->ctx = NULL;
+
+    if (decoder->frame_group) {
+        mpp_buffer_group_put(decoder->frame_group);
+        decoder->frame_group = NULL;
+    }
+
+    av_buffer_unref(&decoder->frames_ref);
+    av_buffer_unref(&decoder->device_ref);
+
+    av_free(decoder);
+}
+
+static int rkmpp_init_decoder(AVCodecContext *avctx)
+{
+    RKMPPDecodeContext *rk_context = avctx->priv_data;
+    RKMPPDecoder *decoder = NULL;
+    MppCodingType codectype = MPP_VIDEO_CodingUnused;
+    int ret = MPP_NOK;
+    RK_S64 paramS64;
+    RK_S32 paramS32;
+
+    avctx->pix_fmt = AV_PIX_FMT_DRM_PRIME;
+
+    // create a decoder and a ref to it
+    decoder = av_mallocz(sizeof(RKMPPDecoder));
+    if (!decoder) {
+        ret = AVERROR(ENOMEM);
+        goto fail;
+    }
+
+    rk_context->decoder_ref = av_buffer_create((uint8_t *)decoder, sizeof(*decoder), rkmpp_release_decoder,
+                                               NULL, AV_BUFFER_FLAG_READONLY);
+    if (!rk_context->decoder_ref) {
+        av_free(decoder);
+        ret = AVERROR(ENOMEM);
+        goto fail;
+    }
+
+    av_log(avctx, AV_LOG_DEBUG, "Initializing RKMPP decoder.\n");
+
+    codectype = rkmpp_get_codingtype(avctx);
+    if (codectype == MPP_VIDEO_CodingUnused) {
+        av_log(avctx, AV_LOG_ERROR, "Unknown codec type (%d).\n", avctx->codec_id);
+        ret = AVERROR_UNKNOWN;
+        goto fail;
+    }
+
+    // Create the MPP context
+    ret = mpp_create(&decoder->ctx, &decoder->mpi);
+    if (ret != MPP_OK) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to create MPP context (code = %d).\n", ret);
+        ret = AVERROR_UNKNOWN;
+        goto fail;
+    }
+
+    // initialize mpp
+    ret = mpp_init(decoder->ctx, MPP_CTX_DEC, codectype);
+    if (ret != MPP_OK) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to initialize MPP context (code = %d).\n", ret);
+        ret = AVERROR_UNKNOWN;
+        goto fail;
+    }
+
+    // make decode calls blocking with a timeout
+    paramS32 = MPP_POLL_BLOCK;
+    ret = decoder->mpi->control(decoder->ctx, MPP_SET_OUTPUT_BLOCK, &paramS32);
+    if (ret != MPP_OK) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to set blocking mode on MPI (code = %d).\n", ret);
+        ret = AVERROR_UNKNOWN;
+        goto fail;
+    }
+
+    paramS64 = RECEIVE_FRAME_TIMEOUT;
+    ret = decoder->mpi->control(decoder->ctx, MPP_SET_OUTPUT_BLOCK_TIMEOUT, &paramS64);
+    if (ret != MPP_OK) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to set block timeout on MPI (code = %d).\n", ret);
+        ret = AVERROR_UNKNOWN;
+        goto fail;
+    }
+
+    ret = mpp_buffer_group_get_internal(&decoder->frame_group, MPP_BUFFER_TYPE_ION);
+    if (ret) {
+       av_log(avctx, AV_LOG_ERROR, "Failed to retrieve buffer group (code = %d)\n", ret);
+       ret = AVERROR_UNKNOWN;
+       goto fail;
+    }
+
+    ret = decoder->mpi->control(decoder->ctx, MPP_DEC_SET_EXT_BUF_GROUP, decoder->frame_group);
+    if (ret) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to assign buffer group (code = %d)\n", ret);
+        ret = AVERROR_UNKNOWN;
+        goto fail;
+    }
+
+    ret = mpp_buffer_group_limit_config(decoder->frame_group, 0, FRAMEGROUP_MAX_FRAMES);
+    if (ret) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to set buffer group limit (code = %d)\n", ret);
+        ret = AVERROR_UNKNOWN;
+        goto fail;
+    }
+
+    decoder->first_packet = 1;
+
+    av_log(avctx, AV_LOG_DEBUG, "RKMPP decoder initialized successfully.\n");
+
+    decoder->device_ref = av_hwdevice_ctx_alloc(AV_HWDEVICE_TYPE_DRM);
+    if (!decoder->device_ref) {
+        ret = AVERROR(ENOMEM);
+        goto fail;
+    }
+    ret = av_hwdevice_ctx_init(decoder->device_ref);
+    if (ret < 0)
+        goto fail;
+
+    return 0;
+
+fail:
+    av_log(avctx, AV_LOG_ERROR, "Failed to initialize RKMPP decoder.\n");
+    rkmpp_close_decoder(avctx);
+    return ret;
+}
+
+static int rkmpp_send_packet(AVCodecContext *avctx, const AVPacket *avpkt)
+{
+    RKMPPDecodeContext *rk_context = avctx->priv_data;
+    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
+    int ret = MPP_NOK;
+
+    // handle EOF
+    if (!avpkt->size) {
+        av_log(avctx, AV_LOG_DEBUG, "End of stream.\n");
+        decoder->eos_reached = 1;
+        ret = rkmpp_write_data(avctx, NULL, 0, 0);
+        if (ret)
+            av_log(avctx, AV_LOG_ERROR, "Failed to send EOS to decoder (code = %d)\n", ret);
+        return ret;
+    }
+
+    // on first packet, send extradata
+    if (decoder->first_packet) {
+        if (avctx->extradata_size) {
+            ret = rkmpp_write_data(avctx, avctx->extradata,
+                                            avctx->extradata_size,
+                                            avpkt->pts);
+            if (ret) {
+                av_log(avctx, AV_LOG_ERROR, "Failed to write extradata to decoder (code = %d)\n", ret);
+                return ret;
+            }
+        }
+        decoder->first_packet = 0;
+    }
+
+    // now send packet
+    ret = rkmpp_write_data(avctx, avpkt->data, avpkt->size, avpkt->pts);
+    if (ret && ret!=AVERROR(EAGAIN))
+        av_log(avctx, AV_LOG_ERROR, "Failed to write data to decoder (code = %d)\n", ret);
+
+    return ret;
+}
+
+static void rkmpp_release_frame(void *opaque, uint8_t *data)
+{
+    AVDRMFrameDescriptor *desc = (AVDRMFrameDescriptor *)data;
+    AVBufferRef *framecontextref = (AVBufferRef *)opaque;
+    RKMPPFrameContext *framecontext = (RKMPPFrameContext *)framecontextref->data;
+
+    mpp_frame_deinit(&framecontext->frame);
+    av_buffer_unref(&framecontext->decoder_ref);
+    av_buffer_unref(&framecontextref);
+
+    av_free(desc);
+}
+
+static int rkmpp_retrieve_frame(AVCodecContext *avctx, AVFrame *frame)
+{
+    RKMPPDecodeContext *rk_context = avctx->priv_data;
+    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
+    RKMPPFrameContext *framecontext = NULL;
+    AVBufferRef *framecontextref = NULL;
+    int ret = MPP_NOK;
+    MppFrame mppframe = NULL;
+    MppBuffer buffer = NULL;
+    AVDRMFrameDescriptor *desc = NULL;
+    AVDRMLayerDescriptor *layer = NULL;
+    int retrycount = 0;
+    int format;
+
+    // on start of decoding, MPP can return -1, which is supposed to be expected
+    // this is due to some internal MPP init which is not completed, that will
+    // only happen in the first few frames queries, but should not be interpreted
+    // as an error, Therefore we need to retry a couple times when we get -1
+    // in order to let it time to complete it's init, then we sleep a bit between retries.
+retry_get_frame:
+    ret = decoder->mpi->decode_get_frame(decoder->ctx, &mppframe);
+    if (ret != MPP_OK && ret != MPP_ERR_TIMEOUT && !decoder->first_frame) {
+        if (retrycount < 5) {
+            av_log(avctx, AV_LOG_DEBUG, "Failed to get a frame, retrying (code = %d, retrycount = %d)\n", ret, retrycount);
+            usleep(10000);
+            retrycount++;
+            goto retry_get_frame;
+        } else {
+            av_log(avctx, AV_LOG_ERROR, "Failed to get a frame from MPP (code = %d)\n", ret);
+            goto fail;
+        }
+    }
+
+    if (mppframe) {
+        // Check wether we have a special frame or not
+        if (mpp_frame_get_info_change(mppframe)) {
+            AVHWFramesContext *hwframes;
+
+            av_log(avctx, AV_LOG_INFO, "Decoder noticed an info change (%dx%d), format=%d\n",
+                                        (int)mpp_frame_get_width(mppframe), (int)mpp_frame_get_height(mppframe),
+                                        (int)mpp_frame_get_fmt(mppframe));
+
+            avctx->width = mpp_frame_get_width(mppframe);
+            avctx->height = mpp_frame_get_height(mppframe);
+
+            decoder->mpi->control(decoder->ctx, MPP_DEC_SET_INFO_CHANGE_READY, NULL);
+            decoder->first_frame = 1;
+
+            av_buffer_unref(&decoder->frames_ref);
+
+            decoder->frames_ref = av_hwframe_ctx_alloc(decoder->device_ref);
+            if (!decoder->frames_ref) {
+                ret = AVERROR(ENOMEM);
+                goto fail;
+            }
+
+            format = (int)mpp_frame_get_fmt(mppframe);
+
+            hwframes = (AVHWFramesContext*)decoder->frames_ref->data;
+            hwframes->format    = AV_PIX_FMT_DRM_PRIME;
+            hwframes->sw_format = rkmpp_get_frameformat(format) == DRM_FORMAT_NV12 ? AV_PIX_FMT_NV12 : AV_PIX_FMT_NONE;
+            hwframes->width     = avctx->width;
+            hwframes->height    = avctx->height;
+            ret = av_hwframe_ctx_init(decoder->frames_ref);
+            if (ret < 0)
+                goto fail;
+
+            // here decoder is fully initialized, we need to feed it again with data
+            ret = AVERROR(EAGAIN);
+            goto fail;
+        } else if (mpp_frame_get_eos(mppframe)) {
+            av_log(avctx, AV_LOG_DEBUG, "Received a EOS frame.\n");
+            decoder->eos_reached = 1;
+            ret = AVERROR_EOF;
+            goto fail;
+        } else if (mpp_frame_get_discard(mppframe)) {
+            av_log(avctx, AV_LOG_DEBUG, "Received a discard frame.\n");
+            ret = AVERROR(EAGAIN);
+            goto fail;
+        } else if (mpp_frame_get_errinfo(mppframe)) {
+            av_log(avctx, AV_LOG_ERROR, "Received a errinfo frame.\n");
+            ret = AVERROR_UNKNOWN;
+            goto fail;
+        }
+
+        // here we should have a valid frame
+        av_log(avctx, AV_LOG_DEBUG, "Received a frame.\n");
+
+        // setup general frame fields
+        frame->format           = AV_PIX_FMT_DRM_PRIME;
+        frame->width            = mpp_frame_get_width(mppframe);
+        frame->height           = mpp_frame_get_height(mppframe);
+        frame->pts              = mpp_frame_get_pts(mppframe);
+        frame->color_range      = mpp_frame_get_color_range(mppframe);
+        frame->color_primaries  = mpp_frame_get_color_primaries(mppframe);
+        frame->color_trc        = mpp_frame_get_color_trc(mppframe);
+        frame->colorspace       = mpp_frame_get_colorspace(mppframe);
+
+        int mode = mpp_frame_get_mode(mppframe);
+        frame->interlaced_frame = ((mode & MPP_FRAME_FLAG_FIELD_ORDER_MASK) == MPP_FRAME_FLAG_DEINTERLACED);
+        frame->top_field_first  = ((mode & MPP_FRAME_FLAG_FIELD_ORDER_MASK) == MPP_FRAME_FLAG_TOP_FIRST);
+
+
+        // now setup the frame buffer info
+        buffer = mpp_frame_get_buffer(mppframe);
+        if (buffer) {
+            desc = av_mallocz(sizeof(AVDRMFrameDescriptor));
+            if (!desc) {
+                ret = AVERROR(ENOMEM);
+                goto fail;
+            }
+
+            desc->nb_objects = 1;
+            desc->objects[0].fd = mpp_buffer_get_fd(buffer);
+            desc->objects[0].size = mpp_buffer_get_size(buffer);
+
+            desc->nb_layers = 1;
+            layer = &desc->layers[0];
+            layer->format = rkmpp_get_frameformat(mpp_frame_get_fmt(mppframe));
+            layer->nb_planes = 2;
+
+            layer->planes[0].object_index = 0;
+            layer->planes[0].offset = 0;
+            layer->planes[0].pitch = mpp_frame_get_hor_stride(mppframe);
+
+            layer->planes[1].object_index = 0;
+            layer->planes[1].offset = layer->planes[0].pitch * mpp_frame_get_ver_stride(mppframe);
+            layer->planes[1].pitch = layer->planes[0].pitch;
+
+            // we also allocate a struct in buf[0] that will allow to hold additionnal information
+            // for releasing properly MPP frames and decoder
+            framecontextref = av_buffer_allocz(sizeof(*framecontext));
+            if (!framecontextref) {
+                ret = AVERROR(ENOMEM);
+                goto fail;
+            }
+
+            // MPP decoder needs to be closed only when all frames have been released.
+            framecontext = (RKMPPFrameContext *)framecontextref->data;
+            framecontext->decoder_ref = av_buffer_ref(rk_context->decoder_ref);
+            framecontext->frame = mppframe;
+
+            frame->data[0]  = (uint8_t *)desc;
+            frame->buf[0]   = av_buffer_create((uint8_t *)desc, sizeof(*desc), rkmpp_release_frame,
+                                               framecontextref, AV_BUFFER_FLAG_READONLY);
+
+            if (!frame->buf[0]) {
+                ret = AVERROR(ENOMEM);
+                goto fail;
+            }
+
+            frame->hw_frames_ctx = av_buffer_ref(decoder->frames_ref);
+            if (!frame->hw_frames_ctx) {
+                ret = AVERROR(ENOMEM);
+                goto fail;
+            }
+
+            decoder->first_frame = 0;
+            return 0;
+        } else {
+            av_log(avctx, AV_LOG_ERROR, "Failed to retrieve the frame buffer, frame is dropped (code = %d)\n", ret);
+            mpp_frame_deinit(&mppframe);
+        }
+    } else if (decoder->eos_reached) {
+        return AVERROR_EOF;
+    } else if (ret == MPP_ERR_TIMEOUT) {
+        av_log(avctx, AV_LOG_DEBUG, "Timeout when trying to get a frame from MPP\n");
+    }
+
+    return AVERROR(EAGAIN);
+
+fail:
+    if (mppframe)
+        mpp_frame_deinit(&mppframe);
+
+    if (framecontext)
+        av_buffer_unref(&framecontext->decoder_ref);
+
+    if (framecontextref)
+        av_buffer_unref(&framecontextref);
+
+    if (desc)
+        av_free(desc);
+
+    return ret;
+}
+
+static int rkmpp_receive_frame(AVCodecContext *avctx, AVFrame *frame)
+{
+    RKMPPDecodeContext *rk_context = avctx->priv_data;
+    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
+    int ret = MPP_NOK;
+    AVPacket pkt = {0};
+    RK_S32 freeslots;
+
+    if (!decoder->eos_reached) {
+        // we get the available slots in decoder
+        ret = decoder->mpi->control(decoder->ctx, MPP_DEC_GET_FREE_PACKET_SLOT_COUNT, &freeslots);
+        if (ret != MPP_OK) {
+            av_log(avctx, AV_LOG_ERROR, "Failed to get decoder free slots (code = %d).\n", ret);
+            return ret;
+        }
+
+        if (freeslots > 0) {
+            ret = ff_decode_get_packet(avctx, &pkt);
+            if (ret < 0 && ret != AVERROR_EOF) {
+                return ret;
+            }
+
+            ret = rkmpp_send_packet(avctx, &pkt);
+            av_packet_unref(&pkt);
+
+            if (ret < 0) {
+                av_log(avctx, AV_LOG_ERROR, "Failed to send packet to decoder (code = %d)\n", ret);
+                return ret;
+            }
+        }
+
+        // make sure we keep decoder full
+        if (freeslots > 1 && decoder->first_frame)
+            return AVERROR(EAGAIN);
+    }
+
+    return rkmpp_retrieve_frame(avctx, frame);
+}
+
+static void rkmpp_flush(AVCodecContext *avctx)
+{
+    RKMPPDecodeContext *rk_context = avctx->priv_data;
+    RKMPPDecoder *decoder = (RKMPPDecoder *)rk_context->decoder_ref->data;
+    int ret = MPP_NOK;
+
+    av_log(avctx, AV_LOG_DEBUG, "Flush.\n");
+
+    ret = decoder->mpi->reset(decoder->ctx);
+    if (ret == MPP_OK) {
+        decoder->first_frame = 1;
+        decoder->first_packet = 1;
+    } else
+        av_log(avctx, AV_LOG_ERROR, "Failed to reset MPI (code = %d)\n", ret);
+}
+
+
+#define RKMPP_DEC_CLASS(NAME) \
+    static const AVClass rkmpp_##NAME##_dec_class = { \
+        .class_name = "rkmpp_" #NAME "_dec", \
+        .version    = LIBAVUTIL_VERSION_INT, \
+    };
+
+#define RKMPP_DEC(NAME, ID, BSFS) \
+    RKMPP_DEC_CLASS(NAME) \
+    AVCodec ff_##NAME##_rkmpp_decoder = { \
+        .name           = #NAME "_rkmpp", \
+        .long_name      = NULL_IF_CONFIG_SMALL(#NAME " (rkmpp)"), \
+        .type           = AVMEDIA_TYPE_VIDEO, \
+        .id             = ID, \
+        .priv_data_size = sizeof(RKMPPDecodeContext), \
+        .init           = rkmpp_init_decoder, \
+        .close          = rkmpp_close_decoder, \
+        .receive_frame  = rkmpp_receive_frame, \
+        .flush          = rkmpp_flush, \
+        .priv_class     = &rkmpp_##NAME##_dec_class, \
+        .capabilities   = AV_CODEC_CAP_DELAY, \
+        .caps_internal  = FF_CODEC_CAP_SETS_PKT_DTS, \
+        .pix_fmts       = (const enum AVPixelFormat[]) { AV_PIX_FMT_DRM_PRIME, \
+                                                         AV_PIX_FMT_NONE}, \
+        .bsfs           = BSFS, \
+    };
+
+RKMPP_DEC(h264,  AV_CODEC_ID_H264,          "h264_mp4toannexb")
+RKMPP_DEC(hevc,  AV_CODEC_ID_HEVC,          "hevc_mp4toannexb")
+RKMPP_DEC(mpeg4, AV_CODEC_ID_MPEG4,         NULL)
+RKMPP_DEC(vp8,   AV_CODEC_ID_VP8,           NULL)
+RKMPP_DEC(vp9,   AV_CODEC_ID_VP9,           NULL)