| Message ID | 1504383070-27130-1-git-send-email-jorge.ramirez-ortiz@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
On 02/09/17 21:11, Jorge Ramirez-Ortiz wrote: > This patchset enhances Alexis Ballier's original patch and validates > it using Qualcomm's Venus hardware (driver recently landed upstream > [1]). > > This has been tested on Qualcomm's DragonBoard 410c and 820c > Configure/make scripts have been validated on Ubuntu 10.04 and > 16.04. > > Tested decoders: > - h264 > - h263 > - mpeg4 > - vp8 > - vp9 > - hevc > > Tested encoders: > - h264 > - h263 > - mpeg4 > > Tested transcoding (concurrent encoding/decoding) > > Some of the changes introduced: > - v4l2: code cleanup and abstractions added > - v4l2: follow the new encode/decode api. > - v4l2: fix display size for NV12 output pool. > - v4l2: handle EOS. > - v4l2: vp8 and mpeg4 decoding and encoding. > - v4l2: hevc and vp9 support. > - v4l2: generate EOF on dequeue errors. > - v4l2: h264_mp4toannexb filtering. > - v4l2: fixed make install and fate issues. > - v4l2: codecs enabled/disabled depending on pixfmt defined > - v4l2: pass timebase/framerate to the context > - v4l2: runtime decoder reconfiguration. > - v4l2: add more frame information > - v4l2: free hardware resources on last reference being released > - v4l2 encoding: disable b-frames for upstreaming (patch required) > > [1] https://lwn.net/Articles/697956/ > > Reviewed-by: Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > Reviewed-by: Alexis Ballier <aballier@gentoo.org> > Tested-by: Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > --- > Changelog | 1 + > configure | 30 +- > libavcodec/Makefile | 16 + > libavcodec/allcodecs.c | 9 + > libavcodec/v4l2_buffers.c | 918 ++++++++++++++++++++++++++++++++++++++++++ > libavcodec/v4l2_buffers.h | 236 +++++++++++ > libavcodec/v4l2_fmt.c | 184 +++++++++ > libavcodec/v4l2_fmt.h | 34 ++ > libavcodec/v4l2_m2m.c | 452 +++++++++++++++++++++ > libavcodec/v4l2_m2m.h | 70 ++++ > libavcodec/v4l2_m2m_avcodec.h | 32 ++ > libavcodec/v4l2_m2m_dec.c | 213 ++++++++++ > libavcodec/v4l2_m2m_enc.c | 340 ++++++++++++++++ > 13 files changed, 2534 insertions(+), 1 deletion(-) > create mode 100644 libavcodec/v4l2_buffers.c > create mode 100644 libavcodec/v4l2_buffers.h > create mode 100644 libavcodec/v4l2_fmt.c > create mode 100644 libavcodec/v4l2_fmt.h > create mode 100644 libavcodec/v4l2_m2m.c > create mode 100644 libavcodec/v4l2_m2m.h > create mode 100644 libavcodec/v4l2_m2m_avcodec.h > create mode 100644 libavcodec/v4l2_m2m_dec.c > create mode 100644 libavcodec/v4l2_m2m_enc.c git am notes: .git/rebase-apply/patch:1244: new blank line at EOF. + .git/rebase-apply/patch:1676: new blank line at EOF. + .git/rebase-apply/patch:2175: new blank line at EOF. + .git/rebase-apply/patch:2508: new blank line at EOF. + > > diff --git a/Changelog b/Changelog > index 8309417..c6fcda3 100644 > --- a/Changelog > +++ b/Changelog > @@ -40,6 +40,7 @@ version <next>: > They must always be used by name. > - FITS demuxer and decoder > - FITS muxer and encoder > +- V4L2 mem2mem HW accelerated codecs support They aren't really accelerated - they are just hardware, there is no acceleration. > version 3.3: > - CrystalHD decoder moved to new decode API > diff --git a/configure b/configure > index 4f1c172..d9244e7 100755 > --- a/configure > +++ b/configure > @@ -149,6 +149,7 @@ Component options: > --disable-pixelutils disable pixel utils in libavutil > > Individual component options: > + --disable-v4l2_m2m disable V4L2 mem2mem code [autodetect] Should be somewhere further down. > --disable-everything disable all components listed below > --disable-encoder=NAME disable encoder NAME > --enable-encoder=NAME enable encoder NAME > @@ -1433,6 +1434,7 @@ AVCODEC_COMPONENTS=" > > AVDEVICE_COMPONENTS=" > indevs > + v4l2_m2m Likewise; it's not an avdevice component. > outdevs > " > AVFILTER_COMPONENTS=" > @@ -2271,6 +2273,7 @@ map 'eval ${v}_inline_deps=inline_asm' $ARCH_EXT_LIST_ARM > > loongson2_deps="mips" > loongson3_deps="mips" > +v4l2_deps_any="linux_videodev2_h" I don't think this fits in a section with MIPs CPU capabilities. > mipsfpu_deps="mips" > mipsdsp_deps="mips" > mipsdspr2_deps="mips" > @@ -2743,6 +2746,8 @@ nvenc_deps="cuda" > nvenc_deps_any="dlopen LoadLibrary" > nvenc_encoder_deps="nvenc" > > +h263_v4l2m2m_decoder_deps="v4l2_m2m h263_v4l2_m2m" > +h263_v4l2m2m_encoder_deps="v4l2_m2m h263_v4l2_m2m" > h264_crystalhd_decoder_select="crystalhd h264_mp4toannexb_bsf h264_parser" > h264_cuvid_decoder_deps="cuda cuvid" > h264_cuvid_decoder_select="h264_mp4toannexb_bsf" > @@ -2761,6 +2766,8 @@ h264_vda_decoder_deps="vda" > h264_vda_decoder_select="h264_decoder" > h264_vdpau_decoder_deps="vdpau" > h264_vdpau_decoder_select="h264_decoder" > +h264_v4l2m2m_decoder_deps="v4l2_m2m h264_v4l2_m2m" > +h264_v4l2m2m_encoder_deps="v4l2_m2m h264_v4l2_m2m" > hevc_cuvid_decoder_deps="cuda cuvid" > hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf" > hevc_mediacodec_decoder_deps="mediacodec" > @@ -2772,12 +2779,15 @@ hevc_qsv_encoder_deps="libmfx" > hevc_qsv_encoder_select="hevcparse qsvenc" > hevc_vaapi_encoder_deps="VAEncPictureParameterBufferHEVC" > hevc_vaapi_encoder_select="vaapi_encode golomb" > +hevc_v4l2m2m_decoder_deps="v4l2_m2m hevc_v4l2_m2m" > +hevc_v4l2m2m_encoder_deps="v4l2_m2m hevc_v4l2_m2m" > mjpeg_cuvid_decoder_deps="cuda cuvid" > mjpeg_vaapi_encoder_deps="VAEncPictureParameterBufferJPEG" > mjpeg_vaapi_encoder_select="vaapi_encode jpegtables" > mpeg1_cuvid_decoder_deps="cuda cuvid" > mpeg1_vdpau_decoder_deps="vdpau" > mpeg1_vdpau_decoder_select="mpeg1video_decoder" > +mpeg1_v4l2m2m_decoder_deps="v4l2_m2m mpeg1_v4l2_m2m" > mpeg2_crystalhd_decoder_select="crystalhd" > mpeg2_cuvid_decoder_deps="cuda cuvid" > mpeg2_mmal_decoder_deps="mmal" > @@ -2788,6 +2798,7 @@ mpeg2_qsv_encoder_deps="libmfx" > mpeg2_qsv_encoder_select="qsvenc" > mpeg2_vaapi_encoder_deps="VAEncPictureParameterBufferMPEG2" > mpeg2_vaapi_encoder_select="vaapi_encode" > +mpeg2_v4l2m2m_decoder_deps="v4l2_m2m mpeg2_v4l2_m2m" > mpeg4_crystalhd_decoder_select="crystalhd" > mpeg4_cuvid_decoder_deps="cuda cuvid" > mpeg4_mediacodec_decoder_deps="mediacodec" > @@ -2795,6 +2806,8 @@ mpeg4_mmal_decoder_deps="mmal" > mpeg4_omx_encoder_deps="omx" > mpeg4_vdpau_decoder_deps="vdpau" > mpeg4_vdpau_decoder_select="mpeg4_decoder" > +mpeg4_v4l2m2m_decoder_deps="v4l2_m2m mpeg4_v4l2_m2m" > +mpeg4_v4l2m2m_encoder_deps="v4l2_m2m mpeg4_v4l2_m2m" > mpeg_vdpau_decoder_deps="vdpau" > mpeg_vdpau_decoder_select="mpeg2video_decoder" > msmpeg4_crystalhd_decoder_select="crystalhd" > @@ -2805,16 +2818,20 @@ vc1_cuvid_decoder_deps="cuda cuvid" > vc1_mmal_decoder_deps="mmal" > vc1_vdpau_decoder_deps="vdpau" > vc1_vdpau_decoder_select="vc1_decoder" > +vc1_v4l2m2m_decoder_deps="v4l2_m2m vc1_v4l2_m2m" > vp8_cuvid_decoder_deps="cuda cuvid" > vp8_mediacodec_decoder_deps="mediacodec" > vp8_qsv_decoder_deps="libmfx" > vp8_qsv_decoder_select="qsvdec vp8_qsv_hwaccel vp8_parser" > vp8_vaapi_encoder_deps="VAEncPictureParameterBufferVP8" > vp8_vaapi_encoder_select="vaapi_encode" > +vp8_v4l2m2m_decoder_deps="v4l2_m2m vp8_v4l2_m2m" > +vp8_v4l2m2m_encoder_deps="v4l2_m2m vp8_v4l2_m2m" > vp9_cuvid_decoder_deps="cuda cuvid" > vp9_mediacodec_decoder_deps="mediacodec" > vp9_vaapi_encoder_deps="VAEncPictureParameterBufferVP9" > vp9_vaapi_encoder_select="vaapi_encode" > +vp9_v4l2m2m_decoder_deps="v4l2_m2m vp9_v4l2_m2m" > wmv3_crystalhd_decoder_select="crystalhd" > wmv3_vdpau_decoder_select="vc1_vdpau_decoder" > > @@ -3597,7 +3614,7 @@ done > enable_weak audiotoolbox > > # Enable hwaccels by default. > -enable_weak d3d11va dxva2 vaapi vda vdpau videotoolbox_hwaccel xvmc > +enable_weak d3d11va dxva2 vaapi v4l2_m2m vda vdpau videotoolbox_hwaccel xvmc [This line has disappeared on rebase.] > enable_weak xlib > > enable_weak cuda cuvid nvenc vda_framework videotoolbox videotoolbox_encoder > @@ -6066,10 +6083,21 @@ perl -v > /dev/null 2>&1 && enable perl || disable perl > pod2man --help > /dev/null 2>&1 && enable pod2man || disable pod2man > rsync --help 2> /dev/null | grep -q 'contimeout' && enable rsync_contimeout || disable rsync_contimeout > > +# check V4L2 codecs available in the API > check_header linux/fb.h > check_header linux/videodev.h > check_header linux/videodev2.h > check_code cc linux/videodev2.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete > +check_code cc linux/videodev2.h "int i = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_VIDEO_M2M | V4L2_BUF_FLAG_LAST;" || disable v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VC1_ANNEX_G;" && enable vc1_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG1;" && enable mpeg1_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG2;" && enable mpeg2_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG4;" && enable mpeg4_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_HEVC;" && enable hevc_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H263;" && enable h263_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H264;" && enable h264_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP8;" && enable vp8_v4l2_m2m > +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP9;" && enable vp9_v4l2_m2m I think these need to appear as named config options somewhere. It doesn't currently detect anything for me, even with e.g. --enable-v4l2-m2m --enable-encoder=h264_v4l2m2m. > > check_header sys/videoio.h > check_code cc sys/videoio.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete > diff --git a/libavcodec/Makefile b/libavcodec/Makefile > index 999632c..6e1de37 100644 > --- a/libavcodec/Makefile > +++ b/libavcodec/Makefile > @@ -137,6 +137,7 @@ OBJS-$(CONFIG_VIDEODSP) += videodsp.o > OBJS-$(CONFIG_VP3DSP) += vp3dsp.o > OBJS-$(CONFIG_VP56DSP) += vp56dsp.o > OBJS-$(CONFIG_VP8DSP) += vp8dsp.o > +OBJS-$(CONFIG_V4L2_M2M) += v4l2_m2m.o v4l2_buffers.o v4l2_fmt.o > OBJS-$(CONFIG_WMA_FREQS) += wma_freqs.o > OBJS-$(CONFIG_WMV2DSP) += wmv2dsp.o > > @@ -323,6 +324,8 @@ OBJS-$(CONFIG_H263_DECODER) += h263dec.o h263.o ituh263dec.o \ > intelh263dec.o h263data.o > OBJS-$(CONFIG_H263_ENCODER) += mpeg4videoenc.o mpeg4video.o \ > h263.o ituh263enc.o flvenc.o h263data.o > +OBJS-$(CONFIG_H263_V4L2M2M_DECODER) += v4l2_m2m_dec.o > +OBJS-$(CONFIG_H263_V4L2M2M_ENCODER) += v4l2_m2m_enc.o > OBJS-$(CONFIG_H264_DECODER) += h264dec.o h264_cabac.o h264_cavlc.o \ > h264_direct.o h264_loopfilter.o \ > h264_mb.o h264_picture.o \ > @@ -340,6 +343,8 @@ OBJS-$(CONFIG_H264_QSV_DECODER) += qsvdec_h2645.o > OBJS-$(CONFIG_H264_QSV_ENCODER) += qsvenc_h264.o > OBJS-$(CONFIG_H264_VAAPI_ENCODER) += vaapi_encode_h264.o vaapi_encode_h26x.o > OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o > +OBJS-$(CONFIG_H264_V4L2M2M_DECODER) += v4l2_m2m_dec.o > +OBJS-$(CONFIG_H264_V4L2M2M_ENCODER) += v4l2_m2m_enc.o > OBJS-$(CONFIG_HAP_DECODER) += hapdec.o hap.o > OBJS-$(CONFIG_HAP_ENCODER) += hapenc.o hap.o > OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o hevc_mvs.o \ > @@ -353,6 +358,8 @@ OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec_h2645.o > OBJS-$(CONFIG_HEVC_QSV_ENCODER) += qsvenc_hevc.o hevc_ps_enc.o \ > hevc_data.o > OBJS-$(CONFIG_HEVC_VAAPI_ENCODER) += vaapi_encode_h265.o vaapi_encode_h26x.o > +OBJS-$(CONFIG_HEVC_V4L2M2M_DECODER) += v4l2_m2m_dec.o > +OBJS-$(CONFIG_HEVC_V4L2M2M_ENCODER) += v4l2_m2m_enc.o > OBJS-$(CONFIG_HNM4_VIDEO_DECODER) += hnm4video.o > OBJS-$(CONFIG_HQ_HQA_DECODER) += hq_hqa.o hq_hqadata.o hq_hqadsp.o \ > canopus.o > @@ -422,6 +429,7 @@ OBJS-$(CONFIG_MPC8_DECODER) += mpc8.o mpc.o > OBJS-$(CONFIG_MPEGVIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o > OBJS-$(CONFIG_MPEG1VIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o > OBJS-$(CONFIG_MPEG1VIDEO_ENCODER) += mpeg12enc.o mpeg12.o > +OBJS-$(CONFIG_MPEG1_V4L2M2M_DECODER) += v4l2_m2m_dec.o > OBJS-$(CONFIG_MPEG2_MMAL_DECODER) += mmaldec.o > OBJS-$(CONFIG_MPEG2_QSV_DECODER) += qsvdec_other.o > OBJS-$(CONFIG_MPEG2_QSV_ENCODER) += qsvenc_mpeg2.o > @@ -429,9 +437,12 @@ OBJS-$(CONFIG_MPEG2VIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o > OBJS-$(CONFIG_MPEG2VIDEO_ENCODER) += mpeg12enc.o mpeg12.o > OBJS-$(CONFIG_MPEG2_MEDIACODEC_DECODER) += mediacodecdec.o > OBJS-$(CONFIG_MPEG2_VAAPI_ENCODER) += vaapi_encode_mpeg2.o > +OBJS-$(CONFIG_MPEG2_V4L2M2M_DECODER) += v4l2_m2m_dec.o > OBJS-$(CONFIG_MPEG4_DECODER) += xvididct.o > OBJS-$(CONFIG_MPEG4_MEDIACODEC_DECODER) += mediacodecdec.o > OBJS-$(CONFIG_MPEG4_OMX_ENCODER) += omx.o > +OBJS-$(CONFIG_MPEG4_V4L2M2M_DECODER) += v4l2_m2m_dec.o > +OBJS-$(CONFIG_MPEG4_V4L2M2M_ENCODER) += v4l2_m2m_enc.o > OBJS-$(CONFIG_MPL2_DECODER) += mpl2dec.o ass.o > OBJS-$(CONFIG_MSA1_DECODER) += mss3.o > OBJS-$(CONFIG_MSCC_DECODER) += mscc.o > @@ -605,6 +616,7 @@ OBJS-$(CONFIG_VC1_DECODER) += vc1dec.o vc1_block.o vc1_loopfilter.o > OBJS-$(CONFIG_VC1_CUVID_DECODER) += cuvid.o > OBJS-$(CONFIG_VC1_MMAL_DECODER) += mmaldec.o > OBJS-$(CONFIG_VC1_QSV_DECODER) += qsvdec_other.o > +OBJS-$(CONFIG_VC1_V4L2M2M_DECODER) += v4l2_m2m_dec.o > OBJS-$(CONFIG_VC2_ENCODER) += vc2enc.o vc2enc_dwt.o diractab.o > OBJS-$(CONFIG_VCR1_DECODER) += vcr1.o > OBJS-$(CONFIG_VMDAUDIO_DECODER) += vmdaudio.o > @@ -614,6 +626,7 @@ OBJS-$(CONFIG_VORBIS_DECODER) += vorbisdec.o vorbisdsp.o vorbis.o \ > vorbis_data.o > OBJS-$(CONFIG_VORBIS_ENCODER) += vorbisenc.o vorbis.o \ > vorbis_data.o > +OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o Stray change? > OBJS-$(CONFIG_VP3_DECODER) += vp3.o > OBJS-$(CONFIG_VP5_DECODER) += vp5.o vp56.o vp56data.o vp56rac.o > OBJS-$(CONFIG_VP6_DECODER) += vp6.o vp56.o vp56data.o \ > @@ -624,6 +637,8 @@ OBJS-$(CONFIG_VP8_CUVID_DECODER) += cuvid.o > OBJS-$(CONFIG_VP8_MEDIACODEC_DECODER) += mediacodecdec.o > OBJS-$(CONFIG_VP8_QSV_DECODER) += qsvdec_other.o > OBJS-$(CONFIG_VP8_VAAPI_ENCODER) += vaapi_encode_vp8.o > +OBJS-$(CONFIG_VP8_V4L2M2M_DECODER) += v4l2_m2m_dec.o > +OBJS-$(CONFIG_VP8_V4L2M2M_ENCODER) += v4l2_m2m_enc.o > OBJS-$(CONFIG_VP9_DECODER) += vp9.o vp9data.o vp9dsp.o vp9lpf.o vp9recon.o \ > vp9block.o vp9prob.o vp9mvs.o vp56rac.o \ > vp9dsp_8bpp.o vp9dsp_10bpp.o vp9dsp_12bpp.o > @@ -631,6 +646,7 @@ OBJS-$(CONFIG_VP9_CUVID_DECODER) += cuvid.o > OBJS-$(CONFIG_VP9_MEDIACODEC_DECODER) += mediacodecdec.o > OBJS-$(CONFIG_VP9_VAAPI_ENCODER) += vaapi_encode_vp9.o > OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o > +OBJS-$(CONFIG_VP9_V4L2M2M_DECODER) += v4l2_m2m_dec.o > OBJS-$(CONFIG_VQA_DECODER) += vqavideo.o > OBJS-$(CONFIG_WAVPACK_DECODER) += wavpack.o > OBJS-$(CONFIG_WAVPACK_ENCODER) += wavpackenc.o > diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c > index ce0bc7e..eea1ae1 100644 > --- a/libavcodec/allcodecs.c > +++ b/libavcodec/allcodecs.c > @@ -208,8 +208,10 @@ static void register_all(void) > REGISTER_ENCDEC (H263, h263); > REGISTER_DECODER(H263I, h263i); > REGISTER_ENCDEC (H263P, h263p); > + REGISTER_ENCDEC (H263_V4L2M2M, h263_v4l2m2m); > REGISTER_DECODER(H264, h264); > REGISTER_DECODER(H264_CRYSTALHD, h264_crystalhd); > + REGISTER_ENCDEC (H264_V4L2M2M, h264_v4l2m2m); > REGISTER_DECODER(H264_MEDIACODEC, h264_mediacodec); > REGISTER_DECODER(H264_MMAL, h264_mmal); > REGISTER_DECODER(H264_QSV, h264_qsv); > @@ -220,6 +222,7 @@ static void register_all(void) > REGISTER_ENCDEC (HAP, hap); > REGISTER_DECODER(HEVC, hevc); > REGISTER_DECODER(HEVC_QSV, hevc_qsv); > + REGISTER_ENCDEC(HEVC_V4L2M2M, hevc_v4l2m2m); Alignment. > REGISTER_DECODER(HNM4_VIDEO, hnm4_video); > REGISTER_DECODER(HQ_HQA, hq_hqa); > REGISTER_DECODER(HQX, hqx); > @@ -254,6 +257,7 @@ static void register_all(void) > REGISTER_ENCDEC (MPEG2VIDEO, mpeg2video); > REGISTER_ENCDEC (MPEG4, mpeg4); > REGISTER_DECODER(MPEG4_CRYSTALHD, mpeg4_crystalhd); > + REGISTER_ENCDEC (MPEG4_V4L2M2M, mpeg4_v4l2m2m); > REGISTER_DECODER(MPEG4_MMAL, mpeg4_mmal); > #if FF_API_VDPAU > REGISTER_DECODER(MPEG4_VDPAU, mpeg4_vdpau); > @@ -263,8 +267,10 @@ static void register_all(void) > REGISTER_DECODER(MPEG_VDPAU, mpeg_vdpau); > REGISTER_DECODER(MPEG1_VDPAU, mpeg1_vdpau); > #endif > + REGISTER_DECODER(MPEG1_V4L2M2M, mpeg1_v4l2m2m); > REGISTER_DECODER(MPEG2_MMAL, mpeg2_mmal); > REGISTER_DECODER(MPEG2_CRYSTALHD, mpeg2_crystalhd); > + REGISTER_DECODER(MPEG2_V4L2M2M, mpeg2_v4l2m2m); > REGISTER_DECODER(MPEG2_QSV, mpeg2_qsv); > REGISTER_DECODER(MPEG2_MEDIACODEC, mpeg2_mediacodec); > REGISTER_DECODER(MSA1, msa1); > @@ -362,6 +368,7 @@ static void register_all(void) > REGISTER_DECODER(VC1IMAGE, vc1image); > REGISTER_DECODER(VC1_MMAL, vc1_mmal); > REGISTER_DECODER(VC1_QSV, vc1_qsv); > + REGISTER_DECODER(VC1_V4L2M2M, vc1_v4l2m2m); > REGISTER_ENCODER(VC2, vc2); > REGISTER_DECODER(VCR1, vcr1); > REGISTER_DECODER(VMDVIDEO, vmdvideo); > @@ -373,7 +380,9 @@ static void register_all(void) > REGISTER_DECODER(VP6F, vp6f); > REGISTER_DECODER(VP7, vp7); > REGISTER_DECODER(VP8, vp8); > + REGISTER_ENCDEC (VP8_V4L2M2M, vp8_v4l2m2m); > REGISTER_DECODER(VP9, vp9); > + REGISTER_DECODER(VP9_V4L2M2M, vp9_v4l2m2m); > REGISTER_DECODER(VQA, vqa); > REGISTER_DECODER(BITPACKED, bitpacked); > REGISTER_DECODER(WEBP, webp); > diff --git a/libavcodec/v4l2_buffers.c b/libavcodec/v4l2_buffers.c > new file mode 100644 > index 0000000..c3c0d03 > --- /dev/null > +++ b/libavcodec/v4l2_buffers.c > @@ -0,0 +1,918 @@ > +/* > + * V4L2 buffer helper functions. > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#include <linux/videodev2.h> > +#include <sys/ioctl.h> > +#include <sys/mman.h> > +#include <unistd.h> > +#include <fcntl.h> > +#include <poll.h> > +#include "libavcodec/avcodec.h" > +#include "libavcodec/internal.h" > +#include "v4l2_buffers.h" > +#include "v4l2_m2m.h" > + > +#define USEC_PER_SEC 1000000 > + > +enum V4L2Buffer_status { > + V4L2BUF_AVAILABLE, > + V4L2BUF_IN_DRIVER, > + V4L2BUF_RET_USER, > +}; > + > +/* buffer transform */ > +typedef int (*pkt_to_buf_f)(const AVPacket *, V4L2Buffer *); > +typedef int (*frm_to_buf_f)(const AVFrame *, V4L2Buffer *); > +typedef int (*buf_to_pkt_f)(AVPacket *, V4L2Buffer *); > +typedef int (*buf_to_frm_f)(AVFrame *, V4L2Buffer *); > + > +typedef int (*buf_to_bufref_f)(V4L2Buffer *in, int plane, AVBufferRef **buf); > +typedef int (*bufref_to_buf_f)(V4L2Buffer *out, int plane, const uint8_t* data, int size, AVBufferRef* bref); > + > +struct V4L2Buffer_ops { > + pkt_to_buf_f pkt_to_buf; > + frm_to_buf_f frm_to_buf; > + buf_to_pkt_f buf_to_pkt; > + buf_to_frm_f buf_to_frm; > + > + bufref_to_buf_f bufref_to_buf; > + buf_to_bufref_f buf_to_bufref; > +}; > + > +struct V4L2Buffer { > + /* each buffer needs to have a reference to its context */ > + struct V4L2Context *context; > + > + struct V4L2Plane_info { > + void * mm_addr; > + size_t lengths; > + } plane_info[VIDEO_MAX_PLANES]; > + > + /* some common buffer operations */ > + struct V4L2Buffer_ops ops; > + > + /* memcpy to the v4l2_buffer planes array when needed */ > + struct v4l2_plane planes[VIDEO_MAX_PLANES]; > + struct v4l2_buffer buf; > + > + int bytesperline[4]; Why 4? Should this be VIDEO_MAX_PLANES again? > + int num_planes; > + > + int flags; > + enum V4L2Buffer_status status; > + > +}; > ... I'll read the buffer stuff more carefully later. > diff --git a/libavcodec/v4l2_m2m.c b/libavcodec/v4l2_m2m.c > new file mode 100644 > index 0000000..809e591 > --- /dev/null > +++ b/libavcodec/v4l2_m2m.c > @@ -0,0 +1,452 @@ > +/* > + * V4L mem2mem > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#include <linux/videodev2.h> > +#include <sys/ioctl.h> > +#include <sys/mman.h> > +#include <unistd.h> > +#include <dirent.h> > +#include <fcntl.h> > +#include "libavcodec/avcodec.h" > +#include "libavcodec/internal.h" > +#include "libavutil/pixdesc.h" > +#include "libavutil/imgutils.h" > +#include "libavutil/pixfmt.h" > +#include "v4l2_m2m_avcodec.h" > +#include "v4l2_buffers.h" > +#include "v4l2_fmt.h" > +#include "v4l2_m2m.h" > + > +static inline int try_raw_format(V4L2Context* ctx, enum AVPixelFormat pixfmt) > +{ > + struct v4l2_format *fmt = &ctx->format; > + uint32_t v4l2_fmt; > + int ret; > + > + v4l2_fmt = ff_v4l2_avfmt_to_v4l2fmt(pixfmt); > + if (!v4l2_fmt) > + return AVERROR(EINVAL); > + > + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) > + fmt->fmt.pix_mp.pixelformat = v4l2_fmt; > + else > + fmt->fmt.pix.pixelformat = v4l2_fmt; > + > + fmt->type = ctx->type; > + > + ret = ioctl(ctx->fd, VIDIOC_TRY_FMT, fmt); > + if (ret) > + return AVERROR(EINVAL); > + > + return 0; > +} > + > +static int query_raw_format(V4L2Context* ctx, int set) > +{ > + enum AVPixelFormat pixfmt = ctx->av_pix_fmt; > + struct v4l2_fmtdesc fdesc; > + int ret; > + > + memset(&fdesc, 0, sizeof(fdesc)); > + fdesc.type = ctx->type; > + > + if (pixfmt != AV_PIX_FMT_NONE) { > + ret = try_raw_format(ctx, pixfmt); > + if (ret) > + pixfmt = AV_PIX_FMT_NONE; > + else > + return 0; > + } > + > + for (;;) { > + ret = ioctl(ctx->fd, VIDIOC_ENUM_FMT, &fdesc); > + if (ret) > + return AVERROR(EINVAL); > + > + pixfmt = ff_v4l2_v4l2fmt_to_avfmt(fdesc.pixelformat, AV_CODEC_ID_RAWVIDEO); > + ret = try_raw_format(ctx, pixfmt); > + if (ret){ > + fdesc.index++; > + continue; > + } > + > + if (set) > + ctx->av_pix_fmt = pixfmt; > + > + return 0; > + } > + > + return AVERROR(EINVAL); > +} > + > +static int query_coded_format(V4L2Context* ctx, uint32_t *p) > +{ > + struct v4l2_fmtdesc fdesc; > + uint32_t v4l2_fmt; > + int ret; > + > + v4l2_fmt = ff_v4l2_avcodec_to_v4l2fmt(ctx->av_codec_id); > + if (!v4l2_fmt) > + return AVERROR(EINVAL); > + > + memset(&fdesc, 0, sizeof(fdesc)); > + fdesc.type = ctx->type; > + > + for (;;) { > + ret = ioctl(ctx->fd, VIDIOC_ENUM_FMT, &fdesc); > + if (ret) > + return AVERROR(EINVAL); > + > + if (fdesc.pixelformat == v4l2_fmt) { > + break; > + } > + > + fdesc.index++; > + } > + > + *p = v4l2_fmt; > + > + return 0; > +} > + > +static inline int splane_video(struct v4l2_capability *cap) > +{ > + if (cap->capabilities & (V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT) & V4L2_CAP_STREAMING) From videodev2.h: #define V4L2_CAP_VIDEO_CAPTURE 0x00000001 /* Is a video capture device */ #define V4L2_CAP_VIDEO_OUTPUT 0x00000002 /* Is a video output device */ #define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */ So this is always zero? > + return 1; > + > + if (cap->capabilities & V4L2_CAP_VIDEO_M2M) > + return 1; > + > + return 0; > +} > + > +static inline int mplane_video(struct v4l2_capability *cap) > +{ > + if (cap->capabilities & (V4L2_CAP_VIDEO_CAPTURE_MPLANE | V4L2_CAP_VIDEO_OUTPUT_MPLANE) & V4L2_CAP_STREAMING) Similarly. > + return 1; > + > + if (cap->capabilities & V4L2_CAP_VIDEO_M2M_MPLANE) > + return 1; > + > + return 0; > +} > + > +static int prepare_contexts(V4L2m2mContext* s, void *log_ctx) > +{ > + struct v4l2_capability cap; > + int ret; > + > + s->capture.log_ctx = s->output.log_ctx = log_ctx; > + s->capture.done = s->output.done = 0; > + s->capture.fd = s->output.fd = s->fd; > + s->capture.name = "v4l2_cap"; > + s->output.name = "v4l2_out"; > + atomic_init(&s->refcount, 0); > + sem_init(&s->refsync, 0, 0); > + > + memset(&cap, 0, sizeof(cap)); > + ret = ioctl(s->fd, VIDIOC_QUERYCAP, &cap); > + if (ret < 0) > + return ret; > + > + av_log(log_ctx, AV_LOG_INFO, "driver '%s' on card '%s'\n", cap.driver, cap.card); > + > + if (mplane_video(&cap)) { > + s->capture.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; > + s->output.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; > + return 0; > + } > + > + if (splane_video(&cap)) { > + s->capture.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; > + s->output.type = V4L2_BUF_TYPE_VIDEO_OUTPUT; > + return 0; > + } > + > + return AVERROR(EINVAL); > +} > + > +static int probe_v4l2_driver(V4L2m2mContext* s, void *log_ctx) > +{ > + int ret; > + > + s->fd = open(s->devname, O_RDWR | O_NONBLOCK, 0); > + if (s->fd < 0) > + return AVERROR(errno); > + > + ret = prepare_contexts(s, log_ctx); > + if (ret < 0) > + goto done; > + > + ret = ff_v4l2_m2m_codec_format_context(&s->output, 0); > + if (ret) { > + av_log(log_ctx, AV_LOG_DEBUG, "can't set input format\n"); > + goto done; > + } > + > + ret = ff_v4l2_m2m_codec_format_context(&s->capture, 0); > + if (ret) { > + av_log(log_ctx, AV_LOG_DEBUG, "can't to set output format\n"); > + goto done; > + } > + > +done: > + close(s->fd); > + s->fd = 0; > + > + return ret; > +} > + > +static int configure_contexts(V4L2m2mContext* s, void *log_ctx) > +{ > + int ret; > + > + s->fd = open(s->devname, O_RDWR | O_NONBLOCK, 0); > + if (s->fd < 0) > + return AVERROR(errno); > + > + ret = prepare_contexts(s, log_ctx); > + if (ret < 0) > + goto error; > + > + ret = ff_v4l2_m2m_codec_format_context(&s->output, 1); > + if (ret) { > + av_log(log_ctx, AV_LOG_ERROR, "can't set input format\n"); > + goto error; > + } > + > + ret = ff_v4l2_m2m_codec_format_context(&s->capture, 1); > + if (ret) { > + av_log(log_ctx, AV_LOG_ERROR, "can't to set output format\n"); > + goto error; > + } > + > + ret = ff_v4l2_context_init(&s->output, s->output.lazy_init); > + if (ret) { > + av_log(log_ctx, AV_LOG_ERROR, "no output context's buffers\n"); > + goto error; > + } > + > + ret = ff_v4l2_context_init(&s->capture, s->capture.lazy_init); > + if (ret) { > + av_log(log_ctx, AV_LOG_ERROR, "no capture context's buffers\n"); > + goto error; > + } > + > +error: > + if (ret) { > + close(s->fd); > + s->fd = 0; > + } > + > + return 0; > +} > + > +static void save_to_context(V4L2Context* ctx, uint32_t v4l2_fmt) > +{ > + ctx->format.type = ctx->type; > + > + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { > + /* this is to handle the reconfiguration of the capture stream at runtime */ > + ctx->format.fmt.pix_mp.height = ctx->height; > + ctx->format.fmt.pix_mp.width = ctx->width; > + if (v4l2_fmt) > + ctx->format.fmt.pix_mp.pixelformat = v4l2_fmt; > + } else { > + ctx->format.fmt.pix.height = ctx->height; > + ctx->format.fmt.pix.width = ctx->width; > + if (v4l2_fmt) > + ctx->format.fmt.pix_mp.pixelformat = v4l2_fmt; > + } > +} > + > +int ff_v4l2_m2m_codec_format_context(V4L2Context* ctx, int set) > +{ > + uint32_t v4l2_fmt; > + int ret; > + > + if (ctx->av_codec_id == AV_CODEC_ID_RAWVIDEO) { > + ret = query_raw_format(ctx, set); > + if (ret) > + return ret; > + > + save_to_context(ctx, 0); > + if (set) > + return ioctl(ctx->fd, VIDIOC_S_FMT, &ctx->format); > + > + return ret; > + } > + > + ret = query_coded_format(ctx, &v4l2_fmt); > + if (ret) > + return ret; > + > + save_to_context(ctx, v4l2_fmt); > + if (set) > + return ioctl(ctx->fd, VIDIOC_S_FMT, &ctx->format); > + > + return ioctl(ctx->fd, VIDIOC_TRY_FMT, &ctx->format); > +} > + > +int ff_v4l2_m2m_codec_end(V4L2m2mContext* s) > +{ > + int ret; > + > + ret = ff_v4l2_context_set_status(&s->output, VIDIOC_STREAMOFF); > + if (ret) > + av_log(s->output.log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", s->output.name); > + > + ret = ff_v4l2_context_set_status(&s->capture, VIDIOC_STREAMOFF); > + if (ret) > + av_log(s->capture.log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", s->capture.name); > + > + ff_v4l2_context_release(&s->output); > + > + if (atomic_load(&s->refcount)) { > + av_log(s->capture.log_ctx, AV_LOG_DEBUG, "avpriv_v4l2m2m_end leaving pending buffers \n"); > + > + return 0; > + } > + > + ff_v4l2_context_release(&s->capture); > + sem_destroy(&s->refsync); > + > + /* release the hardware */ > + close(s->fd); > + > + return 0; > +} > + > +int ff_v4l2_m2m_codec_init(V4L2m2mContext* s, void* log_ctx) > +{ > + char *devname_save = s->devname; > + int ret = AVERROR(EINVAL); > + struct dirent *entry; > + char node[PATH_MAX]; > + DIR *dirp; > + > + if (s->devname && *s->devname) > + return configure_contexts(s, log_ctx); > + > + dirp = opendir("/dev"); > + if (!dirp) > + return AVERROR(errno); > + > + for (entry = readdir(dirp); entry; entry = readdir(dirp)) { > + > + if (strncmp(entry->d_name, "video", 5)) > + continue; > + > + snprintf(node, sizeof(node), "/dev/%s", entry->d_name); > + > + av_log(log_ctx, AV_LOG_DEBUG, "probing device %s\n", node); > + > + s->devname = node; > + ret = probe_v4l2_driver(s, log_ctx); > + if (!ret) > + break; > + } > + > + closedir(dirp); > + > + if (!ret) { > + av_log(log_ctx, AV_LOG_INFO, "Using device %s\n", node); > + ret = configure_contexts(s, log_ctx); > + } else { > + av_log(log_ctx, AV_LOG_ERROR, "Could not find a valid device\n"); > + } > + s->devname = devname_save; > + > + return ret; > +} > + > +int ff_v4l2_m2m_codec_reinit(V4L2m2mContext* s) > +{ > + int ret; > + > + /* 1. reinit in progress */ > + s->reinit = 1; > + > + av_log(s->avctx, AV_LOG_DEBUG, "reinit context\n"); > + > + /* 2. streamoff */ > + ret = ff_v4l2_context_set_status(&s->capture, VIDIOC_STREAMOFF); > + if (ret) > + av_log(s->avctx, AV_LOG_ERROR, "capture VIDIOC_STREAMOFF\n"); > + > + /* 3. unmap the capture buffers (v4l2 and ffmpeg): > + * we must wait for all references to be released before being allowed > + * to queue new buffers. > + */ > + av_log(s->avctx, AV_LOG_DEBUG, "capture wait for user to release AVBufferRefs \n"); > + if (atomic_load(&s->refcount)) { > + while(sem_wait(&s->refsync) == -1 && errno == EINTR); > + } > + > + ff_v4l2_context_release(&s->capture); > + > + /* 4. query the new format */ > + ret = ff_v4l2_m2m_codec_format_context(&s->capture, 1); > + if (ret) { > + av_log(s->avctx, AV_LOG_ERROR, "setting capture format\n"); > + return ret; > + } > + > + /* 5. do lazy initialization */ > + ret = ff_v4l2_context_init(&s->capture, s->capture.lazy_init); > + if (ret) { > + av_log(s->avctx, AV_LOG_ERROR, "capture buffers lazy init\n"); > + return ret; > + } > + > + /* 6. update AVCodecContext */ > + ret = ff_set_dimensions(s->avctx, s->capture.width, s->capture.height); > + if (ret < 0) > + av_log(s->avctx, AV_LOG_WARNING, "update avcodec height and width\n"); > + > + /* 7. complete reinit */ > + sem_destroy(&s->refsync); > + sem_init(&s->refsync, 0, 0); > + s->draining = 0; > + s->reinit = 0; > + > + return 0; > +} Seems like it might be nicer if you just passed the AVCodecContext to all of these functions rather than separate V4L2m2mContext and log context. (Feel free to ignore that.) > +int ff_v4l2m2m_codec_end(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + > + av_log(avctx, AV_LOG_DEBUG, "Closing context\n"); > + > + return ff_v4l2_m2m_codec_end(s); > +} > + > +int ff_v4l2m2m_codec_init(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + s->avctx = avctx; > + > + return ff_v4l2_m2m_codec_init(s, avctx); > +} There's also some funny indenting in a few places in this file. > diff --git a/libavcodec/v4l2_m2m.h b/libavcodec/v4l2_m2m.h > new file mode 100644 > index 0000000..63a4326 > --- /dev/null > +++ b/libavcodec/v4l2_m2m.h > @@ -0,0 +1,70 @@ > +/* > + * V4L2 mem2mem helper functions > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef AVCODEC_V4L2_M2M_H > +#define AVCODEC_V4L2_M2M_H > + > +#include <semaphore.h> > +#include "v4l2_buffers.h" > + > +#define container_of(ptr, type, member) ({ \ > + const __typeof__(((type *)0)->member ) *__mptr = (ptr); \ > + (type *)((char *)__mptr - offsetof(type,member) );}) > + > +#define V4L_M2M_DEFAULT_OPTS \ > + { "device",\ > + "Path to the device to use",\ > + OFFSET(devname), AV_OPT_TYPE_STRING, {.str = NULL }, 0, 0, FLAGS },\ > + { "num_output_buffers",\ > + "Number of buffers in the output context",\ > + OFFSET(output.num_buffers), AV_OPT_TYPE_INT, { .i64 = 16 }, 6, INT_MAX, FLAGS } > + > +typedef struct V4L2m2mContext > +{ > + AVClass *class; > + int fd; > + char *devname; > + > + /* the codec context queues */ > + V4L2Context capture; > + V4L2Context output; > + > + /* refcount of buffers held by the user */ > + atomic_uint refcount; > + > + /* dynamic stream reconfig */ > + AVCodecContext *avctx; > + sem_t refsync; > + int reinit; > + > + /* null frame or packet received */ > + int draining; > +} V4L2m2mContext; > + > +int ff_v4l2_m2m_codec_init(V4L2m2mContext *ctx, void* log_ctx); > +int ff_v4l2_m2m_codec_reinit(V4L2m2mContext *ctx); > +int ff_v4l2_m2m_codec_end(V4L2m2mContext *ctx); > +int ff_v4l2_m2m_codec_format_context(V4L2Context *ctx, int set); > + > + > +#endif /* AVCODEC_V4L2_M2M_H */ > diff --git a/libavcodec/v4l2_m2m_avcodec.h b/libavcodec/v4l2_m2m_avcodec.h > new file mode 100644 > index 0000000..c6ad5d4 > --- /dev/null > +++ b/libavcodec/v4l2_m2m_avcodec.h > @@ -0,0 +1,32 @@ > +/* > + * V4L2 mem2mem avcodec helper functions > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef AVCODEC_V4L2_M2M_AVCODEC_H > +#define AVCODEC_V4L2_M2M_AVCODEC_H > + > +#include "libavcodec/avcodec.h" > + > +int ff_v4l2m2m_codec_init(AVCodecContext *avctx); > +int ff_v4l2m2m_codec_end(AVCodecContext *avctx); > + > +#endif > diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c > new file mode 100644 > index 0000000..205f695 > --- /dev/null > +++ b/libavcodec/v4l2_m2m_dec.c > @@ -0,0 +1,213 @@ > +/* > + * V4L2 mem2mem decoders > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#include <linux/videodev2.h> > +#include <sys/ioctl.h> > +#include "libavutil/pixfmt.h" > +#include "libavutil/pixdesc.h" > +#include "libavutil/opt.h" > +#include "libavcodec/avcodec.h" > +#include "libavcodec/decode.h" > +#include "v4l2_m2m_avcodec.h" > +#include "v4l2_buffers.h" > +#include "v4l2_fmt.h" > +#include "v4l2_m2m.h" > + > +static int try_start(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const capture = &s->capture; > + V4L2Context *const output = &s->output; > + struct v4l2_event_subscription sub; > + struct v4l2_selection selection; > + int ret; > + > + if (output->streamon && capture->streamon) > + return 0; > + > + /* 0. subscribe to source change event */ > + memset(&sub, 0, sizeof(sub)); > + sub.type = V4L2_EVENT_SOURCE_CHANGE; > + ret = ioctl(s->fd, VIDIOC_SUBSCRIBE_EVENT, &sub); > + if ( ret < 0) > + av_log(avctx, AV_LOG_WARNING, "decoding does not support resolution change\n"); How fatal should that be? What happens if it changes anyway? > + > + /* 1. start the output process */ > + if (!output->streamon) { > + ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON); > + if (ret < 0) { > + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON on output context\n"); > + return ret; > + } > + } > + > + /* 2. get the capture format */ > + capture->format.type = capture->type; > + ret = ioctl(capture->fd, VIDIOC_G_FMT, &capture->format); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_FMT ioctl\n"); > + return ret; > + } > + > + /* 2.1 update the AVCodecContext */ > + avctx->pix_fmt = ff_v4l2_v4l2fmt_to_avfmt(capture->format.fmt.pix_mp.pixelformat, AV_CODEC_ID_RAWVIDEO); > + capture->av_pix_fmt = avctx->pix_fmt; > + > + /* 3. set the crop parameters */ > + selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; > + selection.r.height = avctx->coded_height; > + selection.r.width = avctx->coded_width;> + ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection); > + if (!ret) { > + ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_SELECTION ioctl\n"); > + } else { > + av_log(avctx, AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height); > + /* update the size of the resulting frame */ > + capture->height = selection.r.height; > + capture->width = selection.r.width; > + } > + } What is this trying to do? I'm not sure that the coded_width|height values are something you should be using here - in many cases they are larger than the actual output (what does 1920x1080 H.264, which is coded as 1920x1088, do?). Also, while they are often sensible for the stream if it comes from lavf and a normal file container, they need not be set at all here (0x0). > + > + /* 4. init the capture context now that we have the capture format */ > + if (!capture->buffers) { > + av_log(capture->log_ctx, AV_LOG_DEBUG, "%s requested (%dx%d)\n", > + capture->name, capture->format.fmt.pix_mp.width, capture->format.fmt.pix_mp.height); > + > + ret = ff_v4l2_context_init(capture, 0); > + if (ret) { > + av_log(avctx, AV_LOG_DEBUG, "can't request output buffers\n"); > + return ret; > + } > + } > + > + /* 6. start the capture process */ > + ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON); > + if (ret) { > + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON, on capture context\n"); > + return ret; > + } > + > + return 0; > +} > + > +static av_cold int v4l2m2m_decode_init(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *capture = &s->capture; > + V4L2Context *output = &s->output; > + > + output->height = capture->height = avctx->coded_height; > + output->width = capture->width =avctx->coded_width; Again, I don't think you should be using these. > + > + output->av_codec_id = avctx->codec_id; > + output->av_pix_fmt = AV_PIX_FMT_NONE; > + > + /* > + * the buffers associated to this context can not be initialized without > + * additional information available in the kernel driver, > + * so let's postpone requesting the buffers until we know more about the frames > + */ > + capture->lazy_init = 1; > + capture->av_codec_id = AV_CODEC_ID_RAWVIDEO; > + capture->av_pix_fmt = avctx->pix_fmt; > + > + return ff_v4l2m2m_codec_init(avctx); > +} > +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a > + * timeout is * required when retrieving a frame in case the driver has not received > + * enough input * to start generating output. > + * > + * once decoding starts, the timeout should not be hit. This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? (I might need to think more about how the semantics of this work.) > + */ > +static int v4l2m2m_receive_frame(AVCodecContext *avctx, AVFrame *frame) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const capture = &s->capture; > + V4L2Context *const output = &s->output; > + AVPacket avpkt = {0}; > + int timeout = 50; > + int ret; > + > + ret = ff_decode_get_packet(avctx, &avpkt); > + if (ret < 0 && ret != AVERROR_EOF) > + return ret; > + > + if (s->draining) > + goto dequeue; > + > + ret = ff_v4l2_enqueue_packet(output, &avpkt); > + if (ret < 0) > + return ret; > + > + if (avpkt.size) { > + ret = try_start(avctx); > + if (ret) > + return 0; > + } > + > +dequeue: > + return ff_v4l2_dequeue_frame(capture, frame, timeout); > +} > + > +#define OFFSET(x) offsetof(V4L2m2mContext, x) > +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_DECODING_PARAM > + > +static const AVOption options[] = { > + V4L_M2M_DEFAULT_OPTS,{ "num_capture_extra_buffers","Number of extra buffers in the capture context", > + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT,{.i64 = 6}, 6, INT_MAX, FLAGS}, > + { NULL}, > +}; > + > +#define M2MDEC(NAME, LONGNAME, CODEC, bsf_name) \ > +static const AVClass v4l2_m2m_ ## NAME ## _dec_class = {\ > + .class_name = #NAME "_v4l2_m2m_decoder",\ > + .item_name = av_default_item_name,\ > + .option = options,\ > + .version = LIBAVUTIL_VERSION_INT,\ > +};\ > +\ > +AVCodec ff_ ## NAME ## _v4l2m2m_decoder = { \ > + .name = #NAME "_v4l2m2m" ,\ > + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ > + .type = AVMEDIA_TYPE_VIDEO,\ > + .id = CODEC ,\ > + .priv_data_size = sizeof(V4L2m2mContext),\ > + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ > + .init = v4l2m2m_decode_init,\ > + .receive_frame = v4l2m2m_receive_frame,\ > + .close = ff_v4l2m2m_codec_end,\ > + .bsfs = bsf_name, \ > +}; > + > +M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); > +M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); h264_mp4toannexb is for H.264, and will refuse to do anything with H.265: "[AVBSFContext @ 0x55f4d346c620] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28)." (Was this decoder really tested as the commit message claims?) > +M2MDEC(mpeg1, "MPEG1", AV_CODEC_ID_MPEG1VIDEO, NULL); > +M2MDEC(mpeg2, "MPEG2", AV_CODEC_ID_MPEG2VIDEO, NULL); > +M2MDEC(mpeg4, "MPEG4", AV_CODEC_ID_MPEG4, NULL); > +M2MDEC(h263, "H.263", AV_CODEC_ID_H263, NULL); > +M2MDEC(vc1 , "VC1", AV_CODEC_ID_VC1, NULL); > +M2MDEC(vp8, "VP8", AV_CODEC_ID_VP8, NULL); > +M2MDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL); > + > diff --git a/libavcodec/v4l2_m2m_enc.c b/libavcodec/v4l2_m2m_enc.c > new file mode 100644 > index 0000000..47bb160 > --- /dev/null > +++ b/libavcodec/v4l2_m2m_enc.c > @@ -0,0 +1,340 @@ > +/* > + * V4L2 mem2mem encoders > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#include <linux/videodev2.h> > +#include <sys/ioctl.h> > +#include <search.h> > +#include "libavcodec/avcodec.h" > +#include "libavutil/pixdesc.h" > +#include "libavutil/pixfmt.h" > +#include "libavutil/opt.h" > +#include "v4l2_m2m_avcodec.h" > +#include "v4l2_buffers.h" > +#include "v4l2_m2m.h" > + > +#define MPEG_CID(x) V4L2_CID_MPEG_VIDEO_##x > +#define MPEG_VIDEO(x) V4L2_MPEG_VIDEO_##x > + > +static inline void v4l2_set_timeperframe(V4L2m2mContext *s, unsigned int num, unsigned int den) > +{ > + struct v4l2_streamparm parm = { 0 }; > + > + parm.parm.output.timeperframe.denominator = den; > + parm.parm.output.timeperframe.numerator = num; > + parm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; What does this do if the output isn't MPLANE? > + > + if (ioctl(s->fd, VIDIOC_S_PARM, &parm) < 0) > + av_log(s->avctx, AV_LOG_WARNING, "Failed to set timeperframe"); > +} > + > +static inline void v4l2_set_ext_ctrl(V4L2m2mContext *s, unsigned int id, signed int value, const char *name) > +{ > + struct v4l2_ext_controls ctrls = { 0 }; > + struct v4l2_ext_control ctrl = { 0 }; > + > + /* set ctrls */ > + ctrls.ctrl_class = V4L2_CTRL_CLASS_MPEG; > + ctrls.controls = &ctrl; > + ctrls.count = 1; > + > + /* set ctrl*/ > + ctrl.value = value; > + ctrl.id = id ; > + > + if (ioctl(s->fd, VIDIOC_S_EXT_CTRLS, &ctrls) < 0) > + av_log(s->avctx, AV_LOG_WARNING, "Failed to set %s\n", name); > + else > + av_log(s->avctx, AV_LOG_DEBUG, "Encoder: %s = %d\n", name, value); > +} > + > +static inline int v4l2_get_ext_ctrl(V4L2m2mContext *s, unsigned int id, signed int *value, const char *name) > +{ > + struct v4l2_ext_controls ctrls = { 0 }; > + struct v4l2_ext_control ctrl = { 0 }; > + int ret; > + > + /* set ctrls */ > + ctrls.ctrl_class = V4L2_CTRL_CLASS_MPEG; > + ctrls.controls = &ctrl; > + ctrls.count = 1; > + > + /* set ctrl*/ > + ctrl.id = id ; > + > + ret = ioctl(s->fd, VIDIOC_G_EXT_CTRLS, &ctrls); > + if (ret < 0) { > + av_log(s->avctx, AV_LOG_WARNING, "Failed to set %s\n", name); > + return ret; > + } > + > + *value = ctrl.value; > + > + return 0; > +} > + > +static int match_profile(const void *a, const void *b) > +{ > + if (*(unsigned int *)a == *(unsigned int *)b) > + return 0; > + > + return 1; > +} > + > +static inline unsigned int v4l2_h264_profile_from_ff(int p) > +{ > + struct h264_profile { > + unsigned int ffmpeg_val; > + unsigned int v4l2_val; > + } *val, profile[] = { > + { FF_PROFILE_H264_CONSTRAINED_BASELINE, MPEG_VIDEO(H264_PROFILE_CONSTRAINED_BASELINE) }, > + { FF_PROFILE_H264_HIGH_444_PREDICTIVE, MPEG_VIDEO(H264_PROFILE_HIGH_444_PREDICTIVE) }, > + { FF_PROFILE_H264_HIGH_422_INTRA, MPEG_VIDEO(H264_PROFILE_HIGH_422_INTRA) }, > + { FF_PROFILE_H264_HIGH_444_INTRA, MPEG_VIDEO(H264_PROFILE_HIGH_444_INTRA) }, > + { FF_PROFILE_H264_HIGH_10_INTRA, MPEG_VIDEO(H264_PROFILE_HIGH_10_INTRA) }, > + { FF_PROFILE_H264_HIGH_422, MPEG_VIDEO(H264_PROFILE_HIGH_422) }, > + { FF_PROFILE_H264_BASELINE, MPEG_VIDEO(H264_PROFILE_BASELINE) }, > + { FF_PROFILE_H264_EXTENDED, MPEG_VIDEO(H264_PROFILE_EXTENDED) }, > + { FF_PROFILE_H264_HIGH_10, MPEG_VIDEO(H264_PROFILE_HIGH_10) }, > + { FF_PROFILE_H264_MAIN, MPEG_VIDEO(H264_PROFILE_MAIN) }, > + { FF_PROFILE_H264_HIGH, MPEG_VIDEO(H264_PROFILE_HIGH) }, > + }; > + size_t len = sizeof(profile) / sizeof(profile[0]); FF_ARRAY_LENGTH(). > + > + val = lfind(&p, profile, &len, sizeof(profile[0]), match_profile); > + if (val) > + return val->v4l2_val; > + > + return FF_PROFILE_UNKNOWN; Returning FF_PROFILE_UNKNOWN here is bit weird because it mixes the namespaces. I know that's how you handle it below, but still... > +} > + > +static inline int v4l2_mpeg4_profile_from_ff(int p) > +{ > + struct mpeg4_profile { > + unsigned int ffmpeg_val; > + unsigned int v4l2_val; > + } *val, profile[] = { > + { FF_PROFILE_MPEG4_ADVANCED_CODING, MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_CODING_EFFICIENCY) }, > + { FF_PROFILE_MPEG4_ADVANCED_SIMPLE, MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_SIMPLE) }, > + { FF_PROFILE_MPEG4_SIMPLE_SCALABLE, MPEG_VIDEO(MPEG4_PROFILE_SIMPLE_SCALABLE) }, > + { FF_PROFILE_MPEG4_SIMPLE, MPEG_VIDEO(MPEG4_PROFILE_SIMPLE) }, > + { FF_PROFILE_MPEG4_CORE, MPEG_VIDEO(MPEG4_PROFILE_CORE) }, > + }; > + size_t len = sizeof(profile) / sizeof(profile[0]); FF_ARRAY_LENGTH(). > + > + val = lfind(&p, profile, &len, sizeof(profile[0]), match_profile); > + if (val) > + return val->v4l2_val; > + > + return FF_PROFILE_UNKNOWN; > +} > + > +static int check_b_frame_support(V4L2m2mContext *s) > +{ > + if (s->avctx->max_b_frames) > + av_log(s->avctx, AV_LOG_WARNING, "Encoder does not support b-frames yet\n"); > + > + v4l2_set_ext_ctrl(s, MPEG_CID(B_FRAMES), 0, "number of B-frames"); > + > + v4l2_get_ext_ctrl(s, MPEG_CID(B_FRAMES), &s->avctx->max_b_frames, "number of B-frames"); > + if (s->avctx->max_b_frames == 0) > + return 0; > + > + avpriv_report_missing_feature(s->avctx, "DTS/PTS calculation for V4L2 encoding"); > + > + return AVERROR_PATCHWELCOME; > +} > + > +static av_cold int v4l2m2m_encode_init(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *capture = &s->capture; > + V4L2Context *output = &s->output; > + int qmin_cid, qmax_cid, ret, val; > + int qmin, qmax; > + > + /* common settings output/capture */ > + output->height = capture->height = avctx->height; > + output->width = capture->width = avctx->width; > + > + /* output context */ > + output->av_codec_id = AV_CODEC_ID_RAWVIDEO; > + output->av_pix_fmt = avctx->pix_fmt; > + > + /* capture context */ > + capture->av_codec_id = avctx->codec_id; > + capture->av_pix_fmt = AV_PIX_FMT_NONE; > + > + ret = ff_v4l2m2m_codec_init(avctx); > + if (ret) > + return ret; > + > + ret = check_b_frame_support(s); > + if (ret) > + return ret; > + > + /* set params */ > + v4l2_set_timeperframe(s, avctx->framerate.num, avctx->framerate.den); Framerate need not be set if the input is VFR. What does timeperframe actually get used for here? > + > + /* set ext ctrls */ > + v4l2_set_ext_ctrl(s, MPEG_CID(HEADER_MODE), MPEG_VIDEO(HEADER_MODE_SEPARATE), "header mode"); > + v4l2_set_ext_ctrl(s, MPEG_CID(BITRATE) , avctx->bit_rate, "bit rate"); > + v4l2_set_ext_ctrl(s, MPEG_CID(GOP_SIZE), avctx->gop_size,"gop size"); > + > + av_log(avctx, AV_LOG_DEBUG, "Encoder Context: id (%d), profile (%d), frame rate(%d/%d), number b-frames (%d), " > + "gop size (%d), bit rate (%ld), qmin (%d), qmax (%d)\n", > + avctx->codec_id, avctx->profile, avctx->framerate.num, avctx->framerate.den, > + avctx->max_b_frames, avctx->gop_size, avctx->bit_rate, avctx->qmin, avctx->qmax); > + > + switch (avctx->codec_id) { > + case AV_CODEC_ID_H264: > + val = v4l2_h264_profile_from_ff(avctx->profile); > + if (val != FF_PROFILE_UNKNOWN) > + v4l2_set_ext_ctrl(s, MPEG_CID(H264_PROFILE), val, "h264 profile"); > + else > + av_log(avctx, AV_LOG_WARNING, "h264 profile unknown)\n"); > + qmin_cid = MPEG_CID(H264_MIN_QP); > + qmax_cid = MPEG_CID(H264_MAX_QP); > + > + qmin = 0; > + qmax = 51; > + break; > + case AV_CODEC_ID_MPEG4: > + val = v4l2_mpeg4_profile_from_ff(avctx->profile); > + if (val != FF_PROFILE_UNKNOWN) > + v4l2_set_ext_ctrl(s, MPEG_CID(MPEG4_PROFILE), val, "mpeg4 profile"); > + else > + av_log(avctx, AV_LOG_WARNING, "mpeg4 profile unknown)\n"); > + qmin_cid = MPEG_CID(MPEG4_MIN_QP); > + qmax_cid = MPEG_CID(MPEG4_MAX_QP); > + if (avctx->flags & CODEC_FLAG_QPEL) > + v4l2_set_ext_ctrl(s, MPEG_CID(MPEG4_QPEL), 1, "qpel"); > + qmax = 51; > + qmin = 0; MPEG-4 part 2 should have the same ranges as H.263, I think. > + break; > + case AV_CODEC_ID_H263: > + qmin_cid = MPEG_CID(H263_MIN_QP); > + qmax_cid = MPEG_CID(H263_MAX_QP); > + qmin = 1; > + qmax = 31; > + break; > + case AV_CODEC_ID_VP8: > + qmin_cid = MPEG_CID(VPX_MIN_QP); > + qmax_cid = MPEG_CID(VPX_MAX_QP); > + qmin = 0; > + qmax = 127; > + break; > + case AV_CODEC_ID_VP9: > + qmin_cid = MPEG_CID(VPX_MIN_QP); > + qmax_cid = MPEG_CID(VPX_MAX_QP); > + qmin = 0; > + qmax = 255; > + break; > + default: > + return 0; > + } > + > + if (qmin != avctx->qmin || qmax != avctx->qmax) > + av_log(avctx, AV_LOG_WARNING, "Encoder adjusted: qmin (%d), qmax (%d)\n", qmin, qmax); > + > + v4l2_set_ext_ctrl(s, qmin_cid, qmin, "minimum video quantizer scale"); > + v4l2_set_ext_ctrl(s, qmax_cid, qmax, "maximum video quantizer scale"); > + > + return 0; > +} > + > +static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const output = &s->output; > + > + return ff_v4l2_enqueue_frame(output, frame); > +} > + > +/* Send and receive frame happen on the same thread, hence the need for a polling timeout */ Maybe this should only block if there isn't any space to queue more input? > +static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const capture = &s->capture; > + V4L2Context *const output = &s->output; > + unsigned int timeout = 50; > + int ret; > + > + if (s->draining) > + goto dequeue; > + > + if (!output->streamon) { > + ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n"); > + return ret; > + } > + } > + > + if (!capture->streamon) { > + ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n"); > + return ret; > + } > + } > + > +dequeue: > + return ff_v4l2_dequeue_packet(capture, avpkt, timeout); > +} > + > +#define OFFSET(x) offsetof(V4L2m2mContext, x) > +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM > + > +static const AVOption options[] = { > + V4L_M2M_DEFAULT_OPTS, > + { "num_capture_buffers", "Number of buffers in the capture context", > + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT, {.i64 = 4 }, 4, INT_MAX, FLAGS }, > + { NULL }, > +}; > + > +#define M2MENC(NAME, LONGNAME, CODEC) \ > +static const AVClass v4l2_m2m_ ## NAME ## _enc_class = {\ > + .class_name = #NAME "_v4l2_m2m_encoder",\ > + .item_name = av_default_item_name,\ > + .option = options,\ > + .version = LIBAVUTIL_VERSION_INT,\ > +};\ > +\ > +AVCodec ff_ ## NAME ## _v4l2m2m_encoder = { \ > + .name = #NAME "_v4l2m2m" ,\ > + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " encoder wrapper"),\ > + .type = AVMEDIA_TYPE_VIDEO,\ > + .id = CODEC ,\ > + .priv_data_size = sizeof(V4L2m2mContext),\ > + .priv_class = &v4l2_m2m_ ## NAME ##_enc_class,\ > + .init = v4l2m2m_encode_init,\ > + .send_frame = v4l2m2m_send_frame,\ > + .receive_packet = v4l2m2m_receive_packet,\ > + .close = ff_v4l2m2m_codec_end,\ > +}; > + > +M2MENC(mpeg4,"MPEG4", AV_CODEC_ID_MPEG4); > +M2MENC(h263, "H.263", AV_CODEC_ID_H263); > +M2MENC(h264, "H.264", AV_CODEC_ID_H264); > +M2MENC(hevc, "HEVC", AV_CODEC_ID_HEVC); > +M2MENC(vp8, "VP8", AV_CODEC_ID_VP8); >
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> diff --git a/Changelog b/Changelog >> index 8309417..c6fcda3 100644 >> --- a/Changelog >> +++ b/Changelog >> @@ -40,6 +40,7 @@ version <next>: >> They must always be used by name. >> - FITS demuxer and decoder >> - FITS muxer and encoder >> +- V4L2 mem2mem HW accelerated codecs support > They aren't really accelerated - they are just hardware, there is no acceleration. > I guess I am confused about the semantics: the typical v4l2 IP blocks provide acceleration in the sense that there is hardware to offload the computation away from the CPU cores. if that is not it, what does acceleration mean in the ffmpeg domain?
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> +# check V4L2 codecs available in the API >> check_header linux/fb.h >> check_header linux/videodev.h >> check_header linux/videodev2.h >> check_code cc linux/videodev2.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete >> +check_code cc linux/videodev2.h "int i = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_VIDEO_M2M | V4L2_BUF_FLAG_LAST;" || disable v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VC1_ANNEX_G;" && enable vc1_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG1;" && enable mpeg1_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG2;" && enable mpeg2_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG4;" && enable mpeg4_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_HEVC;" && enable hevc_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H263;" && enable h263_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H264;" && enable h264_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP8;" && enable vp8_v4l2_m2m >> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP9;" && enable vp9_v4l2_m2m > I think these need to appear as named config options somewhere. It doesn't currently detect anything for me, even with e.g. --enable-v4l2-m2m --enable-encoder=h264_v4l2m2m. > I dont think that would bring much value to be honest: since the ffmpeg project doesn't contain a copy of linux/videodev2.h - something strongly rejected by the maintainers - forcing the options would just cause a build error so you would not gain anything. do you have videodev2.h in your system? (in /usr/include/linux for instance)
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> vorbis_data.o >> OBJS-$(CONFIG_VORBIS_ENCODER) += vorbisenc.o vorbis.o \ >> vorbis_data.o >> +OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o > Stray change? > yes, sorry!
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + REGISTER_ENCDEC (H264_V4L2M2M, h264_v4l2m2m); >> REGISTER_DECODER(H264_MEDIACODEC, h264_mediacodec); >> REGISTER_DECODER(H264_MMAL, h264_mmal); >> REGISTER_DECODER(H264_QSV, h264_qsv); >> @@ -220,6 +222,7 @@ static void register_all(void) >> REGISTER_ENCDEC (HAP, hap); >> REGISTER_DECODER(HEVC, hevc); >> REGISTER_DECODER(HEVC_QSV, hevc_qsv); >> + REGISTER_ENCDEC(HEVC_V4L2M2M, hevc_v4l2m2m); > Alignment. > I am sorry, what do you mean?
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + >> + /* memcpy to the v4l2_buffer planes array when needed */ >> + struct v4l2_plane planes[VIDEO_MAX_PLANES]; >> + struct v4l2_buffer buf; >> + >> + int bytesperline[4]; > Why 4? Should this be VIDEO_MAX_PLANES again? > yes
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + >> +static inline int splane_video(struct v4l2_capability *cap) >> +{ >> + if (cap->capabilities & (V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT) & V4L2_CAP_STREAMING) > From videodev2.h: > > #define V4L2_CAP_VIDEO_CAPTURE 0x00000001 /* Is a video capture device */ > #define V4L2_CAP_VIDEO_OUTPUT 0x00000002 /* Is a video output device */ > #define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */ > > So this is always zero? > um, my bad - I wrongly refactored this some versions ago. unfortunately the subsequent test seem to have caught the right setting so I didnt notice. thanks for checking!
On 03/09/17 13:20, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> +# check V4L2 codecs available in the API >>> check_header linux/fb.h >>> check_header linux/videodev.h >>> check_header linux/videodev2.h >>> check_code cc linux/videodev2.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete >>> +check_code cc linux/videodev2.h "int i = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_VIDEO_M2M | V4L2_BUF_FLAG_LAST;" || disable v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VC1_ANNEX_G;" && enable vc1_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG1;" && enable mpeg1_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG2;" && enable mpeg2_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG4;" && enable mpeg4_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_HEVC;" && enable hevc_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H263;" && enable h263_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H264;" && enable h264_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP8;" && enable vp8_v4l2_m2m >>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP9;" && enable vp9_v4l2_m2m >> I think these need to appear as named config options somewhere. It doesn't currently detect anything for me, even with e.g. --enable-v4l2-m2m --enable-encoder=h264_v4l2m2m. >> > > I dont think that would bring much value to be honest: since the ffmpeg project doesn't contain a copy of linux/videodev2.h - something strongly rejected by the maintainers - forcing the options would just cause a build error so you would not gain anything. > > do you have videodev2.h in your system? (in /usr/include/linux for instance) Yes, it detected correctly in a previous version of the patch. I think it might be something to do with the recent build system updates - maybe rebase and try again yourself. - Mark
On 03/09/17 13:25, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> + REGISTER_ENCDEC (H264_V4L2M2M, h264_v4l2m2m); >>> REGISTER_DECODER(H264_MEDIACODEC, h264_mediacodec); >>> REGISTER_DECODER(H264_MMAL, h264_mmal); >>> REGISTER_DECODER(H264_QSV, h264_qsv); >>> @@ -220,6 +222,7 @@ static void register_all(void) >>> REGISTER_ENCDEC (HAP, hap); >>> REGISTER_DECODER(HEVC, hevc); >>> REGISTER_DECODER(HEVC_QSV, hevc_qsv); >>> + REGISTER_ENCDEC(HEVC_V4L2M2M, hevc_v4l2m2m); >> Alignment. >> > > I am sorry, what do you mean? The other ( are all aligned by adding a space after ENCDEC, this one isn't.
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + av_log(s->avctx, AV_LOG_ERROR, "setting capture format\n"); >> + return ret; >> + } >> + >> + /* 5. do lazy initialization */ >> + ret = ff_v4l2_context_init(&s->capture, s->capture.lazy_init); >> + if (ret) { >> + av_log(s->avctx, AV_LOG_ERROR, "capture buffers lazy init\n"); >> + return ret; >> + } >> + >> + /* 6. update AVCodecContext */ >> + ret = ff_set_dimensions(s->avctx, s->capture.width, s->capture.height); >> + if (ret < 0) >> + av_log(s->avctx, AV_LOG_WARNING, "update avcodec height and width\n"); >> + >> + /* 7. complete reinit */ >> + sem_destroy(&s->refsync); >> + sem_init(&s->refsync, 0, 0); >> + s->draining = 0; >> + s->reinit = 0; >> + >> + return 0; >> +} > Seems like it might be nicer if you just passed the AVCodecContext to all of these functions rather than separate V4L2m2mContext and log context. (Feel free to ignore that.) sure I accept your input. even though from where I am coming from (less ffmpeg experience) I think it increases the semantic distance in the header file maybe making it a bit harder to read (now it will look as below) typedef struct V4L2m2mContext { AVClass *class; int fd; char *devname; /* the codec context queues */ V4L2Context capture; V4L2Context output; /* refcount of buffers held by the user */ atomic_uint refcount; /* dynamic stream reconfig */ AVCodecContext *avctx; sem_t refsync; int reinit; /* null frame or packet received */ int draining; } V4L2m2mContext; int ff_v4l2_m2m_codec_init(AVCodecContext *); int ff_v4l2_m2m_codec_reinit(AVCodecContext *); int ff_v4l2_m2m_codec_end(AVCodecContext *); int ff_v4l2_m2m_codec_format_context(V4L2Context *ctx, int set); while in the past all functions accepted the V4L2m2mContext only needing the AVCodecContext in the marginal case when the buffers need to be reinit while encoding/decoding due to new sizes/formats.... but sure will do. > >> +int ff_v4l2m2m_codec_end(AVCodecContext *avctx) >> +{ >> + V4L2m2mContext *s = avctx->priv_data; >> + >> + av_log(avctx, AV_LOG_DEBUG, "Closing context\n"); >> + >> + return ff_v4l2_m2m_codec_end(s); >> +} >> + >> +int ff_v4l2m2m_codec_init(AVCodecContext *avctx) >> +{ >> + V4L2m2mContext *s = avctx->priv_data; >> + s->avctx = avctx; >> + >> + return ff_v4l2_m2m_codec_init(s, avctx); >> +}
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> +int ff_v4l2m2m_codec_end(AVCodecContext *avctx) >> +{ >> + V4L2m2mContext *s = avctx->priv_data; >> + >> + av_log(avctx, AV_LOG_DEBUG, "Closing context\n"); >> + >> + return ff_v4l2_m2m_codec_end(s); >> +} >> + >> +int ff_v4l2m2m_codec_init(AVCodecContext *avctx) >> +{ >> + V4L2m2mContext *s = avctx->priv_data; >> + s->avctx = avctx; >> + >> + return ff_v4l2_m2m_codec_init(s, avctx); >> +} > There's also some funny indenting in a few places in this file. > ACK
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + /* 0. subscribe to source change event */ >> + memset(&sub, 0, sizeof(sub)); >> + sub.type = V4L2_EVENT_SOURCE_CHANGE; >> + ret = ioctl(s->fd, VIDIOC_SUBSCRIBE_EVENT, &sub); >> + if ( ret < 0) >> + av_log(avctx, AV_LOG_WARNING, "decoding does not support resolution change\n"); > How fatal should that be? What happens if it changes anyway? > the buffers are allocated and queued in the driver with a size for a resolution and format: if the resolution changes down-up the image will not fit in the previously allocated buffers - not sure if that will cause some drivers to generate bus errors....I guess it will depend on each driver; also the sizes reported back with the frame after decoding to ffmpeg will not have been fixed properly since we couldnt receive the event so I doubt the images can be recomposed. Basically the v4l2 kernel driver will have no way to report changes back to ffmpeg...
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + } >> + >> + /* 2.1 update the AVCodecContext */ >> + avctx->pix_fmt = ff_v4l2_v4l2fmt_to_avfmt(capture->format.fmt.pix_mp.pixelformat, AV_CODEC_ID_RAWVIDEO); >> + capture->av_pix_fmt = avctx->pix_fmt; >> + >> + /* 3. set the crop parameters */ >> + selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; >> + selection.r.height = avctx->coded_height; >> + selection.r.width = avctx->coded_width;> + ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection); >> + if (!ret) { >> + ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection); >> + if (ret) { >> + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_SELECTION ioctl\n"); >> + } else { >> + av_log(avctx, AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height); >> + /* update the size of the resulting frame */ >> + capture->height = selection.r.height; >> + capture->width = selection.r.width; >> + } >> + } > What is this trying to do? I'm not sure that the coded_width|height values are something you should be using here - in many cases they are larger than the actual output (what does 1920x1080 H.264, which is coded as 1920x1088, do?). Also, while they are often sensible for the stream if it comes from lavf and a normal file container, they need not be set at all here (0x0). > I am just trying to crop the decoded frame size to what I understood are the expected dimensions; if you look at buffer_ops_v4l2buf_to_avframe (used to return the dequeued buffer to the frame) we use these variables to return the frame dimensions back to ffmpeg. When ffplaying 1920x1080 this is what xwininfo (window information utility for X) returns: [jramirez@igloo ~]$ xwininfo xwininfo: Please select the window about which you would like information by clicking the mouse in that window. xwininfo: Window id: 0x560000c "Big Buck Bunny, Sunflower version - /home/linaro/Videos/avc1.bbb_sunflower_1080p_30fps_normal.mp4" Absolute upper-left X: 2240 Absolute upper-left Y: 204 Relative upper-left X: 0 Relative upper-left Y: 0 Width: 1920 Height: 1088 Depth: 24 Visual: 0xeb Visual Class: DirectColor Border width: 0 Class: InputOutput Colormap: 0x560000b (not installed) Bit Gravity State: ForgetGravity Window Gravity State: NorthWestGravity Backing Store State: NotUseful Save Under State: no Map State: IsViewable Override Redirect State: no Corners: +2240+204 -2240+204 -2240-148 +2240-148 -geometry 1920x1088+2230+166 maybe it works out nicely because the frames returned by the decoder are also 32 byte aligned? this is what is encoded in the video [jramirez@igloo Videos]$ ffprobe -show_frames avc1.bbb_sunflower_1080p_30fps_normal.mp4 | more 1 [FRAME] 2 media_type=video 3 stream_index=0 4 key_frame=1 5 pkt_pts=2000 6 pkt_pts_time=0.066667 7 pkt_dts=2000 8 pkt_dts_time=0.066667 9 best_effort_timestamp=2000 10 best_effort_timestamp_time=0.066667 11 pkt_duration=1000 12 pkt_duration_time=0.033333 13 pkt_pos=236997 14 pkt_size=1116 15 width=1920 16 height=1080 17 pix_fmt=yuv420p 18 sample_aspect_ratio=1:1 19 pict_type=I 20 coded_picture_number=0 21 display_picture_number=0 22 interlaced_frame=0 23 top_field_first=0 24 repeat_pict=0 25 [/FRAME] so if this is not correct, could you suggest or hint for a proper way please?
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ >> + .type = AVMEDIA_TYPE_VIDEO,\ >> + .id = CODEC ,\ >> + .priv_data_size = sizeof(V4L2m2mContext),\ >> + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ >> + .init = v4l2m2m_decode_init,\ >> + .receive_frame = v4l2m2m_receive_frame,\ >> + .close = ff_v4l2m2m_codec_end,\ >> + .bsfs = bsf_name, \ >> +}; >> + >> +M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); >> +M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); > h264_mp4toannexb is for H.264, and will refuse to do anything with H.265: > > "[AVBSFContext @ 0x55f4d346c620] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28)." > > (Was this decoder really tested as the commit message claims?) > yes it was a couple of versions ago though: without it, the decoded frames could not be displayed using the dragon board 820c. however just tried again and got the error message that you mentioned; [...] [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) yeah now, after removing the h264_mp4toannexb from the decoder, I can not play any HEVC videos on the db820c.....I'll then remove HEVC from the list of supported decoders then. It is a pity.
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + >> + val = lfind(&p, profile, &len, sizeof(profile[0]), match_profile); >> + if (val) >> + return val->v4l2_val; >> + >> + return FF_PROFILE_UNKNOWN; > Returning FF_PROFILE_UNKNOWN here is bit weird because it mixes the namespaces. I know that's how you handle it below, but still... > shall I return a instead AVERROR(ENOENT)?
On 09/03/2017 04:56 PM, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ >>> + .type = AVMEDIA_TYPE_VIDEO,\ >>> + .id = CODEC ,\ >>> + .priv_data_size = sizeof(V4L2m2mContext),\ >>> + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ >>> + .init = v4l2m2m_decode_init,\ >>> + .receive_frame = v4l2m2m_receive_frame,\ >>> + .close = ff_v4l2m2m_codec_end,\ >>> + .bsfs = bsf_name, \ >>> +}; >>> + >>> +M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); >>> +M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); >> h264_mp4toannexb is for H.264, and will refuse to do anything with H.265: >> >> "[AVBSFContext @ 0x55f4d346c620] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28)." >> >> (Was this decoder really tested as the commit message claims?) >> > > yes it was a couple of versions ago though: without it, the decoded > frames could not be displayed using the dragon board 820c. > however just tried again and got the error message that you mentioned; > [...] > [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by > the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) > [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by > the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) > [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by > the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) > [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by > the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) > [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by > the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) > > yeah now, after removing the h264_mp4toannexb from the decoder, I can > not play any HEVC videos on the db820c.....I'll then remove HEVC from > the list of supported decoders then. It is a pity. > > > actually I will just remove the filter from the hevc decoder instantiation in case some v4l2 codec (different from the one in the db820) can still benefit
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> # Enable hwaccels by default. >> -enable_weak d3d11va dxva2 vaapi vda vdpau videotoolbox_hwaccel xvmc >> +enable_weak d3d11va dxva2 vaapi v4l2_m2m vda vdpau videotoolbox_hwaccel xvmc > [This line has disappeared on rebase.] > ok, just rebased today and saw that happening (didnt happen yesterday) shall I add v4l2_m2m to the HWACCEL_AUTODETECT_LIBRARYLIST or should I add a enable_weak line just for v4l2_m2m?
On 09/03/2017 02:53 PM, Mark Thompson wrote: > On 03/09/17 13:20, Jorge Ramirez wrote: >> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>> +# check V4L2 codecs available in the API >>>> check_header linux/fb.h >>>> check_header linux/videodev.h >>>> check_header linux/videodev2.h >>>> check_code cc linux/videodev2.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete >>>> +check_code cc linux/videodev2.h "int i = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_VIDEO_M2M | V4L2_BUF_FLAG_LAST;" || disable v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VC1_ANNEX_G;" && enable vc1_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG1;" && enable mpeg1_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG2;" && enable mpeg2_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG4;" && enable mpeg4_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_HEVC;" && enable hevc_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H263;" && enable h263_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H264;" && enable h264_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP8;" && enable vp8_v4l2_m2m >>>> +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP9;" && enable vp9_v4l2_m2m >>> I think these need to appear as named config options somewhere. It doesn't currently detect anything for me, even with e.g. --enable-v4l2-m2m --enable-encoder=h264_v4l2m2m. >>> >> I dont think that would bring much value to be honest: since the ffmpeg project doesn't contain a copy of linux/videodev2.h - something strongly rejected by the maintainers - forcing the options would just cause a build error so you would not gain anything. >> >> do you have videodev2.h in your system? (in /usr/include/linux for instance) > Yes, it detected correctly in a previous version of the patch. I think it might be something to do with the recent build system updates - maybe rebase and try again yourself. yes after rebasing today a configure line that enabled the setting by default was removed. on a previous post is the culprit: [...] On 09/03/2017 02:27 AM, Mark Thompson wrote: >> # Enable hwaccels by default. >> -enable_weak d3d11va dxva2 vaapi vda vdpau videotoolbox_hwaccel xvmc >> +enable_weak d3d11va dxva2 vaapi v4l2_m2m vda vdpau videotoolbox_hwaccel xvmc > [This line has disappeared on rebase.] > ok, just rebased today and saw that happening (didnt happen yesterday) shall I add v4l2_m2m to the HWACCEL_AUTODETECT_LIBRARYLIST or should I add a enable_weak line just for v4l2_m2m? [...]
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >> + * timeout is * required when retrieving a frame in case the driver has not received >> + * enough input * to start generating output. >> + * >> + * once decoding starts, the timeout should not be hit. > This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? > > (I might need to think more about how the semantics of this work.) > if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output?
On 09/03/2017 02:27 AM, Mark Thompson wrote: >> + >> + return 0; >> +} >> + >> +static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) >> +{ >> + V4L2m2mContext *s = avctx->priv_data; >> + V4L2Context *const output = &s->output; >> + >> + return ff_v4l2_enqueue_frame(output, frame); >> +} >> + >> +/* Send and receive frame happen on the same thread, hence the need for a polling timeout */ > Maybe this should only block if there isn't any space to queue more input? > as it happens on the decoding side, the issue is the same. before being able to receive frames from the capture queue (encoded/decoded) the user needs to push a number of buffers to the output queue. this number varies from IP to IP...I guess also depend on the formats. so yeah, if we knew before hand this number then it would be easy to make sure we have queued enough input buffers to get the pipe going and then just block for data safely. you proposed that this number is calibrated (ie, track the timeout); however it makes me feel unsafe not having a fallback option (ie, if the pipe blocks, then we are stuck).... having said that, I will post the calibrated solution if you'd rather have that....we can easily go back to a timeout if needed.
On 03/09/17 14:10, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> + av_log(s->avctx, AV_LOG_ERROR, "setting capture format\n"); >>> + return ret; >>> + } >>> + >>> + /* 5. do lazy initialization */ >>> + ret = ff_v4l2_context_init(&s->capture, s->capture.lazy_init); >>> + if (ret) { >>> + av_log(s->avctx, AV_LOG_ERROR, "capture buffers lazy init\n"); >>> + return ret; >>> + } >>> + >>> + /* 6. update AVCodecContext */ >>> + ret = ff_set_dimensions(s->avctx, s->capture.width, s->capture.height); >>> + if (ret < 0) >>> + av_log(s->avctx, AV_LOG_WARNING, "update avcodec height and width\n"); >>> + >>> + /* 7. complete reinit */ >>> + sem_destroy(&s->refsync); >>> + sem_init(&s->refsync, 0, 0); >>> + s->draining = 0; >>> + s->reinit = 0; >>> + >>> + return 0; >>> +} >> Seems like it might be nicer if you just passed the AVCodecContext to all of these functions rather than separate V4L2m2mContext and log context. (Feel free to ignore that.) > > sure I accept your input. > even though from where I am coming from (less ffmpeg experience) I think it increases the semantic distance in the header file maybe making it a bit harder to read (now it will look as below) > > > typedef struct V4L2m2mContext > { > AVClass *class; > int fd; > char *devname; > > /* the codec context queues */ > V4L2Context capture; > V4L2Context output; > > /* refcount of buffers held by the user */ > atomic_uint refcount; > > /* dynamic stream reconfig */ > AVCodecContext *avctx; > sem_t refsync; > int reinit; > > /* null frame or packet received */ > int draining; > } V4L2m2mContext; > > int ff_v4l2_m2m_codec_init(AVCodecContext *); > int ff_v4l2_m2m_codec_reinit(AVCodecContext *); > int ff_v4l2_m2m_codec_end(AVCodecContext *); > int ff_v4l2_m2m_codec_format_context(V4L2Context *ctx, int set); > > while in the past all functions accepted the V4L2m2mContext only needing the AVCodecContext in the marginal case when the buffers need to be reinit while encoding/decoding due to new sizes/formats.... > > but sure will do. It isn't just those functions - it's all of the ones of the form "foo(V4L2m2mContext* s, void* log_ctx)" as well. Still, do ignore it if you prefer your way. > > >> >>> +int ff_v4l2m2m_codec_end(AVCodecContext *avctx) >>> +{ >>> + V4L2m2mContext *s = avctx->priv_data; >>> + >>> + av_log(avctx, AV_LOG_DEBUG, "Closing context\n"); >>> + >>> + return ff_v4l2_m2m_codec_end(s); >>> +} >>> + >>> +int ff_v4l2m2m_codec_init(AVCodecContext *avctx) >>> +{ >>> + V4L2m2mContext *s = avctx->priv_data; >>> + s->avctx = avctx; >>> + >>> + return ff_v4l2_m2m_codec_init(s, avctx); >>> +} > >
On 03/09/17 14:21, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> + /* 0. subscribe to source change event */ >>> + memset(&sub, 0, sizeof(sub)); >>> + sub.type = V4L2_EVENT_SOURCE_CHANGE; >>> + ret = ioctl(s->fd, VIDIOC_SUBSCRIBE_EVENT, &sub); >>> + if ( ret < 0) >>> + av_log(avctx, AV_LOG_WARNING, "decoding does not support resolution change\n"); >> How fatal should that be? What happens if it changes anyway? >> > the buffers are allocated and queued in the driver with a size for a resolution and format: if the resolution changes down-up the image will not fit in the previously allocated buffers - not sure if that will cause some drivers to generate bus errors....I guess it will depend on each driver; also the sizes reported back with the frame after decoding to ffmpeg will not have been fixed properly since we couldnt receive the event so I doubt the images can be recomposed. > > Basically the v4l2 kernel driver will have no way to report changes back to ffmpeg... Well, if this is likely to be entirely broken then maybe it should be disabled if the event isn't available?
On 03/09/17 15:36, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> + } >>> + >>> + /* 2.1 update the AVCodecContext */ >>> + avctx->pix_fmt = ff_v4l2_v4l2fmt_to_avfmt(capture->format.fmt.pix_mp.pixelformat, AV_CODEC_ID_RAWVIDEO); >>> + capture->av_pix_fmt = avctx->pix_fmt; >>> + >>> + /* 3. set the crop parameters */ >>> + selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; >>> + selection.r.height = avctx->coded_height; >>> + selection.r.width = avctx->coded_width;> + ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection); >>> + if (!ret) { >>> + ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection); >>> + if (ret) { >>> + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_SELECTION ioctl\n"); >>> + } else { >>> + av_log(avctx, AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height); >>> + /* update the size of the resulting frame */ >>> + capture->height = selection.r.height; >>> + capture->width = selection.r.width; >>> + } >>> + } >> What is this trying to do? I'm not sure that the coded_width|height values are something you should be using here - in many cases they are larger than the actual output (what does 1920x1080 H.264, which is coded as 1920x1088, do?). Also, while they are often sensible for the stream if it comes from lavf and a normal file container, they need not be set at all here (0x0). >> > > I am just trying to crop the decoded frame size to what I understood are the expected dimensions; if you look at buffer_ops_v4l2buf_to_avframe (used to return the dequeued buffer to the frame) we use these variables to return the frame dimensions back to ffmpeg. > > When ffplaying 1920x1080 this is what xwininfo (window information utility for X) returns: > > [jramirez@igloo ~]$ xwininfo > > xwininfo: Please select the window about which you > would like information by clicking the > mouse in that window. > > xwininfo: Window id: 0x560000c "Big Buck Bunny, Sunflower version - /home/linaro/Videos/avc1.bbb_sunflower_1080p_30fps_normal.mp4" > > Absolute upper-left X: 2240 > Absolute upper-left Y: 204 > Relative upper-left X: 0 > Relative upper-left Y: 0 > Width: 1920 > Height: 1088 > Depth: 24 > Visual: 0xeb > Visual Class: DirectColor > Border width: 0 > Class: InputOutput > Colormap: 0x560000b (not installed) > Bit Gravity State: ForgetGravity > Window Gravity State: NorthWestGravity > Backing Store State: NotUseful > Save Under State: no > Map State: IsViewable > Override Redirect State: no > Corners: +2240+204 -2240+204 -2240-148 +2240-148 > -geometry 1920x1088+2230+166 > > > maybe it works out nicely because the frames returned by the decoder are also 32 byte aligned? > this is what is encoded in the video > > [jramirez@igloo Videos]$ ffprobe -show_frames avc1.bbb_sunflower_1080p_30fps_normal.mp4 | more > > 1 [FRAME] > 2 media_type=video > 3 stream_index=0 > 4 key_frame=1 > 5 pkt_pts=2000 > 6 pkt_pts_time=0.066667 > 7 pkt_dts=2000 > 8 pkt_dts_time=0.066667 > 9 best_effort_timestamp=2000 > 10 best_effort_timestamp_time=0.066667 > 11 pkt_duration=1000 > 12 pkt_duration_time=0.033333 > 13 pkt_pos=236997 > 14 pkt_size=1116 > 15 width=1920 > 16 height=1080 > 17 pix_fmt=yuv420p > 18 sample_aspect_ratio=1:1 > 19 pict_type=I > 20 coded_picture_number=0 > 21 display_picture_number=0 > 22 interlaced_frame=0 > 23 top_field_first=0 > 24 repeat_pict=0 > 25 [/FRAME] > > > so if this is not correct, could you suggest or hint for a proper way please? I think I've failed to emphasise the important point here: you can't use the coded_width/coded_height fields in AVCodecContext to do something like this because they need not be set. It is completely valid for the user to not set them when using libavcodec, such that the decoder receives coded_width = coded_height = 0 on input. It looks like that will then do something nasty in your code, when it tries to crop the output to 0x0. If you really need the width and height as found in the bitstream here then you will need to find it yourself. (For a different case which does something very similar to this, see the use of parsers qsvdec.) (There is also the secondary point that coded_width/coded_height do not actually contain the output width/height because of stream cropping, but that is probably less important and easier to fix.)
On 03/09/17 16:26, Jorge Ramirez wrote: > On 09/03/2017 04:56 PM, Jorge Ramirez wrote: >> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>> + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ >>>> + .type = AVMEDIA_TYPE_VIDEO,\ >>>> + .id = CODEC ,\ >>>> + .priv_data_size = sizeof(V4L2m2mContext),\ >>>> + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ >>>> + .init = v4l2m2m_decode_init,\ >>>> + .receive_frame = v4l2m2m_receive_frame,\ >>>> + .close = ff_v4l2m2m_codec_end,\ >>>> + .bsfs = bsf_name, \ >>>> +}; >>>> + >>>> +M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); >>>> +M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); >>> h264_mp4toannexb is for H.264, and will refuse to do anything with H.265: >>> >>> "[AVBSFContext @ 0x55f4d346c620] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28)." >>> >>> (Was this decoder really tested as the commit message claims?) >>> >> >> yes it was a couple of versions ago though: without it, the decoded frames could not be displayed using the dragon board 820c. >> however just tried again and got the error message that you mentioned; >> [...] >> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >> >> yeah now, after removing the h264_mp4toannexb from the decoder, I can not play any HEVC videos on the db820c.....I'll then remove HEVC from the list of supported decoders then. It is a pity. >> >> >> > > actually I will just remove the filter from the hevc decoder instantiation in case some v4l2 codec (different from the one in the db820) can still benefit > What do you actually need here? An H.265 annex B bitstream, such as created by hevc_mp4toannexb?
On 03/09/17 17:07, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> # Enable hwaccels by default. >>> -enable_weak d3d11va dxva2 vaapi vda vdpau videotoolbox_hwaccel xvmc >>> +enable_weak d3d11va dxva2 vaapi v4l2_m2m vda vdpau videotoolbox_hwaccel xvmc >> [This line has disappeared on rebase.] >> > ok, just rebased today and saw that happening (didnt happen yesterday) > > shall I add v4l2_m2m to the HWACCEL_AUTODETECT_LIBRARYLIST or should I add a enable_weak line just for v4l2_m2m? The list seems more consistent with everything else. Whatever you do, make sure that it gets autodetected if you configure with no options, and that it does not appear if you configure with either --disable-everything or --disable-autodetect.
On 09/03/2017 08:13 PM, Mark Thompson wrote: > On 03/09/17 16:26, Jorge Ramirez wrote: >> On 09/03/2017 04:56 PM, Jorge Ramirez wrote: >>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>> + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ >>>>> + .type = AVMEDIA_TYPE_VIDEO,\ >>>>> + .id = CODEC ,\ >>>>> + .priv_data_size = sizeof(V4L2m2mContext),\ >>>>> + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ >>>>> + .init = v4l2m2m_decode_init,\ >>>>> + .receive_frame = v4l2m2m_receive_frame,\ >>>>> + .close = ff_v4l2m2m_codec_end,\ >>>>> + .bsfs = bsf_name, \ >>>>> +}; >>>>> + >>>>> +M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); >>>>> +M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); >>>> h264_mp4toannexb is for H.264, and will refuse to do anything with H.265: >>>> >>>> "[AVBSFContext @ 0x55f4d346c620] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28)." >>>> >>>> (Was this decoder really tested as the commit message claims?) >>>> >>> yes it was a couple of versions ago though: without it, the decoded frames could not be displayed using the dragon board 820c. >>> however just tried again and got the error message that you mentioned; >>> [...] >>> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >>> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >>> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >>> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >>> [AVBSFContext @ 0xffff74000b90] Codec 'hevc' (174) is not supported by the bitstream filter 'h264_mp4toannexb'. Supported codecs are: h264 (28) >>> >>> yeah now, after removing the h264_mp4toannexb from the decoder, I can not play any HEVC videos on the db820c.....I'll then remove HEVC from the list of supported decoders then. It is a pity. >>> >>> >>> >> actually I will just remove the filter from the hevc decoder instantiation in case some v4l2 codec (different from the one in the db820) can still benefit >> > What do you actually need here? An H.265 annex B bitstream, such as created by hevc_mp4toannexb? um, yes, I think this was a regression then (I originally had it on v3). my fault. https://github.com/ldts/FFmpeg/commit/4cbba823a6ab92609b59cb0e1b1ded470403cd09#diff-143a78d3b451c891d27ad5a4ce321515R224 > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 03/09/17 17:54, Jorge Ramirez wrote: > On 09/03/2017 02:27 AM, Mark Thompson wrote: >>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>> + * timeout is * required when retrieving a frame in case the driver has not received >>> + * enough input * to start generating output. >>> + * >>> + * once decoding starts, the timeout should not be hit. >> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >> >> (I might need to think more about how the semantics of this work.) >> > > if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) > > the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases.
On 09/03/2017 08:23 PM, Mark Thompson wrote: > On 03/09/17 17:54, Jorge Ramirez wrote: >> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>> + * enough input * to start generating output. >>>> + * >>>> + * once decoding starts, the timeout should not be hit. >>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>> >>> (I might need to think more about how the semantics of this work.) >>> >> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >> >> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? > Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. sure, can do that as well, not a problem. the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) is this intentional or is it a bug? > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 09/03/2017 08:06 PM, Mark Thompson wrote: > On 03/09/17 15:36, Jorge Ramirez wrote: >> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>> + } >>>> + >>>> + /* 2.1 update the AVCodecContext */ >>>> + avctx->pix_fmt = ff_v4l2_v4l2fmt_to_avfmt(capture->format.fmt.pix_mp.pixelformat, AV_CODEC_ID_RAWVIDEO); >>>> + capture->av_pix_fmt = avctx->pix_fmt; >>>> + >>>> + /* 3. set the crop parameters */ >>>> + selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; >>>> + selection.r.height = avctx->coded_height; >>>> + selection.r.width = avctx->coded_width;> + ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection); >>>> + if (!ret) { >>>> + ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection); >>>> + if (ret) { >>>> + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_SELECTION ioctl\n"); >>>> + } else { >>>> + av_log(avctx, AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height); >>>> + /* update the size of the resulting frame */ >>>> + capture->height = selection.r.height; >>>> + capture->width = selection.r.width; >>>> + } >>>> + } >>> What is this trying to do? I'm not sure that the coded_width|height values are something you should be using here - in many cases they are larger than the actual output (what does 1920x1080 H.264, which is coded as 1920x1088, do?). Also, while they are often sensible for the stream if it comes from lavf and a normal file container, they need not be set at all here (0x0). >>> >> I am just trying to crop the decoded frame size to what I understood are the expected dimensions; if you look at buffer_ops_v4l2buf_to_avframe (used to return the dequeued buffer to the frame) we use these variables to return the frame dimensions back to ffmpeg. >> >> When ffplaying 1920x1080 this is what xwininfo (window information utility for X) returns: >> >> [jramirez@igloo ~]$ xwininfo >> >> xwininfo: Please select the window about which you >> would like information by clicking the >> mouse in that window. >> >> xwininfo: Window id: 0x560000c "Big Buck Bunny, Sunflower version - /home/linaro/Videos/avc1.bbb_sunflower_1080p_30fps_normal.mp4" >> >> Absolute upper-left X: 2240 >> Absolute upper-left Y: 204 >> Relative upper-left X: 0 >> Relative upper-left Y: 0 >> Width: 1920 >> Height: 1088 >> Depth: 24 >> Visual: 0xeb >> Visual Class: DirectColor >> Border width: 0 >> Class: InputOutput >> Colormap: 0x560000b (not installed) >> Bit Gravity State: ForgetGravity >> Window Gravity State: NorthWestGravity >> Backing Store State: NotUseful >> Save Under State: no >> Map State: IsViewable >> Override Redirect State: no >> Corners: +2240+204 -2240+204 -2240-148 +2240-148 >> -geometry 1920x1088+2230+166 >> >> >> maybe it works out nicely because the frames returned by the decoder are also 32 byte aligned? >> this is what is encoded in the video >> >> [jramirez@igloo Videos]$ ffprobe -show_frames avc1.bbb_sunflower_1080p_30fps_normal.mp4 | more >> >> 1 [FRAME] >> 2 media_type=video >> 3 stream_index=0 >> 4 key_frame=1 >> 5 pkt_pts=2000 >> 6 pkt_pts_time=0.066667 >> 7 pkt_dts=2000 >> 8 pkt_dts_time=0.066667 >> 9 best_effort_timestamp=2000 >> 10 best_effort_timestamp_time=0.066667 >> 11 pkt_duration=1000 >> 12 pkt_duration_time=0.033333 >> 13 pkt_pos=236997 >> 14 pkt_size=1116 >> 15 width=1920 >> 16 height=1080 >> 17 pix_fmt=yuv420p >> 18 sample_aspect_ratio=1:1 >> 19 pict_type=I >> 20 coded_picture_number=0 >> 21 display_picture_number=0 >> 22 interlaced_frame=0 >> 23 top_field_first=0 >> 24 repeat_pict=0 >> 25 [/FRAME] >> >> >> so if this is not correct, could you suggest or hint for a proper way please? > I think I've failed to emphasise the important point here: you can't use the coded_width/coded_height fields in AVCodecContext to do something like this because they need not be set. It is completely valid for the user to not set them when using libavcodec, such that the decoder receives coded_width = coded_height = 0 on input. It looks like that will then do something nasty in your code, when it tries to crop the output to 0x0. yep. > > If you really need the width and height as found in the bitstream here then you will need to find it yourself. (For a different case which does something very similar to this, see the use of parsers qsvdec.) that is ok, I can let the driver detect the discrepancy and raise an event with the proper dimensions. this will be slightly different to reinit since I will have to reconfigure the output queue as well but not a major issue I think....maybe another 10 LOC or so > > (There is also the secondary point that coded_width/coded_height do not actually contain the output width/height because of stream cropping, but that is probably less important and easier to fix.) ok > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 04/09/17 17:00, Jorge Ramirez wrote: > On 09/03/2017 08:23 PM, Mark Thompson wrote: >> On 03/09/17 17:54, Jorge Ramirez wrote: >>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>> + * enough input * to start generating output. >>>>> + * >>>>> + * once decoding starts, the timeout should not be hit. >>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>> >>>> (I might need to think more about how the semantics of this work.) >>>> >>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>> >>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. > > sure, can do that as well, not a problem. > > the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) > > is this intentional or is it a bug? The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. - Mark
On 09/04/2017 06:31 PM, Mark Thompson wrote: > On 04/09/17 17:00, Jorge Ramirez wrote: >> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>> + * enough input * to start generating output. >>>>>> + * >>>>>> + * once decoding starts, the timeout should not be hit. >>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>> >>>>> (I might need to think more about how the semantics of this work.) >>>>> >>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>> >>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >> sure, can do that as well, not a problem. >> >> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >> >> is this intentional or is it a bug? > The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). > > If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. thanks for the prompt answer. yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real) anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN. ret = avcodec_send_frame(enc, in_picture); if (ret < 0) goto error; while (1) { ret = avcodec_receive_packet(enc, &pkt); update_benchmark("encode_video %d.%d", ost->file_index, ost->index); if (ret == AVERROR(EAGAIN)) break; if (ret < 0) goto error; if (debug_ts) { av_log(NULL, AV_LOG_INFO, "encoder -> type:video " "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base), av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base)); } if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY)) pkt.pts = ost->sync_opts; av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase); if (debug_ts) { av_log(NULL, AV_LOG_INFO, "encoder -> type:video " "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase), av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase)); } frame_size = pkt.size; output_packet(of, &pkt, ost, 0); /* if two pass, output log */ if (ost->logfile && enc->stats_out) { fprintf(ost->logfile, "%s", enc->stats_out); } } } so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... does the above look ok to you? > > - Mark > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 04/09/17 18:01, Jorge Ramirez wrote: > On 09/04/2017 06:31 PM, Mark Thompson wrote: >> On 04/09/17 17:00, Jorge Ramirez wrote: >>> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>>> + * enough input * to start generating output. >>>>>>> + * >>>>>>> + * once decoding starts, the timeout should not be hit. >>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>>> >>>>>> (I might need to think more about how the semantics of this work.) >>>>>> >>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>>> >>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >>> sure, can do that as well, not a problem. >>> >>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >>> >>> is this intentional or is it a bug? >> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). >> >> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. > > thanks for the prompt answer. > > yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. > But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) > > seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real) > > anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN. > > > ret = avcodec_send_frame(enc, in_picture); > if (ret < 0) > goto error; > > while (1) { > ret = avcodec_receive_packet(enc, &pkt); > update_benchmark("encode_video %d.%d", ost->file_index, ost->index); > if (ret == AVERROR(EAGAIN)) > break; > if (ret < 0) > goto error; > > if (debug_ts) { > av_log(NULL, AV_LOG_INFO, "encoder -> type:video " > "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", > av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base), > av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base)); > } > > if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY)) > pkt.pts = ost->sync_opts; > > av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase); > > if (debug_ts) { > av_log(NULL, AV_LOG_INFO, "encoder -> type:video " > "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", > av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase), > av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase)); > } > > frame_size = pkt.size; > output_packet(of, &pkt, ost, 0); > > /* if two pass, output log */ > if (ost->logfile && enc->stats_out) { > fprintf(ost->logfile, "%s", enc->stats_out); > } > } > } > > > so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... > > > > does the above look ok to you? Yes: send_frame() is always callable once, and then receive_packet() is callable repeatedly until it returns EAGAIN; once it does, send_frame() is necessarily callable again. Can you offer a sequence of valid returns from the lavc API which would break it? (Ignoring what the implementation behind it actually is for a moment.) - Mark
On 04/09/17 18:01, Jorge Ramirez wrote: > On 09/04/2017 06:31 PM, Mark Thompson wrote: >> On 04/09/17 17:00, Jorge Ramirez wrote: >>> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>>> + * enough input * to start generating output. >>>>>>> + * >>>>>>> + * once decoding starts, the timeout should not be hit. >>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>>> >>>>>> (I might need to think more about how the semantics of this work.) >>>>>> >>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>>> >>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >>> sure, can do that as well, not a problem. >>> >>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >>> >>> is this intentional or is it a bug? >> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). >> >> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. > > thanks for the prompt answer. > > yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. > But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) > > ... > > so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... I've thought about this a bit further and I just don't understand the case you are hitting here. receive_packet() will only need to block if both the the packet queue is full (all buffers in the kernel) and the frame queue is empty (also all buffers in the kernel), right? If the packet queue is not full then we can return EAGAIN to indicate that send_packet() should be called to add more input, and if the frame queue is not empty then we can immediately return a frame. If we do get blocked in that state (polling for read on both queues), then one must become readable eventually. If the packet queue becomes readable then we dequeue a free buffer and return EAGAIN, if the frame queue becomes readable then we dequeue a frame and return it. If neither happens then something is very wrong - either we don't have enough input buffers to make a frame, or the decoder itself is somehow broken. (Ignoring end-of-stream behaviour, but I don't think that's the case you are talking about.) So, can you describe in more detail the problem case? Thanks, - Mark
On 09/04/2017 07:55 PM, Mark Thompson wrote: > On 04/09/17 18:01, Jorge Ramirez wrote: >> On 09/04/2017 06:31 PM, Mark Thompson wrote: >>> On 04/09/17 17:00, Jorge Ramirez wrote: >>>> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>>>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>>>> + * enough input * to start generating output. >>>>>>>> + * >>>>>>>> + * once decoding starts, the timeout should not be hit. >>>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>>>> >>>>>>> (I might need to think more about how the semantics of this work.) >>>>>>> >>>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>>>> >>>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >>>> sure, can do that as well, not a problem. >>>> >>>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >>>> >>>> is this intentional or is it a bug? >>> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). >>> >>> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. >> thanks for the prompt answer. >> >> yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. >> But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) >> >> seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real) >> >> anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN. >> >> >> ret = avcodec_send_frame(enc, in_picture); >> if (ret < 0) >> goto error; >> >> while (1) { >> ret = avcodec_receive_packet(enc, &pkt); >> update_benchmark("encode_video %d.%d", ost->file_index, ost->index); >> if (ret == AVERROR(EAGAIN)) >> break; >> if (ret < 0) >> goto error; >> >> if (debug_ts) { >> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base), >> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base)); >> } >> >> if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY)) >> pkt.pts = ost->sync_opts; >> >> av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase); >> >> if (debug_ts) { >> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase), >> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase)); >> } >> >> frame_size = pkt.size; >> output_packet(of, &pkt, ost, 0); >> >> /* if two pass, output log */ >> if (ost->logfile && enc->stats_out) { >> fprintf(ost->logfile, "%s", enc->stats_out); >> } >> } >> } >> >> >> so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... >> >> >> >> does the above look ok to you? > Yes: send_frame() is always callable once, and then receive_packet() is callable repeatedly until it returns EAGAIN; once it does, send_frame() is necessarily callable again. > > Can you offer a sequence of valid returns from the lavc API which would break it? (Ignoring what the implementation behind it actually is for a moment.) yes, just the code I have in the patch I am preparing for v9 for instance where I remove the timeout. https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_enc.c#L267 encoding just blocks. After having queued all input buffers it dequeues the first packet ...but then it just and keeps on trying to do that for ever since we block for output ready and never send another frame. decoding works as expected. https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_dec.c#L148
On 04/09/17 21:34, Jorge Ramirez wrote: > On 09/04/2017 07:55 PM, Mark Thompson wrote: >> On 04/09/17 18:01, Jorge Ramirez wrote: >>> On 09/04/2017 06:31 PM, Mark Thompson wrote: >>>> On 04/09/17 17:00, Jorge Ramirez wrote: >>>>> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>>>>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>>>>> + * enough input * to start generating output. >>>>>>>>> + * >>>>>>>>> + * once decoding starts, the timeout should not be hit. >>>>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>>>>> >>>>>>>> (I might need to think more about how the semantics of this work.) >>>>>>>> >>>>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>>>>> >>>>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>>>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >>>>> sure, can do that as well, not a problem. >>>>> >>>>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >>>>> >>>>> is this intentional or is it a bug? >>>> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). >>>> >>>> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. >>> thanks for the prompt answer. >>> >>> yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. >>> But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) >>> >>> seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real) >>> >>> anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN. >>> >>> >>> ret = avcodec_send_frame(enc, in_picture); >>> if (ret < 0) >>> goto error; >>> >>> while (1) { >>> ret = avcodec_receive_packet(enc, &pkt); >>> update_benchmark("encode_video %d.%d", ost->file_index, ost->index); >>> if (ret == AVERROR(EAGAIN)) >>> break; >>> if (ret < 0) >>> goto error; >>> >>> if (debug_ts) { >>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base), >>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base)); >>> } >>> >>> if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY)) >>> pkt.pts = ost->sync_opts; >>> >>> av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase); >>> >>> if (debug_ts) { >>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase), >>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase)); >>> } >>> >>> frame_size = pkt.size; >>> output_packet(of, &pkt, ost, 0); >>> >>> /* if two pass, output log */ >>> if (ost->logfile && enc->stats_out) { >>> fprintf(ost->logfile, "%s", enc->stats_out); >>> } >>> } >>> } >>> >>> >>> so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... >>> >>> >>> >>> does the above look ok to you? >> Yes: send_frame() is always callable once, and then receive_packet() is callable repeatedly until it returns EAGAIN; once it does, send_frame() is necessarily callable again. >> >> Can you offer a sequence of valid returns from the lavc API which would break it? (Ignoring what the implementation behind it actually is for a moment.) > > > yes, just the code I have in the patch I am preparing for v9 for instance where I remove the timeout. > https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_enc.c#L267 > > encoding just blocks. > After having queued all input buffers it dequeues the first packet ...but then it just and keeps on trying to do that for ever since we block for output ready and never send another frame. > > decoding works as expected. > https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_dec.c#L148 There is too much code there for me to follow exactly what the sequence is without hardware to run it on. Could you post an ordered list of all of the avcodec_send_frame() and avcodec_receive_packet() calls and results which result in this deadlock? If it is an error in ffmpeg.c then those calls must be violating the API contact somehow independent of the actual implementation being used, and that could be verified and fixed without the v4l2 encoder. - Mark
On 09/04/2017 10:45 PM, Mark Thompson wrote: > On 04/09/17 21:34, Jorge Ramirez wrote: >> On 09/04/2017 07:55 PM, Mark Thompson wrote: >>> On 04/09/17 18:01, Jorge Ramirez wrote: >>>> On 09/04/2017 06:31 PM, Mark Thompson wrote: >>>>> On 04/09/17 17:00, Jorge Ramirez wrote: >>>>>> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>>>>>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>>>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>>>>>> + * enough input * to start generating output. >>>>>>>>>> + * >>>>>>>>>> + * once decoding starts, the timeout should not be hit. >>>>>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>>>>>> >>>>>>>>> (I might need to think more about how the semantics of this work.) >>>>>>>>> >>>>>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>>>>>> >>>>>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>>>>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >>>>>> sure, can do that as well, not a problem. >>>>>> >>>>>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >>>>>> >>>>>> is this intentional or is it a bug? >>>>> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). >>>>> >>>>> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. >>>> thanks for the prompt answer. >>>> >>>> yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. >>>> But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) >>>> >>>> seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real) >>>> >>>> anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN. >>>> >>>> >>>> ret = avcodec_send_frame(enc, in_picture); >>>> if (ret < 0) >>>> goto error; >>>> >>>> while (1) { >>>> ret = avcodec_receive_packet(enc, &pkt); >>>> update_benchmark("encode_video %d.%d", ost->file_index, ost->index); >>>> if (ret == AVERROR(EAGAIN)) >>>> break; >>>> if (ret < 0) >>>> goto error; >>>> >>>> if (debug_ts) { >>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base), >>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base)); >>>> } >>>> >>>> if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY)) >>>> pkt.pts = ost->sync_opts; >>>> >>>> av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase); >>>> >>>> if (debug_ts) { >>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase), >>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase)); >>>> } >>>> >>>> frame_size = pkt.size; >>>> output_packet(of, &pkt, ost, 0); >>>> >>>> /* if two pass, output log */ >>>> if (ost->logfile && enc->stats_out) { >>>> fprintf(ost->logfile, "%s", enc->stats_out); >>>> } >>>> } >>>> } >>>> >>>> >>>> so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... >>>> >>>> >>>> >>>> does the above look ok to you? >>> Yes: send_frame() is always callable once, and then receive_packet() is callable repeatedly until it returns EAGAIN; once it does, send_frame() is necessarily callable again. >>> >>> Can you offer a sequence of valid returns from the lavc API which would break it? (Ignoring what the implementation behind it actually is for a moment.) >> >> yes, just the code I have in the patch I am preparing for v9 for instance where I remove the timeout. >> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_enc.c#L267 >> >> encoding just blocks. >> After having queued all input buffers it dequeues the first packet ...but then it just and keeps on trying to do that for ever since we block for output ready and never send another frame. >> >> decoding works as expected. >> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_dec.c#L148 > There is too much code there for me to follow exactly what the sequence is without hardware to run it on. Could you post an ordered list of all of the avcodec_send_frame() and avcodec_receive_packet() calls and results which result in this deadlock? If it is an error in ffmpeg.c then those calls must be violating the API contact somehow independent of the actual implementation being used, and that could be verified and fixed without the v4l2 encoder. sorry for the rather long trace (I thought it was better than missing info or using http://haste.bin or similar) linaro@db820 raw]$ cat encode.h264.sh ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 linaro@db820 raw]$ ./encode.h264.sh ~/Src/git.zoltan.ffmpeg/ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 ffmpeg version N-87159-ga1e41bc747 Copyright (c) 2000-2017 the FFmpeg developers built with gcc 6.3.0 (Debian 6.3.0-21) 20170628 configuration: --disable-optimizations --disable-stripping libavutil 55. 74.100 / 55. 74.100 libavcodec 57.104.101 / 57.104.101 libavformat 57. 80.100 / 57. 80.100 libavdevice 57. 8.100 / 57. 8.100 libavfilter 6.101.100 / 6.101.100 libswscale 4. 7.103 / 4. 7.103 libswresample 2. 8.100 / 2. 8.100 Splitting the commandline. Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'. Reading option '-f' ... matched as option 'f' (force format) with argument 'rawvideo'. Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'nv12'. Reading option '-s:v' ... matched as option 's' (set frame size (WxH or abbreviation)) with argument '1280:720'. Reading option '-r' ... matched as option 'r' (set frame rate (Hz value, fraction or abbreviation)) with argument '25'. Reading option '-i' ... matched as input url with argument '/home/linaro/Videos/raw/freeway.yuv'. Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_v4l2m2m'. Reading option 'out/out.h264.mp4' ... matched as output url. Finished splitting the commandline. Parsing a group of options: global . Applying option loglevel (set logging level) with argument debug. Successfully parsed a group of options. Parsing a group of options: input url /home/linaro/Videos/raw/freeway.yuv. Applying option f (force format) with argument rawvideo. Applying option pix_fmt (set pixel format) with argument nv12. Applying option s:v (set frame size (WxH or abbreviation)) with argument 1280:720. Applying option r (set frame rate (Hz value, fraction or abbreviation)) with argument 25. Successfully parsed a group of options. Opening an input file: /home/linaro/Videos/raw/freeway.yuv. [rawvideo @ 0xaaaafee46490] Opening '/home/linaro/Videos/raw/freeway.yuv' for reading [file @ 0xaaaafee46d10] Setting default whitelist 'file,crypto' [rawvideo @ 0xaaaafee46490] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1 [rawvideo @ 0xaaaafee46490] All info found [rawvideo @ 0xaaaafee46490] Estimating duration from bitrate, this may be inaccurate [rawvideo @ 0xaaaafee46490] After avformat_find_stream_info() pos: 1382400 bytes read:1382400 seeks:0 frames:1 Input #0, rawvideo, from '/home/linaro/Videos/raw/freeway.yuv': Duration: 00:00:09.28, start: 0.000000, bitrate: 276480 kb/s Stream #0:0, 1, 1/25: Video: rawvideo, 1 reference frame (NV12 / 0x3231564E), nv12, 1280x720, 0/1, 276480 kb/s, 25 tbr, 25 tbn, 25 tbc Successfully opened the file. Parsing a group of options: output url out/out.h264.mp4. Applying option c:v (codec name) with argument h264_v4l2m2m. Successfully parsed a group of options. Opening an output file: out/out.h264.mp4. [file @ 0xaaaafee53930] Setting default whitelist 'file,crypto' Successfully opened the file. Stream mapping: Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m)) Press [q] to stop, [?] for help cur_dts is invalid (this is harmless if it occurs once at the start per stream) [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 detected 4 logical cores [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'video_size' to value '1280x720' [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pix_fmt' to value '25' [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'time_base' to value '1/25' [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pixel_aspect' to value '0/1' [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'sws_param' to value 'flags=2' [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'frame_rate' to value '25/1' [graph 0 input from stream 0:0 @ 0xaaaafee55430] w:1280 h:720 pixfmt:nv12 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2 [AVFilterGraph @ 0xaaaafee47080] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video1 [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video decoder' [h264_v4l2m2m @ 0xaaaafee51b70] output format not supported [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video0 [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder' [h264_v4l2m2m @ 0xaaaafee51b70] Using device /dev/video0 [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder' [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_out queuing 16 buffers [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_cap queuing 4 buffers [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: number of B-frames = 0 [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: header mode = 0 [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: bit rate = 200000 [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: gop size = 12 [h264_v4l2m2m @ 0xaaaafee51b70] Encoder Context: id (28), profile (-99), frame rate(25/1), number b-frames (0), gop size (12), bit rate (200000), qmin (2), qmax (31) [h264_v4l2m2m @ 0xaaaafee51b70] h264 profile not found [h264_v4l2m2m @ 0xaaaafee51b70] Encoder adjusted: qmin (0), qmax (51) [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: minimum video quantizer scale = 0 [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: maximum video quantizer scale = 51 Output #0, mp4, to 'out/out.h264.mp4': Metadata: encoder : Lavf57.80.100 Stream #0:0, 0, 1/12800: Video: h264 (h264_v4l2m2m), 1 reference frame (avc1 / 0x31637661), nv12, 1280x720, 0/1, q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc Metadata: encoder : Lavc57.104.101 h264_v4l2m2m Clipping frame in rate conversion by 0.000008 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 0 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 1 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 2 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 3 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 4 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 5 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 6 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 7 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 8 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 9 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 10 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 11 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 12 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 13 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 14 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 15 [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 16 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 0 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 1 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 2 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 3 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 4 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 5 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 6 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 7 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 8 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 9 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 10 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 11 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 12 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 13 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 14 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 15 [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 16 [at this point it blocks for ever since it has consumed all the input and we are blocking) The code in the encoder: static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) { V4L2m2mContext *s = avctx->priv_data; V4L2Context *const capture = &s->capture; V4L2Context *const output = &s->output; int ret; static int cnt = 0; if (!output->streamon) { ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON); if (ret) { av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n"); return ret; } } if (!capture->streamon) { ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON); if (ret) { av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n"); return ret; } } ret = ff_v4l2_enqueue_frame(output, frame); if (!ret) { av_log(avctx, AV_LOG_ERROR, "enqued raw frame %d\n", cnt++); } return ret; } /* Send and receive frame happen on the same thread, hence the need for a polling timeout */ static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt) { V4L2m2mContext *s = avctx->priv_data; V4L2Context *const capture = &s->capture; V4L2Context *const output = &s->output; unsigned int timeout = -1; /* block for ever until packets are ready to dequeue */ static int cnt = 0; int ret; if (s->draining) goto dequeue; if (!s->data_queued) { if (output->ops.get_buffer(output)) return AVERROR(EAGAIN); else s->data_queued = 1; } dequeue: ret = ff_v4l2_dequeue_packet(capture, avpkt, timeout); if (!ret) av_log(avctx, AV_LOG_ERROR, "dequeued encoded packet %d\n", cnt++); return ret; } > > - Mark > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 04/09/17 22:07, Jorge Ramirez wrote: > On 09/04/2017 10:45 PM, Mark Thompson wrote: >> On 04/09/17 21:34, Jorge Ramirez wrote: >>> On 09/04/2017 07:55 PM, Mark Thompson wrote: >>>> On 04/09/17 18:01, Jorge Ramirez wrote: >>>>> On 09/04/2017 06:31 PM, Mark Thompson wrote: >>>>>> On 04/09/17 17:00, Jorge Ramirez wrote: >>>>>>> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>>>>>>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>>>>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>>>>>>> + * enough input * to start generating output. >>>>>>>>>>> + * >>>>>>>>>>> + * once decoding starts, the timeout should not be hit. >>>>>>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>>>>>>> >>>>>>>>>> (I might need to think more about how the semantics of this work.) >>>>>>>>>> >>>>>>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>>>>>>> >>>>>>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>>>>>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >>>>>>> sure, can do that as well, not a problem. >>>>>>> >>>>>>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >>>>>>> >>>>>>> is this intentional or is it a bug? >>>>>> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). >>>>>> >>>>>> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. >>>>> thanks for the prompt answer. >>>>> >>>>> yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. >>>>> But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) >>>>> >>>>> seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real) >>>>> >>>>> anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN. >>>>> >>>>> >>>>> ret = avcodec_send_frame(enc, in_picture); >>>>> if (ret < 0) >>>>> goto error; >>>>> >>>>> while (1) { >>>>> ret = avcodec_receive_packet(enc, &pkt); >>>>> update_benchmark("encode_video %d.%d", ost->file_index, ost->index); >>>>> if (ret == AVERROR(EAGAIN)) >>>>> break; >>>>> if (ret < 0) >>>>> goto error; >>>>> >>>>> if (debug_ts) { >>>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base), >>>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base)); >>>>> } >>>>> >>>>> if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY)) >>>>> pkt.pts = ost->sync_opts; >>>>> >>>>> av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase); >>>>> >>>>> if (debug_ts) { >>>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase), >>>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase)); >>>>> } >>>>> >>>>> frame_size = pkt.size; >>>>> output_packet(of, &pkt, ost, 0); >>>>> >>>>> /* if two pass, output log */ >>>>> if (ost->logfile && enc->stats_out) { >>>>> fprintf(ost->logfile, "%s", enc->stats_out); >>>>> } >>>>> } >>>>> } >>>>> >>>>> >>>>> so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... >>>>> >>>>> >>>>> >>>>> does the above look ok to you? >>>> Yes: send_frame() is always callable once, and then receive_packet() is callable repeatedly until it returns EAGAIN; once it does, send_frame() is necessarily callable again. >>>> >>>> Can you offer a sequence of valid returns from the lavc API which would break it? (Ignoring what the implementation behind it actually is for a moment.) >>> >>> yes, just the code I have in the patch I am preparing for v9 for instance where I remove the timeout. >>> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_enc.c#L267 >>> >>> encoding just blocks. >>> After having queued all input buffers it dequeues the first packet ...but then it just and keeps on trying to do that for ever since we block for output ready and never send another frame. >>> >>> decoding works as expected. >>> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_dec.c#L148 >> There is too much code there for me to follow exactly what the sequence is without hardware to run it on. Could you post an ordered list of all of the avcodec_send_frame() and avcodec_receive_packet() calls and results which result in this deadlock? If it is an error in ffmpeg.c then those calls must be violating the API contact somehow independent of the actual implementation being used, and that could be verified and fixed without the v4l2 encoder. > > sorry for the rather long trace (I thought it was better than missing info or using http://haste.bin or similar) > > > linaro@db820 raw]$ cat encode.h264.sh > ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 > > > linaro@db820 raw]$ ./encode.h264.sh > ~/Src/git.zoltan.ffmpeg/ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 > -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 > ffmpeg version N-87159-ga1e41bc747 Copyright (c) 2000-2017 the FFmpeg developers > built with gcc 6.3.0 (Debian 6.3.0-21) 20170628 > configuration: --disable-optimizations --disable-stripping > libavutil 55. 74.100 / 55. 74.100 > libavcodec 57.104.101 / 57.104.101 > libavformat 57. 80.100 / 57. 80.100 > libavdevice 57. 8.100 / 57. 8.100 > libavfilter 6.101.100 / 6.101.100 > libswscale 4. 7.103 / 4. 7.103 > libswresample 2. 8.100 / 2. 8.100 > Splitting the commandline. > Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'. > Reading option '-f' ... matched as option 'f' (force format) with argument 'rawvideo'. > Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'nv12'. > Reading option '-s:v' ... matched as option 's' (set frame size (WxH or abbreviation)) with argument '1280:720'. > Reading option '-r' ... matched as option 'r' (set frame rate (Hz value, fraction or abbreviation)) with argument '25'. > Reading option '-i' ... matched as input url with argument '/home/linaro/Videos/raw/freeway.yuv'. > Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_v4l2m2m'. > Reading option 'out/out.h264.mp4' ... matched as output url. > Finished splitting the commandline. > Parsing a group of options: global . > Applying option loglevel (set logging level) with argument debug. > Successfully parsed a group of options. > Parsing a group of options: input url /home/linaro/Videos/raw/freeway.yuv. > Applying option f (force format) with argument rawvideo. > Applying option pix_fmt (set pixel format) with argument nv12. > Applying option s:v (set frame size (WxH or abbreviation)) with argument 1280:720. > Applying option r (set frame rate (Hz value, fraction or abbreviation)) with argument 25. > Successfully parsed a group of options. > Opening an input file: /home/linaro/Videos/raw/freeway.yuv. > [rawvideo @ 0xaaaafee46490] Opening '/home/linaro/Videos/raw/freeway.yuv' for reading > [file @ 0xaaaafee46d10] Setting default whitelist 'file,crypto' > [rawvideo @ 0xaaaafee46490] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1 > [rawvideo @ 0xaaaafee46490] All info found > [rawvideo @ 0xaaaafee46490] Estimating duration from bitrate, this may be inaccurate > [rawvideo @ 0xaaaafee46490] After avformat_find_stream_info() pos: 1382400 bytes read:1382400 seeks:0 frames:1 > Input #0, rawvideo, from '/home/linaro/Videos/raw/freeway.yuv': > Duration: 00:00:09.28, start: 0.000000, bitrate: 276480 kb/s > Stream #0:0, 1, 1/25: Video: rawvideo, 1 reference frame (NV12 / 0x3231564E), nv12, 1280x720, 0/1, 276480 kb/s, 25 tbr, 25 tbn, 25 tbc > Successfully opened the file. > Parsing a group of options: output url out/out.h264.mp4. > Applying option c:v (codec name) with argument h264_v4l2m2m. > Successfully parsed a group of options. > Opening an output file: out/out.h264.mp4. > [file @ 0xaaaafee53930] Setting default whitelist 'file,crypto' > Successfully opened the file. > Stream mapping: > Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m)) > Press [q] to stop, [?] for help > cur_dts is invalid (this is harmless if it occurs once at the start per stream) > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > detected 4 logical cores > [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'video_size' to value '1280x720' > [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pix_fmt' to value '25' > [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'time_base' to value '1/25' > [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pixel_aspect' to value '0/1' > [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'sws_param' to value 'flags=2' > [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'frame_rate' to value '25/1' > [graph 0 input from stream 0:0 @ 0xaaaafee55430] w:1280 h:720 pixfmt:nv12 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2 > [AVFilterGraph @ 0xaaaafee47080] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed > [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video1 > [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video decoder' > [h264_v4l2m2m @ 0xaaaafee51b70] output format not supported > [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video0 > [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder' > [h264_v4l2m2m @ 0xaaaafee51b70] Using device /dev/video0 > [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder' > [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_out queuing 16 buffers > [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_cap queuing 4 buffers > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: number of B-frames = 0 > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: header mode = 0 > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: bit rate = 200000 > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: gop size = 12 > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder Context: id (28), profile (-99), frame rate(25/1), number b-frames (0), gop size (12), bit rate (200000), qmin (2), qmax (31) > [h264_v4l2m2m @ 0xaaaafee51b70] h264 profile not found > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder adjusted: qmin (0), qmax (51) > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: minimum video quantizer scale = 0 > [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: maximum video quantizer scale = 51 > Output #0, mp4, to 'out/out.h264.mp4': > Metadata: > encoder : Lavf57.80.100 > Stream #0:0, 0, 1/12800: Video: h264 (h264_v4l2m2m), 1 reference frame (avc1 / 0x31637661), nv12, 1280x720, 0/1, q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc > Metadata: > encoder : Lavc57.104.101 h264_v4l2m2m > Clipping frame in rate conversion by 0.000008 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 0 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 1 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 2 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 3 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 4 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 5 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 6 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 7 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 8 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 9 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 10 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 11 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 12 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 13 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 14 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 15 > [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 > [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 16 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 0 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 1 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 2 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 3 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 4 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 5 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 6 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 7 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 8 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 9 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 10 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 11 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 12 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 13 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 14 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 15 > [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 16 > > [at this point it blocks for ever since it has consumed all the input and we are blocking) > > The code in the encoder: > > static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) > { > V4L2m2mContext *s = avctx->priv_data; > V4L2Context *const capture = &s->capture; > V4L2Context *const output = &s->output; > int ret; > static int cnt = 0; > > if (!output->streamon) { > ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON); > if (ret) { > av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n"); > return ret; > } > } > > if (!capture->streamon) { > ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON); > if (ret) { > av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n"); > return ret; > } > } > > ret = ff_v4l2_enqueue_frame(output, frame); > if (!ret) { > av_log(avctx, AV_LOG_ERROR, "enqued raw frame %d\n", cnt++); > } > > return ret; > } > > /* Send and receive frame happen on the same thread, hence the need for a polling timeout */ > static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt) > { > V4L2m2mContext *s = avctx->priv_data; > V4L2Context *const capture = &s->capture; > V4L2Context *const output = &s->output; > unsigned int timeout = -1; /* block for ever until packets are ready to dequeue */ > static int cnt = 0; > int ret; > > if (s->draining) > goto dequeue; > > if (!s->data_queued) { > if (output->ops.get_buffer(output)) > return AVERROR(EAGAIN); > else s->data_queued = 1; > } > > dequeue: > > ret = ff_v4l2_dequeue_packet(capture, avpkt, timeout); > if (!ret) > av_log(avctx, AV_LOG_ERROR, "dequeued encoded packet %d\n", cnt++); > > return ret; > } So the sequence of calls is: send_frame(frame 0) -> success receive_packet() -> EAGAIN send_frame(frame 1) -> success receive_packet() -> EAGAIN ... send_frame(frame 15) -> success receive_packet() -> EAGAIN send_frame(frame 16) -> success receive_packet() -> packet 0 receive_packet() -> packet 1 ... receive_packet() -> packet 15 receive_packet() -> packet 16 receive_packet() -> blocks This appears correct to me - since EAGAIN has not been returned by a receive_packet() call, it can be called again repeatedly (as documented in avcodec.h). Do you disagree? I think that the problem is that you are only polling on the frame buffer queue when blocking, so you don't see the packet buffers becoming free in the packet buffer queue - if you did see and dequeue them, you would then return EAGAIN to indicate that more input is needed. (See comments in <e4c6a8d7-798a-dfdb-b748-42936e944829@jkqxz.net>.) - Mark
On 09/04/2017 11:29 PM, Mark Thompson wrote: > On 04/09/17 22:07, Jorge Ramirez wrote: >> On 09/04/2017 10:45 PM, Mark Thompson wrote: >>> On 04/09/17 21:34, Jorge Ramirez wrote: >>>> On 09/04/2017 07:55 PM, Mark Thompson wrote: >>>>> On 04/09/17 18:01, Jorge Ramirez wrote: >>>>>> On 09/04/2017 06:31 PM, Mark Thompson wrote: >>>>>>> On 04/09/17 17:00, Jorge Ramirez wrote: >>>>>>>> On 09/03/2017 08:23 PM, Mark Thompson wrote: >>>>>>>>> On 03/09/17 17:54, Jorge Ramirez wrote: >>>>>>>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote: >>>>>>>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a >>>>>>>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received >>>>>>>>>>>> + * enough input * to start generating output. >>>>>>>>>>>> + * >>>>>>>>>>>> + * once decoding starts, the timeout should not be hit. >>>>>>>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds? >>>>>>>>>>> >>>>>>>>>>> (I might need to think more about how the semantics of this work.) >>>>>>>>>>> >>>>>>>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact) >>>>>>>>>> >>>>>>>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output? >>>>>>>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases. >>>>>>>> sure, can do that as well, not a problem. >>>>>>>> >>>>>>>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever) >>>>>>>> >>>>>>>> is this intentional or is it a bug? >>>>>>> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h). >>>>>>> >>>>>>> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong. >>>>>> thanks for the prompt answer. >>>>>> >>>>>> yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet. >>>>>> But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing) >>>>>> >>>>>> seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real) >>>>>> >>>>>> anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN. >>>>>> >>>>>> >>>>>> ret = avcodec_send_frame(enc, in_picture); >>>>>> if (ret < 0) >>>>>> goto error; >>>>>> >>>>>> while (1) { >>>>>> ret = avcodec_receive_packet(enc, &pkt); >>>>>> update_benchmark("encode_video %d.%d", ost->file_index, ost->index); >>>>>> if (ret == AVERROR(EAGAIN)) >>>>>> break; >>>>>> if (ret < 0) >>>>>> goto error; >>>>>> >>>>>> if (debug_ts) { >>>>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>>>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>>>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base), >>>>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base)); >>>>>> } >>>>>> >>>>>> if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY)) >>>>>> pkt.pts = ost->sync_opts; >>>>>> >>>>>> av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase); >>>>>> >>>>>> if (debug_ts) { >>>>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video " >>>>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n", >>>>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase), >>>>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase)); >>>>>> } >>>>>> >>>>>> frame_size = pkt.size; >>>>>> output_packet(of, &pkt, ost, 0); >>>>>> >>>>>> /* if two pass, output log */ >>>>>> if (ost->logfile && enc->stats_out) { >>>>>> fprintf(ost->logfile, "%s", enc->stats_out); >>>>>> } >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input... >>>>>> >>>>>> >>>>>> >>>>>> does the above look ok to you? >>>>> Yes: send_frame() is always callable once, and then receive_packet() is callable repeatedly until it returns EAGAIN; once it does, send_frame() is necessarily callable again. >>>>> >>>>> Can you offer a sequence of valid returns from the lavc API which would break it? (Ignoring what the implementation behind it actually is for a moment.) >>>> yes, just the code I have in the patch I am preparing for v9 for instance where I remove the timeout. >>>> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_enc.c#L267 >>>> >>>> encoding just blocks. >>>> After having queued all input buffers it dequeues the first packet ...but then it just and keeps on trying to do that for ever since we block for output ready and never send another frame. >>>> >>>> decoding works as expected. >>>> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_dec.c#L148 >>> There is too much code there for me to follow exactly what the sequence is without hardware to run it on. Could you post an ordered list of all of the avcodec_send_frame() and avcodec_receive_packet() calls and results which result in this deadlock? If it is an error in ffmpeg.c then those calls must be violating the API contact somehow independent of the actual implementation being used, and that could be verified and fixed without the v4l2 encoder. >> sorry for the rather long trace (I thought it was better than missing info or using http://haste.bin or similar) >> >> >> linaro@db820 raw]$ cat encode.h264.sh >> ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 >> >> >> linaro@db820 raw]$ ./encode.h264.sh >> ~/Src/git.zoltan.ffmpeg/ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 >> -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4 >> ffmpeg version N-87159-ga1e41bc747 Copyright (c) 2000-2017 the FFmpeg developers >> built with gcc 6.3.0 (Debian 6.3.0-21) 20170628 >> configuration: --disable-optimizations --disable-stripping >> libavutil 55. 74.100 / 55. 74.100 >> libavcodec 57.104.101 / 57.104.101 >> libavformat 57. 80.100 / 57. 80.100 >> libavdevice 57. 8.100 / 57. 8.100 >> libavfilter 6.101.100 / 6.101.100 >> libswscale 4. 7.103 / 4. 7.103 >> libswresample 2. 8.100 / 2. 8.100 >> Splitting the commandline. >> Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'. >> Reading option '-f' ... matched as option 'f' (force format) with argument 'rawvideo'. >> Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'nv12'. >> Reading option '-s:v' ... matched as option 's' (set frame size (WxH or abbreviation)) with argument '1280:720'. >> Reading option '-r' ... matched as option 'r' (set frame rate (Hz value, fraction or abbreviation)) with argument '25'. >> Reading option '-i' ... matched as input url with argument '/home/linaro/Videos/raw/freeway.yuv'. >> Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_v4l2m2m'. >> Reading option 'out/out.h264.mp4' ... matched as output url. >> Finished splitting the commandline. >> Parsing a group of options: global . >> Applying option loglevel (set logging level) with argument debug. >> Successfully parsed a group of options. >> Parsing a group of options: input url /home/linaro/Videos/raw/freeway.yuv. >> Applying option f (force format) with argument rawvideo. >> Applying option pix_fmt (set pixel format) with argument nv12. >> Applying option s:v (set frame size (WxH or abbreviation)) with argument 1280:720. >> Applying option r (set frame rate (Hz value, fraction or abbreviation)) with argument 25. >> Successfully parsed a group of options. >> Opening an input file: /home/linaro/Videos/raw/freeway.yuv. >> [rawvideo @ 0xaaaafee46490] Opening '/home/linaro/Videos/raw/freeway.yuv' for reading >> [file @ 0xaaaafee46d10] Setting default whitelist 'file,crypto' >> [rawvideo @ 0xaaaafee46490] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1 >> [rawvideo @ 0xaaaafee46490] All info found >> [rawvideo @ 0xaaaafee46490] Estimating duration from bitrate, this may be inaccurate >> [rawvideo @ 0xaaaafee46490] After avformat_find_stream_info() pos: 1382400 bytes read:1382400 seeks:0 frames:1 >> Input #0, rawvideo, from '/home/linaro/Videos/raw/freeway.yuv': >> Duration: 00:00:09.28, start: 0.000000, bitrate: 276480 kb/s >> Stream #0:0, 1, 1/25: Video: rawvideo, 1 reference frame (NV12 / 0x3231564E), nv12, 1280x720, 0/1, 276480 kb/s, 25 tbr, 25 tbn, 25 tbc >> Successfully opened the file. >> Parsing a group of options: output url out/out.h264.mp4. >> Applying option c:v (codec name) with argument h264_v4l2m2m. >> Successfully parsed a group of options. >> Opening an output file: out/out.h264.mp4. >> [file @ 0xaaaafee53930] Setting default whitelist 'file,crypto' >> Successfully opened the file. >> Stream mapping: >> Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m)) >> Press [q] to stop, [?] for help >> cur_dts is invalid (this is harmless if it occurs once at the start per stream) >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> detected 4 logical cores >> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'video_size' to value '1280x720' >> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pix_fmt' to value '25' >> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'time_base' to value '1/25' >> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pixel_aspect' to value '0/1' >> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'sws_param' to value 'flags=2' >> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'frame_rate' to value '25/1' >> [graph 0 input from stream 0:0 @ 0xaaaafee55430] w:1280 h:720 pixfmt:nv12 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2 >> [AVFilterGraph @ 0xaaaafee47080] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed >> [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video1 >> [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video decoder' >> [h264_v4l2m2m @ 0xaaaafee51b70] output format not supported >> [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video0 >> [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder' >> [h264_v4l2m2m @ 0xaaaafee51b70] Using device /dev/video0 >> [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder' >> [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_out queuing 16 buffers >> [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_cap queuing 4 buffers >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: number of B-frames = 0 >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: header mode = 0 >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: bit rate = 200000 >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: gop size = 12 >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder Context: id (28), profile (-99), frame rate(25/1), number b-frames (0), gop size (12), bit rate (200000), qmin (2), qmax (31) >> [h264_v4l2m2m @ 0xaaaafee51b70] h264 profile not found >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder adjusted: qmin (0), qmax (51) >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: minimum video quantizer scale = 0 >> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: maximum video quantizer scale = 51 >> Output #0, mp4, to 'out/out.h264.mp4': >> Metadata: >> encoder : Lavf57.80.100 >> Stream #0:0, 0, 1/12800: Video: h264 (h264_v4l2m2m), 1 reference frame (avc1 / 0x31637661), nv12, 1280x720, 0/1, q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc >> Metadata: >> encoder : Lavc57.104.101 h264_v4l2m2m >> Clipping frame in rate conversion by 0.000008 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 0 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 1 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 2 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 3 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 4 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 5 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 6 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 7 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 8 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 9 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 10 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 11 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 12 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 13 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 14 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 15 >> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920 >> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 16 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 0 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 1 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 2 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 3 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 4 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 5 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 6 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 7 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 8 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 9 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 10 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 11 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 12 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 13 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 14 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 15 >> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 16 >> >> [at this point it blocks for ever since it has consumed all the input and we are blocking) >> >> The code in the encoder: >> >> static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) >> { >> V4L2m2mContext *s = avctx->priv_data; >> V4L2Context *const capture = &s->capture; >> V4L2Context *const output = &s->output; >> int ret; >> static int cnt = 0; >> >> if (!output->streamon) { >> ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON); >> if (ret) { >> av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n"); >> return ret; >> } >> } >> >> if (!capture->streamon) { >> ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON); >> if (ret) { >> av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n"); >> return ret; >> } >> } >> >> ret = ff_v4l2_enqueue_frame(output, frame); >> if (!ret) { >> av_log(avctx, AV_LOG_ERROR, "enqued raw frame %d\n", cnt++); >> } >> >> return ret; >> } >> >> /* Send and receive frame happen on the same thread, hence the need for a polling timeout */ >> static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt) >> { >> V4L2m2mContext *s = avctx->priv_data; >> V4L2Context *const capture = &s->capture; >> V4L2Context *const output = &s->output; >> unsigned int timeout = -1; /* block for ever until packets are ready to dequeue */ >> static int cnt = 0; >> int ret; >> >> if (s->draining) >> goto dequeue; >> >> if (!s->data_queued) { >> if (output->ops.get_buffer(output)) >> return AVERROR(EAGAIN); >> else s->data_queued = 1; >> } >> >> dequeue: >> >> ret = ff_v4l2_dequeue_packet(capture, avpkt, timeout); >> if (!ret) >> av_log(avctx, AV_LOG_ERROR, "dequeued encoded packet %d\n", cnt++); >> >> return ret; >> } > So the sequence of calls is: > > send_frame(frame 0) -> success > receive_packet() -> EAGAIN > send_frame(frame 1) -> success > receive_packet() -> EAGAIN > ... > send_frame(frame 15) -> success > receive_packet() -> EAGAIN > send_frame(frame 16) -> success > receive_packet() -> packet 0 > receive_packet() -> packet 1 > ... > receive_packet() -> packet 15 > receive_packet() -> packet 16 > receive_packet() -> blocks > > This appears correct to me - since EAGAIN has not been returned by a receive_packet() call, it can be called again repeatedly (as documented in avcodec.h). Do you disagree? I would have expected that after a packet is received, a new frame is enqueued instead of executing again receive_packet (why do we that? what is the benefit?) under these circumsntances I can't block in receive_packet blindly, I have to track in user-space what the driver has in its guts and predict how much it needs to keep working....I dont think it is a good idea. > > I think that the problem is that you are only polling on the frame buffer queue when blocking, so you don't see the packet buffers becoming free in the packet buffer queue - if you did see and dequeue them, you would then return EAGAIN to indicate that more input is needed. (See comments in <e4c6a8d7-798a-dfdb-b748-42936e944829@jkqxz.net>.) I could manually track it that way with additional counters - so before blocking I could see count many frames are enqueued in the input and if there is not at least one I could just return EAGAIN. but the behaviour differs from encoding. in decoding I queue all buffers and then block in capture; everytime it unblocks and returns a new frame back to ffmpeg, the next call just enqueues a packet (it doesnt try to dequeue another frame) so the pipeline never stops. I think the code should to be symmetrical for encoding and decoding... > > - Mark > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 04/09/17 22:55, Jorge Ramirez wrote: > On 09/04/2017 11:29 PM, Mark Thompson wrote: >>> ... stuff ... >> >> So the sequence of calls is: >> >> send_frame(frame 0) -> success >> receive_packet() -> EAGAIN >> send_frame(frame 1) -> success >> receive_packet() -> EAGAIN >> ... >> send_frame(frame 15) -> success >> receive_packet() -> EAGAIN >> send_frame(frame 16) -> success >> receive_packet() -> packet 0 >> receive_packet() -> packet 1 >> ... >> receive_packet() -> packet 15 >> receive_packet() -> packet 16 >> receive_packet() -> blocks >> >> This appears correct to me - since EAGAIN has not been returned by a receive_packet() call, it can be called again repeatedly (as documented in avcodec.h). Do you disagree? > > I would have expected that after a packet is received, a new frame is enqueued instead of executing again receive_packet (why do we that? what is the benefit?) > under these circumsntances I can't block in receive_packet blindly, I have to track in user-space what the driver has in its guts and predict how much it needs to keep working....I dont think it is a good idea. Feel free to submit a patch to avcodec.h which changes the definition of the API. >> I think that the problem is that you are only polling on the frame buffer queue when blocking, so you don't see the packet buffers becoming free in the packet buffer queue - if you did see and dequeue them, you would then return EAGAIN to indicate that more input is needed. (See comments in <e4c6a8d7-798a-dfdb-b748-42936e944829@jkqxz.net>.) > > I could manually track it that way with additional counters - so before blocking I could see count many frames are enqueued in the input and if there is not at least one I could just return EAGAIN. > but the behaviour differs from encoding. The behaviour is identical. You return the output buffer if there is one available on the output queue, otherwise you return EAGAIN if there is any space on the input queue, otherwise you block waiting for either of those to become possible. > in decoding I queue all buffers and then block in capture; everytime it unblocks and returns a new frame back to ffmpeg, the next call just enqueues a packet (it doesnt try to dequeue another frame) so the pipeline never stops. > > I think the code should to be symmetrical for encoding and decoding... Yes.
On 09/05/2017 12:16 AM, Mark Thompson wrote: > On 04/09/17 22:55, Jorge Ramirez wrote: >> On 09/04/2017 11:29 PM, Mark Thompson wrote: >>>> ... stuff ... >>> So the sequence of calls is: >>> >>> send_frame(frame 0) -> success >>> receive_packet() -> EAGAIN >>> send_frame(frame 1) -> success >>> receive_packet() -> EAGAIN >>> ... >>> send_frame(frame 15) -> success >>> receive_packet() -> EAGAIN >>> send_frame(frame 16) -> success >>> receive_packet() -> packet 0 >>> receive_packet() -> packet 1 >>> ... >>> receive_packet() -> packet 15 >>> receive_packet() -> packet 16 >>> receive_packet() -> blocks >>> >>> This appears correct to me - since EAGAIN has not been returned by a receive_packet() call, it can be called again repeatedly (as documented in avcodec.h). Do you disagree? >> I would have expected that after a packet is received, a new frame is enqueued instead of executing again receive_packet (why do we that? what is the benefit?) >> under these circumsntances I can't block in receive_packet blindly, I have to track in user-space what the driver has in its guts and predict how much it needs to keep working....I dont think it is a good idea. > Feel free to submit a patch to avcodec.h which changes the definition of the API. no, that is ok. I can work around it easily (maybe v4l2 has special needs compared to the rest of ffmpeg) > >>> I think that the problem is that you are only polling on the frame buffer queue when blocking, so you don't see the packet buffers becoming free in the packet buffer queue - if you did see and dequeue them, you would then return EAGAIN to indicate that more input is needed. (See comments in <e4c6a8d7-798a-dfdb-b748-42936e944829@jkqxz.net>.) >> I could manually track it that way with additional counters - so before blocking I could see count many frames are enqueued in the input and if there is not at least one I could just return EAGAIN. >> but the behaviour differs from encoding. > The behaviour is identical. You return the output buffer if there is one available on the output queue, otherwise you return EAGAIN if there is any space on the input queue, otherwise you block waiting for either of those to become possible. but we must be talking about different things because what I observe in decode it is 180 degrees away from identical In the decode case the receive_frame API (a single call for input and output) can implement a simple model [1] - receive a packet, - push it to the input queue (push till the input queue is full at init) - block for data to be ready in the output queue. static int v4l2m2m_receive_frame(AVCodecContext *avctx, AVFrame *frame) { V4L2m2mContext *s = avctx->priv_data; V4L2Context *const capture = &s->capture; V4L2Context *const output = &s->output; AVPacket avpkt = {0}; int timeout = -1; /* block for capture ready */ int ret; ret = ff_decode_get_packet(avctx, &avpkt); if (ret < 0 && ret != AVERROR_EOF) return ret; if (s->draining) goto dequeue; ret = ff_v4l2_enqueue_packet(output, &avpkt); if (ret < 0) return ret; if (avpkt.size) { ret = try_start(avctx); if (ret) return 0; } if (!s->data_queued) { if (output->ops.get_buffer(output)) return AVERROR(EAGAIN); else s->data_queued = 1; } dequeue: return ff_v4l2_dequeue_frame(capture, frame, timeout); } To sum up,. every time a packet is pushed to input, a frame can be dequeued. However the encode API seems wrongly designed (heavy coupled) since receive_frame has to check if there is space in the input queue. Receive frame should only care about the output queue - otherwise what is the reason for having two calls send_frame and receive_packet? In my view, receive_packet should just block. And send_frame should be called everytime a packet is received in receive_packet. that way I can 1. fill the input pipe 2. start streaming always blocking for output. > >> in decoding I queue all buffers and then block in capture; everytime it unblocks and returns a new frame back to ffmpeg, the next call just enqueues a packet (it doesnt try to dequeue another frame) so the pipeline never stops. >> >> I think the code should to be symmetrical for encoding and decoding... > Yes. cool. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 09/05/2017 09:16 AM, Jorge Ramirez wrote: > On 09/05/2017 12:16 AM, Mark Thompson wrote: >> On 04/09/17 22:55, Jorge Ramirez wrote: >>> On 09/04/2017 11:29 PM, Mark Thompson wrote: >>>>> ... stuff ... >>>> So the sequence of calls is: >>>> >>>> send_frame(frame 0) -> success >>>> receive_packet() -> EAGAIN >>>> send_frame(frame 1) -> success >>>> receive_packet() -> EAGAIN >>>> ... >>>> send_frame(frame 15) -> success >>>> receive_packet() -> EAGAIN >>>> send_frame(frame 16) -> success >>>> receive_packet() -> packet 0 >>>> receive_packet() -> packet 1 >>>> ... >>>> receive_packet() -> packet 15 >>>> receive_packet() -> packet 16 >>>> receive_packet() -> blocks >>>> >>>> This appears correct to me - since EAGAIN has not been returned by >>>> a receive_packet() call, it can be called again repeatedly (as >>>> documented in avcodec.h). Do you disagree? >>> I would have expected that after a packet is received, a new frame >>> is enqueued instead of executing again receive_packet (why do we >>> that? what is the benefit?) >>> under these circumsntances I can't block in receive_packet blindly, >>> I have to track in user-space what the driver has in its guts and >>> predict how much it needs to keep working....I dont think it is a >>> good idea. >> Feel free to submit a patch to avcodec.h which changes the definition >> of the API. > > no, that is ok. I can work around it easily (maybe v4l2 has special > needs compared to the rest of ffmpeg) > >> >>>> I think that the problem is that you are only polling on the frame >>>> buffer queue when blocking, so you don't see the packet buffers >>>> becoming free in the packet buffer queue - if you did see and >>>> dequeue them, you would then return EAGAIN to indicate that more >>>> input is needed. (See comments in >>>> <e4c6a8d7-798a-dfdb-b748-42936e944829@jkqxz.net>.) >>> I could manually track it that way with additional counters - so >>> before blocking I could see count many frames are enqueued in the >>> input and if there is not at least one I could just return EAGAIN. >>> but the behaviour differs from encoding. >> The behaviour is identical. You return the output buffer if there is >> one available on the output queue, otherwise you return EAGAIN if >> there is any space on the input queue, otherwise you block waiting >> for either of those to become possible. > I made some modifications to the way I was dequeuing buffers so now it polls for both input and output queues. you were right, things work much smoother this way...thanks. having said that I still I find the encoding API wrongly defined (no need for two calls that are in fact highly couple to each other in my opinion)
On Tue, 5 Sep 2017 10:03:49 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 09/05/2017 09:16 AM, Jorge Ramirez wrote: > > On 09/05/2017 12:16 AM, Mark Thompson wrote: > >> On 04/09/17 22:55, Jorge Ramirez wrote: > >>> On 09/04/2017 11:29 PM, Mark Thompson wrote: > >>>>> ... stuff ... > >>>> So the sequence of calls is: > >>>> > >>>> send_frame(frame 0) -> success > >>>> receive_packet() -> EAGAIN > >>>> send_frame(frame 1) -> success > >>>> receive_packet() -> EAGAIN > >>>> ... > >>>> send_frame(frame 15) -> success > >>>> receive_packet() -> EAGAIN > >>>> send_frame(frame 16) -> success > >>>> receive_packet() -> packet 0 > >>>> receive_packet() -> packet 1 > >>>> ... > >>>> receive_packet() -> packet 15 > >>>> receive_packet() -> packet 16 > >>>> receive_packet() -> blocks > >>>> > >>>> This appears correct to me - since EAGAIN has not been returned by > >>>> a receive_packet() call, it can be called again repeatedly (as > >>>> documented in avcodec.h). Do you disagree? > >>> I would have expected that after a packet is received, a new frame > >>> is enqueued instead of executing again receive_packet (why do we > >>> that? what is the benefit?) > >>> under these circumsntances I can't block in receive_packet blindly, > >>> I have to track in user-space what the driver has in its guts and > >>> predict how much it needs to keep working....I dont think it is a > >>> good idea. > >> Feel free to submit a patch to avcodec.h which changes the definition > >> of the API. > > > > no, that is ok. I can work around it easily (maybe v4l2 has special > > needs compared to the rest of ffmpeg) > > > >> > >>>> I think that the problem is that you are only polling on the frame > >>>> buffer queue when blocking, so you don't see the packet buffers > >>>> becoming free in the packet buffer queue - if you did see and > >>>> dequeue them, you would then return EAGAIN to indicate that more > >>>> input is needed. (See comments in > >>>> <e4c6a8d7-798a-dfdb-b748-42936e944829@jkqxz.net>.) > >>> I could manually track it that way with additional counters - so > >>> before blocking I could see count many frames are enqueued in the > >>> input and if there is not at least one I could just return EAGAIN. > >>> but the behaviour differs from encoding. > >> The behaviour is identical. You return the output buffer if there is > >> one available on the output queue, otherwise you return EAGAIN if > >> there is any space on the input queue, otherwise you block waiting > >> for either of those to become possible. > > > > > I made some modifications to the way I was dequeuing buffers so now it > polls for both input and output queues. you were right, things work much > smoother this way...thanks. > > having said that I still I find the encoding API wrongly defined (no > need for two calls that are in fact highly couple to each other in my > opinion) The encoding API works exactly the same way as the decoding API, just that the decoding API has an helper internally that makes it easier to merge send/receive into 1 function. You could do the same on the encoding side. Other than that, the API is meant for single-threaded use, so we can't do things like randomly blocking calls and requiring reentrant calls from other threads to unblock them.
On 09/05/2017 10:27 AM, wm4 wrote: > On Tue, 5 Sep 2017 10:03:49 +0200 > Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > >> On 09/05/2017 09:16 AM, Jorge Ramirez wrote: >>> On 09/05/2017 12:16 AM, Mark Thompson wrote: >>>> On 04/09/17 22:55, Jorge Ramirez wrote: >>>>> On 09/04/2017 11:29 PM, Mark Thompson wrote: >>>>>>> ... stuff ... >>>>>> So the sequence of calls is: >>>>>> >>>>>> send_frame(frame 0) -> success >>>>>> receive_packet() -> EAGAIN >>>>>> send_frame(frame 1) -> success >>>>>> receive_packet() -> EAGAIN >>>>>> ... >>>>>> send_frame(frame 15) -> success >>>>>> receive_packet() -> EAGAIN >>>>>> send_frame(frame 16) -> success >>>>>> receive_packet() -> packet 0 >>>>>> receive_packet() -> packet 1 >>>>>> ... >>>>>> receive_packet() -> packet 15 >>>>>> receive_packet() -> packet 16 >>>>>> receive_packet() -> blocks >>>>>> >>>>>> This appears correct to me - since EAGAIN has not been returned by >>>>>> a receive_packet() call, it can be called again repeatedly (as >>>>>> documented in avcodec.h). Do you disagree? >>>>> I would have expected that after a packet is received, a new frame >>>>> is enqueued instead of executing again receive_packet (why do we >>>>> that? what is the benefit?) >>>>> under these circumsntances I can't block in receive_packet blindly, >>>>> I have to track in user-space what the driver has in its guts and >>>>> predict how much it needs to keep working....I dont think it is a >>>>> good idea. >>>> Feel free to submit a patch to avcodec.h which changes the definition >>>> of the API. >>> no, that is ok. I can work around it easily (maybe v4l2 has special >>> needs compared to the rest of ffmpeg) >>> >>>> >>>>>> I think that the problem is that you are only polling on the frame >>>>>> buffer queue when blocking, so you don't see the packet buffers >>>>>> becoming free in the packet buffer queue - if you did see and >>>>>> dequeue them, you would then return EAGAIN to indicate that more >>>>>> input is needed. (See comments in >>>>>> <e4c6a8d7-798a-dfdb-b748-42936e944829@jkqxz.net>.) >>>>> I could manually track it that way with additional counters - so >>>>> before blocking I could see count many frames are enqueued in the >>>>> input and if there is not at least one I could just return EAGAIN. >>>>> but the behaviour differs from encoding. >>>> The behaviour is identical. You return the output buffer if there is >>>> one available on the output queue, otherwise you return EAGAIN if >>>> there is any space on the input queue, otherwise you block waiting >>>> for either of those to become possible. >>> >> >> I made some modifications to the way I was dequeuing buffers so now it >> polls for both input and output queues. you were right, things work much >> smoother this way...thanks. >> >> having said that I still I find the encoding API wrongly defined (no >> need for two calls that are in fact highly couple to each other in my >> opinion) > The encoding API works exactly the same way as the decoding API, just > that the decoding API has an helper internally that makes it easier to > merge send/receive into 1 function. You could do the same on the > encoding side. I see. But doesnt the helper makes the API confusing? I quite liked the way Mark described the operation: " You return the output buffer if there is one available on the output queue, otherwise you return EAGAIN if there is any space on the input queue, otherwise you block waiting for either of those to become possible." I now think that having a single function "muds" the behavioral definition above since we can just queue many packets on input and then be sure that on each dequeue we will be given the chance to enqueue again _even_ if there is another processed buffer ready to be read. > > Other than that, the API is meant for single-threaded use, so we can't > do things like randomly blocking calls and requiring reentrant calls > from other threads to unblock them. ok, understood. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
diff --git a/Changelog b/Changelog index 8309417..c6fcda3 100644 --- a/Changelog +++ b/Changelog @@ -40,6 +40,7 @@ version <next>: They must always be used by name. - FITS demuxer and decoder - FITS muxer and encoder +- V4L2 mem2mem HW accelerated codecs support version 3.3: - CrystalHD decoder moved to new decode API diff --git a/configure b/configure index 4f1c172..d9244e7 100755 --- a/configure +++ b/configure @@ -149,6 +149,7 @@ Component options: --disable-pixelutils disable pixel utils in libavutil Individual component options: + --disable-v4l2_m2m disable V4L2 mem2mem code [autodetect] --disable-everything disable all components listed below --disable-encoder=NAME disable encoder NAME --enable-encoder=NAME enable encoder NAME @@ -1433,6 +1434,7 @@ AVCODEC_COMPONENTS=" AVDEVICE_COMPONENTS=" indevs + v4l2_m2m outdevs " AVFILTER_COMPONENTS=" @@ -2271,6 +2273,7 @@ map 'eval ${v}_inline_deps=inline_asm' $ARCH_EXT_LIST_ARM loongson2_deps="mips" loongson3_deps="mips" +v4l2_deps_any="linux_videodev2_h" mipsfpu_deps="mips" mipsdsp_deps="mips" mipsdspr2_deps="mips" @@ -2743,6 +2746,8 @@ nvenc_deps="cuda" nvenc_deps_any="dlopen LoadLibrary" nvenc_encoder_deps="nvenc" +h263_v4l2m2m_decoder_deps="v4l2_m2m h263_v4l2_m2m" +h263_v4l2m2m_encoder_deps="v4l2_m2m h263_v4l2_m2m" h264_crystalhd_decoder_select="crystalhd h264_mp4toannexb_bsf h264_parser" h264_cuvid_decoder_deps="cuda cuvid" h264_cuvid_decoder_select="h264_mp4toannexb_bsf" @@ -2761,6 +2766,8 @@ h264_vda_decoder_deps="vda" h264_vda_decoder_select="h264_decoder" h264_vdpau_decoder_deps="vdpau" h264_vdpau_decoder_select="h264_decoder" +h264_v4l2m2m_decoder_deps="v4l2_m2m h264_v4l2_m2m" +h264_v4l2m2m_encoder_deps="v4l2_m2m h264_v4l2_m2m" hevc_cuvid_decoder_deps="cuda cuvid" hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf" hevc_mediacodec_decoder_deps="mediacodec" @@ -2772,12 +2779,15 @@ hevc_qsv_encoder_deps="libmfx" hevc_qsv_encoder_select="hevcparse qsvenc" hevc_vaapi_encoder_deps="VAEncPictureParameterBufferHEVC" hevc_vaapi_encoder_select="vaapi_encode golomb" +hevc_v4l2m2m_decoder_deps="v4l2_m2m hevc_v4l2_m2m" +hevc_v4l2m2m_encoder_deps="v4l2_m2m hevc_v4l2_m2m" mjpeg_cuvid_decoder_deps="cuda cuvid" mjpeg_vaapi_encoder_deps="VAEncPictureParameterBufferJPEG" mjpeg_vaapi_encoder_select="vaapi_encode jpegtables" mpeg1_cuvid_decoder_deps="cuda cuvid" mpeg1_vdpau_decoder_deps="vdpau" mpeg1_vdpau_decoder_select="mpeg1video_decoder" +mpeg1_v4l2m2m_decoder_deps="v4l2_m2m mpeg1_v4l2_m2m" mpeg2_crystalhd_decoder_select="crystalhd" mpeg2_cuvid_decoder_deps="cuda cuvid" mpeg2_mmal_decoder_deps="mmal" @@ -2788,6 +2798,7 @@ mpeg2_qsv_encoder_deps="libmfx" mpeg2_qsv_encoder_select="qsvenc" mpeg2_vaapi_encoder_deps="VAEncPictureParameterBufferMPEG2" mpeg2_vaapi_encoder_select="vaapi_encode" +mpeg2_v4l2m2m_decoder_deps="v4l2_m2m mpeg2_v4l2_m2m" mpeg4_crystalhd_decoder_select="crystalhd" mpeg4_cuvid_decoder_deps="cuda cuvid" mpeg4_mediacodec_decoder_deps="mediacodec" @@ -2795,6 +2806,8 @@ mpeg4_mmal_decoder_deps="mmal" mpeg4_omx_encoder_deps="omx" mpeg4_vdpau_decoder_deps="vdpau" mpeg4_vdpau_decoder_select="mpeg4_decoder" +mpeg4_v4l2m2m_decoder_deps="v4l2_m2m mpeg4_v4l2_m2m" +mpeg4_v4l2m2m_encoder_deps="v4l2_m2m mpeg4_v4l2_m2m" mpeg_vdpau_decoder_deps="vdpau" mpeg_vdpau_decoder_select="mpeg2video_decoder" msmpeg4_crystalhd_decoder_select="crystalhd" @@ -2805,16 +2818,20 @@ vc1_cuvid_decoder_deps="cuda cuvid" vc1_mmal_decoder_deps="mmal" vc1_vdpau_decoder_deps="vdpau" vc1_vdpau_decoder_select="vc1_decoder" +vc1_v4l2m2m_decoder_deps="v4l2_m2m vc1_v4l2_m2m" vp8_cuvid_decoder_deps="cuda cuvid" vp8_mediacodec_decoder_deps="mediacodec" vp8_qsv_decoder_deps="libmfx" vp8_qsv_decoder_select="qsvdec vp8_qsv_hwaccel vp8_parser" vp8_vaapi_encoder_deps="VAEncPictureParameterBufferVP8" vp8_vaapi_encoder_select="vaapi_encode" +vp8_v4l2m2m_decoder_deps="v4l2_m2m vp8_v4l2_m2m" +vp8_v4l2m2m_encoder_deps="v4l2_m2m vp8_v4l2_m2m" vp9_cuvid_decoder_deps="cuda cuvid" vp9_mediacodec_decoder_deps="mediacodec" vp9_vaapi_encoder_deps="VAEncPictureParameterBufferVP9" vp9_vaapi_encoder_select="vaapi_encode" +vp9_v4l2m2m_decoder_deps="v4l2_m2m vp9_v4l2_m2m" wmv3_crystalhd_decoder_select="crystalhd" wmv3_vdpau_decoder_select="vc1_vdpau_decoder" @@ -3597,7 +3614,7 @@ done enable_weak audiotoolbox # Enable hwaccels by default. -enable_weak d3d11va dxva2 vaapi vda vdpau videotoolbox_hwaccel xvmc +enable_weak d3d11va dxva2 vaapi v4l2_m2m vda vdpau videotoolbox_hwaccel xvmc enable_weak xlib enable_weak cuda cuvid nvenc vda_framework videotoolbox videotoolbox_encoder @@ -6066,10 +6083,21 @@ perl -v > /dev/null 2>&1 && enable perl || disable perl pod2man --help > /dev/null 2>&1 && enable pod2man || disable pod2man rsync --help 2> /dev/null | grep -q 'contimeout' && enable rsync_contimeout || disable rsync_contimeout +# check V4L2 codecs available in the API check_header linux/fb.h check_header linux/videodev.h check_header linux/videodev2.h check_code cc linux/videodev2.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete +check_code cc linux/videodev2.h "int i = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_VIDEO_M2M | V4L2_BUF_FLAG_LAST;" || disable v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VC1_ANNEX_G;" && enable vc1_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG1;" && enable mpeg1_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG2;" && enable mpeg2_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG4;" && enable mpeg4_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_HEVC;" && enable hevc_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H263;" && enable h263_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H264;" && enable h264_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP8;" && enable vp8_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP9;" && enable vp9_v4l2_m2m check_header sys/videoio.h check_code cc sys/videoio.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 999632c..6e1de37 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -137,6 +137,7 @@ OBJS-$(CONFIG_VIDEODSP) += videodsp.o OBJS-$(CONFIG_VP3DSP) += vp3dsp.o OBJS-$(CONFIG_VP56DSP) += vp56dsp.o OBJS-$(CONFIG_VP8DSP) += vp8dsp.o +OBJS-$(CONFIG_V4L2_M2M) += v4l2_m2m.o v4l2_buffers.o v4l2_fmt.o OBJS-$(CONFIG_WMA_FREQS) += wma_freqs.o OBJS-$(CONFIG_WMV2DSP) += wmv2dsp.o @@ -323,6 +324,8 @@ OBJS-$(CONFIG_H263_DECODER) += h263dec.o h263.o ituh263dec.o \ intelh263dec.o h263data.o OBJS-$(CONFIG_H263_ENCODER) += mpeg4videoenc.o mpeg4video.o \ h263.o ituh263enc.o flvenc.o h263data.o +OBJS-$(CONFIG_H263_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_H263_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_H264_DECODER) += h264dec.o h264_cabac.o h264_cavlc.o \ h264_direct.o h264_loopfilter.o \ h264_mb.o h264_picture.o \ @@ -340,6 +343,8 @@ OBJS-$(CONFIG_H264_QSV_DECODER) += qsvdec_h2645.o OBJS-$(CONFIG_H264_QSV_ENCODER) += qsvenc_h264.o OBJS-$(CONFIG_H264_VAAPI_ENCODER) += vaapi_encode_h264.o vaapi_encode_h26x.o OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o +OBJS-$(CONFIG_H264_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_H264_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_HAP_DECODER) += hapdec.o hap.o OBJS-$(CONFIG_HAP_ENCODER) += hapenc.o hap.o OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o hevc_mvs.o \ @@ -353,6 +358,8 @@ OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec_h2645.o OBJS-$(CONFIG_HEVC_QSV_ENCODER) += qsvenc_hevc.o hevc_ps_enc.o \ hevc_data.o OBJS-$(CONFIG_HEVC_VAAPI_ENCODER) += vaapi_encode_h265.o vaapi_encode_h26x.o +OBJS-$(CONFIG_HEVC_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_HEVC_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_HNM4_VIDEO_DECODER) += hnm4video.o OBJS-$(CONFIG_HQ_HQA_DECODER) += hq_hqa.o hq_hqadata.o hq_hqadsp.o \ canopus.o @@ -422,6 +429,7 @@ OBJS-$(CONFIG_MPC8_DECODER) += mpc8.o mpc.o OBJS-$(CONFIG_MPEGVIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o OBJS-$(CONFIG_MPEG1VIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o OBJS-$(CONFIG_MPEG1VIDEO_ENCODER) += mpeg12enc.o mpeg12.o +OBJS-$(CONFIG_MPEG1_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_MPEG2_MMAL_DECODER) += mmaldec.o OBJS-$(CONFIG_MPEG2_QSV_DECODER) += qsvdec_other.o OBJS-$(CONFIG_MPEG2_QSV_ENCODER) += qsvenc_mpeg2.o @@ -429,9 +437,12 @@ OBJS-$(CONFIG_MPEG2VIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o OBJS-$(CONFIG_MPEG2VIDEO_ENCODER) += mpeg12enc.o mpeg12.o OBJS-$(CONFIG_MPEG2_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_MPEG2_VAAPI_ENCODER) += vaapi_encode_mpeg2.o +OBJS-$(CONFIG_MPEG2_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_MPEG4_DECODER) += xvididct.o OBJS-$(CONFIG_MPEG4_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_MPEG4_OMX_ENCODER) += omx.o +OBJS-$(CONFIG_MPEG4_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_MPEG4_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_MPL2_DECODER) += mpl2dec.o ass.o OBJS-$(CONFIG_MSA1_DECODER) += mss3.o OBJS-$(CONFIG_MSCC_DECODER) += mscc.o @@ -605,6 +616,7 @@ OBJS-$(CONFIG_VC1_DECODER) += vc1dec.o vc1_block.o vc1_loopfilter.o OBJS-$(CONFIG_VC1_CUVID_DECODER) += cuvid.o OBJS-$(CONFIG_VC1_MMAL_DECODER) += mmaldec.o OBJS-$(CONFIG_VC1_QSV_DECODER) += qsvdec_other.o +OBJS-$(CONFIG_VC1_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_VC2_ENCODER) += vc2enc.o vc2enc_dwt.o diractab.o OBJS-$(CONFIG_VCR1_DECODER) += vcr1.o OBJS-$(CONFIG_VMDAUDIO_DECODER) += vmdaudio.o @@ -614,6 +626,7 @@ OBJS-$(CONFIG_VORBIS_DECODER) += vorbisdec.o vorbisdsp.o vorbis.o \ vorbis_data.o OBJS-$(CONFIG_VORBIS_ENCODER) += vorbisenc.o vorbis.o \ vorbis_data.o +OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o OBJS-$(CONFIG_VP3_DECODER) += vp3.o OBJS-$(CONFIG_VP5_DECODER) += vp5.o vp56.o vp56data.o vp56rac.o OBJS-$(CONFIG_VP6_DECODER) += vp6.o vp56.o vp56data.o \ @@ -624,6 +637,8 @@ OBJS-$(CONFIG_VP8_CUVID_DECODER) += cuvid.o OBJS-$(CONFIG_VP8_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_VP8_QSV_DECODER) += qsvdec_other.o OBJS-$(CONFIG_VP8_VAAPI_ENCODER) += vaapi_encode_vp8.o +OBJS-$(CONFIG_VP8_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_VP8_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_VP9_DECODER) += vp9.o vp9data.o vp9dsp.o vp9lpf.o vp9recon.o \ vp9block.o vp9prob.o vp9mvs.o vp56rac.o \ vp9dsp_8bpp.o vp9dsp_10bpp.o vp9dsp_12bpp.o @@ -631,6 +646,7 @@ OBJS-$(CONFIG_VP9_CUVID_DECODER) += cuvid.o OBJS-$(CONFIG_VP9_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_VP9_VAAPI_ENCODER) += vaapi_encode_vp9.o OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o +OBJS-$(CONFIG_VP9_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_VQA_DECODER) += vqavideo.o OBJS-$(CONFIG_WAVPACK_DECODER) += wavpack.o OBJS-$(CONFIG_WAVPACK_ENCODER) += wavpackenc.o diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c index ce0bc7e..eea1ae1 100644 --- a/libavcodec/allcodecs.c +++ b/libavcodec/allcodecs.c @@ -208,8 +208,10 @@ static void register_all(void) REGISTER_ENCDEC (H263, h263); REGISTER_DECODER(H263I, h263i); REGISTER_ENCDEC (H263P, h263p); + REGISTER_ENCDEC (H263_V4L2M2M, h263_v4l2m2m); REGISTER_DECODER(H264, h264); REGISTER_DECODER(H264_CRYSTALHD, h264_crystalhd); + REGISTER_ENCDEC (H264_V4L2M2M, h264_v4l2m2m); REGISTER_DECODER(H264_MEDIACODEC, h264_mediacodec); REGISTER_DECODER(H264_MMAL, h264_mmal); REGISTER_DECODER(H264_QSV, h264_qsv); @@ -220,6 +222,7 @@ static void register_all(void) REGISTER_ENCDEC (HAP, hap); REGISTER_DECODER(HEVC, hevc); REGISTER_DECODER(HEVC_QSV, hevc_qsv); + REGISTER_ENCDEC(HEVC_V4L2M2M, hevc_v4l2m2m); REGISTER_DECODER(HNM4_VIDEO, hnm4_video); REGISTER_DECODER(HQ_HQA, hq_hqa); REGISTER_DECODER(HQX, hqx); @@ -254,6 +257,7 @@ static void register_all(void) REGISTER_ENCDEC (MPEG2VIDEO, mpeg2video); REGISTER_ENCDEC (MPEG4, mpeg4); REGISTER_DECODER(MPEG4_CRYSTALHD, mpeg4_crystalhd); + REGISTER_ENCDEC (MPEG4_V4L2M2M, mpeg4_v4l2m2m); REGISTER_DECODER(MPEG4_MMAL, mpeg4_mmal); #if FF_API_VDPAU REGISTER_DECODER(MPEG4_VDPAU, mpeg4_vdpau); @@ -263,8 +267,10 @@ static void register_all(void) REGISTER_DECODER(MPEG_VDPAU, mpeg_vdpau); REGISTER_DECODER(MPEG1_VDPAU, mpeg1_vdpau); #endif + REGISTER_DECODER(MPEG1_V4L2M2M, mpeg1_v4l2m2m); REGISTER_DECODER(MPEG2_MMAL, mpeg2_mmal); REGISTER_DECODER(MPEG2_CRYSTALHD, mpeg2_crystalhd); + REGISTER_DECODER(MPEG2_V4L2M2M, mpeg2_v4l2m2m); REGISTER_DECODER(MPEG2_QSV, mpeg2_qsv); REGISTER_DECODER(MPEG2_MEDIACODEC, mpeg2_mediacodec); REGISTER_DECODER(MSA1, msa1); @@ -362,6 +368,7 @@ static void register_all(void) REGISTER_DECODER(VC1IMAGE, vc1image); REGISTER_DECODER(VC1_MMAL, vc1_mmal); REGISTER_DECODER(VC1_QSV, vc1_qsv); + REGISTER_DECODER(VC1_V4L2M2M, vc1_v4l2m2m); REGISTER_ENCODER(VC2, vc2); REGISTER_DECODER(VCR1, vcr1); REGISTER_DECODER(VMDVIDEO, vmdvideo); @@ -373,7 +380,9 @@ static void register_all(void) REGISTER_DECODER(VP6F, vp6f); REGISTER_DECODER(VP7, vp7); REGISTER_DECODER(VP8, vp8); + REGISTER_ENCDEC (VP8_V4L2M2M, vp8_v4l2m2m); REGISTER_DECODER(VP9, vp9); + REGISTER_DECODER(VP9_V4L2M2M, vp9_v4l2m2m); REGISTER_DECODER(VQA, vqa); REGISTER_DECODER(BITPACKED, bitpacked); REGISTER_DECODER(WEBP, webp); diff --git a/libavcodec/v4l2_buffers.c b/libavcodec/v4l2_buffers.c new file mode 100644 index 0000000..c3c0d03 --- /dev/null +++ b/libavcodec/v4l2_buffers.c @@ -0,0 +1,918 @@ +/* + * V4L2 buffer helper functions. + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <linux/videodev2.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <unistd.h> +#include <fcntl.h> +#include <poll.h> +#include "libavcodec/avcodec.h" +#include "libavcodec/internal.h" +#include "v4l2_buffers.h" +#include "v4l2_m2m.h" + +#define USEC_PER_SEC 1000000 + +enum V4L2Buffer_status { + V4L2BUF_AVAILABLE, + V4L2BUF_IN_DRIVER, + V4L2BUF_RET_USER, +}; + +/* buffer transform */ +typedef int (*pkt_to_buf_f)(const AVPacket *, V4L2Buffer *); +typedef int (*frm_to_buf_f)(const AVFrame *, V4L2Buffer *); +typedef int (*buf_to_pkt_f)(AVPacket *, V4L2Buffer *); +typedef int (*buf_to_frm_f)(AVFrame *, V4L2Buffer *); + +typedef int (*buf_to_bufref_f)(V4L2Buffer *in, int plane, AVBufferRef **buf); +typedef int (*bufref_to_buf_f)(V4L2Buffer *out, int plane, const uint8_t* data, int size, AVBufferRef* bref); + +struct V4L2Buffer_ops { + pkt_to_buf_f pkt_to_buf; + frm_to_buf_f frm_to_buf; + buf_to_pkt_f buf_to_pkt; + buf_to_frm_f buf_to_frm; + + bufref_to_buf_f bufref_to_buf; + buf_to_bufref_f buf_to_bufref; +}; + +struct V4L2Buffer { + /* each buffer needs to have a reference to its context */ + struct V4L2Context *context; + + struct V4L2Plane_info { + void * mm_addr; + size_t lengths; + } plane_info[VIDEO_MAX_PLANES]; + + /* some common buffer operations */ + struct V4L2Buffer_ops ops; + + /* memcpy to the v4l2_buffer planes array when needed */ + struct v4l2_plane planes[VIDEO_MAX_PLANES]; + struct v4l2_buffer buf; + + int bytesperline[4]; + int num_planes; + + int flags; + enum V4L2Buffer_status status; + +}; + +static inline int buffer_type_supported(V4L2Context *ctx) +{ + return ctx->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE || + ctx->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE || + ctx->type == V4L2_BUF_TYPE_VIDEO_CAPTURE || + ctx->type == V4L2_BUF_TYPE_VIDEO_OUTPUT; +} + +static inline unsigned int get_width(V4L2Context *ctx, struct v4l2_format *fmt) +{ + return V4L2_TYPE_IS_MULTIPLANAR(ctx->type) ? fmt->fmt.pix_mp.width : fmt->fmt.pix.width; +} + +static inline unsigned int get_height(V4L2Context *ctx, struct v4l2_format *fmt) +{ + return V4L2_TYPE_IS_MULTIPLANAR(ctx->type) ? fmt->fmt.pix_mp.height : fmt->fmt.pix.height; +} + +static inline enum AVColorSpace get_colorspace(V4L2Buffer *buf) +{ + enum v4l2_ycbcr_encoding ycbcr; + enum v4l2_colorspace cs; + + cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.colorspace : + buf->context->format.fmt.pix.colorspace; + + ycbcr = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.ycbcr_enc: + buf->context->format.fmt.pix.ycbcr_enc; + + switch(cs) { + case V4L2_COLORSPACE_SRGB: return AVCOL_SPC_RGB; + case V4L2_COLORSPACE_REC709: return AVCOL_SPC_BT709; + case V4L2_COLORSPACE_470_SYSTEM_M: return AVCOL_SPC_FCC; + case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_SPC_BT470BG; + case V4L2_COLORSPACE_SMPTE170M: return AVCOL_SPC_SMPTE170M; + case V4L2_COLORSPACE_SMPTE240M: return AVCOL_SPC_SMPTE240M; + case V4L2_COLORSPACE_BT2020: + if (ycbcr == V4L2_YCBCR_ENC_BT2020_CONST_LUM) + return AVCOL_SPC_BT2020_CL; + else + return AVCOL_SPC_BT2020_NCL; + default: + break; + } + + return AVCOL_SPC_UNSPECIFIED; +} + +static inline enum AVColorPrimaries get_colorprimaries(V4L2Buffer *buf) +{ + enum v4l2_ycbcr_encoding ycbcr; + enum v4l2_colorspace cs; + + cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.colorspace : + buf->context->format.fmt.pix.colorspace; + + ycbcr = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.ycbcr_enc: + buf->context->format.fmt.pix.ycbcr_enc; + + switch(ycbcr) { + case V4L2_YCBCR_ENC_XV709: + case V4L2_YCBCR_ENC_709: return AVCOL_PRI_BT709; + case V4L2_YCBCR_ENC_XV601: + case V4L2_YCBCR_ENC_601:return AVCOL_PRI_BT470M; + default: + break; + } + + switch(cs) { + case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_PRI_BT470BG; + case V4L2_COLORSPACE_SMPTE170M: return AVCOL_PRI_SMPTE170M; + case V4L2_COLORSPACE_SMPTE240M: return AVCOL_PRI_SMPTE240M; + case V4L2_COLORSPACE_BT2020: return AVCOL_PRI_BT2020; + default: + break; + } + + return AVCOL_PRI_UNSPECIFIED; +} + +static inline enum AVColorTransferCharacteristic get_colortrc(V4L2Buffer *buf) +{ + enum v4l2_ycbcr_encoding ycbcr; + enum v4l2_xfer_func xfer; + enum v4l2_colorspace cs; + + cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.colorspace : + buf->context->format.fmt.pix.colorspace; + + ycbcr = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.ycbcr_enc: + buf->context->format.fmt.pix.ycbcr_enc; + + xfer = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.xfer_func: + buf->context->format.fmt.pix.xfer_func; + + switch (xfer) { + case V4L2_XFER_FUNC_709: return AVCOL_TRC_BT709; + case V4L2_XFER_FUNC_SRGB: return AVCOL_TRC_IEC61966_2_1; + default: + break; + } + + switch (cs) { + case V4L2_COLORSPACE_470_SYSTEM_M: return AVCOL_TRC_GAMMA22; + case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_TRC_GAMMA28; + case V4L2_COLORSPACE_SMPTE170M: return AVCOL_TRC_SMPTE170M; + case V4L2_COLORSPACE_SMPTE240M: return AVCOL_TRC_SMPTE240M; + default: + break; + } + + switch (ycbcr) { + case V4L2_YCBCR_ENC_XV709: + case V4L2_YCBCR_ENC_XV601: return AVCOL_TRC_BT1361_ECG; + default: + break; + } + + return AVCOL_TRC_UNSPECIFIED; +} + +static inline enum AVColorPrimaries get_colorrange(V4L2Buffer *buf) +{ + enum v4l2_quantization qt; + + qt = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? + buf->context->format.fmt.pix_mp.quantization : + buf->context->format.fmt.pix.quantization; + + switch (qt) { + case V4L2_QUANTIZATION_LIM_RANGE: return AVCOL_RANGE_MPEG; + case V4L2_QUANTIZATION_FULL_RANGE: return AVCOL_RANGE_JPEG; + default: + break; + } + + return AVCOL_RANGE_UNSPECIFIED; +} + +static inline void set_pts(V4L2Buffer *out, int64_t pts) +{ + V4L2m2mContext *s = container_of(out->context, V4L2m2mContext, output); + AVRational v4l2_timebase = { 1, USEC_PER_SEC }; + int64_t v4l2_pts; + + if (pts == AV_NOPTS_VALUE) + pts = 0; + + /* convert pts to v4l2 timebase */ + v4l2_pts = av_rescale_q(pts, s->avctx->time_base, v4l2_timebase); + out->buf.timestamp.tv_usec = v4l2_pts % USEC_PER_SEC; + out->buf.timestamp.tv_sec = v4l2_pts / USEC_PER_SEC; +} + +static inline uint64_t get_pts(V4L2Buffer *avbuf) +{ + V4L2m2mContext *s = container_of(avbuf->context, V4L2m2mContext, capture); + AVRational v4l2_timebase = { 1, USEC_PER_SEC }; + int64_t v4l2_pts; + + /* convert pts back to encoder timebase */ + v4l2_pts = avbuf->buf.timestamp.tv_sec * USEC_PER_SEC + avbuf->buf.timestamp.tv_usec; + + return av_rescale_q(v4l2_pts, v4l2_timebase, s->avctx->time_base); +} + +static void free_v4l2buf_cb(void *opaque, uint8_t *unused) +{ + V4L2Buffer* avbuf = opaque; + V4L2m2mContext *s = container_of(avbuf->context, V4L2m2mContext, capture); + + atomic_fetch_sub_explicit(&s->refcount, 1, memory_order_acq_rel); + + if (s->reinit) { + if (!atomic_load(&s->refcount)) + sem_post(&s->refsync); + return; + } + + if (avbuf->context->streamon) { + avbuf->context->ops.enqueue(avbuf); + return; + } + + if (!atomic_load(&s->refcount)) + ff_v4l2_m2m_codec_end(s); +} + +/*** + Buffer Operations + */ +static int buffer_ops_bufref_to_v4l2buf(V4L2Buffer *out, int plane, const uint8_t* data, int size, AVBufferRef* bref) +{ + if (plane >= out->num_planes) + return AVERROR(EINVAL); + + memcpy(out->plane_info[plane].mm_addr, data, FFMIN(size, out->plane_info[plane].lengths)); + + out->planes[plane].bytesused = FFMIN(size, out->plane_info[plane].lengths); + out->planes[plane].length = out->plane_info[plane].lengths; + + return 0; +} + +static inline int buffer_ops_v4l2buf_to_bufref(V4L2Buffer *in, int plane, AVBufferRef **buf) +{ + V4L2m2mContext *s = container_of(in->context, V4L2m2mContext, capture); + + if (plane >= in->num_planes) + return AVERROR(EINVAL); + + /* even though most encoders return 0 in data_offset encoding vp8 does require this value */ + *buf = av_buffer_create((char *)in->plane_info[plane].mm_addr + in->planes[plane].data_offset, + in->plane_info[plane].lengths, free_v4l2buf_cb, in, 0); + if (!*buf) + return AVERROR(ENOMEM); + + in->status = V4L2BUF_RET_USER; + atomic_fetch_add_explicit(&s->refcount, 1, memory_order_relaxed); + + return 0; +} + +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) +{ + int i, ret; + + av_frame_unref(frame); + + /* 1. get references to the actual data */ + for (i = 0; i < avbuf->num_planes; i++) { + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); + if (ret) + return ret; + + frame->linesize[i] = avbuf->bytesperline[i]; + frame->data[i] = frame->buf[i]->data; + } + + /* 1.1 fixup special cases */ + switch (avbuf->context->av_pix_fmt) { + case AV_PIX_FMT_NV12: + if (avbuf->num_planes > 1) + break; + frame->linesize[1] = avbuf->bytesperline[0]; + frame->data[1] = frame->buf[0]->data + avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; + break; + default: + break; + } + + /* 2. get frame information */ + frame->key_frame = !!(avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME); + frame->format = avbuf->context->av_pix_fmt; + frame->color_primaries = get_colorprimaries(avbuf); + frame->colorspace = get_colorspace(avbuf); + frame->color_range = get_colorrange(avbuf); + frame->color_trc = get_colortrc(avbuf); + frame->pts = get_pts(avbuf); + /* these two values are updated also during re-init in process_video_event */ + frame->height = avbuf->context->height; + frame->width = avbuf->context->width; + + /* 3. report errors upstream */ + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: driver decode error\n", avbuf->context->name); + frame->decode_error_flags |= FF_DECODE_ERROR_INVALID_BITSTREAM; + } + + return 0; +} + +static int buffer_ops_avpkt_to_v4l2buf(const AVPacket *pkt, V4L2Buffer *out) +{ + int ret; + + ret = out->ops.bufref_to_buf(out, 0, pkt->data, pkt->size, pkt->buf); + if (ret) + return ret; + + set_pts(out, pkt->pts); + + if (pkt->flags & AV_PKT_FLAG_KEY) + out->flags = V4L2_BUF_FLAG_KEYFRAME; + + return 0; +} + +static int buffer_ops_avframe_to_v4l2buf(const AVFrame *frame, V4L2Buffer* out) +{ + int i, ret; + + for(i = 0; i < out->num_planes; i++) { + ret = out->ops.bufref_to_buf(out, i, frame->buf[i]->data, frame->buf[i]->size, frame->buf[i]); + if (ret) + return ret; + } + + set_pts(out, frame->pts); + + return 0; +} + +static int buffer_ops_v4l2buf_to_avpkt(AVPacket *pkt, V4L2Buffer *avbuf) +{ + int ret; + + av_packet_unref(pkt); + ret = avbuf->ops.buf_to_bufref(avbuf, 0, &pkt->buf); + if (ret) + return ret; + + pkt->size = V4L2_TYPE_IS_MULTIPLANAR(avbuf->context->type) ? avbuf->buf.m.planes[0].bytesused : avbuf->buf.bytesused; + pkt->data = pkt->buf->data; + + if (avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME) + pkt->flags |= AV_PKT_FLAG_KEY; + + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s driver encode error\n", avbuf->context->name); + pkt->flags |= AV_PKT_FLAG_CORRUPT; + } + + pkt->dts = pkt->pts = get_pts(avbuf); + + return 0; +} + +/*** + Context Operations + */ +static int context_ops_stop_decode(V4L2Context *ctx) +{ + struct v4l2_decoder_cmd cmd = { + .cmd = V4L2_DEC_CMD_STOP, + }; + int ret; + + ret = ioctl(ctx->fd, VIDIOC_DECODER_CMD, &cmd); + if (ret) { + /* DECODER_CMD is optional */ + if (errno == ENOTTY) + return ff_v4l2_context_set_status(ctx, VIDIOC_STREAMOFF); + } + + return 0; +} + +static int context_ops_stop_encode(V4L2Context *ctx) +{ + struct v4l2_encoder_cmd cmd = { + .cmd = V4L2_ENC_CMD_STOP, + }; + int ret; + + ret = ioctl(ctx->fd, VIDIOC_ENCODER_CMD, &cmd); + if (ret) { + /* ENCODER_CMD is optional */ + if (errno == ENOTTY) + return ff_v4l2_context_set_status(ctx, VIDIOC_STREAMOFF); + } + + return 0; +} + +static int context_ops_enqueue_v4l2buf(V4L2Buffer* avbuf) +{ + int ret; + + avbuf->buf.flags = avbuf->flags; + + ret = ioctl(avbuf->context->fd, VIDIOC_QBUF, &avbuf->buf); + if (ret < 0) + return AVERROR(errno); + + avbuf->status = V4L2BUF_IN_DRIVER; + + return 0; +} + +static int process_video_event(V4L2Context *ctx) +{ + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, capture); + struct v4l2_format cap_fmt = s->capture.format; + struct v4l2_format out_fmt = s->output.format; + struct v4l2_event evt; + int ret; + + ret = ioctl(ctx->fd, VIDIOC_DQEVENT, &evt); + if (ret < 0) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s VIDIOC_DQEVENT\n", ctx->name); + return 0; + } + + ret = ioctl(s->fd, VIDIOC_G_FMT, &cap_fmt); + if (ret) { + av_log(s->capture.log_ctx, AV_LOG_ERROR, "%s VIDIOC_G_FMT\n", s->capture.name); + return 0; + } + + ret = ioctl(s->fd, VIDIOC_G_FMT, &out_fmt); + if (ret) { + av_log(s->output.log_ctx, AV_LOG_ERROR, "%s VIDIOC_G_FMT\n", s->output.name); + return 0; + } + + if (evt.type != V4L2_EVENT_SOURCE_CHANGE) + return 0; + + if (get_height(&s->output, &s->output.format) != get_height(&s->output, &out_fmt) || + get_width(&s->output, &s->output.format) != get_width(&s->output, &out_fmt)) { + + av_log(s->output.log_ctx, AV_LOG_DEBUG, "%s changed (%dx%d) -> (%dx%d)\n", + s->output.name, + get_width(&s->output, &s->output.format), get_height(&s->output, &s->output.format), + get_width(&s->output, &out_fmt), get_height(&s->output, &out_fmt)); + + /* 0. update the output context */ + s->output.height = get_height(ctx, &out_fmt); + s->output.width = get_width(ctx, &out_fmt); + + /* 1. store the new dimensions in the capture context so the resulting frame + can be cropped */ + s->capture.height = get_height(ctx, &out_fmt); + s->capture.width = get_width(ctx, &out_fmt); + } + + if (get_height(&s->capture, &s->capture.format) != get_height(&s->capture, &cap_fmt) || + get_width(&s->capture, &s->capture.format) != get_width(&s->capture, &cap_fmt)) { + + av_log(s->capture.log_ctx, AV_LOG_DEBUG, "%s changed (%dx%d) -> (%dx%d)\n", + s->capture.name, + get_width(&s->capture, &s->capture.format), get_height(&s->capture, &s->capture.format), + get_width(&s->capture, &cap_fmt), get_height(&s->capture, &cap_fmt)); + + /* streamoff capture and unmap and remap new buffers */ + ret = ff_v4l2_m2m_codec_reinit(s); + if (ret) + av_log(ctx->log_ctx, AV_LOG_ERROR, "avpriv_v4l2m2m_reinit\n"); + + /* let the caller function know that reinit was executed */ + return 1; + } + + return 0; +} + +static V4L2Buffer* context_ops_dequeue_v4l2buf(V4L2Context *ctx, int timeout) +{ + struct v4l2_plane planes[VIDEO_MAX_PLANES]; + struct v4l2_buffer buf = { 0 }; + V4L2Buffer* avbuf = NULL; + struct pollfd pfd = { + .events = POLLIN | POLLRDNORM | POLLPRI, /* default capture context */ + .fd = ctx->fd, + }; + int ret; + + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) + pfd.events = POLLOUT | POLLWRNORM; + + for (;;) { + /* timeout: + * - capture: not enough buffers are queued to generate output + * - output: no more free buffers to reuse, keep going + */ + ret = poll(&pfd, 1, timeout); + if (ret > 0) + break; + if (errno == EINTR) + continue; + return NULL; + } + + /* 0. handle errors */ + if (pfd.revents & POLLERR) { + av_log(ctx->log_ctx, AV_LOG_WARNING, "%s POLLERR\n", ctx->name); + return NULL; + } + + /* 1. dequeue the buffer */ + if (pfd.revents & (POLLIN | POLLRDNORM) || pfd.revents & (POLLOUT | POLLWRNORM)) { + memset(&buf, 0, sizeof(buf)); + buf.memory = V4L2_MEMORY_MMAP; + buf.type = ctx->type; + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + memset(planes, 0, sizeof(planes)); + buf.length = VIDEO_MAX_PLANES; + buf.m.planes = planes; + } + + ret = ioctl(ctx->fd, VIDIOC_DQBUF, &buf); + if (ret) { + if (errno != EAGAIN) { + ctx->done = errno; + if (errno != EPIPE) + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s VIDIOC_DQBUF, errno (%s)\n", + ctx->name, av_err2str(AVERROR(errno))); + } + } else { + avbuf = &ctx->buffers[buf.index]; + avbuf->status = V4L2BUF_AVAILABLE; + avbuf->buf = buf; + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + memcpy(avbuf->planes, planes, sizeof(planes)); + avbuf->buf.m.planes = avbuf->planes; + } + } + } + + /* 2. handle resolution changes */ + if (pfd.revents & POLLPRI) { + ret = process_video_event(ctx); + if (ret) { + /* drop the buffer (if there was one) since we had to reconfigure capture (unmap all buffers) */ + return NULL; + } + } + + return avbuf; +} + +static V4L2Buffer* context_ops_getfree_v4l2buf(V4L2Context *ctx) +{ + int timeout = 0; /* return when no more buffers to dequeue */ + int i; + + /* get back as many output buffers as possible */ + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) { + do { + } while (ctx->ops.dequeue(ctx, timeout)); + } + + for (i = 0; i < ctx->num_buffers; i++) { + if (ctx->buffers[i].status == V4L2BUF_AVAILABLE) + return &ctx->buffers[i]; + } + + return NULL; +} + +static int context_ops_release_v4l2_buffers(V4L2Context* ctx) +{ + struct v4l2_requestbuffers req = { + .memory = V4L2_MEMORY_MMAP, + .type = ctx->type, + .count = 0, /* 0 unmaps buffers from the driver */ + }; + int i, j; + + + for (i = 0; i < ctx->num_buffers; i++) { + V4L2Buffer *buffer = &ctx->buffers[i]; + + for (j = 0; j < buffer->num_planes; j++) { + struct V4L2Plane_info *p = &buffer->plane_info[j]; + if (p->mm_addr && p->lengths) + if (munmap(p->mm_addr, p->lengths) < 0) + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s unmap plane (%s))\n", ctx->name, av_err2str(AVERROR(errno))); + } + } + + return ioctl(ctx->fd, VIDIOC_REQBUFS, &req); +} + +static int context_ops_initialize_v4l2buf(V4L2Context *ctx, V4L2Buffer* avbuf, int index) +{ + int ret, i; + + /* keep a reference to the context */ + avbuf->context = ctx; + + /* initalize the buffer operations */ + avbuf->ops.buf_to_frm = buffer_ops_v4l2buf_to_avframe; + avbuf->ops.frm_to_buf = buffer_ops_avframe_to_v4l2buf; + avbuf->ops.buf_to_pkt = buffer_ops_v4l2buf_to_avpkt; + avbuf->ops.pkt_to_buf = buffer_ops_avpkt_to_v4l2buf; + + avbuf->ops.bufref_to_buf = buffer_ops_bufref_to_v4l2buf; + avbuf->ops.buf_to_bufref = buffer_ops_v4l2buf_to_bufref; + + avbuf->buf.memory = V4L2_MEMORY_MMAP; + avbuf->buf.type = ctx->type; + avbuf->buf.index = index; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->buf.length = VIDEO_MAX_PLANES; + avbuf->buf.m.planes = avbuf->planes; + } + + ret = ioctl(ctx->fd, VIDIOC_QUERYBUF, &avbuf->buf); + if (ret < 0) + return AVERROR(errno); + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->num_planes = 0; + for (;;) { + /* in MP, the V4L2 API states that buf.length means num_planes */ + if (avbuf->num_planes >= avbuf->buf.length) + break; + if (avbuf->buf.m.planes[avbuf->num_planes].length) + avbuf->num_planes++; + } + } else + avbuf->num_planes = 1; + + for (i = 0; i < avbuf->num_planes; i++) { + + avbuf->bytesperline[i] = V4L2_TYPE_IS_MULTIPLANAR(ctx->type) ? + ctx->format.fmt.pix_mp.plane_fmt[i].bytesperline : + ctx->format.fmt.pix.bytesperline; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->plane_info[i].lengths = avbuf->buf.m.planes[i].length; + avbuf->plane_info[i].mm_addr = mmap(NULL, avbuf->buf.m.planes[i].length, + PROT_READ | PROT_WRITE, MAP_SHARED, + ctx->fd, avbuf->buf.m.planes[i].m.mem_offset); + } else { + avbuf->plane_info[i].lengths = avbuf->buf.length; + avbuf->plane_info[i].mm_addr = mmap(NULL, avbuf->buf.length, + PROT_READ | PROT_WRITE, MAP_SHARED, + ctx->fd, avbuf->buf.m.offset); + } + + if (avbuf->plane_info[i].mm_addr == MAP_FAILED) + return AVERROR(ENOMEM); + } + + avbuf->status = V4L2BUF_AVAILABLE; + + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) + return 0; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->buf.m.planes = avbuf->planes; + avbuf->buf.length = avbuf->num_planes; + + } else { + avbuf->buf.bytesused = avbuf->planes[index].bytesused; + avbuf->buf.length = avbuf->planes[index].length; + } + + return ctx->ops.enqueue(avbuf); +} + +int ff_v4l2_enqueue_frame(V4L2Context* ctx, const AVFrame* frame) +{ + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, output); + V4L2Buffer* avbuf; + int ret; + + if (!frame) { + ret = ctx->ops.stop_encode(ctx); + if (ret) + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s stop_encode\n", ctx->name); + s->draining= 1; + return 0; + } + + avbuf = ctx->ops.get_buffer(ctx); + if (!avbuf) + return AVERROR(ENOMEM); + + ret = avbuf->ops.frm_to_buf(frame, avbuf); + if (ret) + return ret; + + ret = ctx->ops.enqueue(avbuf); + if (ret) + return ret; + + return 0; +} + +int ff_v4l2_enqueue_packet(V4L2Context* ctx, const AVPacket* pkt) +{ + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, output); + V4L2Buffer* avbuf; + int ret; + + if (!pkt->size) { + ret = ctx->ops.stop_decode(ctx); + if (ret) + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s stop_decode\n", ctx->name); + s->draining = 1; + return 0; + } + + avbuf = ctx->ops.get_buffer(ctx); + if (!avbuf) + return AVERROR(ENOMEM); + + ret = avbuf->ops.pkt_to_buf(pkt, avbuf); + if (ret) + return ret; + + ret = ctx->ops.enqueue(avbuf); + if (ret) + return ret; + + return 0; +} + +int ff_v4l2_dequeue_frame(V4L2Context* ctx, AVFrame* frame, int timeout) +{ + V4L2Buffer* avbuf = NULL; + + avbuf = ctx->ops.dequeue(ctx, timeout); + if (!avbuf) { + if (ctx->done) + return AVERROR_EOF; + + return AVERROR(EAGAIN); + } + + return avbuf->ops.buf_to_frm(frame, avbuf); +} + +int ff_v4l2_dequeue_packet(V4L2Context* ctx, AVPacket* pkt, int timeout) +{ + V4L2Buffer* avbuf = NULL; + + avbuf = ctx->ops.dequeue(ctx, timeout); + if (!avbuf) { + if (ctx->done) + return AVERROR_EOF; + + return AVERROR(EAGAIN); + } + + return avbuf->ops.buf_to_pkt(pkt, avbuf); +} + +int ff_v4l2_context_set_status(V4L2Context* ctx, int cmd) +{ + int type = ctx->type; + int ret; + + ret = ioctl(ctx->fd, cmd, &type); + if (ret < 0) + return AVERROR(errno); + + ctx->streamon = (cmd == VIDIOC_STREAMON); + + return 0; +} + +void ff_v4l2_context_release(V4L2Context* ctx) +{ + int ret; + + if (!ctx->buffers) + return; + + ret = ctx->ops.release_buffers(ctx); + if (ret) + av_log(ctx->log_ctx, AV_LOG_WARNING, "V4L2 failed to unmap the %s buffers\n", ctx->name); + else + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s all buffers unmapped\n", ctx->name); + + av_free(ctx->buffers); + ctx->buffers = NULL; +} + +int ff_v4l2_context_init(V4L2Context* ctx, int lazy_init) +{ + struct v4l2_requestbuffers req; + struct v4l2_control ctrl; + int min_buffers = 6; + int ret, i; + + if (!buffer_type_supported(ctx)) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "type %i not supported\n", ctx->type); + return AVERROR_PATCHWELCOME; + } + + ctx->ops.release_buffers = context_ops_release_v4l2_buffers; + ctx->ops.init_buffer = context_ops_initialize_v4l2buf; + ctx->ops.get_buffer = context_ops_getfree_v4l2buf; + ctx->ops.dequeue = context_ops_dequeue_v4l2buf; + ctx->ops.enqueue = context_ops_enqueue_v4l2buf; + ctx->ops.stop_decode = context_ops_stop_decode; + ctx->ops.stop_encode = context_ops_stop_encode; + + if (lazy_init) + return 0; + + /* get the minimum number of buffers required by the hardware */ + memset(&ctrl, 0, sizeof(ctrl)); + ctrl.id = V4L2_TYPE_IS_OUTPUT(ctx->type) ? V4L2_CID_MIN_BUFFERS_FOR_OUTPUT : V4L2_CID_MIN_BUFFERS_FOR_CAPTURE; + ret = ioctl(ctx->fd, VIDIOC_G_CTRL, &ctrl); + if (!ret) + min_buffers = ctrl.value; + else { + av_log(ctx->log_ctx, AV_LOG_WARNING, "%s min_buffers to default %d\n", ctx->name, min_buffers); + } + + memset(&req, 0, sizeof(req)); + req.count = ctx->num_buffers + min_buffers; + req.memory = V4L2_MEMORY_MMAP; + req.type = ctx->type; + ret = ioctl(ctx->fd, VIDIOC_REQBUFS, &req); + if (ret < 0) + return AVERROR(errno); + + ctx->num_buffers = req.count; + ctx->buffers = av_mallocz(ctx->num_buffers * sizeof(V4L2Buffer)); + if (!ctx->buffers) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s buffer initialization ENOMEM\n", ctx->name); + return AVERROR(ENOMEM); + } + + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s queuing %d buffers\n", ctx->name, req.count); + for (i = 0; i < req.count; i++) { + ret = ctx->ops.init_buffer(ctx, &ctx->buffers[i], i); + if (ret < 0) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s buffer initialization (%s)\n", ctx->name, av_err2str(ret)); + av_free(ctx->buffers); + return ret; + } + } + + return 0; +} + + diff --git a/libavcodec/v4l2_buffers.h b/libavcodec/v4l2_buffers.h new file mode 100644 index 0000000..6123eb8 --- /dev/null +++ b/libavcodec/v4l2_buffers.h @@ -0,0 +1,236 @@ +/* + * V4L2 buffer{,context} helper functions. + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_V4L2_BUFFERS_H +#define AVCODEC_V4L2_BUFFERS_H + +#include <stdatomic.h> +#include "libavcodec/avcodec.h" +#include "libavutil/pixfmt.h" +#include "libavutil/frame.h" +#include "libavutil/buffer.h" + +struct V4L2Buffer; +typedef struct V4L2Buffer V4L2Buffer; + +struct V4L2Context; +typedef struct V4L2Context V4L2Context; + +/** + * V4L2Context_operations: + */ +typedef V4L2Buffer* (*v4l2_dequeue_buffer_f)(V4L2Context *ctx, int timeout); +typedef V4L2Buffer* (*v4l2_getfree_buffer_f)(V4L2Context *ctx); + +typedef int (*v4l2_initialize_buffer_f)(V4L2Context *ctx, V4L2Buffer* avbuf, int index); +typedef int (*v4l2_enqueue_buffer_f)(V4L2Buffer *buf); +typedef int (*v4l2_release_buffers_f)(V4L2Context *ctx); +typedef int (*v4l2_stop_decode_f)(V4L2Context *ctx); +typedef int (*v4l2_stop_encode_f)(V4L2Context *ctx); + +typedef struct V4L2Context_ops { + v4l2_stop_decode_f stop_decode; + v4l2_stop_encode_f stop_encode; + + /* operations on all buffers at once */ + v4l2_release_buffers_f release_buffers; + + /* operations on single buffers */ + v4l2_initialize_buffer_f init_buffer; + v4l2_getfree_buffer_f get_buffer; + v4l2_dequeue_buffer_f dequeue; + v4l2_enqueue_buffer_f enqueue; + +} V4L2Context_ops; + +typedef struct V4L2Context { + /** + * Buffer context operations + * queue a V4L2Buffer into the context + * dequeue a V4L2Buffer into the context + * get a free V4L2Buffer from the context + * release all V4L2Buffers allocated to the context + */ + V4L2Context_ops ops; + + /** + * Log context (for av_log()). Can be NULL. + */ + void *log_ctx; + + /** + * Lazy Initialization: set to one if the context can not initialize its + * buffers until it first queries the driver for formats and sizes. + */ + int lazy_init; + + /** + * context name: must be set before calling avpriv_v4l2_context_init(). + */ + const char* name; + + /** + * File descriptor obtained from opening the associated device. + * Must be set before calling avpriv_v4l2_context_init(). + * Readonly after init. + */ + int fd; + + /** + * Type of this buffer context. + * See V4L2_BUF_TYPE_VIDEO_* in videodev2.h + * Must be set before calling avpriv_v4l2_context_init(). + * Readonly after init. + */ + enum v4l2_buf_type type; + + /** + * AVPixelFormat corresponding to this buffer context. + * AV_PIX_FMT_NONE means this is an encoded stream. + */ + enum AVPixelFormat av_pix_fmt; + + /** + * AVCodecID corresponding to this buffer context. + * AV_CODEC_ID_RAWVIDEO means this is a raw stream and av_pix_fmt must be set to a valid value. + */ + enum AVCodecID av_codec_id; + + /** + * Format returned by the driver after initializing the buffer context. + * Must be set before calling avpriv_v4l2_context_init(). + * avpriv_v4l2m2m_format() can set it. + * Readonly after init. + */ + struct v4l2_format format; + + /** + * Width and height of the frames it produces (in case of a capture context, e.g. when decoding) + * or accepts (in case of an output context, e.g. when encoding). + * + * For output context, this must must be set before calling avpriv_v4l2_context_init(). + * For capture context during decoding, it will be set after having received the + * information from the driver. at which point we can initialize the buffers. + */ + int width, height; + + /** + * Whether the stream has been started (VIDIOC_STREAMON has been sent). + */ + int streamon; + + /** + * + * Before calling avpriv_v4l2_context_init() this is the number of buffers we would like to have available. + * avpriv_v4l2_context_init() asks for (min_buffers + num_buffers) and sets this value to the actual number + * of buffers the driver gave us. + * + * Readonly after init. + */ + int num_buffers; + + /** + * Indexed array of V4L2Buffers + */ + V4L2Buffer *buffers; + + /** + * Either no more buffers available or an unrecoverable error was notified + * by the V4L2 kernel driver: either way, we can't continue using this context.. + */ + int done; + +} V4L2Context; + +/** + * Initializes a V4L2Context. + * + * @param[in] ctx A pointer to a V4L2Context. See V4L2Context description for required variables. + * @return 0 in case of success, a negative value representing the error otherwise. + */ +int ff_v4l2_context_init(V4L2Context* ctx, int lazy_init); + +/** + * Releases a V4L2Context. + * + * @param[in] ctx A pointer to a V4L2Context. + * The caller is reponsible for freeing it. + * It must not be used after calling this function. + */ +void ff_v4l2_context_release(V4L2Context* ctx); + +/** + * Sets the status of a V4L2Context. + * + * @param[in] ctx A pointer to a V4L2Context. + * @param[in] cmd The status to set (VIDIOC_STREAMON or VIDIOC_STREAMOFF). + * Warning: If VIDIOC_STREAMOFF is sent to a buffer context that still has some frames buffered, + * those frames will be dropped. + * @return 0 in case of success, a negative value representing the error otherwise. + */ +int ff_v4l2_context_set_status(V4L2Context* ctx, int cmd); + +/** + * Dequeues a buffer from a V4L2Context to an AVPacket. + * + * The pkt must be non NULL. + * @param[in] ctx The V4L2Context to dequeue from. + * @param[inout] pkt The AVPacket to dequeue to. + * @param[ino] timeout The number of milliseconds to wait for the dequeue. + * @return 0 in case of success, AVERROR(EAGAIN) if no buffer was ready, another negative error in case of error. + */ +int ff_v4l2_dequeue_packet(V4L2Context* ctx, AVPacket* pkt, int timeout); + +/** + * Dequeues a buffer from a V4L2Context to an AVFrame. + * + * The frame must be non NULL. + * @param[in] ctx The V4L2Context to dequeue from. + * @param[inout] f The AVFrame to dequeue to. + * @param[ino] timeout The number of milliseconds to wait for the dequeue. + * @return 0 in case of success, AVERROR(EAGAIN) if no buffer was ready, another negative error in case of error. + */ +int ff_v4l2_dequeue_frame(V4L2Context* ctx, AVFrame* f, int timeout); + +/** + * Enqueues a buffer to a V4L2Context from an AVPacket + * The packet must be non NULL. + * When the size of the pkt is null, the buffer is not queued but a V4L2_DEC_CMD_STOP command is sent instead to the driver. + * + * @param[in] ctx The V4L2Context to enqueue to. + * @param[in] pkt A pointer to an AVPacket. + * @return 0 in case of success, a negative error otherwise. + */ +int ff_v4l2_enqueue_packet(V4L2Context* ctx, const AVPacket* pkt); + +/** + * Enqueues a buffer to a V4L2Context from an AVFrame + * The frame must be non NULL. + * + * @param[in] ctx The V4L2Context to enqueue to. + * @param[in] f A pointer to an AVFrame to enqueue. + * @return 0 in case of success, a negative error otherwise. + */ +int ff_v4l2_enqueue_frame(V4L2Context* ctx, const AVFrame* f); + +#endif // AVCODEC_V4L2_BUFFERS_H diff --git a/libavcodec/v4l2_fmt.c b/libavcodec/v4l2_fmt.c new file mode 100644 index 0000000..85fda7d --- /dev/null +++ b/libavcodec/v4l2_fmt.c @@ -0,0 +1,184 @@ +/* + * V4L2 format helper functions + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <linux/videodev2.h> +#include <search.h> +#include "v4l2_fmt.h" + +#define V4L2_FMT(x) V4L2_PIX_FMT_##x +#define AV_CODEC(x) AV_CODEC_ID_##x +#define AV_FMT(x) AV_PIX_FMT_##x + +static const struct fmt_conversion { + enum AVPixelFormat avfmt; + enum AVCodecID avcodec; + uint32_t v4l2_fmt; +} fmt_map[] = { + { AV_FMT(RGB555LE), AV_CODEC(RAWVIDEO), V4L2_FMT(RGB555) }, + { AV_FMT(RGB555BE), AV_CODEC(RAWVIDEO), V4L2_FMT(RGB555X) }, + { AV_FMT(RGB565LE), AV_CODEC(RAWVIDEO), V4L2_FMT(RGB565) }, + { AV_FMT(RGB565BE), AV_CODEC(RAWVIDEO), V4L2_FMT(RGB565X) }, + { AV_FMT(BGR24), AV_CODEC(RAWVIDEO), V4L2_FMT(BGR24) }, + { AV_FMT(RGB24), AV_CODEC(RAWVIDEO), V4L2_FMT(RGB24) }, + { AV_FMT(BGR0), AV_CODEC(RAWVIDEO), V4L2_FMT(BGR32) }, + { AV_FMT(0RGB), AV_CODEC(RAWVIDEO), V4L2_FMT(RGB32) }, + { AV_FMT(GRAY8), AV_CODEC(RAWVIDEO), V4L2_FMT(GREY) }, + { AV_FMT(YUV420P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV420) }, + { AV_FMT(YUYV422), AV_CODEC(RAWVIDEO), V4L2_FMT(YUYV) }, + { AV_FMT(UYVY422), AV_CODEC(RAWVIDEO), V4L2_FMT(UYVY) }, + { AV_FMT(YUV422P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV422P) }, + { AV_FMT(YUV411P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV411P) }, + { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV410) }, + { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YVU410) }, + { AV_FMT(NV12), AV_CODEC(RAWVIDEO), V4L2_FMT(NV12) }, + { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, + { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(JPEG) }, +#ifdef V4L2_PIX_FMT_SRGGB8 + { AV_FMT(BAYER_BGGR8), AV_CODEC(RAWVIDEO), V4L2_FMT(SBGGR8) }, + { AV_FMT(BAYER_GBRG8), AV_CODEC(RAWVIDEO), V4L2_FMT(SGBRG8) }, + { AV_FMT(BAYER_GRBG8), AV_CODEC(RAWVIDEO), V4L2_FMT(SGRBG8) }, + { AV_FMT(BAYER_RGGB8), AV_CODEC(RAWVIDEO), V4L2_FMT(SRGGB8) }, +#endif +#ifdef V4L2_PIX_FMT_Y16 + { AV_FMT(GRAY16LE), AV_CODEC(RAWVIDEO), V4L2_FMT(Y16) }, +#endif +#ifdef V4L2_PIX_FMT_NV12M + { AV_FMT(NV12), AV_CODEC(RAWVIDEO), V4L2_FMT(NV12M) }, +#endif +#ifdef V4L2_PIX_FMT_NV21M + { AV_FMT(NV21), AV_CODEC(RAWVIDEO), V4L2_FMT(NV21M) }, +#endif +#ifdef V4L2_PIX_FMT_YUV420M + { AV_FMT(YUV420P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV420M) }, +#endif +#ifdef V4L2_PIX_FMT_NV16M + { AV_FMT(NV16), AV_CODEC(RAWVIDEO), V4L2_FMT(NV16M) }, +#endif +#ifdef V4L2_PIX_FMT_H263 + { AV_FMT(NONE), AV_CODEC(H263), V4L2_FMT(H263) }, +#endif +#ifdef V4L2_PIX_FMT_H264 + { AV_FMT(NONE), AV_CODEC(H264), V4L2_FMT(H264) }, +#endif +#ifdef V4L2_PIX_FMT_MPEG4 + { AV_FMT(NONE), AV_CODEC(MPEG4), V4L2_FMT(MPEG4) }, +#endif +#ifdef V4L2_PIX_FMT_CPIA1 + { AV_FMT(NONE), AV_CODEC(CPIA), V4L2_FMT(CPIA1) }, +#endif +#ifdef V4L2_PIX_FMT_DV + { AV_FMT(NONE), AV_CODEC(DVVIDEO), V4L2_FMT(DV) }, +#endif +#ifdef V4L2_PIX_FMT_MPEG1 + { AV_FMT(NONE), AV_CODEC(MPEG1VIDEO), V4L2_FMT(MPEG1) }, +#endif +#ifdef V4L2_PIX_FMT_MPEG2 + { AV_FMT(NONE), AV_CODEC(MPEG2VIDEO), V4L2_FMT(MPEG2) }, +#endif +#ifdef V4L2_PIX_FMT_VP8 + { AV_FMT(NONE), AV_CODEC(VP8), V4L2_FMT(VP8) }, +#endif +#ifdef V4L2_PIX_FMT_VP9 + { AV_FMT(NONE), AV_CODEC(VP9), V4L2_FMT(VP9) }, +#endif +#ifdef V4L2_PIX_FMT_HEVC + { AV_FMT(NONE), AV_CODEC(HEVC), V4L2_FMT(HEVC) }, +#endif +#ifdef V4L2_PIX_FMT_VC1_ANNEX_G + { AV_FMT(NONE), AV_CODEC(VC1), V4L2_FMT(VC1_ANNEX_G) }, +#endif +}; + +static int match_codec(const void *a, const void *b) +{ + if (*(enum AVCodecID *)a == ((struct fmt_conversion *)b)->avcodec) + return 0; + + return 1; +} + +uint32_t ff_v4l2_avcodec_to_v4l2fmt(enum AVCodecID avcodec) +{ + size_t len = sizeof(fmt_map) / sizeof(fmt_map[0]); + struct fmt_conversion *item; + + item = lfind(&avcodec, fmt_map, &len, sizeof(fmt_map[0]), match_codec); + if (item) + return item->v4l2_fmt; + + return 0; +} + +static int match_fmt(const void *a, const void *b) +{ + if ( *(enum AVPixelFormat *)a == ((struct fmt_conversion *)b)->avfmt) + return 0; + + return 1; +} + +uint32_t ff_v4l2_avfmt_to_v4l2fmt(enum AVPixelFormat avfmt) +{ + size_t len = sizeof(fmt_map) / sizeof(fmt_map[0]); + struct fmt_conversion *item; + + item = lfind(&avfmt, fmt_map, &len, sizeof(fmt_map[0]), match_fmt); + if (item) + return item->v4l2_fmt; + + return 0; +} + +struct v4l2fmt_avcodec_pair { + enum AVCodecID avcodec; + uint32_t v4l2_fmt; +}; + +static int match_codecfmt(const void *a, const void *b) +{ + struct v4l2fmt_avcodec_pair *key = (struct v4l2fmt_avcodec_pair *) a; + struct fmt_conversion *item = (struct fmt_conversion *) b; + + if (key->avcodec == item->avcodec && key->v4l2_fmt == item->v4l2_fmt) + return 0; + + return 1; +} + +enum AVPixelFormat ff_v4l2_v4l2fmt_to_avfmt(uint32_t v4l2_fmt, enum AVCodecID avcodec) +{ + size_t len = sizeof(fmt_map) / sizeof(fmt_map[0]); + struct v4l2fmt_avcodec_pair const key = { + .v4l2_fmt = v4l2_fmt, + .avcodec = avcodec, + }; + struct fmt_conversion *item; + + item = lfind(&key, fmt_map, &len, sizeof(fmt_map[0]), match_codecfmt); + if (item) + return item->avfmt; + + return AV_PIX_FMT_NONE; +} + + diff --git a/libavcodec/v4l2_fmt.h b/libavcodec/v4l2_fmt.h new file mode 100644 index 0000000..e8323f1 --- /dev/null +++ b/libavcodec/v4l2_fmt.h @@ -0,0 +1,34 @@ +/* + * V4L2 format helper functions + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_V4L2_FMT_H +#define AVCODEC_V4L2_FMT_H + +#include "libavcodec/avcodec.h" +#include "libavutil/pixfmt.h" + +enum AVPixelFormat ff_v4l2_v4l2fmt_to_avfmt(uint32_t v4l2_fmt, enum AVCodecID avcodec); +uint32_t ff_v4l2_avcodec_to_v4l2fmt(enum AVCodecID avcodec); +uint32_t ff_v4l2_avfmt_to_v4l2fmt(enum AVPixelFormat avfmt); + +#endif /* AVCODEC_V4L2_FMT_H*/ diff --git a/libavcodec/v4l2_m2m.c b/libavcodec/v4l2_m2m.c new file mode 100644 index 0000000..809e591 --- /dev/null +++ b/libavcodec/v4l2_m2m.c @@ -0,0 +1,452 @@ +/* + * V4L mem2mem + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <linux/videodev2.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <unistd.h> +#include <dirent.h> +#include <fcntl.h> +#include "libavcodec/avcodec.h" +#include "libavcodec/internal.h" +#include "libavutil/pixdesc.h" +#include "libavutil/imgutils.h" +#include "libavutil/pixfmt.h" +#include "v4l2_m2m_avcodec.h" +#include "v4l2_buffers.h" +#include "v4l2_fmt.h" +#include "v4l2_m2m.h" + +static inline int try_raw_format(V4L2Context* ctx, enum AVPixelFormat pixfmt) +{ + struct v4l2_format *fmt = &ctx->format; + uint32_t v4l2_fmt; + int ret; + + v4l2_fmt = ff_v4l2_avfmt_to_v4l2fmt(pixfmt); + if (!v4l2_fmt) + return AVERROR(EINVAL); + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) + fmt->fmt.pix_mp.pixelformat = v4l2_fmt; + else + fmt->fmt.pix.pixelformat = v4l2_fmt; + + fmt->type = ctx->type; + + ret = ioctl(ctx->fd, VIDIOC_TRY_FMT, fmt); + if (ret) + return AVERROR(EINVAL); + + return 0; +} + +static int query_raw_format(V4L2Context* ctx, int set) +{ + enum AVPixelFormat pixfmt = ctx->av_pix_fmt; + struct v4l2_fmtdesc fdesc; + int ret; + + memset(&fdesc, 0, sizeof(fdesc)); + fdesc.type = ctx->type; + + if (pixfmt != AV_PIX_FMT_NONE) { + ret = try_raw_format(ctx, pixfmt); + if (ret) + pixfmt = AV_PIX_FMT_NONE; + else + return 0; + } + + for (;;) { + ret = ioctl(ctx->fd, VIDIOC_ENUM_FMT, &fdesc); + if (ret) + return AVERROR(EINVAL); + + pixfmt = ff_v4l2_v4l2fmt_to_avfmt(fdesc.pixelformat, AV_CODEC_ID_RAWVIDEO); + ret = try_raw_format(ctx, pixfmt); + if (ret){ + fdesc.index++; + continue; + } + + if (set) + ctx->av_pix_fmt = pixfmt; + + return 0; + } + + return AVERROR(EINVAL); +} + +static int query_coded_format(V4L2Context* ctx, uint32_t *p) +{ + struct v4l2_fmtdesc fdesc; + uint32_t v4l2_fmt; + int ret; + + v4l2_fmt = ff_v4l2_avcodec_to_v4l2fmt(ctx->av_codec_id); + if (!v4l2_fmt) + return AVERROR(EINVAL); + + memset(&fdesc, 0, sizeof(fdesc)); + fdesc.type = ctx->type; + + for (;;) { + ret = ioctl(ctx->fd, VIDIOC_ENUM_FMT, &fdesc); + if (ret) + return AVERROR(EINVAL); + + if (fdesc.pixelformat == v4l2_fmt) { + break; + } + + fdesc.index++; + } + + *p = v4l2_fmt; + + return 0; +} + +static inline int splane_video(struct v4l2_capability *cap) +{ + if (cap->capabilities & (V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT) & V4L2_CAP_STREAMING) + return 1; + + if (cap->capabilities & V4L2_CAP_VIDEO_M2M) + return 1; + + return 0; +} + +static inline int mplane_video(struct v4l2_capability *cap) +{ + if (cap->capabilities & (V4L2_CAP_VIDEO_CAPTURE_MPLANE | V4L2_CAP_VIDEO_OUTPUT_MPLANE) & V4L2_CAP_STREAMING) + return 1; + + if (cap->capabilities & V4L2_CAP_VIDEO_M2M_MPLANE) + return 1; + + return 0; +} + +static int prepare_contexts(V4L2m2mContext* s, void *log_ctx) +{ + struct v4l2_capability cap; + int ret; + + s->capture.log_ctx = s->output.log_ctx = log_ctx; + s->capture.done = s->output.done = 0; + s->capture.fd = s->output.fd = s->fd; + s->capture.name = "v4l2_cap"; + s->output.name = "v4l2_out"; + atomic_init(&s->refcount, 0); + sem_init(&s->refsync, 0, 0); + + memset(&cap, 0, sizeof(cap)); + ret = ioctl(s->fd, VIDIOC_QUERYCAP, &cap); + if (ret < 0) + return ret; + + av_log(log_ctx, AV_LOG_INFO, "driver '%s' on card '%s'\n", cap.driver, cap.card); + + if (mplane_video(&cap)) { + s->capture.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; + s->output.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; + return 0; + } + + if (splane_video(&cap)) { + s->capture.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + s->output.type = V4L2_BUF_TYPE_VIDEO_OUTPUT; + return 0; + } + + return AVERROR(EINVAL); +} + +static int probe_v4l2_driver(V4L2m2mContext* s, void *log_ctx) +{ + int ret; + + s->fd = open(s->devname, O_RDWR | O_NONBLOCK, 0); + if (s->fd < 0) + return AVERROR(errno); + + ret = prepare_contexts(s, log_ctx); + if (ret < 0) + goto done; + + ret = ff_v4l2_m2m_codec_format_context(&s->output, 0); + if (ret) { + av_log(log_ctx, AV_LOG_DEBUG, "can't set input format\n"); + goto done; + } + + ret = ff_v4l2_m2m_codec_format_context(&s->capture, 0); + if (ret) { + av_log(log_ctx, AV_LOG_DEBUG, "can't to set output format\n"); + goto done; + } + +done: + close(s->fd); + s->fd = 0; + + return ret; +} + +static int configure_contexts(V4L2m2mContext* s, void *log_ctx) +{ + int ret; + + s->fd = open(s->devname, O_RDWR | O_NONBLOCK, 0); + if (s->fd < 0) + return AVERROR(errno); + + ret = prepare_contexts(s, log_ctx); + if (ret < 0) + goto error; + + ret = ff_v4l2_m2m_codec_format_context(&s->output, 1); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "can't set input format\n"); + goto error; + } + + ret = ff_v4l2_m2m_codec_format_context(&s->capture, 1); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "can't to set output format\n"); + goto error; + } + + ret = ff_v4l2_context_init(&s->output, s->output.lazy_init); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "no output context's buffers\n"); + goto error; + } + + ret = ff_v4l2_context_init(&s->capture, s->capture.lazy_init); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "no capture context's buffers\n"); + goto error; + } + +error: + if (ret) { + close(s->fd); + s->fd = 0; + } + + return 0; +} + +static void save_to_context(V4L2Context* ctx, uint32_t v4l2_fmt) +{ + ctx->format.type = ctx->type; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + /* this is to handle the reconfiguration of the capture stream at runtime */ + ctx->format.fmt.pix_mp.height = ctx->height; + ctx->format.fmt.pix_mp.width = ctx->width; + if (v4l2_fmt) + ctx->format.fmt.pix_mp.pixelformat = v4l2_fmt; + } else { + ctx->format.fmt.pix.height = ctx->height; + ctx->format.fmt.pix.width = ctx->width; + if (v4l2_fmt) + ctx->format.fmt.pix_mp.pixelformat = v4l2_fmt; + } +} + +int ff_v4l2_m2m_codec_format_context(V4L2Context* ctx, int set) +{ + uint32_t v4l2_fmt; + int ret; + + if (ctx->av_codec_id == AV_CODEC_ID_RAWVIDEO) { + ret = query_raw_format(ctx, set); + if (ret) + return ret; + + save_to_context(ctx, 0); + if (set) + return ioctl(ctx->fd, VIDIOC_S_FMT, &ctx->format); + + return ret; + } + + ret = query_coded_format(ctx, &v4l2_fmt); + if (ret) + return ret; + + save_to_context(ctx, v4l2_fmt); + if (set) + return ioctl(ctx->fd, VIDIOC_S_FMT, &ctx->format); + + return ioctl(ctx->fd, VIDIOC_TRY_FMT, &ctx->format); +} + +int ff_v4l2_m2m_codec_end(V4L2m2mContext* s) +{ + int ret; + + ret = ff_v4l2_context_set_status(&s->output, VIDIOC_STREAMOFF); + if (ret) + av_log(s->output.log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", s->output.name); + + ret = ff_v4l2_context_set_status(&s->capture, VIDIOC_STREAMOFF); + if (ret) + av_log(s->capture.log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", s->capture.name); + + ff_v4l2_context_release(&s->output); + + if (atomic_load(&s->refcount)) { + av_log(s->capture.log_ctx, AV_LOG_DEBUG, "avpriv_v4l2m2m_end leaving pending buffers \n"); + + return 0; + } + + ff_v4l2_context_release(&s->capture); + sem_destroy(&s->refsync); + + /* release the hardware */ + close(s->fd); + + return 0; +} + +int ff_v4l2_m2m_codec_init(V4L2m2mContext* s, void* log_ctx) +{ + char *devname_save = s->devname; + int ret = AVERROR(EINVAL); + struct dirent *entry; + char node[PATH_MAX]; + DIR *dirp; + + if (s->devname && *s->devname) + return configure_contexts(s, log_ctx); + + dirp = opendir("/dev"); + if (!dirp) + return AVERROR(errno); + + for (entry = readdir(dirp); entry; entry = readdir(dirp)) { + + if (strncmp(entry->d_name, "video", 5)) + continue; + + snprintf(node, sizeof(node), "/dev/%s", entry->d_name); + + av_log(log_ctx, AV_LOG_DEBUG, "probing device %s\n", node); + + s->devname = node; + ret = probe_v4l2_driver(s, log_ctx); + if (!ret) + break; + } + + closedir(dirp); + + if (!ret) { + av_log(log_ctx, AV_LOG_INFO, "Using device %s\n", node); + ret = configure_contexts(s, log_ctx); + } else { + av_log(log_ctx, AV_LOG_ERROR, "Could not find a valid device\n"); + } + s->devname = devname_save; + + return ret; +} + +int ff_v4l2_m2m_codec_reinit(V4L2m2mContext* s) +{ + int ret; + + /* 1. reinit in progress */ + s->reinit = 1; + + av_log(s->avctx, AV_LOG_DEBUG, "reinit context\n"); + + /* 2. streamoff */ + ret = ff_v4l2_context_set_status(&s->capture, VIDIOC_STREAMOFF); + if (ret) + av_log(s->avctx, AV_LOG_ERROR, "capture VIDIOC_STREAMOFF\n"); + + /* 3. unmap the capture buffers (v4l2 and ffmpeg): + * we must wait for all references to be released before being allowed + * to queue new buffers. + */ + av_log(s->avctx, AV_LOG_DEBUG, "capture wait for user to release AVBufferRefs \n"); + if (atomic_load(&s->refcount)) { + while(sem_wait(&s->refsync) == -1 && errno == EINTR); + } + + ff_v4l2_context_release(&s->capture); + + /* 4. query the new format */ + ret = ff_v4l2_m2m_codec_format_context(&s->capture, 1); + if (ret) { + av_log(s->avctx, AV_LOG_ERROR, "setting capture format\n"); + return ret; + } + + /* 5. do lazy initialization */ + ret = ff_v4l2_context_init(&s->capture, s->capture.lazy_init); + if (ret) { + av_log(s->avctx, AV_LOG_ERROR, "capture buffers lazy init\n"); + return ret; + } + + /* 6. update AVCodecContext */ + ret = ff_set_dimensions(s->avctx, s->capture.width, s->capture.height); + if (ret < 0) + av_log(s->avctx, AV_LOG_WARNING, "update avcodec height and width\n"); + + /* 7. complete reinit */ + sem_destroy(&s->refsync); + sem_init(&s->refsync, 0, 0); + s->draining = 0; + s->reinit = 0; + + return 0; +} + +int ff_v4l2m2m_codec_end(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + + av_log(avctx, AV_LOG_DEBUG, "Closing context\n"); + + return ff_v4l2_m2m_codec_end(s); +} + +int ff_v4l2m2m_codec_init(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + s->avctx = avctx; + + return ff_v4l2_m2m_codec_init(s, avctx); +} + diff --git a/libavcodec/v4l2_m2m.h b/libavcodec/v4l2_m2m.h new file mode 100644 index 0000000..63a4326 --- /dev/null +++ b/libavcodec/v4l2_m2m.h @@ -0,0 +1,70 @@ +/* + * V4L2 mem2mem helper functions + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_V4L2_M2M_H +#define AVCODEC_V4L2_M2M_H + +#include <semaphore.h> +#include "v4l2_buffers.h" + +#define container_of(ptr, type, member) ({ \ + const __typeof__(((type *)0)->member ) *__mptr = (ptr); \ + (type *)((char *)__mptr - offsetof(type,member) );}) + +#define V4L_M2M_DEFAULT_OPTS \ + { "device",\ + "Path to the device to use",\ + OFFSET(devname), AV_OPT_TYPE_STRING, {.str = NULL }, 0, 0, FLAGS },\ + { "num_output_buffers",\ + "Number of buffers in the output context",\ + OFFSET(output.num_buffers), AV_OPT_TYPE_INT, { .i64 = 16 }, 6, INT_MAX, FLAGS } + +typedef struct V4L2m2mContext +{ + AVClass *class; + int fd; + char *devname; + + /* the codec context queues */ + V4L2Context capture; + V4L2Context output; + + /* refcount of buffers held by the user */ + atomic_uint refcount; + + /* dynamic stream reconfig */ + AVCodecContext *avctx; + sem_t refsync; + int reinit; + + /* null frame or packet received */ + int draining; +} V4L2m2mContext; + +int ff_v4l2_m2m_codec_init(V4L2m2mContext *ctx, void* log_ctx); +int ff_v4l2_m2m_codec_reinit(V4L2m2mContext *ctx); +int ff_v4l2_m2m_codec_end(V4L2m2mContext *ctx); +int ff_v4l2_m2m_codec_format_context(V4L2Context *ctx, int set); + + +#endif /* AVCODEC_V4L2_M2M_H */ diff --git a/libavcodec/v4l2_m2m_avcodec.h b/libavcodec/v4l2_m2m_avcodec.h new file mode 100644 index 0000000..c6ad5d4 --- /dev/null +++ b/libavcodec/v4l2_m2m_avcodec.h @@ -0,0 +1,32 @@ +/* + * V4L2 mem2mem avcodec helper functions + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_V4L2_M2M_AVCODEC_H +#define AVCODEC_V4L2_M2M_AVCODEC_H + +#include "libavcodec/avcodec.h" + +int ff_v4l2m2m_codec_init(AVCodecContext *avctx); +int ff_v4l2m2m_codec_end(AVCodecContext *avctx); + +#endif diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c new file mode 100644 index 0000000..205f695 --- /dev/null +++ b/libavcodec/v4l2_m2m_dec.c @@ -0,0 +1,213 @@ +/* + * V4L2 mem2mem decoders + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <linux/videodev2.h> +#include <sys/ioctl.h> +#include "libavutil/pixfmt.h" +#include "libavutil/pixdesc.h" +#include "libavutil/opt.h" +#include "libavcodec/avcodec.h" +#include "libavcodec/decode.h" +#include "v4l2_m2m_avcodec.h" +#include "v4l2_buffers.h" +#include "v4l2_fmt.h" +#include "v4l2_m2m.h" + +static int try_start(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const capture = &s->capture; + V4L2Context *const output = &s->output; + struct v4l2_event_subscription sub; + struct v4l2_selection selection; + int ret; + + if (output->streamon && capture->streamon) + return 0; + + /* 0. subscribe to source change event */ + memset(&sub, 0, sizeof(sub)); + sub.type = V4L2_EVENT_SOURCE_CHANGE; + ret = ioctl(s->fd, VIDIOC_SUBSCRIBE_EVENT, &sub); + if ( ret < 0) + av_log(avctx, AV_LOG_WARNING, "decoding does not support resolution change\n"); + + /* 1. start the output process */ + if (!output->streamon) { + ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON); + if (ret < 0) { + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON on output context\n"); + return ret; + } + } + + /* 2. get the capture format */ + capture->format.type = capture->type; + ret = ioctl(capture->fd, VIDIOC_G_FMT, &capture->format); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_FMT ioctl\n"); + return ret; + } + + /* 2.1 update the AVCodecContext */ + avctx->pix_fmt = ff_v4l2_v4l2fmt_to_avfmt(capture->format.fmt.pix_mp.pixelformat, AV_CODEC_ID_RAWVIDEO); + capture->av_pix_fmt = avctx->pix_fmt; + + /* 3. set the crop parameters */ + selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + selection.r.height = avctx->coded_height; + selection.r.width = avctx->coded_width; + ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection); + if (!ret) { + ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_SELECTION ioctl\n"); + } else { + av_log(avctx, AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height); + /* update the size of the resulting frame */ + capture->height = selection.r.height; + capture->width = selection.r.width; + } + } + + /* 4. init the capture context now that we have the capture format */ + if (!capture->buffers) { + av_log(capture->log_ctx, AV_LOG_DEBUG, "%s requested (%dx%d)\n", + capture->name, capture->format.fmt.pix_mp.width, capture->format.fmt.pix_mp.height); + + ret = ff_v4l2_context_init(capture, 0); + if (ret) { + av_log(avctx, AV_LOG_DEBUG, "can't request output buffers\n"); + return ret; + } + } + + /* 6. start the capture process */ + ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON); + if (ret) { + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON, on capture context\n"); + return ret; + } + + return 0; +} + +static av_cold int v4l2m2m_decode_init(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *capture = &s->capture; + V4L2Context *output = &s->output; + + output->height = capture->height = avctx->coded_height; + output->width = capture->width =avctx->coded_width; + + output->av_codec_id = avctx->codec_id; + output->av_pix_fmt = AV_PIX_FMT_NONE; + + /* + * the buffers associated to this context can not be initialized without + * additional information available in the kernel driver, + * so let's postpone requesting the buffers until we know more about the frames + */ + capture->lazy_init = 1; + capture->av_codec_id = AV_CODEC_ID_RAWVIDEO; + capture->av_pix_fmt = avctx->pix_fmt; + + return ff_v4l2m2m_codec_init(avctx); +} +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a + * timeout is * required when retrieving a frame in case the driver has not received + * enough input * to start generating output. + * + * once decoding starts, the timeout should not be hit. + */ +static int v4l2m2m_receive_frame(AVCodecContext *avctx, AVFrame *frame) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const capture = &s->capture; + V4L2Context *const output = &s->output; + AVPacket avpkt = {0}; + int timeout = 50; + int ret; + + ret = ff_decode_get_packet(avctx, &avpkt); + if (ret < 0 && ret != AVERROR_EOF) + return ret; + + if (s->draining) + goto dequeue; + + ret = ff_v4l2_enqueue_packet(output, &avpkt); + if (ret < 0) + return ret; + + if (avpkt.size) { + ret = try_start(avctx); + if (ret) + return 0; + } + +dequeue: + return ff_v4l2_dequeue_frame(capture, frame, timeout); +} + +#define OFFSET(x) offsetof(V4L2m2mContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_DECODING_PARAM + +static const AVOption options[] = { + V4L_M2M_DEFAULT_OPTS,{ "num_capture_extra_buffers","Number of extra buffers in the capture context", + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT,{.i64 = 6}, 6, INT_MAX, FLAGS}, + { NULL}, +}; + +#define M2MDEC(NAME, LONGNAME, CODEC, bsf_name) \ +static const AVClass v4l2_m2m_ ## NAME ## _dec_class = {\ + .class_name = #NAME "_v4l2_m2m_decoder",\ + .item_name = av_default_item_name,\ + .option = options,\ + .version = LIBAVUTIL_VERSION_INT,\ +};\ +\ +AVCodec ff_ ## NAME ## _v4l2m2m_decoder = { \ + .name = #NAME "_v4l2m2m" ,\ + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ + .type = AVMEDIA_TYPE_VIDEO,\ + .id = CODEC ,\ + .priv_data_size = sizeof(V4L2m2mContext),\ + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ + .init = v4l2m2m_decode_init,\ + .receive_frame = v4l2m2m_receive_frame,\ + .close = ff_v4l2m2m_codec_end,\ + .bsfs = bsf_name, \ +}; + +M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); +M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); +M2MDEC(mpeg1, "MPEG1", AV_CODEC_ID_MPEG1VIDEO, NULL); +M2MDEC(mpeg2, "MPEG2", AV_CODEC_ID_MPEG2VIDEO, NULL); +M2MDEC(mpeg4, "MPEG4", AV_CODEC_ID_MPEG4, NULL); +M2MDEC(h263, "H.263", AV_CODEC_ID_H263, NULL); +M2MDEC(vc1 , "VC1", AV_CODEC_ID_VC1, NULL); +M2MDEC(vp8, "VP8", AV_CODEC_ID_VP8, NULL); +M2MDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL); + diff --git a/libavcodec/v4l2_m2m_enc.c b/libavcodec/v4l2_m2m_enc.c new file mode 100644 index 0000000..47bb160 --- /dev/null +++ b/libavcodec/v4l2_m2m_enc.c @@ -0,0 +1,340 @@ +/* + * V4L2 mem2mem encoders + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <linux/videodev2.h> +#include <sys/ioctl.h> +#include <search.h> +#include "libavcodec/avcodec.h" +#include "libavutil/pixdesc.h" +#include "libavutil/pixfmt.h" +#include "libavutil/opt.h" +#include "v4l2_m2m_avcodec.h" +#include "v4l2_buffers.h" +#include "v4l2_m2m.h" + +#define MPEG_CID(x) V4L2_CID_MPEG_VIDEO_##x +#define MPEG_VIDEO(x) V4L2_MPEG_VIDEO_##x + +static inline void v4l2_set_timeperframe(V4L2m2mContext *s, unsigned int num, unsigned int den) +{ + struct v4l2_streamparm parm = { 0 }; + + parm.parm.output.timeperframe.denominator = den; + parm.parm.output.timeperframe.numerator = num; + parm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; + + if (ioctl(s->fd, VIDIOC_S_PARM, &parm) < 0) + av_log(s->avctx, AV_LOG_WARNING, "Failed to set timeperframe"); +} + +static inline void v4l2_set_ext_ctrl(V4L2m2mContext *s, unsigned int id, signed int value, const char *name) +{ + struct v4l2_ext_controls ctrls = { 0 }; + struct v4l2_ext_control ctrl = { 0 }; + + /* set ctrls */ + ctrls.ctrl_class = V4L2_CTRL_CLASS_MPEG; + ctrls.controls = &ctrl; + ctrls.count = 1; + + /* set ctrl*/ + ctrl.value = value; + ctrl.id = id ; + + if (ioctl(s->fd, VIDIOC_S_EXT_CTRLS, &ctrls) < 0) + av_log(s->avctx, AV_LOG_WARNING, "Failed to set %s\n", name); + else + av_log(s->avctx, AV_LOG_DEBUG, "Encoder: %s = %d\n", name, value); +} + +static inline int v4l2_get_ext_ctrl(V4L2m2mContext *s, unsigned int id, signed int *value, const char *name) +{ + struct v4l2_ext_controls ctrls = { 0 }; + struct v4l2_ext_control ctrl = { 0 }; + int ret; + + /* set ctrls */ + ctrls.ctrl_class = V4L2_CTRL_CLASS_MPEG; + ctrls.controls = &ctrl; + ctrls.count = 1; + + /* set ctrl*/ + ctrl.id = id ; + + ret = ioctl(s->fd, VIDIOC_G_EXT_CTRLS, &ctrls); + if (ret < 0) { + av_log(s->avctx, AV_LOG_WARNING, "Failed to set %s\n", name); + return ret; + } + + *value = ctrl.value; + + return 0; +} + +static int match_profile(const void *a, const void *b) +{ + if (*(unsigned int *)a == *(unsigned int *)b) + return 0; + + return 1; +} + +static inline unsigned int v4l2_h264_profile_from_ff(int p) +{ + struct h264_profile { + unsigned int ffmpeg_val; + unsigned int v4l2_val; + } *val, profile[] = { + { FF_PROFILE_H264_CONSTRAINED_BASELINE, MPEG_VIDEO(H264_PROFILE_CONSTRAINED_BASELINE) }, + { FF_PROFILE_H264_HIGH_444_PREDICTIVE, MPEG_VIDEO(H264_PROFILE_HIGH_444_PREDICTIVE) }, + { FF_PROFILE_H264_HIGH_422_INTRA, MPEG_VIDEO(H264_PROFILE_HIGH_422_INTRA) }, + { FF_PROFILE_H264_HIGH_444_INTRA, MPEG_VIDEO(H264_PROFILE_HIGH_444_INTRA) }, + { FF_PROFILE_H264_HIGH_10_INTRA, MPEG_VIDEO(H264_PROFILE_HIGH_10_INTRA) }, + { FF_PROFILE_H264_HIGH_422, MPEG_VIDEO(H264_PROFILE_HIGH_422) }, + { FF_PROFILE_H264_BASELINE, MPEG_VIDEO(H264_PROFILE_BASELINE) }, + { FF_PROFILE_H264_EXTENDED, MPEG_VIDEO(H264_PROFILE_EXTENDED) }, + { FF_PROFILE_H264_HIGH_10, MPEG_VIDEO(H264_PROFILE_HIGH_10) }, + { FF_PROFILE_H264_MAIN, MPEG_VIDEO(H264_PROFILE_MAIN) }, + { FF_PROFILE_H264_HIGH, MPEG_VIDEO(H264_PROFILE_HIGH) }, + }; + size_t len = sizeof(profile) / sizeof(profile[0]); + + val = lfind(&p, profile, &len, sizeof(profile[0]), match_profile); + if (val) + return val->v4l2_val; + + return FF_PROFILE_UNKNOWN; +} + +static inline int v4l2_mpeg4_profile_from_ff(int p) +{ + struct mpeg4_profile { + unsigned int ffmpeg_val; + unsigned int v4l2_val; + } *val, profile[] = { + { FF_PROFILE_MPEG4_ADVANCED_CODING, MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_CODING_EFFICIENCY) }, + { FF_PROFILE_MPEG4_ADVANCED_SIMPLE, MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_SIMPLE) }, + { FF_PROFILE_MPEG4_SIMPLE_SCALABLE, MPEG_VIDEO(MPEG4_PROFILE_SIMPLE_SCALABLE) }, + { FF_PROFILE_MPEG4_SIMPLE, MPEG_VIDEO(MPEG4_PROFILE_SIMPLE) }, + { FF_PROFILE_MPEG4_CORE, MPEG_VIDEO(MPEG4_PROFILE_CORE) }, + }; + size_t len = sizeof(profile) / sizeof(profile[0]); + + val = lfind(&p, profile, &len, sizeof(profile[0]), match_profile); + if (val) + return val->v4l2_val; + + return FF_PROFILE_UNKNOWN; +} + +static int check_b_frame_support(V4L2m2mContext *s) +{ + if (s->avctx->max_b_frames) + av_log(s->avctx, AV_LOG_WARNING, "Encoder does not support b-frames yet\n"); + + v4l2_set_ext_ctrl(s, MPEG_CID(B_FRAMES), 0, "number of B-frames"); + + v4l2_get_ext_ctrl(s, MPEG_CID(B_FRAMES), &s->avctx->max_b_frames, "number of B-frames"); + if (s->avctx->max_b_frames == 0) + return 0; + + avpriv_report_missing_feature(s->avctx, "DTS/PTS calculation for V4L2 encoding"); + + return AVERROR_PATCHWELCOME; +} + +static av_cold int v4l2m2m_encode_init(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *capture = &s->capture; + V4L2Context *output = &s->output; + int qmin_cid, qmax_cid, ret, val; + int qmin, qmax; + + /* common settings output/capture */ + output->height = capture->height = avctx->height; + output->width = capture->width = avctx->width; + + /* output context */ + output->av_codec_id = AV_CODEC_ID_RAWVIDEO; + output->av_pix_fmt = avctx->pix_fmt; + + /* capture context */ + capture->av_codec_id = avctx->codec_id; + capture->av_pix_fmt = AV_PIX_FMT_NONE; + + ret = ff_v4l2m2m_codec_init(avctx); + if (ret) + return ret; + + ret = check_b_frame_support(s); + if (ret) + return ret; + + /* set params */ + v4l2_set_timeperframe(s, avctx->framerate.num, avctx->framerate.den); + + /* set ext ctrls */ + v4l2_set_ext_ctrl(s, MPEG_CID(HEADER_MODE), MPEG_VIDEO(HEADER_MODE_SEPARATE), "header mode"); + v4l2_set_ext_ctrl(s, MPEG_CID(BITRATE) , avctx->bit_rate, "bit rate"); + v4l2_set_ext_ctrl(s, MPEG_CID(GOP_SIZE), avctx->gop_size,"gop size"); + + av_log(avctx, AV_LOG_DEBUG, "Encoder Context: id (%d), profile (%d), frame rate(%d/%d), number b-frames (%d), " + "gop size (%d), bit rate (%ld), qmin (%d), qmax (%d)\n", + avctx->codec_id, avctx->profile, avctx->framerate.num, avctx->framerate.den, + avctx->max_b_frames, avctx->gop_size, avctx->bit_rate, avctx->qmin, avctx->qmax); + + switch (avctx->codec_id) { + case AV_CODEC_ID_H264: + val = v4l2_h264_profile_from_ff(avctx->profile); + if (val != FF_PROFILE_UNKNOWN) + v4l2_set_ext_ctrl(s, MPEG_CID(H264_PROFILE), val, "h264 profile"); + else + av_log(avctx, AV_LOG_WARNING, "h264 profile unknown)\n"); + qmin_cid = MPEG_CID(H264_MIN_QP); + qmax_cid = MPEG_CID(H264_MAX_QP); + + qmin = 0; + qmax = 51; + break; + case AV_CODEC_ID_MPEG4: + val = v4l2_mpeg4_profile_from_ff(avctx->profile); + if (val != FF_PROFILE_UNKNOWN) + v4l2_set_ext_ctrl(s, MPEG_CID(MPEG4_PROFILE), val, "mpeg4 profile"); + else + av_log(avctx, AV_LOG_WARNING, "mpeg4 profile unknown)\n"); + qmin_cid = MPEG_CID(MPEG4_MIN_QP); + qmax_cid = MPEG_CID(MPEG4_MAX_QP); + if (avctx->flags & CODEC_FLAG_QPEL) + v4l2_set_ext_ctrl(s, MPEG_CID(MPEG4_QPEL), 1, "qpel"); + qmax = 51; + qmin = 0; + break; + case AV_CODEC_ID_H263: + qmin_cid = MPEG_CID(H263_MIN_QP); + qmax_cid = MPEG_CID(H263_MAX_QP); + qmin = 1; + qmax = 31; + break; + case AV_CODEC_ID_VP8: + qmin_cid = MPEG_CID(VPX_MIN_QP); + qmax_cid = MPEG_CID(VPX_MAX_QP); + qmin = 0; + qmax = 127; + break; + case AV_CODEC_ID_VP9: + qmin_cid = MPEG_CID(VPX_MIN_QP); + qmax_cid = MPEG_CID(VPX_MAX_QP); + qmin = 0; + qmax = 255; + break; + default: + return 0; + } + + if (qmin != avctx->qmin || qmax != avctx->qmax) + av_log(avctx, AV_LOG_WARNING, "Encoder adjusted: qmin (%d), qmax (%d)\n", qmin, qmax); + + v4l2_set_ext_ctrl(s, qmin_cid, qmin, "minimum video quantizer scale"); + v4l2_set_ext_ctrl(s, qmax_cid, qmax, "maximum video quantizer scale"); + + return 0; +} + +static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const output = &s->output; + + return ff_v4l2_enqueue_frame(output, frame); +} + +/* Send and receive frame happen on the same thread, hence the need for a polling timeout */ +static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const capture = &s->capture; + V4L2Context *const output = &s->output; + unsigned int timeout = 50; + int ret; + + if (s->draining) + goto dequeue; + + if (!output->streamon) { + ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n"); + return ret; + } + } + + if (!capture->streamon) { + ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n"); + return ret; + } + } + +dequeue: + return ff_v4l2_dequeue_packet(capture, avpkt, timeout); +} + +#define OFFSET(x) offsetof(V4L2m2mContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM + +static const AVOption options[] = { + V4L_M2M_DEFAULT_OPTS, + { "num_capture_buffers", "Number of buffers in the capture context", + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT, {.i64 = 4 }, 4, INT_MAX, FLAGS }, + { NULL }, +}; + +#define M2MENC(NAME, LONGNAME, CODEC) \ +static const AVClass v4l2_m2m_ ## NAME ## _enc_class = {\ + .class_name = #NAME "_v4l2_m2m_encoder",\ + .item_name = av_default_item_name,\ + .option = options,\ + .version = LIBAVUTIL_VERSION_INT,\ +};\ +\ +AVCodec ff_ ## NAME ## _v4l2m2m_encoder = { \ + .name = #NAME "_v4l2m2m" ,\ + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " encoder wrapper"),\ + .type = AVMEDIA_TYPE_VIDEO,\ + .id = CODEC ,\ + .priv_data_size = sizeof(V4L2m2mContext),\ + .priv_class = &v4l2_m2m_ ## NAME ##_enc_class,\ + .init = v4l2m2m_encode_init,\ + .send_frame = v4l2m2m_send_frame,\ + .receive_packet = v4l2m2m_receive_packet,\ + .close = ff_v4l2m2m_codec_end,\ +}; + +M2MENC(mpeg4,"MPEG4", AV_CODEC_ID_MPEG4); +M2MENC(h263, "H.263", AV_CODEC_ID_H263); +M2MENC(h264, "H.264", AV_CODEC_ID_H264); +M2MENC(hevc, "HEVC", AV_CODEC_ID_HEVC); +M2MENC(vp8, "VP8", AV_CODEC_ID_VP8);
This patchset enhances Alexis Ballier's original patch and validates it using Qualcomm's Venus hardware (driver recently landed upstream [1]). This has been tested on Qualcomm's DragonBoard 410c and 820c Configure/make scripts have been validated on Ubuntu 10.04 and 16.04. Tested decoders: - h264 - h263 - mpeg4 - vp8 - vp9 - hevc Tested encoders: - h264 - h263 - mpeg4 Tested transcoding (concurrent encoding/decoding) Some of the changes introduced: - v4l2: code cleanup and abstractions added - v4l2: follow the new encode/decode api. - v4l2: fix display size for NV12 output pool. - v4l2: handle EOS. - v4l2: vp8 and mpeg4 decoding and encoding. - v4l2: hevc and vp9 support. - v4l2: generate EOF on dequeue errors. - v4l2: h264_mp4toannexb filtering. - v4l2: fixed make install and fate issues. - v4l2: codecs enabled/disabled depending on pixfmt defined - v4l2: pass timebase/framerate to the context - v4l2: runtime decoder reconfiguration. - v4l2: add more frame information - v4l2: free hardware resources on last reference being released - v4l2 encoding: disable b-frames for upstreaming (patch required) [1] https://lwn.net/Articles/697956/ Reviewed-by: Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> Reviewed-by: Alexis Ballier <aballier@gentoo.org> Tested-by: Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> --- Changelog | 1 + configure | 30 +- libavcodec/Makefile | 16 + libavcodec/allcodecs.c | 9 + libavcodec/v4l2_buffers.c | 918 ++++++++++++++++++++++++++++++++++++++++++ libavcodec/v4l2_buffers.h | 236 +++++++++++ libavcodec/v4l2_fmt.c | 184 +++++++++ libavcodec/v4l2_fmt.h | 34 ++ libavcodec/v4l2_m2m.c | 452 +++++++++++++++++++++ libavcodec/v4l2_m2m.h | 70 ++++ libavcodec/v4l2_m2m_avcodec.h | 32 ++ libavcodec/v4l2_m2m_dec.c | 213 ++++++++++ libavcodec/v4l2_m2m_enc.c | 340 ++++++++++++++++ 13 files changed, 2534 insertions(+), 1 deletion(-) create mode 100644 libavcodec/v4l2_buffers.c create mode 100644 libavcodec/v4l2_buffers.h create mode 100644 libavcodec/v4l2_fmt.c create mode 100644 libavcodec/v4l2_fmt.h create mode 100644 libavcodec/v4l2_m2m.c create mode 100644 libavcodec/v4l2_m2m.h create mode 100644 libavcodec/v4l2_m2m_avcodec.h create mode 100644 libavcodec/v4l2_m2m_dec.c create mode 100644 libavcodec/v4l2_m2m_enc.c