| Message ID | 01e9296a-d2ef-64f8-a501-ddc18f64fbfd@jkqxz.net |
|---|---|
| State | New |
| Headers | show |
Hi Jorge, unfortunately don't have a SBC that features a V4L2 M2M MJPEG/JPEG encoder so I couldn't test it. Hi Mark, thanks for the extensive review. Too bad neither scenario worked fully. 2018-08-12 19:24 GMT+02:00 Mark Thompson <sw@jkqxz.net>: > On 12/08/18 15:40, Maxime Jourdan wrote: >> Tested on an Odroid-C2 with a V4L2 M2M MJPEG decoder. >> >> Signed-off-by: Maxime Jourdan <maxi.jourdan@wanadoo.fr> >> --- >> configure | 3 +++ >> libavcodec/Makefile | 2 ++ >> libavcodec/allcodecs.c | 2 ++ >> libavcodec/v4l2_m2m_dec.c | 1 + >> libavcodec/v4l2_m2m_enc.c | 1 + >> 5 files changed, 9 insertions(+) >> >> diff --git a/configure b/configure >> index e718c1531c..5a5ba1ebcf 100755 >> --- a/configure >> +++ b/configure >> @@ -2939,6 +2939,8 @@ hevc_vaapi_encoder_deps="VAEncPictureParameterBufferHEVC" >> hevc_vaapi_encoder_select="cbs_h265 vaapi_encode" >> hevc_v4l2m2m_decoder_deps="v4l2_m2m hevc_v4l2_m2m" >> hevc_v4l2m2m_encoder_deps="v4l2_m2m hevc_v4l2_m2m" >> +mjpeg_v4l2m2m_decoder_deps="v4l2_m2m mjpeg_v4l2_m2m" >> +mjpeg_v4l2m2m_encoder_deps="v4l2_m2m mjpeg_v4l2_m2m" >> mjpeg_cuvid_decoder_deps="cuvid" >> mjpeg_qsv_encoder_deps="libmfx" >> mjpeg_qsv_encoder_select="qsvenc" >> @@ -6252,6 +6254,7 @@ check_header linux/videodev2.h >> test_code cc linux/videodev2.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_sanitized struct_v4l2_frmivalenum_discrete >> check_cc v4l2_m2m linux/videodev2.h "int i = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_VIDEO_M2M | V4L2_BUF_FLAG_LAST;" >> check_cc vc1_v4l2_m2m linux/videodev2.h "int i = V4L2_PIX_FMT_VC1_ANNEX_G;" >> +check_cc mjpeg_v4l2_m2m linux/videodev2.h "int i = V4L2_PIX_FMT_MJPEG;" > > The existing format code unconditionally assumes this value is present (libavcodec/v4l2_fmt.c:54 - it's been around forever for input devices), so the check isn't needed. ack. >> check_cc mpeg1_v4l2_m2m linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG1;" >> check_cc mpeg2_v4l2_m2m linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG2;" >> check_cc mpeg4_v4l2_m2m linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG4;" >> diff --git a/libavcodec/Makefile b/libavcodec/Makefile >> index 6e40702947..cd975739ef 100644 >> --- a/libavcodec/Makefile >> +++ b/libavcodec/Makefile >> @@ -415,7 +415,9 @@ OBJS-$(CONFIG_METASOUND_DECODER) += metasound.o metasound_data.o \ >> twinvq.o >> OBJS-$(CONFIG_MICRODVD_DECODER) += microdvddec.o ass.o >> OBJS-$(CONFIG_MIMIC_DECODER) += mimic.o >> +OBJS-$(CONFIG_MJPEG_V4L2M2M_DECODER) += v4l2_m2m_dec.o >> OBJS-$(CONFIG_MJPEG_DECODER) += mjpegdec.o >> +OBJS-$(CONFIG_MJPEG_V4L2M2M_ENCODER) += v4l2_m2m_enc.o > > Use the same ordering as for other codecs. ack. >> OBJS-$(CONFIG_MJPEG_ENCODER) += mjpegenc.o mjpegenc_common.o \ >> mjpegenc_huffman.o >> OBJS-$(CONFIG_MJPEGB_DECODER) += mjpegbdec.o >> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c >> index ab3ec04251..af4870a54d 100644 >> --- a/libavcodec/allcodecs.c >> +++ b/libavcodec/allcodecs.c >> @@ -173,7 +173,9 @@ extern AVCodec ff_magicyuv_encoder; >> extern AVCodec ff_magicyuv_decoder; >> extern AVCodec ff_mdec_decoder; >> extern AVCodec ff_mimic_decoder; > > This file gets parsed in order to make the table of decoders/encoders, so: > >> +extern AVCodec ff_mjpeg_v4l2m2m_encoder; >> extern AVCodec ff_mjpeg_encoder; > > The new entry for the encoder should be in the non-default section at the bottom. > >> +extern AVCodec ff_mjpeg_v4l2m2m_decoder; >> extern AVCodec ff_mjpeg_decoder; > > The new entry for the decoder should be after the normal decoder. ack. >> extern AVCodec ff_mjpegb_decoder; >> extern AVCodec ff_mmvideo_decoder; >> diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c >> index a4e2f02596..80a09f7a43 100644 >> --- a/libavcodec/v4l2_m2m_dec.c >> +++ b/libavcodec/v4l2_m2m_dec.c >> @@ -231,6 +231,7 @@ AVCodec ff_ ## NAME ## _v4l2m2m_decoder = { \ >> >> M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); >> M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "hevc_mp4toannexb"); >> +M2MDEC(mjpeg, "MJPEG", AV_CODEC_ID_MJPEG, NULL); >> M2MDEC(mpeg1, "MPEG1", AV_CODEC_ID_MPEG1VIDEO, NULL); >> M2MDEC(mpeg2, "MPEG2", AV_CODEC_ID_MPEG2VIDEO, NULL); >> M2MDEC(mpeg4, "MPEG4", AV_CODEC_ID_MPEG4, NULL); >> diff --git a/libavcodec/v4l2_m2m_enc.c b/libavcodec/v4l2_m2m_enc.c >> index 4c9ea1fd92..b025f59275 100644 >> --- a/libavcodec/v4l2_m2m_enc.c >> +++ b/libavcodec/v4l2_m2m_enc.c >> @@ -345,6 +345,7 @@ AVCodec ff_ ## NAME ## _v4l2m2m_encoder = { \ >> .wrapper_name = "v4l2m2m", \ >> }; >> >> +M2MENC(mjpeg,"MJPEG", AV_CODEC_ID_MJPEG); >> M2MENC(mpeg4,"MPEG4", AV_CODEC_ID_MPEG4); >> M2MENC(h263, "H.263", AV_CODEC_ID_H263); >> M2MENC(h264, "H.264", AV_CODEC_ID_H264); > > Since MJPEG is intra-only, it probably wants to avoid setting options like GOP size and B-frames (currently this gives me warnings for each of these options that it failed to set them, visible in log below). Ack, although right now I'm not sure what's the cleanest way to enforce this. > Trying this on an Odroid XU4 (Exynos S5P running Linux 4.14.55), it looks like we need to somehow allow both "MJPG" and "JPEG" as fourccs. Reordering the two lines in libavcodec/v4l2_fmt.c to put "JPEG" first makes it get past that test, but obviously that will stop it from working on an "MJPG" device. I see, so some rework will be needed to account for AV_CODECs that appear multiple times in the list. > > So, for testing I used this patch: > > diff --git a/libavcodec/v4l2_fmt.c b/libavcodec/v4l2_fmt.c > index 6df47e3..06b57db 100644 > --- a/libavcodec/v4l2_fmt.c > +++ b/libavcodec/v4l2_fmt.c > @@ -51,8 +51,8 @@ static const struct fmt_conversion { > { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV410) }, > { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YVU410) }, > { AV_FMT(NV12), AV_CODEC(RAWVIDEO), V4L2_FMT(NV12) }, > - { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, > { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(JPEG) }, > + { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, > #ifdef V4L2_PIX_FMT_SRGGB8 > { AV_FMT(BAYER_BGGR8), AV_CODEC(RAWVIDEO), V4L2_FMT(SBGGR8) }, > { AV_FMT(BAYER_GBRG8), AV_CODEC(RAWVIDEO), V4L2_FMT(SGBRG8) }, > > > After getting past that, the decoder seems to hang for me pretty much immediately on a DQBUF call. Did dmesg have anything interesting to say ? > Input from a UVC webcam (Logitech C920, making 640x480 YUV 4:2:2 JPEGs): > > $ gdb --args ./ffmpeg_g -y -v 55 -f v4l2 -format mjpeg -framerate 30 -video_size 640x480 -c:v mjpeg_v4l2m2m -i /dev/video0 -f null - > ... > [video4linux2,v4l2 @ 0x1a19140] All info found > Input #0, video4linux2,v4l2, from '/dev/video0': > Duration: N/A, start: 4490.517673, bitrate: N/A > Stream #0:0, 1, 1/1000000: Video: mjpeg, 1 reference frame, yuvj422p(pc, bt470bg/unknown/unknown, center), 640x480, 0/1, 30 fps, 30 tbr, 1000k tbn, 1000k tbc > Successfully opened the file. > Parsing a group of options: output url -. > Applying option f (force format) with argument null. > Successfully parsed a group of options. > Opening an output file: -. > Successfully opened the file. > [mjpeg_v4l2m2m @ 0x1a1a9d0] probing device /dev/video0 > [mjpeg_v4l2m2m @ 0x1a1a9d0] driver 'uvcvideo' on card 'HD Pro Webcam C920' > [mjpeg_v4l2m2m @ 0x1a1a9d0] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1a9d0] probing device /dev/video21 > [mjpeg_v4l2m2m @ 0x1a1a9d0] driver 'exynos-gsc' on card 'exynos-gsc gscaler' > [mjpeg_v4l2m2m @ 0x1a1a9d0] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1a9d0] probing device /dev/video20 > [mjpeg_v4l2m2m @ 0x1a1a9d0] driver 'exynos-gsc' on card 'exynos-gsc gscaler' > [mjpeg_v4l2m2m @ 0x1a1a9d0] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1a9d0] probing device /dev/video11 > [mjpeg_v4l2m2m @ 0x1a1a9d0] driver 's5p-mfc' on card 's5p-mfc-enc' > [mjpeg_v4l2m2m @ 0x1a1a9d0] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1a9d0] probing device /dev/video10 > [mjpeg_v4l2m2m @ 0x1a1a9d0] driver 's5p-mfc' on card 's5p-mfc-dec' > [mjpeg_v4l2m2m @ 0x1a1a9d0] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1a9d0] probing device /dev/video33 > [mjpeg_v4l2m2m @ 0x1a1a9d0] driver 's5p-jpeg' on card 's5p-jpeg decoder' > [mjpeg_v4l2m2m @ 0x1a1a9d0] Using device /dev/video33 > [mjpeg_v4l2m2m @ 0x1a1a9d0] driver 's5p-jpeg' on card 's5p-jpeg decoder' > [mjpeg_v4l2m2m @ 0x1a1a9d0] output : JPEG 01 buffers initialized: 0000x0000, sizeimage 00230528, bytesperline 00000000 > Stream mapping: > Stream #0:0 -> #0:0 (mjpeg (mjpeg_v4l2m2m) -> wrapped_avframe (native)) > Press [q] to stop, [?] for help > cur_dts is invalid (this is harmless if it occurs once at the start per stream) > [mjpeg_v4l2m2m @ 0x1a1a9d0] capture: YUYV 01 buffers initialized: 0640x0480, sizeimage 00614400, bytesperline 00001280 > [mjpeg_v4l2m2m @ 0x1a1a9d0] output changed (0x0) -> (640x480) > [mjpeg_v4l2m2m @ 0x1a1a9d0] /dev/video33 full reinit > [mjpeg_v4l2m2m @ 0x1a1a9d0] output : JPEG 01 buffers initialized: 0640x0480, sizeimage 00230528, bytesperline 00000000 > cur_dts is invalid (this is harmless if it occurs once at the start per stream) > Last message repeated 498 times > [mjpeg_v4l2m2m @ 0x1a1a9d0] capture: YUYV 01 buffers initialized: 0640x0480, sizeimage 00614400, bytesperline 00001280 > ... > [graph 0 input from stream 0:0 @ 0x1a1fa90] Setting 'video_size' to value '640x480' > [graph 0 input from stream 0:0 @ 0x1a1fa90] Setting 'pix_fmt' to value '1' > [graph 0 input from stream 0:0 @ 0x1a1fa90] Setting 'time_base' to value '1/1000000' > [graph 0 input from stream 0:0 @ 0x1a1fa90] Setting 'pixel_aspect' to value '0/1' > [graph 0 input from stream 0:0 @ 0x1a1fa90] Setting 'sws_param' to value 'flags=2' > [graph 0 input from stream 0:0 @ 0x1a1fa90] Setting 'frame_rate' to value '30/1' > [graph 0 input from stream 0:0 @ 0x1a1fa90] w:640 h:480 pixfmt:yuyv422 tb:1/1000000 fr:30/1 sar:0/1 sws_param:flags=2 > [AVFilterGraph @ 0x1a1c080] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed > Output #0, null, to 'pipe:': > Metadata: > encoder : Lavf58.17.101 > Stream #0:0, 0, 1/30: Video: wrapped_avframe, 1 reference frame, yuyv422(center), 640x480, 0/1, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc > Metadata: > encoder : Lavc58.22.101 wrapped_avframe > > Thread 1 "ffmpeg_g" received signal SIGINT, Interrupt. > 0xb5d05bf0 in poll () at ../sysdeps/unix/syscall-template.S:84 > 84 ../sysdeps/unix/syscall-template.S: No such file or directory. > (gdb) bt > #0 0xb5d05bf0 in poll () at ../sysdeps/unix/syscall-template.S:84 > #1 0x00c45266 in v4l2_dequeue_v4l2buf (ctx=0x1a1e2c4, timeout=-1) at src/libavcodec/v4l2_context.c:283 > #2 0x00c45b00 in ff_v4l2_context_dequeue_frame (ctx=0x1a1e2c4, frame=0x1a1cf90) at src/libavcodec/v4l2_context.c:583 > #3 0x007a05da in v4l2_receive_frame (avctx=0x1a1a9d0, frame=0x1a1cf90) at src/libavcodec/v4l2_m2m_dec.c:160 > #4 0x00466d20 in decode_receive_frame_internal (avctx=0x1a1a9d0, frame=0x1a1cf90) at src/libavcodec/decode.c:645 > #5 0x00466ed4 in avcodec_send_packet (avctx=0x1a1a9d0, avpkt=0xbeffe930) at src/libavcodec/decode.c:709 > #6 0x0002cf0e in decode (avctx=0x1a1a9d0, frame=0x1a1e7a0, got_frame=0xbeffea78, pkt=0xbeffe930) at src/fftools/ffmpeg.c:2238 > #7 0x0002d4b2 in decode_video (ist=0x1a1f5a0, pkt=0xbeffea90, got_output=0xbeffea78, duration_pts=0xbeffea88, eof=0, decode_failed=0xbeffea80) at src/fftools/ffmpeg.c:2382 > #8 0x0002dfb6 in process_input_packet (ist=0x1a1f5a0, pkt=0xbeffecb0, no_eof=0) at src/fftools/ffmpeg.c:2623 > #9 0x00033e30 in process_input (file_index=0) at src/fftools/ffmpeg.c:4505 > #10 0x00034256 in transcode_step () at src/fftools/ffmpeg.c:4625 > #11 0x00034388 in transcode () at src/fftools/ffmpeg.c:4679 > #12 0x00034a38 in main (argc=19, argv=0xbefff654) at src/fftools/ffmpeg.c:4886 > (gdb) > > > Input from a file (made with "./ffmpeg_g -y -f lavfi -i 'smptebars=size=640x480:rate=30' -an -pix_fmt yuvj420p -c:v mjpeg -frames:v 10 test.mp4"): > > $ gdb --args ./ffmpeg_g -v 55 -c:v mjpeg_v4l2m2m -i test.mp4 -f null - > ... > [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1a19000] All info found > [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1a19000] After avformat_find_stream_info() pos: 5758 bytes read:66347 seeks:2 frames:1 > Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4': > Metadata: > major_brand : isom > minor_version : 512 > compatible_brands: isomiso2mp41 > encoder : Lavf58.17.101 > Duration: 00:00:00.33, start: 0.000000, bitrate: 1363 kb/s > Stream #0:0(und), 1, 1/15360: Video: mjpeg, 1 reference frame (mp4v / 0x7634706D), yuvj420p(pc, bt470bg/unknown/unknown, progressive, center), 640x480 [SAR 1:1 DAR 4:3], 0/1, 1345 kb/s, 30 fps, 30 tbr, 15360 tbn, 15360 tbc (default) > Metadata: > handler_name : VideoHandler > Successfully opened the file. > Parsing a group of options: output url -. > Applying option f (force format) with argument null. > Successfully parsed a group of options. > Opening an output file: -. > Successfully opened the file. > [mjpeg_v4l2m2m @ 0x1a1b220] probing device /dev/video0 > [mjpeg_v4l2m2m @ 0x1a1b220] driver 'uvcvideo' on card 'HD Pro Webcam C920' > [mjpeg_v4l2m2m @ 0x1a1b220] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1b220] probing device /dev/video21 > [mjpeg_v4l2m2m @ 0x1a1b220] driver 'exynos-gsc' on card 'exynos-gsc gscaler' > [mjpeg_v4l2m2m @ 0x1a1b220] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1b220] probing device /dev/video20 > [mjpeg_v4l2m2m @ 0x1a1b220] driver 'exynos-gsc' on card 'exynos-gsc gscaler' > [mjpeg_v4l2m2m @ 0x1a1b220] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1b220] probing device /dev/video11 > [mjpeg_v4l2m2m @ 0x1a1b220] driver 's5p-mfc' on card 's5p-mfc-enc' > [mjpeg_v4l2m2m @ 0x1a1b220] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1b220] probing device /dev/video10 > [mjpeg_v4l2m2m @ 0x1a1b220] driver 's5p-mfc' on card 's5p-mfc-dec' > [mjpeg_v4l2m2m @ 0x1a1b220] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a1b220] probing device /dev/video33 > [mjpeg_v4l2m2m @ 0x1a1b220] driver 's5p-jpeg' on card 's5p-jpeg decoder' > [mjpeg_v4l2m2m @ 0x1a1b220] Using device /dev/video33 > [mjpeg_v4l2m2m @ 0x1a1b220] driver 's5p-jpeg' on card 's5p-jpeg decoder' > [mjpeg_v4l2m2m @ 0x1a1b220] output : JPEG 01 buffers initialized: 0000x0000, sizeimage 00230528, bytesperline 00000000 > Stream mapping: > Stream #0:0 -> #0:0 (mjpeg (mjpeg_v4l2m2m) -> wrapped_avframe (native)) > Press [q] to stop, [?] for help > cur_dts is invalid (this is harmless if it occurs once at the start per stream) > [mjpeg_v4l2m2m @ 0x1a1b220] capture: YUYV 01 buffers initialized: 0640x0480, sizeimage 00614400, bytesperline 00001280 > [mjpeg_v4l2m2m @ 0x1a1b220] output changed (0x0) -> (640x480) > [mjpeg_v4l2m2m @ 0x1a1b220] /dev/video33 full reinit > [mjpeg_v4l2m2m @ 0x1a1b220] output : JPEG 01 buffers initialized: 0640x0480, sizeimage 00230528, bytesperline 00000000 > cur_dts is invalid (this is harmless if it occurs once at the start per stream) > [mjpeg_v4l2m2m @ 0x1a1b220] capture: YUYV 01 buffers initialized: 0640x0480, sizeimage 00614400, bytesperline 00001280 > ... > [graph 0 input from stream 0:0 @ 0x1a1fe10] Setting 'video_size' to value '640x480' > [graph 0 input from stream 0:0 @ 0x1a1fe10] Setting 'pix_fmt' to value '1' > [graph 0 input from stream 0:0 @ 0x1a1fe10] Setting 'time_base' to value '1/15360' > [graph 0 input from stream 0:0 @ 0x1a1fe10] Setting 'pixel_aspect' to value '1/1' > [graph 0 input from stream 0:0 @ 0x1a1fe10] Setting 'sws_param' to value 'flags=2' > [graph 0 input from stream 0:0 @ 0x1a1fe10] Setting 'frame_rate' to value '30/1' > [graph 0 input from stream 0:0 @ 0x1a1fe10] w:640 h:480 pixfmt:yuyv422 tb:1/15360 fr:30/1 sar:1/1 sws_param:flags=2 > [AVFilterGraph @ 0x1a1ad50] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed > Output #0, null, to 'pipe:': > Metadata: > major_brand : isom > minor_version : 512 > compatible_brands: isomiso2mp41 > encoder : Lavf58.17.101 > Stream #0:0(und), 0, 1/30: Video: wrapped_avframe, 1 reference frame, yuyv422(center), 640x480 [SAR 1:1 DAR 4:3], 0/1, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default) > Metadata: > handler_name : VideoHandler > encoder : Lavc58.22.101 wrapped_avframe > > Thread 1 "ffmpeg_g" received signal SIGINT, Interrupt. > 0xb5d05bf0 in poll () at ../sysdeps/unix/syscall-template.S:84 > 84 ../sysdeps/unix/syscall-template.S: No such file or directory. > (gdb) bt > #0 0xb5d05bf0 in poll () at ../sysdeps/unix/syscall-template.S:84 > #1 0x00c45266 in v4l2_dequeue_v4l2buf (ctx=0x1a1cd14, timeout=-1) at src/libavcodec/v4l2_context.c:283 > #2 0x00c45b00 in ff_v4l2_context_dequeue_frame (ctx=0x1a1cd14, frame=0x1a1ba20) at src/libavcodec/v4l2_context.c:583 > #3 0x007a05da in v4l2_receive_frame (avctx=0x1a1b220, frame=0x1a1ba20) at src/libavcodec/v4l2_m2m_dec.c:160 > #4 0x00466d20 in decode_receive_frame_internal (avctx=0x1a1b220, frame=0x1a1ba20) at src/libavcodec/decode.c:645 > #5 0x00466ed4 in avcodec_send_packet (avctx=0x1a1b220, avpkt=0xbeffe990) at src/libavcodec/decode.c:709 > #6 0x0002cf0e in decode (avctx=0x1a1b220, frame=0x1a1d1f0, got_frame=0xbeffead8, pkt=0xbeffe990) at src/fftools/ffmpeg.c:2238 > #7 0x0002d4b2 in decode_video (ist=0x1a1b090, pkt=0xbeffeaf0, got_output=0xbeffead8, duration_pts=0xbeffeae8, eof=0, decode_failed=0xbeffeae0) at src/fftools/ffmpeg.c:2382 > #8 0x0002dfb6 in process_input_packet (ist=0x1a1b090, pkt=0xbeffed10, no_eof=0) at src/fftools/ffmpeg.c:2623 > #9 0x00033e30 in process_input (file_index=0) at src/fftools/ffmpeg.c:4505 > #10 0x00034256 in transcode_step () at src/fftools/ffmpeg.c:4625 > #11 0x00034388 in transcode () at src/fftools/ffmpeg.c:4679 > #12 0x00034a38 in main (argc=10, argv=0xbefff6b4) at src/fftools/ffmpeg.c:4886 > (gdb) > > > For the second run it looks suspicious that it's making a YUYV (4:2:2) output from a YUV420P (4:2:0) input file, but given that it fails in the same way as the first one I'm not sure that's actually relevant. Is there anything else I should be trying here? AFAIK there is currently no way to select the CAPTURE pixfmt, so the current code keeps the default format returned by the driver from G_FMT. In this case s5p-jpeg sets YUYV as default, which explains that. I think we'll need a -capture_pixfmt option to try setting it ourselves, unless there's a way to retrieve the -pix_fmt ffmpeg output option to try and match it ? > > The encoder does work, but seems to have a similar problem where it hangs at end-of-file: > > $ gdb --args ./ffmpeg_g -y -v 55 -f lavfi -i 'smptebars=size=640x480:rate=30' -an -pix_fmt yuyv422 -c:v mjpeg_v4l2m2m -frames:v 10 out.mjpeg > ... > [lavfi @ 0x1a19050] All info found > Input #0, lavfi, from 'smptebars=size=640x480:rate=30': > Duration: N/A, start: 0.000000, bitrate: N/A > Stream #0:0, 1, 1/30: Video: rawvideo, 1 reference frame (I420 / 0x30323449), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 0/1, 30 tbr, 30 tbn, 30 tbc > Successfully opened the file. > Parsing a group of options: output url out.mjpeg. > Applying option an (disable audio) with argument 1. > Applying option pix_fmt (set pixel format) with argument yuyv422. > Applying option c:v (codec name) with argument mjpeg_v4l2m2m. > Applying option frames:v (set the number of frames to output) with argument 10. > Successfully parsed a group of options. > Opening an output file: out.mjpeg. > [file @ 0x1a43840] Setting default whitelist 'file,crypto' > Successfully opened the file. > Stream mapping: > Stream #0:0 -> #0:0 (rawvideo (native) -> mjpeg (mjpeg_v4l2m2m)) > Press [q] to stop, [?] for help > cur_dts is invalid (this is harmless if it occurs once at the start per stream) > [rawvideo @ 0x1a41c80] PACKET SIZE: 460800, STRIDE: 960 > ... > [graph 0 input from stream 0:0 @ 0x1a45210] Setting 'video_size' to value '640x480' > [graph 0 input from stream 0:0 @ 0x1a45210] Setting 'pix_fmt' to value '0' > [graph 0 input from stream 0:0 @ 0x1a45210] Setting 'time_base' to value '1/30' > [graph 0 input from stream 0:0 @ 0x1a45210] Setting 'pixel_aspect' to value '1/1' > [graph 0 input from stream 0:0 @ 0x1a45210] Setting 'sws_param' to value 'flags=2' > [graph 0 input from stream 0:0 @ 0x1a45210] Setting 'frame_rate' to value '30/1' > [graph 0 input from stream 0:0 @ 0x1a45210] w:640 h:480 pixfmt:yuv420p tb:1/30 fr:30/1 sar:1/1 sws_param:flags=2 > [format @ 0x1a45890] Setting 'pix_fmts' to value 'yuyv422' > [auto_scaler_0 @ 0x1a45a80] Setting 'flags' to value 'bicubic' > [auto_scaler_0 @ 0x1a45a80] w:iw h:ih flags:'bicubic' interl:0 > [format @ 0x1a45890] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format' > [AVFilterGraph @ 0x1a448f0] query_formats: 4 queried, 2 merged, 1 already done, 0 delayed > [auto_scaler_0 @ 0x1a45a80] w:640 h:480 fmt:yuv420p sar:1/1 -> w:640 h:480 fmt:yuyv422 sar:1/1 flags:0x4 > [mjpeg_v4l2m2m @ 0x1a42f70] probing device /dev/video0 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 'uvcvideo' on card 'HD Pro Webcam C920' > [mjpeg_v4l2m2m @ 0x1a42f70] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a42f70] probing device /dev/video21 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 'exynos-gsc' on card 'exynos-gsc gscaler' > [mjpeg_v4l2m2m @ 0x1a42f70] v4l2 capture format not supported > [mjpeg_v4l2m2m @ 0x1a42f70] probing device /dev/video20 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 'exynos-gsc' on card 'exynos-gsc gscaler' > [mjpeg_v4l2m2m @ 0x1a42f70] v4l2 capture format not supported > [mjpeg_v4l2m2m @ 0x1a42f70] probing device /dev/video11 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 's5p-mfc' on card 's5p-mfc-enc' > [mjpeg_v4l2m2m @ 0x1a42f70] v4l2 capture format not supported > [mjpeg_v4l2m2m @ 0x1a42f70] probing device /dev/video10 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 's5p-mfc' on card 's5p-mfc-dec' > [mjpeg_v4l2m2m @ 0x1a42f70] v4l2 output format not supported > [mjpeg_v4l2m2m @ 0x1a42f70] probing device /dev/video33 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 's5p-jpeg' on card 's5p-jpeg decoder' > [mjpeg_v4l2m2m @ 0x1a42f70] v4l2 capture format not supported > [mjpeg_v4l2m2m @ 0x1a42f70] probing device /dev/video32 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 's5p-jpeg' on card 's5p-jpeg encoder' > [mjpeg_v4l2m2m @ 0x1a42f70] Using device /dev/video32 > [mjpeg_v4l2m2m @ 0x1a42f70] driver 's5p-jpeg' on card 's5p-jpeg encoder' > [mjpeg_v4l2m2m @ 0x1a42f70] output : YUYV 16 buffers initialized: 0640x0480, sizeimage 00614400, bytesperline 00001280 > [mjpeg_v4l2m2m @ 0x1a42f70] capture: JPEG 04 buffers initialized: 0640x0480, sizeimage 00233472, bytesperline 00000000 > [mjpeg_v4l2m2m @ 0x1a42f70] Failed to set number of B-frames > Last message repeated 1 times > [mjpeg_v4l2m2m @ 0x1a42f70] Failed to set timeperframeFailed to set header mode > [mjpeg_v4l2m2m @ 0x1a42f70] Failed to set bit rate > [mjpeg_v4l2m2m @ 0x1a42f70] Failed to set gop size > [mjpeg_v4l2m2m @ 0x1a42f70] Encoder Context: id (7), profile (-99), frame rate(30/1), number b-frames (0), gop size (12), bit rate (200000), qmin (2), qmax (31) > Output #0, mjpeg, to 'out.mjpeg': > Metadata: > encoder : Lavf58.17.101 > Stream #0:0, 0, 1/30: Video: mjpeg (mjpeg_v4l2m2m), 1 reference frame, yuyv422, 640x480 [SAR 1:1 DAR 4:3], 0/1, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc > Metadata: > encoder : Lavc58.22.101 mjpeg_v4l2m2m > Clipping frame in rate conversion by 0.000008 > [rawvideo @ 0x1a41c80] PACKET SIZE: 460800, STRIDE: 960 > Last message repeated 3 times > [rawvideo @ 0x1a41c80] PACKET SIZE: 460800, STRIDE: 9600.16 bitrate=3769.2kbits/s speed=0.299x > Last message repeated 4 times > No more output streams to write to, finishing. > > Thread 1 "ffmpeg_g" received signal SIGINT, Interrupt. > 0xb5d05bf0 in poll () at ../sysdeps/unix/syscall-template.S:84 > 84 ../sysdeps/unix/syscall-template.S: No such file or directory. > (gdb) bt > #0 0xb5d05bf0 in poll () at ../sysdeps/unix/syscall-template.S:84 > #1 0x00c45266 in v4l2_dequeue_v4l2buf (ctx=0x1a64834, timeout=-1) at src/libavcodec/v4l2_context.c:283 > #2 0x00c45b4c in ff_v4l2_context_dequeue_packet (ctx=0x1a64834, pkt=0xbeffeef8) at src/libavcodec/v4l2_context.c:603 > #3 0x007a0dd2 in v4l2_receive_packet (avctx=0x1a42f70, avpkt=0xbeffeef8) at src/libavcodec/v4l2_m2m_enc.c:278 > #4 0x004a50fa in avcodec_receive_packet (avctx=0x1a42f70, avpkt=0xbeffeef8) at src/libavcodec/encode.c:430 > #5 0x0002c260 in flush_encoders () at src/fftools/ffmpeg.c:1938 > #6 0x0003444e in transcode () at src/fftools/ffmpeg.c:4699 > #7 0x00034a38 in main (argc=16, argv=0xbefff654) at src/fftools/ffmpeg.c:4886 > (gdb) > > The output file is correct here, though it's truncated because libavformat doesn't get an opportunity to finish it. (I had to give it a YUYV input explicitly - the output is messed up with another input format, but I don't think that's related to this patch.) I've glanced at the s5p-jpeg driver and I haven't seen any code to handle EOS (things like CMD_STOP or an empty input buffer), so I'm actually curious as to how they usually handle EOS with this driver. > Thanks, > > - Mark > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 13/08/18 00:33, Maxime Jourdan wrote: > Hi Jorge, unfortunately don't have a SBC that features a V4L2 M2M > MJPEG/JPEG encoder so I couldn't test it. > > Hi Mark, thanks for the extensive review. Too bad neither scenario worked fully. > > 2018-08-12 19:24 GMT+02:00 Mark Thompson <sw@jkqxz.net>: >> On 12/08/18 15:40, Maxime Jourdan wrote: >>> Tested on an Odroid-C2 with a V4L2 M2M MJPEG decoder. >>> >>> Signed-off-by: Maxime Jourdan <maxi.jourdan@wanadoo.fr> >>> --- >>> configure | 3 +++ >>> libavcodec/Makefile | 2 ++ >>> libavcodec/allcodecs.c | 2 ++ >>> libavcodec/v4l2_m2m_dec.c | 1 + >>> libavcodec/v4l2_m2m_enc.c | 1 + >>> 5 files changed, 9 insertions(+) >>> >>> ... >>> diff --git a/libavcodec/v4l2_m2m_enc.c b/libavcodec/v4l2_m2m_enc.c >>> index 4c9ea1fd92..b025f59275 100644 >>> --- a/libavcodec/v4l2_m2m_enc.c >>> +++ b/libavcodec/v4l2_m2m_enc.c >>> @@ -345,6 +345,7 @@ AVCodec ff_ ## NAME ## _v4l2m2m_encoder = { \ >>> .wrapper_name = "v4l2m2m", \ >>> }; >>> >>> +M2MENC(mjpeg,"MJPEG", AV_CODEC_ID_MJPEG); >>> M2MENC(mpeg4,"MPEG4", AV_CODEC_ID_MPEG4); >>> M2MENC(h263, "H.263", AV_CODEC_ID_H263); >>> M2MENC(h264, "H.264", AV_CODEC_ID_H264); >> >> Since MJPEG is intra-only, it probably wants to avoid setting options like GOP size and B-frames (currently this gives me warnings for each of these options that it failed to set them, visible in log below). > > Ack, although right now I'm not sure what's the cleanest way to enforce this. I guess make an intra-only flag based on the codec and then add "if (!intra-only)" around applying those settings in the encoder. >> Trying this on an Odroid XU4 (Exynos S5P running Linux 4.14.55), it looks like we need to somehow allow both "MJPG" and "JPEG" as fourccs. Reordering the two lines in libavcodec/v4l2_fmt.c to put "JPEG" first makes it get past that test, but obviously that will stop it from working on an "MJPG" device. > > I see, so some rework will be needed to account for AV_CODECs that > appear multiple times in the list. Yeah, probably. (Assuming there isn't some special difference between them which I have missed.) >> >> So, for testing I used this patch: >> >> diff --git a/libavcodec/v4l2_fmt.c b/libavcodec/v4l2_fmt.c >> index 6df47e3..06b57db 100644 >> --- a/libavcodec/v4l2_fmt.c >> +++ b/libavcodec/v4l2_fmt.c >> @@ -51,8 +51,8 @@ static const struct fmt_conversion { >> { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV410) }, >> { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YVU410) }, >> { AV_FMT(NV12), AV_CODEC(RAWVIDEO), V4L2_FMT(NV12) }, >> - { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, >> { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(JPEG) }, >> + { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, >> #ifdef V4L2_PIX_FMT_SRGGB8 >> { AV_FMT(BAYER_BGGR8), AV_CODEC(RAWVIDEO), V4L2_FMT(SBGGR8) }, >> { AV_FMT(BAYER_GBRG8), AV_CODEC(RAWVIDEO), V4L2_FMT(SGBRG8) }, >> >> >> After getting past that, the decoder seems to hang for me pretty much immediately on a DQBUF call. > > Did dmesg have anything interesting to say ? Huh, I did not think to look there. Running again, I get the following when SIGINTing to kill the decoder (nothing before that): [ 196.346037] ------------[ cut here ]------------ [ 196.346137] WARNING: CPU: 0 PID: 972 at drivers/media/v4l2-core/videobuf2-core.c:1655 __vb2_queue_cancel+0x17c/0x1e4 [ 196.346154] Modules linked in: cpufreq_conservative cpufreq_userspace cpufreq_powersave spidev spi_s3c64xx evdev gpio_keys uio_pdrv_genirq uio extcon_usb_gpio exynos_gpiomem uvcvideo snd_usb_audio snd_hwdep snd_usbmidi_lib videobuf2_vmalloc snd_rawmidi snd_seq_device ipv6 [ 196.346424] CPU: 0 PID: 972 Comm: ffmpeg_g Tainted: G W 4.14.55-133 #1 [ 196.346441] Hardware name: ODROID-XU4 [ 196.346522] [<c0110b14>] (unwind_backtrace) from [<c010ce98>] (show_stack+0x10/0x14) [ 196.346585] [<c010ce98>] (show_stack) from [<c08a9c94>] (dump_stack+0x84/0x98) [ 196.346655] [<c08a9c94>] (dump_stack) from [<c01242cc>] (__warn+0xec/0x104) [ 196.346717] [<c01242cc>] (__warn) from [<c0124394>] (warn_slowpath_null+0x20/0x28) [ 196.346780] [<c0124394>] (warn_slowpath_null) from [<c06adf14>] (__vb2_queue_cancel+0x17c/0x1e4) [ 196.346846] [<c06adf14>] (__vb2_queue_cancel) from [<c06ae5d4>] (vb2_core_queue_release+0x18/0x38) [ 196.346906] [<c06ae5d4>] (vb2_core_queue_release) from [<c06abc84>] (v4l2_m2m_ctx_release+0x1c/0x28) [ 196.346969] [<c06abc84>] (v4l2_m2m_ctx_release) from [<c06cb514>] (s5p_jpeg_release+0x28/0x58) [ 196.347030] [<c06cb514>] (s5p_jpeg_release) from [<c069716c>] (v4l2_release+0x38/0x74) [ 196.347096] [<c069716c>] (v4l2_release) from [<c0286704>] (__fput+0x84/0x1cc) [ 196.347156] [<c0286704>] (__fput) from [<c0141d40>] (task_work_run+0x90/0xb0) [ 196.347209] [<c0141d40>] (task_work_run) from [<c01291cc>] (do_exit+0x390/0xb50) [ 196.347259] [<c01291cc>] (do_exit) from [<c01299f4>] (do_group_exit+0x3c/0xbc) [ 196.347310] [<c01299f4>] (do_group_exit) from [<c01342c8>] (get_signal+0x300/0x678) [ 196.347373] [<c01342c8>] (get_signal) from [<c010bfb8>] (do_signal+0x64/0x430) [ 196.347433] [<c010bfb8>] (do_signal) from [<c010c540>] (do_work_pending+0xac/0xc4) [ 196.347491] [<c010c540>] (do_work_pending) from [<c01087ec>] (slow_work_pending+0xc/0x20) [ 196.348088] ---[ end trace a4a2a65a0e7292e0 ]--- >> Input from a UVC webcam (Logitech C920, making 640x480 YUV 4:2:2 JPEGs): >> >> $ gdb --args ./ffmpeg_g -y -v 55 -f v4l2 -format mjpeg -framerate 30 -video_size 640x480 -c:v mjpeg_v4l2m2m -i /dev/video0 -f null - >> ... >> >> >> Input from a file (made with "./ffmpeg_g -y -f lavfi -i 'smptebars=size=640x480:rate=30' -an -pix_fmt yuvj420p -c:v mjpeg -frames:v 10 test.mp4"): >> >> $ gdb --args ./ffmpeg_g -v 55 -c:v mjpeg_v4l2m2m -i test.mp4 -f null - >> ... >> >> >> For the second run it looks suspicious that it's making a YUYV (4:2:2) output from a YUV420P (4:2:0) input file, but given that it fails in the same way as the first one I'm not sure that's actually relevant. Is there anything else I should be trying here? > > AFAIK there is currently no way to select the CAPTURE pixfmt, so the > current code keeps the default format returned by the driver from > G_FMT. In this case s5p-jpeg sets YUYV as default, which explains > that. > > I think we'll need a -capture_pixfmt option to try setting it > ourselves, unless there's a way to retrieve the -pix_fmt ffmpeg output > option to try and match it ? If the device can tell us what the output format should be then a decoder can just output that pixfmt. The problem comes when it can't actually tell us in advance, but does offer a list of possible formats: it seems likely that only one of those choices will actually work - the YUV 4:2:0 JPEGs probably won't upsample into a YUYV output buffer, rather you need to pick the right output format for them. That might require parsing some of the input stream to decide which possible output formats actually match the input. Maybe it's similar to how the VAAPI hwaccel works for JPEG - the Intel driver offers a set of possible formats, but you need to pick the one which has the same chroma subsampling as your input stream for it to actually work. (Pushing the choice onto the user as an extra option doesn't seem right to me.) >> >> The encoder does work, but seems to have a similar problem where it hangs at end-of-file: >> >> $ gdb --args ./ffmpeg_g -y -v 55 -f lavfi -i 'smptebars=size=640x480:rate=30' -an -pix_fmt yuyv422 -c:v mjpeg_v4l2m2m -frames:v 10 out.mjpeg Encoder kernel outout: The format test produces some spam when the encoder is starting: [ 678.815745] vidioc_try_fmt:1070: failed to try output format [ 678.820253] vidioc_try_fmt:1070: failed to try output format [ 678.830969] vidioc_try_fmt:387: Unsupported format for source. [ 678.835371] vidioc_try_fmt:387: Unsupported format for source. [ 678.841224] vidioc_try_fmt:387: Unsupported format for source. [ 678.847002] vidioc_try_fmt:387: Unsupported format for source. [ 678.852840] vidioc_try_fmt:387: Unsupported format for source. [ 678.858609] vidioc_try_fmt:387: Unsupported format for source. [ 678.864452] vidioc_try_fmt:387: Unsupported format for source. [ 678.870227] vidioc_try_fmt:387: Unsupported format for source. [ 678.876061] vidioc_try_fmt:387: Unsupported format for source. [ 678.881838] vidioc_try_fmt:387: Unsupported format for source. [ 678.887678] vidioc_try_fmt:387: Unsupported format for source. [ 678.894671] s5p-jpeg 11f60000.jpeg: Fourcc format (0x31324d4e) invalid. And then we get a similar backtrace to the decoder when interrupting it: [ 689.648208] ------------[ cut here ]------------ [ 689.648237] WARNING: CPU: 4 PID: 995 at drivers/media/v4l2-core/videobuf2-core.c:1655 __vb2_queue_cancel+0x17c/0x1e4 [ 689.648243] Modules linked in: cpufreq_conservative cpufreq_userspace cpufreq_powersave spidev spi_s3c64xx evdev gpio_keys uio_pdrv_genirq uio extcon_usb_gpio exynos_gpiomem uvcvideo snd_usb_audio snd_hwdep snd_usbmidi_lib videobuf2_vmalloc snd_rawmidi snd_seq_device ipv6 [ 689.648356] CPU: 4 PID: 995 Comm: ffmpeg_g Tainted: G W 4.14.55-133 #1 [ 689.648363] Hardware name: ODROID-XU4 [ 689.648384] [<c0110b14>] (unwind_backtrace) from [<c010ce98>] (show_stack+0x10/0x14) [ 689.648400] [<c010ce98>] (show_stack) from [<c08a9c94>] (dump_stack+0x84/0x98) [ 689.648417] [<c08a9c94>] (dump_stack) from [<c01242cc>] (__warn+0xec/0x104) [ 689.648432] [<c01242cc>] (__warn) from [<c0124394>] (warn_slowpath_null+0x20/0x28) [ 689.648447] [<c0124394>] (warn_slowpath_null) from [<c06adf14>] (__vb2_queue_cancel+0x17c/0x1e4) [ 689.648462] [<c06adf14>] (__vb2_queue_cancel) from [<c06ae5d4>] (vb2_core_queue_release+0x18/0x38) [ 689.648475] [<c06ae5d4>] (vb2_core_queue_release) from [<c06abc7c>] (v4l2_m2m_ctx_release+0x14/0x28) [ 689.648490] [<c06abc7c>] (v4l2_m2m_ctx_release) from [<c06cb514>] (s5p_jpeg_release+0x28/0x58) [ 689.648503] [<c06cb514>] (s5p_jpeg_release) from [<c069716c>] (v4l2_release+0x38/0x74) [ 689.648518] [<c069716c>] (v4l2_release) from [<c0286704>] (__fput+0x84/0x1cc) [ 689.648533] [<c0286704>] (__fput) from [<c0141d40>] (task_work_run+0x90/0xb0) [ 689.648545] [<c0141d40>] (task_work_run) from [<c01291cc>] (do_exit+0x390/0xb50) [ 689.648556] [<c01291cc>] (do_exit) from [<c01299f4>] (do_group_exit+0x3c/0xbc) [ 689.648568] [<c01299f4>] (do_group_exit) from [<c01342c8>] (get_signal+0x300/0x678) [ 689.648582] [<c01342c8>] (get_signal) from [<c010bfb8>] (do_signal+0x64/0x430) [ 689.648595] [<c010bfb8>] (do_signal) from [<c010c540>] (do_work_pending+0xac/0xc4) [ 689.648608] [<c010c540>] (do_work_pending) from [<c01087ec>] (slow_work_pending+0xc/0x20) [ 689.648617] ---[ end trace a4a2a65a0e7292e1 ]--- >> The output file is correct here, though it's truncated because libavformat doesn't get an opportunity to finish it. (I had to give it a YUYV input explicitly - the output is messed up with another input format, but I don't think that's related to this patch.) > > I've glanced at the s5p-jpeg driver and I haven't seen any code to > handle EOS (things like CMD_STOP or an empty input buffer), so I'm > actually curious as to how they usually handle EOS with this driver. (Note that it's not required that it works on this particular hardware to apply the patch, but it may well indicate problems and cause confusion later if it doesn't. It's definitely a good idea to see it working on more than one implementation, though.) Thanks, - Mark
2018-08-13 2:26 GMT+02:00 Mark Thompson <sw@jkqxz.net>: > On 13/08/18 00:33, Maxime Jourdan wrote: >> Hi Jorge, unfortunately don't have a SBC that features a V4L2 M2M >> MJPEG/JPEG encoder so I couldn't test it. >> >> Hi Mark, thanks for the extensive review. Too bad neither scenario worked fully. >> >> 2018-08-12 19:24 GMT+02:00 Mark Thompson <sw@jkqxz.net>: >>> On 12/08/18 15:40, Maxime Jourdan wrote: >>>> Tested on an Odroid-C2 with a V4L2 M2M MJPEG decoder. >>>> >>>> Signed-off-by: Maxime Jourdan <maxi.jourdan@wanadoo.fr> >>>> --- >>>> configure | 3 +++ >>>> libavcodec/Makefile | 2 ++ >>>> libavcodec/allcodecs.c | 2 ++ >>>> libavcodec/v4l2_m2m_dec.c | 1 + >>>> libavcodec/v4l2_m2m_enc.c | 1 + >>>> 5 files changed, 9 insertions(+) >>>> >>>> ... >>>> diff --git a/libavcodec/v4l2_m2m_enc.c b/libavcodec/v4l2_m2m_enc.c >>>> index 4c9ea1fd92..b025f59275 100644 >>>> --- a/libavcodec/v4l2_m2m_enc.c >>>> +++ b/libavcodec/v4l2_m2m_enc.c >>>> @@ -345,6 +345,7 @@ AVCodec ff_ ## NAME ## _v4l2m2m_encoder = { \ >>>> .wrapper_name = "v4l2m2m", \ >>>> }; >>>> >>>> +M2MENC(mjpeg,"MJPEG", AV_CODEC_ID_MJPEG); >>>> M2MENC(mpeg4,"MPEG4", AV_CODEC_ID_MPEG4); >>>> M2MENC(h263, "H.263", AV_CODEC_ID_H263); >>>> M2MENC(h264, "H.264", AV_CODEC_ID_H264); >>> >>> Since MJPEG is intra-only, it probably wants to avoid setting options like GOP size and B-frames (currently this gives me warnings for each of these options that it failed to set them, visible in log below). >> >> Ack, although right now I'm not sure what's the cleanest way to enforce this. > > I guess make an intra-only flag based on the codec and then add "if (!intra-only)" around applying those settings in the encoder. Sounds good >>> Trying this on an Odroid XU4 (Exynos S5P running Linux 4.14.55), it looks like we need to somehow allow both "MJPG" and "JPEG" as fourccs. Reordering the two lines in libavcodec/v4l2_fmt.c to put "JPEG" first makes it get past that test, but obviously that will stop it from working on an "MJPG" device. >> >> I see, so some rework will be needed to account for AV_CODECs that >> appear multiple times in the list. > > Yeah, probably. (Assuming there isn't some special difference between them which I have missed.) > >>> >>> So, for testing I used this patch: >>> >>> diff --git a/libavcodec/v4l2_fmt.c b/libavcodec/v4l2_fmt.c >>> index 6df47e3..06b57db 100644 >>> --- a/libavcodec/v4l2_fmt.c >>> +++ b/libavcodec/v4l2_fmt.c >>> @@ -51,8 +51,8 @@ static const struct fmt_conversion { >>> { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV410) }, >>> { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YVU410) }, >>> { AV_FMT(NV12), AV_CODEC(RAWVIDEO), V4L2_FMT(NV12) }, >>> - { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, >>> { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(JPEG) }, >>> + { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, >>> #ifdef V4L2_PIX_FMT_SRGGB8 >>> { AV_FMT(BAYER_BGGR8), AV_CODEC(RAWVIDEO), V4L2_FMT(SBGGR8) }, >>> { AV_FMT(BAYER_GBRG8), AV_CODEC(RAWVIDEO), V4L2_FMT(SGBRG8) }, >>> >>> >>> After getting past that, the decoder seems to hang for me pretty much immediately on a DQBUF call. >> >> Did dmesg have anything interesting to say ? > > Huh, I did not think to look there. > > Running again, I get the following when SIGINTing to kill the decoder (nothing before that): > > [ 196.346037] ------------[ cut here ]------------ > [ 196.346137] WARNING: CPU: 0 PID: 972 at drivers/media/v4l2-core/videobuf2-core.c:1655 __vb2_queue_cancel+0x17c/0x1e4 > [ 196.346154] Modules linked in: cpufreq_conservative cpufreq_userspace cpufreq_powersave spidev spi_s3c64xx evdev gpio_keys uio_pdrv_genirq uio extcon_usb_gpio exynos_gpiomem uvcvideo snd_usb_audio snd_hwdep snd_usbmidi_lib videobuf2_vmalloc snd_rawmidi snd_seq_device ipv6 > [ 196.346424] CPU: 0 PID: 972 Comm: ffmpeg_g Tainted: G W 4.14.55-133 #1 > [ 196.346441] Hardware name: ODROID-XU4 > [ 196.346522] [<c0110b14>] (unwind_backtrace) from [<c010ce98>] (show_stack+0x10/0x14) > [ 196.346585] [<c010ce98>] (show_stack) from [<c08a9c94>] (dump_stack+0x84/0x98) > [ 196.346655] [<c08a9c94>] (dump_stack) from [<c01242cc>] (__warn+0xec/0x104) > [ 196.346717] [<c01242cc>] (__warn) from [<c0124394>] (warn_slowpath_null+0x20/0x28) > [ 196.346780] [<c0124394>] (warn_slowpath_null) from [<c06adf14>] (__vb2_queue_cancel+0x17c/0x1e4) > [ 196.346846] [<c06adf14>] (__vb2_queue_cancel) from [<c06ae5d4>] (vb2_core_queue_release+0x18/0x38) > [ 196.346906] [<c06ae5d4>] (vb2_core_queue_release) from [<c06abc84>] (v4l2_m2m_ctx_release+0x1c/0x28) > [ 196.346969] [<c06abc84>] (v4l2_m2m_ctx_release) from [<c06cb514>] (s5p_jpeg_release+0x28/0x58) > [ 196.347030] [<c06cb514>] (s5p_jpeg_release) from [<c069716c>] (v4l2_release+0x38/0x74) > [ 196.347096] [<c069716c>] (v4l2_release) from [<c0286704>] (__fput+0x84/0x1cc) > [ 196.347156] [<c0286704>] (__fput) from [<c0141d40>] (task_work_run+0x90/0xb0) > [ 196.347209] [<c0141d40>] (task_work_run) from [<c01291cc>] (do_exit+0x390/0xb50) > [ 196.347259] [<c01291cc>] (do_exit) from [<c01299f4>] (do_group_exit+0x3c/0xbc) > [ 196.347310] [<c01299f4>] (do_group_exit) from [<c01342c8>] (get_signal+0x300/0x678) > [ 196.347373] [<c01342c8>] (get_signal) from [<c010bfb8>] (do_signal+0x64/0x430) > [ 196.347433] [<c010bfb8>] (do_signal) from [<c010c540>] (do_work_pending+0xac/0xc4) > [ 196.347491] [<c010c540>] (do_work_pending) from [<c01087ec>] (slow_work_pending+0xc/0x20) > [ 196.348088] ---[ end trace a4a2a65a0e7292e0 ]--- Thanks for looking, unfortunately those dmesg traces don't bring much more to the table :< . >>> Input from a UVC webcam (Logitech C920, making 640x480 YUV 4:2:2 JPEGs): >>> >>> $ gdb --args ./ffmpeg_g -y -v 55 -f v4l2 -format mjpeg -framerate 30 -video_size 640x480 -c:v mjpeg_v4l2m2m -i /dev/video0 -f null - >>> ... >>> >>> >>> Input from a file (made with "./ffmpeg_g -y -f lavfi -i 'smptebars=size=640x480:rate=30' -an -pix_fmt yuvj420p -c:v mjpeg -frames:v 10 test.mp4"): >>> >>> $ gdb --args ./ffmpeg_g -v 55 -c:v mjpeg_v4l2m2m -i test.mp4 -f null - >>> ... >>> >>> >>> For the second run it looks suspicious that it's making a YUYV (4:2:2) output from a YUV420P (4:2:0) input file, but given that it fails in the same way as the first one I'm not sure that's actually relevant. Is there anything else I should be trying here? >> >> AFAIK there is currently no way to select the CAPTURE pixfmt, so the >> current code keeps the default format returned by the driver from >> G_FMT. In this case s5p-jpeg sets YUYV as default, which explains >> that. >> >> I think we'll need a -capture_pixfmt option to try setting it >> ourselves, unless there's a way to retrieve the -pix_fmt ffmpeg output >> option to try and match it ? > > If the device can tell us what the output format should be then a decoder can just output that pixfmt. > > The problem comes when it can't actually tell us in advance, but does offer a list of possible formats: it seems likely that only one of those choices will actually work - the YUV 4:2:0 JPEGs probably won't upsample into a YUYV output buffer, rather you need to pick the right output format for them. That might require parsing some of the input stream to decide which possible output formats actually match the input. Maybe it's similar to how the VAAPI hwaccel works for JPEG - the Intel driver offers a set of possible formats, but you need to pick the one which has the same chroma subsampling as your input stream for it to actually work. > > (Pushing the choice onto the user as an extra option doesn't seem right to me.) I agree, but this is how it's done in V4L2 as well: everything must be set prior to decoding. I think the most crucial aspect is that you need to have the buffers - with the correct size/planes, which depend on the pixfmt - ready before decoding. So, some code will be needed to match the jpeg's pixfmt to the driver's CAPTURE pixfmt. >>> >>> The encoder does work, but seems to have a similar problem where it hangs at end-of-file: >>> >>> $ gdb --args ./ffmpeg_g -y -v 55 -f lavfi -i 'smptebars=size=640x480:rate=30' -an -pix_fmt yuyv422 -c:v mjpeg_v4l2m2m -frames:v 10 out.mjpeg > > Encoder kernel outout: > > The format test produces some spam when the encoder is starting: > > [ 678.815745] vidioc_try_fmt:1070: failed to try output format > [ 678.820253] vidioc_try_fmt:1070: failed to try output format > [ 678.830969] vidioc_try_fmt:387: Unsupported format for source. > [ 678.835371] vidioc_try_fmt:387: Unsupported format for source. > [ 678.841224] vidioc_try_fmt:387: Unsupported format for source. > [ 678.847002] vidioc_try_fmt:387: Unsupported format for source. > [ 678.852840] vidioc_try_fmt:387: Unsupported format for source. > [ 678.858609] vidioc_try_fmt:387: Unsupported format for source. > [ 678.864452] vidioc_try_fmt:387: Unsupported format for source. > [ 678.870227] vidioc_try_fmt:387: Unsupported format for source. > [ 678.876061] vidioc_try_fmt:387: Unsupported format for source. > [ 678.881838] vidioc_try_fmt:387: Unsupported format for source. > [ 678.887678] vidioc_try_fmt:387: Unsupported format for source. > [ 678.894671] s5p-jpeg 11f60000.jpeg: Fourcc format (0x31324d4e) invalid. > > And then we get a similar backtrace to the decoder when interrupting it: > > [ 689.648208] ------------[ cut here ]------------ > [ 689.648237] WARNING: CPU: 4 PID: 995 at drivers/media/v4l2-core/videobuf2-core.c:1655 __vb2_queue_cancel+0x17c/0x1e4 > [ 689.648243] Modules linked in: cpufreq_conservative cpufreq_userspace cpufreq_powersave spidev spi_s3c64xx evdev gpio_keys uio_pdrv_genirq uio extcon_usb_gpio exynos_gpiomem uvcvideo snd_usb_audio snd_hwdep snd_usbmidi_lib videobuf2_vmalloc snd_rawmidi snd_seq_device ipv6 > [ 689.648356] CPU: 4 PID: 995 Comm: ffmpeg_g Tainted: G W 4.14.55-133 #1 > [ 689.648363] Hardware name: ODROID-XU4 > [ 689.648384] [<c0110b14>] (unwind_backtrace) from [<c010ce98>] (show_stack+0x10/0x14) > [ 689.648400] [<c010ce98>] (show_stack) from [<c08a9c94>] (dump_stack+0x84/0x98) > [ 689.648417] [<c08a9c94>] (dump_stack) from [<c01242cc>] (__warn+0xec/0x104) > [ 689.648432] [<c01242cc>] (__warn) from [<c0124394>] (warn_slowpath_null+0x20/0x28) > [ 689.648447] [<c0124394>] (warn_slowpath_null) from [<c06adf14>] (__vb2_queue_cancel+0x17c/0x1e4) > [ 689.648462] [<c06adf14>] (__vb2_queue_cancel) from [<c06ae5d4>] (vb2_core_queue_release+0x18/0x38) > [ 689.648475] [<c06ae5d4>] (vb2_core_queue_release) from [<c06abc7c>] (v4l2_m2m_ctx_release+0x14/0x28) > [ 689.648490] [<c06abc7c>] (v4l2_m2m_ctx_release) from [<c06cb514>] (s5p_jpeg_release+0x28/0x58) > [ 689.648503] [<c06cb514>] (s5p_jpeg_release) from [<c069716c>] (v4l2_release+0x38/0x74) > [ 689.648518] [<c069716c>] (v4l2_release) from [<c0286704>] (__fput+0x84/0x1cc) > [ 689.648533] [<c0286704>] (__fput) from [<c0141d40>] (task_work_run+0x90/0xb0) > [ 689.648545] [<c0141d40>] (task_work_run) from [<c01291cc>] (do_exit+0x390/0xb50) > [ 689.648556] [<c01291cc>] (do_exit) from [<c01299f4>] (do_group_exit+0x3c/0xbc) > [ 689.648568] [<c01299f4>] (do_group_exit) from [<c01342c8>] (get_signal+0x300/0x678) > [ 689.648582] [<c01342c8>] (get_signal) from [<c010bfb8>] (do_signal+0x64/0x430) > [ 689.648595] [<c010bfb8>] (do_signal) from [<c010c540>] (do_work_pending+0xac/0xc4) > [ 689.648608] [<c010c540>] (do_work_pending) from [<c01087ec>] (slow_work_pending+0xc/0x20) > [ 689.648617] ---[ end trace a4a2a65a0e7292e1 ]--- > >>> The output file is correct here, though it's truncated because libavformat doesn't get an opportunity to finish it. (I had to give it a YUYV input explicitly - the output is messed up with another input format, but I don't think that's related to this patch.) >> >> I've glanced at the s5p-jpeg driver and I haven't seen any code to >> handle EOS (things like CMD_STOP or an empty input buffer), so I'm >> actually curious as to how they usually handle EOS with this driver. > > (Note that it's not required that it works on this particular hardware to apply the patch, but it may well indicate problems and cause confusion later if it doesn't. It's definitely a good idea to see it working on more than one implementation, though.) > Even on the C2 it's not perfect. For instance, I tried to feed 4:2:2 MJPEG to the decoder and it produced garbage: turns out it only supports yuv420p. I'm thinking that ffmpeg shouldn't even try to decode with v4l2 if the jpeg's pixfmt isn't in the CAPTURE pixfmt list. And in case the driver supports up/downsampling, well that's too bad because we won't be able to know. Btw, this issue is not specific to MJPEG. Some HW decoders can only do H.264 4:2:0 8-bit, but in the current code ffmpeg will happily send H.264 4:2:2 10-bit to a v4l2 decoder that doesn't support it. You can only pray that the HW decoder has some kind of failsafe to abort decoding if it doesn't support the input. I need to think more about this, but on the V4L2 side we should have fine-tuned input parameters that allow specifying more than just the compression standard. > Thanks, > > - Mark > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel Maxime
diff --git a/libavcodec/v4l2_fmt.c b/libavcodec/v4l2_fmt.c index 6df47e3..06b57db 100644 --- a/libavcodec/v4l2_fmt.c +++ b/libavcodec/v4l2_fmt.c @@ -51,8 +51,8 @@ static const struct fmt_conversion { { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YUV410) }, { AV_FMT(YUV410P), AV_CODEC(RAWVIDEO), V4L2_FMT(YVU410) }, { AV_FMT(NV12), AV_CODEC(RAWVIDEO), V4L2_FMT(NV12) }, - { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(JPEG) }, + { AV_FMT(NONE), AV_CODEC(MJPEG), V4L2_FMT(MJPEG) }, #ifdef V4L2_PIX_FMT_SRGGB8 { AV_FMT(BAYER_BGGR8), AV_CODEC(RAWVIDEO), V4L2_FMT(SBGGR8) }, { AV_FMT(BAYER_GBRG8), AV_CODEC(RAWVIDEO), V4L2_FMT(SGBRG8) },