| Message ID | 1503660168-24757-5-git-send-email-jorge.ramirez-ortiz@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
That looks generally OK. Is there any chance a hwaccel approach would be possible instead? If I've learned anything about hardware decoding, then that hwaccel is vastly superior to vendor-implemented full stream decoders. I don't think I like the attempt of sharing the v4l helper functions between libavdevice and libavcodec, but I can't tell how much it helps. On Fri, 25 Aug 2017 13:22:48 +0200 Jorge Ramirez-Ortiz <jorge.ramirez-ortiz@linaro.org> wrote: > +#define WIDTH(__ctx, __fmt) \ > + (V4L2_TYPE_IS_MULTIPLANAR((__ctx)->type) ? __fmt.fmt.pix_mp.width : __fmt.fmt.pix.width) > + > +#define HEIGHT(__ctx, __fmt) \ > + (V4L2_TYPE_IS_MULTIPLANAR((__ctx)->type) ? __fmt.fmt.pix_mp.height : __fmt.fmt.pix.height) These names are a bit generic. Also, identifiers starting with __ are always implementation reserved (i.e. using them undefined behavior). You're forgetting to quote the __fmt macro parameter too. > +static inline void set_pts(V4L2Buffer *out, int64_t pts) > +{ > + if (pts == AV_NOPTS_VALUE) { > + /* invalid timestamp: not sure how to handle this case */ > + out->timestamp.tv_sec = 0; > + out->timestamp.tv_usec = 0; > + } else { > + AVRational v4l2_timebase = { 1, 1000000 }; > + int64_t v4l2_pts = av_rescale_q(pts, out->context->time_base, v4l2_timebase); > + out->timestamp.tv_sec = v4l2_pts / INT64_C(1000000); > + out->timestamp.tv_usec = v4l2_pts % INT64_C(1000000); > + } > +} Why does it require a fixed timebase? A decoder shouldn't even look at the timestamps, it should only pass them though. Also, not using DTS will make it a nightmare to support containers like avi. I suspect the decoder tries to "fix" timestamps, or maybe even does something particularly bad like reordering frames by timestamps. This is NOT something that should be in a kernel API. (FFmpeg native decoders _and_ hwaccels pass through both PTS and DTS, and don't touch their values.) > +static void free_v4l2buf_cb(void *opaque, uint8_t *unused) > +{ > + V4L2Buffer* avbuf = opaque; > + > + if (V4L2BUF_IN_DRIVER == avbuf->status) > + return; > + > + if (V4L2_TYPE_IS_OUTPUT(avbuf->context->type)) > + avbuf->status = V4L2BUF_AVAILABLE; > + else > + avbuf->context->ops.enqueue(avbuf); > +} > +static inline int buffer_ops_v4l2buf_to_bufref(V4L2Buffer *in, int plane, AVBufferRef **buf) > +{ > + if (plane >= in->num_planes) > + return AVERROR(EINVAL); > + > + /* even though most encoders return 0 in data_offset encoding vp8 does require this value*/ > + *buf = av_buffer_create((char *)in->plane_info[plane].mm_addr + in->planes[plane].data_offset, > + in->plane_info[plane].lengths, free_v4l2buf_cb, in, 0); > + if (!*buf) > + return AVERROR(ENOMEM); > + > + in->status = V4L2BUF_RET_USER; > + > + return 0; > +} This looks like it would trigger massive UB if you keep a frame after the decoder is closed. This should not happen, an AVBufferRef must stay valid forever. At least it looks like it assumes that the decoder is somehow still around, without unreffing it, which hints towards that this is done incorrectly. > +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) > +{ > + int i, ret; > + > + av_frame_unref(frame); > + > + /* 1. get references to the actual data */ > + for (i = 0; i < avbuf->num_planes; i++) { > + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); > + if (ret) > + return ret; > + > + frame->linesize[i] = avbuf->bytesperline[i]; > + frame->data[i] = frame->buf[i]->data; > + } > + > + /* 1.1 fixup special cases */ > + switch (avbuf->context->av_pix_fmt) { > + case AV_PIX_FMT_NV12: > + if (avbuf->num_planes > 1) > + break; > + frame->linesize[1] = avbuf->bytesperline[0]; > + frame->data[1] = frame->buf[0]->data + avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; > + break; > + default: > + break; > + } > + > + /* 2. get frame information */ > + frame->key_frame = !!(avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME); > + frame->format = avbuf->context->av_pix_fmt; > + > + /* these values are updated also during re-init in process_video_event */ > + frame->height = avbuf->context->height; > + frame->width = avbuf->context->width; > + frame->pts = get_pts(avbuf); > + > + /* 3. report errors upstream */ > + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { > + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: driver decode error\n", avbuf->context->name); > + frame->decode_error_flags |= FF_DECODE_ERROR_INVALID_BITSTREAM; > + } > + > + return 0; > +} This function seems to lack setting typically required metadata like colorspace. > +static int buffer_ops_v4l2buf_to_avpkt(AVPacket *pkt, V4L2Buffer *avbuf) > +{ > + int ret; > + > + av_packet_unref(pkt); > + ret = avbuf->ops.buf_to_bufref(avbuf, 0, &pkt->buf); > + if (ret) > + return ret; > + > + pkt->size = V4L2_TYPE_IS_MULTIPLANAR(avbuf->context->type) ? avbuf->buf.m.planes[0].bytesused : avbuf->buf.bytesused; > + pkt->data = pkt->buf->data; > + > + if (avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME) > + pkt->flags |= AV_PKT_FLAG_KEY; > + > + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { > + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s driver encode error\n", avbuf->context->name); > + pkt->flags |= AV_PKT_FLAG_CORRUPT; > + } > + > + pkt->pts = get_pts(avbuf); > + > + return 0; > +} (Didn't check whether it's done correctly here, but again, AVPackets must stay valid even if the encoder is closed.) Why does this not set the DTS? The DTS is required for muxing with some important containers. > +static V4L2Buffer* context_ops_dequeue_v4l2buf(V4L2Context *ctx, unsigned int timeout) > +{ > + struct v4l2_plane planes[VIDEO_MAX_PLANES]; > + struct v4l2_buffer buf = { 0 }; > + V4L2Buffer* avbuf = NULL; > + struct pollfd pfd = { > + .events = POLLIN | POLLRDNORM | POLLPRI, /* default capture context */ > + .fd = ctx->fd, > + }; > + int ret; > + > + if (ctx->num_queued < ctx->min_queued_buffers) > + return NULL; > + > + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) > + pfd.events = POLLOUT | POLLWRNORM; > + > + for (;;) { > + ret = poll(&pfd, 1, timeout); Why does this have a timeout? This makes no sense: either a frame will be decoded/packet can be encoded, or you need to give it more input frames or packets, or an error happened. > +/** > + * Initializes a V4L2Context. > + * > + * @param[in] ctx A pointer to a V4L2Context. See V4L2Context description for required variables. > + * @return 0 in case of success, a negative value representing the error otherwise. > + */ > +int avpriv_v4l2_context_init(V4L2Context* ctx, int lazy_init); > + > +/** > + * Releases a V4L2Context. > + * > + * @param[in] ctx A pointer to a V4L2Context. > + * The caller is reponsible for freeing it. > + * It must not be used after calling this function. > + */ > +void avpriv_v4l2_context_release(V4L2Context* ctx); We shouldn't add new avpriv functions. The situation is bad enough as it is. > +int avpriv_v4l2m2m_init(V4L2m2mContext* s, void* log_ctx) > +{ > + char *devname_save = s->devname; > + int ret = AVERROR(EINVAL); > + char node[PATH_MAX]; > + struct dirent *entry; > + DIR *dirp; > + > + if (s->devname && *s->devname) > + return configure_contexts(s, log_ctx); > + > + dirp = opendir("/dev"); > + if (!dirp) > + return AVERROR(errno); > + > + for (entry = readdir(dirp); entry; entry = readdir(dirp)) { > + > + if (strncmp(entry->d_name, "video", 5)) > + continue; > + > + snprintf(node, sizeof(node), "/dev/%s", entry->d_name); > + > + av_log(log_ctx, AV_LOG_DEBUG, "probing device %s\n", node); > + > + s->devname = node; > + ret = probe_v4l2_driver(s, log_ctx); > + if (!ret) > + break; > + } This doesn't really look like a good idea. Even somehow trying to enumerate stuff in /sys/class/ sounds better. Just because device filename starts with "video" it doesn't need to be relevant, and poking random ioctl()s at arbitrary devices doesn't sound very reliable. > + closedir(dirp); > + > + if (!ret) { > + av_log(log_ctx, AV_LOG_INFO, "Using device %s\n", node); > + ret = configure_contexts(s, log_ctx); > + } else { > + av_log(log_ctx, AV_LOG_ERROR, "Could not find a valid device\n"); > + } > + s->devname = devname_save; > + > + return ret; > +} > + > +int avpriv_v4l2m2m_reinit(V4L2Context* ctx) > +{ > + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, capture); > + int ret; > + > + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s reinit context\n", ctx->name); > + > + /* 1. streamoff */ > + ret = avpriv_v4l2_context_set_status(ctx, VIDIOC_STREAMOFF); > + if (ret) > + av_log(ctx->log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", ctx->name); > + > + /* 2. unmap the buffers (v4l2 and ffmpeg) */ > + avpriv_v4l2_context_release(ctx); What happens to AVFrames or AVPackets that are still referenced? > + > + /* 3. query the new format */ > + ret = avpriv_v4l2m2m_format(ctx, 1); > + if (ret) { > + av_log(ctx->log_ctx, AV_LOG_ERROR, "setting %s format\n", ctx->name); > + return ret; > + } > + > + /* 4. do lazy initialization */ > + ret = avpriv_v4l2_context_init(ctx, ctx->lazy_init); > + if (ret) { > + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s buffers lazy init\n", ctx->name); > + return ret; > + } > + > + /* 5. update AVCodecContext after re-init */ > + ret = ff_set_dimensions(s->avctx, ctx->width, ctx->height); > + if (ret < 0) > + av_log(ctx->log_ctx, AV_LOG_WARNING, "update avcodec height and width\n"); > + > + return 0; > +} > + > +int ff_v4l2m2m_codec_end(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + > + av_log(avctx, AV_LOG_DEBUG, "Closing context\n"); > + > + return avpriv_v4l2m2m_end(s); > +} > + > +int ff_v4l2m2m_codec_init(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + s->avctx = avctx; > + > + return avpriv_v4l2m2m_init(s, avctx); > +} > + > diff --git a/libavcodec/v4l2_m2m.h b/libavcodec/v4l2_m2m.h > new file mode 100644 > index 0000000..41cd6e9 > --- /dev/null > +++ b/libavcodec/v4l2_m2m.h > @@ -0,0 +1,59 @@ > +/* > + * V4L2 mem2mem helper functions > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef AVCODEC_V4L2_M2M_H > +#define AVCODEC_V4L2_M2M_H > + > +#include "v4l2_buffers.h" > +#include "v4l2_fmt.h" > + > +#define container_of(ptr, type, member) ({ \ > + const __typeof__(((type *)0)->member ) *__mptr = (ptr); \ > + (type *)((char *)__mptr - offsetof(type,member) );}) > + > +#define V4L_M2M_DEFAULT_OPTS \ > + { "device",\ > + "Path to the device to use",\ > + OFFSET(devname), AV_OPT_TYPE_STRING, {.str = NULL }, 0, 0, FLAGS },\ > + { "num_output_buffers",\ > + "Number of buffers in the output context",\ > + OFFSET(output.num_buffers), AV_OPT_TYPE_INT, { .i64 = 16 }, 4, INT_MAX, FLAGS } > + > +typedef struct V4L2m2mContext > +{ > + AVClass *class; > + int fd; > + char *devname; > + struct v4l2_capability cap; > + V4L2Context capture; > + V4L2Context output; > + /* update codec while dynamic stream reconfig */ > + AVCodecContext *avctx; > +} V4L2m2mContext; > + > +int avpriv_v4l2m2m_init(V4L2m2mContext *ctx, void* log_ctx); > +int avpriv_v4l2m2m_reinit(V4L2Context *ctx); > +int avpriv_v4l2m2m_format(V4L2Context *ctx, int set); > +int avpriv_v4l2m2m_end(V4L2m2mContext *ctx); > + > +#endif /* AVCODEC_V4L2_M2M_H */ > diff --git a/libavcodec/v4l2_m2m_avcodec.h b/libavcodec/v4l2_m2m_avcodec.h > new file mode 100644 > index 0000000..3e17ae9 > --- /dev/null > +++ b/libavcodec/v4l2_m2m_avcodec.h > @@ -0,0 +1,32 @@ > +/* > + * V4L2 mem2mem avcodec helper functions > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef AVCODEC_V4L2_M2M_AVCODEC_H > +#define AVCODEC_V4L2_M2M_AVCODEC_H > + > +#include "avcodec.h" > + > +int ff_v4l2m2m_codec_init(AVCodecContext *avctx); > +int ff_v4l2m2m_codec_end(AVCodecContext *avctx); > + > +#endif > diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c > new file mode 100644 > index 0000000..19fef72 > --- /dev/null > +++ b/libavcodec/v4l2_m2m_dec.c > @@ -0,0 +1,252 @@ > +/* > + * V4L2 mem2mem decoders > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#include <sys/ioctl.h> > +#include "libavutil/pixfmt.h" > +#include "libavutil/pixdesc.h" > +#include "libavutil/opt.h" > +#include "v4l2_m2m_avcodec.h" > +#include "v4l2_fmt.h" > +#include "v4l2_buffers.h" > +#include "v4l2_m2m.h" > +#include "decode.h" > +#include "avcodec.h" > + > +static int try_start(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const capture = &s->capture; > + V4L2Context *const output = &s->output; > + struct v4l2_event_subscription sub; > + struct v4l2_selection selection; > + struct v4l2_control ctrl; > + int ret; > + > + if (output->streamon && capture->streamon) > + return 0; > + > + /* 0. subscribe to source change event */ > + memset(&sub, 0, sizeof(sub)); > + sub.type = V4L2_EVENT_SOURCE_CHANGE; > + ret = ioctl(s->fd, VIDIOC_SUBSCRIBE_EVENT, &sub); > + if ( ret < 0) > + av_log(avctx, AV_LOG_WARNING, "decoding does not support resolution change\n"); > + > + /* 1. start the output process */ > + if (!output->streamon) { > + ret = avpriv_v4l2_context_set_status(output, VIDIOC_STREAMON); > + if (ret < 0) { > + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON on output context\n"); > + return ret; > + } > + } > + > + /* 2. get the capture format */ > + capture->format.type = capture->type; > + ret = ioctl(capture->fd, VIDIOC_G_FMT, &capture->format); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_FMT ioctl\n"); > + return ret; > + } > + > + /* 2.1 update the AVCodecContext */ > + avctx->pix_fmt = avpriv_v4l_fmt_v4l2ff(capture->format.fmt.pix_mp.pixelformat, AV_CODEC_ID_RAWVIDEO); > + capture->av_pix_fmt = avctx->pix_fmt; > + > + /* 3. set the crop parameters */ > + selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; > + selection.r.height = avctx->coded_height; > + selection.r.width = avctx->coded_width; > + ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection); > + if (!ret) { > + ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_SELECTION ioctl\n"); > + } else { > + av_log(avctx, AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height); > + /* update the size of the resulting frame */ > + capture->height = selection.r.height; > + capture->width = selection.r.width; > + } > + } > + > + /* 4. get the minimum number of buffers required by capture (or default) */ > + memset(&ctrl, 0, sizeof(ctrl)); > + ctrl.id = V4L2_CID_MIN_BUFFERS_FOR_CAPTURE; > + ret = ioctl(s->fd, VIDIOC_G_CTRL, &ctrl); > + if (!ret) > + capture->min_queued_buffers = ctrl.value; > + > + /* 5. init the capture context now that we have the capture format */ > + if (!capture->buffers) { > + > + av_log(capture->log_ctx, AV_LOG_DEBUG, "%s requested (%dx%d)\n", > + capture->name, capture->format.fmt.pix_mp.width, capture->format.fmt.pix_mp.height); > + > + ret = avpriv_v4l2_context_init(capture, 0); > + if (ret) { > + av_log(avctx, AV_LOG_DEBUG, "can't request output buffers\n"); > + return ret; > + } > + } > + > + /* 6. start the capture process */ > + ret = avpriv_v4l2_context_set_status(capture, VIDIOC_STREAMON); > + if (ret) { > + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON, on capture context\n"); > + return ret; > + } > + > + return 0; > +} > + > +static av_cold int v4l2m2m_decode_init(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *capture = &s->capture; > + V4L2Context *output = &s->output; > + > + output->default_flags = capture->default_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; > + output->time_base = capture->time_base = avctx->framerate; > + output->height = capture->height = avctx->coded_height; > + output->width = capture->width =avctx->coded_width; > + > + output->av_codec_id = avctx->codec_id; > + output->av_pix_fmt = AV_PIX_FMT_NONE; > + output->min_queued_buffers = 6; > + > + /* > + * increase the number of buffers to support h.264/h.265 > + * default (configurable via option) is 16 > + */ > + switch (avctx->codec_id) { > + case AV_CODEC_ID_H264: > + case AV_CODEC_ID_HEVC: > + output->num_buffers += 4; > + break; > + } That sounds questionable. The maximum DPB for h264 and HEVC are usually put at 16. So it should add 14 buffers, assuming traditional MPEG codecs use 2. VP9 uses something between. > + > + /* > + * the buffers associated to this context cant be initialized without > + * additional information available in the kernel driver, so do let's postpone it > + */ > + capture->lazy_init = 1; > + > + capture->av_codec_id = AV_CODEC_ID_RAWVIDEO; > + capture->av_pix_fmt = avctx->pix_fmt; > + capture->min_queued_buffers = 6; > + > + return ff_v4l2m2m_codec_init(avctx); > +} > + > +static int v4l2m2m_receive_frame(AVCodecContext *avctx, AVFrame *frame) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const capture = &s->capture; > + V4L2Context *const output = &s->output; > + unsigned int timeout = 50; > + AVPacket avpkt = {0}; I think we've been arguing that we should avoid allocating AVPackets on the stack because we want to remove sizeof(AVPacket) from the ABI, but I guess it doesn't actually matter for now. > + int ret; > + > + ret = ff_decode_get_packet(avctx, &avpkt); > + if (ret < 0 && ret != AVERROR_EOF) > + return ret; > + > + ret = avpriv_v4l2_enqueue_packet(output, &avpkt); > + if (ret < 0) > + return ret; > + > + if (avpkt.size) { > + ret = try_start(avctx); > + if (ret) > + return 0; > + } > + > + return avpriv_v4l2_dequeue_frame(capture, frame, timeout); > +} > + > +#define OFFSET(x) offsetof(V4L2m2mContext, x) > +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_DECODING_PARAM > + > + static const AVOption options[] = { > + V4L_M2M_DEFAULT_OPTS,{ "num_capture_extra_buffers","Number of extra buffers in the capture context", > + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT,{.i64 = 6}, 6, INT_MAX, FLAGS}, > + { NULL}, > + }; > + > +#define M2MDEC(NAME, LONGNAME, CODEC, bsf_name) \ > +static const AVClass v4l2_m2m_ ## NAME ## _dec_class = {\ > + .class_name = #NAME "_v4l2_m2m_decoder",\ > + .item_name = av_default_item_name,\ > + .option = options,\ > + .version = LIBAVUTIL_VERSION_INT,\ > +};\ > +\ > +AVCodec ff_ ## NAME ## _v4l2m2m_decoder = { \ > + .name = #NAME "_v4l2m2m" ,\ > + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ > + .type = AVMEDIA_TYPE_VIDEO,\ > + .id = CODEC ,\ > + .priv_data_size = sizeof(V4L2m2mContext),\ > + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ > + .init = v4l2m2m_decode_init,\ > + .receive_frame = v4l2m2m_receive_frame,\ > + .close = ff_v4l2m2m_codec_end,\ > + .bsfs = bsf_name, \ > +}; > + > +#if CONFIG_H263_V4L2M2M_DECODER > + M2MDEC(h263, "H.263", AV_CODEC_ID_H263, NULL); > +#endif > + > +#if CONFIG_H264_V4L2M2M_DECODER > + M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); > +#endif > + > +#if CONFIG_MPEG1_V4L2M2M_DECODER > + M2MDEC(mpeg1, "MPEG1", AV_CODEC_ID_MPEG1VIDEO, NULL); > +#endif > + > +#if CONFIG_MPEG2_V4L2M2M_DECODER > + M2MDEC(mpeg2, "MPEG2", AV_CODEC_ID_MPEG2VIDEO, NULL); > +#endif > + > +#if CONFIG_MPEG4_V4L2M2M_DECODER > + M2MDEC(mpeg4, "MPEG4", AV_CODEC_ID_MPEG4, NULL); > +#endif > + > +#if CONFIG_VC1_V4L2M2M_DECODER > + M2MDEC(vc1 , "VC1", AV_CODEC_ID_VC1, NULL); > +#endif > + > +#if CONFIG_VP8_V4L2M2M_DECODER > + M2MDEC(vp8, "VP8", AV_CODEC_ID_VP8, NULL); > +#endif > + > +#if CONFIG_VP9_V4L2M2M_DECODER > + M2MDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL); > +#endif > + > +#if CONFIG_HEVC_V4L2M2M_DECODER > + M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); > +#endif You don't need all this ifdeffery. A codec entry doesn't pull in any additional functions, so a disabled codec entry will simply not reference the AVCodec. > diff --git a/libavcodec/v4l2_m2m_enc.c b/libavcodec/v4l2_m2m_enc.c > new file mode 100644 > index 0000000..db9c090 > --- /dev/null > +++ b/libavcodec/v4l2_m2m_enc.c > @@ -0,0 +1,288 @@ > +/* > + * V4L2 mem2mem encoders > + * > + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> > + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#include <sys/ioctl.h> > + > +#include "libavutil/pixfmt.h" > +#include "libavutil/pixdesc.h" > +#include "libavutil/opt.h" > +#include "v4l2_m2m_avcodec.h" > +#include "v4l2_buffers.h" > +#include "v4l2_fmt.h" > +#include "v4l2_m2m.h" > +#include "avcodec.h" > + > +#define STR(s) AV_TOSTRING(s) > +#define MPEG_CID(x) V4L2_CID_MPEG_VIDEO_##x > +#define MPEG_VIDEO(x) V4L2_MPEG_VIDEO_##x > + > +#define SET_V4L2_EXT_CTRL(TYPE, ID, VALUE, NAME) \ > +{ \ > + struct v4l2_ext_control ctrl = { 0 }; \ > + struct v4l2_ext_controls ctrls = { 0 }; \ > + ctrls.ctrl_class = V4L2_CTRL_CLASS_MPEG; \ > + ctrls.controls = &ctrl; \ > + ctrl.TYPE = VALUE ; \ > + ctrl.id = ID ; \ > + ctrls.count = 1; \ > + \ > + if ((ret = ioctl(s->fd, VIDIOC_S_EXT_CTRLS, &ctrls)) < 0) \ > + av_log(avctx, AV_LOG_WARNING, "Failed to set " NAME "%s\n", STR(ID)); \ > +} > + > +#define SET_V4L2_TIME_PER_FRAME(NUM, DEN) \ > +{ \ > + struct v4l2_streamparm parm = { 0 }; \ > + parm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; \ > + parm.parm.output.timeperframe.numerator = NUM; \ > + parm.parm.output.timeperframe.denominator = DEN; \ > + \ > + if ((ret = ioctl(s->fd, VIDIOC_S_PARM, &parm)) < 0) \ > + av_log(avctx, AV_LOG_WARNING, "Failed to set timeperframe"); \ > +} I think those should be functions. ctrl has only 3 fields, so it's ok to have the caller set it up. (If you use C99 struct literals it won't be more code than before, maybe.) > + > +static inline int v4l2_h264_profile_from_ff(int p) > +{ > + switch(p) { > + case FF_PROFILE_H264_CONSTRAINED_BASELINE: > + return MPEG_VIDEO(H264_PROFILE_CONSTRAINED_BASELINE); > + case FF_PROFILE_H264_HIGH_444_PREDICTIVE: > + return MPEG_VIDEO(H264_PROFILE_HIGH_444_PREDICTIVE); > + case FF_PROFILE_H264_HIGH_422_INTRA: > + return MPEG_VIDEO(H264_PROFILE_HIGH_422_INTRA); > + case FF_PROFILE_H264_HIGH_444_INTRA: > + return MPEG_VIDEO(H264_PROFILE_HIGH_444_INTRA); > + case FF_PROFILE_H264_HIGH_10_INTRA: > + return MPEG_VIDEO(H264_PROFILE_HIGH_10_INTRA); > + case FF_PROFILE_H264_HIGH_422: > + return MPEG_VIDEO(H264_PROFILE_HIGH_422); > + case FF_PROFILE_H264_BASELINE: > + return MPEG_VIDEO(H264_PROFILE_BASELINE); > + case FF_PROFILE_H264_EXTENDED: > + return MPEG_VIDEO(H264_PROFILE_EXTENDED); > + case FF_PROFILE_H264_HIGH_10: > + return MPEG_VIDEO(H264_PROFILE_HIGH_10); > + case FF_PROFILE_H264_MAIN: > + return MPEG_VIDEO(H264_PROFILE_MAIN); > + case FF_PROFILE_H264_HIGH: > + return MPEG_VIDEO(H264_PROFILE_HIGH); > + } > + > + return -1; > +} > + > +static inline int v4l2_mpeg4_profile_from_ff(int p) > +{ > + switch(p) { > + case FF_PROFILE_MPEG4_ADVANCED_CODING: > + return MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_CODING_EFFICIENCY); > + case FF_PROFILE_MPEG4_ADVANCED_SIMPLE: > + return MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_SIMPLE); > + case FF_PROFILE_MPEG4_SIMPLE_SCALABLE: > + > + return MPEG_VIDEO(MPEG4_PROFILE_SIMPLE_SCALABLE); > + case FF_PROFILE_MPEG4_SIMPLE: > + return MPEG_VIDEO(MPEG4_PROFILE_SIMPLE); > + case FF_PROFILE_MPEG4_CORE: > + return MPEG_VIDEO(MPEG4_PROFILE_CORE); > + } > + > + return -1; > +} Would a table be better maybe? > + > +static av_cold int v4l2m2m_encode_init(AVCodecContext *avctx) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *capture = &s->capture; > + V4L2Context *output = &s->output; > + int qmin_cid, qmax_cid, ret, val; > + int qmin, qmax; > + > + output->default_flags = capture->default_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; > + output->time_base = capture->time_base = avctx->time_base; > + output->height = capture->height = avctx->height; > + output->width = capture->width = avctx->width; > + > + /* output context */ > + output->av_codec_id = AV_CODEC_ID_RAWVIDEO; > + output->av_pix_fmt = avctx->pix_fmt; > + > + /* capture context */ > + capture->av_codec_id = avctx->codec_id; > + capture->av_pix_fmt = AV_PIX_FMT_NONE; > + capture->min_queued_buffers = 1; > + > + if (ret = ff_v4l2m2m_codec_init(avctx)) > + return ret; > + > + SET_V4L2_TIME_PER_FRAME(avctx->framerate.num, avctx->framerate.den); > + SET_V4L2_EXT_CTRL(value, MPEG_CID(HEADER_MODE), MPEG_VIDEO(HEADER_MODE_SEPARATE), "header mode"); > + SET_V4L2_EXT_CTRL(value, MPEG_CID(B_FRAMES), avctx->max_b_frames, "number of B-frames"); > + SET_V4L2_EXT_CTRL(value, MPEG_CID(GOP_SIZE), avctx->gop_size,"gop size"); > + SET_V4L2_EXT_CTRL(value, MPEG_CID(BITRATE) , avctx->bit_rate, "bit rate"); > + > + av_log(avctx, AV_LOG_DEBUG, "Encoder Context: frame rate(%d/%d), number b-frames (%d)," > + "gop size (%d), bit rate (%ld), qmin (%d), qmax (%d)\n", > + avctx->framerate.num, avctx->framerate.den, avctx->max_b_frames, avctx->gop_size, avctx->bit_rate, avctx->qmin, avctx->qmax); > + > + switch(avctx->codec_id) { > + case AV_CODEC_ID_H264: > + val = v4l2_h264_profile_from_ff(avctx->profile); > + if (val >= 0) { > + SET_V4L2_EXT_CTRL(value, MPEG_CID(H264_PROFILE), val, "h264 profile"); > + } > + qmin_cid = MPEG_CID(H264_MIN_QP); > + qmax_cid = MPEG_CID(H264_MAX_QP); > + qmin = 0; > + qmax = 51; > + break; > + case AV_CODEC_ID_MPEG4: > + val = v4l2_mpeg4_profile_from_ff(avctx->profile); > + if (val >= 0) { > + SET_V4L2_EXT_CTRL(value, MPEG_CID(MPEG4_PROFILE), val, "mpeg4 profile"); > + } > + qmin_cid = MPEG_CID(MPEG4_MIN_QP); > + qmax_cid = MPEG_CID(MPEG4_MAX_QP); > + if (avctx->flags & CODEC_FLAG_QPEL) { > + SET_V4L2_EXT_CTRL(value, MPEG_CID(MPEG4_QPEL), 1, "qpel"); > + } > + qmax = 51; > + qmin = 0; > + break; > + case AV_CODEC_ID_H263: > + qmin_cid = MPEG_CID(H263_MIN_QP); > + qmax_cid = MPEG_CID(H263_MAX_QP); > + qmin = 1; > + qmax = 31; > + break; > + case AV_CODEC_ID_VP8: > + qmin_cid = MPEG_CID(VPX_MIN_QP); > + qmax_cid = MPEG_CID(VPX_MAX_QP); > + qmin = 0; > + qmax = 127; > + break; > + case AV_CODEC_ID_VP9: > + qmin_cid = MPEG_CID(VPX_MIN_QP); > + qmax_cid = MPEG_CID(VPX_MAX_QP); > + qmin = 0; > + qmax = 255; > + break; > + default: > + return 0; > + } > + > + if (qmin != avctx->qmin || qmax != avctx->qmax) > + av_log(avctx, AV_LOG_WARNING, "Encoder adjusted: qmin (%d), qmax (%d)\n", qmin, qmax); > + > + SET_V4L2_EXT_CTRL(value, qmin_cid, qmin, "minimum video quantizer scale"); > + SET_V4L2_EXT_CTRL(value, qmax_cid, qmax, "maximum video quantizer scale"); > + > + return 0; > +} > + > +static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const output = &s->output; > + > + return avpriv_v4l2_enqueue_frame(output, frame); > +} > + > +static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt) > +{ > + V4L2m2mContext *s = avctx->priv_data; > + V4L2Context *const capture = &s->capture; > + V4L2Context *const output = &s->output; > + unsigned int timeout = 50; > + int ret; > + > + if (!output->streamon) { > + ret = avpriv_v4l2_context_set_status(output, VIDIOC_STREAMON); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n"); > + return ret; > + } > + } > + > + if (!capture->streamon) { > + ret = avpriv_v4l2_context_set_status(capture, VIDIOC_STREAMON); > + if (ret) { > + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n"); > + return ret; > + } > + } > + > + return avpriv_v4l2_dequeue_packet(capture, avpkt, timeout); > +} > + > +#define OFFSET(x) offsetof(V4L2m2mContext, x) > +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM > + > +static const AVOption options[] = { > + V4L_M2M_DEFAULT_OPTS, > + { "num_capture_buffers", "Number of buffers in the capture context", > + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT, {.i64 = 4 }, 4, INT_MAX, FLAGS }, > + { NULL }, > +}; > + > +#define M2MENC(NAME, LONGNAME, CODEC) \ > +static const AVClass v4l2_m2m_ ## NAME ## _enc_class = {\ > + .class_name = #NAME "_v4l2_m2m_encoder",\ > + .item_name = av_default_item_name,\ > + .option = options,\ > + .version = LIBAVUTIL_VERSION_INT,\ > +};\ > +\ > +AVCodec ff_ ## NAME ## _v4l2m2m_encoder = { \ > + .name = #NAME "_v4l2m2m" ,\ > + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " encoder wrapper"),\ > + .type = AVMEDIA_TYPE_VIDEO,\ > + .id = CODEC ,\ > + .priv_data_size = sizeof(V4L2m2mContext),\ > + .priv_class = &v4l2_m2m_ ## NAME ##_enc_class,\ > + .init = v4l2m2m_encode_init,\ > + .send_frame = v4l2m2m_send_frame,\ > + .receive_packet = v4l2m2m_receive_packet,\ > + .close = ff_v4l2m2m_codec_end,\ > +}; > + > +#if CONFIG_H263_V4L2M2M_ENCODER > +M2MENC(h263, "H.263", AV_CODEC_ID_H263); > +#endif > + > +#if CONFIG_H264_V4L2M2M_ENCODER > +M2MENC(h264, "H.264", AV_CODEC_ID_H264); > +#endif > + > +#if CONFIG_MPEG4_V4L2M2M_ENCODER > +M2MENC(mpeg4, "MPEG4", AV_CODEC_ID_MPEG4); > +#endif > + > +#if CONFIG_VP8_V4L2M2M_ENCODER > +M2MENC(vp8, "VP8", AV_CODEC_ID_VP8); > +#endif > + > +#if CONFIG_HEVC_V4L2M2M_ENCODER > +M2MENC(hevc, "HEVC", AV_CODEC_ID_HEVC); > +#endif > + Same comment about the ifdeffery as on the decoder side.
On 08/25/2017 05:35 PM, wm4 wrote: > That looks generally OK. Is there any chance a hwaccel approach would > be possible instead? If I've learned anything about hardware decoding, > then that hwaccel is vastly superior to vendor-implemented full stream > decoders. could you help me understand what would that entitle and what how would that be beneficial to the users? I just dont feel I can answer that question properly... v4l2 provides a generic API which is what the patchset uses to perform encoding/decoding on any v4l2 supported hardware (it is completely vendor independent) From the layer above (libavcodec) all it needs is a way to get the frame information and after processing to pass it back; so in principle, if the hwaccel API provides that, I could just move it all to use those calls if you think that fits better with ffmpeg. but I dont think I understand the benefit of changing from the ffmpeg encoding/decoding API to hwaccel API. > > I don't think I like the attempt of sharing the v4l helper functions > between libavdevice and libavcodec, but I can't tell how much it helps. ok. I am of course open to suggestions on this (I didnt see any issues with what the patchset provides or I would have done it differently).
On 08/25/2017 05:35 PM, wm4 wrote: >> +#define WIDTH(__ctx, __fmt) \ >> + (V4L2_TYPE_IS_MULTIPLANAR((__ctx)->type) ? __fmt.fmt.pix_mp.width : __fmt.fmt.pix.width) >> + >> +#define HEIGHT(__ctx, __fmt) \ >> + (V4L2_TYPE_IS_MULTIPLANAR((__ctx)->type) ? __fmt.fmt.pix_mp.height : __fmt.fmt.pix.height) > These names are a bit generic. Also, identifiers starting with __ are > always implementation reserved (i.e. using them undefined behavior). > You're forgetting to quote the __fmt macro parameter too. > ok, will just do inline functions instead. I shouldnt have done a macro...
On 08/25/2017 05:35 PM, wm4 wrote: >> +static inline int buffer_ops_v4l2buf_to_bufref(V4L2Buffer *in, int plane, AVBufferRef **buf) >> +{ >> + if (plane >= in->num_planes) >> + return AVERROR(EINVAL); >> + >> + /* even though most encoders return 0 in data_offset encoding vp8 does require this value*/ >> + *buf = av_buffer_create((char *)in->plane_info[plane].mm_addr + in->planes[plane].data_offset, >> + in->plane_info[plane].lengths, free_v4l2buf_cb, in, 0); >> + if (!*buf) >> + return AVERROR(ENOMEM); >> + >> + in->status = V4L2BUF_RET_USER; >> + >> + return 0; >> +} > This looks like it would trigger massive UB if you keep a frame after > the decoder is closed. This should not happen, an AVBufferRef must > stay valid forever. Yes you are right. v4l2 uses a limited number of buffers in physical memory that it reuses during the video operation. these buffers must be mapped to the process address space before they can be used. for decoding, ffmpeg maps the V4L2 physical buffers, memcpies the input data to the output queue and uses the _references_ from the capture queue to create AVBufferRefs (these references are mmaped addresses of physical memory buffers). In the current design buffers are not be reused until their corresponding AVBufferRefs are released (notice that freeing an AVBufferRef executes the callback that sets the V4L2BUF_AVAILABLE flag making it ready to be enqueued again). So: 1. would it be acceptable if I stopped the codec from ummaping its buffers until all AVBufferRefs have been freed? From what you are saying I think this has to be done. 2. also notice that if ffmpeg keeps all the AVBufferRefs for 'too long' this would cause the v4l2 pipeline to starve. I have not seen this case (I have tested all decoders with ffplay) but from what you are saying it could be possible (typical hardware can handle 32 physical buffers although v4l2 applications rarely request so many). Is there any way in ffmpeg to catch this condition and force the release of some of the AVBuffers?
On 08/25/2017 05:35 PM, wm4 wrote: >> +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) >> +{ >> + int i, ret; >> + >> + av_frame_unref(frame); >> + >> + /* 1. get references to the actual data */ >> + for (i = 0; i < avbuf->num_planes; i++) { >> + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); >> + if (ret) >> + return ret; >> + >> + frame->linesize[i] = avbuf->bytesperline[i]; >> + frame->data[i] = frame->buf[i]->data; >> + } >> + >> + /* 1.1 fixup special cases */ >> + switch (avbuf->context->av_pix_fmt) { >> + case AV_PIX_FMT_NV12: >> + if (avbuf->num_planes > 1) >> + break; >> + frame->linesize[1] = avbuf->bytesperline[0]; >> + frame->data[1] = frame->buf[0]->data + avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; >> + break; >> + default: >> + break; >> + } >> + >> + /* 2. get frame information */ >> + frame->key_frame = !!(avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME); >> + frame->format = avbuf->context->av_pix_fmt; >> + >> + /* these values are updated also during re-init in process_video_event */ >> + frame->height = avbuf->context->height; >> + frame->width = avbuf->context->width; >> + frame->pts = get_pts(avbuf); >> + >> + /* 3. report errors upstream */ >> + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { >> + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: driver decode error\n", avbuf->context->name); >> + frame->decode_error_flags |= FF_DECODE_ERROR_INVALID_BITSTREAM; >> + } >> + >> + return 0; >> +} > This function seems to lack setting typically required metadata like > colorspace. > ok I will retrieve the colorspace from the v4l2 format structure and set it in the frame. is there anything else I am missing?
On 08/27/2017 09:09 PM, Jorge Ramirez wrote: > On 08/25/2017 05:35 PM, wm4 wrote: >>> +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) >>> +{ >>> + int i, ret; >>> + >>> + av_frame_unref(frame); >>> + >>> + /* 1. get references to the actual data */ >>> + for (i = 0; i < avbuf->num_planes; i++) { >>> + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); >>> + if (ret) >>> + return ret; >>> + >>> + frame->linesize[i] = avbuf->bytesperline[i]; >>> + frame->data[i] = frame->buf[i]->data; >>> + } >>> + >>> + /* 1.1 fixup special cases */ >>> + switch (avbuf->context->av_pix_fmt) { >>> + case AV_PIX_FMT_NV12: >>> + if (avbuf->num_planes > 1) >>> + break; >>> + frame->linesize[1] = avbuf->bytesperline[0]; >>> + frame->data[1] = frame->buf[0]->data + avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; >>> + break; >>> + default: >>> + break; >>> + } >>> + >>> + /* 2. get frame information */ >>> + frame->key_frame = !!(avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME); >>> + frame->format = avbuf->context->av_pix_fmt; >>> + >>> + /* these values are updated also during re-init in process_video_event */ >>> + frame->height = avbuf->context->height; >>> + frame->width = avbuf->context->width; >>> + frame->pts = get_pts(avbuf); >>> + >>> + /* 3. report errors upstream */ >>> + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { >>> + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: driver decode error\n", avbuf->context->name); >>> + frame->decode_error_flags |= FF_DECODE_ERROR_INVALID_BITSTREAM; >>> + } >>> + >>> + return 0; >>> +} >> This function seems to lack setting typically required metadata like >> colorspace. >> > > ok I will retrieve the colorspace from the v4l2 format structure and > set it in the frame. um, I dont see a 1:1 mapping between the colorspaces reported from v4l2 and what ffmpeg expects (I am also not a subject matter expert on this :( ). Could I get some guidance on the translation table below please? btw in my simple tests - ffplay decoded vp8, vp9, mpeg4, mpeg2, h263, h264 and hevc I didnt need this field so I am not sure if I am getting it right. TIA static inline enum AVColorSpace get_colorspace(V4L2Buffer *buf) { enum v4l2_colorspace cs; cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? buf->context->format.fmt.pix_mp.colorspace : buf->context->format.fmt.pix.colorspace; /* */ switch (cs) { case V4L2_COLORSPACE_SMPTE170M: return AVCOL_SPC_SMPTE170M; case V4L2_COLORSPACE_SMPTE240M: return AVCOL_SPC_SMPTE240M; case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_SPC_BT470BG; case V4L2_COLORSPACE_BT2020: return AVCOL_SPC_BT2020_CL; case V4L2_COLORSPACE_REC709: return AVCOL_SPC_BT709; case V4L2_COLORSPACE_ADOBERGB: case V4L2_COLORSPACE_SRGB: case V4L2_COLORSPACE_DCI_P3: case V4L2_COLORSPACE_JPEG: case V4L2_COLORSPACE_RAW: default: return AVCOL_SPC_RGB; } }
On 08/25/2017 05:35 PM, wm4 wrote: >> +static V4L2Buffer* context_ops_dequeue_v4l2buf(V4L2Context *ctx, unsigned int timeout) >> +{ >> + struct v4l2_plane planes[VIDEO_MAX_PLANES]; >> + struct v4l2_buffer buf = { 0 }; >> + V4L2Buffer* avbuf = NULL; >> + struct pollfd pfd = { >> + .events = POLLIN | POLLRDNORM | POLLPRI, /* default capture context */ >> + .fd = ctx->fd, >> + }; >> + int ret; >> + >> + if (ctx->num_queued < ctx->min_queued_buffers) >> + return NULL; >> + >> + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) >> + pfd.events = POLLOUT | POLLWRNORM; >> + >> + for (;;) { >> + ret = poll(&pfd, 1, timeout); > Why does this have a timeout? This makes no sense: either a frame will > be decoded/packet can be encoded, or you need to give it more input > frames or packets, or an error happened. > yes, it does makes no sense when we consider only the capture buffers in a decoding scenario (those don't time out for the reasons that you pointed out); however we also should look into how the output buffers are handled (in the same decoding case...see context_ops_getfree_v4l2buf). every time a new output buffer is required to input data to the driver, we poll (with a timeout) until we have recovered _all_ the buffers that are no longer needed in the driver. I dont think this is an issue.
On 08/25/2017 05:35 PM, wm4 wrote: >> + >> + dirp = opendir("/dev"); >> + if (!dirp) >> + return AVERROR(errno); >> + >> + for (entry = readdir(dirp); entry; entry = readdir(dirp)) { >> + >> + if (strncmp(entry->d_name, "video", 5)) >> + continue; >> + >> + snprintf(node, sizeof(node), "/dev/%s", entry->d_name); >> + >> + av_log(log_ctx, AV_LOG_DEBUG, "probing device %s\n", node); >> + >> + s->devname = node; >> + ret = probe_v4l2_driver(s, log_ctx); >> + if (!ret) >> + break; >> + } > This doesn't really look like a good idea. Even somehow trying to > enumerate stuff in/sys/class/ sounds better. Just because device > filename starts with "video" it doesn't need to be relevant, and poking > random ioctl()s at arbitrary devices doesn't sound very reliable. > yes it might seem that way but this is pretty much the standard way of opening v4l2 devices (I am not reimplementing the API to my understanding). for an example already present in ffmpeg check libavdevice/v4l2.c v4l2_get_device_list.
On 08/25/2017 05:35 PM, wm4 wrote: >> +int avpriv_v4l2m2m_reinit(V4L2Context* ctx) >> +{ >> + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, capture); >> + int ret; >> + >> + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s reinit context\n", ctx->name); >> + >> + /* 1. streamoff */ >> + ret = avpriv_v4l2_context_set_status(ctx, VIDIOC_STREAMOFF); >> + if (ret) >> + av_log(ctx->log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", ctx->name); >> + >> + /* 2. unmap the buffers (v4l2 and ffmpeg) */ >> + avpriv_v4l2_context_release(ctx); > What happens to AVFrames or AVPackets that are still referenced? > yes in response to your previous comment on the subject, I am trying to address that in v7 not releasing the context while there are buffers on the fly.... This is what I came up with (if you can suggest something better - ie, a way to block until the AVBufferRefs are unref- please let me know) void avpriv_v4l2_context_release(V4L2Context* ctx) { int i, ret; if (!ctx->buffers) return; /* wait until all buffers are no longer in use */ for (i = 0; i < ctx->num_buffers; i++) { if (ctx->buffers[i].status & (V4L2BUF_IN_DRIVER | V4L2BUF_AVAILABLE)) continue; while (ctx->buffers[i].status & V4L2BUF_RET_USER) usleep(100); } ret = ctx->ops.release_buffers(ctx); if (ret) av_log(ctx->log_ctx, AV_LOG_WARNING, "V4L2 failed to unmap the %s buffers\n", ctx->name); else av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s buffers unmapped\n", ctx->name); av_free(ctx->buffers); ctx->buffers = NULL; ctx->num_queued = 0; }
On 08/25/2017 05:35 PM, wm4 wrote: >> + /* >> + * increase the number of buffers to support h.264/h.265 >> + * default (configurable via option) is 16 >> + */ >> + switch (avctx->codec_id) { >> + case AV_CODEC_ID_H264: >> + case AV_CODEC_ID_HEVC: >> + output->num_buffers += 4; >> + break; >> + } > That sounds questionable. The maximum DPB for h264 and HEVC are usually > put at 16. So it should add 14 buffers, assuming traditional MPEG > codecs use 2. VP9 uses something between. > ffmpeg will request 20 out buffers (ie, input frames to the driver) in this case (16 + 4). The kernel _driver_ might return more or less depending on what the hardware can handle and the codec_id. this is just a tentative...
On 08/25/2017 05:35 PM, wm4 wrote: >> +#if CONFIG_VP8_V4L2M2M_DECODER >> + M2MDEC(vp8, "VP8", AV_CODEC_ID_VP8, NULL); >> +#endif >> + >> +#if CONFIG_VP9_V4L2M2M_DECODER >> + M2MDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL); >> +#endif >> + >> +#if CONFIG_HEVC_V4L2M2M_DECODER >> + M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); >> +#endif > You don't need all this ifdeffery. A codec entry doesn't pull in > any additional functions, so a disabled codec entry will simply not > reference the AVCodec. > ok. will fix.
Hi, On Sun, 27 Aug 2017 21:41:06 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/27/2017 09:09 PM, Jorge Ramirez wrote: > > On 08/25/2017 05:35 PM, wm4 wrote: > >>> +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, > >>> V4L2Buffer *avbuf) +{ > >>> + int i, ret; > >>> + > >>> + av_frame_unref(frame); > >>> + > >>> + /* 1. get references to the actual data */ > >>> + for (i = 0; i < avbuf->num_planes; i++) { > >>> + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); > >>> + if (ret) > >>> + return ret; > >>> + > >>> + frame->linesize[i] = avbuf->bytesperline[i]; > >>> + frame->data[i] = frame->buf[i]->data; > >>> + } > >>> + > >>> + /* 1.1 fixup special cases */ > >>> + switch (avbuf->context->av_pix_fmt) { > >>> + case AV_PIX_FMT_NV12: > >>> + if (avbuf->num_planes > 1) > >>> + break; > >>> + frame->linesize[1] = avbuf->bytesperline[0]; > >>> + frame->data[1] = frame->buf[0]->data + > >>> avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; > >>> + break; > >>> + default: > >>> + break; > >>> + } > >>> + > >>> + /* 2. get frame information */ > >>> + frame->key_frame = !!(avbuf->buf.flags & > >>> V4L2_BUF_FLAG_KEYFRAME); > >>> + frame->format = avbuf->context->av_pix_fmt; > >>> + > >>> + /* these values are updated also during re-init in > >>> process_video_event */ > >>> + frame->height = avbuf->context->height; > >>> + frame->width = avbuf->context->width; > >>> + frame->pts = get_pts(avbuf); > >>> + > >>> + /* 3. report errors upstream */ > >>> + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { > >>> + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: > >>> driver decode error\n", avbuf->context->name); > >>> + frame->decode_error_flags |= > >>> FF_DECODE_ERROR_INVALID_BITSTREAM; > >>> + } > >>> + > >>> + return 0; > >>> +} > >> This function seems to lack setting typically required metadata > >> like colorspace. > >> > > > > ok I will retrieve the colorspace from the v4l2 format structure > > and set it in the frame. > > um, I dont see a 1:1 mapping between the colorspaces reported from > v4l2 and what ffmpeg expects (I am also not a subject matter expert > on this :( ). > > Could I get some guidance on the translation table below please? > btw in my simple tests - ffplay decoded vp8, vp9, mpeg4, mpeg2, h263, > h264 and hevc I didnt need this field so I am not sure if I am > getting it right. I don't think it will affect decoding, but the image will be something like too pale if it's set wrongly > static inline enum AVColorSpace get_colorspace(V4L2Buffer *buf) > { > enum v4l2_colorspace cs; > > cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? > buf->context->format.fmt.pix_mp.colorspace : > buf->context->format.fmt.pix.colorspace; > > /* */ > switch (cs) { > case V4L2_COLORSPACE_SMPTE170M: return AVCOL_SPC_SMPTE170M; > case V4L2_COLORSPACE_SMPTE240M: return AVCOL_SPC_SMPTE240M; > case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_SPC_BT470BG; > case V4L2_COLORSPACE_BT2020: return AVCOL_SPC_BT2020_CL; > case V4L2_COLORSPACE_REC709: return AVCOL_SPC_BT709; > case V4L2_COLORSPACE_ADOBERGB: > case V4L2_COLORSPACE_SRGB: > case V4L2_COLORSPACE_DCI_P3: > case V4L2_COLORSPACE_JPEG: > case V4L2_COLORSPACE_RAW: > default: > return AVCOL_SPC_RGB; > } FWIW, here is the conversion I had been using in other projects: switch(f->colorspace) { case V4L2_COLORSPACE_SRGB: return AVCOL_SPC_RGB; case V4L2_COLORSPACE_REC709: return AVCOL_SPC_BT709; case V4L2_COLORSPACE_470_SYSTEM_M: return AVCOL_SPC_FCC; case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_SPC_BT470BG; case V4L2_COLORSPACE_SMPTE170M: return AVCOL_SPC_SMPTE170M; case V4L2_COLORSPACE_SMPTE240M: return AVCOL_SPC_SMPTE240M; case V4L2_COLORSPACE_BT2020: if(f->ycbcr_enc == V4L2_YCBCR_ENC_BT2020_CONST_LUM) return AVCOL_SPC_BT2020_CL; return AVCOL_SPC_BT2020_NCL; default: return AVCOL_SPC_UNSPECIFIED; } you'd likely need AVCOL_PRI*: switch(f->ycbcr_enc) { case V4L2_YCBCR_ENC_XV709: case V4L2_YCBCR_ENC_709 : return AVCOL_PRI_BT709; case V4L2_YCBCR_ENC_XV601: case V4L2_YCBCR_ENC_601: return AVCOL_PRI_BT470M; default: break; } switch(f->colorspace) { case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_PRI_BT470BG; case V4L2_COLORSPACE_SMPTE170M: return AVCOL_PRI_SMPTE170M; case V4L2_COLORSPACE_SMPTE240M: return AVCOL_PRI_SMPTE240M; case V4L2_COLORSPACE_BT2020: return AVCOL_PRI_BT2020; default: break; } return AVCOL_PRI_UNSPECIFIED; AVCOL_TRC*: switch(f->xfer_func) { case V4L2_XFER_FUNC_709: switch(f->bit_depth) { case 10: return AVCOL_TRC_BT2020_10; case 12: return AVCOL_TRC_BT2020_12; default: break; } return AVCOL_TRC_BT709; case V4L2_XFER_FUNC_SRGB: return AVCOL_TRC_IEC61966_2_1; default: break; } switch(f->colorspace) { case V4L2_COLORSPACE_470_SYSTEM_M: return AVCOL_TRC_GAMMA22; case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_TRC_GAMMA28; case V4L2_COLORSPACE_SMPTE170M: return AVCOL_TRC_SMPTE170M; case V4L2_COLORSPACE_SMPTE240M: return AVCOL_TRC_SMPTE240M; default: break; } switch(f->ycbcr_enc) { case V4L2_YCBCR_ENC_XV709: case V4L2_YCBCR_ENC_XV601: return AVCOL_TRC_BT1361_ECG; default: break; } return AVCOL_TRC_UNSPECIFIED; AVCOL_RANGE*: switch(f->quantization) { case V4L2_QUANTIZATION_LIM_RANGE: return AVCOL_RANGE_MPEG; case V4L2_QUANTIZATION_FULL_RANGE: return AVCOL_RANGE_JPEG; default: break; } return AVCOL_RANGE_UNSPECIFIED; Feel free to re-use, point and/or fix bugs here (it's been long since I wrote this and I don't think I've ever visually checked all the possibilities). This might also be useful for the V4L2 output and capture drivers in lavd. Bests, Alexis.
On Sun, 27 Aug 2017 21:41:06 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/27/2017 09:09 PM, Jorge Ramirez wrote: > > On 08/25/2017 05:35 PM, wm4 wrote: > >>> +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) > >>> +{ > >>> + int i, ret; > >>> + > >>> + av_frame_unref(frame); > >>> + > >>> + /* 1. get references to the actual data */ > >>> + for (i = 0; i < avbuf->num_planes; i++) { > >>> + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); > >>> + if (ret) > >>> + return ret; > >>> + > >>> + frame->linesize[i] = avbuf->bytesperline[i]; > >>> + frame->data[i] = frame->buf[i]->data; > >>> + } > >>> + > >>> + /* 1.1 fixup special cases */ > >>> + switch (avbuf->context->av_pix_fmt) { > >>> + case AV_PIX_FMT_NV12: > >>> + if (avbuf->num_planes > 1) > >>> + break; > >>> + frame->linesize[1] = avbuf->bytesperline[0]; > >>> + frame->data[1] = frame->buf[0]->data + avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; > >>> + break; > >>> + default: > >>> + break; > >>> + } > >>> + > >>> + /* 2. get frame information */ > >>> + frame->key_frame = !!(avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME); > >>> + frame->format = avbuf->context->av_pix_fmt; > >>> + > >>> + /* these values are updated also during re-init in process_video_event */ > >>> + frame->height = avbuf->context->height; > >>> + frame->width = avbuf->context->width; > >>> + frame->pts = get_pts(avbuf); > >>> + > >>> + /* 3. report errors upstream */ > >>> + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { > >>> + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: driver decode error\n", avbuf->context->name); > >>> + frame->decode_error_flags |= FF_DECODE_ERROR_INVALID_BITSTREAM; > >>> + } > >>> + > >>> + return 0; > >>> +} > >> This function seems to lack setting typically required metadata like > >> colorspace. > >> > > > > ok I will retrieve the colorspace from the v4l2 format structure and > > set it in the frame. > > um, I dont see a 1:1 mapping between the colorspaces reported from v4l2 > and what ffmpeg expects (I am also not a subject matter expert on this :( ). > > Could I get some guidance on the translation table below please? > btw in my simple tests - ffplay decoded vp8, vp9, mpeg4, mpeg2, h263, > h264 and hevc I didnt need this field so I am not sure if I am getting > it right. > > TIA > > static inline enum AVColorSpace get_colorspace(V4L2Buffer *buf) > { > enum v4l2_colorspace cs; > > cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? > buf->context->format.fmt.pix_mp.colorspace : > buf->context->format.fmt.pix.colorspace; > > /* */ > switch (cs) { > case V4L2_COLORSPACE_SMPTE170M: return AVCOL_SPC_SMPTE170M; > case V4L2_COLORSPACE_SMPTE240M: return AVCOL_SPC_SMPTE240M; > case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_SPC_BT470BG; > case V4L2_COLORSPACE_BT2020: return AVCOL_SPC_BT2020_CL; > case V4L2_COLORSPACE_REC709: return AVCOL_SPC_BT709; > case V4L2_COLORSPACE_ADOBERGB: > case V4L2_COLORSPACE_SRGB: > case V4L2_COLORSPACE_DCI_P3: > case V4L2_COLORSPACE_JPEG: > case V4L2_COLORSPACE_RAW: > default: > return AVCOL_SPC_RGB; > } > } I don't know either, but for one, I don't think there's a special jpeg colorspace. It's just BT.601. Also, since you have to do these conversions in both direction, using a table might be better to avoid duplication? Also I just noticed I sent the previous mails to you personally, instead of the mailing list. Curse this mail client...
On Mon, 28 Aug 2017 10:21:54 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/25/2017 05:35 PM, wm4 wrote: > >> +int avpriv_v4l2m2m_reinit(V4L2Context* ctx) > >> +{ > >> + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, capture); > >> + int ret; > >> + > >> + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s reinit context\n", ctx->name); > >> + > >> + /* 1. streamoff */ > >> + ret = avpriv_v4l2_context_set_status(ctx, VIDIOC_STREAMOFF); > >> + if (ret) > >> + av_log(ctx->log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", ctx->name); > >> + > >> + /* 2. unmap the buffers (v4l2 and ffmpeg) */ > >> + avpriv_v4l2_context_release(ctx); > > What happens to AVFrames or AVPackets that are still referenced? > > > > yes in response to your previous comment on the subject, I am trying to > address that in v7 not releasing the context while there are buffers on > the fly.... > This is what I came up with (if you can suggest something better - ie, a > way to block until the AVBufferRefs are unref- please let me know) > > void avpriv_v4l2_context_release(V4L2Context* ctx) > { > int i, ret; > > if (!ctx->buffers) > return; > > /* wait until all buffers are no longer in use */ > for (i = 0; i < ctx->num_buffers; i++) { > > if (ctx->buffers[i].status & (V4L2BUF_IN_DRIVER | > V4L2BUF_AVAILABLE)) > continue; > > while (ctx->buffers[i].status & V4L2BUF_RET_USER) > usleep(100); > } > > ret = ctx->ops.release_buffers(ctx); > if (ret) > av_log(ctx->log_ctx, AV_LOG_WARNING, "V4L2 failed to unmap the > %s buffers\n", ctx->name); > else > av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s buffers unmapped\n", > ctx->name); > > av_free(ctx->buffers); > ctx->buffers = NULL; > ctx->num_queued = 0; > } > No, this is not possible. You must make the V4L state reference counted, and AVBufferRefs must implicitly reference this context. AVHWFramesContext does this in a more elegant way - it's a frame pool specifically made for hw decoders. I suspect the V4L code should be ported to that. I hope Mark Thompson has some helpful remarks on this.
On 08/28/2017 11:26 AM, Alexis Ballier wrote: > Hi, > > > On Sun, 27 Aug 2017 21:41:06 +0200 > Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > >> On 08/27/2017 09:09 PM, Jorge Ramirez wrote: >>> On 08/25/2017 05:35 PM, wm4 wrote: >>>>> +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, >>>>> V4L2Buffer *avbuf) +{ >>>>> + int i, ret; >>>>> + >>>>> + av_frame_unref(frame); >>>>> + >>>>> + /* 1. get references to the actual data */ >>>>> + for (i = 0; i < avbuf->num_planes; i++) { >>>>> + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); >>>>> + if (ret) >>>>> + return ret; >>>>> + >>>>> + frame->linesize[i] = avbuf->bytesperline[i]; >>>>> + frame->data[i] = frame->buf[i]->data; >>>>> + } >>>>> + >>>>> + /* 1.1 fixup special cases */ >>>>> + switch (avbuf->context->av_pix_fmt) { >>>>> + case AV_PIX_FMT_NV12: >>>>> + if (avbuf->num_planes > 1) >>>>> + break; >>>>> + frame->linesize[1] = avbuf->bytesperline[0]; >>>>> + frame->data[1] = frame->buf[0]->data + >>>>> avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; >>>>> + break; >>>>> + default: >>>>> + break; >>>>> + } >>>>> + >>>>> + /* 2. get frame information */ >>>>> + frame->key_frame = !!(avbuf->buf.flags & >>>>> V4L2_BUF_FLAG_KEYFRAME); >>>>> + frame->format = avbuf->context->av_pix_fmt; >>>>> + >>>>> + /* these values are updated also during re-init in >>>>> process_video_event */ >>>>> + frame->height = avbuf->context->height; >>>>> + frame->width = avbuf->context->width; >>>>> + frame->pts = get_pts(avbuf); >>>>> + >>>>> + /* 3. report errors upstream */ >>>>> + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { >>>>> + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: >>>>> driver decode error\n", avbuf->context->name); >>>>> + frame->decode_error_flags |= >>>>> FF_DECODE_ERROR_INVALID_BITSTREAM; >>>>> + } >>>>> + >>>>> + return 0; >>>>> +} >>>> This function seems to lack setting typically required metadata >>>> like colorspace. >>>> >>> ok I will retrieve the colorspace from the v4l2 format structure >>> and set it in the frame. >> um, I dont see a 1:1 mapping between the colorspaces reported from >> v4l2 and what ffmpeg expects (I am also not a subject matter expert >> on this :( ). >> >> Could I get some guidance on the translation table below please? >> btw in my simple tests - ffplay decoded vp8, vp9, mpeg4, mpeg2, h263, >> h264 and hevc I didnt need this field so I am not sure if I am >> getting it right. > > I don't think it will affect decoding, but the image will be something like too pale if it's set wrongly > > >> static inline enum AVColorSpace get_colorspace(V4L2Buffer *buf) >> { >> enum v4l2_colorspace cs; >> >> cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? >> buf->context->format.fmt.pix_mp.colorspace : >> buf->context->format.fmt.pix.colorspace; >> >> /* */ >> switch (cs) { >> case V4L2_COLORSPACE_SMPTE170M: return AVCOL_SPC_SMPTE170M; >> case V4L2_COLORSPACE_SMPTE240M: return AVCOL_SPC_SMPTE240M; >> case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_SPC_BT470BG; >> case V4L2_COLORSPACE_BT2020: return AVCOL_SPC_BT2020_CL; >> case V4L2_COLORSPACE_REC709: return AVCOL_SPC_BT709; >> case V4L2_COLORSPACE_ADOBERGB: >> case V4L2_COLORSPACE_SRGB: >> case V4L2_COLORSPACE_DCI_P3: >> case V4L2_COLORSPACE_JPEG: >> case V4L2_COLORSPACE_RAW: >> default: >> return AVCOL_SPC_RGB; >> } > > FWIW, here is the conversion I had been using in other projects: > > switch(f->colorspace) > { > case V4L2_COLORSPACE_SRGB: > return AVCOL_SPC_RGB; > case V4L2_COLORSPACE_REC709: > return AVCOL_SPC_BT709; > case V4L2_COLORSPACE_470_SYSTEM_M: > return AVCOL_SPC_FCC; > case V4L2_COLORSPACE_470_SYSTEM_BG: > return AVCOL_SPC_BT470BG; > case V4L2_COLORSPACE_SMPTE170M: > return AVCOL_SPC_SMPTE170M; > case V4L2_COLORSPACE_SMPTE240M: > return AVCOL_SPC_SMPTE240M; > case V4L2_COLORSPACE_BT2020: > if(f->ycbcr_enc == V4L2_YCBCR_ENC_BT2020_CONST_LUM) > return AVCOL_SPC_BT2020_CL; > return AVCOL_SPC_BT2020_NCL; > default: return AVCOL_SPC_UNSPECIFIED; > } > > > you'd likely need AVCOL_PRI*: > > switch(f->ycbcr_enc) > { > case V4L2_YCBCR_ENC_XV709: > case V4L2_YCBCR_ENC_709 : > return AVCOL_PRI_BT709; > case V4L2_YCBCR_ENC_XV601: > case V4L2_YCBCR_ENC_601: > return AVCOL_PRI_BT470M; > default: break; > } > > switch(f->colorspace) > { > case V4L2_COLORSPACE_470_SYSTEM_BG: > return AVCOL_PRI_BT470BG; > case V4L2_COLORSPACE_SMPTE170M: > return AVCOL_PRI_SMPTE170M; > case V4L2_COLORSPACE_SMPTE240M: > return AVCOL_PRI_SMPTE240M; > case V4L2_COLORSPACE_BT2020: > return AVCOL_PRI_BT2020; > default: break; > } > return AVCOL_PRI_UNSPECIFIED; > > > AVCOL_TRC*: > > switch(f->xfer_func) > { > case V4L2_XFER_FUNC_709: > switch(f->bit_depth) > { > case 10: return AVCOL_TRC_BT2020_10; > case 12: return AVCOL_TRC_BT2020_12; > default: break; > } > return AVCOL_TRC_BT709; > case V4L2_XFER_FUNC_SRGB: > return AVCOL_TRC_IEC61966_2_1; > default: break; > } > > switch(f->colorspace) > { > case V4L2_COLORSPACE_470_SYSTEM_M: > return AVCOL_TRC_GAMMA22; > case V4L2_COLORSPACE_470_SYSTEM_BG: > return AVCOL_TRC_GAMMA28; > case V4L2_COLORSPACE_SMPTE170M: > return AVCOL_TRC_SMPTE170M; > case V4L2_COLORSPACE_SMPTE240M: > return AVCOL_TRC_SMPTE240M; > default: break; > } > > switch(f->ycbcr_enc) > { > case V4L2_YCBCR_ENC_XV709: > case V4L2_YCBCR_ENC_XV601: > return AVCOL_TRC_BT1361_ECG; > default: break; > } > return AVCOL_TRC_UNSPECIFIED; > > AVCOL_RANGE*: > > switch(f->quantization) > { > case V4L2_QUANTIZATION_LIM_RANGE: > return AVCOL_RANGE_MPEG; > case V4L2_QUANTIZATION_FULL_RANGE: > return AVCOL_RANGE_JPEG; > default: break; > } > > return AVCOL_RANGE_UNSPECIFIED; > > > > Feel free to re-use, point and/or fix bugs here (it's been long since I wrote this and I don't think I've ever visually checked all the possibilities). > This might also be useful for the V4L2 output and capture drivers in lavd. awesome. thanks a lot for this information, will merge!
On 27/08/17 20:41, Jorge Ramirez wrote: > On 08/27/2017 09:09 PM, Jorge Ramirez wrote: >> On 08/25/2017 05:35 PM, wm4 wrote: >>>> +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) >>>> +{ >>>> + int i, ret; >>>> + >>>> + av_frame_unref(frame); >>>> + >>>> + /* 1. get references to the actual data */ >>>> + for (i = 0; i < avbuf->num_planes; i++) { >>>> + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); >>>> + if (ret) >>>> + return ret; >>>> + >>>> + frame->linesize[i] = avbuf->bytesperline[i]; >>>> + frame->data[i] = frame->buf[i]->data; >>>> + } >>>> + >>>> + /* 1.1 fixup special cases */ >>>> + switch (avbuf->context->av_pix_fmt) { >>>> + case AV_PIX_FMT_NV12: >>>> + if (avbuf->num_planes > 1) >>>> + break; >>>> + frame->linesize[1] = avbuf->bytesperline[0]; >>>> + frame->data[1] = frame->buf[0]->data + avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; >>>> + break; >>>> + default: >>>> + break; >>>> + } >>>> + >>>> + /* 2. get frame information */ >>>> + frame->key_frame = !!(avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME); >>>> + frame->format = avbuf->context->av_pix_fmt; >>>> + >>>> + /* these values are updated also during re-init in process_video_event */ >>>> + frame->height = avbuf->context->height; >>>> + frame->width = avbuf->context->width; >>>> + frame->pts = get_pts(avbuf); >>>> + >>>> + /* 3. report errors upstream */ >>>> + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { >>>> + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: driver decode error\n", avbuf->context->name); >>>> + frame->decode_error_flags |= FF_DECODE_ERROR_INVALID_BITSTREAM; >>>> + } >>>> + >>>> + return 0; >>>> +} >>> This function seems to lack setting typically required metadata like >>> colorspace. >>> >> >> ok I will retrieve the colorspace from the v4l2 format structure and set it in the frame. > > um, I dont see a 1:1 mapping between the colorspaces reported from v4l2 and what ffmpeg expects (I am also not a subject matter expert on this :( ). > > Could I get some guidance on the translation table below please? > btw in my simple tests - ffplay decoded vp8, vp9, mpeg4, mpeg2, h263, h264 and hevc I didnt need this field so I am not sure if I am getting it right. > > TIA > > static inline enum AVColorSpace get_colorspace(V4L2Buffer *buf) > { > enum v4l2_colorspace cs; > > cs = V4L2_TYPE_IS_MULTIPLANAR(buf->context->type) ? > buf->context->format.fmt.pix_mp.colorspace : > buf->context->format.fmt.pix.colorspace; > > /* */ > switch (cs) { > case V4L2_COLORSPACE_SMPTE170M: return AVCOL_SPC_SMPTE170M; > case V4L2_COLORSPACE_SMPTE240M: return AVCOL_SPC_SMPTE240M; > case V4L2_COLORSPACE_470_SYSTEM_BG: return AVCOL_SPC_BT470BG; > case V4L2_COLORSPACE_BT2020: return AVCOL_SPC_BT2020_CL; > case V4L2_COLORSPACE_REC709: return AVCOL_SPC_BT709; > case V4L2_COLORSPACE_ADOBERGB: > case V4L2_COLORSPACE_SRGB: > case V4L2_COLORSPACE_DCI_P3: > case V4L2_COLORSPACE_JPEG: > case V4L2_COLORSPACE_RAW: > default: > return AVCOL_SPC_RGB; > } > } The ffmpeg enums use the same ISO 11664-1 values as H.264, H.265 and MPEG-2 do. Since there must be code in your V4L2 driver to translate the V4L2 enums both to and from that (for encode and decode respectively), I suggest just copying the logic of that code to do the matching opposite direction on the ffmpeg side. (Look for code reading/writing colour_primaries, transfer_characteristics and matrix_coefficients in the VUI section of the SPS.) - Mark
On 08/28/2017 12:04 PM, wm4 wrote: >> yes in response to your previous comment on the subject, I am trying to >> address that in v7 not releasing the context while there are buffers on >> the fly.... >> This is what I came up with (if you can suggest something better - ie, a >> way to block until the AVBufferRefs are unref- please let me know) >> >> void avpriv_v4l2_context_release(V4L2Context* ctx) >> { >> int i, ret; >> >> if (!ctx->buffers) >> return; >> >> /* wait until all buffers are no longer in use */ >> for (i = 0; i < ctx->num_buffers; i++) { >> >> if (ctx->buffers[i].status & (V4L2BUF_IN_DRIVER | >> V4L2BUF_AVAILABLE)) >> continue; >> >> while (ctx->buffers[i].status & V4L2BUF_RET_USER) >> usleep(100); >> } >> >> ret = ctx->ops.release_buffers(ctx); >> if (ret) >> av_log(ctx->log_ctx, AV_LOG_WARNING, "V4L2 failed to unmap the >> %s buffers\n", ctx->name); >> else >> av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s buffers unmapped\n", >> ctx->name); >> >> av_free(ctx->buffers); >> ctx->buffers = NULL; >> ctx->num_queued = 0; >> } >> > No, this is not possible. ok > You must make the V4L state reference > counted, and AVBufferRefs must implicitly reference this context. that is what we tried to do: the AVBufferRefs "free" callback changes the state to AVAILABLE thus implicitly referencing the context. maybe I am misunderstanding what you are trying to say? > > AVHWFramesContext does this in a more elegant way - it's a frame pool > specifically made for hw decoders. I suspect the V4L code should be > ported to that. I hope Mark Thompson has some helpful remarks on this. I guess that instead of polling for the AVBufferRef to be unreferenced, I can associate a sync (ie a sempahore) to each buffer, take it on release and post the semaphore on the AVBufferRefs being unreferenced. that is actually pretty clean in terms of cpu usage.
On 25/08/17 16:35, wm4 wrote: > That looks generally OK. Is there any chance a hwaccel approach would > be possible instead? If I've learned anything about hardware decoding, > then that hwaccel is vastly superior to vendor-implemented full stream > decoders. Unfortunately I think the whole point of the V4L2 M2M push is to define a common way for vendors to implement full-stream decoders in the kernel without it affecting userspace at all. It's pretty much OpenMAX IL, except with all of the code pushed into the kernel so that users don't need to deal with extra userspace libraries. So, I don't think there is any chance of it working like this at all, though there might be something said for parsing the headers externally to extract all the metadata correctly before passing it to the kernel. - Mark
On Mon, 28 Aug 2017 12:33:38 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/28/2017 12:04 PM, wm4 wrote: > >> yes in response to your previous comment on the subject, I am trying to > >> address that in v7 not releasing the context while there are buffers on > >> the fly.... > >> This is what I came up with (if you can suggest something better - ie, a > >> way to block until the AVBufferRefs are unref- please let me know) > >> > >> void avpriv_v4l2_context_release(V4L2Context* ctx) > >> { > >> int i, ret; > >> > >> if (!ctx->buffers) > >> return; > >> > >> /* wait until all buffers are no longer in use */ > >> for (i = 0; i < ctx->num_buffers; i++) { > >> > >> if (ctx->buffers[i].status & (V4L2BUF_IN_DRIVER | > >> V4L2BUF_AVAILABLE)) > >> continue; > >> > >> while (ctx->buffers[i].status & V4L2BUF_RET_USER) > >> usleep(100); > >> } > >> > >> ret = ctx->ops.release_buffers(ctx); > >> if (ret) > >> av_log(ctx->log_ctx, AV_LOG_WARNING, "V4L2 failed to unmap the > >> %s buffers\n", ctx->name); > >> else > >> av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s buffers unmapped\n", > >> ctx->name); > >> > >> av_free(ctx->buffers); > >> ctx->buffers = NULL; > >> ctx->num_queued = 0; > >> } > >> > > No, this is not possible. > ok > > > You must make the V4L state reference > > counted, and AVBufferRefs must implicitly reference this context. > that is what we tried to do: the AVBufferRefs "free" callback changes > the state to AVAILABLE thus implicitly referencing the context. > maybe I am misunderstanding what you are trying to say? > > > > > AVHWFramesContext does this in a more elegant way - it's a frame pool > > specifically made for hw decoders. I suspect the V4L code should be > > ported to that. I hope Mark Thompson has some helpful remarks on this. > > I guess that instead of polling for the AVBufferRef to be unreferenced, > I can associate a sync (ie a sempahore) to each buffer, take it on > release and post the semaphore on the AVBufferRefs being unreferenced. > that is actually pretty clean in terms of cpu usage. That would just freeze an API user calling avcodec_close(), when it keeps around decoded AVFrames for later use.
On 08/28/2017 12:47 PM, wm4 wrote: >> I guess that instead of polling for the AVBufferRef to be unreferenced, >> I can associate a sync (ie a sempahore) to each buffer, take it on >> release and post the semaphore on the AVBufferRefs being unreferenced. >> that is actually pretty clean in terms of cpu usage. > That would just freeze an API user calling avcodec_close(), when it > keeps around decoded AVFrames for later use. yes I understand, but it does avoid using the CPU to poll for the buffer release (an incremental improvement) but yes I think that the message is that even though this proposal might suffice for simple video players (my tests) is not good enough for other users requiring the decoded frame for post processing. is this a blocker to upstream or could I continue working with it flagging the encoder/decoder as EXPERIMENTAL? the current situation at least keeps video players happy.
On 08/25/2017 05:35 PM, wm4 wrote: >> + >> +#define SET_V4L2_EXT_CTRL(TYPE, ID, VALUE, NAME) \ >> +{ \ >> + struct v4l2_ext_control ctrl = { 0 }; \ >> + struct v4l2_ext_controls ctrls = { 0 }; \ >> + ctrls.ctrl_class = V4L2_CTRL_CLASS_MPEG; \ >> + ctrls.controls = &ctrl; \ >> + ctrl.TYPE = VALUE ; \ >> + ctrl.id = ID ; \ >> + ctrls.count = 1; \ >> + \ >> + if ((ret = ioctl(s->fd, VIDIOC_S_EXT_CTRLS, &ctrls)) < 0) \ >> + av_log(avctx, AV_LOG_WARNING, "Failed to set " NAME "%s\n", STR(ID)); \ >> +} >> + >> +#define SET_V4L2_TIME_PER_FRAME(NUM, DEN) \ >> +{ \ >> + struct v4l2_streamparm parm = { 0 }; \ >> + parm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; \ >> + parm.parm.output.timeperframe.numerator = NUM; \ >> + parm.parm.output.timeperframe.denominator = DEN; \ >> + \ >> + if ((ret = ioctl(s->fd, VIDIOC_S_PARM, &parm)) < 0) \ >> + av_log(avctx, AV_LOG_WARNING, "Failed to set timeperframe"); \ >> +} > I think those should be functions. ctrl has only 3 fields, so it's ok > to have the caller set it up. (If you use C99 struct literals it won't > be more code than before, maybe.) > ok.
On 08/25/2017 05:35 PM, wm4 wrote: >> +static inline int v4l2_h264_profile_from_ff(int p) >> +{ >> + switch(p) { >> + case FF_PROFILE_H264_CONSTRAINED_BASELINE: >> + return MPEG_VIDEO(H264_PROFILE_CONSTRAINED_BASELINE); >> + case FF_PROFILE_H264_HIGH_444_PREDICTIVE: >> + return MPEG_VIDEO(H264_PROFILE_HIGH_444_PREDICTIVE); >> + case FF_PROFILE_H264_HIGH_422_INTRA: >> + return MPEG_VIDEO(H264_PROFILE_HIGH_422_INTRA); >> + case FF_PROFILE_H264_HIGH_444_INTRA: >> + return MPEG_VIDEO(H264_PROFILE_HIGH_444_INTRA); >> + case FF_PROFILE_H264_HIGH_10_INTRA: >> + return MPEG_VIDEO(H264_PROFILE_HIGH_10_INTRA); >> + case FF_PROFILE_H264_HIGH_422: >> + return MPEG_VIDEO(H264_PROFILE_HIGH_422); >> + case FF_PROFILE_H264_BASELINE: >> + return MPEG_VIDEO(H264_PROFILE_BASELINE); >> + case FF_PROFILE_H264_EXTENDED: >> + return MPEG_VIDEO(H264_PROFILE_EXTENDED); >> + case FF_PROFILE_H264_HIGH_10: >> + return MPEG_VIDEO(H264_PROFILE_HIGH_10); >> + case FF_PROFILE_H264_MAIN: >> + return MPEG_VIDEO(H264_PROFILE_MAIN); >> + case FF_PROFILE_H264_HIGH: >> + return MPEG_VIDEO(H264_PROFILE_HIGH); >> + } >> + >> + return -1; >> +} >> + >> +static inline int v4l2_mpeg4_profile_from_ff(int p) >> +{ >> + switch(p) { >> + case FF_PROFILE_MPEG4_ADVANCED_CODING: >> + return MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_CODING_EFFICIENCY); >> + case FF_PROFILE_MPEG4_ADVANCED_SIMPLE: >> + return MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_SIMPLE); >> + case FF_PROFILE_MPEG4_SIMPLE_SCALABLE: >> + >> + return MPEG_VIDEO(MPEG4_PROFILE_SIMPLE_SCALABLE); >> + case FF_PROFILE_MPEG4_SIMPLE: >> + return MPEG_VIDEO(MPEG4_PROFILE_SIMPLE); >> + case FF_PROFILE_MPEG4_CORE: >> + return MPEG_VIDEO(MPEG4_PROFILE_CORE); >> + } >> + >> + return -1; >> +} > Would a table be better maybe? > ok, switch replace with tables.
On 08/28/2017 02:16 PM, Jorge Ramirez wrote: > On 08/28/2017 12:47 PM, wm4 wrote: >>> I guess that instead of polling for the AVBufferRef to be unreferenced, >>> I can associate a sync (ie a sempahore) to each buffer, take it on >>> release and post the semaphore on the AVBufferRefs being unreferenced. >>> that is actually pretty clean in terms of cpu usage. >> That would just freeze an API user calling avcodec_close(), when it >> keeps around decoded AVFrames for later use. > > yes I understand, but it does avoid using the CPU to poll for the > buffer release (an incremental improvement) > > but yes I think that the message is that even though this proposal > might suffice for simple video players (my tests) is not good enough > for other users requiring the decoded frame for post processing. > > is this a blocker to upstream or could I continue working with it > flagging the encoder/decoder as EXPERIMENTAL? the current situation at > least keeps video players happy. > > just wondering, if the AVBufferRefs must live for ever (ie, after the codecs have been closed), what do other codecs dequeuing from a limited number of re-usable hardware allocated buffers do? do they use the CPU allocate and copy the data from those buffers to the heap?
On Mon, 28 Aug 2017 21:24:26 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/28/2017 02:16 PM, Jorge Ramirez wrote: > > On 08/28/2017 12:47 PM, wm4 wrote: > >>> I guess that instead of polling for the AVBufferRef to be unreferenced, > >>> I can associate a sync (ie a sempahore) to each buffer, take it on > >>> release and post the semaphore on the AVBufferRefs being unreferenced. > >>> that is actually pretty clean in terms of cpu usage. > >> That would just freeze an API user calling avcodec_close(), when it > >> keeps around decoded AVFrames for later use. > > > > yes I understand, but it does avoid using the CPU to poll for the > > buffer release (an incremental improvement) > > > > but yes I think that the message is that even though this proposal > > might suffice for simple video players (my tests) is not good enough > > for other users requiring the decoded frame for post processing. > > > > is this a blocker to upstream or could I continue working with it > > flagging the encoder/decoder as EXPERIMENTAL? the current situation at > > least keeps video players happy. I'd say yes this is a blocker. We usually try to avoid committing half-finished code, because it often means it will be never finished. > > > > > > just wondering, if the AVBufferRefs must live for ever (ie, after the > codecs have been closed), what do other codecs dequeuing from a limited > number of re-usable hardware allocated buffers do? > do they use the CPU allocate and copy the data from those buffers to the > heap? > Like I wrote before: hwaccels use AVHWFramesContext, which was made more or less for this situation. If you want FD support later (for something like zero-copy transcoding or playback), AVHWFramesContext will probably be mandatory anyway. But I guess it's a big change for someone not familiar with the codebase. But manually "nesting" AVBufferRefs to make any underlying state refcounted would also work.
On 08/28/2017 09:53 PM, wm4 wrote: > On Mon, 28 Aug 2017 21:24:26 +0200 > Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > >> On 08/28/2017 02:16 PM, Jorge Ramirez wrote: >>> On 08/28/2017 12:47 PM, wm4 wrote: >>>>> I guess that instead of polling for the AVBufferRef to be unreferenced, >>>>> I can associate a sync (ie a sempahore) to each buffer, take it on >>>>> release and post the semaphore on the AVBufferRefs being unreferenced. >>>>> that is actually pretty clean in terms of cpu usage. >>>> That would just freeze an API user calling avcodec_close(), when it >>>> keeps around decoded AVFrames for later use. >>> yes I understand, but it does avoid using the CPU to poll for the >>> buffer release (an incremental improvement) >>> >>> but yes I think that the message is that even though this proposal >>> might suffice for simple video players (my tests) is not good enough >>> for other users requiring the decoded frame for post processing. >>> >>> is this a blocker to upstream or could I continue working with it >>> flagging the encoder/decoder as EXPERIMENTAL? the current situation at >>> least keeps video players happy. > I'd say yes this is a blocker. We usually try to avoid committing > half-finished code, because it often means it will be never finished. hi, I forgot to say earlier, thanks for all the review over the past couple of days (it has been of much help). on the half finished matter, the issue that I face is that the current code doesn't cover the use case where _all_ the processed frames have to be kept available indefinitely (this is why I thought that perhaps setting .capabilities to AV_CODEC_CAP_EXPERIMENTAL could be an option to upstream and get more exposure to other users; I do plan to continue supporting v4l2 ffmpeg integration - mmaped filters, DRM and so on...having invested this long I do want to see this through; and since I can't guaranteed that some "force majeure" wont happen I think the sooner the code I have been working on can get exposure the sooner we will start seeing contributions. Anyhow, the current code does support the typical use case of most video players so it would benefit a considerable amount of users. does it have to be an all or nothing at this point or could we flag the v4l2 m2m as experimental codecs? > >>> >> just wondering, if the AVBufferRefs must live for ever (ie, after the >> codecs have been closed), what do other codecs dequeuing from a limited >> number of re-usable hardware allocated buffers do? >> do they use the CPU allocate and copy the data from those buffers to the >> heap? >> > Like I wrote before: hwaccels use AVHWFramesContext, which was made > more or less for this situation. If you want FD support later (for > something like zero-copy transcoding or playback), AVHWFramesContext > will probably be mandatory anyway. But I guess it's a big change for > someone not familiar with the codebase. Yes I had a look and it seems not an easy change to integrate. Still I'd like to make sure we are talking about the same requirement because if AVHWFramesContext works around the issue [1] , I can probably do the same with a few more lines of code (including the FD support for v4l2 which is pretty straight forward) [1] When: a) there is a limited number of buffers allocated by the hardware and b) these buffers are mapped to the process address space and c) the application can choose to keep _all_ decoded buffers for post processing, then there is no other option than copying each of the processed buffers to newly allocated areas in the heap (there can be no magic on this since the hardware buffers are always limited and have to be reused). I had a look a AVHWFRamesContext and it seems to me that under the transfer frames semantics it performs some sort of memcpy in/out (something I could do on every capture buffer dequeue if this is the requirement). I could be wrong and therefore would appreciate the clarification if the previous comment is incorrect. notice that I do insist on continue using V4L2_MEMORY_MMAP (instead of switching to V4L2_MEMORY_USERPTR) because it is easy to export the buffers as DMABUFs (~30 lines of code) and then pass these in FDs (which could be associated to short lived AVBufferRefs for DRM) > > But manually "nesting" AVBufferRefs to make any underlying state > refcounted would also work. I think so, context release now looks like this (it raises an ERROR to the user) but will not lock or poll. void avpriv_v4l2_context_release(V4L2Context* ctx) { struct timespec timeout = { 0, 0}; int i, ret; if (!ctx->buffers) return; timeout.tv_sec = av_gettime() / 1000000 + 10; /* wait until all buffers owned by the user are returned */ for (i = 0; i < ctx->num_buffers; i++) { for (;;) { ret = sem_timedwait(&ctx->buffers[i].delete_sync, &timeout); if (!ret) break; if (errno == EINTR) continue; if (errno == ETIMEDOUT) av_log(ctx->log_ctx, AV_LOG_ERROR, "AVBufferRef nbr %d in use, bad things might happen\n", i); else av_log(ctx->log_ctx, AV_LOG_ERROR, "sem_wait %s\n", av_err2str(AVERROR(errno))); break; } } ret = ctx->ops.release_buffers(ctx); if (ret) av_log(ctx->log_ctx, AV_LOG_WARNING, "V4L2 failed to unmap the %s buffers\n", ctx->name); else av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s buffers unmapped\n", ctx->name); av_free(ctx->buffers); ctx->buffers = NULL; } thanks, > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On Mon, Aug 28, 2017 at 11:36 PM, Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: >> >> But manually "nesting" AVBufferRefs to make any underlying state >> refcounted would also work. > > > I think so, context release now looks like this (it raises an ERROR to the > user) but will not lock or poll. > Thats not really what he was referring to, but to avoid the "blocking" entirely. Basically, what you would do is have an AVBufferRef that basically "holds" your v4l2 context, and every frame holds a reference on this AVBufferRef - and only when all frames are released, the "free" function on your context AVBufferRef is called, and you can then release all its resources - without having to block the avcodec_close function. This is basically what AVHWFramesContext does, just with more frills around it. - Hendrik
On 08/28/2017 11:42 PM, Hendrik Leppkes wrote: > On Mon, Aug 28, 2017 at 11:36 PM, Jorge Ramirez > <jorge.ramirez-ortiz@linaro.org> wrote: >>> But manually "nesting" AVBufferRefs to make any underlying state >>> refcounted would also work. >> >> I think so, context release now looks like this (it raises an ERROR to the >> user) but will not lock or poll. >> > Thats not really what he was referring to, but to avoid the "blocking" entirely. > > Basically, what you would do is have an AVBufferRef that basically > "holds" your v4l2 context, and every frame holds a reference on this > AVBufferRef - and only when all frames are released, the "free" > function on your context AVBufferRef is called, and you can then > release all its resources - without having to block the avcodec_close > function. I see, thanks for the info still, with the current v4l2 buffer design (previous thread in the discussion), I can't allow avcodec_close to complete - so I have to block or at least timeblock- when AVBuffersRefs pointing to v4l2 buffers have not been released by the ffmpeg user (if the user tried to access that memory it would result in bus errors since the mmaped addresses would not be valid) > This is basically what AVHWFramesContext does, just with more frills around it. ah! but what about memcpies (do you know if the AVHWFramesContext framework copies to the hardware buffers before processing and then back to user buffers? because I see no alternative if the AVBufferRefs must be kept alive for ever... > > - Hendrik > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On 08/29/2017 12:18 AM, Hendrik Leppkes wrote: > On Tue, Aug 29, 2017 at 12:07 AM, Jorge Ramirez > <jorge.ramirez-ortiz@linaro.org> wrote: >> On 08/28/2017 11:42 PM, Hendrik Leppkes wrote: >>> On Mon, Aug 28, 2017 at 11:36 PM, Jorge Ramirez >>> <jorge.ramirez-ortiz@linaro.org> wrote: >>>>> But manually "nesting" AVBufferRefs to make any underlying state >>>>> refcounted would also work. >>>> >>>> I think so, context release now looks like this (it raises an ERROR to >>>> the >>>> user) but will not lock or poll. >>>> >>> Thats not really what he was referring to, but to avoid the "blocking" >>> entirely. >>> >>> Basically, what you would do is have an AVBufferRef that basically >>> "holds" your v4l2 context, and every frame holds a reference on this >>> AVBufferRef - and only when all frames are released, the "free" >>> function on your context AVBufferRef is called, and you can then >>> release all its resources - without having to block the avcodec_close >>> function. >> >> I see, thanks for the info >> >> still, with the current v4l2 buffer design (previous thread in the >> discussion), I can't allow avcodec_close to complete - so I have to block or >> at least timeblock- when AVBuffersRefs pointing to v4l2 buffers have not >> been released by the ffmpeg user (if the user tried to access that memory it >> would result in bus errors since the mmaped addresses would not be valid) > > The entire point here is that avcodec_close would basically not close > much (only things that can be free'ed without consequences) as long as > buffers are still referenced, and the actual closing of hardware > resources would happen in the free callback fired once all AVBufferRef > are unused. ok will try to delegate closing the hardware and doing any operations that might invalidate any of the buffers that are on the fly to the last reference being released (avcodec_close will probably not be doing much then) > >> >>> This is basically what AVHWFramesContext does, just with more frills >>> around it. >> >> ah! >> but what about memcpies (do you know if the AVHWFramesContext framework >> copies to the hardware buffers before processing and then back to user >> buffers? because I see no alternative if the AVBufferRefs must be kept alive >> for ever... > Most hardware formats cannot be accessed by "software" without some > sort of conversion, thats where those memcpys come in - they download > the frame from special gpu memory to ordinary memory so they can be > further processed by any ordinary code. > This is handy for plugging hardware decoding into any old workflow > that already exists. > > However, it is not required - you can have a fully optimized path and > keep everything in gpu memory at all times, assuming all components in > between support any special access requirements imposed by the gpu. ok. makes sense. thanks a lot Hendrik. > > - Hendrik [apologies, it seems I took the thread out of the ML when I initally replied. bringing it back]
On Mon, 28 Aug 2017 23:36:08 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/28/2017 09:53 PM, wm4 wrote: > > On Mon, 28 Aug 2017 21:24:26 +0200 > > Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > > > >> On 08/28/2017 02:16 PM, Jorge Ramirez wrote: > >>> On 08/28/2017 12:47 PM, wm4 wrote: > >>>>> I guess that instead of polling for the AVBufferRef to be unreferenced, > >>>>> I can associate a sync (ie a sempahore) to each buffer, take it on > >>>>> release and post the semaphore on the AVBufferRefs being unreferenced. > >>>>> that is actually pretty clean in terms of cpu usage. > >>>> That would just freeze an API user calling avcodec_close(), when it > >>>> keeps around decoded AVFrames for later use. > >>> yes I understand, but it does avoid using the CPU to poll for the > >>> buffer release (an incremental improvement) > >>> > >>> but yes I think that the message is that even though this proposal > >>> might suffice for simple video players (my tests) is not good enough > >>> for other users requiring the decoded frame for post processing. > >>> > >>> is this a blocker to upstream or could I continue working with it > >>> flagging the encoder/decoder as EXPERIMENTAL? the current situation at > >>> least keeps video players happy. > > I'd say yes this is a blocker. We usually try to avoid committing > > half-finished code, because it often means it will be never finished. > > hi, I forgot to say earlier, thanks for all the review over the past > couple of days (it has been of much help). > > on the half finished matter, the issue that I face is that the current > code doesn't cover the use case where _all_ the processed frames have to > be kept available indefinitely (this is why I thought that perhaps > setting .capabilities to AV_CODEC_CAP_EXPERIMENTAL could be an option to > upstream and get more exposure to other users; > > I do plan to continue supporting v4l2 ffmpeg integration - mmaped > filters, DRM and so on...having invested this long I do want to see this > through; and since I can't guaranteed that some "force majeure" wont > happen I think the sooner the code I have been working on can get > exposure the sooner we will start seeing contributions. > > Anyhow, the current code does support the typical use case of most video > players so it would benefit a considerable amount of users. > > does it have to be an all or nothing at this point or could we flag the > v4l2 m2m as experimental codecs? You could just copy the frames before returning them to the user to avoid breaking refcounting. > > > >>> > >> just wondering, if the AVBufferRefs must live for ever (ie, after the > >> codecs have been closed), what do other codecs dequeuing from a limited > >> number of re-usable hardware allocated buffers do? > >> do they use the CPU allocate and copy the data from those buffers to the > >> heap? > >> > > Like I wrote before: hwaccels use AVHWFramesContext, which was made > > more or less for this situation. If you want FD support later (for > > something like zero-copy transcoding or playback), AVHWFramesContext > > will probably be mandatory anyway. But I guess it's a big change for > > someone not familiar with the codebase. > > Yes I had a look and it seems not an easy change to integrate. > > Still I'd like to make sure we are talking about the same requirement > because if AVHWFramesContext works around the issue [1] , I can probably > do the same with a few more lines of code (including the FD support for > v4l2 which is pretty straight forward) > > [1] When: > a) there is a limited number of buffers allocated by the hardware and > b) these buffers are mapped to the process address space and > c) the application can choose to keep _all_ decoded buffers for post > processing, > > then there is no other option than copying each of the processed buffers > to newly allocated areas in the heap (there can be no magic on this > since the hardware buffers are always limited and have to be reused). The semantics of AVHWFramesContext are such that if the user keeps enough AVFrames references to exhaust the frame pool, trying to continue decoding will result in an error. The whole point is to make the limited and fixed buffer allocation visible to the API user. We are also thinking about adding an API that lets the decoder communicate to the user how many references are required (inherently, and due to codec reference frame requirements). > I had a look a AVHWFRamesContext and it seems to me that under the > transfer frames semantics it performs some sort of memcpy in/out > (something I could do on every capture buffer dequeue if this is the > requirement). I could be wrong and therefore would appreciate the > clarification if the previous comment is incorrect. The AVFrames in the context pool would be opaque frames (using FDs), and there are entry points for temporary and permanent mapping of opaque frames, which in your code would call mmap. > notice that I do insist on continue using V4L2_MEMORY_MMAP (instead of > switching to V4L2_MEMORY_USERPTR) because it is easy to export the > buffers as DMABUFs (~30 lines of code) and then pass these in FDs (which > could be associated to short lived AVBufferRefs for DRM) No idea what's the difference between those. If you want to support direct FD/dmabuf export, adapting to AVHWFramesContext now would probably be easier in total. Especially because of the implied API change. But I'd wait for Mark Thompson's comments on that before making any big changes. AFAIK he posted a proposal patch for a DRM AVHWFramesContext too. > > > > > But manually "nesting" AVBufferRefs to make any underlying state > > refcounted would also work. > > I think so, context release now looks like this (it raises an ERROR to > the user) but will not lock or poll. > > void avpriv_v4l2_context_release(V4L2Context* ctx) > { > struct timespec timeout = { 0, 0}; > int i, ret; > > if (!ctx->buffers) > return; > > timeout.tv_sec = av_gettime() / 1000000 + 10; > > /* wait until all buffers owned by the user are returned */ > for (i = 0; i < ctx->num_buffers; i++) { > for (;;) { > ret = sem_timedwait(&ctx->buffers[i].delete_sync, &timeout); > if (!ret) > break; > if (errno == EINTR) > continue; > if (errno == ETIMEDOUT) > av_log(ctx->log_ctx, AV_LOG_ERROR, "AVBufferRef nbr %d > in use, bad things might happen\n", i); Undefined behavior after timeout? How is that reasonable in any way?
On 08/29/2017 10:56 AM, wm4 wrote: > On Mon, 28 Aug 2017 23:36:08 +0200 > Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > >> On 08/28/2017 09:53 PM, wm4 wrote: >>> On Mon, 28 Aug 2017 21:24:26 +0200 >>> Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: >>> >>>> On 08/28/2017 02:16 PM, Jorge Ramirez wrote: >>>>> On 08/28/2017 12:47 PM, wm4 wrote: >>>>>>> I guess that instead of polling for the AVBufferRef to be unreferenced, >>>>>>> I can associate a sync (ie a sempahore) to each buffer, take it on >>>>>>> release and post the semaphore on the AVBufferRefs being unreferenced. >>>>>>> that is actually pretty clean in terms of cpu usage. >>>>>> That would just freeze an API user calling avcodec_close(), when it >>>>>> keeps around decoded AVFrames for later use. >>>>> yes I understand, but it does avoid using the CPU to poll for the >>>>> buffer release (an incremental improvement) >>>>> >>>>> but yes I think that the message is that even though this proposal >>>>> might suffice for simple video players (my tests) is not good enough >>>>> for other users requiring the decoded frame for post processing. >>>>> >>>>> is this a blocker to upstream or could I continue working with it >>>>> flagging the encoder/decoder as EXPERIMENTAL? the current situation at >>>>> least keeps video players happy. >>> I'd say yes this is a blocker. We usually try to avoid committing >>> half-finished code, because it often means it will be never finished. >> hi, I forgot to say earlier, thanks for all the review over the past >> couple of days (it has been of much help). >> >> on the half finished matter, the issue that I face is that the current >> code doesn't cover the use case where _all_ the processed frames have to >> be kept available indefinitely (this is why I thought that perhaps >> setting .capabilities to AV_CODEC_CAP_EXPERIMENTAL could be an option to >> upstream and get more exposure to other users; >> >> I do plan to continue supporting v4l2 ffmpeg integration - mmaped >> filters, DRM and so on...having invested this long I do want to see this >> through; and since I can't guaranteed that some "force majeure" wont >> happen I think the sooner the code I have been working on can get >> exposure the sooner we will start seeing contributions. >> >> Anyhow, the current code does support the typical use case of most video >> players so it would benefit a considerable amount of users. >> >> does it have to be an all or nothing at this point or could we flag the >> v4l2 m2m as experimental codecs? > You could just copy the frames before returning them to the user to > avoid breaking refcounting. thinking again about this I'd rather not do that (it will impact performance too much) and Hendrik gave me some pointers yesterday in line with what you said as well. I implemented reference counting delegating the closing of _some_ resources needed to keep the buffers alive. closing the codec now doesnt wait or leave dangling buffers. the AVBufferRef free callback looks just like this static void free_v4l2buf_cb(void *opaque, uint8_t *unused) { V4L2Buffer* avbuf = opaque; V4L2m2mContext *s = container_of(avbuf->context, V4L2m2mContext, capture); atomic_fetch_sub_explicit(&s->refcount, 1, memory_order_acq_rel); if (s->reinit) { if (!atomic_load(&s->refcount)) sem_post(&s->refsync); return; } if (avbuf->context->streamon) { avbuf->context->ops.enqueue(avbuf); return; } if (!atomic_load(&s->refcount)) avpriv_v4l2m2m_end(s); } The only case where I can't get away without waiting for the AVBufferRef to be released is when re-initializing the frame dimensions (ie, resolution changes/format) _during_ streaming since I need to release _all_ hardware buffers and queue them again. will this be acceptable? I have just tested these changes and works as expected. > >>> >>>>> >>>> just wondering, if the AVBufferRefs must live for ever (ie, after the >>>> codecs have been closed), what do other codecs dequeuing from a limited >>>> number of re-usable hardware allocated buffers do? >>>> do they use the CPU allocate and copy the data from those buffers to the >>>> heap? >>>> >>> Like I wrote before: hwaccels use AVHWFramesContext, which was made >>> more or less for this situation. If you want FD support later (for >>> something like zero-copy transcoding or playback), AVHWFramesContext >>> will probably be mandatory anyway. But I guess it's a big change for >>> someone not familiar with the codebase. >> Yes I had a look and it seems not an easy change to integrate. >> >> Still I'd like to make sure we are talking about the same requirement >> because if AVHWFramesContext works around the issue [1] , I can probably >> do the same with a few more lines of code (including the FD support for >> v4l2 which is pretty straight forward) >> >> [1] When: >> a) there is a limited number of buffers allocated by the hardware and >> b) these buffers are mapped to the process address space and >> c) the application can choose to keep _all_ decoded buffers for post >> processing, >> >> then there is no other option than copying each of the processed buffers >> to newly allocated areas in the heap (there can be no magic on this >> since the hardware buffers are always limited and have to be reused). > The semantics of AVHWFramesContext are such that if the user keeps > enough AVFrames references to exhaust the frame pool, trying to continue > decoding will result in an error. The whole point is to make the > limited and fixed buffer allocation visible to the API user. makes sense although I havent found such an interface; in my view, the user should be able to register an observer to receive async events from codecs (be these from hardware or codec state machines) could you point me where that is? the way I understand ffmpeg is that everything seem to be working synchronously with no room for events like this (so the user would only be reported of an error after it tries to get a frame for instance..) > > We are also thinking about adding an API that lets the decoder > communicate to the user how many references are required (inherently, > and due to codec reference frame requirements). that is an interface that I would welcome as well. > >> I had a look a AVHWFRamesContext and it seems to me that under the >> transfer frames semantics it performs some sort of memcpy in/out >> (something I could do on every capture buffer dequeue if this is the >> requirement). I could be wrong and therefore would appreciate the >> clarification if the previous comment is incorrect. > The AVFrames in the context pool would be opaque frames (using FDs), > and there are entry points for temporary and permanent mapping of > opaque frames, which in your code would call mmap. thanks. I'll have a look at this. > >> notice that I do insist on continue using V4L2_MEMORY_MMAP (instead of >> switching to V4L2_MEMORY_USERPTR) because it is easy to export the >> buffers as DMABUFs (~30 lines of code) and then pass these in FDs (which >> could be associated to short lived AVBufferRefs for DRM) > No idea what's the difference between those. > > If you want to support direct FD/dmabuf export, adapting to > AVHWFramesContext now would probably be easier in total. Especially > because of the implied API change. But I'd wait for Mark Thompson's > comments on that before making any big changes. AFAIK he posted a > proposal patch for a DRM AVHWFramesContext too. why not just add a FD to the AVBufferRef and let the user decide whether to use it or not? > >>> But manually "nesting" AVBufferRefs to make any underlying state >>> refcounted would also work. >> I think so, context release now looks like this (it raises an ERROR to >> the user) but will not lock or poll. >> >> void avpriv_v4l2_context_release(V4L2Context* ctx) >> { >> struct timespec timeout = { 0, 0}; >> int i, ret; >> >> if (!ctx->buffers) >> return; >> >> timeout.tv_sec = av_gettime() / 1000000 + 10; >> >> /* wait until all buffers owned by the user are returned */ >> for (i = 0; i < ctx->num_buffers; i++) { >> for (;;) { >> ret = sem_timedwait(&ctx->buffers[i].delete_sync, &timeout); >> if (!ret) >> break; >> if (errno == EINTR) >> continue; >> if (errno == ETIMEDOUT) >> av_log(ctx->log_ctx, AV_LOG_ERROR, "AVBufferRef nbr %d >> in use, bad things might happen\n", i); > Undefined behavior after timeout? How is that reasonable in any way? is not! yes, please forget about the above... > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
On Tue, 29 Aug 2017 12:03:42 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/29/2017 10:56 AM, wm4 wrote: > > On Mon, 28 Aug 2017 23:36:08 +0200 > > Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > > > >> On 08/28/2017 09:53 PM, wm4 wrote: > >>> On Mon, 28 Aug 2017 21:24:26 +0200 > >>> Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > >>> > >>>> On 08/28/2017 02:16 PM, Jorge Ramirez wrote: > >>>>> On 08/28/2017 12:47 PM, wm4 wrote: > >>>>>>> I guess that instead of polling for the AVBufferRef to be unreferenced, > >>>>>>> I can associate a sync (ie a sempahore) to each buffer, take it on > >>>>>>> release and post the semaphore on the AVBufferRefs being unreferenced. > >>>>>>> that is actually pretty clean in terms of cpu usage. > >>>>>> That would just freeze an API user calling avcodec_close(), when it > >>>>>> keeps around decoded AVFrames for later use. > >>>>> yes I understand, but it does avoid using the CPU to poll for the > >>>>> buffer release (an incremental improvement) > >>>>> > >>>>> but yes I think that the message is that even though this proposal > >>>>> might suffice for simple video players (my tests) is not good enough > >>>>> for other users requiring the decoded frame for post processing. > >>>>> > >>>>> is this a blocker to upstream or could I continue working with it > >>>>> flagging the encoder/decoder as EXPERIMENTAL? the current situation at > >>>>> least keeps video players happy. > >>> I'd say yes this is a blocker. We usually try to avoid committing > >>> half-finished code, because it often means it will be never finished. > >> hi, I forgot to say earlier, thanks for all the review over the past > >> couple of days (it has been of much help). > >> > >> on the half finished matter, the issue that I face is that the current > >> code doesn't cover the use case where _all_ the processed frames have to > >> be kept available indefinitely (this is why I thought that perhaps > >> setting .capabilities to AV_CODEC_CAP_EXPERIMENTAL could be an option to > >> upstream and get more exposure to other users; > >> > >> I do plan to continue supporting v4l2 ffmpeg integration - mmaped > >> filters, DRM and so on...having invested this long I do want to see this > >> through; and since I can't guaranteed that some "force majeure" wont > >> happen I think the sooner the code I have been working on can get > >> exposure the sooner we will start seeing contributions. > >> > >> Anyhow, the current code does support the typical use case of most video > >> players so it would benefit a considerable amount of users. > >> > >> does it have to be an all or nothing at this point or could we flag the > >> v4l2 m2m as experimental codecs? > > You could just copy the frames before returning them to the user to > > avoid breaking refcounting. > > thinking again about this I'd rather not do that (it will impact > performance too much) and Hendrik gave me some pointers yesterday in > line with what you said as well. > I implemented reference counting delegating the closing of _some_ > resources needed to keep the buffers alive. > > closing the codec now doesnt wait or leave dangling buffers. > > the AVBufferRef free callback looks just like this > > static void free_v4l2buf_cb(void *opaque, uint8_t *unused) > { > V4L2Buffer* avbuf = opaque; > V4L2m2mContext *s = container_of(avbuf->context, V4L2m2mContext, > capture); > > atomic_fetch_sub_explicit(&s->refcount, 1, memory_order_acq_rel); > > if (s->reinit) { > if (!atomic_load(&s->refcount)) > sem_post(&s->refsync); > return; > } > > if (avbuf->context->streamon) { > avbuf->context->ops.enqueue(avbuf); > return; > } > > if (!atomic_load(&s->refcount)) > avpriv_v4l2m2m_end(s); > } > > The only case where I can't get away without waiting for the AVBufferRef > to be released is when re-initializing the frame dimensions (ie, > resolution changes/format) _during_ streaming since I need to release > _all_ hardware buffers and queue them again. > > will this be acceptable? > I have just tested these changes and works as expected. The implementation seems rather roundabout and complex - why not use AVBufferRef? But apart from that, yes. > > > >>> > >>>>> > >>>> just wondering, if the AVBufferRefs must live for ever (ie, after the > >>>> codecs have been closed), what do other codecs dequeuing from a limited > >>>> number of re-usable hardware allocated buffers do? > >>>> do they use the CPU allocate and copy the data from those buffers to the > >>>> heap? > >>>> > >>> Like I wrote before: hwaccels use AVHWFramesContext, which was made > >>> more or less for this situation. If you want FD support later (for > >>> something like zero-copy transcoding or playback), AVHWFramesContext > >>> will probably be mandatory anyway. But I guess it's a big change for > >>> someone not familiar with the codebase. > >> Yes I had a look and it seems not an easy change to integrate. > >> > >> Still I'd like to make sure we are talking about the same requirement > >> because if AVHWFramesContext works around the issue [1] , I can probably > >> do the same with a few more lines of code (including the FD support for > >> v4l2 which is pretty straight forward) > >> > >> [1] When: > >> a) there is a limited number of buffers allocated by the hardware and > >> b) these buffers are mapped to the process address space and > >> c) the application can choose to keep _all_ decoded buffers for post > >> processing, > >> > >> then there is no other option than copying each of the processed buffers > >> to newly allocated areas in the heap (there can be no magic on this > >> since the hardware buffers are always limited and have to be reused). > > The semantics of AVHWFramesContext are such that if the user keeps > > enough AVFrames references to exhaust the frame pool, trying to continue > > decoding will result in an error. The whole point is to make the > > limited and fixed buffer allocation visible to the API user. > > makes sense although I havent found such an interface; in my view, the > user should be able to register an observer to receive async events from > codecs (be these from hardware or codec state machines) > could you point me where that is? the way I understand ffmpeg is that > everything seem to be working synchronously with no room for events like > this (so the user would only be reported of an error after it tries to > get a frame for instance..) that doesn't seem to have anything to do with our previous discussion. Async behavior could be easily added to the API. But the current API is a state machine that doesn't know about time. > >> notice that I do insist on continue using V4L2_MEMORY_MMAP (instead of > >> switching to V4L2_MEMORY_USERPTR) because it is easy to export the > >> buffers as DMABUFs (~30 lines of code) and then pass these in FDs (which > >> could be associated to short lived AVBufferRefs for DRM) > > No idea what's the difference between those. > > > > If you want to support direct FD/dmabuf export, adapting to > > AVHWFramesContext now would probably be easier in total. Especially > > because of the implied API change. But I'd wait for Mark Thompson's > > comments on that before making any big changes. AFAIK he posted a > > proposal patch for a DRM AVHWFramesContext too. > > why not just add a FD to the AVBufferRef and let the user decide whether > to use it or not? Because there's too much framework-y stuff in ffmpeg that expects AVHWFramesContext. It's basically needed to do anything with it, and avoids ad-hoc approaches the old hwaccel API was full of.
On 08/29/2017 01:50 PM, wm4 wrote: >> static void free_v4l2buf_cb(void *opaque, uint8_t *unused) >> { >> V4L2Buffer* avbuf = opaque; >> V4L2m2mContext *s = container_of(avbuf->context, V4L2m2mContext, >> capture); >> >> atomic_fetch_sub_explicit(&s->refcount, 1, memory_order_acq_rel); >> >> if (s->reinit) { >> if (!atomic_load(&s->refcount)) >> sem_post(&s->refsync); >> return; >> } >> >> if (avbuf->context->streamon) { >> avbuf->context->ops.enqueue(avbuf); >> return; >> } >> >> if (!atomic_load(&s->refcount)) >> avpriv_v4l2m2m_end(s); >> } >> >> The only case where I can't get away without waiting for the AVBufferRef >> to be released is when re-initializing the frame dimensions (ie, >> resolution changes/format)_during_ streaming since I need to release >> _all_ hardware buffers and queue them again. >> >> will this be acceptable? >> I have just tested these changes and works as expected. > The implementation seems rather roundabout and complex - why not use > AVBufferRef? But apart from that, yes. > I thought about using an AVBufferRef for this but really all I need are an atomic_uint (refcount), the atomic operations and a sem_t (refsync). then it is pretty straight forward. I'll post it like it is in v7 and if you still believe an AVBufferRef is a more maintainable solution I'll change it, is not a big deal (although I'll still need the sem_t).
On 08/29/2017 01:50 PM, wm4 wrote: >>> If you want to support direct FD/dmabuf export, adapting to >>> AVHWFramesContext now would probably be easier in total. Especially >>> because of the implied API change. But I'd wait for Mark Thompson's >>> comments on that before making any big changes. AFAIK he posted a >>> proposal patch for a DRM AVHWFramesContext too. >> why not just add a FD to the AVBufferRef and let the user decide whether >> to use it or not? > Because there's too much framework-y stuff in ffmpeg that expects > AVHWFramesContext. It's basically needed to do anything with it, and > avoids ad-hoc approaches the old hwaccel API was full of. understood...so at some point I will have to use the AVHWFramesContext (to integrate dmabuf)...will add it to my list. thanks
On 08/25/2017 05:35 PM, wm4 wrote: >> +static inline void set_pts(V4L2Buffer *out, int64_t pts) >> +{ >> + if (pts == AV_NOPTS_VALUE) { >> + /* invalid timestamp: not sure how to handle this case */ >> + out->timestamp.tv_sec = 0; >> + out->timestamp.tv_usec = 0; >> + } else { >> + AVRational v4l2_timebase = { 1, 1000000 }; >> + int64_t v4l2_pts = av_rescale_q(pts, out->context->time_base, v4l2_timebase); >> + out->timestamp.tv_sec = v4l2_pts / INT64_C(1000000); >> + out->timestamp.tv_usec = v4l2_pts % INT64_C(1000000); >> + } >> +} > Why does it require a fixed timebase? A decoder shouldn't even look at > the timestamps, it should only pass them though. Also, not using DTS > will make it a nightmare to support containers like avi. > > I suspect the decoder tries to "fix" timestamps, or maybe even does > something particularly bad like reordering frames by timestamps. This > is NOT something that should be in a kernel API. > > (FFmpeg native decoders_and_ hwaccels pass through both PTS and DTS, > and don't touch their values.) > ok I will just pass through the dts/pts and return them unmodified for decoding and retest. what about for encoding? should I just read the pts from the driver, rescale it back to time_base and set both pts&&dts on the returned packet to the this same value? after this I think I am ready to post v7 with all the review comments addressed so thanks a lot for all the responses!
On Tue, 29 Aug 2017 15:11:55 +0200 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> wrote: > On 08/25/2017 05:35 PM, wm4 wrote: > >> +static inline void set_pts(V4L2Buffer *out, int64_t pts) > >> +{ > >> + if (pts == AV_NOPTS_VALUE) { > >> + /* invalid timestamp: not sure how to handle this case */ > >> + out->timestamp.tv_sec = 0; > >> + out->timestamp.tv_usec = 0; > >> + } else { > >> + AVRational v4l2_timebase = { 1, 1000000 }; > >> + int64_t v4l2_pts = av_rescale_q(pts, out->context->time_base, v4l2_timebase); > >> + out->timestamp.tv_sec = v4l2_pts / INT64_C(1000000); > >> + out->timestamp.tv_usec = v4l2_pts % INT64_C(1000000); > >> + } > >> +} > > Why does it require a fixed timebase? A decoder shouldn't even look at > > the timestamps, it should only pass them though. Also, not using DTS > > will make it a nightmare to support containers like avi. > > > > I suspect the decoder tries to "fix" timestamps, or maybe even does > > something particularly bad like reordering frames by timestamps. This > > is NOT something that should be in a kernel API. > > > > (FFmpeg native decoders_and_ hwaccels pass through both PTS and DTS, > > and don't touch their values.) > > > > ok I will just pass through the dts/pts and return them unmodified for > decoding and retest. > > what about for encoding? should I just read the pts from the driver, > rescale it back to time_base and set both pts&&dts on the returned > packet to the this same value? > > after this I think I am ready to post v7 with all the review comments > addressed so thanks a lot for all the responses! Things are different from encoders. These should synthesize correct pts and dts from the pts set on input frames.
On 08/28/2017 11:52 AM, wm4 wrote: > On Sun, 27 Aug 2017 18:26:32 +0200 > Jorge Ramirez<jorge.ramirez-ortiz@linaro.org> wrote: > >> On 08/25/2017 05:35 PM, wm4 wrote: >>> That looks generally OK. Is there any chance a hwaccel approach would >>> be possible instead? If I've learned anything about hardware decoding, >>> then that hwaccel is vastly superior to vendor-implemented full stream >>> decoders. >> could you help me understand what would that entitle and what how would >> that be beneficial to the users? >> I just dont feel I can answer that question properly... > With hwaccels, the hardware gets only the slice data (and a bunch of > API parameters with information about the bitstream etc.) instead of > the full bitstream. The advantage is that higher level bitstream > parsing, metadata retrieval, reference frame management, frame > reordering, etc. are all done by the already existing native codec > implementation. > >> v4l2 provides a generic API which is what the patchset uses to perform >> encoding/decoding on any v4l2 supported hardware (it is completely >> vendor independent) >> >> From the layer above (libavcodec) all it needs is a way to get the >> frame information and after processing to pass it back; so in principle, >> if the hwaccel API provides that, I could just move it all to use those >> calls if you think that fits better with ffmpeg. >> >> but I dont think I understand the benefit of changing from the ffmpeg >> encoding/decoding API to hwaccel API. >> >>> I don't think I like the attempt of sharing the v4l helper functions >>> between libavdevice and libavcodec, but I can't tell how much it helps. >> ok. I am of course open to suggestions on this (I didnt see any issues >> with what the patchset provides or I would have done it differently). > Just forget that libavdevice exists, and keep it correct and simple in > libavcodec. [resending to the ML]; but I do need to use some of the functions supported in libavdevice for its v4l2 frame grabber support; ignoring that code means I will have to pretty much duplicate it with a couple of additions that said.. is this what you are suggesting or am I misunderstanding you? I'll post v7 after this last bit (this dependency seemed to also be a source of concern for Paul Mahol)
diff --git a/Changelog b/Changelog index f5dc1da..9156568 100644 --- a/Changelog +++ b/Changelog @@ -34,6 +34,7 @@ version <next>: - floodfill video filter - pseudocolor video filter - raw G.726 demuxer, left- and right-justified +- V4L2 mem2mem HW accelerated codecs support version 3.3: - CrystalHD decoder moved to new decode API diff --git a/configure b/configure index e048b02..7a8c0e6 100755 --- a/configure +++ b/configure @@ -149,6 +149,7 @@ Component options: --disable-pixelutils disable pixel utils in libavutil Individual component options: + --disable-v4l2_m2m disable V4L2 mem2mem code [autodetect] --disable-everything disable all components listed below --disable-encoder=NAME disable encoder NAME --enable-encoder=NAME enable encoder NAME @@ -1432,6 +1433,7 @@ AVCODEC_COMPONENTS=" AVDEVICE_COMPONENTS=" indevs + v4l2_m2m outdevs " AVFILTER_COMPONENTS=" @@ -2271,6 +2273,7 @@ map 'eval ${v}_inline_deps=inline_asm' $ARCH_EXT_LIST_ARM loongson2_deps="mips" loongson3_deps="mips" v4l2_deps_any="linux_videodev2_h sys_videoio_h" +v4l2_m2m_select="v4l2" mipsfpu_deps="mips" mipsdsp_deps="mips" mipsdspr2_deps="mips" @@ -2743,6 +2746,8 @@ nvenc_deps="cuda" nvenc_deps_any="dlopen LoadLibrary" nvenc_encoder_deps="nvenc" +h263_v4l2m2m_decoder_deps="v4l2_m2m h263_v4l2_m2m" +h263_v4l2m2m_encoder_deps="v4l2_m2m h263_v4l2_m2m" h264_crystalhd_decoder_select="crystalhd h264_mp4toannexb_bsf h264_parser" h264_cuvid_decoder_deps="cuda cuvid" h264_cuvid_decoder_select="h264_mp4toannexb_bsf" @@ -2761,6 +2766,8 @@ h264_vda_decoder_deps="vda" h264_vda_decoder_select="h264_decoder" h264_vdpau_decoder_deps="vdpau" h264_vdpau_decoder_select="h264_decoder" +h264_v4l2m2m_decoder_deps="v4l2_m2m h264_v4l2_m2m" +h264_v4l2m2m_encoder_deps="v4l2_m2m h264_v4l2_m2m" hevc_cuvid_decoder_deps="cuda cuvid" hevc_cuvid_decoder_select="hevc_mp4toannexb_bsf" hevc_mediacodec_decoder_deps="mediacodec" @@ -2772,12 +2779,15 @@ hevc_qsv_encoder_deps="libmfx" hevc_qsv_encoder_select="hevcparse qsvenc" hevc_vaapi_encoder_deps="VAEncPictureParameterBufferHEVC" hevc_vaapi_encoder_select="vaapi_encode golomb" +hevc_v4l2m2m_decoder_deps="v4l2_m2m hevc_v4l2_m2m" +hevc_v4l2m2m_encoder_deps="v4l2_m2m hevc_v4l2_m2m" mjpeg_cuvid_decoder_deps="cuda cuvid" mjpeg_vaapi_encoder_deps="VAEncPictureParameterBufferJPEG" mjpeg_vaapi_encoder_select="vaapi_encode jpegtables" mpeg1_cuvid_decoder_deps="cuda cuvid" mpeg1_vdpau_decoder_deps="vdpau" mpeg1_vdpau_decoder_select="mpeg1video_decoder" +mpeg1_v4l2m2m_decoder_deps="v4l2_m2m mpeg1_v4l2_m2m" mpeg2_crystalhd_decoder_select="crystalhd" mpeg2_cuvid_decoder_deps="cuda cuvid" mpeg2_mmal_decoder_deps="mmal" @@ -2788,6 +2798,7 @@ mpeg2_qsv_encoder_deps="libmfx" mpeg2_qsv_encoder_select="qsvenc" mpeg2_vaapi_encoder_deps="VAEncPictureParameterBufferMPEG2" mpeg2_vaapi_encoder_select="vaapi_encode" +mpeg2_v4l2m2m_decoder_deps="v4l2_m2m mpeg2_v4l2_m2m" mpeg4_crystalhd_decoder_select="crystalhd" mpeg4_cuvid_decoder_deps="cuda cuvid" mpeg4_mediacodec_decoder_deps="mediacodec" @@ -2795,6 +2806,8 @@ mpeg4_mmal_decoder_deps="mmal" mpeg4_omx_encoder_deps="omx" mpeg4_vdpau_decoder_deps="vdpau" mpeg4_vdpau_decoder_select="mpeg4_decoder" +mpeg4_v4l2m2m_decoder_deps="v4l2_m2m mpeg4_v4l2_m2m" +mpeg4_v4l2m2m_encoder_deps="v4l2_m2m mpeg4_v4l2_m2m" mpeg_vdpau_decoder_deps="vdpau" mpeg_vdpau_decoder_select="mpeg2video_decoder" msmpeg4_crystalhd_decoder_select="crystalhd" @@ -2805,16 +2818,20 @@ vc1_cuvid_decoder_deps="cuda cuvid" vc1_mmal_decoder_deps="mmal" vc1_vdpau_decoder_deps="vdpau" vc1_vdpau_decoder_select="vc1_decoder" +vc1_v4l2m2m_decoder_deps="v4l2_m2m vc1_v4l2_m2m" vp8_cuvid_decoder_deps="cuda cuvid" vp8_mediacodec_decoder_deps="mediacodec" vp8_qsv_decoder_deps="libmfx" vp8_qsv_decoder_select="qsvdec vp8_qsv_hwaccel vp8_parser" vp8_vaapi_encoder_deps="VAEncPictureParameterBufferVP8" vp8_vaapi_encoder_select="vaapi_encode" +vp8_v4l2m2m_decoder_deps="v4l2_m2m vp8_v4l2_m2m" +vp8_v4l2m2m_encoder_deps="v4l2_m2m vp8_v4l2_m2m" vp9_cuvid_decoder_deps="cuda cuvid" vp9_mediacodec_decoder_deps="mediacodec" vp9_vaapi_encoder_deps="VAEncPictureParameterBufferVP9" vp9_vaapi_encoder_select="vaapi_encode" +vp9_v4l2m2m_decoder_deps="v4l2_m2m vp9_v4l2_m2m" wmv3_crystalhd_decoder_select="crystalhd" wmv3_vdpau_decoder_select="vc1_vdpau_decoder" @@ -3593,7 +3610,7 @@ done enable_weak audiotoolbox # Enable hwaccels by default. -enable_weak d3d11va dxva2 vaapi vda vdpau videotoolbox_hwaccel xvmc +enable_weak d3d11va dxva2 vaapi v4l2_m2m vda vdpau videotoolbox_hwaccel xvmc enable_weak xlib enable_weak cuda cuvid nvenc vda_framework videotoolbox videotoolbox_encoder @@ -6058,10 +6075,21 @@ perl -v > /dev/null 2>&1 && enable perl || disable perl pod2man --help > /dev/null 2>&1 && enable pod2man || disable pod2man rsync --help 2> /dev/null | grep -q 'contimeout' && enable rsync_contimeout || disable rsync_contimeout +# check V4L2 codecs available in the API check_header linux/fb.h check_header linux/videodev.h check_header linux/videodev2.h check_code cc linux/videodev2.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete +check_code cc linux/videodev2.h "int i = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_VIDEO_M2M | V4L2_BUF_FLAG_LAST;" || disable v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VC1_ANNEX_G;" && enable vc1_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG1;" && enable mpeg1_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG2;" && enable mpeg2_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_MPEG4;" && enable mpeg4_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_HEVC;" && enable hevc_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H263;" && enable h263_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_H264;" && enable h264_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP8;" && enable vp8_v4l2_m2m +check_code cc linux/videodev2.h "int i = V4L2_PIX_FMT_VP9;" && enable vp9_v4l2_m2m check_header sys/videoio.h check_code cc sys/videoio.h "struct v4l2_frmsizeenum vfse; vfse.discrete.width = 0;" && enable_safe struct_v4l2_frmivalenum_discrete diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 153247f..4984ee5 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -101,7 +101,6 @@ OBJS-$(CONFIG_LZF) += lzf.o OBJS-$(CONFIG_MDCT) += mdct_fixed.o mdct_float.o mdct_fixed_32.o OBJS-$(CONFIG_ME_CMP) += me_cmp.o OBJS-$(CONFIG_MEDIACODEC) += mediacodecdec_common.o mediacodec_surface.o mediacodec_wrapper.o mediacodec_sw_buffer.o -OBJS-$(CONFIG_V4L2) += v4l2_fmt.o OBJS-$(CONFIG_MPEG_ER) += mpeg_er.o OBJS-$(CONFIG_MPEGAUDIO) += mpegaudio.o OBJS-$(CONFIG_MPEGAUDIODSP) += mpegaudiodsp.o \ @@ -138,6 +137,8 @@ OBJS-$(CONFIG_VIDEODSP) += videodsp.o OBJS-$(CONFIG_VP3DSP) += vp3dsp.o OBJS-$(CONFIG_VP56DSP) += vp56dsp.o OBJS-$(CONFIG_VP8DSP) += vp8dsp.o +OBJS-$(CONFIG_V4L2) += v4l2_fmt.o +OBJS-$(CONFIG_V4L2_M2M) += v4l2_m2m.o v4l2_buffers.o OBJS-$(CONFIG_WMA_FREQS) += wma_freqs.o OBJS-$(CONFIG_WMV2DSP) += wmv2dsp.o @@ -322,6 +323,8 @@ OBJS-$(CONFIG_H263_DECODER) += h263dec.o h263.o ituh263dec.o \ intelh263dec.o h263data.o OBJS-$(CONFIG_H263_ENCODER) += mpeg4videoenc.o mpeg4video.o \ h263.o ituh263enc.o flvenc.o h263data.o +OBJS-$(CONFIG_H263_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_H263_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_H264_DECODER) += h264dec.o h264_cabac.o h264_cavlc.o \ h264_direct.o h264_loopfilter.o \ h264_mb.o h264_picture.o \ @@ -339,6 +342,8 @@ OBJS-$(CONFIG_H264_QSV_DECODER) += qsvdec_h2645.o OBJS-$(CONFIG_H264_QSV_ENCODER) += qsvenc_h264.o OBJS-$(CONFIG_H264_VAAPI_ENCODER) += vaapi_encode_h264.o vaapi_encode_h26x.o OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o +OBJS-$(CONFIG_H264_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_H264_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_HAP_DECODER) += hapdec.o hap.o OBJS-$(CONFIG_HAP_ENCODER) += hapenc.o hap.o OBJS-$(CONFIG_HEVC_DECODER) += hevcdec.o hevc_mvs.o \ @@ -352,6 +357,8 @@ OBJS-$(CONFIG_HEVC_QSV_DECODER) += qsvdec_h2645.o OBJS-$(CONFIG_HEVC_QSV_ENCODER) += qsvenc_hevc.o hevc_ps_enc.o \ hevc_data.o OBJS-$(CONFIG_HEVC_VAAPI_ENCODER) += vaapi_encode_h265.o vaapi_encode_h26x.o +OBJS-$(CONFIG_HEVC_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_HEVC_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_HNM4_VIDEO_DECODER) += hnm4video.o OBJS-$(CONFIG_HQ_HQA_DECODER) += hq_hqa.o hq_hqadata.o hq_hqadsp.o \ canopus.o @@ -421,6 +428,7 @@ OBJS-$(CONFIG_MPC8_DECODER) += mpc8.o mpc.o OBJS-$(CONFIG_MPEGVIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o OBJS-$(CONFIG_MPEG1VIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o OBJS-$(CONFIG_MPEG1VIDEO_ENCODER) += mpeg12enc.o mpeg12.o +OBJS-$(CONFIG_MPEG1_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_MPEG2_MMAL_DECODER) += mmaldec.o OBJS-$(CONFIG_MPEG2_QSV_DECODER) += qsvdec_other.o OBJS-$(CONFIG_MPEG2_QSV_ENCODER) += qsvenc_mpeg2.o @@ -428,9 +436,12 @@ OBJS-$(CONFIG_MPEG2VIDEO_DECODER) += mpeg12dec.o mpeg12.o mpeg12data.o OBJS-$(CONFIG_MPEG2VIDEO_ENCODER) += mpeg12enc.o mpeg12.o OBJS-$(CONFIG_MPEG2_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_MPEG2_VAAPI_ENCODER) += vaapi_encode_mpeg2.o +OBJS-$(CONFIG_MPEG2_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_MPEG4_DECODER) += xvididct.o OBJS-$(CONFIG_MPEG4_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_MPEG4_OMX_ENCODER) += omx.o +OBJS-$(CONFIG_MPEG4_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_MPEG4_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_MPL2_DECODER) += mpl2dec.o ass.o OBJS-$(CONFIG_MSA1_DECODER) += mss3.o OBJS-$(CONFIG_MSCC_DECODER) += mscc.o @@ -604,6 +615,7 @@ OBJS-$(CONFIG_VC1_DECODER) += vc1dec.o vc1_block.o vc1_loopfilter.o OBJS-$(CONFIG_VC1_CUVID_DECODER) += cuvid.o OBJS-$(CONFIG_VC1_MMAL_DECODER) += mmaldec.o OBJS-$(CONFIG_VC1_QSV_DECODER) += qsvdec_other.o +OBJS-$(CONFIG_VC1_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_VC2_ENCODER) += vc2enc.o vc2enc_dwt.o diractab.o OBJS-$(CONFIG_VCR1_DECODER) += vcr1.o OBJS-$(CONFIG_VMDAUDIO_DECODER) += vmdaudio.o @@ -613,6 +625,7 @@ OBJS-$(CONFIG_VORBIS_DECODER) += vorbisdec.o vorbisdsp.o vorbis.o \ vorbis_data.o OBJS-$(CONFIG_VORBIS_ENCODER) += vorbisenc.o vorbis.o \ vorbis_data.o +OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o OBJS-$(CONFIG_VP3_DECODER) += vp3.o OBJS-$(CONFIG_VP5_DECODER) += vp5.o vp56.o vp56data.o vp56rac.o OBJS-$(CONFIG_VP6_DECODER) += vp6.o vp56.o vp56data.o \ @@ -623,6 +636,8 @@ OBJS-$(CONFIG_VP8_CUVID_DECODER) += cuvid.o OBJS-$(CONFIG_VP8_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_VP8_QSV_DECODER) += qsvdec_other.o OBJS-$(CONFIG_VP8_VAAPI_ENCODER) += vaapi_encode_vp8.o +OBJS-$(CONFIG_VP8_V4L2M2M_DECODER) += v4l2_m2m_dec.o +OBJS-$(CONFIG_VP8_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_VP9_DECODER) += vp9.o vp9data.o vp9dsp.o vp9lpf.o vp9recon.o \ vp9block.o vp9prob.o vp9mvs.o vp56rac.o \ vp9dsp_8bpp.o vp9dsp_10bpp.o vp9dsp_12bpp.o @@ -630,6 +645,7 @@ OBJS-$(CONFIG_VP9_CUVID_DECODER) += cuvid.o OBJS-$(CONFIG_VP9_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_VP9_VAAPI_ENCODER) += vaapi_encode_vp9.o OBJS-$(CONFIG_VPLAYER_DECODER) += textdec.o ass.o +OBJS-$(CONFIG_VP9_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_VQA_DECODER) += vqavideo.o OBJS-$(CONFIG_WAVPACK_DECODER) += wavpack.o OBJS-$(CONFIG_WAVPACK_ENCODER) += wavpackenc.o diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c index 1e5942d..4bd78a8 100644 --- a/libavcodec/allcodecs.c +++ b/libavcodec/allcodecs.c @@ -207,8 +207,10 @@ static void register_all(void) REGISTER_ENCDEC (H263, h263); REGISTER_DECODER(H263I, h263i); REGISTER_ENCDEC (H263P, h263p); + REGISTER_ENCDEC (H263_V4L2M2M, h263_v4l2m2m); REGISTER_DECODER(H264, h264); REGISTER_DECODER(H264_CRYSTALHD, h264_crystalhd); + REGISTER_ENCDEC (H264_V4L2M2M, h264_v4l2m2m); REGISTER_DECODER(H264_MEDIACODEC, h264_mediacodec); REGISTER_DECODER(H264_MMAL, h264_mmal); REGISTER_DECODER(H264_QSV, h264_qsv); @@ -219,6 +221,7 @@ static void register_all(void) REGISTER_ENCDEC (HAP, hap); REGISTER_DECODER(HEVC, hevc); REGISTER_DECODER(HEVC_QSV, hevc_qsv); + REGISTER_ENCDEC(HEVC_V4L2M2M, hevc_v4l2m2m); REGISTER_DECODER(HNM4_VIDEO, hnm4_video); REGISTER_DECODER(HQ_HQA, hq_hqa); REGISTER_DECODER(HQX, hqx); @@ -253,6 +256,7 @@ static void register_all(void) REGISTER_ENCDEC (MPEG2VIDEO, mpeg2video); REGISTER_ENCDEC (MPEG4, mpeg4); REGISTER_DECODER(MPEG4_CRYSTALHD, mpeg4_crystalhd); + REGISTER_ENCDEC (MPEG4_V4L2M2M, mpeg4_v4l2m2m); REGISTER_DECODER(MPEG4_MMAL, mpeg4_mmal); #if FF_API_VDPAU REGISTER_DECODER(MPEG4_VDPAU, mpeg4_vdpau); @@ -262,8 +266,10 @@ static void register_all(void) REGISTER_DECODER(MPEG_VDPAU, mpeg_vdpau); REGISTER_DECODER(MPEG1_VDPAU, mpeg1_vdpau); #endif + REGISTER_DECODER(MPEG1_V4L2M2M, mpeg1_v4l2m2m); REGISTER_DECODER(MPEG2_MMAL, mpeg2_mmal); REGISTER_DECODER(MPEG2_CRYSTALHD, mpeg2_crystalhd); + REGISTER_DECODER(MPEG2_V4L2M2M, mpeg2_v4l2m2m); REGISTER_DECODER(MPEG2_QSV, mpeg2_qsv); REGISTER_DECODER(MPEG2_MEDIACODEC, mpeg2_mediacodec); REGISTER_DECODER(MSA1, msa1); @@ -361,6 +367,7 @@ static void register_all(void) REGISTER_DECODER(VC1IMAGE, vc1image); REGISTER_DECODER(VC1_MMAL, vc1_mmal); REGISTER_DECODER(VC1_QSV, vc1_qsv); + REGISTER_DECODER(VC1_V4L2M2M, vc1_v4l2m2m); REGISTER_ENCODER(VC2, vc2); REGISTER_DECODER(VCR1, vcr1); REGISTER_DECODER(VMDVIDEO, vmdvideo); @@ -372,7 +379,9 @@ static void register_all(void) REGISTER_DECODER(VP6F, vp6f); REGISTER_DECODER(VP7, vp7); REGISTER_DECODER(VP8, vp8); + REGISTER_ENCDEC (VP8_V4L2M2M, vp8_v4l2m2m); REGISTER_DECODER(VP9, vp9); + REGISTER_DECODER(VP9_V4L2M2M, vp9_v4l2m2m); REGISTER_DECODER(VQA, vqa); REGISTER_DECODER(BITPACKED, bitpacked); REGISTER_DECODER(WEBP, webp); diff --git a/libavcodec/v4l2_buffers.c b/libavcodec/v4l2_buffers.c new file mode 100644 index 0000000..1d32c7e --- /dev/null +++ b/libavcodec/v4l2_buffers.c @@ -0,0 +1,741 @@ +/* + * V4L2 buffer{,context} helper functions. + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <poll.h> +#include <fcntl.h> +#include <unistd.h> +#include "avcodec.h" +#include "internal.h" +#include "v4l2_buffers.h" +#include "v4l2_m2m.h" +#include "v4l2_fmt.h" + +#define WIDTH(__ctx, __fmt) \ + (V4L2_TYPE_IS_MULTIPLANAR((__ctx)->type) ? __fmt.fmt.pix_mp.width : __fmt.fmt.pix.width) + +#define HEIGHT(__ctx, __fmt) \ + (V4L2_TYPE_IS_MULTIPLANAR((__ctx)->type) ? __fmt.fmt.pix_mp.height : __fmt.fmt.pix.height) + +#define BUFFER_TYPE_SUPPORTED(ctx) \ + ((ctx->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) || \ + (ctx->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) || \ + (ctx->type == V4L2_BUF_TYPE_VIDEO_CAPTURE) || \ + (ctx->type == V4L2_BUF_TYPE_VIDEO_OUTPUT)) + +enum V4L2Buffer_status { + V4L2BUF_AVAILABLE, + V4L2BUF_IN_DRIVER, + V4L2BUF_RET_USER, +}; + +/* buffer transform */ +typedef int (*pkt_to_buf_f)(const AVPacket *, V4L2Buffer *); +typedef int (*frm_to_buf_f)(const AVFrame *, V4L2Buffer *); +typedef int (*buf_to_pkt_f)(AVPacket *, V4L2Buffer *); +typedef int (*buf_to_frm_f)(AVFrame *, V4L2Buffer *); + +typedef int (*buf_to_bufref_f)(V4L2Buffer *in, int plane, AVBufferRef **buf); +typedef int (*bufref_to_buf_f)(V4L2Buffer *out, int plane, const uint8_t* data, int size, AVBufferRef* bref); + +struct V4L2Buffer_ops { + pkt_to_buf_f pkt_to_buf; + frm_to_buf_f frm_to_buf; + buf_to_pkt_f buf_to_pkt; + buf_to_frm_f buf_to_frm; + + bufref_to_buf_f bufref_to_buf; + buf_to_bufref_f buf_to_bufref; +}; + +struct V4L2Buffer { + /* each buffer needs to have a reference to its context */ + struct V4L2Context *context; + + struct V4L2Plane_info { + void * mm_addr; + size_t lengths; + } plane_info[VIDEO_MAX_PLANES]; + + /* some common buffer operations */ + struct V4L2Buffer_ops ops; + + /* memcpy to the v4l2_buffer planes array when needed */ + struct v4l2_plane planes[VIDEO_MAX_PLANES]; + struct v4l2_buffer buf; + + int bytesperline[4]; + int num_planes; + + int flags; + struct timeval timestamp; + enum V4L2Buffer_status status; +}; + +static inline void set_pts(V4L2Buffer *out, int64_t pts) +{ + if (pts == AV_NOPTS_VALUE) { + /* invalid timestamp: not sure how to handle this case */ + out->timestamp.tv_sec = 0; + out->timestamp.tv_usec = 0; + } else { + AVRational v4l2_timebase = { 1, 1000000 }; + int64_t v4l2_pts = av_rescale_q(pts, out->context->time_base, v4l2_timebase); + out->timestamp.tv_sec = v4l2_pts / INT64_C(1000000); + out->timestamp.tv_usec = v4l2_pts % INT64_C(1000000); + } +} + +static inline uint64_t get_pts(V4L2Buffer *avbuf) +{ + if (avbuf->buf.timestamp.tv_sec || avbuf->buf.timestamp.tv_usec) { + int64_t pts = (avbuf->buf.timestamp.tv_sec) * INT64_C(1000000) + avbuf->buf.timestamp.tv_usec; + AVRational v4l2_timebase = { 1, 1000000 }; + pts = av_rescale_q(pts, v4l2_timebase, avbuf->context->time_base); + return pts; + } + + return AV_NOPTS_VALUE; +} + +static void free_v4l2buf_cb(void *opaque, uint8_t *unused) +{ + V4L2Buffer* avbuf = opaque; + + if (V4L2BUF_IN_DRIVER == avbuf->status) + return; + + if (V4L2_TYPE_IS_OUTPUT(avbuf->context->type)) + avbuf->status = V4L2BUF_AVAILABLE; + else + avbuf->context->ops.enqueue(avbuf); +} + +/*** + Buffer Operations + */ +static int buffer_ops_bufref_to_v4l2buf(V4L2Buffer *out, int plane, const uint8_t* data, int size, AVBufferRef* bref) +{ + if (plane >= out->num_planes) + return AVERROR(EINVAL); + + memcpy(out->plane_info[plane].mm_addr, data, FFMIN(size, out->plane_info[plane].lengths)); + + out->planes[plane].bytesused = FFMIN(size, out->plane_info[plane].lengths); + out->planes[plane].length = out->plane_info[plane].lengths; + + return 0; +} + +static inline int buffer_ops_v4l2buf_to_bufref(V4L2Buffer *in, int plane, AVBufferRef **buf) +{ + if (plane >= in->num_planes) + return AVERROR(EINVAL); + + /* even though most encoders return 0 in data_offset encoding vp8 does require this value*/ + *buf = av_buffer_create((char *)in->plane_info[plane].mm_addr + in->planes[plane].data_offset, + in->plane_info[plane].lengths, free_v4l2buf_cb, in, 0); + if (!*buf) + return AVERROR(ENOMEM); + + in->status = V4L2BUF_RET_USER; + + return 0; +} + +static int buffer_ops_v4l2buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) +{ + int i, ret; + + av_frame_unref(frame); + + /* 1. get references to the actual data */ + for (i = 0; i < avbuf->num_planes; i++) { + ret = avbuf->ops.buf_to_bufref(avbuf, i, &frame->buf[i]); + if (ret) + return ret; + + frame->linesize[i] = avbuf->bytesperline[i]; + frame->data[i] = frame->buf[i]->data; + } + + /* 1.1 fixup special cases */ + switch (avbuf->context->av_pix_fmt) { + case AV_PIX_FMT_NV12: + if (avbuf->num_planes > 1) + break; + frame->linesize[1] = avbuf->bytesperline[0]; + frame->data[1] = frame->buf[0]->data + avbuf->bytesperline[0] * avbuf->context->format.fmt.pix_mp.height; + break; + default: + break; + } + + /* 2. get frame information */ + frame->key_frame = !!(avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME); + frame->format = avbuf->context->av_pix_fmt; + + /* these values are updated also during re-init in process_video_event */ + frame->height = avbuf->context->height; + frame->width = avbuf->context->width; + frame->pts = get_pts(avbuf); + + /* 3. report errors upstream */ + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s: driver decode error\n", avbuf->context->name); + frame->decode_error_flags |= FF_DECODE_ERROR_INVALID_BITSTREAM; + } + + return 0; +} + +static int buffer_ops_avpkt_to_v4l2buf(const AVPacket *pkt, V4L2Buffer *out) { + int ret; + + ret = out->ops.bufref_to_buf(out, 0, pkt->data, pkt->size, pkt->buf); + if (ret) + return ret; + + set_pts(out, pkt->pts); + + if (pkt->flags & AV_PKT_FLAG_KEY) + out->flags = V4L2_BUF_FLAG_KEYFRAME; + + return 0; +} + +static int buffer_ops_avframe_to_v4l2buf(const AVFrame *frame, V4L2Buffer* out) +{ + int i, ret; + + for(i = 0; i < out->num_planes; i++) { + ret = out->ops.bufref_to_buf(out, i, frame->buf[i]->data, frame->buf[i]->size, frame->buf[i]); + if (ret) + return ret; + } + + set_pts(out, frame->pts); + + return 0; +} + +static int buffer_ops_v4l2buf_to_avpkt(AVPacket *pkt, V4L2Buffer *avbuf) +{ + int ret; + + av_packet_unref(pkt); + ret = avbuf->ops.buf_to_bufref(avbuf, 0, &pkt->buf); + if (ret) + return ret; + + pkt->size = V4L2_TYPE_IS_MULTIPLANAR(avbuf->context->type) ? avbuf->buf.m.planes[0].bytesused : avbuf->buf.bytesused; + pkt->data = pkt->buf->data; + + if (avbuf->buf.flags & V4L2_BUF_FLAG_KEYFRAME) + pkt->flags |= AV_PKT_FLAG_KEY; + + if (avbuf->buf.flags & V4L2_BUF_FLAG_ERROR) { + av_log(avbuf->context->log_ctx, AV_LOG_ERROR, "%s driver encode error\n", avbuf->context->name); + pkt->flags |= AV_PKT_FLAG_CORRUPT; + } + + pkt->pts = get_pts(avbuf); + + return 0; +} + +/*** + Context Operations + */ +static int context_ops_stop_decode(V4L2Context *ctx) +{ + struct v4l2_decoder_cmd cmd = { + .cmd = V4L2_DEC_CMD_STOP, + }; + int ret; + + ret = ioctl(ctx->fd, VIDIOC_DECODER_CMD, &cmd); + if (ret) { + /* DECODER_CMD is optional */ + if (errno == ENOTTY) + return ioctl(ctx->fd, VIDIOC_STREAMOFF); + } + + return ret; +} + +static int context_ops_stop_encode(V4L2Context *ctx) +{ + + struct v4l2_encoder_cmd cmd = { + .cmd = V4L2_ENC_CMD_STOP, + }; + int ret; + + ret = ioctl(ctx->fd, VIDIOC_ENCODER_CMD, &cmd); + if (ret) { + /* ENCODER_CMD is optional */ + if (errno == ENOTTY) + return ioctl(ctx->fd, VIDIOC_STREAMOFF); + } + + return ret; +} + +static int context_ops_enqueue_v4l2buf(V4L2Buffer* avbuf) +{ + int ret; + + avbuf->buf.flags = avbuf->context->default_flags | avbuf->flags; + avbuf->buf.timestamp = avbuf->timestamp; + + ret = ioctl(avbuf->context->fd, VIDIOC_QBUF, &avbuf->buf); + if (ret < 0) + return AVERROR(errno); + + avbuf->status = V4L2BUF_IN_DRIVER; + avbuf->context->num_queued++; + + return 0; +} + +static int process_video_event(V4L2Context *ctx) +{ + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, capture); + struct v4l2_format cap_fmt = s->capture.format; + struct v4l2_format out_fmt = s->output.format; + struct v4l2_event evt; + int ret; + + ret = ioctl(ctx->fd, VIDIOC_DQEVENT, &evt); + if (ret < 0) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s VIDIOC_DQEVENT\n", ctx->name); + return 0; + } + + ret = ioctl(s->fd, VIDIOC_G_FMT, &cap_fmt); + if (ret) { + av_log(s->capture.log_ctx, AV_LOG_ERROR, "%s VIDIOC_G_FMT\n", s->capture.name); + return 0; + } + + ret = ioctl(s->fd, VIDIOC_G_FMT, &out_fmt); + if (ret) { + av_log(s->output.log_ctx, AV_LOG_ERROR, "%s VIDIOC_G_FMT\n", s->output.name); + return 0; + } + + if (evt.type != V4L2_EVENT_SOURCE_CHANGE) + return 0; + + if (HEIGHT(&s->output, s->output.format) != HEIGHT(&s->output, out_fmt) || + WIDTH(&s->output, s->output.format) != WIDTH(&s->output, out_fmt)) { + + av_log(s->output.log_ctx, AV_LOG_DEBUG, "%s changed (%dx%d) -> (%dx%d)\n", + s->output.name, + WIDTH(&s->output, s->output.format), HEIGHT(&s->output, s->output.format), + WIDTH(&s->output, out_fmt), HEIGHT(&s->output, out_fmt)); + + /* 0. update the output context */ + s->output.height = HEIGHT(ctx, out_fmt); + s->output.width = WIDTH(ctx, out_fmt); + + /* 1. store the new dimensions in the capture context so the resulting frame + can be cropped */ + s->capture.height = HEIGHT(ctx, out_fmt); + s->capture.width = WIDTH(ctx, out_fmt); + } + + if (HEIGHT(&s->capture, s->capture.format) != HEIGHT(&s->capture, cap_fmt) || + WIDTH(&s->capture, s->capture.format) != WIDTH(&s->capture, cap_fmt)) { + + av_log(s->capture.log_ctx, AV_LOG_DEBUG, "%s changed (%dx%d) -> (%dx%d)\n", + s->capture.name, + WIDTH(&s->capture, s->capture.format), HEIGHT(&s->capture, s->capture.format), + WIDTH(&s->capture, cap_fmt), HEIGHT(&s->capture, cap_fmt)); + + /* streamoff capture and unmap and remap new buffers */ + ret = avpriv_v4l2m2m_reinit(ctx); + if (ret) + av_log(ctx->log_ctx, AV_LOG_ERROR, "avpriv_v4l2m2m_reinit\n"); + + /* let the caller function know that reinit was executed */ + return 1; + } + + return 0; +} + +static V4L2Buffer* context_ops_dequeue_v4l2buf(V4L2Context *ctx, unsigned int timeout) +{ + struct v4l2_plane planes[VIDEO_MAX_PLANES]; + struct v4l2_buffer buf = { 0 }; + V4L2Buffer* avbuf = NULL; + struct pollfd pfd = { + .events = POLLIN | POLLRDNORM | POLLPRI, /* default capture context */ + .fd = ctx->fd, + }; + int ret; + + if (ctx->num_queued < ctx->min_queued_buffers) + return NULL; + + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) + pfd.events = POLLOUT | POLLWRNORM; + + for (;;) { + ret = poll(&pfd, 1, timeout); + if (ret > 0) + break; + if (errno == EINTR) + continue; + return NULL; + } + + /* 0. handle errors */ + if (pfd.revents & POLLERR) { + av_log(ctx->log_ctx, AV_LOG_WARNING, "%s POLLERR\n", ctx->name); + return NULL; + } + + /* 1. dequeue the buffer */ + if (pfd.revents & (POLLIN | POLLRDNORM) || pfd.revents & (POLLOUT | POLLWRNORM)) { + memset(&buf, 0, sizeof(buf)); + buf.memory = V4L2_MEMORY_MMAP; + buf.type = ctx->type; + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + memset(planes, 0, sizeof(planes)); + buf.length = VIDEO_MAX_PLANES; + buf.m.planes = planes; + } + + ret = ioctl(ctx->fd, VIDIOC_DQBUF, &buf); + if (ret) { + if (errno != EAGAIN) { + ctx->broken = errno; + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s VIDIOC_DQBUF, errno (%d)\n", ctx->name, errno); + } + } else { + + avbuf = &ctx->buffers[buf.index]; + avbuf->status = V4L2BUF_AVAILABLE; + avbuf->context->num_queued--; + avbuf->buf = buf; + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + memcpy(avbuf->planes, planes, sizeof(planes)); + avbuf->buf.m.planes = avbuf->planes; + } + } + } + + /* 2. handle resolution changes */ + if (pfd.revents & POLLPRI) { + ret = process_video_event(ctx); + if (ret) + return NULL; + } + + return avbuf; +} + +static V4L2Buffer* context_ops_getfree_v4l2buf(V4L2Context *ctx) +{ + unsigned int timeout = 0; + int i; + + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) { + do { + } while (ctx->ops.dequeue(ctx, timeout)); + } + + for (i = 0; i < ctx->num_buffers; i++) { + if (ctx->buffers[i].status == V4L2BUF_AVAILABLE) + return &ctx->buffers[i]; + } + + return NULL; +} + +static int context_ops_release_v4l2_buffers(V4L2Context* ctx) +{ + struct v4l2_requestbuffers req = { + .memory = V4L2_MEMORY_MMAP, + .type = ctx->type, + .count = 0, /* count = 0 unmaps buffers from the driver */ + }; + int i, j; + + for (i = 0; i < ctx->num_buffers; i++) { + V4L2Buffer *buffer = &ctx->buffers[i]; + + for (j = 0; j < buffer->num_planes; j++) { + struct V4L2Plane_info *p = &buffer->plane_info[j]; + if (p->mm_addr && p->lengths) + munmap(p->mm_addr, p->lengths); + } + } + + return ioctl(ctx->fd, VIDIOC_REQBUFS, &req); +} + +static int context_ops_initialize_v4l2buf(V4L2Context *ctx, V4L2Buffer* avbuf, int index) +{ + int ret, i; + + /* keep a reference to the context */ + avbuf->context = ctx; + + /* initalize the buffer operations */ + avbuf->ops.buf_to_frm = buffer_ops_v4l2buf_to_avframe; + avbuf->ops.frm_to_buf = buffer_ops_avframe_to_v4l2buf; + avbuf->ops.buf_to_pkt = buffer_ops_v4l2buf_to_avpkt; + avbuf->ops.pkt_to_buf = buffer_ops_avpkt_to_v4l2buf; + + avbuf->ops.bufref_to_buf = buffer_ops_bufref_to_v4l2buf; + avbuf->ops.buf_to_bufref = buffer_ops_v4l2buf_to_bufref; + + avbuf->buf.memory = V4L2_MEMORY_MMAP; + avbuf->buf.type = ctx->type; + avbuf->buf.index = index; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->buf.length = VIDEO_MAX_PLANES; + avbuf->buf.m.planes = avbuf->planes; + } + + ret = ioctl(ctx->fd, VIDIOC_QUERYBUF, &avbuf->buf); + if (ret < 0) + return AVERROR(errno); + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->num_planes = 0; + for (;;) { + /* in MP, the V4L2 API states that buf.length means num_planes */ + if (avbuf->num_planes >= avbuf->buf.length) + break; + if (avbuf->buf.m.planes[avbuf->num_planes].length) + avbuf->num_planes++; + } + } else + avbuf->num_planes = 1; + + for (i = 0; i < avbuf->num_planes; i++) { + + avbuf->bytesperline[i] = V4L2_TYPE_IS_MULTIPLANAR(ctx->type) ? + ctx->format.fmt.pix_mp.plane_fmt[i].bytesperline : + ctx->format.fmt.pix.bytesperline; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->plane_info[i].lengths = avbuf->buf.m.planes[i].length; + avbuf->plane_info[i].mm_addr = mmap(NULL, avbuf->buf.m.planes[i].length, + PROT_READ | PROT_WRITE, MAP_SHARED, + ctx->fd, avbuf->buf.m.planes[i].m.mem_offset); + } else { + avbuf->plane_info[i].lengths = avbuf->buf.length; + avbuf->plane_info[i].mm_addr = mmap(NULL, avbuf->buf.length, + PROT_READ | PROT_WRITE, MAP_SHARED, + ctx->fd, avbuf->buf.m.offset); + } + + if (avbuf->plane_info[i].mm_addr == MAP_FAILED) + return AVERROR(ENOMEM); + } + + avbuf->status = V4L2BUF_AVAILABLE; + + if (V4L2_TYPE_IS_OUTPUT(ctx->type)) + return 0; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + avbuf->buf.m.planes = avbuf->planes; + avbuf->buf.length = avbuf->num_planes; + + } else { + avbuf->buf.bytesused = avbuf->planes[index].bytesused; + avbuf->buf.length = avbuf->planes[index].length; + } + + return ctx->ops.enqueue(avbuf); +} + +int avpriv_v4l2_enqueue_frame(V4L2Context* ctx, const AVFrame* frame) +{ + V4L2Buffer* avbuf; + int ret; + + if (!frame) + return ctx->ops.stop_encode(ctx); + + avbuf = ctx->ops.get_buffer(ctx); + if (!avbuf) + return AVERROR(ENOMEM); + + ret = avbuf->ops.frm_to_buf(frame, avbuf); + if (ret) + return ret; + + ret = ctx->ops.enqueue(avbuf); + if (ret) + return ret; + + return 0; +} + +int avpriv_v4l2_enqueue_packet(V4L2Context* ctx, const AVPacket* pkt) +{ + V4L2Buffer* avbuf; + int ret; + + if (pkt->size == 0) + return ctx->ops.stop_decode(ctx); + + avbuf = ctx->ops.get_buffer(ctx); + if (!avbuf) + return AVERROR(ENOMEM); + + ret = avbuf->ops.pkt_to_buf(pkt, avbuf); + if (ret) + return ret; + + ret = ctx->ops.enqueue(avbuf); + if (ret) + return ret; + + return 0; +} + +int avpriv_v4l2_dequeue_frame(V4L2Context* ctx, AVFrame* frame, unsigned int timeout) +{ + + V4L2Buffer* avbuf = NULL; + + avbuf = ctx->ops.dequeue(ctx, timeout); + if (!avbuf) { + if (ctx->broken) + return AVERROR_EOF; + + return AVERROR(EAGAIN); + } + + return avbuf->ops.buf_to_frm(frame, avbuf); +} + +int avpriv_v4l2_dequeue_packet(V4L2Context* ctx, AVPacket* pkt, unsigned int timeout) +{ + V4L2Buffer* avbuf = NULL; + + avbuf = ctx->ops.dequeue(ctx, timeout); + if (!avbuf) { + if (ctx->broken) + return AVERROR_EOF; + + return AVERROR(EAGAIN); + } + + return avbuf->ops.buf_to_pkt(pkt, avbuf); +} + +int avpriv_v4l2_context_set_status(V4L2Context* ctx, int cmd) +{ + int type = ctx->type; + int ret; + + ret = ioctl(ctx->fd, cmd, &type); + if (ret < 0) + return AVERROR(errno); + + ctx->streamon = (cmd == VIDIOC_STREAMON); + + return 0; +} + +void avpriv_v4l2_context_release(V4L2Context* ctx) +{ + int ret; + + if (!ctx->buffers) + return; + + ret = ctx->ops.release_buffers(ctx); + if (ret) + av_log(ctx->log_ctx, AV_LOG_WARNING, "V4L2 failed to unmap the %s buffers\n", ctx->name); + else + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s buffers unmapped\n", ctx->name); + + av_free(ctx->buffers); + ctx->buffers = NULL; + ctx->num_queued = 0; +} + +int avpriv_v4l2_context_init(V4L2Context* ctx, int lazy_init) +{ + struct v4l2_requestbuffers req; + int ret, i; + + if (!BUFFER_TYPE_SUPPORTED(ctx)) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "type %i not supported\n", ctx->type); + return AVERROR_PATCHWELCOME; + } + + ctx->ops.release_buffers = context_ops_release_v4l2_buffers; + ctx->ops.init_buffer = context_ops_initialize_v4l2buf; + ctx->ops.get_buffer = context_ops_getfree_v4l2buf; + ctx->ops.dequeue = context_ops_dequeue_v4l2buf; + ctx->ops.enqueue = context_ops_enqueue_v4l2buf; + ctx->ops.stop_decode = context_ops_stop_decode; + ctx->ops.stop_encode = context_ops_stop_encode; + + if (lazy_init) + return 0; + + memset(&req, 0, sizeof(req)); + req.count = ctx->num_buffers + ctx->min_queued_buffers; + req.memory = V4L2_MEMORY_MMAP; + req.type = ctx->type; + ret = ioctl(ctx->fd, VIDIOC_REQBUFS, &req); + if (ret < 0) + return AVERROR(errno); + + ctx->num_buffers = req.count; + ctx->num_queued = 0; + ctx->buffers = av_mallocz(ctx->num_buffers * sizeof(V4L2Buffer)); + if (!ctx->buffers) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s buffer initialization ENOMEM\n", ctx->name); + return AVERROR(ENOMEM); + } + + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s queuing %d buffers\n", ctx->name, req.count); + for (i = 0; i < req.count; i++) { + ret = ctx->ops.init_buffer(ctx, &ctx->buffers[i], i); + if (ret < 0) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s buffer initialization (%s)\n", ctx->name, av_err2str(ret)); + av_free(ctx->buffers); + return ret; + } + } + + return 0; +} + + diff --git a/libavcodec/v4l2_buffers.h b/libavcodec/v4l2_buffers.h new file mode 100644 index 0000000..affa81f --- /dev/null +++ b/libavcodec/v4l2_buffers.h @@ -0,0 +1,259 @@ +/* + * V4L2 buffer{,context} helper functions. + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_V4L2_BUFFERS_H +#define AVCODEC_V4L2_BUFFERS_H + + +#include "v4l2_fmt.h" +#include "avcodec.h" +#include "libavutil/pixfmt.h" +#include "libavutil/frame.h" +#include "libavutil/buffer.h" + +struct V4L2Buffer; +typedef struct V4L2Buffer V4L2Buffer; + +struct V4L2Context; +typedef struct V4L2Context V4L2Context; + +/** + * V4L2Context_operations: + */ +typedef V4L2Buffer* (*v4l2_dequeue_buffer_f)(V4L2Context *ctx, unsigned int timeout); +typedef V4L2Buffer* (*v4l2_getfree_buffer_f)(V4L2Context *ctx); + +typedef int (*v4l2_initialize_buffer_f)(V4L2Context *ctx, V4L2Buffer* avbuf, int index); +typedef int (*v4l2_enqueue_buffer_f)(V4L2Buffer *buf); +typedef int (*v4l2_release_buffers_f)(V4L2Context *ctx); +typedef int (*v4l2_stop_decode_f)(V4L2Context *ctx); +typedef int (*v4l2_stop_encode_f)(V4L2Context *ctx); + +typedef struct V4L2Context_ops { + v4l2_stop_decode_f stop_decode; + v4l2_stop_encode_f stop_encode; + + /* all buffers operations */ + v4l2_release_buffers_f release_buffers; + + /* single buffer operations */ + v4l2_initialize_buffer_f init_buffer; + v4l2_getfree_buffer_f get_buffer; + v4l2_dequeue_buffer_f dequeue; + v4l2_enqueue_buffer_f enqueue; + +} V4L2Context_ops; + +typedef struct V4L2Context { + /** + * Buffer context operations + * queue a V4L2Buffer into the context + * dequeue a V4L2Buffer into the context + * get a free V4L2Buffer from the context + * release all V4L2Buffers allocated to the context + */ + V4L2Context_ops ops; + + /** + * Log context (for av_log()). Can be NULL. + */ + void *log_ctx; + + /** + * Lazy Initialization: set to one if the context can not initialize its + * buffers until it first queries the driver for formats and sizes. + */ + int lazy_init; + + /** + * context name: must be set before calling avpriv_v4l2_context_init(). + */ + const char* name; + + /** + * File descriptor obtained from opening the associated device. + * Must be set before calling avpriv_v4l2_context_init(). + * Readonly after init. + */ + int fd; + + /** + * Type of this buffer context. + * See V4L2_BUF_TYPE_VIDEO_* in videodev2.h + * Must be set before calling avpriv_v4l2_context_init(). + * Readonly after init. + */ + enum v4l2_buf_type type; + + /** + * AVPixelFormat corresponding to this buffer context. + * AV_PIX_FMT_NONE means this is an encoded stream. + */ + enum AVPixelFormat av_pix_fmt; + + /** + * AVCodecID corresponding to this buffer context. + * AV_CODEC_ID_RAWVIDEO means this is a raw stream and av_pix_fmt must be set to a valid value. + */ + enum AVCodecID av_codec_id; + + /** + * Format returned by the driver after initializing the buffer context. + * Must be set before calling avpriv_v4l2_context_init(). + * avpriv_v4l2m2m_format() can set it. + * Readonly after init. + */ + struct v4l2_format format; + + /** + * Width and height of the frames it produces (in case of a capture context, e.g. when decoding) + * or accepts (in case of an output context, e.g. when encoding). + * + * For output context, this must must be set before calling avpriv_v4l2_context_init(). + * For capture context during decoding, it will be set after having received the + * information from the driver. at which point we can initialize the buffers. + */ + int width, height; + + /** + * Default flags to set on buffers to enqueue. + * See V4L2_BUF_FLAG_*. + */ + int default_flags; + + /** + * Time base + */ + AVRational time_base; + + /** + * Whether the stream has been started (VIDIOC_STREAMON has been sent). + */ + int streamon; + + /** + * Number of queued buffers. + */ + int num_queued; + + /** + * Minimum number of buffers that must be kept queued in this queue. + * + * E.g. for decoders, the drivers might have such requirements to produce proper output. + */ + int min_queued_buffers; + + /** + * The actual number of buffers. + * + * Before calling avpriv_v4l2_context_init() this is the number of buffers we would like to have available. + * avpriv_v4l2_context_init() asks for (min_buffers + num_buffers) and sets this value to the actual number + * of buffers the driver gave us. + * Readonly after init. + */ + int num_buffers; + + /** + * Indexed array of V4L2Buffers + */ + V4L2Buffer *buffers; + + /** + * Unrecoverable error notified by the V4L2 kernel driver + */ + int broken; + +} V4L2Context; + +/** + * Initializes a V4L2Context. + * + * @param[in] ctx A pointer to a V4L2Context. See V4L2Context description for required variables. + * @return 0 in case of success, a negative value representing the error otherwise. + */ +int avpriv_v4l2_context_init(V4L2Context* ctx, int lazy_init); + +/** + * Releases a V4L2Context. + * + * @param[in] ctx A pointer to a V4L2Context. + * The caller is reponsible for freeing it. + * It must not be used after calling this function. + */ +void avpriv_v4l2_context_release(V4L2Context* ctx); + +/** + * Sets the status of a V4L2Context. + * + * @param[in] ctx A pointer to a V4L2Context. + * @param[in] cmd The status to set (VIDIOC_STREAMON or VIDIOC_STREAMOFF). + * Warning: If VIDIOC_STREAMOFF is sent to a buffer context that still has some frames buffered, + * those frames will be dropped. + * @return 0 in case of success, a negative value representing the error otherwise. + */ +int avpriv_v4l2_context_set_status(V4L2Context* ctx, int cmd); + +/** + * Dequeues a buffer from a V4L2Context to an AVPacket. + * + * The pkt must be non NULL. + * @param[in] ctx The V4L2Context to dequeue from. + * @param[inout] pkt The AVPacket to dequeue to. + * @param[ino] timeout The number of milliseconds to wait for the dequeue. + * @return 0 in case of success, AVERROR(EAGAIN) if no buffer was ready, another negative error in case of error. + */ +int avpriv_v4l2_dequeue_packet(V4L2Context* ctx, AVPacket* pkt, unsigned int timeout); + +/** + * Dequeues a buffer from a V4L2Context to an AVFrame. + * + * The frame must be non NULL. + * @param[in] ctx The V4L2Context to dequeue from. + * @param[inout] f The AVFrame to dequeue to. + * @param[ino] timeout The number of milliseconds to wait for the dequeue. + * @return 0 in case of success, AVERROR(EAGAIN) if no buffer was ready, another negative error in case of error. + */ +int avpriv_v4l2_dequeue_frame(V4L2Context* ctx, AVFrame* f, unsigned int timeout); + +/** + * Enqueues a buffer to a V4L2Context from an AVPacket + * The packet must be non NULL. + * When the size of the pkt is null, the buffer is not queued but a V4L2_DEC_CMD_STOP command is sent instead to the driver. + * + * @param[in] ctx The V4L2Context to enqueue to. + * @param[in] pkt A pointer to an AVPacket. + * @return 0 in case of success, a negative error otherwise. + */ +int avpriv_v4l2_enqueue_packet(V4L2Context* ctx, const AVPacket* pkt); + +/** + * Enqueues a buffer to a V4L2Context from an AVFrame + * The frame must be non NULL. + * + * @param[in] ctx The V4L2Context to enqueue to. + * @param[in] f A pointer to an AVFrame to enqueue. + * @return 0 in case of success, a negative error otherwise. + */ +int avpriv_v4l2_enqueue_frame(V4L2Context* ctx, const AVFrame* f); + +#endif // AVCODEC_V4L2_BUFFERS_H diff --git a/libavcodec/v4l2_fmt.c b/libavcodec/v4l2_fmt.c index 2cda6b2..d25ff42 100644 --- a/libavcodec/v4l2_fmt.c +++ b/libavcodec/v4l2_fmt.c @@ -19,9 +19,8 @@ #include "v4l2_fmt.h" const struct v4l_fmt_map avpriv_v4l_fmt_conversion_table[] = { - /* ff_fmt codec_id v4l2_fmt pack_flags */ - { AV_PIX_FMT_YUV420P, AV_CODEC_ID_RAWVIDEO, V4L2_PIX_FMT_YUV420 , FF_V4L_PACK_AVPACKET }, - { AV_PIX_FMT_YUV420P, AV_CODEC_ID_RAWVIDEO, V4L2_PIX_FMT_YVU420 , FF_V4L_PACK_AVPACKET }, + /* ff_fmt codec_id v4l2_fmt pack_flags */ + { AV_PIX_FMT_YUV420P, AV_CODEC_ID_RAWVIDEO, V4L2_PIX_FMT_YUV420 , FF_V4L_PACK_AVPACKET | FF_V4L_PACK_AVFRAME }, { AV_PIX_FMT_YUV422P, AV_CODEC_ID_RAWVIDEO, V4L2_PIX_FMT_YUV422P , FF_V4L_PACK_AVPACKET }, { AV_PIX_FMT_YUYV422, AV_CODEC_ID_RAWVIDEO, V4L2_PIX_FMT_YUYV , FF_V4L_PACK_AVPACKET | FF_V4L_PACK_AVFRAME }, { AV_PIX_FMT_UYVY422, AV_CODEC_ID_RAWVIDEO, V4L2_PIX_FMT_UYVY , FF_V4L_PACK_AVPACKET | FF_V4L_PACK_AVFRAME }, @@ -74,7 +73,7 @@ const struct v4l_fmt_map avpriv_v4l_fmt_conversion_table[] = { { AV_PIX_FMT_NONE, AV_CODEC_ID_DVVIDEO, V4L2_PIX_FMT_DV , FF_V4L_PACK_AVPACKET }, #endif #ifdef V4L2_PIX_FMT_H263 - { AV_PIX_FMT_NONE, AV_CODEC_ID_H263, V4L2_PIX_FMT_H263 , FF_V4L_PACK_AVPACKET }, + { AV_PIX_FMT_NONE, AV_CODEC_ID_H263, V4L2_PIX_FMT_H263 , FF_V4L_PACK_AVPACKET | FF_V4L_PACK_AVFRAME }, #endif #ifdef V4L2_PIX_FMT_MPEG1 { AV_PIX_FMT_NONE, AV_CODEC_ID_MPEG1VIDEO, V4L2_PIX_FMT_MPEG1 , FF_V4L_PACK_AVPACKET }, @@ -89,7 +88,7 @@ const struct v4l_fmt_map avpriv_v4l_fmt_conversion_table[] = { { AV_PIX_FMT_NONE, AV_CODEC_ID_VP8, V4L2_PIX_FMT_VP8 , FF_V4L_PACK_AVPACKET }, #endif #ifdef V4L2_PIX_FMT_HEVC - { AV_PIX_FMT_NONE, AV_CODEC_ID_HEVC, V4L2_PIX_FMT_HEVC , FF_V4L_PACK_AVPACKET}, + { AV_PIX_FMT_NONE, AV_CODEC_ID_HEVC, V4L2_PIX_FMT_HEVC , FF_V4L_PACK_AVPACKET }, #endif #ifdef V4L2_PIX_FMT_VP9 { AV_PIX_FMT_NONE, AV_CODEC_ID_VP9, V4L2_PIX_FMT_VP9 , FF_V4L_PACK_AVPACKET }, diff --git a/libavcodec/v4l2_m2m.c b/libavcodec/v4l2_m2m.c new file mode 100644 index 0000000..52986b6 --- /dev/null +++ b/libavcodec/v4l2_m2m.c @@ -0,0 +1,428 @@ +/* + * V4L mem2mem wrapper + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <unistd.h> +#include <dirent.h> +#include <fcntl.h> +#include "libavutil/imgutils.h" +#include "libavcodec/internal.h" +#include "libavutil/pixfmt.h" +#include "libavutil/pixdesc.h" +#include "avcodec.h" +#include "v4l2_m2m_avcodec.h" +#include "v4l2_buffers.h" +#include "v4l2_fmt.h" +#include "v4l2_m2m.h" + +static inline int try_raw_format(V4L2Context* ctx, enum AVPixelFormat pixfmt) +{ + struct v4l2_format *fmt = &ctx->format; + uint32_t v4l2_fmt; + int ret; + + v4l2_fmt = avpriv_v4l_fmt_ff2v4l(pixfmt, ctx->av_codec_id, FF_V4L_PACK_AVFRAME); + if (!v4l2_fmt) + return AVERROR(EINVAL); + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) + fmt->fmt.pix_mp.pixelformat = v4l2_fmt; + else + fmt->fmt.pix.pixelformat = v4l2_fmt; + + fmt->type = ctx->type; + + ret = ioctl(ctx->fd, VIDIOC_TRY_FMT, fmt); + if (ret) + return AVERROR(EINVAL); + + return 0; +} + +static int query_raw_format(V4L2Context* ctx, int set) +{ + enum AVPixelFormat pixfmt = ctx->av_pix_fmt; + struct v4l2_fmtdesc fdesc; + int ret; + + memset(&fdesc, 0, sizeof(fdesc)); + fdesc.type = ctx->type; + + if (pixfmt != AV_PIX_FMT_NONE) { + ret = try_raw_format(ctx, pixfmt); + if (ret) + pixfmt = AV_PIX_FMT_NONE; + else + return 0; + } + + for (;;) { + ret = ioctl(ctx->fd, VIDIOC_ENUM_FMT, &fdesc); + if (ret) + return AVERROR(EINVAL); + + pixfmt = avpriv_v4l_fmt_v4l2ff(fdesc.pixelformat, AV_CODEC_ID_RAWVIDEO); + ret = try_raw_format(ctx, pixfmt); + if (ret){ + fdesc.index++; + continue; + } + + if (set) + ctx->av_pix_fmt = pixfmt; + + return 0; + } + + return AVERROR(EINVAL); +} + +static int query_coded_format(V4L2Context* ctx, uint32_t *p) +{ + struct v4l2_fmtdesc fdesc; + uint32_t v4l2_fmt; + int ret; + + v4l2_fmt = avpriv_v4l_fmt_ff2v4l(ctx->av_pix_fmt, ctx->av_codec_id, FF_V4L_PACK_AVPACKET); + if (!v4l2_fmt) + return AVERROR(EINVAL); + + memset(&fdesc, 0, sizeof(fdesc)); + fdesc.type = ctx->type; + + for (;;) { + ret = ioctl(ctx->fd, VIDIOC_ENUM_FMT, &fdesc); + if (ret) + return AVERROR(EINVAL); + + if (fdesc.pixelformat == v4l2_fmt) { + break; + } + + fdesc.index++; + } + + *p = v4l2_fmt; + + return 0; +} + +static inline int splane_video(struct v4l2_capability *cap) +{ + if (cap->capabilities & (V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT) & V4L2_CAP_STREAMING) + return 1; + + if (cap->capabilities & V4L2_CAP_VIDEO_M2M) + return 1; + + return 0; +} + +static inline int mplane_video(struct v4l2_capability *cap) +{ + if (cap->capabilities & (V4L2_CAP_VIDEO_CAPTURE_MPLANE | V4L2_CAP_VIDEO_OUTPUT_MPLANE) & V4L2_CAP_STREAMING) + return 1; + + if (cap->capabilities & V4L2_CAP_VIDEO_M2M_MPLANE) + return 1; + + return 0; +} + +static int prepare_contexts(V4L2m2mContext* s, void *log_ctx) +{ + int ret; + + s->capture.log_ctx = s->output.log_ctx = log_ctx; + s->capture.broken = s->output.broken = 0; + s->capture.fd = s->output.fd = s->fd; + s->capture.name = "v4l2_cap"; + s->output.name = "v4l2_out"; + + memset(&s->cap, 0, sizeof(s->cap)); + ret = ioctl(s->fd, VIDIOC_QUERYCAP, &s->cap); + if (ret < 0) + return ret; + + if (mplane_video(&s->cap)) { + s->capture.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; + s->output.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; + return 0; + } + + if (splane_video(&s->cap)) { + s->capture.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + s->output.type = V4L2_BUF_TYPE_VIDEO_OUTPUT; + return 0; + } + + return AVERROR(EINVAL); +} + +static int probe_v4l2_driver(V4L2m2mContext* s, void *log_ctx) +{ + int ret; + + s->fd = open(s->devname, O_RDWR | O_NONBLOCK, 0); + if (s->fd < 0) + return AVERROR(errno); + + ret = prepare_contexts(s, log_ctx); + if (ret < 0) + goto done; + + ret = avpriv_v4l2m2m_format(&s->output, 0); + if (ret) { + av_log(log_ctx, AV_LOG_DEBUG, "can't set input format\n"); + goto done; + } + + ret = avpriv_v4l2m2m_format(&s->capture, 0); + if (ret) { + av_log(log_ctx, AV_LOG_DEBUG, "can't to set output format\n"); + goto done; + } + +done: + + close(s->fd); + s->fd = 0; + + return ret; +} + +static int configure_contexts(V4L2m2mContext* s, void *log_ctx) +{ + int ret; + + s->fd = open(s->devname, O_RDWR | O_NONBLOCK, 0); + if (s->fd < 0) + return AVERROR(errno); + + ret = prepare_contexts(s, log_ctx); + if (ret < 0) + goto error; + + ret = avpriv_v4l2m2m_format(&s->output, 1); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "can't set input format\n"); + goto error; + } + + ret = avpriv_v4l2m2m_format(&s->capture, 1); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "can't to set output format\n"); + goto error; + } + + ret = avpriv_v4l2_context_init(&s->output, s->output.lazy_init); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "no output context's buffers\n"); + goto error; + } + + ret = avpriv_v4l2_context_init(&s->capture, s->capture.lazy_init); + if (ret) { + av_log(log_ctx, AV_LOG_ERROR, "no capture context's buffers\n"); + goto error; + } + + av_log(log_ctx, AV_LOG_INFO, "using driver '%s' on card '%s'\n", s->cap.driver, s->cap.card); + +error: + + if (ret) { + close(s->fd); + s->fd = 0; + } + + return 0; +} + +static void save_to_context(V4L2Context* ctx, uint32_t v4l2_fmt) +{ + ctx->format.type = ctx->type; + + if (V4L2_TYPE_IS_MULTIPLANAR(ctx->type)) { + /* this is to handle the reconfiguration of the capture stream at runtime */ + ctx->format.fmt.pix_mp.height = ctx->height; + ctx->format.fmt.pix_mp.width = ctx->width; + if (v4l2_fmt) + ctx->format.fmt.pix_mp.pixelformat = v4l2_fmt; + } else { + ctx->format.fmt.pix.height = ctx->height; + ctx->format.fmt.pix.width = ctx->width; + if (v4l2_fmt) + ctx->format.fmt.pix_mp.pixelformat = v4l2_fmt; + } +} + +int avpriv_v4l2m2m_format(V4L2Context* ctx, int set) +{ + uint32_t v4l2_fmt; + int ret; + + switch (ctx->av_codec_id) { + case AV_CODEC_ID_RAWVIDEO: + ret = query_raw_format(ctx, set); + if (ret) + return ret; + + save_to_context(ctx, 0); + + if (set) + return ioctl(ctx->fd, VIDIOC_S_FMT, &ctx->format); + + return ret; + default: + ret = query_coded_format(ctx, &v4l2_fmt); + if (ret) + return ret; + + save_to_context(ctx, v4l2_fmt); + + if (set) + return ioctl(ctx->fd, VIDIOC_S_FMT, &ctx->format); + + return ioctl(ctx->fd, VIDIOC_TRY_FMT, &ctx->format); + }; +} + +int avpriv_v4l2m2m_end(V4L2m2mContext* s) +{ + int ret; + + ret = avpriv_v4l2_context_set_status(&s->output, VIDIOC_STREAMOFF); + if (ret) + av_log(s->output.log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", s->output.name); + + ret = avpriv_v4l2_context_set_status(&s->capture, VIDIOC_STREAMOFF); + if (ret) + av_log(s->capture.log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", s->capture.name); + + avpriv_v4l2_context_release(&s->output); + avpriv_v4l2_context_release(&s->capture); + + close(s->fd); + + return 0; +} + +int avpriv_v4l2m2m_init(V4L2m2mContext* s, void* log_ctx) +{ + char *devname_save = s->devname; + int ret = AVERROR(EINVAL); + char node[PATH_MAX]; + struct dirent *entry; + DIR *dirp; + + if (s->devname && *s->devname) + return configure_contexts(s, log_ctx); + + dirp = opendir("/dev"); + if (!dirp) + return AVERROR(errno); + + for (entry = readdir(dirp); entry; entry = readdir(dirp)) { + + if (strncmp(entry->d_name, "video", 5)) + continue; + + snprintf(node, sizeof(node), "/dev/%s", entry->d_name); + + av_log(log_ctx, AV_LOG_DEBUG, "probing device %s\n", node); + + s->devname = node; + ret = probe_v4l2_driver(s, log_ctx); + if (!ret) + break; + } + + closedir(dirp); + + if (!ret) { + av_log(log_ctx, AV_LOG_INFO, "Using device %s\n", node); + ret = configure_contexts(s, log_ctx); + } else { + av_log(log_ctx, AV_LOG_ERROR, "Could not find a valid device\n"); + } + s->devname = devname_save; + + return ret; +} + +int avpriv_v4l2m2m_reinit(V4L2Context* ctx) +{ + V4L2m2mContext *s = container_of(ctx, V4L2m2mContext, capture); + int ret; + + av_log(ctx->log_ctx, AV_LOG_DEBUG, "%s reinit context\n", ctx->name); + + /* 1. streamoff */ + ret = avpriv_v4l2_context_set_status(ctx, VIDIOC_STREAMOFF); + if (ret) + av_log(ctx->log_ctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF %s\n", ctx->name); + + /* 2. unmap the buffers (v4l2 and ffmpeg) */ + avpriv_v4l2_context_release(ctx); + + /* 3. query the new format */ + ret = avpriv_v4l2m2m_format(ctx, 1); + if (ret) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "setting %s format\n", ctx->name); + return ret; + } + + /* 4. do lazy initialization */ + ret = avpriv_v4l2_context_init(ctx, ctx->lazy_init); + if (ret) { + av_log(ctx->log_ctx, AV_LOG_ERROR, "%s buffers lazy init\n", ctx->name); + return ret; + } + + /* 5. update AVCodecContext after re-init */ + ret = ff_set_dimensions(s->avctx, ctx->width, ctx->height); + if (ret < 0) + av_log(ctx->log_ctx, AV_LOG_WARNING, "update avcodec height and width\n"); + + return 0; +} + +int ff_v4l2m2m_codec_end(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + + av_log(avctx, AV_LOG_DEBUG, "Closing context\n"); + + return avpriv_v4l2m2m_end(s); +} + +int ff_v4l2m2m_codec_init(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + s->avctx = avctx; + + return avpriv_v4l2m2m_init(s, avctx); +} + diff --git a/libavcodec/v4l2_m2m.h b/libavcodec/v4l2_m2m.h new file mode 100644 index 0000000..41cd6e9 --- /dev/null +++ b/libavcodec/v4l2_m2m.h @@ -0,0 +1,59 @@ +/* + * V4L2 mem2mem helper functions + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_V4L2_M2M_H +#define AVCODEC_V4L2_M2M_H + +#include "v4l2_buffers.h" +#include "v4l2_fmt.h" + +#define container_of(ptr, type, member) ({ \ + const __typeof__(((type *)0)->member ) *__mptr = (ptr); \ + (type *)((char *)__mptr - offsetof(type,member) );}) + +#define V4L_M2M_DEFAULT_OPTS \ + { "device",\ + "Path to the device to use",\ + OFFSET(devname), AV_OPT_TYPE_STRING, {.str = NULL }, 0, 0, FLAGS },\ + { "num_output_buffers",\ + "Number of buffers in the output context",\ + OFFSET(output.num_buffers), AV_OPT_TYPE_INT, { .i64 = 16 }, 4, INT_MAX, FLAGS } + +typedef struct V4L2m2mContext +{ + AVClass *class; + int fd; + char *devname; + struct v4l2_capability cap; + V4L2Context capture; + V4L2Context output; + /* update codec while dynamic stream reconfig */ + AVCodecContext *avctx; +} V4L2m2mContext; + +int avpriv_v4l2m2m_init(V4L2m2mContext *ctx, void* log_ctx); +int avpriv_v4l2m2m_reinit(V4L2Context *ctx); +int avpriv_v4l2m2m_format(V4L2Context *ctx, int set); +int avpriv_v4l2m2m_end(V4L2m2mContext *ctx); + +#endif /* AVCODEC_V4L2_M2M_H */ diff --git a/libavcodec/v4l2_m2m_avcodec.h b/libavcodec/v4l2_m2m_avcodec.h new file mode 100644 index 0000000..3e17ae9 --- /dev/null +++ b/libavcodec/v4l2_m2m_avcodec.h @@ -0,0 +1,32 @@ +/* + * V4L2 mem2mem avcodec helper functions + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_V4L2_M2M_AVCODEC_H +#define AVCODEC_V4L2_M2M_AVCODEC_H + +#include "avcodec.h" + +int ff_v4l2m2m_codec_init(AVCodecContext *avctx); +int ff_v4l2m2m_codec_end(AVCodecContext *avctx); + +#endif diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c new file mode 100644 index 0000000..19fef72 --- /dev/null +++ b/libavcodec/v4l2_m2m_dec.c @@ -0,0 +1,252 @@ +/* + * V4L2 mem2mem decoders + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <sys/ioctl.h> +#include "libavutil/pixfmt.h" +#include "libavutil/pixdesc.h" +#include "libavutil/opt.h" +#include "v4l2_m2m_avcodec.h" +#include "v4l2_fmt.h" +#include "v4l2_buffers.h" +#include "v4l2_m2m.h" +#include "decode.h" +#include "avcodec.h" + +static int try_start(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const capture = &s->capture; + V4L2Context *const output = &s->output; + struct v4l2_event_subscription sub; + struct v4l2_selection selection; + struct v4l2_control ctrl; + int ret; + + if (output->streamon && capture->streamon) + return 0; + + /* 0. subscribe to source change event */ + memset(&sub, 0, sizeof(sub)); + sub.type = V4L2_EVENT_SOURCE_CHANGE; + ret = ioctl(s->fd, VIDIOC_SUBSCRIBE_EVENT, &sub); + if ( ret < 0) + av_log(avctx, AV_LOG_WARNING, "decoding does not support resolution change\n"); + + /* 1. start the output process */ + if (!output->streamon) { + ret = avpriv_v4l2_context_set_status(output, VIDIOC_STREAMON); + if (ret < 0) { + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON on output context\n"); + return ret; + } + } + + /* 2. get the capture format */ + capture->format.type = capture->type; + ret = ioctl(capture->fd, VIDIOC_G_FMT, &capture->format); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_FMT ioctl\n"); + return ret; + } + + /* 2.1 update the AVCodecContext */ + avctx->pix_fmt = avpriv_v4l_fmt_v4l2ff(capture->format.fmt.pix_mp.pixelformat, AV_CODEC_ID_RAWVIDEO); + capture->av_pix_fmt = avctx->pix_fmt; + + /* 3. set the crop parameters */ + selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + selection.r.height = avctx->coded_height; + selection.r.width = avctx->coded_width; + ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection); + if (!ret) { + ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_G_SELECTION ioctl\n"); + } else { + av_log(avctx, AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height); + /* update the size of the resulting frame */ + capture->height = selection.r.height; + capture->width = selection.r.width; + } + } + + /* 4. get the minimum number of buffers required by capture (or default) */ + memset(&ctrl, 0, sizeof(ctrl)); + ctrl.id = V4L2_CID_MIN_BUFFERS_FOR_CAPTURE; + ret = ioctl(s->fd, VIDIOC_G_CTRL, &ctrl); + if (!ret) + capture->min_queued_buffers = ctrl.value; + + /* 5. init the capture context now that we have the capture format */ + if (!capture->buffers) { + + av_log(capture->log_ctx, AV_LOG_DEBUG, "%s requested (%dx%d)\n", + capture->name, capture->format.fmt.pix_mp.width, capture->format.fmt.pix_mp.height); + + ret = avpriv_v4l2_context_init(capture, 0); + if (ret) { + av_log(avctx, AV_LOG_DEBUG, "can't request output buffers\n"); + return ret; + } + } + + /* 6. start the capture process */ + ret = avpriv_v4l2_context_set_status(capture, VIDIOC_STREAMON); + if (ret) { + av_log(avctx, AV_LOG_DEBUG, "VIDIOC_STREAMON, on capture context\n"); + return ret; + } + + return 0; +} + +static av_cold int v4l2m2m_decode_init(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *capture = &s->capture; + V4L2Context *output = &s->output; + + output->default_flags = capture->default_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + output->time_base = capture->time_base = avctx->framerate; + output->height = capture->height = avctx->coded_height; + output->width = capture->width =avctx->coded_width; + + output->av_codec_id = avctx->codec_id; + output->av_pix_fmt = AV_PIX_FMT_NONE; + output->min_queued_buffers = 6; + + /* + * increase the number of buffers to support h.264/h.265 + * default (configurable via option) is 16 + */ + switch (avctx->codec_id) { + case AV_CODEC_ID_H264: + case AV_CODEC_ID_HEVC: + output->num_buffers += 4; + break; + } + + /* + * the buffers associated to this context cant be initialized without + * additional information available in the kernel driver, so do let's postpone it + */ + capture->lazy_init = 1; + + capture->av_codec_id = AV_CODEC_ID_RAWVIDEO; + capture->av_pix_fmt = avctx->pix_fmt; + capture->min_queued_buffers = 6; + + return ff_v4l2m2m_codec_init(avctx); +} + +static int v4l2m2m_receive_frame(AVCodecContext *avctx, AVFrame *frame) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const capture = &s->capture; + V4L2Context *const output = &s->output; + unsigned int timeout = 50; + AVPacket avpkt = {0}; + int ret; + + ret = ff_decode_get_packet(avctx, &avpkt); + if (ret < 0 && ret != AVERROR_EOF) + return ret; + + ret = avpriv_v4l2_enqueue_packet(output, &avpkt); + if (ret < 0) + return ret; + + if (avpkt.size) { + ret = try_start(avctx); + if (ret) + return 0; + } + + return avpriv_v4l2_dequeue_frame(capture, frame, timeout); +} + +#define OFFSET(x) offsetof(V4L2m2mContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_DECODING_PARAM + + static const AVOption options[] = { + V4L_M2M_DEFAULT_OPTS,{ "num_capture_extra_buffers","Number of extra buffers in the capture context", + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT,{.i64 = 6}, 6, INT_MAX, FLAGS}, + { NULL}, + }; + +#define M2MDEC(NAME, LONGNAME, CODEC, bsf_name) \ +static const AVClass v4l2_m2m_ ## NAME ## _dec_class = {\ + .class_name = #NAME "_v4l2_m2m_decoder",\ + .item_name = av_default_item_name,\ + .option = options,\ + .version = LIBAVUTIL_VERSION_INT,\ +};\ +\ +AVCodec ff_ ## NAME ## _v4l2m2m_decoder = { \ + .name = #NAME "_v4l2m2m" ,\ + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " decoder wrapper"),\ + .type = AVMEDIA_TYPE_VIDEO,\ + .id = CODEC ,\ + .priv_data_size = sizeof(V4L2m2mContext),\ + .priv_class = &v4l2_m2m_ ## NAME ## _dec_class,\ + .init = v4l2m2m_decode_init,\ + .receive_frame = v4l2m2m_receive_frame,\ + .close = ff_v4l2m2m_codec_end,\ + .bsfs = bsf_name, \ +}; + +#if CONFIG_H263_V4L2M2M_DECODER + M2MDEC(h263, "H.263", AV_CODEC_ID_H263, NULL); +#endif + +#if CONFIG_H264_V4L2M2M_DECODER + M2MDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb"); +#endif + +#if CONFIG_MPEG1_V4L2M2M_DECODER + M2MDEC(mpeg1, "MPEG1", AV_CODEC_ID_MPEG1VIDEO, NULL); +#endif + +#if CONFIG_MPEG2_V4L2M2M_DECODER + M2MDEC(mpeg2, "MPEG2", AV_CODEC_ID_MPEG2VIDEO, NULL); +#endif + +#if CONFIG_MPEG4_V4L2M2M_DECODER + M2MDEC(mpeg4, "MPEG4", AV_CODEC_ID_MPEG4, NULL); +#endif + +#if CONFIG_VC1_V4L2M2M_DECODER + M2MDEC(vc1 , "VC1", AV_CODEC_ID_VC1, NULL); +#endif + +#if CONFIG_VP8_V4L2M2M_DECODER + M2MDEC(vp8, "VP8", AV_CODEC_ID_VP8, NULL); +#endif + +#if CONFIG_VP9_V4L2M2M_DECODER + M2MDEC(vp9, "VP9", AV_CODEC_ID_VP9, NULL); +#endif + +#if CONFIG_HEVC_V4L2M2M_DECODER + M2MDEC(hevc, "HEVC", AV_CODEC_ID_HEVC, "h264_mp4toannexb"); +#endif diff --git a/libavcodec/v4l2_m2m_enc.c b/libavcodec/v4l2_m2m_enc.c new file mode 100644 index 0000000..db9c090 --- /dev/null +++ b/libavcodec/v4l2_m2m_enc.c @@ -0,0 +1,288 @@ +/* + * V4L2 mem2mem encoders + * + * Copyright (C) 2017 Alexis Ballier <aballier@gentoo.org> + * Copyright (C) 2017 Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <sys/ioctl.h> + +#include "libavutil/pixfmt.h" +#include "libavutil/pixdesc.h" +#include "libavutil/opt.h" +#include "v4l2_m2m_avcodec.h" +#include "v4l2_buffers.h" +#include "v4l2_fmt.h" +#include "v4l2_m2m.h" +#include "avcodec.h" + +#define STR(s) AV_TOSTRING(s) +#define MPEG_CID(x) V4L2_CID_MPEG_VIDEO_##x +#define MPEG_VIDEO(x) V4L2_MPEG_VIDEO_##x + +#define SET_V4L2_EXT_CTRL(TYPE, ID, VALUE, NAME) \ +{ \ + struct v4l2_ext_control ctrl = { 0 }; \ + struct v4l2_ext_controls ctrls = { 0 }; \ + ctrls.ctrl_class = V4L2_CTRL_CLASS_MPEG; \ + ctrls.controls = &ctrl; \ + ctrl.TYPE = VALUE ; \ + ctrl.id = ID ; \ + ctrls.count = 1; \ + \ + if ((ret = ioctl(s->fd, VIDIOC_S_EXT_CTRLS, &ctrls)) < 0) \ + av_log(avctx, AV_LOG_WARNING, "Failed to set " NAME "%s\n", STR(ID)); \ +} + +#define SET_V4L2_TIME_PER_FRAME(NUM, DEN) \ +{ \ + struct v4l2_streamparm parm = { 0 }; \ + parm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; \ + parm.parm.output.timeperframe.numerator = NUM; \ + parm.parm.output.timeperframe.denominator = DEN; \ + \ + if ((ret = ioctl(s->fd, VIDIOC_S_PARM, &parm)) < 0) \ + av_log(avctx, AV_LOG_WARNING, "Failed to set timeperframe"); \ +} + +static inline int v4l2_h264_profile_from_ff(int p) +{ + switch(p) { + case FF_PROFILE_H264_CONSTRAINED_BASELINE: + return MPEG_VIDEO(H264_PROFILE_CONSTRAINED_BASELINE); + case FF_PROFILE_H264_HIGH_444_PREDICTIVE: + return MPEG_VIDEO(H264_PROFILE_HIGH_444_PREDICTIVE); + case FF_PROFILE_H264_HIGH_422_INTRA: + return MPEG_VIDEO(H264_PROFILE_HIGH_422_INTRA); + case FF_PROFILE_H264_HIGH_444_INTRA: + return MPEG_VIDEO(H264_PROFILE_HIGH_444_INTRA); + case FF_PROFILE_H264_HIGH_10_INTRA: + return MPEG_VIDEO(H264_PROFILE_HIGH_10_INTRA); + case FF_PROFILE_H264_HIGH_422: + return MPEG_VIDEO(H264_PROFILE_HIGH_422); + case FF_PROFILE_H264_BASELINE: + return MPEG_VIDEO(H264_PROFILE_BASELINE); + case FF_PROFILE_H264_EXTENDED: + return MPEG_VIDEO(H264_PROFILE_EXTENDED); + case FF_PROFILE_H264_HIGH_10: + return MPEG_VIDEO(H264_PROFILE_HIGH_10); + case FF_PROFILE_H264_MAIN: + return MPEG_VIDEO(H264_PROFILE_MAIN); + case FF_PROFILE_H264_HIGH: + return MPEG_VIDEO(H264_PROFILE_HIGH); + } + + return -1; +} + +static inline int v4l2_mpeg4_profile_from_ff(int p) +{ + switch(p) { + case FF_PROFILE_MPEG4_ADVANCED_CODING: + return MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_CODING_EFFICIENCY); + case FF_PROFILE_MPEG4_ADVANCED_SIMPLE: + return MPEG_VIDEO(MPEG4_PROFILE_ADVANCED_SIMPLE); + case FF_PROFILE_MPEG4_SIMPLE_SCALABLE: + + return MPEG_VIDEO(MPEG4_PROFILE_SIMPLE_SCALABLE); + case FF_PROFILE_MPEG4_SIMPLE: + return MPEG_VIDEO(MPEG4_PROFILE_SIMPLE); + case FF_PROFILE_MPEG4_CORE: + return MPEG_VIDEO(MPEG4_PROFILE_CORE); + } + + return -1; +} + +static av_cold int v4l2m2m_encode_init(AVCodecContext *avctx) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *capture = &s->capture; + V4L2Context *output = &s->output; + int qmin_cid, qmax_cid, ret, val; + int qmin, qmax; + + output->default_flags = capture->default_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + output->time_base = capture->time_base = avctx->time_base; + output->height = capture->height = avctx->height; + output->width = capture->width = avctx->width; + + /* output context */ + output->av_codec_id = AV_CODEC_ID_RAWVIDEO; + output->av_pix_fmt = avctx->pix_fmt; + + /* capture context */ + capture->av_codec_id = avctx->codec_id; + capture->av_pix_fmt = AV_PIX_FMT_NONE; + capture->min_queued_buffers = 1; + + if (ret = ff_v4l2m2m_codec_init(avctx)) + return ret; + + SET_V4L2_TIME_PER_FRAME(avctx->framerate.num, avctx->framerate.den); + SET_V4L2_EXT_CTRL(value, MPEG_CID(HEADER_MODE), MPEG_VIDEO(HEADER_MODE_SEPARATE), "header mode"); + SET_V4L2_EXT_CTRL(value, MPEG_CID(B_FRAMES), avctx->max_b_frames, "number of B-frames"); + SET_V4L2_EXT_CTRL(value, MPEG_CID(GOP_SIZE), avctx->gop_size,"gop size"); + SET_V4L2_EXT_CTRL(value, MPEG_CID(BITRATE) , avctx->bit_rate, "bit rate"); + + av_log(avctx, AV_LOG_DEBUG, "Encoder Context: frame rate(%d/%d), number b-frames (%d)," + "gop size (%d), bit rate (%ld), qmin (%d), qmax (%d)\n", + avctx->framerate.num, avctx->framerate.den, avctx->max_b_frames, avctx->gop_size, avctx->bit_rate, avctx->qmin, avctx->qmax); + + switch(avctx->codec_id) { + case AV_CODEC_ID_H264: + val = v4l2_h264_profile_from_ff(avctx->profile); + if (val >= 0) { + SET_V4L2_EXT_CTRL(value, MPEG_CID(H264_PROFILE), val, "h264 profile"); + } + qmin_cid = MPEG_CID(H264_MIN_QP); + qmax_cid = MPEG_CID(H264_MAX_QP); + qmin = 0; + qmax = 51; + break; + case AV_CODEC_ID_MPEG4: + val = v4l2_mpeg4_profile_from_ff(avctx->profile); + if (val >= 0) { + SET_V4L2_EXT_CTRL(value, MPEG_CID(MPEG4_PROFILE), val, "mpeg4 profile"); + } + qmin_cid = MPEG_CID(MPEG4_MIN_QP); + qmax_cid = MPEG_CID(MPEG4_MAX_QP); + if (avctx->flags & CODEC_FLAG_QPEL) { + SET_V4L2_EXT_CTRL(value, MPEG_CID(MPEG4_QPEL), 1, "qpel"); + } + qmax = 51; + qmin = 0; + break; + case AV_CODEC_ID_H263: + qmin_cid = MPEG_CID(H263_MIN_QP); + qmax_cid = MPEG_CID(H263_MAX_QP); + qmin = 1; + qmax = 31; + break; + case AV_CODEC_ID_VP8: + qmin_cid = MPEG_CID(VPX_MIN_QP); + qmax_cid = MPEG_CID(VPX_MAX_QP); + qmin = 0; + qmax = 127; + break; + case AV_CODEC_ID_VP9: + qmin_cid = MPEG_CID(VPX_MIN_QP); + qmax_cid = MPEG_CID(VPX_MAX_QP); + qmin = 0; + qmax = 255; + break; + default: + return 0; + } + + if (qmin != avctx->qmin || qmax != avctx->qmax) + av_log(avctx, AV_LOG_WARNING, "Encoder adjusted: qmin (%d), qmax (%d)\n", qmin, qmax); + + SET_V4L2_EXT_CTRL(value, qmin_cid, qmin, "minimum video quantizer scale"); + SET_V4L2_EXT_CTRL(value, qmax_cid, qmax, "maximum video quantizer scale"); + + return 0; +} + +static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const output = &s->output; + + return avpriv_v4l2_enqueue_frame(output, frame); +} + +static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt) +{ + V4L2m2mContext *s = avctx->priv_data; + V4L2Context *const capture = &s->capture; + V4L2Context *const output = &s->output; + unsigned int timeout = 50; + int ret; + + if (!output->streamon) { + ret = avpriv_v4l2_context_set_status(output, VIDIOC_STREAMON); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n"); + return ret; + } + } + + if (!capture->streamon) { + ret = avpriv_v4l2_context_set_status(capture, VIDIOC_STREAMON); + if (ret) { + av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n"); + return ret; + } + } + + return avpriv_v4l2_dequeue_packet(capture, avpkt, timeout); +} + +#define OFFSET(x) offsetof(V4L2m2mContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM + +static const AVOption options[] = { + V4L_M2M_DEFAULT_OPTS, + { "num_capture_buffers", "Number of buffers in the capture context", + OFFSET(capture.num_buffers), AV_OPT_TYPE_INT, {.i64 = 4 }, 4, INT_MAX, FLAGS }, + { NULL }, +}; + +#define M2MENC(NAME, LONGNAME, CODEC) \ +static const AVClass v4l2_m2m_ ## NAME ## _enc_class = {\ + .class_name = #NAME "_v4l2_m2m_encoder",\ + .item_name = av_default_item_name,\ + .option = options,\ + .version = LIBAVUTIL_VERSION_INT,\ +};\ +\ +AVCodec ff_ ## NAME ## _v4l2m2m_encoder = { \ + .name = #NAME "_v4l2m2m" ,\ + .long_name = NULL_IF_CONFIG_SMALL("V4L2 mem2mem " LONGNAME " encoder wrapper"),\ + .type = AVMEDIA_TYPE_VIDEO,\ + .id = CODEC ,\ + .priv_data_size = sizeof(V4L2m2mContext),\ + .priv_class = &v4l2_m2m_ ## NAME ##_enc_class,\ + .init = v4l2m2m_encode_init,\ + .send_frame = v4l2m2m_send_frame,\ + .receive_packet = v4l2m2m_receive_packet,\ + .close = ff_v4l2m2m_codec_end,\ +}; + +#if CONFIG_H263_V4L2M2M_ENCODER +M2MENC(h263, "H.263", AV_CODEC_ID_H263); +#endif + +#if CONFIG_H264_V4L2M2M_ENCODER +M2MENC(h264, "H.264", AV_CODEC_ID_H264); +#endif + +#if CONFIG_MPEG4_V4L2M2M_ENCODER +M2MENC(mpeg4, "MPEG4", AV_CODEC_ID_MPEG4); +#endif + +#if CONFIG_VP8_V4L2M2M_ENCODER +M2MENC(vp8, "VP8", AV_CODEC_ID_VP8); +#endif + +#if CONFIG_HEVC_V4L2M2M_ENCODER +M2MENC(hevc, "HEVC", AV_CODEC_ID_HEVC); +#endif +
This patchset enhances Alexis Ballier's original patch and validates it using Qualcomm's Venus hardware (driver recently landed upstream [1]). This has been tested on Qualcomm's DragonBoard 410c and 820c Configure/make scripts have been validated on Ubuntu 10.04 and 16.04. Tested decoders: - h264 - mpeg4 - vp8 - vp9 - hevc Tested encoders: - h264 - h263 - mpeg4 Tested transcoding (concurrent encoding/decoding) Some of the changes introduced: - v4l2: code cleanup and abstractions added - v4l2: follow the new encode/decode api. - v4l2: fix display size for NV12 output pool. - v4l2: handle EOS. - v4l2: vp8 and mpeg4 decoding and encoding. - v4l2: hevc and vp9 support. - v4l2: generate EOF on dequeue errors. - v4l2: h264_mp4toannexb filtering. - v4l2: fixed make install and fate issues. - v4l2: codecs enabled/disabled depending on pixfmt defined - v4l2: pass timebase/framerate to the context - v4l2: runtime decoder reconfiguration. [1] https://lwn.net/Articles/697956/ Reviewed-by: Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> Reviewed-by: Alexis Ballier <aballier@gentoo.org> Tested-by: Jorge Ramirez <jorge.ramirez-ortiz@linaro.org> --- Changelog | 1 + configure | 30 +- libavcodec/Makefile | 18 +- libavcodec/allcodecs.c | 9 + libavcodec/v4l2_buffers.c | 741 ++++++++++++++++++++++++++++++++++++++++++ libavcodec/v4l2_buffers.h | 259 +++++++++++++++ libavcodec/v4l2_fmt.c | 9 +- libavcodec/v4l2_m2m.c | 428 ++++++++++++++++++++++++ libavcodec/v4l2_m2m.h | 59 ++++ libavcodec/v4l2_m2m_avcodec.h | 32 ++ libavcodec/v4l2_m2m_dec.c | 252 ++++++++++++++ libavcodec/v4l2_m2m_enc.c | 288 ++++++++++++++++ 12 files changed, 2119 insertions(+), 7 deletions(-) create mode 100644 libavcodec/v4l2_buffers.c create mode 100644 libavcodec/v4l2_buffers.h create mode 100644 libavcodec/v4l2_m2m.c create mode 100644 libavcodec/v4l2_m2m.h create mode 100644 libavcodec/v4l2_m2m_avcodec.h create mode 100644 libavcodec/v4l2_m2m_dec.c create mode 100644 libavcodec/v4l2_m2m_enc.c