From patchwork Mon Sep 3 21:55:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Thompson X-Patchwork-Id: 10224 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:12c4:0:0:0:0:0 with SMTP id 65-v6csp3151654jap; Mon, 3 Sep 2018 14:55:42 -0700 (PDT) X-Google-Smtp-Source: ANB0VdasiT2gdm/MBlDgMSx8Un7UW1pvFB4kMRnyYXmgCV5msdYVPU/ENXMR1WclOqN9x7ZiGCTb X-Received: by 2002:a1c:9706:: with SMTP id z6-v6mr6591291wmd.0.1536011742251; Mon, 03 Sep 2018 14:55:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536011742; cv=none; d=google.com; s=arc-20160816; b=daYHqMnUNgDi79PmsBXohRY73xYu1TXocvPicceMD6t1kproQNAGdVfr2gv/Kc5698 cU4HlhqXb/RVO/LA/a5nlik+Sdq+mrb/MCLYqtcYiW9h5r7EEYRAFUYKYnwSemSVjEOq aFzh5QrIFWJqcgZL8yT6guJQ/LPFFwAXXZHPr3rp5+rf0y2qgx4EGIc7OfeMLOWEylOG DdL1L/e6dPYO00csTW9uD4buPchLnosgB5f/e8E26uW7NrGAFP4BmjOB7W0niAg+oasG wB2oq9NuFOd+xiKCMOpjWL1aFtOXM7lAfnIEXpSC/5+p1PsiOvXToUikIWi+xqk8tcRF fSoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:dkim-signature :delivered-to; bh=spGpp4G6AUlCKgSqCcuG+uE7QsPvaTQ9q1fv1nhrgf0=; b=0q/zFt6b0CqBnNy/7q0ShQijy2+dL8ANqyg1azpLpJDGE2gSanucQVr55+8rFueZCQ 9sjuAMiTDe4Ltx0OAzUnRHlPmB8K9l7tUdrCyb6DQKwxHx7t2mRW31Pfgp6G3TcVanG2 WrC8rHnyGapBuO201eh0iWjrMAS9TQVT8MmHtjrnEu6Dqqk+MT3Yr9Z6tiUFUkFO7sWW GxnthRQMxxhJZZynqdI7LSU+P4OCKLUUuJcLfaWdryrvn9TSzKn4GZfmGT5rKEEt8xVJ +9zE3rlSYLWr1e4obhq88D3BSCxpavLJBeNO8AmzTtBWZPSY/mXKaqe/Amd0pxczP413 WApw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=YJXMnQxc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y189-v6si10543713wmg.96.2018.09.03.14.55.41; Mon, 03 Sep 2018 14:55:42 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=YJXMnQxc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0E3A9689B89; Tue, 4 Sep 2018 00:55:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9453C68996D for ; Tue, 4 Sep 2018 00:55:26 +0300 (EEST) Received: by mail-wm0-f52.google.com with SMTP id r1-v6so7638634wmh.0 for ; Mon, 03 Sep 2018 14:55:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jkqxz-net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=aPQR4110Gf4MAmsLrfvX9VtZN54YoLYqE8/CRykebmY=; b=YJXMnQxcUiaXTem2P3geHIeIXdqV4a+b62/GRlQHQUYJj/ugcQXMhjwXbM/aQOgpyx +WBQKii3i8dGZcXHb+OP7o903q3bDYAcNOWLBej+FZK7n7lWfbxCfnrWSCiRbKn/tlZ5 D2/7bmA51UnqcR7Bcl3uBcWUUUADHak+W1k+dgycilMTgvQaRyVCLtJZFiOm4XWIiUsE MV7cmshgaF1KNTQKO/P0S4z1muPg8ilSS9dCMnVcEWyQM2SV8NUZH3boJN0b4/t4K9LQ NW5kMIQqV5tYseI6Qyvv7pgTJ85He1wE44l1PNegI6g/SgY7LnGp1jjOEDWSvqAQJgSG PzvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=aPQR4110Gf4MAmsLrfvX9VtZN54YoLYqE8/CRykebmY=; b=kFZofSVgnOkVp0zVYTMv3T2XjUKC+uwbec84gd9JQpDtYlwY3dLXW5EQgqW6dgzQwB ftbuFf5aEx0XN+6NB50Kk3Xz8SKpYzt/aBkTF1Yjpe0JFzdkT6rGRo/myEkMN3PftR9S 06nnOM2m6CvzCeF0yVyrS5L1wdgyZqL90bo8/wCAEtwxGW9TEJkWrHWB9wX1N8QUeaDZ Gs8acUWbRCzh6beCOKROIE6ycRCaSYqwdCMZGO3HOZV8zaSwjRSZdLfzuzDUu547IlSK DqnrKpYSDZnc4gxzrbB4RNU3rDVgM9qUlwk9GUbICRP+UPpc+PlCZwGG0YDwoEaU2Kn7 2bvQ== X-Gm-Message-State: APzg51DFGJV2i7TwC8Gjk0pJfruSOX9J+vGQ5GaTzarcEhhobAb0Bk2Z mCOqcuR+kMI0nHS/oFXuJslIyiV4EDI= X-Received: by 2002:a1c:888e:: with SMTP id k136-v6mr793234wmd.6.1536011731915; Mon, 03 Sep 2018 14:55:31 -0700 (PDT) Received: from [192.168.0.4] (cpc91242-cmbg18-2-0-cust650.5-4.cable.virginm.net. [82.8.130.139]) by smtp.gmail.com with ESMTPSA id f18-v6sm18618261wru.51.2018.09.03.14.55.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Sep 2018 14:55:31 -0700 (PDT) To: ffmpeg-devel@ffmpeg.org References: <20180804004031.12143-1-lorusak@gmail.com> <20180804004031.12143-2-lorusak@gmail.com> <4bbeff66-6123-9296-5775-96d75809eee0@jkqxz.net> From: Mark Thompson Message-ID: <53506918-5c4e-9248-64f2-d8007720d517@jkqxz.net> Date: Mon, 3 Sep 2018 22:55:30 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavcodec: v4l2m2m: output AVDRMFrameDescriptor X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On 17/08/18 04:36, Lukas Rusak wrote: > On Sat, 2018-08-04 at 22:43 +0100, Mark Thompson wrote: >> On 04/08/18 01:40, Lukas Rusak wrote: >>> This allows for a zero-copy output by exporting the v4l2 buffer >>> then wrapping that buffer >>> in the AVDRMFrameDescriptor like it is done in rkmpp. >>> >>> This has been in use for quite some time with great success on many >>> platforms including: >>> - Amlogic S905 >>> - Raspberry Pi >>> - i.MX6 >>> - Dragonboard 410c >>> >>> This was developed in conjunction with Kodi to allow handling the >>> zero-copy buffer rendering. >>> A simply utility for testing is also available here: >>> https://github.com/BayLibre/ffmpeg-drm >>> >>> todo: >>> - allow selecting pixel format output from decoder >>> - allow configuring amount of output and capture buffers >>> >>> V2: >>> - allow selecting AV_PIX_FMT_DRM_PRIME >>> >>> V3: >>> - use get_format to select AV_PIX_FMT_DRM_PRIME >>> - use hw_configs >>> - add handling of AV_PIX_FMT_YUV420P format (for raspberry pi) >>> - add handling of AV_PIX_FMT_YUYV422 format (for i.MX6 coda >>> decoder) >>> --- >>> libavcodec/v4l2_buffers.c | 216 ++++++++++++++++++++++++++++++++ >>> ------ >>> libavcodec/v4l2_buffers.h | 4 + >>> libavcodec/v4l2_context.c | 40 ++++++- >>> libavcodec/v4l2_m2m.c | 4 +- >>> libavcodec/v4l2_m2m.h | 3 + >>> libavcodec/v4l2_m2m_dec.c | 23 ++++ >>> 6 files changed, 253 insertions(+), 37 deletions(-) >>> ... >>> + >>> static int v4l2_bufref_to_buf(V4L2Buffer *out, int plane, const >>> uint8_t* data, int size, AVBufferRef* bref) >>> { >>> unsigned int bytesused, length; >>> @@ -308,31 +442,43 @@ int ff_v4l2_buffer_buf_to_avframe(AVFrame >>> *frame, V4L2Buffer *avbuf) >>> >>> av_frame_unref(frame); >>> >>> - /* 1. get references to the actual data */ >>> - for (i = 0; i < avbuf->num_planes; i++) { >>> - ret = v4l2_buf_to_bufref(avbuf, i, &frame->buf[i]); >>> + if (buf_to_m2mctx(avbuf)->output_drm) { >>> + /* 1. get references to the actual data */ >>> + ret = v4l2_buf_to_bufref_drm(avbuf, &frame->buf[0]); >>> if (ret) >>> return ret; >>> >>> - frame->linesize[i] = avbuf->plane_info[i].bytesperline; >>> - frame->data[i] = frame->buf[i]->data; >>> - } >>> + frame->data[0] = (uint8_t *) v4l2_get_drm_frame(avbuf); >>> + frame->format = AV_PIX_FMT_DRM_PRIME; >> >> frame->hw_frames_ctx needs to be set here as well. (I think you can >> use the same trivial device/frames context setup as in rkmppdec.c.) >> > > Can you help me with this? This is the part I'm confused about. the > v4l2 code here seems to use it's own reference counting so are we > wanting to convert that to use the hw ctx instead or what is actually > happening? Working test patch below without changing anything else. Given that the V4L2 code already has its own custom reference counting I don't think there is any point in changing that part. (Todo: put the initialisation in a more sensible place, actually do cleanup, make the failure modes something other than abort().) With that change (and the other hacky patch), hwdownload does work on the Odroid XU4 as, e.g.: $ ./ffmpeg_g -y -c:v h264_v4l2m2m -hwaccel drm -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format drm_prime -i in.mp4 -an -vf 'hwdownload,format=nv21' -c:v libx264 out.mp4 (The output DRM objects are apparently both linear and CPU-mappable.) Then trying to map to OpenCL to do something with it like: $ ./ffmpeg_g -y -c:v h264_v4l2m2m -hwaccel drm -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format drm_prime -init_hw_device opencl=cl:0.0 -filter_hw_device cl -i in.mp4 -an -vf hwmap,unsharp_opencl -f null - almost works, but not quite because the Mali T628 driver lacks the standard cl_khr_image2d_from_buffer extension. Removing the extension check does get it right up to a failure on creating the image from the imported buffer, though, so the import is indeed working. (The same case is working with the T760 in the RK3288, clearly I need a slightly newer test setup to get this working with V4L2.) >>> ... >>> @@ -186,6 +192,15 @@ static av_cold int >>> v4l2_decode_init(AVCodecContext *avctx) >>> capture->av_codec_id = AV_CODEC_ID_RAWVIDEO; >>> capture->av_pix_fmt = avctx->pix_fmt; >>> >>> + /* the client requests the codec to generate DRM frames: >>> + * - data[0] will therefore point to the returned >>> AVDRMFrameDescriptor >>> + * check the ff_v4l2_buffer_to_avframe conversion >>> function. >>> + * - the DRM frame format is passed in the DRM frame >>> descriptor layer. >>> + * check the v4l2_get_drm_frame function. >>> + */ >>> + if (ff_get_format(avctx, avctx->codec->pix_fmts) == >>> AV_PIX_FMT_DRM_PRIME) >>> + s->output_drm = 1; >> >> This list needs to contain the software pixfmts as well, so that the >> user can pick from the correct list. >> > > Is there a simple way to do this or a list that stays updated with all > the possible formats? Not really - you will need to query what software pixfmts are actually supported by the hardware to make the list, so it's completely dependent on V4L2. >> (If ff_get_format() returns AV_PIX_FMT_NONE it means the user has >> declined to use any of the available pixfmts, and the decoder should >> exit cleanly in that case.) >> > > makes sense. > >>> + >>> ret = ff_v4l2_m2m_codec_init(avctx); >>> if (ret) { >>> av_log(avctx, AV_LOG_ERROR, "can't configure decoder\n"); >>> @@ -205,6 +220,11 @@ static const AVOption options[] = { >>> { NULL}, >>> }; >>> >>> +static const AVCodecHWConfigInternal *v4l2_m2m_hw_configs[] = { >>> + HW_CONFIG_INTERNAL(DRM_PRIME), >>> + NULL >>> +}; >>> + >>> #define M2MDEC_CLASS(NAME) \ >>> static const AVClass v4l2_m2m_ ## NAME ## _dec_class = { \ >>> .class_name = #NAME "_v4l2_m2m_decoder", \ >>> @@ -225,7 +245,10 @@ static const AVOption options[] = { >>> .init = v4l2_decode_init, \ >>> .receive_frame = v4l2_receive_frame, \ >>> .close = ff_v4l2_m2m_codec_end, \ >>> + .pix_fmts = (const enum AVPixelFormat[]) { >>> AV_PIX_FMT_DRM_PRIME, \ >>> + AV_PIX_FM >>> T_NONE}, \ >> >> I'm not entirely sure, but I think this list is meant to be >> exhaustive if provided? >> >>> .bsfs = bsf_name, \ >>> + .hw_configs = v4l2_m2m_hw_configs, \ >>> .capabilities = AV_CODEC_CAP_HARDWARE | >>> AV_CODEC_CAP_DELAY, \ >>> AV_CODEC_CAP_AVOID_PROBING, \ >>> .wrapper_name = "v4l2m2m", \ >>> >> >> I had a go at using this on an Exynos device (Odroid XU4) with OpenCL >> in the ffmpeg utility (using cl_arm_import_memory, which works with >> kmsgrab on this device and can be used with the decoder on Rockchip). >> > > unfortunately the exynos devices aren't the best to test on, the v4l2 > device can decode to the correct formats but the drm plane cannot > accept any of the formats the decoder outputs. So it can be used for > pure conversion testing if you want. I recommend using an RPi with Dave > Stevensons v4l2 patches I'm not sure what you mean by "the drm plane" here? The DRM object export seems fine and it imports to Mali OpenCL - getting tripped up for actually doing anything with it by lack of support in the OpenCL driver is mostly orthogonal. Thanks, - Mark diff --git a/libavcodec/v4l2_buffers.c b/libavcodec/v4l2_buffers.c index 897c3c4..45f7bcf 100644 --- a/libavcodec/v4l2_buffers.c +++ b/libavcodec/v4l2_buffers.c @@ -30,6 +30,7 @@ #include #include "libavcodec/avcodec.h" #include "libavcodec/internal.h" +#include "libavutil/avassert.h" #include "libavutil/hwcontext.h" #include "v4l2_context.h" #include "v4l2_buffers.h" @@ -446,6 +447,31 @@ int ff_v4l2_buffer_buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) av_frame_unref(frame); + if (!s->device_ref) { + AVHWDeviceContext *dev; + AVDRMDeviceContext *drm_dev; + s->device_ref = av_hwdevice_ctx_alloc(AV_HWDEVICE_TYPE_DRM); + av_assert0(s->device_ref); + dev = (AVHWDeviceContext*)s->device_ref->data; + drm_dev = dev->hwctx; + // V4L2 is not associated with any DRM device. + drm_dev->fd = -1; + ret = av_hwdevice_ctx_init(s->device_ref); + av_assert0(ret == 0); + } + if (!s->frames_ref) { + AVHWFramesContext *frames; + s->frames_ref = av_hwframe_ctx_alloc(s->device_ref); + av_assert0(s->frames_ref); + frames = (AVHWFramesContext*)s->frames_ref->data; + frames->width = s->output.width; + frames->height = s->output.height; + frames->format = AV_PIX_FMT_DRM_PRIME; + frames->sw_format = avbuf->context->av_pix_fmt; + ret = av_hwframe_ctx_init(s->frames_ref); + av_assert0(ret == 0); + } + if (buf_to_m2mctx(avbuf)->output_drm) { /* 1. get references to the actual data */ ret = v4l2_buf_to_bufref_drm(avbuf, &frame->buf[0]); @@ -454,6 +480,9 @@ int ff_v4l2_buffer_buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf) frame->data[0] = (uint8_t *) v4l2_get_drm_frame(avbuf); frame->format = AV_PIX_FMT_DRM_PRIME; + + frame->hw_frames_ctx = av_buffer_ref(s->frames_ref); + av_assert0(frame->hw_frames_ctx); } else { /* 1. get references to the actual data */ for (i = 0; i < avbuf->num_planes; i++) { diff --git a/libavcodec/v4l2_m2m.h b/libavcodec/v4l2_m2m.h index 9ac5a24..7b87db6 100644 --- a/libavcodec/v4l2_m2m.h +++ b/libavcodec/v4l2_m2m.h @@ -62,6 +62,9 @@ typedef struct V4L2m2mContext { /* generate DRM frames */ int output_drm; + + AVBufferRef *device_ref; + AVBufferRef *frames_ref; } V4L2m2mContext; typedef struct V4L2m2mPriv