From patchwork Sat Feb 3 02:50:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rodger Combs X-Patchwork-Id: 7498 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.150.1 with SMTP id c1csp199087jai; Fri, 2 Feb 2018 18:56:08 -0800 (PST) X-Google-Smtp-Source: AH8x226bbHFuvK4JvEDhbxaTjCgV4xvP7El4juLvOKuBTjXKIm4GBKySSVYzOT+K/zGEqyR9MWGP X-Received: by 10.28.112.15 with SMTP id l15mr31157130wmc.119.1517626568452; Fri, 02 Feb 2018 18:56:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517626568; cv=none; d=google.com; s=arc-20160816; b=seKgVBGzXB9jLgTCZvl1hhHaMFC0JIQB+L9vN4ZRzinVtef2L6p+05JJNy4Be1q/Ws 1TxMmOszdmaXJJFw6BXWNbWBf47Duu2qkkzOdDWgeBMm7aTY0j9egWMInDmaPVaFYClM xCxuS4E2Gs7++yPmKpi8Z7QyCM+niCHya5we75CR8dF/rQw1v42YECZYV8dQdzeDJRdD K56lfsNnyES/2+P90580OXJDg27ljipiBsBNLrZqTcVNukHp/wl2ORAyIM7tL9SRZdyk 17gakrhj/XlRrFC8C4Rh1cPwNAIu4fLfgAv0JI8itAlh/ON5UseyFDWKievjy2zhTdCO mG4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=Nf8kf61Yw50CtHZWzLuEaxLrsorfLuvfPYbVGxVwYYE=; b=uk9MXVjBSOHFlWvJpRjfIyq3FvyuGlor+YIPGcfvi1lfaSW9s6p5UZ/s1ohKuYeeqq Q0qSaKviO77DaidV9niMBvVFWRgYhHtlg8l8uDjJ7EZcFxRrzksVJBPfMSiOUbHNGQSq DN5i1JYSYSTq+okWABJADRUIZfNonRxltvXFIGSXvf4UNphbmjvU9ullpYJiIkHVJJvd PBo/PM4yfb0GioHyKtcv++H+esv+FpFavHyHPyUqoim1eHzHjG1pqiK2yYbl7vz+iMTw iF1qNEk3mCappvzgfKEI3sYJW94JJfXdbER4b+mjUFphvOJoIKnqIQnUbh8/wFKNxWhw 6v6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=OE5ONg6l; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b68si248774wrd.87.2018.02.02.18.56.08; Fri, 02 Feb 2018 18:56:08 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=OE5ONg6l; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3092568974E; Sat, 3 Feb 2018 04:56:00 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-io0-f196.google.com (mail-io0-f196.google.com [209.85.223.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8E7BA6891D0 for ; Sat, 3 Feb 2018 04:55:53 +0200 (EET) Received: by mail-io0-f196.google.com with SMTP id z6so24932704iob.11 for ; Fri, 02 Feb 2018 18:56:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=IEmtS6rJz2edf18Eivcigbdi5vYlRRwCwuPJgwjMMO4=; b=OE5ONg6lF9P6KnedfWEguZeUoT3OQXq/woqBz/sf8xCBgRd2yhIG4yNOTwhmiUHLHp C7VsJMUC5u6tcV2mCAWS/41JrH158e96UGmgUGUOiSH0Hi2+Uugh2OdzmmBd4kuuK86j i9eb53dqzy2HRWPz8AFuSaxrL0D+Omzbmli6M79n0dNoR7B1lvb0TFJx73eFYST+aaSt f2medbPN7JS+V9ZWyFMFEQncRWeJobo5jMSTvpxIxeao9yM2PjrLj6gjKwFsNUE5fihe OjLnpO4lASHtd/TmjkxfKvxDNTNBj6njJwptuT6DRryiXPXde1Vn/PRQvmM9WRIx+gkC zRUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=IEmtS6rJz2edf18Eivcigbdi5vYlRRwCwuPJgwjMMO4=; b=EME2jqwdYaHnFoirA8eeq+nyKCdtNShVie32nlpPQ50ITQGviWDCVNpqqxwQCmoxXt RwgpLf76uKo6/vyE55lysxIz0sUSOM9s+51nDpzVRQSasfMZH7y/3sEeWCrDB2dievrS OGSZ168TeIEoShpHp8OSYe2bUICe/hA1b64U+Y3xO3WrSKOxz6hAdNhDo7PgHo0L+loF aXQ0DrHZb5G5eAB0O3bGlX5QzLUDtU6Y/9iw9zh9MMZV5ltZPI+8a6lTBeA9hXWbYSgc nvei1DzwFQ8B54nwfaNHgxEQ79tN9T33YrYmKBla0jHtXgiNHjDE/kJwFnVGg+ovmcHr PhMA== X-Gm-Message-State: AKwxytcrYnQBotx9h2FffAkvht9ENPuu0KNo3dToTe4deP0yiCWWYR/T RVudL9fsHQ2SrQIo9KzljhTe9bzK X-Received: by 10.107.159.74 with SMTP id i71mr43945263ioe.300.1517626222709; Fri, 02 Feb 2018 18:50:22 -0800 (PST) Received: from MacBook-Pro.localdomain ([71.201.155.37]) by smtp.gmail.com with ESMTPSA id t123sm1965730itb.14.2018.02.02.18.50.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 02 Feb 2018 18:50:21 -0800 (PST) From: Rodger Combs To: ffmpeg-devel@ffmpeg.org Date: Fri, 2 Feb 2018 20:50:13 -0600 Message-Id: <20180203025013.16470-1-rodger.combs@gmail.com> X-Mailer: git-send-email 2.16.1 Subject: [FFmpeg-devel] [PATCH] lavc/videotoolbox: fix threaded decoding X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" AVHWAccel.end_frame can run on a worker thread. The assumption of the frame threading code is that the worker thread will change the AVFrame image data, not the AVFrame fields. So the AVFrame fields are not synced back to the main thread. But this breaks videotoolbox due to its special requirements (everything else is fine). It actually wants to update AVFrame fields. The actual videotoolbox frame is now stored in the dummy AVBufferRef, so it mimics what happens in non-videotoolbox cases. (Changing the AVBufferRef contents is a bit like changing the image data.) The post_process callback copies that reference to the proper AVFrame field. Based on a patch by wm4. --- libavcodec/h264dec.c | 3 --- libavcodec/videotoolbox.c | 68 +++++++++++++++++++++++++++++++++++------------ libavcodec/vt_internal.h | 1 - 3 files changed, 51 insertions(+), 21 deletions(-) diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c index 8c9c6d9f3b..7494c7a8f2 100644 --- a/libavcodec/h264dec.c +++ b/libavcodec/h264dec.c @@ -838,9 +838,6 @@ static int output_frame(H264Context *h, AVFrame *dst, H264Picture *srcp) AVFrame *src = srcp->f; int ret; - if (src->format == AV_PIX_FMT_VIDEOTOOLBOX && src->buf[0]->size == 1) - return AVERROR_INVALIDDATA; - ret = av_frame_ref(dst, src); if (ret < 0) return ret; diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index afec1edf3f..f82c31c5df 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -45,8 +45,10 @@ enum { kCMVideoCodecType_HEVC = 'hvc1' }; static void videotoolbox_buffer_release(void *opaque, uint8_t *data) { - CVPixelBufferRef cv_buffer = (CVImageBufferRef)data; + CVPixelBufferRef cv_buffer = *(CVPixelBufferRef *)data; CVPixelBufferRelease(cv_buffer); + + av_free(data); } static int videotoolbox_buffer_copy(VTContext *vtctx, @@ -69,19 +71,47 @@ static int videotoolbox_buffer_copy(VTContext *vtctx, return 0; } +static int videotoolbox_postproc_frame(void *avctx, AVFrame *frame) +{ + CVPixelBufferRef ref = *(CVPixelBufferRef *)frame->buf[0]->data; + + if (!ref) { + av_log(avctx, AV_LOG_ERROR, "No frame decoded?\n"); + av_frame_unref(frame); + return AVERROR_EXTERNAL; + } + + frame->data[3] = (uint8_t*)ref; + + return 0; +} + int ff_videotoolbox_alloc_frame(AVCodecContext *avctx, AVFrame *frame) { + size_t size = sizeof(CVPixelBufferRef); + uint8_t *data = NULL; + AVBufferRef *buf = NULL; int ret = ff_attach_decode_data(frame); + FrameDecodeData *fdd; if (ret < 0) return ret; + data = av_mallocz(size); + if (!data) + return AVERROR(ENOMEM); + buf = av_buffer_create(data, size, videotoolbox_buffer_release, NULL, 0); + if (!buf) { + av_freep(&data); + return AVERROR(ENOMEM); + } + frame->buf[0] = buf; + + fdd = (FrameDecodeData*)frame->private_ref->data; + fdd->post_process = videotoolbox_postproc_frame; + frame->width = avctx->width; frame->height = avctx->height; frame->format = avctx->pix_fmt; - frame->buf[0] = av_buffer_alloc(1); - - if (!frame->buf[0]) - return AVERROR(ENOMEM); return 0; } @@ -285,20 +315,24 @@ CFDataRef ff_videotoolbox_hvcc_extradata_create(AVCodecContext *avctx) return data; } -int ff_videotoolbox_buffer_create(VTContext *vtctx, AVFrame *frame) +static int videotoolbox_set_frame(AVCodecContext *avctx, AVFrame *frame) { - av_buffer_unref(&frame->buf[0]); - - frame->buf[0] = av_buffer_create((uint8_t*)vtctx->frame, - sizeof(vtctx->frame), - videotoolbox_buffer_release, - NULL, - AV_BUFFER_FLAG_READONLY); - if (!frame->buf[0]) { - return AVERROR(ENOMEM); + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + if (!frame->buf[0] || frame->data[3]) { + av_log(avctx, AV_LOG_ERROR, "videotoolbox: invalid state\n"); + av_frame_unref(frame); + return AVERROR_EXTERNAL; + } + + CVPixelBufferRef *ref = (CVPixelBufferRef *)frame->buf[0]->data; + + if (*ref) { + av_log(avctx, AV_LOG_ERROR, "videotoolbox: frame already set?\n"); + av_frame_unref(frame); + return AVERROR_EXTERNAL; } - frame->data[3] = (uint8_t*)vtctx->frame; + *ref = vtctx->frame; vtctx->frame = NULL; return 0; @@ -406,7 +440,7 @@ static int videotoolbox_buffer_create(AVCodecContext *avctx, AVFrame *frame) AVHWFramesContext *cached_frames; int ret; - ret = ff_videotoolbox_buffer_create(vtctx, frame); + ret = videotoolbox_set_frame(avctx, frame); if (ret < 0) return ret; diff --git a/libavcodec/vt_internal.h b/libavcodec/vt_internal.h index 929fe423fc..fb64735b8c 100644 --- a/libavcodec/vt_internal.h +++ b/libavcodec/vt_internal.h @@ -46,7 +46,6 @@ typedef struct VTContext { int ff_videotoolbox_alloc_frame(AVCodecContext *avctx, AVFrame *frame); int ff_videotoolbox_uninit(AVCodecContext *avctx); -int ff_videotoolbox_buffer_create(VTContext *vtctx, AVFrame *frame); int ff_videotoolbox_h264_start_frame(AVCodecContext *avctx, const uint8_t *buffer, uint32_t size);