From patchwork Sat Jul 4 17:02:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: hanishkvc X-Patchwork-Id: 20805 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 1E2BF44AB0F for ; Sat, 4 Jul 2020 20:35:14 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E499868B30E; Sat, 4 Jul 2020 20:35:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f196.google.com (mail-oi1-f196.google.com [209.85.167.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 729E368B256 for ; Sat, 4 Jul 2020 20:35:07 +0300 (EEST) Received: by mail-oi1-f196.google.com with SMTP id k6so25072083oij.11 for ; Sat, 04 Jul 2020 10:35:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=6L+f1kBZE1AeasmHZGG9n6Pb7vF/oGhrsoTsU1+4vzU=; b=KZEOJ2KG0cDE3LA+kGy4ix2Kv+w8vataX3XS5WKzU2cJ/U+Gsk10dJa+y81lEzTmIe h+kUq625hHNHt6HZ2XMRLJjQfU/JyQnskBJHDm4cp2HTwtkoV9DkCllMjJMkiWQBdQaX KHo5PG5DQYYSYMcrJioysr63O7ITg+A5RzzKmoD53ssgGZphxLPoBWIyf0xuJb6mIsmK 8RL45lQP2KS+jhGLK7Loi6oJGLecTGgp1szPePSpuFtwzcPwoiLO6rLCmm0h9dWDQpv3 GODKqYBZwJaedwL9GmPG+oJqv4k6wfN2TGhFTxzciWpT/xxmq12Pc5U9jkuMQXeqQrlI A6CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=6L+f1kBZE1AeasmHZGG9n6Pb7vF/oGhrsoTsU1+4vzU=; b=BqAVuN70nFbb4fxF76NxcJIY2RYvSyYr/wEifWXQnWQ2JLKxJkfB8prakaGkSfyfzJ FjpqKVbbLwNv8db+isBVnk/D4LNE7OU6dTzmx/xWnjBhZR5COqiJYPFMazGfbNdG+tK2 8rXmHfaFtSugWJgItp105Rq/7ZHc5uke/oQTvn9gTlGIsyZmziMkCTmdThjOQtSaHdDh Wz41YmU+b6cpQW3RokB0BqxMk0KgBzdyP7Z/doYK+8WPpOx/K38tYmJgdCUurTObIiqf PvUd3Go/RhNS2OvroGqbc11YbXHU1TlnryDJg8ktYdY67Ni9n9oC4DoykEqJ6OCPfXcQ DKkA== X-Gm-Message-State: AOAM531T9dDUauK1DIOu6iPJFNrXVRynSEFnLEZLSNBRvMIogSefaMzu BHm/JkV9TsfkSl8Dk71h3iM7iH/q X-Google-Smtp-Source: ABdhPJye4yLH+XzXDQAGSI8VaC2k39AYH47JzjxGJv7KSLItpivokViTh7O8E0d9AdcyCt68UhAgBQ== X-Received: by 2002:a17:90b:154c:: with SMTP id ig12mr37792035pjb.54.1593882205986; Sat, 04 Jul 2020 10:03:25 -0700 (PDT) Received: from localhost.localdomain ([122.179.70.80]) by smtp.gmail.com with ESMTPSA id f18sm14552050pgv.84.2020.07.04.10.03.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 04 Jul 2020 10:03:25 -0700 (PDT) From: hanishkvc To: ffmpeg-devel@ffmpeg.org Date: Sat, 4 Jul 2020 22:32:25 +0530 Message-Id: <20200704170225.125426-1-hanishkvc@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v06.01] hwcontext_drm: check pixformats before detile X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: hanishkvc Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Verify that the source and destination pixformats are supported by fbtile before detiling the source into destination. NOTE: This patch is relative to my v06 patch set. --- libavutil/hwcontext_drm.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/libavutil/hwcontext_drm.c b/libavutil/hwcontext_drm.c index bd74b3f13d..b80ebc5b2d 100644 --- a/libavutil/hwcontext_drm.c +++ b/libavutil/hwcontext_drm.c @@ -196,17 +196,20 @@ static int drm_transfer_with_detile(const AVFrame *hwAVFrame, AVFrame *dst, cons int err = 0; if (hwAVFrame->format == AV_PIX_FMT_DRM_PRIME) { - AVDRMFrameDescriptor *drmFrame = (AVDRMFrameDescriptor*)hwAVFrame->data[0]; - uint64_t formatModifier = drmFrame->objects[0].format_modifier; - if (formatModifier != DRM_FORMAT_MOD_LINEAR) { - err = detile_this(TILE_AUTO, formatModifier, dst->width, dst->height, - dst->data[0], dst->linesize[0], - src->data[0], src->linesize[0], 4); - if (!err) { + int ok = fbtile_checkpixformats(src->format, dst->format); + if (ok) { + AVDRMFrameDescriptor *drmFrame = (AVDRMFrameDescriptor*)hwAVFrame->data[0]; + uint64_t formatModifier = drmFrame->objects[0].format_modifier; + if (formatModifier != DRM_FORMAT_MOD_LINEAR) { + err = detile_this(TILE_AUTO, formatModifier, dst->width, dst->height, + dst->data[0], dst->linesize[0], + src->data[0], src->linesize[0], 4); + if (!err) { #if HWCTXDRM_SYNCRELATED_FORMATMODIFIER - drmFrame->objects[0].format_modifier = DRM_FORMAT_MOD_LINEAR; + drmFrame->objects[0].format_modifier = DRM_FORMAT_MOD_LINEAR; #endif - return 0; + return 0; + } } } }