From patchwork Sat Apr 14 15:53:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Thompson X-Patchwork-Id: 8443 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.1.70 with SMTP id c67csp1941320jad; Sat, 14 Apr 2018 08:59:53 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/4VCXIlQrEXVB2Dk4NwMPSYiEXBGttujdLRzMOtc6Pb7tysdd+g9tOtEDEZHuff+FRHwsa X-Received: by 10.223.226.66 with SMTP id n2mr6706292wri.228.1523721593514; Sat, 14 Apr 2018 08:59:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523721593; cv=none; d=google.com; s=arc-20160816; b=XCKoPAeCWg1SrP9sjrdpnI4kCZyP2a1p+160j6HLT9oy4FYkoUXNqB++OD8YJiI0K9 y9+7YpjfzFKZ2O/ZqLe15RtEuvJvx7fNuLhnl2p5vwLCtLzMlv+gP/PobovbDqLYs/rn qknnlLQkNn9Azo3eldE7PjXmbTDYK824LVGWMrxgX9lgVCU1c37ftanhz5qRd75Kfm+v 1BAKWN1ksLiWSVPx07mcgboK34unWXbaVWGQFQqRyKLuI1uyhAFKjgxLD6kjLnrGtQrs bKT42Zx9G68sWgQpmD7sjBcVe5XH6tWyDm7Py5I+sOIdiORt+/+Jez8mm13e129rpeWG sqcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=QHKdvwfkR3i7GkVvE5Q6foG+dKrgFSXadysLfROgsZ8=; b=G+Ygz8tud91cTkglzZimQ3mVgeC+O/QeoP3N9I0zk5OEsiqSwdz0kpYIavw1RQgEAa LKAIYuGOT6Jv+OvDBRBPtDYbRJqmqlxr1Tdn602G/Tkhwq9hdQKmkTqmGCsqm2QiPuB3 U/+KYgo4MLV2xOKUo2xRLnEOH+XtuX8stihYkVeEteJqxrRFcCRpI+YvK53MAPqRWzzK ehHrBcyBPsAIGvV0A4bgyoPBdOrqF2NNS1U8BGlh4urFFbY1nJVx0gxVowrSkOEnXS2r jijGi1uQLTrYUyrbutznPx7ynSdkqFfcQOmKSo6KJetFajSgq/Wp4qd9efLIxiPDTfyL wsxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=GwDuS/qb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id n6si2138432wmh.72.2018.04.14.08.59.53; Sat, 14 Apr 2018 08:59:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=GwDuS/qb; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 63CA0689DDE; Sat, 14 Apr 2018 18:59:26 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com [209.85.128.193]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1EE21689228 for ; Sat, 14 Apr 2018 18:59:20 +0300 (EEST) Received: by mail-wr0-f193.google.com with SMTP id o3so15141137wri.2 for ; Sat, 14 Apr 2018 08:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jkqxz-net.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=Ki3d7LQmxCHubK/ia4ew/c7uk7PbI3WA78x7mskCCG4=; b=GwDuS/qbnoYGRYTj5sFLQPSGevDZQlL+dGXPyIzlszEzXBhoQnSIcih4g1fjVrplne CSTNd7/TFXHlirPLnIAFgfDToELWIaIVQGrru0ixtwp9z7oRcA6ZCkSGyxZm05YrWVhK tvBhmby9w94STG9zoik9E0Dk3vHG9f7YweePVrbY5P6PP/Z1elfHimsya1RV8F48opcA P5gurKUleX+dAV/2S7AdR7BFlrowqSS/sbO4Y63PhpfDRBpfe/eY+cyDMIkmeBDcFWVw fJhUOGl0XuZeCXbZGEgDyonEHr/GhwS/hyzbYniKkceBye1rSPTxKxRom5R6CpT725Dw W1vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=Ki3d7LQmxCHubK/ia4ew/c7uk7PbI3WA78x7mskCCG4=; b=gcRzQbB3UubJ7DXiE9ZbWq3R45FXitswkUkcCeUa6oYz2sYNhcNWRGBke7sSKRAKZ3 AyHamtdowY+Om5CruK+6ohaGGLM9Lr+E3vCf6ACntPGCoj/+pezPpdytzWQJkla0VhUf Q1TzXmZssZakEeOnWwHGdH2hRkR3wK2WMwh97WsDNMSbCiJVEsXmGSJnZbcq6kz620SC f56L2l65HiO9U6YSB0M498U+1unhf0KjYAADJvYobdMTspHOvDZBLEFRG9h9wNsLAybX nbdNcCPdnN5n+67Qm6QSnDupPhoa69e1FQ/Y5oVihEzeLpvSYLGe9c7FeZWGQOHzIpT+ pPBg== X-Gm-Message-State: ALQs6tCVGlPRaOnaNHLGc6VrwNJXvECTIRPcJzB3I9eSKEzMcQUr8qK/ ahxmr+6Hp1zUGjwrooWD+kXeZwGw X-Received: by 10.28.213.12 with SMTP id m12mr6287435wmg.98.1523721228910; Sat, 14 Apr 2018 08:53:48 -0700 (PDT) Received: from rywe.jkqxz.net (cpc91242-cmbg18-2-0-cust650.5-4.cable.virginm.net. [82.8.130.139]) by smtp.gmail.com with ESMTPSA id n49sm13651823wrn.50.2018.04.14.08.53.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 Apr 2018 08:53:47 -0700 (PDT) From: Mark Thompson To: ffmpeg-devel@ffmpeg.org Date: Sat, 14 Apr 2018 16:53:37 +0100 Message-Id: <20180414155339.22839-3-sw@jkqxz.net> X-Mailer: git-send-email 2.16.3 In-Reply-To: <20180414155339.22839-1-sw@jkqxz.net> References: <20180414155339.22839-1-sw@jkqxz.net> Subject: [FFmpeg-devel] [PATCH 3/5] amfenc: Ensure that the software format of hardware frames is valid X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Alexander Kravchenko Signed-off-by: Mark Thompson --- libavcodec/amfenc.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/libavcodec/amfenc.c b/libavcodec/amfenc.c index d1a28f13e2..b761cd74bf 100644 --- a/libavcodec/amfenc.c +++ b/libavcodec/amfenc.c @@ -68,7 +68,6 @@ static const FormatMap format_map[] = { AV_PIX_FMT_GRAY8, AMF_SURFACE_GRAY8 }, { AV_PIX_FMT_YUV420P, AMF_SURFACE_YUV420P }, { AV_PIX_FMT_YUYV422, AMF_SURFACE_YUY2 }, - { AV_PIX_FMT_D3D11, AMF_SURFACE_NV12 }, }; static enum AMF_SURFACE_FORMAT amf_av_to_amf_format(enum AVPixelFormat fmt) @@ -263,6 +262,7 @@ static int amf_init_encoder(AVCodecContext *avctx) AmfContext *ctx = avctx->priv_data; const wchar_t *codec_id = NULL; AMF_RESULT res = AMF_OK; + enum AVPixelFormat pix_fmt; switch (avctx->codec->id) { case AV_CODEC_ID_H264: @@ -276,8 +276,14 @@ static int amf_init_encoder(AVCodecContext *avctx) } AMF_RETURN_IF_FALSE(ctx, codec_id != NULL, AVERROR(EINVAL), "Codec %d is not supported\n", avctx->codec->id); - ctx->format = amf_av_to_amf_format(avctx->pix_fmt); - AMF_RETURN_IF_FALSE(ctx, ctx->format != AMF_SURFACE_UNKNOWN, AVERROR(EINVAL), "Format %d is not supported\n", avctx->pix_fmt); + if (ctx->hw_frames_ctx) + pix_fmt = ((AVHWFramesContext*)ctx->hw_frames_ctx->data)->sw_format; + else + pix_fmt = avctx->pix_fmt; + + ctx->format = amf_av_to_amf_format(pix_fmt); + AMF_RETURN_IF_FALSE(ctx, ctx->format != AMF_SURFACE_UNKNOWN, AVERROR(EINVAL), + "Format %s is not supported\n", av_get_pix_fmt_name(pix_fmt)); res = ctx->factory->pVtbl->CreateComponent(ctx->factory, ctx->context, codec_id, &ctx->encoder); AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_ENCODER_NOT_FOUND, "CreateComponent(%ls) failed with error %d\n", codec_id, res);