From patchwork Fri Dec 17 00:12:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: rcombs X-Patchwork-Id: 32671 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp1001809iog; Thu, 16 Dec 2021 16:13:01 -0800 (PST) X-Google-Smtp-Source: ABdhPJyqvzktSCnF1h03CFAt25wk+phoS4Ibq+x769e06YsdAsQDP8Oc9ARZbzKWZgcsbpBCrTJD X-Received: by 2002:a05:6402:c0a:: with SMTP id co10mr362703edb.295.1639699981039; Thu, 16 Dec 2021 16:13:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1639699981; cv=none; d=google.com; s=arc-20160816; b=VnJ/XC4qLKGabxdScvm55HeyPM6LN4ZdQcfwXyFwJ58KEEOO9oYyuyEFTWNCpVf+qn LPhKsDDaQRq9hARj8X5EFB8+6pfUbAtRFOVCDo8rlivy4dh/5MXD4QC2PEusvV2O/VVi m6A3/jQGUc3SvS+AEkUlm7xiKVEL1LdAp4dOnU9SnJBpsQHdh5u30ePcbPHGD+zAFrJ+ JUZ9vKxei9/RKAPoUT3gdntr9fa1/qaG+QcyvcxS9ZS8bV2870WuZhNsXrkHGgJlCtvn 2+w5zgLZhkZeYyFNmnRIq2DxsEB+vzWs7I/BUE8Nwn1C5of6/Kz/+Ec2J3Z8mgF+RSRE eNUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=Xafq2m9x3KuJQx0xF0lZm2s/j8GMT3i7smqtbSAS9KY=; b=Y44A833t0fMKNEMBjJgQPNUS0wIS82dSvmYY0uC530w9W+o+wNkuoWCpB6BsMQmsvn Ete29GkDlq1COVDlQAgK7TuXbrT+rmjA8OLLzO5P1KwFloQ+p45aO7WCSUcS/9GQmfy2 yzR7KIcwurkN73EPjh2X21mDCfkMq+ozfyD/jcdU3rc+jXa3BDuSUB/G13+Dpf4yTMrF Z6sTHlu3h7FKw22KaCDcQthaBzrFKmJuyhoBRxUPak51bZGTY0/v7ZgRSHkwKgBIDQMv Ser7Ubzq713/mrEAEGr6g2a98TvdkP+ZHJy2eVHzdac1Rzw0ozMvmuQV5x7zlYdS4KV2 cGMA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rcombs.me header.s=google header.b=c4rsUJbH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=rcombs.me Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d3si4747600edo.485.2021.12.16.16.13.00; Thu, 16 Dec 2021 16:13:01 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rcombs.me header.s=google header.b=c4rsUJbH; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=rcombs.me Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4028468AFD4; Fri, 17 Dec 2021 02:12:32 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D5B1968A786 for ; Fri, 17 Dec 2021 02:12:22 +0200 (EET) Received: by mail-io1-f52.google.com with SMTP id x10so704181ioj.9 for ; Thu, 16 Dec 2021 16:12:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rcombs.me; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=r7VF92SVjudlxE7DN4zIictBRpLo4pxVucK4VDOjJ5U=; b=c4rsUJbHwAbu0gpUptAxma6VYPDOZcIbCkALY+l/JjJt/N8nZDChvLVT/+BwtBRGYQ G2l2rDhEeq5Y73yUGziyIUCGq6hYvQol7Fw4p9On30P9hTg2rRuqJrr5Q72JZ9WGXkHz gOxbjk1Akv6ljHc4P5LhCvoqAABSBxfMdVGpH3Ph9ijIotUuv0wB8MDizMsXT/94gOCV E/L4Vae9E1mt/kO6ey8XO1unxZQzPhid5PDPKLaz3knRHLi5T7HI/nq6+1Vvn+0PGwy4 9S/zUEha4jyliVkt4sbfiNZVTCR+wJcgR2ZhCgXQd0pm+mvW9HU5Epv3pXY1qJVxFNq1 481g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r7VF92SVjudlxE7DN4zIictBRpLo4pxVucK4VDOjJ5U=; b=NSSA/n1GiWWs8D9PFl2JnJW0pQyyAyEcbIx1yPUGtdDoV3LDJAfuO1KkFx2rXZ2JcX 4FZmVANgLwIoe8LKNJQENVLN5lrA5h7B0btZlp8RXiflloF07TvivQu4WobgWDKcOC/q ts3slDqK0r3TuSlv0dFi6yYLsuoalhHvXqpNe/+EvS7L8zaD+QdtJkDpKf+A/GlpRtcW /ClxvIZpuub9gScBUJJFiS02QELy5zm+fhvjiRe7hisoabasc+8XYpWUMr+KBscTCJO0 cpczgLr0lJmjRDen291ZQgIRLhyImzqX4zyppaLEgNtzOwRASenZycgacJg5mvkSIYKk qxOw== X-Gm-Message-State: AOAM532yBZbs1mVi7qxlsYqqwvhtBVdHr+efThJCLTwvC3fRAXEBc6dN vQf8ua3q0kijwCs3V6KdCe2Y9cbsF/eem60= X-Received: by 2002:a05:6638:33a6:: with SMTP id h38mr238795jav.151.1639699941217; Thu, 16 Dec 2021 16:12:21 -0800 (PST) Received: from rcombs-mbp.localdomain ([2601:243:2000:5ac:1163:bda5:7473:fab3]) by smtp.gmail.com with ESMTPSA id z13sm1571257iln.43.2021.12.16.16.12.20 for (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 16 Dec 2021 16:12:20 -0800 (PST) From: rcombs To: ffmpeg-devel@ffmpeg.org Date: Thu, 16 Dec 2021 18:12:08 -0600 Message-Id: <20211217001215.75135-4-rcombs@rcombs.me> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211217001215.75135-1-rcombs@rcombs.me> References: <20211217001215.75135-1-rcombs@rcombs.me> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 03/10] lavc/videotoolboxenc: detect alpha more generically X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OtYbNwESg5rS Fixes support for alpha in hardware input frames --- libavcodec/videotoolboxenc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c index 9857e55fdc..fa8f717a6c 100644 --- a/libavcodec/videotoolboxenc.c +++ b/libavcodec/videotoolboxenc.c @@ -406,14 +406,14 @@ static int count_nalus(size_t length_code_size, return 0; } -static CMVideoCodecType get_cm_codec_type(enum AVCodecID id, - enum AVPixelFormat fmt, +static CMVideoCodecType get_cm_codec_type(AVCodecContext *avctx, double alpha_quality) { - switch (id) { + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(avctx->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX ? avctx->sw_pix_fmt : avctx->pix_fmt); + switch (avctx->codec_id) { case AV_CODEC_ID_H264: return kCMVideoCodecType_H264; case AV_CODEC_ID_HEVC: - if (fmt == AV_PIX_FMT_BGRA && alpha_quality > 0.0) { + if (desc && (desc->flags & AV_PIX_FMT_FLAG_ALPHA) && alpha_quality > 0.0) { return kCMVideoCodecType_HEVCWithAlpha; } return kCMVideoCodecType_HEVC; @@ -1376,7 +1376,7 @@ static int vtenc_configure_encoder(AVCodecContext *avctx) CFNumberRef gamma_level = NULL; int status; - codec_type = get_cm_codec_type(avctx->codec_id, avctx->pix_fmt, vtctx->alpha_quality); + codec_type = get_cm_codec_type(avctx, vtctx->alpha_quality); if (!codec_type) { av_log(avctx, AV_LOG_ERROR, "Error: no mapping for AVCodecID %d\n", avctx->codec_id); return AVERROR(EINVAL);