From patchwork Wed May 25 17:15:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 35928 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp1180712pzj; Wed, 25 May 2022 10:15:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzu3/B67AyRkGD47xbB7m+79n0zIkVdomsv1Wb2PnPYT4juFSupyjhoS9MjBUvYGl4DaUJv X-Received: by 2002:a17:907:7ea9:b0:6fe:d412:ec2c with SMTP id qb41-20020a1709077ea900b006fed412ec2cmr16356123ejc.613.1653498944639; Wed, 25 May 2022 10:15:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653498944; cv=none; d=google.com; s=arc-20160816; b=0IGvKEeVoiFF6Ku1tJ0jdGaIUhSOKLRtd2h7ayy08kPNZpyFf04/qKRnsb7kAm2z4C O8K9NTgyievcxJBV43HdSZEvIZ6QtmOBu/1bDS8g99ezIjepuAe1Et0CVoeMSgAA2aMb IuybcgkJZZhYfUOhsKzkaB0usrYTyMb2lHLt+1Ml+0UjSG4k9kH97S935ezUE9T1ZcM4 +4LQ4uHbtNGpKKmscFbXzi8rNFKgtYLzJMA5XkENxNB/c0VSQj0ozlNV6+8qebMAfvhj vZkpVswjvE/4rCkyi1IoHYtefbDAeGXXMv+PMqnbsSpG4IUygKbipYEFUjghvjArtjvL vrug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=BUoXGNbO8kTZ5NFJt693jLrTcHij1dyGBi3NqTNhoX8=; b=oof/C4E9gpglv88TelIoIhokBNEXp/Ax61gKLEaKJ8jfT/h72mrBbfc6cWTRcYYD4b zeb3QTJA5W7E2Zoya6pNeFTJtJ3a5HdNVPt7o8X9Xy1hAQJiR4yEhfAVcmu5kf6yt91s ME+QFZ32tmDBXgJVXy1okkEHVUHfpHq1AIJjTc8L2xrOPjxR86pKzqPlw21VffHSt7QC p8bmA99XZHiAmvD+LgTuT1RbI42yvltfrJD5M9QoHwA5kWJ4Z3bC6bINqneRPcWlV/9a qm7zHdQsKqnU0YaO6FmTk8u7f+yZrvivQy9f4WcJOscl0ZHFM0hwlg/64fOCNIxquwwF 9DGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=pm9wix6H; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id p9-20020a056402154900b00427f1a23067si19550095edx.388.2022.05.25.10.15.43; Wed, 25 May 2022 10:15:44 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=pm9wix6H; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 505B668B4D4; Wed, 25 May 2022 20:15:40 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 404EB68B4C3 for ; Wed, 25 May 2022 20:15:33 +0300 (EEST) Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-e93bbb54f9so26744159fac.12 for ; Wed, 25 May 2022 10:15:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=9B0SIHShbuKdqYuCzlY07jmMzwQJ3QPWFFrCToeavxs=; b=pm9wix6H4D9dtJa+vYHxHNMn7IrLQZqlMmuakHXT6akIm2wyq1XMfWm4+0rRnuTr4s gOUSdw7mgPhXIcyqLhpw5DWSRRLgien9DPKaueWjyKclbOEcbhQgfQ1RFgZGGFWbp5fn Vz51iBtI4WlFfNJGw2dhGnLQlQ8w2JBJ2ZzkeaguSv6Bzb6jTtEvCsjnKraTTsrw7dZW X8WqtwK9X7ADSXdgiCRmb8xMycOq5Y//K1b9ak339EmhM6P7Izpq7MqPvi0Nk79yYEhU Fsfj2G/byrjgvMiWEvxaXXsu6/44yAZzmBqkjWs8bcEVwGoupVj6OmsT3BeCgg7M0Osx QTmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=9B0SIHShbuKdqYuCzlY07jmMzwQJ3QPWFFrCToeavxs=; b=zKYIiSkAR5QNIi6FdsWD2ueHcr3Kfitvo7I39dQJY6odHTIzfumeVcNXh+I1COC7tW 0xpi1rOV2Tg5Fj4NQoSNiPC23H2V7rF3t0Qwh5mYDpHgNxNvbMlBI1/NcwneiyCTfH05 EvOa4wPRpgOuOLwce5zem6tH4UPAZVH/NGsyyJo6j93gF6jSyzgs4xfFUDSKyvAt2wvg v3EYIrEcmgClMEQc8H/4+7KqaM85+FCesyGU5ZN1tsYTrg2sNMFW1+ahLjBbvUtWP5E9 thDfoUP7V8qbvnfNqfz5Og558EqJBh+Kyd6Hj4htnd5dVzx8NS8wsouUAFJ3y6WqSeSh uYAQ== X-Gm-Message-State: AOAM5320KfwpeRB0IdeEe3iJ1VhXFj0QLri+6lDuGWpWcU8gURGlylKx 7bYv8uuKOhutj8tEh8illP8pzdGBH2U= X-Received: by 2002:a05:6870:4596:b0:da:b3f:2b1d with SMTP id y22-20020a056870459600b000da0b3f2b1dmr6693256oao.188.1653498931396; Wed, 25 May 2022 10:15:31 -0700 (PDT) Received: from localhost.localdomain ([186.136.131.95]) by smtp.gmail.com with ESMTPSA id g6-20020aca3906000000b00325cda1ffb5sm6503185oia.52.2022.05.25.10.15.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 10:15:30 -0700 (PDT) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Wed, 25 May 2022 14:15:16 -0300 Message-Id: <20220525171516.64893-1-jamrial@gmail.com> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avutil/avframe: fix channel layout checks in av_frame_copy() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: NPfQ8dSFbdS2 Normally, both the source and dest frame would have only the old API fields set, only the new API fields set, or both set. But in some cases, like when calling av_frame_ref() using a non reference counted source frame where only the old channel layout API fields were populated, the result would be the dst frame having both the new and old fields populated. This commit takes this into account and fixes the checks by calling av_channel_layout_compare() only if the source frame has the new API fields set, and doing sanity checks for the source frame old API fields if the new ones are not set. Signed-off-by: James Almer --- libavutil/frame.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/libavutil/frame.c b/libavutil/frame.c index fbb869fffa..0da4bb4768 100644 --- a/libavutil/frame.c +++ b/libavutil/frame.c @@ -727,19 +727,30 @@ static int frame_copy_audio(AVFrame *dst, const AVFrame *src) #if FF_API_OLD_CHANNEL_LAYOUT FF_DISABLE_DEPRECATION_WARNINGS + int channel_layout = dst->ch_layout.order == AV_CHANNEL_ORDER_NATIVE ? + dst->ch_layout.u.mask : 0; if (!channels) { - if (dst->channels != src->channels || - dst->channel_layout != src->channel_layout) - return AVERROR(EINVAL); channels = dst->channels; + channel_layout = dst->channel_layout; planes = planar ? channels : 1; + } + if (!src->ch_layout.nb_channels) { + if (channels != src->channels || + channel_layout != src->channel_layout) + return AVERROR(EINVAL); CHECK_CHANNELS_CONSISTENCY(src); } FF_ENABLE_DEPRECATION_WARNINGS #endif if (dst->nb_samples != src->nb_samples || +#if FF_API_OLD_CHANNEL_LAYOUT + (av_channel_layout_check(&src->ch_layout) && +#endif av_channel_layout_compare(&dst->ch_layout, &src->ch_layout)) +#if FF_API_OLD_CHANNEL_LAYOUT + ) +#endif return AVERROR(EINVAL); for (i = 0; i < planes; i++) @@ -763,7 +774,7 @@ FF_DISABLE_DEPRECATION_WARNINGS else if (dst->nb_samples > 0 && (av_channel_layout_check(&dst->ch_layout) #if FF_API_OLD_CHANNEL_LAYOUT - || dst->channel_layout || dst->channels + || dst->channels > 0 #endif )) return frame_copy_audio(dst, src);