From patchwork Tue Sep 10 18:10:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Koushik Dutta X-Patchwork-Id: 51498 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:612c:41b1:b0:48e:c0f8:d0de with SMTP id le49csp584252vqb; Tue, 10 Sep 2024 11:11:19 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWSI/EYbPm1rCK4Xzgvv+EkYHiCvhL7LAkY+CRZnl2fQ6ro2sqQ/zrz3uqGmolNJrgtt/RtDmvlZ0oCRylij3Au@gmail.com X-Google-Smtp-Source: AGHT+IE1Tbmt2ttaweNTMDS69qGB1P7aK2gK04LOeSeS9t7zo2BKC2BUN6r18Qn5xxokE9t23GiP X-Received: by 2002:a2e:a542:0:b0:2f7:5c23:98fc with SMTP id 38308e7fff4ca-2f75c23991emr22131251fa.11.1725991879369; Tue, 10 Sep 2024 11:11:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1725991879; cv=none; d=google.com; s=arc-20240605; b=XxlqRgzgofmo4SCcC9VVLU8+tIq3alFfv3oc8jwayt5BP5hlLl9ym0+1TsPhZmwkIP CaX2yZf4/wRDtUq9SL+LbFGlZLIkvZQAEBaxLFOTFkmIz6LNqrPj4ZAp6imcJ9Ysh7D5 SjbSsKqSbezLISyj1NsP75e7Z1jEeF3TSAg8BwNiNTQ2JFl/i7APhZ0uG6Li3l73KWyd SBDlc80K6rOpT/TpxPSMGTFaj1FfVI8Zd013U0XLX1WLUnMiq9odFt7Dk4BiIF7aNObS JFeweDlryLWafxEKNjaIP7EEAHc2tmFecLiAsQi/7no5/E2Xx8eLECO6q1qM+sD8oJfW TtOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=PuFD74xKf9jsesj50wQvutIoK1VmODd62lxjPv9NgSA=; fh=vf1rgm8KpEMxcRKiIfYPC5TR7MMWdHSIxg51vjgwMpE=; b=I4vMzN8Ht7q2mS+PFXqlo+X3twCPgucaNdFNG7rvVAC8TCcdozNK2VxEGfVAWebp4T YZMsBawou14+Dne3fzuZjiq5lCJ0IsW3KT0jW7o+QSYmNkTkaLiWEyjgN/kDv6XO7SLk Yf4wA4732sCdPDG7vej+VWMhRt2nF8YzuAzDOQU5a25ntHPnf29/g+ITVCHWOejapeEQ l26BhtEqbWOxRi3TFbSC9x9ch89eVElTUL2xCrgAf0strtryZ6rwlcJJcm7PYJfB41Cp NpF2B8QwMFquqsaFK8PTJ4r3sP5vMS9vQarpaYjgPHsvlpJzHQpWkDdB2/ycos/7vzei +meg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=YXPDub0G; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 38308e7fff4ca-2f75c0b897esi23426181fa.638.2024.09.10.11.11.18; Tue, 10 Sep 2024 11:11:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=YXPDub0G; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CC44D68E0AF; Tue, 10 Sep 2024 21:11:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8E68B68DE63 for ; Tue, 10 Sep 2024 21:11:07 +0300 (EEST) Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-20688fbaeafso59644255ad.0 for ; Tue, 10 Sep 2024 11:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725991865; x=1726596665; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=WkdzRWa7iClx/3SfHY+PO/C9USqFSrwXVyTKzq4vcgk=; b=YXPDub0GV/phBUu3Y453m+tfkKhkvSxKIhzNbVuLJvM+lvapmV8M94kviQXle9LeIv aca/x5op3xfMmjuSewJoPNDBNKz8BUBZ2j49xy0d9SHoFX0Z75V9aAMv8IYYkcrCcamN Nvkx86wsAEfx5cXZnWHj+Kx7O0chdSEB3HMG/ncAmWi7UkmziQPKsFqAuywaYIHP18fg QvvgL3+f4cGHiNB8LwDWGAffreDrmrZpazc0ZtElk3dVD5UW5E51bjYLzgh9mXu9GUDM UmkIx44jMNCOQVtLKG3MKi+epusyf8MchGXxBr6HDURlNPwwS9bgSPaR+Rbi+luvJggW b78Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725991865; x=1726596665; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WkdzRWa7iClx/3SfHY+PO/C9USqFSrwXVyTKzq4vcgk=; b=C9swcwvzo/c4rIlJfN+eszn+cn9qPMuSX1MugFACPTsb/bnbqJwQ84JYeimfWH24TJ 8TzOVTVbS/Kd9NOu6HghGteJHzJBUb0o6MySQKRNnKx+/2AQaGDpN/vQ6LYVihB0d22j psFxZXWzdOoDl0yQm8WktsRLSNoBnnOLtQqaEA/DQJP1vju80d/P/18pf4RZpuCbGaNF 0LnzasB/O1795MqVV14PDEW5hGLgm1RsJx1X3ViALH5nFkrENt96E5yDRXn38YQf/Fuu i4uSnkOnGirP08n90CTU9ktSibDmpnoxyONDaBDMTvd76RR65cvM3qXQZJzVeuMTI70U OI4g== X-Gm-Message-State: AOJu0YzVaGxNEdQSxs5i9Z4YzeOmSKnGsMh3pB5OcJA4n3vLmg7PK/ID iN9cn7H5mjUupGWFPebv/UCfLQfiJL5SiyWkRydaUh5kjgu/jheGBZ3nY+BQ X-Received: by 2002:a17:902:f68e:b0:207:182c:8a52 with SMTP id d9443c01a7336-2074c79fa6amr18690735ad.58.1725991865236; Tue, 10 Sep 2024 11:11:05 -0700 (PDT) Received: from Koushik-MacStudio.tail05204.ts.net ([2001:559:76c:0:ed2e:ef7a:2bf7:c169]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20710e1d5c3sm51265025ad.59.2024.09.10.11.11.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 10 Sep 2024 11:11:04 -0700 (PDT) From: Koushik Dutta To: ffmpeg-devel@ffmpeg.org Date: Tue, 10 Sep 2024 11:10:55 -0700 Message-Id: <20240910181057.43453-1-koushd@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/3] scale_vt frame crop support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Koushik Dutta Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 2JAY4AVPh6gY The crop filter has no effect on scale_vt: -vf crop=100:100,scale_vt=300x300 Hardware frames (AV_PIX_FMT_FLAG_HWACCEL) are expected to use the crop_* properties, as seen in the implementation vf_crop.c. The current workaround is to hwdownload the full frame and perform the crop on CPU. --- libavfilter/vf_scale_vt.c | 50 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/libavfilter/vf_scale_vt.c b/libavfilter/vf_scale_vt.c index 05f4e7b797..3da46a6cd5 100644 --- a/libavfilter/vf_scale_vt.c +++ b/libavfilter/vf_scale_vt.c @@ -109,6 +109,8 @@ static av_cold int scale_vt_init(AVFilterContext *avctx) VTSessionSetProperty(s->transfer, kVTPixelTransferPropertyKey_DestinationYCbCrMatrix, value); } + VTSessionSetProperty(s->transfer, kVTPixelTransferPropertyKey_ScalingMode, kVTScalingMode_CropSourceToCleanAperture); + return 0; } @@ -132,6 +134,18 @@ static int scale_vt_filter_frame(AVFilterLink *link, AVFrame *in) CVPixelBufferRef src; CVPixelBufferRef dst; + int left; + int top; + int width; + int height; + CFNumberRef crop_width_num; + CFNumberRef crop_height_num; + CFNumberRef crop_offset_left_num; + CFNumberRef crop_offset_top_num; + const void *clean_aperture_keys[4]; + const void *source_clean_aperture_values[4]; + CFDictionaryRef source_clean_aperture; + AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); if (!out) { ret = AVERROR(ENOMEM); @@ -153,8 +167,43 @@ static int scale_vt_filter_frame(AVFilterLink *link, AVFrame *in) if (s->colour_matrix != AVCOL_SPC_UNSPECIFIED) out->colorspace = s->colour_matrix; + width = (in->width - in->crop_right) - in->crop_left; + height = (in->height - in->crop_bottom) - in->crop_top; + // The crop offsets are relative to the center of the frame. + // the crop width and crop height are relative to the center of the crop rect, not top left as normal. + left = in->crop_left - in->width / 2 + width / 2; + top = in->crop_top - in->height / 2 + height / 2; + crop_width_num = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &width); + crop_height_num = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &height); + crop_offset_left_num = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &left); + crop_offset_top_num = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &top); + + clean_aperture_keys[0] = kCVImageBufferCleanApertureWidthKey; + clean_aperture_keys[1] = kCVImageBufferCleanApertureHeightKey; + clean_aperture_keys[2] = kCVImageBufferCleanApertureHorizontalOffsetKey; + clean_aperture_keys[3] = kCVImageBufferCleanApertureVerticalOffsetKey; + + source_clean_aperture_values[0] = crop_width_num; + source_clean_aperture_values[1] = crop_height_num; + source_clean_aperture_values[2] = crop_offset_left_num; + source_clean_aperture_values[3] = crop_offset_top_num; + + source_clean_aperture = CFDictionaryCreate(kCFAllocatorDefault, + clean_aperture_keys, + source_clean_aperture_values, + 4, + &kCFTypeDictionaryKeyCallBacks, + &kCFTypeDictionaryValueCallBacks); + + CFRelease(crop_width_num); + CFRelease(crop_height_num); + CFRelease(crop_offset_left_num); + CFRelease(crop_offset_top_num); + src = (CVPixelBufferRef)in->data[3]; dst = (CVPixelBufferRef)out->data[3]; + CVBufferSetAttachment(src, kCVImageBufferCleanApertureKey, + source_clean_aperture, kCVAttachmentMode_ShouldPropagate); ret = VTPixelTransferSessionTransferImage(s->transfer, src, dst); if (ret != noErr) { av_log(ctx, AV_LOG_ERROR, "transfer image failed, %d\n", ret); @@ -162,6 +211,7 @@ static int scale_vt_filter_frame(AVFilterLink *link, AVFrame *in) goto fail; } + CFRelease(source_clean_aperture); av_frame_free(&in); return ff_filter_frame(outlink, out);