From patchwork Mon Dec 11 22:14:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Martin_Storsj=C3=B6?= X-Patchwork-Id: 45080 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp3229261pzf; Mon, 11 Dec 2023 14:14:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IFIvLAXSwyOiQCJCCcPOlQVcj/MqpfqGP1C6qGrU1uV8jNLzRqGM0LP+aJeHYCebw4KZRFB X-Received: by 2002:a17:906:8cc:b0:a1c:4ebe:aa39 with SMTP id o12-20020a17090608cc00b00a1c4ebeaa39mr2806934eje.51.1702332862371; Mon, 11 Dec 2023 14:14:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702332862; cv=none; d=google.com; s=arc-20160816; b=lRdyp2L9VE8xTLkJ9TV/Y8V5N1U7bOO31OUq5Zv8YF5tPnKJpFXdUWu/c6W5vqHRv4 Sjm0s9p6vCP6z7FHxRKB2t4U/QLvTMa8FfpUxrWYaRFTRHRqka5F/Wn0hinmAFYICb6r IYpr3HyhQLPDbLpiEHfdXp1dN4s3lMtYYMzv3P9tKa3by4g/gmJ5Xb4yUoWW+BjkmmKK Qbf6FcUcyv+BHOnDSN0ZJbzuQbnOwmr78PMOJsAIzQvuRqxIPdMOXhfJuevuehTEHpEi ORTMSoFXQALdd7Hq+lw0mL+UNnotYS5SExy1VRZ5GahHGl5xiHvJ9Nc4Al8LgTmeyGPT 89sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=BxMDFV2M6PBG2c+nyw3z7XddosNstJRSjVdUKNuxpTE=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=oPa0/EY+schHWo+TbChfZ0juATJqNDVjG9KeQU6lRDOXwAeWvJItQPp01f7VaVsD26 A27Qbj8Dc9ZFL5bSUeP46YjsSnsmfcnHr7VrCFK4HS5OVF2P6ZUYm/rqgba61L79BFRl x6qvVZ3RRskJuoLEeWNa4i2aQXtosydyAX5MhYdDcNnJXrmtmejlsgLyctqwPzgX70l7 NST50FDIzoT6/1eshRRFHe0gzh1qg4B9+EIcnj3Q+PbuAxQABWxNj5ipJxJpmpDg0fPg kbRS+JWfBo74iRidJiugOkBsdl5CQQWOH7sUk+RStyBZilPdQbpO45C0tiJN20CK3ryR +/2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20230601.gappssmtp.com header.s=20230601 header.b=GKm2ZIOm; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j18-20020a170906475200b00a1cb971fd55si3784537ejs.722.2023.12.11.14.14.20; Mon, 11 Dec 2023 14:14:22 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20230601.gappssmtp.com header.s=20230601 header.b=GKm2ZIOm; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2219068D1E3; Tue, 12 Dec 2023 00:14:16 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8BDF868D13F for ; Tue, 12 Dec 2023 00:14:09 +0200 (EET) Received: by mail-lf1-f52.google.com with SMTP id 2adb3069b0e04-50bee606265so5251278e87.2 for ; Mon, 11 Dec 2023 14:14:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=martin-st.20230601.gappssmtp.com; s=20230601; t=1702332848; x=1702937648; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=mkoMtFva+OWis/JjLOF80me3HfWawWk4eeY/c/8zGI8=; b=GKm2ZIOmNfKFZDLnlfN0PvN86dQKV3Trcapi7eZhJrAM9O6H2Uh79hyuYBt70HX9QR BLTSF5H0J4PQ61mrEcJzHzrkx5ZyB7jXwyXJr8RyNDzTz+QjVzECr7GwfGtxeQ6RJCde +aCqNdaVRDKis95TPqakwvlc27ijVBfjg7RvOSKaLu5/LwJpDv4vd9rwVg3eiXQYl9dU qTc5e0it5lXTqDelhP9D7GU4Qpd9KzXVgTMmePmtN4yyDA9GazDm3l2MJ8I/aYs5g9js wY4OvtYuZcepzNwDADx9owBEcSDsbx31kvQiwn4HGyZJa2zMf3lX5b1NT6087lNVcfD0 HwZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702332848; x=1702937648; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mkoMtFva+OWis/JjLOF80me3HfWawWk4eeY/c/8zGI8=; b=XYncoiXNNP8oFj6rRm1PIvGRZkiwLv2Sx+SmcveQMFu0TF9QoSVL1nAkkt9nA80W3c sTlrX3Z98b1eaOVKESMh9oNxMdbuGNHvneet3NuOJhzoDTmsgLOAYtKfV55sOYjH7wTd E8rZlSlgiwAtXcBgLCXW+N5MsEo4JJknx2CDnNoX35je6w4VpmVKAtLdBmfuizay80i7 BzixePiRpBDdWjbM4LG5gYIaz5+l4FMyxEoHrMOJ1YdEbZFDWxmbcI1NsPQV5dFE0omM p1/swZ9EMNqdNIR8ABFa/wulptxb3RNIdyqhy30/K1MuVfwfoENUa8OmRk7MunteRCqS uFNw== X-Gm-Message-State: AOJu0YyA4VvxakhDFANGtJiUkkgUOFV6aL5HXKQkxX4oeKrbe9QZ6Gps dkO8JnWS/4AkKV82a40EPYPj9LgQ2YyN2osA7ao7Dg== X-Received: by 2002:a05:6512:39cc:b0:50b:ec63:8cf with SMTP id k12-20020a05651239cc00b0050bec6308cfmr3499442lfu.21.1702332847993; Mon, 11 Dec 2023 14:14:07 -0800 (PST) Received: from localhost.localdomain (dsl-tkubng21-58c01c-243.dhcp.inet.fi. [88.192.28.243]) by smtp.gmail.com with ESMTPSA id k4-20020a05651210c400b0050bef21a012sm1192594lfg.191.2023.12.11.14.14.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 14:14:07 -0800 (PST) From: =?utf-8?q?Martin_Storsj=C3=B6?= To: ffmpeg-devel@ffmpeg.org Date: Tue, 12 Dec 2023 00:14:06 +0200 Message-Id: <20231211221406.2385689-1-martin@martin.st> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] fate: Allow running multiple rounds of tests with differing settings X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: G1r93/FTPfrT This can be used to run tests multple times, with e.g. differing QEMU settings, by adding something like this to the FATE configuration file: target_exec="qemu-aarch64-static" fate_targets="fate-checkasm fate-cpu" fate_environments="sve128 sve256 sve512" sve128_env="QEMU_CPU=max,sve128=on" sve256_env="QEMU_CPU=max,sve256=on" sve512_env="QEMU_CPU=max,sve512=on" It's also possible to customize the target_exec command further by injecting a sufficiently quoted variable into it, which then can be updated for each run, e.g. target_exec="\$(CUR_EXEC_CMD)". For each of the environment names in fate_environments, the tests that are run get the name suffixed on the fate tests in the test log and fate report, e.g. "fate-checkasm-h264dsp_sve128". --- For adding the environment to the fate run, there's three ways it can be done: - (${curenv}; make fate) - ${curenv} make fate - make fate ${curenv} The first has most freedom in what one can do, but requires explicit "export VAR=value", while the other two are simpler for just setting "VAR1=val1 VAR2=val2". For now, I picked the third form. --- tests/Makefile | 4 ++-- tests/fate.sh | 14 +++++++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/tests/Makefile b/tests/Makefile index 444c09b3de..744dbcdfb3 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -305,8 +305,8 @@ $(FATE): export PROGSUF = $(PROGSSUF) $(FATE): export EXECSUF = $(EXESUF) $(FATE): export HOSTEXECSUF = $(HOSTEXESUF) $(FATE): $(FATE_UTILS:%=tests/%$(HOSTEXESUF)) | $(FATE_OUTDIRS) - @echo "TEST $(@:fate-%=%)" - $(Q)$(SRC_PATH)/tests/fate-run.sh $@ "$(TARGET_SAMPLES)" "$(TARGET_EXEC)" "$(TARGET_PATH)" '$(CMD)' '$(CMP)' '$(REF)' '$(FUZZ)' '$(THREADS)' '$(THREAD_TYPE)' '$(CPUFLAGS)' '$(CMP_SHIFT)' '$(CMP_TARGET)' '$(SIZE_TOLERANCE)' '$(CMP_UNIT)' '$(GEN)' '$(HWACCEL)' '$(REPORT)' '$(KEEP_FILES)' + @echo "TEST $(@:fate-%=%)$(FATE_SUFFIX)" + $(Q)$(SRC_PATH)/tests/fate-run.sh $@$(FATE_SUFFIX) "$(TARGET_SAMPLES)" "$(TARGET_EXEC)" "$(TARGET_PATH)" '$(CMD)' '$(CMP)' '$(REF)' '$(FUZZ)' '$(THREADS)' '$(THREAD_TYPE)' '$(CPUFLAGS)' '$(CMP_SHIFT)' '$(CMP_TARGET)' '$(SIZE_TOLERANCE)' '$(CMP_UNIT)' '$(GEN)' '$(HWACCEL)' '$(REPORT)' '$(KEEP_FILES)' fate-list: @printf '%s\n' $(sort $(FATE)) diff --git a/tests/fate.sh b/tests/fate.sh index 95408ea109..c04eb41cbe 100755 --- a/tests/fate.sh +++ b/tests/fate.sh @@ -101,7 +101,19 @@ compile_extra()( fate()( test "$build_only" = "yes" && return cd ${build} || return - ${make} ${makeopts_fate-${makeopts}} -k ${fate_targets} + if [ -n "${fate_environments}" ]; then + ret=0 + for e in ${fate_environments}; do + eval "curenv=\${${e}_env}" + echo Testing environment ${e}: ${curenv} + ${make} ${makeopts_fate-${makeopts}} -k ${fate_targets} FATE_SUFFIX=_${e} ${curenv} + cur_ret=$? + test $cur_ret != 0 && ret=$cur_ret + done + return $ret + else + ${make} ${makeopts_fate-${makeopts}} -k ${fate_targets} + fi ) clean(){