From patchwork Sun Dec 17 22:37:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Martin_Storsj=C3=B6?= X-Patchwork-Id: 45196 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp7012788pzf; Sun, 17 Dec 2023 14:37:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IFDm5vcBINR58p8ulXIn0GpnCMsotD42IiR62aZB24v75uQb8FfHo3GUI1MFfmdfLRBChYC X-Received: by 2002:ac2:558c:0:b0:50e:27b8:80d9 with SMTP id v12-20020ac2558c000000b0050e27b880d9mr982459lfg.39.1702852634190; Sun, 17 Dec 2023 14:37:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702852634; cv=none; d=google.com; s=arc-20160816; b=y2/c497HV8VSsoI5SaMReRpAyuykrMYz5qGJ4okDTsgiwdnuPGUfVfNVi/N4Pz1EgZ y20r+TuCurPQJIZQRFQI5/K0W4a8gjlkhq8NQE9BeDofTUpaMHfL0kBRRflwXA+J7APp tu+j4ZOumUt8ft9pw4i7aoTsfyAqAcLTaORyYK2wJ/RUGT3sy+hFwYKMRZIOmJh2CBuB A/Vus8Z1SdAI7jz9HpYzHAIdvQt82uahlE2/X6SCgqnZXl8/m3W++X7JoK8GCeH1w1Xv FpaxNKU1ePtzGjTD5VknFyUDXfZKcC+DYqSX52ZGFAbEtN1TiS2dIvv9asCf3yQr1iBC fL3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=+iJ+XSVRJcLtG0pFSLoADddTBAVeJUKLPK2P2zZeo/Y=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=psPduTEmncShMUgTRn6F7x47O1xW+UvdxhMe2UnM7Y5qqckdnO7ly39E3Pffm57ync mpqy5f8Bukppq9uhuGsyAUsdHJARl3tiJi3PTh7mOM20602q7xLAVmpTQOnO7WVx4Z0i MsvrtG1ZPi+wgXno2RDdZqpcRV9d4tB148GSUc93rdwBVrV4NQj6rrKzhStjgFVVECTi Q9Qk7KZcn3gTlPpiON6pQF5G2bhaKGqMt0LHVfP276NOpcWkr+1ndf4K02Qin3APIOAT ywtSMHqt2knaoeptuZxxwy30goY4Wk6yG6Wu9/50N+wYw5FHV6wFWlhf7UkMdocV7GDI uqwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20230601.gappssmtp.com header.s=20230601 header.b=Vj1esCoz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c14-20020a170906340e00b00a234be6acffsi695731ejb.1041.2023.12.17.14.37.13; Sun, 17 Dec 2023 14:37:14 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@martin-st.20230601.gappssmtp.com header.s=20230601 header.b=Vj1esCoz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 637BE68D1A5; Mon, 18 Dec 2023 00:37:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0336868CACC for ; Mon, 18 Dec 2023 00:37:04 +0200 (EET) Received: by mail-lj1-f170.google.com with SMTP id 38308e7fff4ca-2cc259392a6so27255111fa.2 for ; Sun, 17 Dec 2023 14:37:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=martin-st.20230601.gappssmtp.com; s=20230601; t=1702852623; x=1703457423; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=ywGtUWD7k4cYUUBWtRR1HIJtZdQUhSw1Aa/k5C5AGYk=; b=Vj1esCoz18JjJSmq/DMLizilK8eGI4jmENe80mEwwmxMGmshoSq4zVb+39Vn/hst6X Qqk/ELQkEVstezEUVGf35i/tz8YPS9OLqWp+VJxso7xfGD5wSdtOdP7keesw+06VVvcO WPfOgC5/OBRD8muaFpCUO3hIE2nUSxHb1kn2P580UYoFKWKB4lwq8qoFUMbpyVG7eG8/ hsf43XfdXofh9/MqNjPC5/yYd52XMItu+ke2/zERIZ7zsUTgPTWUONH5FSyAVPslAXrV Me9k9cgTEG5lmNONmKXCEtjQHmI79MRYGqQMRo6+IhVdOXVyega4UArXI8loBU/0Gxqo uDuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702852623; x=1703457423; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ywGtUWD7k4cYUUBWtRR1HIJtZdQUhSw1Aa/k5C5AGYk=; b=ez2CO55452P1AxQ6HWUu0bFgJFeoQsxrA+T9Bp+1/5l4mKX5Bkyu0Hd3/1EAy7Tlm3 6TB2bNHBjvFgFqukLyczxpjDoo+Hl71PA5GzD7/iPWb2S+/PB8vbqbnqIUvwhOdC4iVz SzC5dTmobwAzJg7NiKu1xsMiliLGKyDIIt9581z/Z4qUfAGBy8rDAUZ7fALjQCrEAh4M uaYahftzJUyqP3VcPQ/XsJhkjZTPXgjezf7BBLgy/zIe+7GE2Aq7wGHoMTuBNd9VBXQv r95sGjSOoHFDdkIuybdpA67GKckayDPWgRbhuQSgfVD79vZGAYTeD2T99UYvulzwscm3 k0kA== X-Gm-Message-State: AOJu0YwtC1Um4LiIs9F/TsuuYiKHD5XWs4JBX0i1/sQ7FoD14JTTXjgn XmO8Tds6YPJ1Vw/Pu0DxvUv/S9pUi6K9kvCT0TNcRw== X-Received: by 2002:a2e:bcd1:0:b0:2cc:2b21:6fc7 with SMTP id z17-20020a2ebcd1000000b002cc2b216fc7mr6607919ljp.88.1702852622936; Sun, 17 Dec 2023 14:37:02 -0800 (PST) Received: from localhost (dsl-tkubng21-58c01c-243.dhcp.inet.fi. [88.192.28.243]) by smtp.gmail.com with ESMTPSA id bz27-20020a05651c0c9b00b002cc744a5517sm43813ljb.103.2023.12.17.14.37.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Dec 2023 14:37:02 -0800 (PST) From: =?utf-8?q?Martin_Storsj=C3=B6?= To: ffmpeg-devel@ffmpeg.org Date: Mon, 18 Dec 2023 00:37:02 +0200 Message-Id: <20231217223702.2878619-1-martin@martin.st> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2] fate: Allow running multiple rounds of tests with differing settings X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: yUV8FSAkdFxt This can be used to run tests multple times, with e.g. differing QEMU settings, by adding something like this to the FATE configuration file: target_exec="qemu-aarch64-static" fate_targets="fate-checkasm fate-cpu" fate_environments="sve128 sve256 sve512" sve128_env="QEMU_CPU=max,sve128=on" sve256_env="QEMU_CPU=max,sve256=on" sve512_env="QEMU_CPU=max,sve512=on" It's also possible to customize the target_exec command further by injecting a sufficiently quoted variable into it, which then can be updated for each run, e.g. target_exec="\$(CUR_EXEC_CMD)". For each of the environment names in fate_environments, the tests that are run get the name suffixed on the fate tests in the test log and fate report, e.g. "fate-checkasm-h264dsp_sve128". --- v2: Added examples to fate_config.sh.template. Will push this together with https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231127123118.3622784-1-martin@martin.st/ soon if this seems reasonable. --- doc/fate_config.sh.template | 20 ++++++++++++++++++++ tests/Makefile | 4 ++-- tests/fate.sh | 14 +++++++++++++- 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/doc/fate_config.sh.template b/doc/fate_config.sh.template index 06bb79a832..8681805d65 100644 --- a/doc/fate_config.sh.template +++ b/doc/fate_config.sh.template @@ -33,3 +33,23 @@ makeopts= # extra options passed to 'make' # defaults to 'tar c' #fate_targets= # targets to make when running fate; defaults to "fate", # can be set to run a subset of tests, e.g. "fate-checkasm". + +#fate_environments= # a list of names of configurations to run tests for; + # each round is run with variables from ${${name}_env} set. + +# One example of using fate_environments: + +# target_exec="qemu-aarch64-static" +# fate_targets="fate-checkasm fate-cpu" +# fate_environments="sve128 sve256" +# sve128_env="QEMU_CPU=max,sve128=on" +# sve256_env="QEMU_CPU=max,sve256=on" + +# The variables set by fate_environments can also be used explicitly +# by target_exec, e.g. like this: + +# target_exec="qemu-aarch64-static -cpu \$(MY_CPU)" +# fate_targets="fate-checkasm fate-cpu" +# fate_environments="sve128 sve256" +# sve128_env="MY_CPU=max,sve128=on" +# sve256_env="MY_CPU=max,sve256=on" diff --git a/tests/Makefile b/tests/Makefile index 444c09b3de..744dbcdfb3 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -305,8 +305,8 @@ $(FATE): export PROGSUF = $(PROGSSUF) $(FATE): export EXECSUF = $(EXESUF) $(FATE): export HOSTEXECSUF = $(HOSTEXESUF) $(FATE): $(FATE_UTILS:%=tests/%$(HOSTEXESUF)) | $(FATE_OUTDIRS) - @echo "TEST $(@:fate-%=%)" - $(Q)$(SRC_PATH)/tests/fate-run.sh $@ "$(TARGET_SAMPLES)" "$(TARGET_EXEC)" "$(TARGET_PATH)" '$(CMD)' '$(CMP)' '$(REF)' '$(FUZZ)' '$(THREADS)' '$(THREAD_TYPE)' '$(CPUFLAGS)' '$(CMP_SHIFT)' '$(CMP_TARGET)' '$(SIZE_TOLERANCE)' '$(CMP_UNIT)' '$(GEN)' '$(HWACCEL)' '$(REPORT)' '$(KEEP_FILES)' + @echo "TEST $(@:fate-%=%)$(FATE_SUFFIX)" + $(Q)$(SRC_PATH)/tests/fate-run.sh $@$(FATE_SUFFIX) "$(TARGET_SAMPLES)" "$(TARGET_EXEC)" "$(TARGET_PATH)" '$(CMD)' '$(CMP)' '$(REF)' '$(FUZZ)' '$(THREADS)' '$(THREAD_TYPE)' '$(CPUFLAGS)' '$(CMP_SHIFT)' '$(CMP_TARGET)' '$(SIZE_TOLERANCE)' '$(CMP_UNIT)' '$(GEN)' '$(HWACCEL)' '$(REPORT)' '$(KEEP_FILES)' fate-list: @printf '%s\n' $(sort $(FATE)) diff --git a/tests/fate.sh b/tests/fate.sh index 95408ea109..c04eb41cbe 100755 --- a/tests/fate.sh +++ b/tests/fate.sh @@ -101,7 +101,19 @@ compile_extra()( fate()( test "$build_only" = "yes" && return cd ${build} || return - ${make} ${makeopts_fate-${makeopts}} -k ${fate_targets} + if [ -n "${fate_environments}" ]; then + ret=0 + for e in ${fate_environments}; do + eval "curenv=\${${e}_env}" + echo Testing environment ${e}: ${curenv} + ${make} ${makeopts_fate-${makeopts}} -k ${fate_targets} FATE_SUFFIX=_${e} ${curenv} + cur_ret=$? + test $cur_ret != 0 && ret=$cur_ret + done + return $ret + else + ${make} ${makeopts_fate-${makeopts}} -k ${fate_targets} + fi ) clean(){