QA: refactor, cleanup, fix, and simplify (#938)

While working on https://github.com/ooni/probe/issues/2130, and specifically on the action item related to making sure all workflows are green, I was confronted with the complexity of the QA directory.

There's plenty of cleaning up and simplifying there. The original intent was to A/B test `miniooni` and `measurement_kit` to ensure they were behaving the same. We don't have this need anymore.

Rather, it seems the QA scripts have grown large and flaky, to the point that I am always tempted to ignore them. The underlying censorship engine, jafar, has also not been developed for quite some time.

So, the first step towards improve the QA infrastructure seems to be humble and acknowledge that we cannot realistically maintain these checks using jafar as a backend for so many experiments.

Let us focus on our most important experiment, Web Connectivity, and let us keep QA checks for it.

Additionally, let us simplify and cleanup QA as much as possible, though without introducing radical changes.

The end result is a QA for Web Connectivity that seems reasonable and runs in six minutes.
This commit is contained in:
Simone Basso 2022-09-07 14:55:36 +02:00 committed by GitHub
commit 2389c7cab3
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
20 changed files with 46 additions and 1083 deletions

View file

@ -1,5 +1,16 @@
#!/bin/sh
set -ex
#!/bin/bash
set -euxo pipefail
DOCKER=${DOCKER:-docker}
GOVERSION=$(cat GOVERSION)
cat > QA/Dockerfile << EOF
FROM golang:$GOVERSION-alpine
RUN apk add gcc go git musl-dev iptables tmux bind-tools curl sudo python3
EOF
$DOCKER build -t jafar-qa ./QA/
$DOCKER run --privileged -v`pwd`:/jafar -w/jafar jafar-qa ./QA/pyrun.sh "$@"
$DOCKER run --privileged -v$(pwd):/jafar -w/jafar jafar-qa ./QA/dockermain.sh "$@"