QA: refactor, cleanup, fix, and simplify (#938)
While working on https://github.com/ooni/probe/issues/2130, and specifically on the action item related to making sure all workflows are green, I was confronted with the complexity of the QA directory.
There's plenty of cleaning up and simplifying there. The original intent was to A/B test `miniooni` and `measurement_kit` to ensure they were behaving the same. We don't have this need anymore.
Rather, it seems the QA scripts have grown large and flaky, to the point that I am always tempted to ignore them. The underlying censorship engine, jafar, has also not been developed for quite some time.
So, the first step towards improve the QA infrastructure seems to be humble and acknowledge that we cannot realistically maintain these checks using jafar as a backend for so many experiments.
Let us focus on our most important experiment, Web Connectivity, and let us keep QA checks for it.
Additionally, let us simplify and cleanup QA as much as possible, though without introducing radical changes.
The end result is a QA for Web Connectivity that seems reasonable and runs in six minutes.
2022-09-07 14:55:36 +02:00
|
|
|
#!/bin/bash
|
|
|
|
|
|
|
|
|
|
set -euxo pipefail
|
|
|
|
|
|
2021-02-02 12:05:47 +01:00
|
|
|
DOCKER=${DOCKER:-docker}
|
QA: refactor, cleanup, fix, and simplify (#938)
While working on https://github.com/ooni/probe/issues/2130, and specifically on the action item related to making sure all workflows are green, I was confronted with the complexity of the QA directory.
There's plenty of cleaning up and simplifying there. The original intent was to A/B test `miniooni` and `measurement_kit` to ensure they were behaving the same. We don't have this need anymore.
Rather, it seems the QA scripts have grown large and flaky, to the point that I am always tempted to ignore them. The underlying censorship engine, jafar, has also not been developed for quite some time.
So, the first step towards improve the QA infrastructure seems to be humble and acknowledge that we cannot realistically maintain these checks using jafar as a backend for so many experiments.
Let us focus on our most important experiment, Web Connectivity, and let us keep QA checks for it.
Additionally, let us simplify and cleanup QA as much as possible, though without introducing radical changes.
The end result is a QA for Web Connectivity that seems reasonable and runs in six minutes.
2022-09-07 14:55:36 +02:00
|
|
|
|
|
|
|
|
GOVERSION=$(cat GOVERSION)
|
|
|
|
|
|
|
|
|
|
cat > QA/Dockerfile << EOF
|
|
|
|
|
FROM golang:$GOVERSION-alpine
|
|
|
|
|
RUN apk add gcc go git musl-dev iptables tmux bind-tools curl sudo python3
|
|
|
|
|
EOF
|
|
|
|
|
|
2021-02-02 12:05:47 +01:00
|
|
|
$DOCKER build -t jafar-qa ./QA/
|
QA: refactor, cleanup, fix, and simplify (#938)
While working on https://github.com/ooni/probe/issues/2130, and specifically on the action item related to making sure all workflows are green, I was confronted with the complexity of the QA directory.
There's plenty of cleaning up and simplifying there. The original intent was to A/B test `miniooni` and `measurement_kit` to ensure they were behaving the same. We don't have this need anymore.
Rather, it seems the QA scripts have grown large and flaky, to the point that I am always tempted to ignore them. The underlying censorship engine, jafar, has also not been developed for quite some time.
So, the first step towards improve the QA infrastructure seems to be humble and acknowledge that we cannot realistically maintain these checks using jafar as a backend for so many experiments.
Let us focus on our most important experiment, Web Connectivity, and let us keep QA checks for it.
Additionally, let us simplify and cleanup QA as much as possible, though without introducing radical changes.
The end result is a QA for Web Connectivity that seems reasonable and runs in six minutes.
2022-09-07 14:55:36 +02:00
|
|
|
|
|
|
|
|
$DOCKER run --privileged -v$(pwd):/jafar -w/jafar jafar-qa ./QA/dockermain.sh "$@"
|