c/ooni-probe-cli
Watch
1
0
Fork
You've already forked ooni-probe-cli
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

QA: refactor, cleanup, fix, and simplify (#938)

Browse source 
While working on https://github.com/ooni/probe/issues/2130, and specifically on the action item related to making sure all workflows are green, I was confronted with the complexity of the QA directory.

There's plenty of cleaning up and simplifying there. The original intent was to A/B test `miniooni` and `measurement_kit` to ensure they were behaving the same. We don't have this need anymore.

Rather, it seems the QA scripts have grown large and flaky, to the point that I am always tempted to ignore them. The underlying censorship engine, jafar, has also not been developed for quite some time.

So, the first step towards improve the QA infrastructure seems to be humble and acknowledge that we cannot realistically maintain these checks using jafar as a backend for so many experiments.

Let us focus on our most important experiment, Web Connectivity, and let us keep QA checks for it.

Additionally, let us simplify and cleanup QA as much as possible, though without introducing radical changes.

The end result is a QA for Web Connectivity that seems reasonable and runs in six minutes.
This commit is contained in:
Simone Basso 2022-09-07 14:55:36 +02:00 • committed by GitHub
commit 2389c7cab3
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
20 changed files with 46 additions and 1083 deletions
Download patch file Download diff file Expand all files Collapse all files

36
QA/README.md
View file

@ -1,34 +1,7 @@
# Quality Assurance scripts
This directory contains quality assurance scripts that use Jafar to
ensure that OONI implementations behave. These scripts take on the
command line as argument the path to a binary with a OONI Probe v2.x
like command line interface. We do not care about full compatibility
but rather about having enough similar flags that running these tools
in parallel is not too much of a burden for us.
Tools with this shallow-compatible CLI are:
1. `github.com/ooni/probe-legacy`
2. `github.com/measurement-kit/measurement-kit/src/measurement_kit`
3. `github.com/ooni/probe-engine/internal/cmd/miniooni`
## Run QA on a Linux system
These scripts assume you're on a Linux system with `iptables`, `bash`,
`python3`, and possibly a bunch of other tools installed.
To start the QA script, run this command:
```bash
sudo ./QA/$nettest.py $ooni_exe
```
where `$nettest` is the nettest name (e.g. `telegram`) and `$ooni_exe`
is the OONI Probe v2.x compatible binary to test.
The Python script needs to run as root. Note however that sudo will also
be used to run `$ooni_exe` with the privileges of the `nobody` user.
ensure that OONI implementations behave. These scripts work with miniooni.
## Run QA using a docker container
@ -38,17 +11,14 @@ Run test in a suitable Docker container using:
./QA/rundocker.sh $nettest
```
Note that this will run a `--privileged` docker container. This will
eventually run the Python script you would run on Linux.
For now, the docker scripts only perform QA of `miniooni`.
Note that this will run a `--privileged` docker container.
## Diagnosing issues
The Python script that performs the QA runs a specific OONI test under
different failure conditions and stops at the first unexpected value found
in the resulting JSONL report. You can infer what went wrong by reading
the output of the `$ooni_exe` command itself, which should be above the point
the output of the `miniooni` command itself, which should be above the point
where the Python script stopped, as well as by inspecting the JSONL file on
disk. By convention such file is named `$nettest.jsonl` and only contains
the result of the last run of `$nettest`.