I've just branched off the `release/3.16` branch since we're
really looking good for release modulo minor changes.
Hence, it's time to update `master`'s version.
While working on https://github.com/ooni/probe/issues/2130, and specifically on the action item related to making sure all workflows are green, I was confronted with the complexity of the QA directory.
There's plenty of cleaning up and simplifying there. The original intent was to A/B test `miniooni` and `measurement_kit` to ensure they were behaving the same. We don't have this need anymore.
Rather, it seems the QA scripts have grown large and flaky, to the point that I am always tempted to ignore them. The underlying censorship engine, jafar, has also not been developed for quite some time.
So, the first step towards improve the QA infrastructure seems to be humble and acknowledge that we cannot realistically maintain these checks using jafar as a backend for so many experiments.
Let us focus on our most important experiment, Web Connectivity, and let us keep QA checks for it.
Additionally, let us simplify and cleanup QA as much as possible, though without introducing radical changes.
The end result is a QA for Web Connectivity that seems reasonable and runs in six minutes.
This diff modifies webconnectivity@v0.5 to take decisions regarding
TLS blocking by using the response from the TH rather than using
questionable heuristics based on inspecting the TLSHandshake list
alone. This change should improve correctness _when_ we're using
the improved TH, which is currently used for 50% of the probes.
See https://github.com/ooni/probe/issues/2257
While there, modify `control.go` to specify which control is being used.
As silly as it seems, emojis help _a lot_ when eyeballing logs
to quickly identify unexpected lines.
I'm doing this work as part of https://github.com/ooni/probe/issues/2257
While there, include integration testing to make sure the script
is working as intended before using it.
While there, edit maketarball.bash's comments.
The residual censorship algorithm prevents dnscheck for checking the
same endpoint again in a short time frame under the assumption that a
previous measurement could have caused residual censorship.
In https://github.com/ooni/probe/issues/2234, we mentioned we probably
wanted to disable this algorithm because we didn't want to slow down
dnscheck too much and because we didn't detect this alleged source of
blocking in our previous measurements.
It turns out the algorithm was already disabled. I probably did this
after we published the paper on DNS censorship to make measurements
overall a bit faster.
So, just introduce a new extension field to the measurement telling us
that the residual censorship algorithm is disabled.
It's not super useful except as for the fact that the next time I
read the source code I notice that the algorithm is disabled.
Most of miniooni panics on errors. We should not panic on error with
--repeat-every, rather we should try the next measurement.
See https://github.com/ooni/probe/issues/2250
Part of https://github.com/ooni/probe/issues/2184, because I wanted
to allow swapping commands and options more freely.
As a side effect, this PR closes https://github.com/ooni/probe/issues/2248.
AFAICT, every usage that was legal before is still legal. What has
changed seems the freedom to swap commands and options and a much
better help that lists the available options.
This diff splits miniooni's implementation in smaller and more
easily tractable blocks ahead of future refactoring.
I'm trying to make `miniooni oonirun -i URL` as possible as
`miniooni -i URL oonirun`, because users typically expect this
kind of flexibity from modern Unix commands.
Part of https://github.com/ooni/probe/issues/2184
This work aims to make Linux builds faster to make https://github.com/ooni/probe/issues/2249 more convenient. Since those builds runs inside Docker, the problem to solve here is to save/restore the Go caches notwithstanding Docker. Because Docker runs as root, we need to modify the build a bit to run as a normal user. Otherwise, we will not be able to save the Go cache using actions/cache@v3. (Other approaches such as using `sudo` are possible but running the build as an unprivileged user actually looks cleaner, so I chose to do that.) While there, add a `.editorconfig`.
This diff modifies all the github actions that produce assets to
publish on a release called rolling when we are not building a tag.
If everything goes as planned, we should be able to provide
people with automatically generated fresh binaries for testing.
While there, introduce caching for all builds to make them
as fast as possible. I suspect gomobile based builds will not
see any speed up but other builds most likely will.
See https://github.com/ooni/probe/issues/2249
This diff introduces a build script, makefile rules, and github actions
rules to build and public android CLI releases.
See https://github.com/ooni/probe/issues/1723
Rather than hardcoding the NDK version inside a script, encode it
as a file in the filesystem, which is easier to share.
Make sure we use the desired NDK by setting environment variables.
Use `-androidabi 21`, which:
1. is what rclone did: 8390ba4ca9
2. is the minimum ABI used by probe-android: 994651be52/app/build.gradle (L10)
Part of https://github.com/ooni/probe/issues/2130
Part of https://github.com/ooni/probe/issues/1753.
While there, introduce a rule by which, if the branch is named `fullbuild` we run all possible builds. It helps to test all the builds without creating a release branch. Because release branches are protected, they cannot be deleted easily. On the contrary, the `fullbuild` branch can easily be disposed of.
It seems several CI builds failed for [v3.16.0-alpha](https://github.com/ooni/probe-cli/releases/tag/v3.16.0-alpha). Let's aim to repair miniooni and ooniprobe-windows for now. The other failing builds seem more tricky. (Android fails with an unsupported NDK while Linux fails with issues accessing the git repository from Docker, probably because the the user running inside Docker is not the user that owns the repository.)
Closes https://github.com/ooni/probe/issues/2183
While there, avoid exposing nil values for optional fields of the
THResponse struct (i.e., "ip_info" and "tls_handshake").
While there, fix `measurexlite`'s `OperationLogger` test
and make it deterministic rather than racy.
This commit moves the TH structs and definitions to model. We don't want
oohelperd to depend on web_connectivity@v0.4.
Part of https://github.com/ooni/probe/issues/2240
A bunch of packages (including oohelperd) just need the ability to
use MaxMind-like databases. They don't need the additional functionality
implemented by the geolocate package. Such a package, in fact, is
mostly (if not only) needed by the engine package.
Therefore, move code to query MaxMind-like databases to a separate
package, and avoid depending on geolocate in all the packages for
which it's sufficient to use geoipx.
Part of https://github.com/ooni/probe/issues/2240
