Code based on urlgetter had this event and we would like to have this
event with step-by-step code as well.
Because there's no tracing for HTTP when using step-by-step, we will
need to include emitting these events inside the boilerplate.
By doing that, we emit events out of order, so make sure we sort
them by T, which is "the moment when the event was collected".
Part of https://github.com/ooni/probe/issues/2238
* fix(model/archival.go): more optional keys
Basically, `t0` and `transaction_id` should be optional. Version 0.4.x
of web_connectivity should not include them, version 0.5.x should.
There is a technical reason why v0.4.x should not include them. The code
it is based on, tracex, does not record these two fields.
Whereas, v0.5.x, uses measurexlite, which records these two fields.
Part of https://github.com/ooni/probe/issues/2238
* fix(webconnectivity@v0.5): add more fields
This diff adds the following fields to webconnectivity@v0.5:
1. agent, always set to "redirect" (legacy field);
2. client_resolver, properly initialized w/ the resolver's IPv4 address;
3. retries, legacy field always set to null;
4. socksproxy, legacy field always set to null.
Part of https://github.com/ooni/probe/issues/2238
* fix(webconnectivity@v0.5): register extensions
The general idea behind this field is that we would be able
in the future to tweak the data model for some fields, by declaring
we're using a later version, so it seems useful to add it.
See https://github.com/ooni/probe/issues/2238
* fix(measurexlite): use tcp or quic for tls handshake network
This diff fixes a bug where measurexlite was using "tls" as the
protocol for the TLS handshake when using TCP.
While this choice _could_ make sense, the rest of the code we have
written so far uses "tcp" instead.
Using "tcp" makes more sense because it allows you to search for
the same endpoint across different events by checking for the same
network and for the same endpoint rather than special casing TLS
handshakes for using "tls" when the endpoint is "tcp".
See https://github.com/ooni/probe/issues/2238
* chore: run alltests.yml for "alltestsbuild" branches
Part of https://github.com/ooni/probe/issues/2238
I've just branched off the `release/3.16` branch since we're
really looking good for release modulo minor changes.
Hence, it's time to update `master`'s version.
While working on https://github.com/ooni/probe/issues/2130, and specifically on the action item related to making sure all workflows are green, I was confronted with the complexity of the QA directory.
There's plenty of cleaning up and simplifying there. The original intent was to A/B test `miniooni` and `measurement_kit` to ensure they were behaving the same. We don't have this need anymore.
Rather, it seems the QA scripts have grown large and flaky, to the point that I am always tempted to ignore them. The underlying censorship engine, jafar, has also not been developed for quite some time.
So, the first step towards improve the QA infrastructure seems to be humble and acknowledge that we cannot realistically maintain these checks using jafar as a backend for so many experiments.
Let us focus on our most important experiment, Web Connectivity, and let us keep QA checks for it.
Additionally, let us simplify and cleanup QA as much as possible, though without introducing radical changes.
The end result is a QA for Web Connectivity that seems reasonable and runs in six minutes.
This diff modifies webconnectivity@v0.5 to take decisions regarding
TLS blocking by using the response from the TH rather than using
questionable heuristics based on inspecting the TLSHandshake list
alone. This change should improve correctness _when_ we're using
the improved TH, which is currently used for 50% of the probes.
See https://github.com/ooni/probe/issues/2257
While there, modify `control.go` to specify which control is being used.
As silly as it seems, emojis help _a lot_ when eyeballing logs
to quickly identify unexpected lines.
I'm doing this work as part of https://github.com/ooni/probe/issues/2257
While there, include integration testing to make sure the script
is working as intended before using it.
While there, edit maketarball.bash's comments.
The residual censorship algorithm prevents dnscheck for checking the
same endpoint again in a short time frame under the assumption that a
previous measurement could have caused residual censorship.
In https://github.com/ooni/probe/issues/2234, we mentioned we probably
wanted to disable this algorithm because we didn't want to slow down
dnscheck too much and because we didn't detect this alleged source of
blocking in our previous measurements.
It turns out the algorithm was already disabled. I probably did this
after we published the paper on DNS censorship to make measurements
overall a bit faster.
So, just introduce a new extension field to the measurement telling us
that the residual censorship algorithm is disabled.
It's not super useful except as for the fact that the next time I
read the source code I notice that the algorithm is disabled.
Most of miniooni panics on errors. We should not panic on error with
--repeat-every, rather we should try the next measurement.
See https://github.com/ooni/probe/issues/2250
Part of https://github.com/ooni/probe/issues/2184, because I wanted
to allow swapping commands and options more freely.
As a side effect, this PR closes https://github.com/ooni/probe/issues/2248.
AFAICT, every usage that was legal before is still legal. What has
changed seems the freedom to swap commands and options and a much
better help that lists the available options.
This diff splits miniooni's implementation in smaller and more
easily tractable blocks ahead of future refactoring.
I'm trying to make `miniooni oonirun -i URL` as possible as
`miniooni -i URL oonirun`, because users typically expect this
kind of flexibity from modern Unix commands.
Part of https://github.com/ooni/probe/issues/2184
This work aims to make Linux builds faster to make https://github.com/ooni/probe/issues/2249 more convenient. Since those builds runs inside Docker, the problem to solve here is to save/restore the Go caches notwithstanding Docker. Because Docker runs as root, we need to modify the build a bit to run as a normal user. Otherwise, we will not be able to save the Go cache using actions/cache@v3. (Other approaches such as using `sudo` are possible but running the build as an unprivileged user actually looks cleaner, so I chose to do that.) While there, add a `.editorconfig`.
This diff modifies all the github actions that produce assets to
publish on a release called rolling when we are not building a tag.
If everything goes as planned, we should be able to provide
people with automatically generated fresh binaries for testing.
While there, introduce caching for all builds to make them
as fast as possible. I suspect gomobile based builds will not
see any speed up but other builds most likely will.
See https://github.com/ooni/probe/issues/2249
This diff introduces a build script, makefile rules, and github actions
rules to build and public android CLI releases.
See https://github.com/ooni/probe/issues/1723
Rather than hardcoding the NDK version inside a script, encode it
as a file in the filesystem, which is easier to share.
Make sure we use the desired NDK by setting environment variables.
Use `-androidabi 21`, which:
1. is what rclone did: 8390ba4ca9
2. is the minimum ABI used by probe-android: 994651be52/app/build.gradle (L10)
Part of https://github.com/ooni/probe/issues/2130
Part of https://github.com/ooni/probe/issues/1753.
While there, introduce a rule by which, if the branch is named `fullbuild` we run all possible builds. It helps to test all the builds without creating a release branch. Because release branches are protected, they cannot be deleted easily. On the contrary, the `fullbuild` branch can easily be disposed of.
It seems several CI builds failed for [v3.16.0-alpha](https://github.com/ooni/probe-cli/releases/tag/v3.16.0-alpha). Let's aim to repair miniooni and ooniprobe-windows for now. The other failing builds seem more tricky. (Android fails with an unsupported NDK while Linux fails with issues accessing the git repository from Docker, probably because the the user running inside Docker is not the user that owns the repository.)
Closes https://github.com/ooni/probe/issues/2183
While there, avoid exposing nil values for optional fields of the
THResponse struct (i.e., "ip_info" and "tls_handshake").
While there, fix `measurexlite`'s `OperationLogger` test
and make it deterministic rather than racy.
