This implements the bare minimum needed to have circumvention proxy support in OONI Probe CLI.
The reference issue is https://github.com/ooni/probe/issues/1955. (Most of that issue is implemented, save for the fact that currently we do not have support for `http` and `https` proxies.)
While there, add to `Makefile` a rule for correctly building `ooniprobe` and `miniooni` for "this system" (i.e., the default `GOOS` and `GOARCH` on a system), because we needed this for testing this patch and we needed to figure out the commands instead.
All measurements collected since 2022-10-19 with previous versions
of OONI Probe will wrongly report sfu.voip.signal.org as blocked
as it switched to using a different root CA
This fixes: https://github.com/ooni/probe/issues/2344
This change ensures that, in turn, we're able to "remote" all the traffic generated by the `geolocate` package, rather than missing some bits of it that were still using the standard library and caused _some_ geolocations to geolocate as the local host rather than as the remote host.
Extracted from https://github.com/ooni/probe-cli/pull/969, where we tested this functionality.
Closes https://github.com/ooni/probe/issues/1383 (which was long overdue).
Part of https://github.com/ooni/probe/issues/2340, because it allows us to make progress with that.
This diff re-enables `E2E/miniooni.bash`. To make it working properly, we
needed to figure out which were the right cloudfronts to use.
I looked into the configuration and determined that both cloudfronts
should be used because they basically map to the same host.
I also determined it was backwards to test a mixture of prod and testing
APIs, and probably also flaky. So, I choose to only test the prod.
Additionally, I added support for testing all supported tunnels.
Closes https://github.com/ooni/probe/issues/2336
This diff adds to miniooni support for using the torsf tunnel. Such a
tunnel consists of a snowflake pluggable transport in front of a custom
instance of tor and requires tor to be installed.
The usage is like:
```
./miniooni --tunnel=torsf [...]
```
The default snowflake rendezvous method is "domain_fronting". You can
select the AMP cache instead using "amp":
```
./miniooni --snowflake-rendezvous=amp --tunnel=torsf [...]
```
Part of https://github.com/ooni/probe/issues/1955
Closes https://github.com/ooni/probe/issues/2334.
While there, reinstate integration tests, which were also lost in a previous refactoring. However, only run those tests for linux/amd64 because we can be confident that the Go compiler is WAI for all archs we support.
While there, always use bash for running end-to-end tests.
H/T @ainghazal for discovering and reporting this bug.
We introduce the -f, --input-file FILE option with which we
are able to run an OONI Run v2 descriptor stored locally.
In this running mode, there are no checks related to whether the
descriptor has changed, since we're dealing with a local file.
Closes https://github.com/ooni/probe/issues/2328
We're bumping the experiment's version number because we changed the name of the field used to contain late/duplicate DNS responses. We have also changed the algorithm to determine `#dnsDiff`. However, the change should only impact how we log this information. Overall, here the idea is to provide users with a reasonably clear explanation of how the probe maps observations to blocking and accessible using expected/unexpected as the conceptual framework.
Part of https://github.com/ooni/probe/issues/2237
This diff includes a rule to recover from the "measurement failed" state that kicks in when we have a chain of successful redirects from the client side leading to a webpage _and_ any URL in the chain uses HTTPS. See https://github.com/ooni/probe/issues/2307.
While there, fix `i/e/w/iox.go` to avoid triggering the `./script/nocopyreadall.bash` script.
This diff introduces a special rule to avoid emitting null, null when all the connects failed in both the probe and the TH.
While there, recognize that the subset of null, null we're hunting actually deals with websites that are down, so change the internal naming to reflect that and make the code easier to read/understand.
See https://github.com/ooni/probe/issues/2299
It's confusing to see
```
measuring additional addrs from TH: []
```
when actually nothing is going to be measured.
So, instead, let us log about the additional addrs
discovered by the TH instead, which is less confusing.
Part of https://github.com/ooni/probe/issues/2237
See https://github.com/ooni/probe/issues/2290
While there, notice that in such a case the priority selector would hang because of the WaitGroup, so get rid of the WaitGroup and accept that the priority selector is going to hang around for the whole duration of the measurement in some cases. The cancellable `measurer.go`'s context will cause the priority selector to eventually exit when we return from `measurer.go`'s `Run` method.
This diff changes the data format to prefer "udp" to "quic" everywhere we were previously using "quic".
Previously, the code inconsistently used "quic" for operations where we knew we were using "quic" and "udp" otherwise (e.g., for generic operations like ReadFrom).
While it would be more correct to say that a specific HTTP request used "quic" rather than "udp", using "udp" consistently allows one to see how distinct events such as ReadFrom and an handshake all refer to the same address, port, and protocol triple. Therefore, this change makes it easier to programmatically unpack a single measurement and create endpoint stats.
Before implementing this change, I discussed the problem with @hellais who mentioned that ooni/data is not currently using the "quic" string anywhere. I know that ooni/pipeline also doesn't rely on this string. The only users of this feature have been research-oriented experiments such as urlgetter, for which such a change would actually be acceptable.
See https://github.com/ooni/probe/issues/2238 and https://github.com/ooni/spec/pull/262.
