* fix(model/archival.go): more optional keys
Basically, `t0` and `transaction_id` should be optional. Version 0.4.x
of web_connectivity should not include them, version 0.5.x should.
There is a technical reason why v0.4.x should not include them. The code
it is based on, tracex, does not record these two fields.
Whereas, v0.5.x, uses measurexlite, which records these two fields.
Part of https://github.com/ooni/probe/issues/2238
* fix(webconnectivity@v0.5): add more fields
This diff adds the following fields to webconnectivity@v0.5:
1. agent, always set to "redirect" (legacy field);
2. client_resolver, properly initialized w/ the resolver's IPv4 address;
3. retries, legacy field always set to null;
4. socksproxy, legacy field always set to null.
Part of https://github.com/ooni/probe/issues/2238
* fix(webconnectivity@v0.5): register extensions
The general idea behind this field is that we would be able
in the future to tweak the data model for some fields, by declaring
we're using a later version, so it seems useful to add it.
See https://github.com/ooni/probe/issues/2238
* fix(measurexlite): use tcp or quic for tls handshake network
This diff fixes a bug where measurexlite was using "tls" as the
protocol for the TLS handshake when using TCP.
While this choice _could_ make sense, the rest of the code we have
written so far uses "tcp" instead.
Using "tcp" makes more sense because it allows you to search for
the same endpoint across different events by checking for the same
network and for the same endpoint rather than special casing TLS
handshakes for using "tls" when the endpoint is "tcp".
See https://github.com/ooni/probe/issues/2238
* chore: run alltests.yml for "alltestsbuild" branches
Part of https://github.com/ooni/probe/issues/2238
I've just branched off the `release/3.16` branch since we're
really looking good for release modulo minor changes.
Hence, it's time to update `master`'s version.
This diff modifies webconnectivity@v0.5 to take decisions regarding
TLS blocking by using the response from the TH rather than using
questionable heuristics based on inspecting the TLSHandshake list
alone. This change should improve correctness _when_ we're using
the improved TH, which is currently used for 50% of the probes.
See https://github.com/ooni/probe/issues/2257
While there, modify `control.go` to specify which control is being used.
As silly as it seems, emojis help _a lot_ when eyeballing logs
to quickly identify unexpected lines.
I'm doing this work as part of https://github.com/ooni/probe/issues/2257
The residual censorship algorithm prevents dnscheck for checking the
same endpoint again in a short time frame under the assumption that a
previous measurement could have caused residual censorship.
In https://github.com/ooni/probe/issues/2234, we mentioned we probably
wanted to disable this algorithm because we didn't want to slow down
dnscheck too much and because we didn't detect this alleged source of
blocking in our previous measurements.
It turns out the algorithm was already disabled. I probably did this
after we published the paper on DNS censorship to make measurements
overall a bit faster.
So, just introduce a new extension field to the measurement telling us
that the residual censorship algorithm is disabled.
It's not super useful except as for the fact that the next time I
read the source code I notice that the algorithm is disabled.
Most of miniooni panics on errors. We should not panic on error with
--repeat-every, rather we should try the next measurement.
See https://github.com/ooni/probe/issues/2250
Part of https://github.com/ooni/probe/issues/2184, because I wanted
to allow swapping commands and options more freely.
As a side effect, this PR closes https://github.com/ooni/probe/issues/2248.
AFAICT, every usage that was legal before is still legal. What has
changed seems the freedom to swap commands and options and a much
better help that lists the available options.
This diff splits miniooni's implementation in smaller and more
easily tractable blocks ahead of future refactoring.
I'm trying to make `miniooni oonirun -i URL` as possible as
`miniooni -i URL oonirun`, because users typically expect this
kind of flexibity from modern Unix commands.
Part of https://github.com/ooni/probe/issues/2184
Closes https://github.com/ooni/probe/issues/2183
While there, avoid exposing nil values for optional fields of the
THResponse struct (i.e., "ip_info" and "tls_handshake").
While there, fix `measurexlite`'s `OperationLogger` test
and make it deterministic rather than racy.
This commit moves the TH structs and definitions to model. We don't want
oohelperd to depend on web_connectivity@v0.4.
Part of https://github.com/ooni/probe/issues/2240
A bunch of packages (including oohelperd) just need the ability to
use MaxMind-like databases. They don't need the additional functionality
implemented by the geolocate package. Such a package, in fact, is
mostly (if not only) needed by the engine package.
Therefore, move code to query MaxMind-like databases to a separate
package, and avoid depending on geolocate in all the packages for
which it's sufficient to use geoipx.
Part of https://github.com/ooni/probe/issues/2240
This diff introduces the following `oohelperd` enhancements:
1. measure both IP addresses resolved by the TH and IP addresses resolved by the probe;
2. when the URL scheme is http and there's no explicit port, measure both 80 and 443 (which will pay off big once we introduce support for optionally performing TLS handshakes);
3. include information about the probe and TH IP addresses into the results: who resolved each IP address, whether an address is a bogon, the ASN associated to an address.
This diff is part of https://github.com/ooni/probe/issues/2237
This diff refactors oohelperd to make performing additional measurements easier. We need:
1. to run the DNS task _before_ other tasks such that we can measure both IP addresses returned by the TH and the ones returned by the probe. When we'll introduce TLS measurements, this will allow us to validate probe-provided IP addresses inside the TH call. If probe-provided addresses work with TLS, they are legitimate for the domain.
2. to tie the number of TCP measurements to a list of endpoints collected by the probe _or_ the TH rather than just to the one provided by the probe. Anticipating this change, let us refactor how we read the results of the TCP task to make it independent of the number of addresses provided by the probe.
This work is part of https://github.com/ooni/probe/issues/2237
While working on https://github.com/ooni/probe/issues/2237, I noticed
there's no enforced timeout for measurement tasks.
So, this diff introduces the following timeouts:
1. use a 4 seconds timeout for the DNS lookup;
2. use a 10 seconds timeout for TCP;
3. use a 15 seconds timeout for HTTP.
They are a bit stricter than what we have on the probe because the TH
should supposedly have better bandwidth and connectivity.
See what we documented at https://github.com/ooni/spec/pull/257
Reference issue: https://github.com/ooni/probe/issues/2238
See also the related ooni/spec PR: https://github.com/ooni/spec/pull/257
See also https://github.com/ooni/probe/issues/2237
While there, bump webconnectivity@v0.5 version because this change
has an impact onto the generated data format.
The drop in coverage is unavoidable because we've written some
tests for `measurex` to ensure we deal with DNS resolvers and transport
names correctly depending on the splitting policy we use.
(However, `measurex` is only used for the `tor` experiment and, per
the step-by-step design document, new experiments should use
`measurexlite` instead, so this is hopefully fine(TM).)
While there, fix a broken integration test that does not run in `-short` mode.
In a pure step-by-step model, we don't need to trace HTTP round trips like we did before. We _may_ want in the future to also have some form of HTTP tracing (see https://github.com/ooni/probe-cli/pull/868 for a prototype) but doing that is currently not in scope for moving forward the step-by-step design. For this reason, I only added a public convenience function for formatting an OONI spec compatible request. I also added new fields, which should be documented inside the ooni/spec repository (see https://github.com/ooni/probe/issues/2238).
Required by https://github.com/ooni/probe/issues/2237
The T0 field is the moment when we started collecting data, while T
is the moment when we finished collecting data.
The TransactionID field will be repurposed for step-by-step measurements
to indicate related observations collected as part of the same flow
(e.g., TCP+TLS+HTTP).
Note that, for now, this change will only affect measurexlite and we're
not planning on changing other libraries for measuring.
Part of https://github.com/ooni/probe/issues/2137
* feat: save CNAME into archival data format
When a DNSResponse contains a non-empty CNAME, we include it
into the related list of answers.
Closes https://github.com/ooni/probe/issues/2227
* doc: add design note
While there, make code more compact and robust to a case where
we're going to extract additional answers.
* doc: document the expected growth of extraction function
Based on feedback by @DecFox
* feat(dnsovergetaddrinfo): collect the CNAME
This diff modifies how dnsovergetaddrinfo.go works such that the
returned DNSResponse includes the CNAME.
Closes https://github.com/ooni/probe/issues/2226.
While there, recognize that we can remove getaddrinfoLookupHost and
always call getaddrinfoLookupANY everywhere. (This simplification is
why we did https://github.com/ooni/probe-cli/pull/874.)
* fix: extra debugging because of failing CI
Everything is OK locally (on macOS). However, maybe things are a bit
different on GNU/Linux perhaps?
Here's the error:
```
--- FAIL: TestPass (0.11s)
resolver_test.go:113: unexpected rcode
FAIL
coverage: 95.7% of statements
FAIL github.com/ooni/probe-cli/v3/internal/cmd/jafar/resolver 0.242s
```
I'm a bit confused because jafar's resolver is _unrelated_. But actually this
error never occurred again after a committed the debugging diff.
* feat(netxlite): support extracting the CNAME
Closes https://github.com/ooni/probe/issues/2225
* fix(netxlite): attempt to increase coverage and improve tests
1. dnsovergetaddrinfo: specify the behavior of a DNSResponse returned
by this file to make it line with normal responses and write unit tests
to make sure we adhere to expectations;
2. dnsoverudp: make sure we wait to deferred responses also w/o a
custom context and post on a private channel and test that;
3. utls: recognize that we can actually write a test for NetConn and
what needs to change when we'll use go1.19 by default will just be
a cast that at that point can be removed.
* cleanup: remove UnderlyingNetworkLibrary and TProxy
While there, replace mixture of mocking and real connections inside
quicping with pure mocking of network connections.
Closes https://github.com/ooni/probe/issues/2224
* cleanup: we don't need a SimpleResolver now
This type was only used by UnderlyingNetworkLibrary and all the
rest of the code uses Resolver. So, let's avoid complexity by zapping
the SimpleResolver type and merging it inside Resolver.
Simone Basso
DecFox