This diff re-enables `E2E/miniooni.bash`. To make it working properly, we
needed to figure out which were the right cloudfronts to use.
I looked into the configuration and determined that both cloudfronts
should be used because they basically map to the same host.
I also determined it was backwards to test a mixture of prod and testing
APIs, and probably also flaky. So, I choose to only test the prod.
Additionally, I added support for testing all supported tunnels.
Closes https://github.com/ooni/probe/issues/2336
This diff adds to miniooni support for using the torsf tunnel. Such a
tunnel consists of a snowflake pluggable transport in front of a custom
instance of tor and requires tor to be installed.
The usage is like:
```
./miniooni --tunnel=torsf [...]
```
The default snowflake rendezvous method is "domain_fronting". You can
select the AMP cache instead using "amp":
```
./miniooni --snowflake-rendezvous=amp --tunnel=torsf [...]
```
Part of https://github.com/ooni/probe/issues/1955
We introduce the -f, --input-file FILE option with which we
are able to run an OONI Run v2 descriptor stored locally.
In this running mode, there are no checks related to whether the
descriptor has changed, since we're dealing with a local file.
Closes https://github.com/ooni/probe/issues/2328
As silly as it seems, emojis help _a lot_ when eyeballing logs
to quickly identify unexpected lines.
I'm doing this work as part of https://github.com/ooni/probe/issues/2257
Most of miniooni panics on errors. We should not panic on error with
--repeat-every, rather we should try the next measurement.
See https://github.com/ooni/probe/issues/2250
Part of https://github.com/ooni/probe/issues/2184, because I wanted
to allow swapping commands and options more freely.
As a side effect, this PR closes https://github.com/ooni/probe/issues/2248.
AFAICT, every usage that was legal before is still legal. What has
changed seems the freedom to swap commands and options and a much
better help that lists the available options.
This diff splits miniooni's implementation in smaller and more
easily tractable blocks ahead of future refactoring.
I'm trying to make `miniooni oonirun -i URL` as possible as
`miniooni -i URL oonirun`, because users typically expect this
kind of flexibity from modern Unix commands.
Part of https://github.com/ooni/probe/issues/2184
Closes https://github.com/ooni/probe/issues/2183
While there, avoid exposing nil values for optional fields of the
THResponse struct (i.e., "ip_info" and "tls_handshake").
While there, fix `measurexlite`'s `OperationLogger` test
and make it deterministic rather than racy.
This commit moves the TH structs and definitions to model. We don't want
oohelperd to depend on web_connectivity@v0.4.
Part of https://github.com/ooni/probe/issues/2240
A bunch of packages (including oohelperd) just need the ability to
use MaxMind-like databases. They don't need the additional functionality
implemented by the geolocate package. Such a package, in fact, is
mostly (if not only) needed by the engine package.
Therefore, move code to query MaxMind-like databases to a separate
package, and avoid depending on geolocate in all the packages for
which it's sufficient to use geoipx.
Part of https://github.com/ooni/probe/issues/2240
This diff introduces the following `oohelperd` enhancements:
1. measure both IP addresses resolved by the TH and IP addresses resolved by the probe;
2. when the URL scheme is http and there's no explicit port, measure both 80 and 443 (which will pay off big once we introduce support for optionally performing TLS handshakes);
3. include information about the probe and TH IP addresses into the results: who resolved each IP address, whether an address is a bogon, the ASN associated to an address.
This diff is part of https://github.com/ooni/probe/issues/2237
This diff refactors oohelperd to make performing additional measurements easier. We need:
1. to run the DNS task _before_ other tasks such that we can measure both IP addresses returned by the TH and the ones returned by the probe. When we'll introduce TLS measurements, this will allow us to validate probe-provided IP addresses inside the TH call. If probe-provided addresses work with TLS, they are legitimate for the domain.
2. to tie the number of TCP measurements to a list of endpoints collected by the probe _or_ the TH rather than just to the one provided by the probe. Anticipating this change, let us refactor how we read the results of the TCP task to make it independent of the number of addresses provided by the probe.
This work is part of https://github.com/ooni/probe/issues/2237
While working on https://github.com/ooni/probe/issues/2237, I noticed
there's no enforced timeout for measurement tasks.
So, this diff introduces the following timeouts:
1. use a 4 seconds timeout for the DNS lookup;
2. use a 10 seconds timeout for TCP;
3. use a 15 seconds timeout for HTTP.
They are a bit stricter than what we have on the probe because the TH
should supposedly have better bandwidth and connectivity.
* feat(dnsovergetaddrinfo): collect the CNAME
This diff modifies how dnsovergetaddrinfo.go works such that the
returned DNSResponse includes the CNAME.
Closes https://github.com/ooni/probe/issues/2226.
While there, recognize that we can remove getaddrinfoLookupHost and
always call getaddrinfoLookupANY everywhere. (This simplification is
why we did https://github.com/ooni/probe-cli/pull/874.)
* fix: extra debugging because of failing CI
Everything is OK locally (on macOS). However, maybe things are a bit
different on GNU/Linux perhaps?
Here's the error:
```
--- FAIL: TestPass (0.11s)
resolver_test.go:113: unexpected rcode
FAIL
coverage: 95.7% of statements
FAIL github.com/ooni/probe-cli/v3/internal/cmd/jafar/resolver 0.242s
```
I'm a bit confused because jafar's resolver is _unrelated_. But actually this
error never occurred again after a committed the debugging diff.
I made a mistake while adapting code from an experimental branch thus
breaking these two experiments because of interface conversion.
This diff fixes it.
While there, remove the panic trap for miniooni. Because miniooni is
an experimental tool, we want to see the full panic text, which definitely
leads to a more pleasant and effective debugging experience.
See https://github.com/ooni/probe/issues/2216 for context on why we
were trying to change how we register experiments.
The broken commit is 6a0ae5c70b.
Until OONI Run v2 has support for repeating the measurement with a schedule, introduce a command line flag requested by users to repeat a measurement every given number of seconds.
Part of https://github.com/ooni/probe/issues/2184
This diff adds support for running OONIRun v1 links.
Run with `miniooni` using:
```
./miniooni -i LINK oonirun
```
Part of https://github.com/ooni/probe/issues/2184
This diff refactors the ./internal/cmd/miniooni pkg and moves the code
for running experiments inside of the ./internal/oonirun pkg.
It's the first concrete step towards https://github.com/ooni/probe/issues/2184.
This option has been disabled for a long time and we said in the
codebase we were going to remove it after 2021-11-01.
So, it feels okay to remove it.
This diff is a cleanup in preparation for https://github.com/ooni/probe/issues/2184.
This diff refactors how we set options for experiments to accept
in input an any value or a map[string]any, depending on which method
we choose to actually set options.
There should be no functional change, except that now we're not
guessing the type and then attempting to set the value of the selected
field: now, instead, we match the provided type and the field's type
as part of the same function (i.e., SetOptionAny).
This diff is functional to https://github.com/ooni/probe/issues/2184,
because it will allow us to load options from a map[string]any,
which will be part of the OONI Run v2 JSON descriptor.
If we didn't apply this change, we would only have been to set options
from a map[string]string, which is good enough as a solution for the
CLI but is definitely clumsy when you have to write stuff like:
```JSON
{
"options": {
"HTTP3Enabled": "true"
}
}
```
when you could instead more naturally write:
```JSON
{
"options": {
"HTTP3Enabled": true
}
}
```
In https://github.com/ooni/probe-cli/pull/832's initial diff, I
mentioned it would be cool to flatten oohelperd's hier.
I'm doing this now, and just for the master branch.
This diff is mostly a mechanical refactoring with very light
and apparently rather safe manual changes.
This diff modifies the implementation of oohelperd in the master branch
to always use throw-away HTTPClient, Dialer, and Resolver.
The rationale of this change is to ensure we're not hitting limits of the
HTTPClient regarding the max number of connections per host.
This issue is described at https://github.com/ooni/probe/issues/2182.
While there, it feels more correct to use throw-away Dialer and Resolver.
We have a different patch for the release/3.15 branch because of
netx-related refactorings: https://github.com/ooni/probe-cli/pull/832.
The oohelperd implementation did not actually need using netx because
it was just constructing default types with logging, which is what
netxlite already does. Hence, let's avoid using netx here.
See https://github.com/ooni/probe/issues/2121
The oohelper does not need to use netx and it's enough to use
netxlite, hence let us apply this refactor.
The original code used DoT but the explanatory comment said we were
using DoT because of unclear issues inside GitHub actions.
We are now using DoH and this is fine as well. The comment implied
that any encrypted transport would do.
See https://github.com/ooni/probe/issues/2121
This diff refactors netx and netxlite to ensure we're not using
netxlite legacy names inside of netx.
To this end, we're cheating a bit. We're exposing a new factory to
get an unwrapped stdlib resolver rather than defining a legacy name
to export the private name of the same factory.
This is actually a fine place to stop, for now, the next and
netxlite refactoring at https://github.com/ooni/probe/issues/2121.
* refactor: move tracex outside of engine/netx
Consistently with https://github.com/ooni/probe/issues/2121 and
https://github.com/ooni/probe/issues/2115, we can now move tracex
outside of engine/netx. The main reason why this makes sense now
is that the package is now changed significantly from the one
that we imported from ooni/probe-engine.
We have improved its implementation, which had not been touched
significantly for quite some time, and converted it to unit
testing. I will document tomorrow some extra work I'd like to
do with this package but likely could not do $soon.
* go fmt
* regen tutorials
This diff modifies the system resolver to use a getaddrinf transport.
Obviously the transport is a fake, but its existence will allow us
to observe DNS events more naturally.
A lookup using the system resolver would be a ANY lookup that will
contain all the resolved IP addresses into the same response.
This change was also part of websteps-illustrated, albeit the way in
which I did it there was less clean than what we have here.
Ref issue: https://github.com/ooni/probe/issues/2096
This diff creates a new package under netx called tracex that
contains everything we need to perform measurements using events
tracing and postprocessing (which is the technique with which
we implement most network experiments).
The general idea here is to (1) create a unique package out of
all of these packages; (2) clean up the code a bit (improve tests,
docs, apply more recent code patterns); (3) move the resulting
code as a toplevel package inside of internal.
Once this is done, netx can be further refactored to avoid
subpackages and we can search for more code to salvage/refactor.
See https://github.com/ooni/probe/issues/2121
These two small packages could easily be merged into the model
package, since they're clearly model-like packages.
Part of https://github.com/ooni/probe/issues/2115
The objective is to make PR checks run much faster.
See https://github.com/ooni/probe/issues/2113 for context.
Regarding netxlite's tests:
Checking for every commit on master or on a release branch is
good enough and makes pull requests faster than one minute since
netxlite for windows is now 1m slower than coverage.
We're losing some coverage but coverage from integration tests
is not so good anyway, so I'm not super sad about this loss.
