Commit Graph

327 Commits

Author SHA1 Message Date
Simone Basso
a0dc65641d
refactor: pass experiment arguments using a struct (#983)
Closes https://github.com/ooni/probe/issues/2358.
2022-11-22 10:43:47 +01:00
Simone Basso
c2ea0b4704
feat(webconnectivity): try all the available THs (#980)
We introduce a fork of internal/httpx, named internal/httpapi, where there is a clear split between the concept of an API endpoint (such as https://0.th.ooni.org/) and of an API descriptor (such as using `GET` to access /api/v1/test-list/url).

Additionally, httpapi allows to create a SequenceCaller that tries to call a given API descriptor using multiple API endpoints. The SequenceCaller will stop once an endpoint works or when all the available endpoints have been tried unsuccessfully.

The definition of "success" is the following: we consider "failure" any error that occurs during the HTTP round trip or when reading the response body. We DO NOT consider "failure" errors (1) when parsing the input URL; (2) when the server returns >= 400; (3) when the server returns a string that does not parse as valid JSON. The idea of this classification of failures is that we ONLY want to retry when we see what looks like a network error that may be caused by (collateral or targeted) censorship.

We take advantage of the availability of this new package and we refactor web_connectivity@v0.4 and web_connectivity@v0.5 to use a SequenceCaller for calling the web connectivity TH API. This means that we will now try all the available THs advertised by the backend rather than just selecting and using the first one provided by the backend.

Because this diff is designed to be backported to the `release/3.16` branch, we have omitted additional changes to always use httpapi where we are currently using httpx. Yet, to remind ourselves about the need to do that, we have deprecated the httpx package. We will rewrite all the code currently using httpx to use httpapi as part of future work.

It is also worth noting that httpapi will allow us to refactor the backend code such that (1) we remove code to select a backend URL endpoint at the beginning and (2) we try several endpoints. The design of the code is such that we can add to the mix some endpoints using as `http.Client` a special client using a tunnel. This will allow us to automatically fallback backend queries.

Closes https://github.com/ooni/probe/issues/2353.

Related to https://github.com/ooni/probe/issues/1519.
2022-11-21 16:28:53 +01:00
DecFox
6b01264373
refactor(ooniprobe): migrate database to internal (#979)
See https://github.com/ooni/probe/issues/2352

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-11-15 10:35:30 +01:00
Arturo Filastò
606bc93ad2
Fix the signal root CA (#978)
See https://github.com/ooni/probe/issues/2344
2022-10-31 17:30:12 +01:00
Arturo Filastò
fa281f259a
Add new root CA to the signal test (#977)
All measurements collected since 2022-10-19 with previous versions
of OONI Probe will wrongly report sfu.voip.signal.org as blocked
as it switched to using a different root CA

This fixes: https://github.com/ooni/probe/issues/2344
2022-10-26 12:59:31 +02:00
Simone Basso
57a3919d2a
fix(geolocate): always use netxlite functionality (#976)
This change ensures that, in turn, we're able to "remote" all the traffic generated by the `geolocate` package, rather than missing some bits of it that were still using the standard library and caused _some_ geolocations to geolocate as the local host rather than as the remote host.

Extracted from https://github.com/ooni/probe-cli/pull/969, where we tested this functionality.

Closes https://github.com/ooni/probe/issues/1383 (which was long overdue).

Part of https://github.com/ooni/probe/issues/2340, because it allows us to make progress with that.
2022-10-12 18:07:42 +02:00
Simone Basso
0fc5d0e904
fix(E2E): ensure miniooni.bash is WAI (#972)
This diff re-enables `E2E/miniooni.bash`. To make it working properly, we
needed to figure out which were the right cloudfronts to use.

I looked into the configuration and determined that both cloudfronts
should be used because they basically map to the same host.

I also determined it was backwards to test a mixture of prod and testing
APIs, and probably also flaky. So, I  choose to only test the prod.

Additionally, I added support for testing all supported tunnels.

Closes https://github.com/ooni/probe/issues/2336
2022-10-08 13:14:11 +02:00
Simone Basso
18a9523496
feat(miniooni): implement torsf tunnel (#921)
This diff adds to miniooni support for using the torsf tunnel. Such a
tunnel consists of a snowflake pluggable transport in front of a custom
instance of tor and requires tor to be installed.

The usage is like:

```
./miniooni --tunnel=torsf [...]
```

The default snowflake rendezvous method is "domain_fronting". You can
select the AMP cache instead using "amp":

```
./miniooni --snowflake-rendezvous=amp --tunnel=torsf [...]
```

Part of https://github.com/ooni/probe/issues/1955
2022-10-03 16:52:20 +02:00
Simone Basso
89a584f93b
fix(go-build-alpine): honour OONI_PSIPHON_TAGS (#968)
Closes https://github.com/ooni/probe/issues/2334.

While there, reinstate integration tests, which were also lost in a previous refactoring. However, only run those tests for linux/amd64 because we can be confident that the Go compiler is WAI for all archs we support.

While there, always use bash for running end-to-end tests.

H/T @ainghazal for discovering and reporting this bug.
2022-10-03 11:55:47 +02:00
DecFox
d6a362d96f
feat: port-filtering experiment (#891)
Part of https://github.com/ooni/probe/issues/2005
2022-09-14 19:54:43 +02:00
Simone Basso
1638c450f0
refactor(engine): scrub the whole measurement (#956)
Part of https://github.com/ooni/probe/issues/2297
2022-09-12 22:22:25 +02:00
DecFox
f2b88ddb4a
feat: tlsmiddlebox experiment (#817)
See https://github.com/ooni/probe/issues/2124
2022-09-08 17:31:03 +02:00
Simone Basso
b78b9aca51
refactor(datafmt): use "udp" instead of "quic" (#946)
This diff changes the data format to prefer "udp" to "quic" everywhere we were previously using "quic".

Previously, the code inconsistently used "quic" for operations where we knew we were using "quic" and "udp" otherwise (e.g., for generic operations like ReadFrom).

While it would be more correct to say that a specific HTTP request used "quic" rather than "udp", using "udp" consistently allows one to see how distinct events such as ReadFrom and an handshake all refer to the same address, port, and protocol triple. Therefore, this change makes it easier to programmatically unpack a single measurement and create endpoint stats.

Before implementing this change, I discussed the problem with @hellais who mentioned that ooni/data is not currently using the "quic" string anywhere. I know that ooni/pipeline also doesn't rely on this string. The only users of this feature have been research-oriented experiments such as urlgetter, for which such a change would actually be acceptable.

See https://github.com/ooni/probe/issues/2238 and https://github.com/ooni/spec/pull/262.
2022-09-08 17:19:59 +02:00
Simone Basso
7df25795c0
fix(probeservices): use api.ooni.io (#926)
See https://github.com/ooni/probe/issues/2147.

Note that this PR also tries to reduce usage of legacy names inside unit/integration tests.
2022-09-02 16:48:14 +02:00
Simone Basso
1153850aca
cleanup: doh.powerdns.org is not working anymore (#924)
While there, `.../internal/sessionresolver` => `.../sessionresolver`

See https://github.com/ooni/probe/issues/2255
2022-09-02 14:44:23 +02:00
Simone Basso
ec73ae20b4
fix(dnscheck): trim the static input list (#923)
This should make the measurements faster since users have been
complaining about "experimental" taking too much time.

See https://github.com/ooni/probe/issues/2234
2022-09-02 13:54:59 +02:00
Simone Basso
7cdcf9f15b
fix(dnscheck): record whether residual is enabled (#922)
The residual censorship algorithm prevents dnscheck for checking the
same endpoint again in a short time frame under the assumption that a
previous measurement could have caused residual censorship.

In https://github.com/ooni/probe/issues/2234, we mentioned we probably
wanted to disable this algorithm because we didn't want to slow down
dnscheck too much and because we didn't detect this alleged source of
blocking in our previous measurements.

It turns out the algorithm was already disabled. I probably did this
after we published the paper on DNS censorship to make measurements
overall a bit faster.

So, just introduce a new extension field to the measurement telling us
that the residual censorship algorithm is disabled.

It's not super useful except as for the fact that the next time I
read the source code I notice that the algorithm is disabled.
2022-09-02 13:31:24 +02:00
Simone Basso
a68472dcee
dnscheck: bump patch version number (#919)
Part of https://github.com/ooni/probe/issues/2234
2022-09-01 19:23:46 +02:00
Simone Basso
cee89132af
fix(dnscheck): lower the default timeouts (#917)
Closes https://github.com/ooni/probe/issues/2234
2022-09-01 15:45:42 +02:00
Simone Basso
d0da224a2a
feat(oonirun): improve tests (#915)
See https://github.com/ooni/probe/issues/2184

While there, rename `runtimex.PanicIfFalse` to `runtimex.Assert` (it was about time...)
2022-08-31 18:40:27 +02:00
Simone Basso
7c1b2bbcb0
refactor: move WebGetTitle inside measurexlite (#895)
Part of https://github.com/ooni/probe/issues/2240
2022-08-28 20:26:40 +02:00
Simone Basso
bb6563f363
refactor: move TH structs and definitions to model (#894)
This commit moves the TH structs and definitions to model. We don't want
oohelperd to depend on web_connectivity@v0.4.

Part of https://github.com/ooni/probe/issues/2240
2022-08-28 20:20:12 +02:00
Simone Basso
110a11828b
refactor: spin geoipx off geolocate (#893)
A bunch of packages (including oohelperd) just need the ability to
use MaxMind-like databases. They don't need the additional functionality
implemented by the geolocate package. Such a package, in fact, is
mostly (if not only) needed by the engine package.

Therefore, move code to query MaxMind-like databases to a separate
package, and avoid depending on geolocate in all the packages for
which it's sufficient to use geoipx.

Part of https://github.com/ooni/probe/issues/2240
2022-08-28 20:00:25 +02:00
Simone Basso
1e7384d1cc
feat(oohelperd): measure TLS for :443 endpoints (#886)
This diff improves oohelperd to measure :443 endpoints with TLS.

Part of https://github.com/ooni/probe/issues/2237.
2022-08-28 14:34:40 +02:00
Simone Basso
df0e099b73
feat(oohelperd): follow (and record) TH and probe endpoints (#890)
This diff introduces the following `oohelperd` enhancements:

1. measure both IP addresses resolved by the TH and IP addresses resolved by the probe;

2. when the URL scheme is http and there's no explicit port, measure both 80 and 443 (which will pay off big once we introduce support for optionally performing TLS handshakes);

3. include information about the probe and TH IP addresses into the results: who resolved each IP address, whether an address is a bogon, the ASN associated to an address.

This diff is part of https://github.com/ooni/probe/issues/2237
2022-08-28 13:49:24 +02:00
Simone Basso
8a0c062844
feat: clearly indicate which resolver we're using (#885)
See what we documented at https://github.com/ooni/spec/pull/257

Reference issue: https://github.com/ooni/probe/issues/2238

See also the related ooni/spec PR: https://github.com/ooni/spec/pull/257

See also https://github.com/ooni/probe/issues/2237

While there, bump webconnectivity@v0.5 version because this change
has an impact onto the generated data format.

The drop in coverage is unavoidable because we've written some
tests for `measurex` to ensure we deal with DNS resolvers and transport
names correctly depending on the splitting policy we use.

(However, `measurex` is only used for the `tor` experiment and, per
the step-by-step design document, new experiments should use
`measurexlite` instead, so this is hopefully fine(TM).)

While there, fix a broken integration test that does not run in `-short` mode.
2022-08-27 15:47:48 +02:00
DecFox
0ef1f24617
feat: record delayed DNS responses in dnsping (#878)
See https://github.com/ooni/probe/issues/2231

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-08-25 13:59:24 +02:00
Simone Basso
da1c13e312
cleanup: remove UnderlyingNetworkLibrary and TProxy (#874)
* cleanup: remove UnderlyingNetworkLibrary and TProxy

While there, replace mixture of mocking and real connections inside
quicping with pure mocking of network connections.

Closes https://github.com/ooni/probe/issues/2224

* cleanup: we don't need a SimpleResolver now

This type was only used by UnderlyingNetworkLibrary and all the
rest of the code uses Resolver. So, let's avoid complexity by zapping
the SimpleResolver type and merging it inside Resolver.
2022-08-23 11:43:44 +02:00
Simone Basso
208ffa253b
fix: disable psiphon when building with go1.19 (#871)
Part of https://github.com/ooni/probe/issues/2211.

See also https://github.com/ooni/probe/issues/2222, which
describes the issue we have with psiphon and go1.19.
2022-08-22 11:50:58 +02:00
Simone Basso
9ffa124511
chore: upgrade deps and attempt to enable using go1.19 (#869)
* upgrade to our go.mod enabled of psiphon-tunnel-core such that
we're now using v2.0.24 of the tunnel-core;

* upgrade to the latest lucas-clemente/quic-go release;

* upgrade to the latest ooni/oohttp release (which is based on go1.19
but the diff seems good enough to continue using go1.18.x as well);

* upgrade to the latest ooni/oocrypto release (for which we can make the
same remarks regarding using go1.18.x);

* deal with changes in lucas-clemente/quic-go API as well as changes
in what a go1.19 *tls.Conn compatible type should look like.

Unfortunately, we cannot switch to go1.19 because psiphon forks quic-go
and their fork's still not building using such a version of go.

Part of ooni/probe#2211.
2022-08-19 11:26:50 +02:00
DecFox
097926c51f
refactor: allow automatically wrap net/quic conn (#867)
See https://github.com/ooni/probe/issues/2219
2022-08-17 20:58:06 +02:00
DecFox
e1d014e826
refactor(measurexlite): make buffered channels private (#864)
Closes https://github.com/ooni/probe/issues/2215
2022-08-17 20:10:48 +02:00
Simone Basso
6a0ae5c70b
refactor(engine): allow scripts to register experiments (#860)
See https://github.com/ooni/probe/issues/2216
2022-08-17 10:57:03 +02:00
DecFox
69602abe8a
refactor(simplequicping): use step-by-step (#852)
See https://github.com/ooni/probe/issues/2159 and https://github.com/ooni/spec/pull/254
2022-08-17 09:19:11 +02:00
DecFox
fc51590a67
feat: refactor dns implementation in measurexlite (#857)
* refactor: remove query-based mapping and introducing resolver wrapper

* refactor dnsping to adapt to measurexlite

* dnsping: extra comments

* Apply suggestions from code review

* Update internal/measurexlite/dns_test.go

See https://github.com/ooni/probe/issues/2208

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-08-11 16:00:37 +02:00
DecFox
5501b2201a
feat: dnsping using step-by-step (#831)
Reference issue for this pull request: https://github.com/ooni/probe/issues/2159

This diff refactors the `dnsping` experiment to use the [step-by-step measurement style](https://github.com/ooni/probe-cli/blob/master/docs/design/dd-003-step-by-step.md).

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-07-08 19:42:24 +02:00
Simone Basso
9a0153a349
feat(oonirun): add support for OONIRun v2 links (#844)
This diff adds support for OONIRun v2 links.

Part of https://github.com/ooni/probe/issues/2184.
2022-07-08 16:53:59 +02:00
Simone Basso
e5697e641e
fix(engine): repair broken integration test (#841)
The integration test that was broken was:

```
--- FAIL: TestCreateInvalidExperiment (0.35s)
    experiment_integration_test.go:192: expected a nil builder here
```

While there improve the documentation of the ExperimentSession
and see there's a method that we are not using.

This diff is a cleanup that I come up with while working
on https://github.com/ooni/probe/issues/2184.
2022-07-08 13:12:12 +02:00
Simone Basso
97864b324f
refactor(engine): more abstract Experiment{,Builder} (#838)
This diff modifies the engine package to make Experiment and
ExperimentBuilder interfaces rather than structs.

The previosuly existing structs are now named experiment{,Builder}.

This diff helps https://github.com/ooni/probe/issues/2184
because it allows us to write unit tests more easily.

There should be no functional change.

While there, I removed a bunch of deprecated functions, which were
unnecessarily complicate the implementation and could be easily
replaced by passing them a context.Context or context.Background().
2022-07-08 12:29:23 +02:00
Simone Basso
086ae43b15
refactor(engine): set options from any value (#837)
This diff refactors how we set options for experiments to accept
in input an any value or a map[string]any, depending on which method
we choose to actually set options.

There should be no functional change, except that now we're not
guessing the type and then attempting to set the value of the selected
field: now, instead, we match the provided type and the field's type
as part of the same function (i.e., SetOptionAny).

This diff is functional to https://github.com/ooni/probe/issues/2184,
because it will allow us to load options from a map[string]any,
which will be part of the OONI Run v2 JSON descriptor.

If we didn't apply this change, we would only have been to set options
from a map[string]string, which is good enough as a solution for the
CLI but is definitely clumsy when you have to write stuff like:

```JSON
{
  "options": {
    "HTTP3Enabled": "true"
  }
}
```

when you could instead more naturally write:

```JSON
{
  "options": {
    "HTTP3Enabled": true
  }
}
```
2022-07-08 11:51:59 +02:00
Simone Basso
6019b25baf
refactor(engine): *http.Client -> model.HTTPClient (#836)
This diff makes the implementation of the engine package more
abstract by changing HTTPClient() to return a model.HTTPClient
as opposed to returning an *http.Client.

Part of https://github.com/ooni/probe/issues/2184
2022-07-08 11:08:10 +02:00
Simone Basso
5ebdeb56ca
feat: tlsping and tcpping using step-by-step (#815)
## Checklist

- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/2158
- [x] if you changed anything related how experiments work and you need to reflect these changes in the ooni/spec repository, please link to the related ooni/spec pull request: https://github.com/ooni/spec/pull/250

## Description

This diff refactors the codebase to reimplement tlsping and tcpping
to use the step-by-step measurements style.

See docs/design/dd-003-step-by-step.md for more information on the
step-by-step measurement style.
2022-07-01 12:22:22 +02:00
Ain Ghazal
74aebedac3
doc(step-by-step): readability improvements (#820)
This diff contains readability improvements for the step-by-step design document.

Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-06-30 09:55:18 +02:00
Simone Basso
be2da83b1b
doc: publish the step-by-step design document (#814)
This pull request publishes the step-by-step design document that I have been discussing with @hellais and @DecFox recently. Compared to the document that was approved, this one has been edited for readability.

While there, I figured it was also be beneficial to publish the few ooni/probe-cli related design documents we produced in the past, because they probably help someone to get acquainted with the codebase.

Reference issue for this pull request: https://github.com/ooni/probe/issues/2148
2022-06-14 14:38:29 +02:00
Simone Basso
1685ef75b5
refactor(netxlite): expose useful HTTPTransport/DNSTransport factories (#813)
These factories will soon be useful to finish with
https://github.com/ooni/probe/issues/2135.
2022-06-09 00:30:18 +02:00
Simone Basso
1a706e47bc
refactor(netxlite): more abstract proxy-enabled dialer construction (#812)
This will help with https://github.com/ooni/probe/issues/2135
2022-06-08 23:10:06 +02:00
Simone Basso
bf7ea423d3
refactor(sessionresolver): replace dnsclientmaker with function (#811)
See https://github.com/ooni/probe/issues/2135
2022-06-08 22:29:01 +02:00
Simone Basso
a02cc6100b
refactor(sessionresolver): minor changes in files and types naming (#810)
Part of https://github.com/ooni/probe/issues/2135
2022-06-08 22:01:51 +02:00
Simone Basso
beba543b98
cleanup: use NewHTTPTransportWithResolver more often (#808)
We can simplify code in a bunch of places using a useful factory.

Part of https://github.com/ooni/probe/issues/2121.
2022-06-08 15:13:01 +02:00
Simone Basso
f1b8071c65
hotfix(sessionresolver): prevent data race inside http3 (#809)
See https://github.com/ooni/probe/issues/2135#issuecomment-1149840579
2022-06-08 15:06:15 +02:00