Commit Graph

702 Commits

Author SHA1 Message Date
Simone Basso
fe6d378a1f
chore: use {go,oohttp,oocrypto} v1.18.5 (#872)
* chore: use {go,oohttp,oocrypto} v1.18.5

This diff pins OONI to use go1.18.5 and oohttp and oocrypto at
go1.18.5 as the version of go with which we build releases.

The same codebase also works with go1.19 although this configuration
cannot include Psiphon (see https://github.com/ooni/probe/issues/2222).

Closes https://github.com/ooni/probe/issues/2223.

* fix: use oocrypto@v0.1.1

This ensures we keep https://github.com/ooni/probe/issues/2122 fixed.
2022-08-22 12:52:37 +02:00
Simone Basso
208ffa253b
fix: disable psiphon when building with go1.19 (#871)
Part of https://github.com/ooni/probe/issues/2211.

See also https://github.com/ooni/probe/issues/2222, which
describes the issue we have with psiphon and go1.19.
2022-08-22 11:50:58 +02:00
Simone Basso
9ffa124511
chore: upgrade deps and attempt to enable using go1.19 (#869)
* upgrade to our go.mod enabled of psiphon-tunnel-core such that
we're now using v2.0.24 of the tunnel-core;

* upgrade to the latest lucas-clemente/quic-go release;

* upgrade to the latest ooni/oohttp release (which is based on go1.19
but the diff seems good enough to continue using go1.18.x as well);

* upgrade to the latest ooni/oocrypto release (for which we can make the
same remarks regarding using go1.18.x);

* deal with changes in lucas-clemente/quic-go API as well as changes
in what a go1.19 *tls.Conn compatible type should look like.

Unfortunately, we cannot switch to go1.19 because psiphon forks quic-go
and their fork's still not building using such a version of go.

Part of ooni/probe#2211.
2022-08-19 11:26:50 +02:00
DecFox
097926c51f
refactor: allow automatically wrap net/quic conn (#867)
See https://github.com/ooni/probe/issues/2219
2022-08-17 20:58:06 +02:00
DecFox
e1d014e826
refactor(measurexlite): make buffered channels private (#864)
Closes https://github.com/ooni/probe/issues/2215
2022-08-17 20:10:48 +02:00
Ain Ghazal
d50a39ae92
feat: avoid safe options to be serialized into the measurement (#859)
Skip options that begin with the `Safe` prefix from appearing in the
serialization of a Measurement that will be submitted to the OONI
backend.

Fixes https://github.com/ooni/probe/issues/2214
2022-08-17 13:48:59 +02:00
Simone Basso
4e99e5030a
fix(registry): ensure we can run telegram and webconnectivity (#862)
I made a mistake while adapting code from an experimental branch thus
breaking these two experiments because of interface conversion.

This diff fixes it.

While there, remove the panic trap for miniooni. Because miniooni is
an experimental tool, we want to see the full panic text, which definitely
leads to a more pleasant and effective debugging experience.

See https://github.com/ooni/probe/issues/2216 for context on why we
were trying to change how we register experiments.

The broken commit is 6a0ae5c70b.
2022-08-17 12:17:52 +02:00
Simone Basso
6a0ae5c70b
refactor(engine): allow scripts to register experiments (#860)
See https://github.com/ooni/probe/issues/2216
2022-08-17 10:57:03 +02:00
DecFox
69602abe8a
refactor(simplequicping): use step-by-step (#852)
See https://github.com/ooni/probe/issues/2159 and https://github.com/ooni/spec/pull/254
2022-08-17 09:19:11 +02:00
DecFox
fbd7cd2b7e
feat: add support for system resolver in measurexlite (#858)
* feat: add support for system resolver in measurexlite

* more tests for coverage

* Apply suggestions from code review

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-08-11 18:18:25 +02:00
DecFox
fc51590a67
feat: refactor dns implementation in measurexlite (#857)
* refactor: remove query-based mapping and introducing resolver wrapper

* refactor dnsping to adapt to measurexlite

* dnsping: extra comments

* Apply suggestions from code review

* Update internal/measurexlite/dns_test.go

See https://github.com/ooni/probe/issues/2208

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-08-11 16:00:37 +02:00
DecFox
576b52b1e3
feat: collect system resolver results using context (#856)
* feat: Introduce context-based tracing to the system resolver

* testing: added tests for context-based tracing in netxlite resolvers

* Apply suggestions from code review

Reference issue: https://github.com/ooni/probe/issues/2207

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-08-11 10:20:28 +02:00
DecFox
5501b2201a
feat: dnsping using step-by-step (#831)
Reference issue for this pull request: https://github.com/ooni/probe/issues/2159

This diff refactors the `dnsping` experiment to use the [step-by-step measurement style](https://github.com/ooni/probe-cli/blob/master/docs/design/dd-003-step-by-step.md).

Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-07-08 19:42:24 +02:00
Simone Basso
8aad36a257
feat(miniooni): introduce the --repeat-every command line flag (#819)
Until OONI Run v2 has support for repeating the measurement with a schedule, introduce a command line flag requested by users to repeat a measurement every given number of seconds.

Part of https://github.com/ooni/probe/issues/2184
2022-07-08 17:04:31 +02:00
Simone Basso
9a0153a349
feat(oonirun): add support for OONIRun v2 links (#844)
This diff adds support for OONIRun v2 links.

Part of https://github.com/ooni/probe/issues/2184.
2022-07-08 16:53:59 +02:00
Simone Basso
ebb78c2848
feat(oonirun): implement OONIRun v1 (#843)
This diff adds support for running OONIRun v1 links.

Run with `miniooni` using:

```
./miniooni -i LINK oonirun
```

Part of https://github.com/ooni/probe/issues/2184
2022-07-08 15:17:52 +02:00
Simone Basso
0b4a49190a
feat: start sketching out the oonirun package (#842)
This diff refactors the ./internal/cmd/miniooni pkg and moves the code
for running experiments inside of the ./internal/oonirun pkg.

It's the first concrete step towards https://github.com/ooni/probe/issues/2184.
2022-07-08 14:20:49 +02:00
Simone Basso
e5697e641e
fix(engine): repair broken integration test (#841)
The integration test that was broken was:

```
--- FAIL: TestCreateInvalidExperiment (0.35s)
    experiment_integration_test.go:192: expected a nil builder here
```

While there improve the documentation of the ExperimentSession
and see there's a method that we are not using.

This diff is a cleanup that I come up with while working
on https://github.com/ooni/probe/issues/2184.
2022-07-08 13:12:12 +02:00
Simone Basso
a960ca51f0
cleanup(miniooni): remove --limit option (#840)
This option has been disabled for a long time and we said in the
codebase we were going to remove it after 2021-11-01.

So, it feels okay to remove it.

This diff is a cleanup in preparation for https://github.com/ooni/probe/issues/2184.
2022-07-08 12:42:02 +02:00
Simone Basso
97864b324f
refactor(engine): more abstract Experiment{,Builder} (#838)
This diff modifies the engine package to make Experiment and
ExperimentBuilder interfaces rather than structs.

The previosuly existing structs are now named experiment{,Builder}.

This diff helps https://github.com/ooni/probe/issues/2184
because it allows us to write unit tests more easily.

There should be no functional change.

While there, I removed a bunch of deprecated functions, which were
unnecessarily complicate the implementation and could be easily
replaced by passing them a context.Context or context.Background().
2022-07-08 12:29:23 +02:00
Simone Basso
5b27df1a37
measurexlite: fix flaky tls_test.go (#839)
Bug reported by @DecFox and subsequently observed in several
CI builds. No need to create an issue.
2022-07-08 12:29:08 +02:00
Simone Basso
086ae43b15
refactor(engine): set options from any value (#837)
This diff refactors how we set options for experiments to accept
in input an any value or a map[string]any, depending on which method
we choose to actually set options.

There should be no functional change, except that now we're not
guessing the type and then attempting to set the value of the selected
field: now, instead, we match the provided type and the field's type
as part of the same function (i.e., SetOptionAny).

This diff is functional to https://github.com/ooni/probe/issues/2184,
because it will allow us to load options from a map[string]any,
which will be part of the OONI Run v2 JSON descriptor.

If we didn't apply this change, we would only have been to set options
from a map[string]string, which is good enough as a solution for the
CLI but is definitely clumsy when you have to write stuff like:

```JSON
{
  "options": {
    "HTTP3Enabled": "true"
  }
}
```

when you could instead more naturally write:

```JSON
{
  "options": {
    "HTTP3Enabled": true
  }
}
```
2022-07-08 11:51:59 +02:00
Simone Basso
6019b25baf
refactor(engine): *http.Client -> model.HTTPClient (#836)
This diff makes the implementation of the engine package more
abstract by changing HTTPClient() to return a model.HTTPClient
as opposed to returning an *http.Client.

Part of https://github.com/ooni/probe/issues/2184
2022-07-08 11:08:10 +02:00
Simone Basso
d419ed8ac8
refactor(oohelperd): improve tests implementation (#835)
After this diff has landed, we have addressed all the points
originally published at https://github.com/ooni/probe/issues/2134.
2022-07-05 20:25:18 +02:00
Simone Basso
535a5d3e00
refactor(oohelperd): flatten package hierarchy (#834)
In https://github.com/ooni/probe-cli/pull/832's initial diff, I
mentioned it would be cool to flatten oohelperd's hier.

I'm doing this now, and just for the master branch.

This diff is mostly a mechanical refactoring with very light
and apparently rather safe manual changes.
2022-07-05 19:10:39 +02:00
Simone Basso
a4d17085f5
fix(oohelperd): use throw-away HTTPClient, Dialer, Resolver (#833)
This diff modifies the implementation of oohelperd in the master branch
to always use throw-away HTTPClient, Dialer, and Resolver.

The rationale of this change is to ensure we're not hitting limits of the
HTTPClient regarding the max number of connections per host.

This issue is described at https://github.com/ooni/probe/issues/2182.

While there, it feels more correct to use throw-away Dialer and Resolver.

We have a different patch for the release/3.15 branch because of
netx-related refactorings: https://github.com/ooni/probe-cli/pull/832.
2022-07-05 18:41:35 +02:00
Simone Basso
5ebdeb56ca
feat: tlsping and tcpping using step-by-step (#815)
## Checklist

- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/2158
- [x] if you changed anything related how experiments work and you need to reflect these changes in the ooni/spec repository, please link to the related ooni/spec pull request: https://github.com/ooni/spec/pull/250

## Description

This diff refactors the codebase to reimplement tlsping and tcpping
to use the step-by-step measurements style.

See docs/design/dd-003-step-by-step.md for more information on the
step-by-step measurement style.
2022-07-01 12:22:22 +02:00
Ain Ghazal
74aebedac3
doc(step-by-step): readability improvements (#820)
This diff contains readability improvements for the step-by-step design document.

Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-06-30 09:55:18 +02:00
Simone Basso
28585fbdc2
doc: mention step-by-step design document
This commit changes the README of `./internal` to make
sure we're referring to the step-by-step design doc.
2022-06-17 11:02:54 +02:00
Simone Basso
c3fb7308d8
doc: mention how to generate a dependency graph
Thanks @ainghazal for mentioning this useful tool to me!
2022-06-17 10:56:47 +02:00
Simone Basso
be2da83b1b
doc: publish the step-by-step design document (#814)
This pull request publishes the step-by-step design document that I have been discussing with @hellais and @DecFox recently. Compared to the document that was approved, this one has been edited for readability.

While there, I figured it was also be beneficial to publish the few ooni/probe-cli related design documents we produced in the past, because they probably help someone to get acquainted with the codebase.

Reference issue for this pull request: https://github.com/ooni/probe/issues/2148
2022-06-14 14:38:29 +02:00
Simone Basso
1685ef75b5
refactor(netxlite): expose useful HTTPTransport/DNSTransport factories (#813)
These factories will soon be useful to finish with
https://github.com/ooni/probe/issues/2135.
2022-06-09 00:30:18 +02:00
Simone Basso
1a706e47bc
refactor(netxlite): more abstract proxy-enabled dialer construction (#812)
This will help with https://github.com/ooni/probe/issues/2135
2022-06-08 23:10:06 +02:00
Simone Basso
bf7ea423d3
refactor(sessionresolver): replace dnsclientmaker with function (#811)
See https://github.com/ooni/probe/issues/2135
2022-06-08 22:29:01 +02:00
Simone Basso
a02cc6100b
refactor(sessionresolver): minor changes in files and types naming (#810)
Part of https://github.com/ooni/probe/issues/2135
2022-06-08 22:01:51 +02:00
Simone Basso
beba543b98
cleanup: use NewHTTPTransportWithResolver more often (#808)
We can simplify code in a bunch of places using a useful factory.

Part of https://github.com/ooni/probe/issues/2121.
2022-06-08 15:13:01 +02:00
Simone Basso
f1b8071c65
hotfix(sessionresolver): prevent data race inside http3 (#809)
See https://github.com/ooni/probe/issues/2135#issuecomment-1149840579
2022-06-08 15:06:15 +02:00
Simone Basso
fe29b432e0
refactor: start refactoring session resolver (#807)
This diff addresses the following points of https://github.com/ooni/probe/issues/2135:

- [x] the `childResolver` type is useless and we can use `model.Resolver` directly;
- [x] we should use `model/mocks` instead of custom fakes;
- [x] we should not use `log.Log` rather we should use `model.DiscardLogger`;
- [x] make `timeLimitedLookup` easier to test with a `-short` tests;
- [x] ensure `timeLimitedLookup` returns as soon as its context expires regardless of the child resolver;

Subsequent diffs will address more points mentioned in there.
2022-06-08 14:06:22 +02:00
Simone Basso
dea23b49d5
refactor(oohelperd): use netxlite rather than netx (#806)
The oohelperd implementation did not actually need using netx because
it was just constructing default types with logging, which is what
netxlite already does. Hence, let's avoid using netx here.

See https://github.com/ooni/probe/issues/2121
2022-06-08 10:14:10 +02:00
Simone Basso
87d35f4225
refactor(oohelper): use netxlite rather than netx (#805)
The oohelper does not need to use netx and it's enough to use
netxlite, hence let us apply this refactor.

The original code used DoT but the explanatory comment said we were
using DoT because of unclear issues inside GitHub actions.

We are now using DoH and this is fine as well. The comment implied
that any encrypted transport would do.

See https://github.com/ooni/probe/issues/2121
2022-06-08 09:52:47 +02:00
Simone Basso
4cf58380c3
[forwardport] fix(stunreachability): don't emit spurious progress events (#803)
This diff forward ports 4fb5f7de69b900cfc2bd211f723108a67bda350d to master.

See https://github.com/ooni/probe/issues/2058#issuecomment-1145847069

This diff WILL need to be forwardported to master.
2022-06-06 16:21:17 +02:00
Simone Basso
504181906a
[forwardport] fix(dnscheck): remove apparently-broken static input (#804)
This diff forward ports 261d1a4cdc88522f6a8f63d6c540f51054566b28 to master
whose original commit message follows:

- - -

It's not working for me from a couple of places and also it does not
seem to be documented upstream, see:

https://docs.namebase.io/guides-1/resolving-handshake-1/hdns.io

This diff WILL need to be forwardported to master.
2022-06-06 16:08:09 +02:00
Simone Basso
57e207e644
doc(netx): reference issue mentioning future improvements (#802)
See https://github.com/ooni/probe/issues/2121#issuecomment-1147424810
2022-06-06 15:16:30 +02:00
Simone Basso
2502a237fb
cleanup: netx does not use netxlite legacy names (#801)
This diff refactors netx and netxlite to ensure we're not using
netxlite legacy names inside of netx.

To this end, we're cheating a bit. We're exposing a new factory to
get an unwrapped stdlib resolver rather than defining a legacy name
to export the private name of the same factory.

This is actually a fine place to stop, for now, the next and
netxlite refactoring at https://github.com/ooni/probe/issues/2121.
2022-06-06 14:46:44 +02:00
Simone Basso
64bffbd941
refactor(netx): reorganize by topic (#800)
Before finishing the ongoing refactoring and leaving whatever
is left of netx in tree, I would like to restructure it so that
we'll have an easy time next time we need to modify it.

Currently, every functionality lives into the `netx.go` file and
we have a support file called `httptransport.go`.

I would like to reorganize by topic, instead. This would allow
future me to more easily perform topic-specific changes.

While there, improve `netx`'s documentation and duplicate some of
this documentation inside `internal/README.md` to provide pointers
to previous documentation, historical context, and some help to
understand the logic architecture of network extensions (aka `netx`).

Part of https://github.com/ooni/probe-cli/pull/396
2022-06-06 14:27:25 +02:00
Simone Basso
5d54aa9c5f
cleanup: move caching resolvers from netx to netxlite (#799)
Now that we have properly refactored the caching resolvers we can
move them into netxlite as optional resolvers created using the
proper abstract factories we just added.

This diff reduces the complexity and the code size of netx.

See https://github.com/ooni/probe/issues/2121.
2022-06-05 21:58:34 +02:00
Simone Basso
6b85dfce88
refactor(netx): move construction logic outside package (#798)
For testability, replace most if-based construction logic with
calls to well-tested factories living in other packages.

While there, acknowledge that a bunch of types could now be private
and make them private, modifying the code to call the public
factories allowing to construct said types instead.

Part of https://github.com/ooni/probe/issues/2121
2022-06-05 21:22:27 +02:00
Simone Basso
2d3d5d9cdc
cleanup(netx): stop using most netxlite resolver internals (#797)
This diff modifies netx to stop using most netxlite resolver internals
but the internal function that creates a new, unwrapped system resolver,
which will be dealt with in a subsequent pull request.

See https://github.com/ooni/probe/issues/2121
2022-06-05 19:52:39 +02:00
Simone Basso
07c0b08505
cleanup(netxlite): drop the DefaultDialer legacy name (#796)
Part of https://github.com/ooni/probe/issues/2121
2022-06-05 18:44:17 +02:00
Simone Basso
c6b3889a33
fix(netx): ensure we create ~same HTTP3 and HTTP2 transports (#795)
1. Use the netxlite.NewHTTPTransport factory for creating a new
HTTP2 (and HTTP1) transport;

2. Recognize the netxlite.NewOOHTTPTransport has now become
an implementation detail so make it private;

3. Recognize that netxlite.NewHTTP3Transport should call
netxlite.WrapTransport so it returns the same typechain
returned by netxlite.NewHTTPTransport (modulo, of course,
the real underlying transport), so ensure that we are
calling netxlite.WrapTransport in NewHTTP3Transport;

4. Recognize that the table based constructor inside of
netx needs a logger to create HTTPTransport instances using
either netxlite.NewHTTP{,3}Transport so pass this argument
along and ensure it's not nil using a constructor inside
model that guarantees that;

5. Cleanup netx's tests to avoid type asserting on the
typechains returned by netxlite since we already test
that inside netxlite;

6. Recognize that now we can make more legacy names inside
of netxlite private because we don't need to use them
inside tests anymore (because of previous point).

Reference issue: https://github.com/ooni/probe/issues/2121
2022-06-05 17:41:06 +02:00
Simone Basso
d5249a6cf7
chore: improve testing and increase coverage (#794)
This diff improves testing and increases coverage inside the
./internal/netxlite and ./internal/tracex packages.

See https://github.com/ooni/probe/issues/2121
2022-06-04 14:58:48 +02:00
Simone Basso
15da0f5344
cleanup(jafar): do not depend on netx and urlgetter (#792)
There's no point in doing that. Also, once this change is merged, it becomes easier to cleanup/simplify netx.

See https://github.com/ooni/probe/issues/2121
2022-06-02 22:25:37 +02:00
Simone Basso
76b65893a1
cleanup(netx): remove redundant config options (#791)
Part of https://github.com/ooni/probe/issues/2121
2022-06-02 18:18:49 +02:00
Simone Basso
e9ed733f07
refactor(netx): use netxlite to build TLSDialer (#790)
This diff modifies netx to use netxlite to build the TLSDialer.

Building the TLSDialer entails building a TLSHandshaker.

While there, hide netxlite names we don't want to be public
and change netx tests to test for functionality.

To this end, refactor filtering to provide an easier to
use TLS server. We don't need the complexity of proxying
rather we need to provoke specific errors.

Part of https://github.com/ooni/probe/issues/2121
2022-06-02 17:39:48 +02:00
Simone Basso
ae24ba644c
cleanup(netx): another batch of small/simple cleanups (#789)
See https://github.com/ooni/probe/issues/2121
2022-06-02 13:50:34 +02:00
Simone Basso
1cb820b19d
cleanup(netx): remove subpackages and unnecessary code (#788)
This pull request consists of several small and obvious cleanups in the netx directory.

See https://github.com/ooni/probe/issues/2121
2022-06-02 11:51:21 +02:00
Simone Basso
9354191b85
refactor(tracex): internally store just the raw certificate (#787)
By just storing the raw certificate we simplify the internal data
structure we use. In turn, this enables us to write better unit tests
using github.com/google/go-cmp where we can construct the expected
result and compare with that. (Yeah, in principle we could also
construct the full certificate but I'm not sure it's worth the effort
since we basically only care about the raw certificate.)

The general idea here is to make tracex more tested. Once it's more
tested, I will create separate structs for each event, which is
something that measurex also does. Once that is done, we can start
ensuring that the code in measurex and the code in tracex do the
same thing in terms of storing observations. When also this is done,
we can then rewrite measurex to use tracex directly.

The overall goal is https://github.com/ooni/probe/issues/2035.
2022-06-02 11:07:02 +02:00
Simone Basso
83e3167ce2
refactor(tracex): internally represent errors as strings (#786)
There are two reasons why this is beneficial:

1. github.com/google/go-cmp is more annoying to use for comparing
data structures when there are interfaces to compare. Sure, there's
a recipe for teaching it to compare errors, but how about making
the errors trivially comparable instead?

2. if we want to send errors over the network, JSON serialization
works but we cannot unmarshal the resulting string back to an error,
so how about making this representation trivial to serialize (we
are not going this now, but we need this property for websteps and
it may be sensible to try to avoid to have duplicate code because
of that -- measurex currently duplicates many tracex functionality
and this is quite unfortunate because it slows development down)

Additionally, if an error is a string:

3. we can very easily use a switch for comparing its possible
values with "" representing the absence of errors, while it is
more complex to do the same when using a nullable string or even
an error (i.e., an interface)

4. if a type is not nullable, it's easier to write safe code for
it and we may want to refactor experiments to use the internal
representation of measurements for more robust processing code

For all these reasons, let's internally use strings in tracex.

The overall aim here is to reduce the duplicated code between pre
and post-measurex measurements (see https://github.com/ooni/probe/issues/2035).
2022-06-02 10:37:07 +02:00
Simone Basso
2556e93050
[forwardport] fix(dnscheck): stop emitting progress (#785)
This diff forward ports b606494db8a9293384efaf5c33a88601f6e1e2a6
to the main development branch.

Dnscheck is emitting progress and the experiment controller is
also emitting progress. This messes up the progress bar.

See https://github.com/ooni/probe/issues/2058#issuecomment-1141638067
2022-06-02 09:31:52 +02:00
Simone Basso
b58cfadb39
hotfix: disable oocrypto until we investigate ciphers selection (#784)
See https://github.com/ooni/probe/issues/2122 for context.
2022-06-02 08:52:15 +02:00
Simone Basso
58adb68b2c
refactor: move tracex outside of engine/netx (#782)
* refactor: move tracex outside of engine/netx

Consistently with https://github.com/ooni/probe/issues/2121 and
https://github.com/ooni/probe/issues/2115, we can now move tracex
outside of engine/netx. The main reason why this makes sense now
is that the package is now changed significantly from the one
that we imported from ooni/probe-engine.

We have improved its implementation, which had not been touched
significantly for quite some time, and converted it to unit
testing. I will document tomorrow some extra work I'd like to
do with this package but likely could not do $soon.

* go fmt

* regen tutorials
2022-06-02 00:50:55 +02:00
Simone Basso
d397036073
refactor(tracex): convert to unit testing (#781)
The exercise already allowed me to notice issues such as fields not
being properly initialized by savers.

This is one of the last steps before moving tracex away from the
internal/netx package and into the internal package.

See https://github.com/ooni/probe/issues/2121
2022-06-01 23:15:47 +02:00
Simone Basso
6212daa54a
fix(tracex): generate archival from single transaction-done event (#780)
Tracex contained some fragile code that assembled HTTP measurements
from scattered events, which worked because we were sure we were
performing a single measurement at any given time.

This diff restructures the code to emit a transaction-start and a
transaction-done events only. We have basically removed all the other
events (which we were not using). We kept the transaction-start
though, because it may be useful to see it when reading events. In
any case, what matters here is that we're now using the transaction-done
event aline to generate the archival HTTP measurement.

Hence, the original issue has been addressed. We will possibly
do more refactoring in the future, but for now this seems sufficient.

Part of https://github.com/ooni/probe/issues/2121
2022-06-01 19:27:47 +02:00
Simone Basso
66fd1569b8
tracex: prepare HTTP code for future refactoring (#778)
The main issue I see inside tracex at the moment is that we
construct the HTTP measurement from separate events.

This is fragile because we cannot be sure that these events
belong to the same round trip. (Currently, they _are_ part
of the same round trip, but this is a fragile assumption and
it would be much more robust to dispose of it.)

To prepare for emitting a single event, it's imperative to
have two distinct fields for HTTP request and response headers,
which is the main contribution in this commit.

Then, we have a bunch of smaller changes including:

1. correctly naming 'response' the DNS response (instead of 'reply')

2. ensure we always use pointer receivers

Reference issue: https://github.com/ooni/probe/issues/2121
2022-06-01 15:20:28 +02:00
Simone Basso
c740be987b
refactor(tracex): do not depend on strings for event names (#777)
Rather than matching a string, match a type.

This is more robust considering future refactorings.

We're confident the names did not change in _this_ refactoring
because we're still testing the same strings in the tests.

Part of https://github.com/ooni/probe/issues/2121
2022-06-01 14:32:16 +02:00
Simone Basso
8f7e3803eb
feat(netxlite): implement DNSTransport wrapping (#776)
Acknowledge that transports MAY be used in isolation (i.e., outside
of a Resolver) and add support for wrapping.

Ensure that every factory that creates an unwrapped type is named
accordingly to hopefully ensure there are no surprises.

Implement DNSTransport wrapping and use a technique similar to the
one used by Dialer to customize the DNSTransport while constructing
more complex data types (e.g., a specific resolver).

Ensure that the stdlib resolver's own "getaddrinfo" transport (1)
is wrapped and (2) could be extended during construction.

This work is part of my ongoing effort to bring to this repository
websteps-illustrated changes relative to netxlite.

Ref issue: https://github.com/ooni/probe/issues/2096
2022-06-01 11:10:08 +02:00
Simone Basso
923d81cdee
refactor(netxlite): introduce the getaddrinfo transport (#775)
This diff modifies the system resolver to use a getaddrinf transport.

Obviously the transport is a fake, but its existence will allow us
to observe DNS events more naturally.

A lookup using the system resolver would be a ANY lookup that will
contain all the resolved IP addresses into the same response.

This change was also part of websteps-illustrated, albeit the way in
which I did it there was less clean than what we have here.

Ref issue: https://github.com/ooni/probe/issues/2096
2022-06-01 09:59:44 +02:00
Simone Basso
7e0b47311d
refactor(netxlite): better integration with tracex (#774)
Rather than passing functions to construct complex objects such
as Dialer and QUICDialer, pass interface implementations.

Ensure that a nil implementation does not cause harm.

Make Saver implement the correct interface either directly or
indirectly. We need to implement the correct interface indirectly
for TCP conns (or connected UDP sockets) because we have two
distinct use cases inside netx: observing just the connect event
and observing just the I/O events.

With this change, the construction of composed Dialers and
QUICDialers is greatly simplified and more obvious.

Part of https://github.com/ooni/probe/issues/2121
2022-06-01 08:31:20 +02:00
Simone Basso
f4f3ed7c42
refactor(tracex): start applying recent code conventions (#773)
The code that is now into the tracex package was written a long
time ago, so let's start to make it more in line with the coding
style of packages that were written more recently.

I didn't apply all the changes I'd like to apply in a single diff
and for now I am committing just this diff.

Broadly, what we need to do is:

1. improve documentation

2. ~always use pointer receivers (object receives have the issue
that they are not mutable by accident meaning that you can mutate
them but their state do not change after the call returns, which
is potentially a source of bugs in case you later refactor to use
a pointer receiver, so always use pointer receivers)

3. ~always avoid embedding (let's say we want to avoid embedding
for types we define and it's instead fine to embed types that are
defined in the stdlib: if later we add a new method, we will not
see a broken build and we'll probably forget to add the new method
to all wrappers -- conversely, if we're wrapping rather than
embedding, we'll see a broken build and act accordingly)

4. prefer unit tests and group tests by type being tested rather
than using a flat structure for tests

There's a coverage slippage that I'll compensate in a follow-up diff where I'll focus on unit testing.

Reference issue: https://github.com/ooni/probe/issues/2121
2022-06-01 07:44:54 +02:00
Simone Basso
bbcd2e2280
refactor(netx): merge archival, trace, and the savers (#772)
This diff creates a new package under netx called tracex that
contains everything we need to perform measurements using events
tracing and postprocessing (which is the technique with which
we implement most network experiments).

The general idea here is to (1) create a unique package out of
all of these packages; (2) clean up the code a bit (improve tests,
docs, apply more recent code patterns); (3) move the resulting
code as a toplevel package inside of internal.

Once this is done, netx can be further refactored to avoid
subpackages and we can search for more code to salvage/refactor.

See https://github.com/ooni/probe/issues/2121
2022-05-31 21:53:01 +02:00
Simone Basso
dd5655eaee
refactor(netxlite): allow easy QUIC dialer chain customization (#771)
Like the previous diff, but for creating QUIC dialers.

See https://github.com/ooni/probe/issues/2121.
2022-05-31 20:28:25 +02:00
Simone Basso
69fd0c5119
refactor(netxlite): allow easy dialer chain customization (#770)
This diff modifies the construction of a dialer to allow one
to insert custom dialer wrappers into the dialers chain.

The point of the chain in which we allow custom wrappers is the
optimal one for connect, read, and write measurements.

This new design is better than the previous netx design since
we don't need to construct the whole chain manually now.

The work in this diff is part of the effort to make engine/netx
just a tiny wrapper around netxlite.

See https://github.com/ooni/probe/issues/2121.
2022-05-31 20:02:11 +02:00
Simone Basso
e4f10eeac2
refactor: continue to simplify engine/netx (#769)
The objective of this diff is to simplify the code inside engine/netx
while moving more bits of code inside netxlite.

See https://github.com/ooni/probe/issues/2121
2022-05-31 08:11:07 +02:00
Simone Basso
3265bc670a
refactor(ndt7): use netxlite rather than netx (#768)
This diff required us to move some code around, but no major
change actually happened, except better tests.

While there, I also slightly refactored ndt7's implementation and
removed the ProxyURL setting, which was actually unused.

See https://github.com/ooni/probe/issues/2121
2022-05-30 23:14:07 +02:00
Simone Basso
314c3c934d
refactor(session.go): replace engine/netx with netxlite (#767)
This diff replaces engine/netx code with netxlite code in
the engine/session.go file. To this end, we needed to move
some code from engine/netx to netxlite. While there, we
did review and improve the unit tests.

A notable change in this diff is (or seems to be) that in
engine/session.go we're not filtering for bogons anymore so
that, in principle, we could believe a resolver returning
to us bogon IP addresses for OONI services. However, I did
not bother with changing bogons filtering because the
sessionresolver package is already filtering for bogons,
so it is actually okay to avoid doing that again the
session.go code. See:

https://github.com/ooni/probe-cli/blob/v3.15.0-alpha.1/internal/engine/internal/sessionresolver/resolvermaker.go#L88

There are two reference issues for this cleanup:

1. https://github.com/ooni/probe/issues/2115

2. https://github.com/ooni/probe/issues/2121
2022-05-30 22:00:45 +02:00
Simone Basso
595d0744db
netxlite: do not call netgo the CGO_ENABLED=0 resolver (#766)
In https://github.com/ooni/probe/issues/2029#issuecomment-1140805266, we
explained why calling it "netgo" would be incorrect.

In other words, we can get the platform's `getaddrinfo` as long as
we're not cross compiling. We do cross compile `ooniprobe`, actually
it's not even possible to cross compile it.

For increased accuracy, we should stop cross compiling `miniooni`
as well, so it would also directly use `getaddrinfo`.

This diff fixes at the same time ooni/probe-cli and ooni/spec
and we'll open two pull requests in parallel.
2022-05-30 10:06:53 +02:00
Simone Basso
f3912188e1
getaddrinfo: fix CGO_ENABLED=0 and record resolver type (#765)
After https://github.com/ooni/probe-cli/pull/764, the build for
CGO_ENABLED=0 has been broken for miniooni:

https://github.com/ooni/probe-cli/runs/6636995859?check_suite_focus=true

Likewise, it's not possible to run tests with CGO_ENABLED=0.

To make tests work with `CGO_ENABLED=0`, I needed to sacrifice some
unit tests run for the CGO case. It is not fully clear to me what was happening
here, but basically `getaddrinfo_cgo_test.go` was compiled with CGO
being disabled, even though the ``//go:build cgo` flag was specified.

Additionally, @hellais previously raised a valid point in the review
of https://github.com/ooni/probe-cli/pull/698:

> Another issue we should consider is that, if I understand how
> this works correctly, depending on whether or not we have built
> with CGO_ENABLED=0 on or not, we are going to be measuring
> things in a different way (using our cgo inspired getaddrinfo
> implementation or using netgo). This might present issues when
> analyzing or interpreting the data.
>
> Do we perhaps want to add some field to the output data format that
> gives us an indication of which DNS resolution code was used to
> generate the the metric?

This comment is relevant to the current commit because
https://github.com/ooni/probe-cli/pull/698 is the previous
iteration of https://github.com/ooni/probe-cli/pull/764.

So, while fixing the build and test issues, let us also distinguish
between the CGO_ENABLED=1 and CGO_ENABLED=0 cases.

Before this commit, OONI used "system" to indicate the case where
we were using net.DefaultResolver. This behavior dates back to the
Measurement Kit days. While it is true that ooni/probe-engine and
ooni/probe-cli could have been using netgo in the past when we
said "system" as the resolver, it also seems reasonable to continue
to use "system" top indicate getaddrinfo.

So, the choice here is basically to use "netgo" from now on to
indicate the cases in which we were built with CGO_ENABLED=0.

This change will need to be documented into ooni/spec along with
the introduction of the `android_dns_cache_no_data` error.

## Checklist

- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/2029
- [x] if you changed anything related how experiments work and you need to reflect these changes in the ooni/spec repository, please link to the related ooni/spec pull request: https://github.com/ooni/spec/pull/242
2022-05-30 07:34:25 +02:00
Simone Basso
cf6dbe48e0
netxlite: call getaddrinfo and handle platform-specific oddities (#764)
This commit changes our system resolver to call getaddrinfo directly when CGO is enabled. This change allows us to:

1. obtain the CNAME easily

2. obtain the real getaddrinfo retval

3. handle platform specific oddities such as `EAI_NODATA`
returned on Android devices

See https://github.com/ooni/probe/issues/2029 and https://github.com/ooni/probe/issues/2029#issuecomment-1140258729 in particular.

See https://github.com/ooni/probe/issues/2033 for documentation regarding the desire to see `getaddrinfo`'s retval.

See https://github.com/ooni/probe/issues/2118 for possible follow-up changes.
2022-05-28 15:10:30 +02:00
Simone Basso
62bd62ece1
fix(dnsoverudp): allow to cancel async round trip immediately (#763)
To this end, we need to refactor the implementation to give the
DNSOverUDPChannel owenership over the net.Conn.

Once this happens, DNSOverUDPChannel.Close closes the conn.

When the conn is closed, the background goroutine will terminate
immediately because any blocking I/O operation will be immediately
unblocked and return net.ErrClosed.

See https://github.com/ooni/probe/issues/2099#issuecomment-1139066946
2022-05-26 23:49:14 +02:00
Simone Basso
16f7407b13
feat(netxlite): observe additional DNS-over-UDP responses (#762)
This diff introduces support for observing additional DNS-over-UDP
responses in some censored environments (e.g. China).

After some uncertainty around whether to use connected or unconnected
UDP sockets, I eventually settled for connected.

Here's a recap:

|                         | connected | unconnected |
| ----------------------- | --------- | ----------- |
| see ICMP errors         | ✔️         |           |
| responses from any server |         | ✔️           |

Because most if not all DNS resolvers expect answers from exactly
the same servers to which they sent the query, I would say that
it's more important to have some limited ability of observing the
effect of ICMP errors (e.g., host_unreachable when we set a low
TTL and send out a query to a server).

Therefore, my choice was to modify the existing DNS-over-UDP transport.

Here's an overview of the changes:

1. introduce a new API for performing an async round trip that returns
a channel wrapper where all responses are posted. The channel will not ever
be closed, so the reader needs to use select for safely reading. If the
reader users the wrapper's Next or TryNextResponses methods, these details
do not matter because they already implement a safe reading pattern.

2. the async round trip API performs the round trip in the background
and stops processing when it sees the first error.

3. the background running code will use an overall deadline derived
from the DNSTransport.IOTimeout field to know when to stop.

4. the background running code will additionally stop running if
noone is reading the channel and there are no empty slots in the
channel's buffer.

5. the RoundTrip method has been rewritten in terms of the async API.

The design I'm using here implements the proposal for async round
trips defined at https://github.com/ooni/probe/issues/2099. I have
chosen not to make all transports async because the DNS transport
seems the only transport that needs to also work in async mode.

While there, I noticed that we were not propagating CloseIdleConnection
to the underlying dialer, which was potentially wrong, so I did it.
2022-05-26 20:09:00 +02:00
Simone Basso
01a513a496
refactor: DNSTransport I/Os DNS messages (#760)
This diff refactors the DNSTransport model to receive in input a DNSQuery and return in output a DNSResponse.

The design of DNSQuery and DNSResponse takes into account the use case of a transport using getaddrinfo, meaning that we don't need to serialize and deserialize messages when using getaddrinfo.

The current codebase does not use a getaddrinfo transport, but I wrote one such a transport in the Websteps Winter 2021 prototype (https://github.com/bassosimone/websteps-illustrated/).

The design conversation that lead to producing this diff is https://github.com/ooni/probe/issues/2099
2022-05-25 17:03:58 +02:00
Simone Basso
7a0a156aec
Spring cleanup: remove unused/unneded code (#761)
* cleanup: remove the archival package

See https://github.com/ooni/probe/issues/2116

* cleanup: remove websteps fall 2021 edition

See https://github.com/ooni/probe/issues/2116

* cleanup: remove JavaScript based testing framework

https://github.com/ooni/probe/issues/2116

* cleanup: remove the unused ooapi package

See https://github.com/ooni/probe/issues/2116
2022-05-25 13:21:39 +02:00
Simone Basso
8b0815efab
cleanup: move legacy from internal/engine to internal (#759)
No functional change.

See https://github.com/ooni/probe/issues/2115
2022-05-25 10:19:03 +02:00
Simone Basso
2d721baa91
cleanup: merge httpheader and httpfailure into model (#758)
These two small packages could easily be merged into the model
package, since they're clearly model-like packages.

Part of https://github.com/ooni/probe/issues/2115
2022-05-25 09:54:50 +02:00
Simone Basso
09523f8efb
cleanup(geolocate): use netxlite rather than netx (#756)
See https://github.com/ooni/probe/issues/2114
2022-05-25 09:12:02 +02:00
Simone Basso
d922bd9afc
cleanup: mark more integration tests as !short mode (#755)
The objective is to make PR checks run much faster.

See https://github.com/ooni/probe/issues/2113 for context.

Regarding netxlite's tests:

Checking for every commit on master or on a release branch is
good enough and makes pull requests faster than one minute since
netxlite for windows is now 1m slower than coverage.

We're losing some coverage but coverage from integration tests
is not so good anyway, so I'm not super sad about this loss.
2022-05-24 21:01:15 +02:00
Simone Basso
6924d1ad81
refactor: only use shaping dialer for ndt7 and dash (#754)
See https://github.com/ooni/probe/issues/2112 for context.

While there, run `go fix -fix buildtag ./...`
2022-05-24 18:23:42 +02:00
Simone Basso
b68b8e1e8f
fix({simplequic,tls}ping): default SNI to URL's hostname (#753)
See https://github.com/ooni/probe/issues/2111
2022-05-24 16:29:13 +02:00
Simone Basso
ebc00a95fe
refactor: use ooni/oocrypto instead of ooni/go (#751)
Rather than building for Android using ooni/go, we're now using
ooni/oocryto as the TLS dependency. Such a repository only forks
crypto/tls and some minor crypto packages and includes the
same set of patches that we have been using in ooni/go.

This new strategy should be better than the previous one in
terms of building for Android, because we can use the vanilla
go1.18.2 build. It also seems that it is easier to track and
merge from upstream with ooni/oocrypto than it is with ooni/go.

Should this assessment be wrong, we can revert back to the
previous scenario where we used ooni/go.

See https://github.com/ooni/probe/issues/2106 for extra context.
2022-05-22 19:53:37 +02:00
Simone Basso
a1df3b4070
chore: upgrade to go1.18.2 (#750)
I have not upgraded ooni/go yet because I am also checking whether it
would be possible instead to use ooni/oocrypto as documented in the
https://github.com/ooni/probe/issues/2106 issue.

Closes https://github.com/ooni/probe/issues/2070

Closes https://github.com/ooni/probe/issues/2077
2022-05-22 18:11:37 +02:00
Simone Basso
395961c4da
chore: we're not hacking on v3.16.0-alpha (#749)
See https://github.com/ooni/probe/issues/2100
2022-05-20 20:03:48 +02:00
Simone Basso
6713fc6bc6
cleanup(geolocate): remove IP lookuppers blocked by sanctions (#747)
Closes https://github.com/ooni/probe/issues/2104
2022-05-20 12:43:56 +02:00
Simone Basso
e5d59e834e
chore: update the user-agent used for measuring (#746)
See https://github.com/ooni/probe/issues/2100
2022-05-20 12:43:37 +02:00
Simone Basso
1a7234e0f1
chore: run go generate ./... (#743)
See https://github.com/ooni/probe/issues/2100
2022-05-19 20:44:39 +02:00
DecFox
cced980387
fix(webconnectivity): pass TestHelpers to measurement (#736)
* Passed the TestHelpers field to RunAsyc and MeasureAsync. This reflects the test_helpers in the measurement.

* Spec already contains the correct output.

See  https://github.com/ooni/probe/issues/2073

Co-authored-by: decfox <decfox>
2022-05-18 15:46:08 +02:00
Simone Basso
1f83c19853
fix(backend api): re-enable previosuly disabled tests (#737)
See https://github.com/ooni/probe/issues/2098
2022-05-18 11:31:46 +02:00
Simone Basso
7c45f7b88c
fix(netxlite): ensure we only accept DNS responses (#735)
Previously, the DNS decoder did not check whether it was parsing
a DNS query or a DNS response, which was wrong.

As a side note, it seems I am using "reply" in the codebase instead
of "response". The latter seems correct DNS terminology.

This diff has been extracted from 9249d14f80

See https://github.com/ooni/probe/issues/2096.
2022-05-16 11:17:30 +02:00
Simone Basso
ce052b665e
feat(netxlite): implements NS queries (#734)
This diff has been extracted from eb0bf38957.

See https://github.com/ooni/probe/issues/2096.

While there, skip the broken tests caused by issue
https://github.com/ooni/probe/issues/2098.
2022-05-16 10:46:53 +02:00
Simone Basso
c1b06a2d09
fix(netxlite): prefer composition over embedding (#733)
This diff has been extracted and adapted from 8848c8c516

The reason to prefer composition over embedding is that we want the
build to break if we add new methods to interfaces we define. If the build
does not break, we may forget about wrapping methods we should
actually be wrapping. I noticed this issue inside netxlite when I was working
on websteps-illustrated and I added support for NS and PTR queries.

See https://github.com/ooni/probe/issues/2096

While there, perform comprehensive netxlite code review
and apply minor changes and improve the docs.
2022-05-15 19:25:27 +02:00
Simone Basso
9d2301cae2
fix(netxlite): reject replies with wrong queryID (#732)
This diff has been extracted from c2f7ccab0e

See https://github.com/ooni/probe/issues/2096

While there, export DecodeReply to decode a raw reply without
interpreting the Rcode or parsing the results, which seems a
nice extra feature to have to more flexibly parse DNS replies
in other parts of the codebase.
2022-05-14 19:38:46 +02:00