Commit Graph

1038 Commits

Author SHA1 Message Date
Ain Ghazal
74aebedac3
doc(step-by-step): readability improvements (#820)
This diff contains readability improvements for the step-by-step design document.

Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-06-30 09:55:18 +02:00
Simone Basso
797dd27ffc
chore: avoid duplicating the code of conduct (#818) 2022-06-28 12:52:18 +02:00
Simone Basso
d4f530b7ea
chore: use go1.18.3 (#816)
This diff changes all github actions and mk to use go1.18.3.

That's what I am using locally.

Also, both oohttp and oocrypto are on go1.18.3 already.
2022-06-17 20:12:14 +02:00
Simone Basso
28585fbdc2
doc: mention step-by-step design document
This commit changes the README of `./internal` to make
sure we're referring to the step-by-step design doc.
2022-06-17 11:02:54 +02:00
Simone Basso
c3fb7308d8
doc: mention how to generate a dependency graph
Thanks @ainghazal for mentioning this useful tool to me!
2022-06-17 10:56:47 +02:00
Simone Basso
be2da83b1b
doc: publish the step-by-step design document (#814)
This pull request publishes the step-by-step design document that I have been discussing with @hellais and @DecFox recently. Compared to the document that was approved, this one has been edited for readability.

While there, I figured it was also be beneficial to publish the few ooni/probe-cli related design documents we produced in the past, because they probably help someone to get acquainted with the codebase.

Reference issue for this pull request: https://github.com/ooni/probe/issues/2148
2022-06-14 14:38:29 +02:00
Simone Basso
1685ef75b5
refactor(netxlite): expose useful HTTPTransport/DNSTransport factories (#813)
These factories will soon be useful to finish with
https://github.com/ooni/probe/issues/2135.
2022-06-09 00:30:18 +02:00
Simone Basso
1a706e47bc
refactor(netxlite): more abstract proxy-enabled dialer construction (#812)
This will help with https://github.com/ooni/probe/issues/2135
2022-06-08 23:10:06 +02:00
Simone Basso
bf7ea423d3
refactor(sessionresolver): replace dnsclientmaker with function (#811)
See https://github.com/ooni/probe/issues/2135
2022-06-08 22:29:01 +02:00
Simone Basso
a02cc6100b
refactor(sessionresolver): minor changes in files and types naming (#810)
Part of https://github.com/ooni/probe/issues/2135
2022-06-08 22:01:51 +02:00
Simone Basso
beba543b98
cleanup: use NewHTTPTransportWithResolver more often (#808)
We can simplify code in a bunch of places using a useful factory.

Part of https://github.com/ooni/probe/issues/2121.
2022-06-08 15:13:01 +02:00
Simone Basso
f1b8071c65
hotfix(sessionresolver): prevent data race inside http3 (#809)
See https://github.com/ooni/probe/issues/2135#issuecomment-1149840579
2022-06-08 15:06:15 +02:00
Simone Basso
fe29b432e0
refactor: start refactoring session resolver (#807)
This diff addresses the following points of https://github.com/ooni/probe/issues/2135:

- [x] the `childResolver` type is useless and we can use `model.Resolver` directly;
- [x] we should use `model/mocks` instead of custom fakes;
- [x] we should not use `log.Log` rather we should use `model.DiscardLogger`;
- [x] make `timeLimitedLookup` easier to test with a `-short` tests;
- [x] ensure `timeLimitedLookup` returns as soon as its context expires regardless of the child resolver;

Subsequent diffs will address more points mentioned in there.
2022-06-08 14:06:22 +02:00
Simone Basso
dea23b49d5
refactor(oohelperd): use netxlite rather than netx (#806)
The oohelperd implementation did not actually need using netx because
it was just constructing default types with logging, which is what
netxlite already does. Hence, let's avoid using netx here.

See https://github.com/ooni/probe/issues/2121
2022-06-08 10:14:10 +02:00
Simone Basso
87d35f4225
refactor(oohelper): use netxlite rather than netx (#805)
The oohelper does not need to use netx and it's enough to use
netxlite, hence let us apply this refactor.

The original code used DoT but the explanatory comment said we were
using DoT because of unclear issues inside GitHub actions.

We are now using DoH and this is fine as well. The comment implied
that any encrypted transport would do.

See https://github.com/ooni/probe/issues/2121
2022-06-08 09:52:47 +02:00
Simone Basso
4cf58380c3
[forwardport] fix(stunreachability): don't emit spurious progress events (#803)
This diff forward ports 4fb5f7de69b900cfc2bd211f723108a67bda350d to master.

See https://github.com/ooni/probe/issues/2058#issuecomment-1145847069

This diff WILL need to be forwardported to master.
2022-06-06 16:21:17 +02:00
Simone Basso
504181906a
[forwardport] fix(dnscheck): remove apparently-broken static input (#804)
This diff forward ports 261d1a4cdc88522f6a8f63d6c540f51054566b28 to master
whose original commit message follows:

- - -

It's not working for me from a couple of places and also it does not
seem to be documented upstream, see:

https://docs.namebase.io/guides-1/resolving-handshake-1/hdns.io

This diff WILL need to be forwardported to master.
2022-06-06 16:08:09 +02:00
Simone Basso
57e207e644
doc(netx): reference issue mentioning future improvements (#802)
See https://github.com/ooni/probe/issues/2121#issuecomment-1147424810
2022-06-06 15:16:30 +02:00
Simone Basso
2502a237fb
cleanup: netx does not use netxlite legacy names (#801)
This diff refactors netx and netxlite to ensure we're not using
netxlite legacy names inside of netx.

To this end, we're cheating a bit. We're exposing a new factory to
get an unwrapped stdlib resolver rather than defining a legacy name
to export the private name of the same factory.

This is actually a fine place to stop, for now, the next and
netxlite refactoring at https://github.com/ooni/probe/issues/2121.
2022-06-06 14:46:44 +02:00
Simone Basso
64bffbd941
refactor(netx): reorganize by topic (#800)
Before finishing the ongoing refactoring and leaving whatever
is left of netx in tree, I would like to restructure it so that
we'll have an easy time next time we need to modify it.

Currently, every functionality lives into the `netx.go` file and
we have a support file called `httptransport.go`.

I would like to reorganize by topic, instead. This would allow
future me to more easily perform topic-specific changes.

While there, improve `netx`'s documentation and duplicate some of
this documentation inside `internal/README.md` to provide pointers
to previous documentation, historical context, and some help to
understand the logic architecture of network extensions (aka `netx`).

Part of https://github.com/ooni/probe-cli/pull/396
2022-06-06 14:27:25 +02:00
Simone Basso
5d54aa9c5f
cleanup: move caching resolvers from netx to netxlite (#799)
Now that we have properly refactored the caching resolvers we can
move them into netxlite as optional resolvers created using the
proper abstract factories we just added.

This diff reduces the complexity and the code size of netx.

See https://github.com/ooni/probe/issues/2121.
2022-06-05 21:58:34 +02:00
Simone Basso
6b85dfce88
refactor(netx): move construction logic outside package (#798)
For testability, replace most if-based construction logic with
calls to well-tested factories living in other packages.

While there, acknowledge that a bunch of types could now be private
and make them private, modifying the code to call the public
factories allowing to construct said types instead.

Part of https://github.com/ooni/probe/issues/2121
2022-06-05 21:22:27 +02:00
Simone Basso
2d3d5d9cdc
cleanup(netx): stop using most netxlite resolver internals (#797)
This diff modifies netx to stop using most netxlite resolver internals
but the internal function that creates a new, unwrapped system resolver,
which will be dealt with in a subsequent pull request.

See https://github.com/ooni/probe/issues/2121
2022-06-05 19:52:39 +02:00
Simone Basso
07c0b08505
cleanup(netxlite): drop the DefaultDialer legacy name (#796)
Part of https://github.com/ooni/probe/issues/2121
2022-06-05 18:44:17 +02:00
Simone Basso
c6b3889a33
fix(netx): ensure we create ~same HTTP3 and HTTP2 transports (#795)
1. Use the netxlite.NewHTTPTransport factory for creating a new
HTTP2 (and HTTP1) transport;

2. Recognize the netxlite.NewOOHTTPTransport has now become
an implementation detail so make it private;

3. Recognize that netxlite.NewHTTP3Transport should call
netxlite.WrapTransport so it returns the same typechain
returned by netxlite.NewHTTPTransport (modulo, of course,
the real underlying transport), so ensure that we are
calling netxlite.WrapTransport in NewHTTP3Transport;

4. Recognize that the table based constructor inside of
netx needs a logger to create HTTPTransport instances using
either netxlite.NewHTTP{,3}Transport so pass this argument
along and ensure it's not nil using a constructor inside
model that guarantees that;

5. Cleanup netx's tests to avoid type asserting on the
typechains returned by netxlite since we already test
that inside netxlite;

6. Recognize that now we can make more legacy names inside
of netxlite private because we don't need to use them
inside tests anymore (because of previous point).

Reference issue: https://github.com/ooni/probe/issues/2121
2022-06-05 17:41:06 +02:00
Simone Basso
d5249a6cf7
chore: improve testing and increase coverage (#794)
This diff improves testing and increases coverage inside the
./internal/netxlite and ./internal/tracex packages.

See https://github.com/ooni/probe/issues/2121
2022-06-04 14:58:48 +02:00
Ain Ghazal
464d03184e
chore: add gosec step (#793)
* add gosec step

* fix: disable build on pull request

Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-06-03 21:41:15 +02:00
Simone Basso
15da0f5344
cleanup(jafar): do not depend on netx and urlgetter (#792)
There's no point in doing that. Also, once this change is merged, it becomes easier to cleanup/simplify netx.

See https://github.com/ooni/probe/issues/2121
2022-06-02 22:25:37 +02:00
Simone Basso
76b65893a1
cleanup(netx): remove redundant config options (#791)
Part of https://github.com/ooni/probe/issues/2121
2022-06-02 18:18:49 +02:00
Simone Basso
e9ed733f07
refactor(netx): use netxlite to build TLSDialer (#790)
This diff modifies netx to use netxlite to build the TLSDialer.

Building the TLSDialer entails building a TLSHandshaker.

While there, hide netxlite names we don't want to be public
and change netx tests to test for functionality.

To this end, refactor filtering to provide an easier to
use TLS server. We don't need the complexity of proxying
rather we need to provoke specific errors.

Part of https://github.com/ooni/probe/issues/2121
2022-06-02 17:39:48 +02:00
Simone Basso
ae24ba644c
cleanup(netx): another batch of small/simple cleanups (#789)
See https://github.com/ooni/probe/issues/2121
2022-06-02 13:50:34 +02:00
Simone Basso
1cb820b19d
cleanup(netx): remove subpackages and unnecessary code (#788)
This pull request consists of several small and obvious cleanups in the netx directory.

See https://github.com/ooni/probe/issues/2121
2022-06-02 11:51:21 +02:00
Simone Basso
9354191b85
refactor(tracex): internally store just the raw certificate (#787)
By just storing the raw certificate we simplify the internal data
structure we use. In turn, this enables us to write better unit tests
using github.com/google/go-cmp where we can construct the expected
result and compare with that. (Yeah, in principle we could also
construct the full certificate but I'm not sure it's worth the effort
since we basically only care about the raw certificate.)

The general idea here is to make tracex more tested. Once it's more
tested, I will create separate structs for each event, which is
something that measurex also does. Once that is done, we can start
ensuring that the code in measurex and the code in tracex do the
same thing in terms of storing observations. When also this is done,
we can then rewrite measurex to use tracex directly.

The overall goal is https://github.com/ooni/probe/issues/2035.
2022-06-02 11:07:02 +02:00
Simone Basso
83e3167ce2
refactor(tracex): internally represent errors as strings (#786)
There are two reasons why this is beneficial:

1. github.com/google/go-cmp is more annoying to use for comparing
data structures when there are interfaces to compare. Sure, there's
a recipe for teaching it to compare errors, but how about making
the errors trivially comparable instead?

2. if we want to send errors over the network, JSON serialization
works but we cannot unmarshal the resulting string back to an error,
so how about making this representation trivial to serialize (we
are not going this now, but we need this property for websteps and
it may be sensible to try to avoid to have duplicate code because
of that -- measurex currently duplicates many tracex functionality
and this is quite unfortunate because it slows development down)

Additionally, if an error is a string:

3. we can very easily use a switch for comparing its possible
values with "" representing the absence of errors, while it is
more complex to do the same when using a nullable string or even
an error (i.e., an interface)

4. if a type is not nullable, it's easier to write safe code for
it and we may want to refactor experiments to use the internal
representation of measurements for more robust processing code

For all these reasons, let's internally use strings in tracex.

The overall aim here is to reduce the duplicated code between pre
and post-measurex measurements (see https://github.com/ooni/probe/issues/2035).
2022-06-02 10:37:07 +02:00
Simone Basso
2556e93050
[forwardport] fix(dnscheck): stop emitting progress (#785)
This diff forward ports b606494db8a9293384efaf5c33a88601f6e1e2a6
to the main development branch.

Dnscheck is emitting progress and the experiment controller is
also emitting progress. This messes up the progress bar.

See https://github.com/ooni/probe/issues/2058#issuecomment-1141638067
2022-06-02 09:31:52 +02:00
Simone Basso
b58cfadb39
hotfix: disable oocrypto until we investigate ciphers selection (#784)
See https://github.com/ooni/probe/issues/2122 for context.
2022-06-02 08:52:15 +02:00
Simone Basso
92ebfd7986
chore: upgrade oocrypto, oohttp, probe-assets (#783)
* oocrypto is now using go1.18.3

* oohttp is now using go1.18.3

* probe-assets is now using db-ip.com's June files
2022-06-02 08:35:02 +02:00
Simone Basso
58adb68b2c
refactor: move tracex outside of engine/netx (#782)
* refactor: move tracex outside of engine/netx

Consistently with https://github.com/ooni/probe/issues/2121 and
https://github.com/ooni/probe/issues/2115, we can now move tracex
outside of engine/netx. The main reason why this makes sense now
is that the package is now changed significantly from the one
that we imported from ooni/probe-engine.

We have improved its implementation, which had not been touched
significantly for quite some time, and converted it to unit
testing. I will document tomorrow some extra work I'd like to
do with this package but likely could not do $soon.

* go fmt

* regen tutorials
2022-06-02 00:50:55 +02:00
Simone Basso
d397036073
refactor(tracex): convert to unit testing (#781)
The exercise already allowed me to notice issues such as fields not
being properly initialized by savers.

This is one of the last steps before moving tracex away from the
internal/netx package and into the internal package.

See https://github.com/ooni/probe/issues/2121
2022-06-01 23:15:47 +02:00
Simone Basso
6212daa54a
fix(tracex): generate archival from single transaction-done event (#780)
Tracex contained some fragile code that assembled HTTP measurements
from scattered events, which worked because we were sure we were
performing a single measurement at any given time.

This diff restructures the code to emit a transaction-start and a
transaction-done events only. We have basically removed all the other
events (which we were not using). We kept the transaction-start
though, because it may be useful to see it when reading events. In
any case, what matters here is that we're now using the transaction-done
event aline to generate the archival HTTP measurement.

Hence, the original issue has been addressed. We will possibly
do more refactoring in the future, but for now this seems sufficient.

Part of https://github.com/ooni/probe/issues/2121
2022-06-01 19:27:47 +02:00
Simone Basso
153bfe1c57
fix: pin to gopkg.in/yaml.v3 v3.0.1 (#779)
There is a security warning about using v3.0.0.

We're not using it directly but some dependency of ours is
using it and perhaps pinning would make the warning disappear?

Thanks to @hellais for help investigating this issue.
2022-06-01 15:29:58 +02:00
Simone Basso
66fd1569b8
tracex: prepare HTTP code for future refactoring (#778)
The main issue I see inside tracex at the moment is that we
construct the HTTP measurement from separate events.

This is fragile because we cannot be sure that these events
belong to the same round trip. (Currently, they _are_ part
of the same round trip, but this is a fragile assumption and
it would be much more robust to dispose of it.)

To prepare for emitting a single event, it's imperative to
have two distinct fields for HTTP request and response headers,
which is the main contribution in this commit.

Then, we have a bunch of smaller changes including:

1. correctly naming 'response' the DNS response (instead of 'reply')

2. ensure we always use pointer receivers

Reference issue: https://github.com/ooni/probe/issues/2121
2022-06-01 15:20:28 +02:00
Simone Basso
c740be987b
refactor(tracex): do not depend on strings for event names (#777)
Rather than matching a string, match a type.

This is more robust considering future refactorings.

We're confident the names did not change in _this_ refactoring
because we're still testing the same strings in the tests.

Part of https://github.com/ooni/probe/issues/2121
2022-06-01 14:32:16 +02:00
Simone Basso
8f7e3803eb
feat(netxlite): implement DNSTransport wrapping (#776)
Acknowledge that transports MAY be used in isolation (i.e., outside
of a Resolver) and add support for wrapping.

Ensure that every factory that creates an unwrapped type is named
accordingly to hopefully ensure there are no surprises.

Implement DNSTransport wrapping and use a technique similar to the
one used by Dialer to customize the DNSTransport while constructing
more complex data types (e.g., a specific resolver).

Ensure that the stdlib resolver's own "getaddrinfo" transport (1)
is wrapped and (2) could be extended during construction.

This work is part of my ongoing effort to bring to this repository
websteps-illustrated changes relative to netxlite.

Ref issue: https://github.com/ooni/probe/issues/2096
2022-06-01 11:10:08 +02:00
Simone Basso
923d81cdee
refactor(netxlite): introduce the getaddrinfo transport (#775)
This diff modifies the system resolver to use a getaddrinf transport.

Obviously the transport is a fake, but its existence will allow us
to observe DNS events more naturally.

A lookup using the system resolver would be a ANY lookup that will
contain all the resolved IP addresses into the same response.

This change was also part of websteps-illustrated, albeit the way in
which I did it there was less clean than what we have here.

Ref issue: https://github.com/ooni/probe/issues/2096
2022-06-01 09:59:44 +02:00
Simone Basso
7e0b47311d
refactor(netxlite): better integration with tracex (#774)
Rather than passing functions to construct complex objects such
as Dialer and QUICDialer, pass interface implementations.

Ensure that a nil implementation does not cause harm.

Make Saver implement the correct interface either directly or
indirectly. We need to implement the correct interface indirectly
for TCP conns (or connected UDP sockets) because we have two
distinct use cases inside netx: observing just the connect event
and observing just the I/O events.

With this change, the construction of composed Dialers and
QUICDialers is greatly simplified and more obvious.

Part of https://github.com/ooni/probe/issues/2121
2022-06-01 08:31:20 +02:00
Simone Basso
f4f3ed7c42
refactor(tracex): start applying recent code conventions (#773)
The code that is now into the tracex package was written a long
time ago, so let's start to make it more in line with the coding
style of packages that were written more recently.

I didn't apply all the changes I'd like to apply in a single diff
and for now I am committing just this diff.

Broadly, what we need to do is:

1. improve documentation

2. ~always use pointer receivers (object receives have the issue
that they are not mutable by accident meaning that you can mutate
them but their state do not change after the call returns, which
is potentially a source of bugs in case you later refactor to use
a pointer receiver, so always use pointer receivers)

3. ~always avoid embedding (let's say we want to avoid embedding
for types we define and it's instead fine to embed types that are
defined in the stdlib: if later we add a new method, we will not
see a broken build and we'll probably forget to add the new method
to all wrappers -- conversely, if we're wrapping rather than
embedding, we'll see a broken build and act accordingly)

4. prefer unit tests and group tests by type being tested rather
than using a flat structure for tests

There's a coverage slippage that I'll compensate in a follow-up diff where I'll focus on unit testing.

Reference issue: https://github.com/ooni/probe/issues/2121
2022-06-01 07:44:54 +02:00
Simone Basso
bbcd2e2280
refactor(netx): merge archival, trace, and the savers (#772)
This diff creates a new package under netx called tracex that
contains everything we need to perform measurements using events
tracing and postprocessing (which is the technique with which
we implement most network experiments).

The general idea here is to (1) create a unique package out of
all of these packages; (2) clean up the code a bit (improve tests,
docs, apply more recent code patterns); (3) move the resulting
code as a toplevel package inside of internal.

Once this is done, netx can be further refactored to avoid
subpackages and we can search for more code to salvage/refactor.

See https://github.com/ooni/probe/issues/2121
2022-05-31 21:53:01 +02:00
Simone Basso
dd5655eaee
refactor(netxlite): allow easy QUIC dialer chain customization (#771)
Like the previous diff, but for creating QUIC dialers.

See https://github.com/ooni/probe/issues/2121.
2022-05-31 20:28:25 +02:00
Simone Basso
69fd0c5119
refactor(netxlite): allow easy dialer chain customization (#770)
This diff modifies the construction of a dialer to allow one
to insert custom dialer wrappers into the dialers chain.

The point of the chain in which we allow custom wrappers is the
optimal one for connect, read, and write measurements.

This new design is better than the previous netx design since
we don't need to construct the whole chain manually now.

The work in this diff is part of the effort to make engine/netx
just a tiny wrapper around netxlite.

See https://github.com/ooni/probe/issues/2121.
2022-05-31 20:02:11 +02:00