refactor(netx): reorganize by topic (#800)
Before finishing the ongoing refactoring and leaving whatever is left of netx in tree, I would like to restructure it so that we'll have an easy time next time we need to modify it. Currently, every functionality lives into the `netx.go` file and we have a support file called `httptransport.go`. I would like to reorganize by topic, instead. This would allow future me to more easily perform topic-specific changes. While there, improve `netx`'s documentation and duplicate some of this documentation inside `internal/README.md` to provide pointers to previous documentation, historical context, and some help to understand the logic architecture of network extensions (aka `netx`). Part of https://github.com/ooni/probe-cli/pull/396
This commit is contained in:
parent
5d54aa9c5f
commit
64bffbd941
17 changed files with 703 additions and 528 deletions
|
|
@ -11,12 +11,86 @@ go doc -all ./internal/$package
|
|||
|
||||
where `$package` is the name of the package.
|
||||
|
||||
Some notable packages:
|
||||
## Tutorials
|
||||
|
||||
- [model](model) contains the interfaces and data model shared
|
||||
by most packages inside this directory;
|
||||
|
||||
- [netxlite](netxlite) is the underlying networking library;
|
||||
|
||||
- [tutorial](tutorial) contains tutorials on writing new experiments,
|
||||
The [tutorial](tutorial) package contains tutorials on writing new experiments,
|
||||
using measurements libraries, and networking code.
|
||||
|
||||
## Network extensions
|
||||
|
||||
This section briefly describes the overall design of the network
|
||||
extensions (aka `netx`) inside `ooni/probe-cli`. In OONI, we have
|
||||
two distinct but complementary needs:
|
||||
|
||||
1. speaking with our backends or accessing other services useful
|
||||
to bootstrap OONI probe and perform measurements;
|
||||
|
||||
2. implementing network experiments.
|
||||
|
||||
We originally implemented these functionality into a separate
|
||||
repository: [ooni/netx](https://github.com/ooni/netx). The
|
||||
original [design document](https://github.com/ooni/netx/blob/master/DESIGN.md)
|
||||
still provides a good overview of the problems we wanted to solve.
|
||||
|
||||
The general idea was to provide interfaces replacing standard library
|
||||
objects that we could further wrap to perform network measurements without
|
||||
deviating from the normal APIs expected by Go programmers. For example,
|
||||
|
||||
```Go
|
||||
type Dialer interface {
|
||||
DialContext(ctx context.Context, network, address string) (net.Conn, error)
|
||||
}
|
||||
```
|
||||
|
||||
is a generic dialer that could be a `&net.Dialer{}` but could also be a
|
||||
*saving* dialer that saves the results of dial events. So, you could write
|
||||
something like:
|
||||
|
||||
```Go
|
||||
saver := &Saver{}
|
||||
var dialer Dialer = NewDialer()
|
||||
dialer = saver.WrapDialer(dialer)
|
||||
conn, err := dialer.DialContext(ctx, network, address)
|
||||
events := saver.ExtractEvents()
|
||||
```
|
||||
|
||||
In short, with the original `netx` you could write measurement code
|
||||
resembling ordinary Go code but you could also save network events
|
||||
from which to derive whether there was censorship.
|
||||
|
||||
Since then, the architecture itself has evolved and `netx` has been
|
||||
merged into `ooni/probe-engine` and later `ooni/probe-cli`. As of
|
||||
2022-06-06, these are the fundamental `netx` packages:
|
||||
|
||||
- [model/netx.go](model/netx.go): contains the interfaces and structs
|
||||
patterned after the Go standard library used by `netx`;
|
||||
|
||||
- [netxlite](netxlite): implements error wrapping (i.e., mapping
|
||||
Go errors to OONI errors), enforces timeouts, and generally ensures
|
||||
that we're using a stdlib-like network API that meet all our
|
||||
constraints and requirements (e.g., logging);
|
||||
|
||||
- [bytecounter](bytecounter): provides support for counting the
|
||||
number of bytes consumed by network interactions;
|
||||
|
||||
- [multierror](multierror): defines an `error` type that contains
|
||||
a list of errors for representing the results of operations where
|
||||
multiple sub-operations may fail (e.g., TCP connect fails for
|
||||
all the IP addresses associated with a domain name);
|
||||
|
||||
- [tracex](tracex): support for collecting events during operations
|
||||
such as TCP connect, QUIC handshake, HTTP round trip. Collecting
|
||||
events allows us to analyze such events and determine whether there
|
||||
was blocking. This measurement strategy is called tracing because
|
||||
we wrap fundamental types (e.g., a dialer or an HTTP transport) to
|
||||
save the result of each operation into a "list of events" type
|
||||
called `Saver;
|
||||
|
||||
- [engine/netx](engine/netx): code surviving from the original `netx`
|
||||
implementation that we're still using for measuring. Issue
|
||||
[ooni/probe#2121](https://github.com/ooni/probe/issues/2121) describes
|
||||
a slow refactoring process where we'll move code outside of `netx`
|
||||
and inside `netxlite` or other packages. We are currently experimenting
|
||||
with step-by-step measurements, an alternative measurement
|
||||
approach where we break down operations in simpler building blocks. This
|
||||
alternative approach may eventually make `netx` obsolete.
|
||||
|
|
|
|||
Loading…
Reference in a new issue