d45e58c14f
Part of https://github.com/ooni/ooni.org/issues/361 Co-authored-by: Arturo Filastò <arturo@openobservatory.org>
105 lines
4.3 KiB
Markdown
105 lines
4.3 KiB
Markdown
# Using the measurex package to write network experiments
|
|
|
|
This tutorial teaches you how to write OONI network
|
|
experiments using the primitives in the `./internal/measurex`
|
|
package. The name of this package means either "measure
|
|
extensions" or "measure crossover".
|
|
|
|
The measure extension interpretation of the name explains
|
|
what this package does. It contains extensions to our
|
|
basic networking code (`./internal/netxlite`) that allow
|
|
us to perform OONI measurements.
|
|
|
|
The measure crossover interpretation explains its history. Since
|
|
OONI has been written in Go, we've mostly performed measurements
|
|
using "tracing". That is, by registering hooks that run when
|
|
specific operations happen (e.g., TCP connect or TLS handshake)
|
|
and then making sense of the network trace. This is the approach
|
|
with which most experiments are written as of 2021-09-24. But
|
|
we have also seen that in several cases a step-by-step approach
|
|
is preferrable. Under this approach, you perform individual
|
|
operations and record their result right away. So, for example,
|
|
you have a connection and a TLS config, you perform a TLS
|
|
handshake, and immediately after you create and store somewhere
|
|
a data structure containing the result. This package is at the
|
|
crossover of these two approaches, because basically it contains
|
|
enough primitives to support both.
|
|
|
|
What we are going to do in this tutorial is the following:
|
|
|
|
- we will start from very simple-step-by-step measurements such
|
|
as TCP connect, DNS lookup, and TLS handshake;
|
|
|
|
- we will see how `measurex` provides support for composing
|
|
these primitives in larger steps, which will lead us to
|
|
eventually perform all the measurements that matter for a
|
|
given input URL (including discovering QUIC endpoints
|
|
and following redirections);
|
|
|
|
- finally, as an exercise, we will use the knowledge
|
|
acquired in the rest of the tutorial to rewrite a
|
|
subset of the Web Connectivity experiment (as of 2021-09-24
|
|
the flagship OONI experiment). This will be an opportunity
|
|
to explore more low level aspects of `measurex`.
|
|
|
|
As part of the process, we'll introduce you to the data
|
|
format used by OONI and there will be proposed exercises
|
|
where we simulate censorship conditions and we see how
|
|
that impacts the generated measurements.
|
|
|
|
Every chapter will show how to write a simple `main.go`
|
|
program that explains how to use some primitives. The
|
|
chapter text itself is autogenerated from comments inside
|
|
the actual `main.go` the we describe in the chapter.
|
|
|
|
For this reason, if you need to change a chapter, you
|
|
need to change the corresponding `main.go` file and then
|
|
follow the instructions at `./internal/tutorial/generate`
|
|
to regenerate the markdown text of the chapter.
|
|
|
|
More in detail, here's the index:
|
|
|
|
- [chapter01](chapter01) explains how to use the "system" resolver
|
|
|
|
- [chapter02](chapter02) deals with establishing TCP connections
|
|
|
|
- [chapter03](chapter03) is about using custom DNS-over-UDP resolvers
|
|
|
|
- [chapter04](chapter04) shows how to measure TLS handshakes
|
|
|
|
- [chapter05](chapter05) is about the QUIC handshake
|
|
|
|
- [chapter06](chapter06) shows how to get a webpage knowing its
|
|
URL and the endpoint (i.e., IP address and TCP/UDP port)
|
|
|
|
- [chapter07](chapter07) shows how to extend what we did in
|
|
chapter06 to cover _all_ the IP addresses in the URL's domain
|
|
|
|
- [chapter08](chapter08) is about HTTPSSvc DNS queries and
|
|
how they can be used to discover and test QUIC endpoints, thus
|
|
extending the work done in chapter07
|
|
|
|
- [chapter09](chapter09) improves upon chapter08 showing
|
|
how to run endpoints measurements in parallel
|
|
|
|
- [chapter10](chapter10) improves upon chapter09 by
|
|
also running DNS queries in parallel
|
|
|
|
- [chapter11](chapter11) tells you that all the code we
|
|
have been writing so far, and specifically the code we have
|
|
in chapter10, is actually the implementation of an API
|
|
of `measurex` called `MeasureURL`, and then shows you how
|
|
you can simplify the code in chapter10 by using this API.
|
|
|
|
- [chapter12](chapter12) extends the work done in
|
|
chapter11 by teaching you about a more high-level API
|
|
that discovers and follows all redirections, calling
|
|
`MeasureURL` for each redirection.
|
|
|
|
- [chapter13](chapter13) contains the exercise regarding
|
|
rewriting WebConnectivity using all the tools you have
|
|
learned so far and pointing you at additional `measurex`
|
|
API that could be useful to solve the problem.
|
|
|
|
- [chapter14](chapter14) contains our solution to the exercise.
|