# Using the measurex package to write network experiments

This tutorial teaches you how to write OONI network
experiments using the primitives in the `./internal/measurex`
package. The name of this package means either "measure
extensions" or "measure crossover".

The measure extension interpretation of the name explains
what this package does. It contains extensions to our
basic networking code (`./internal/netxlite`) that allow
us to perform OONI measurements.

The measure crossover interpretation explains its history. Since
OONI has been written in Go, we've mostly performed measurements
using "tracing". That is, by registering hooks that run when
specific operations happen (e.g., TCP connect or TLS handshake)
and then making sense of the network trace. This is the approach
with which most experiments are written as of 2021-09-24. But
we have also seen that in several cases a step-by-step approach
is preferrable. Under this approach, you perform individual
operations and record their result right away. So, for example,
you have a connection and a TLS config, you perform a TLS
handshake, and immediately after you create and store somewhere
a data structure containing the result. This package is at the
crossover of these two approaches, because basically it contains
enough primitives to support both.

What we are going to do in this tutorial is the following:

- we will start from very simple-step-by-step measurements such
as TCP connect, DNS lookup, and TLS handshake;

- we will see how `measurex` provides support for composing
these primitives in larger steps, which will lead us to
eventually perform all the measurements that matter for a
given input URL (including discovering QUIC endpoints
and following redirections);

- finally, as an exercise, we will use the knowledge
acquired in the rest of the tutorial to rewrite a
subset of the Web Connectivity experiment (as of 2021-09-24
the flagship OONI experiment). This will be an opportunity
to explore more low level aspects of `measurex`.

As part of the process, we'll introduce you to the data
format used by OONI and there will be proposed exercises
where we simulate censorship conditions and we see how
that impacts the generated measurements.

Every chapter will show how to write a simple `main.go`
program that explains how to use some primitives. The
chapter text itself is autogenerated from comments inside
the actual `main.go` the we describe in the chapter.

For this reason, if you need to change a chapter, you
need to change the corresponding `main.go` file and then
follow the instructions at `./internal/tutorial/generate`
to regenerate the markdown text of the chapter.

More in detail, here's the index:

- [chapter01](chapter01) explains how to use the "system" resolver

- [chapter02](chapter02) deals with establishing TCP connections

- [chapter03](chapter03) is about using custom DNS-over-UDP resolvers

- [chapter04](chapter04) shows how to measure TLS handshakes

- [chapter05](chapter05) is about the QUIC handshake

- [chapter06](chapter06) shows how to get a webpage knowing its
URL and the endpoint (i.e., IP address and TCP/UDP port)

- [chapter07](chapter07) shows how to extend what we did in
chapter06 to cover _all_ the IP addresses in the URL's domain

- [chapter08](chapter08) is about HTTPSSvc DNS queries and
how they can be used to discover and test QUIC endpoints, thus
extending the work done in chapter07

- [chapter09](chapter09) improves upon chapter08 showing
how to run endpoints measurements in parallel

- [chapter10](chapter10) improves upon chapter09 by
also running DNS queries in parallel

- [chapter11](chapter11) tells you that all the code we
have been writing so far, and specifically the code we have
in chapter10, is actually the implementation of an API
of `measurex` called `MeasureURL`, and then shows you how
you can simplify the code in chapter10 by using this API.

- [chapter12](chapter12) extends the work done in
chapter11 by teaching you about a more high-level API
that discovers and follows all redirections, calling
`MeasureURL` for each redirection.

- [chapter13](chapter13) contains the exercise regarding
rewriting WebConnectivity using all the tools you have
learned so far and pointing you at additional `measurex`
API that could be useful to solve the problem.

- [chapter14](chapter14) contains our solution to the exercise.