c/ooni-probe-cli
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

doc(measurex): explain how to write experiments (#529)

Browse Source 
Part of https://github.com/ooni/ooni.org/issues/361

Co-authored-by: Arturo Filastò <arturo@openobservatory.org>
This commit is contained in:
Simone Basso 2021-09-30 01:36:03 +02:00 committed by GitHub
parent 399d2f65da
commit d45e58c14f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
31 changed files with 6822 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
internal/tutorial/README.md
View File

@ -10,6 +10,8 @@ real OONI code, it should always be up to date.
- [Rewriting the torsf experiment](experiment/torsf/)
- [Using the measurex package to write network experiments](measurex)
## Regenerating the tutorials

21
internal/tutorial/generator/main.go
View File

@ -90,6 +90,25 @@ func gentorsf() {
gen(path.Join(prefix, "chapter04"), "torsf.go")
}
// genmeasurex generates the measurex chapters.
func genmeasurex() {
prefix := path.Join(".", "measurex")
gen(path.Join(prefix, "chapter01"), "main.go")
gen(path.Join(prefix, "chapter02"), "main.go")
gen(path.Join(prefix, "chapter03"), "main.go")
gen(path.Join(prefix, "chapter04"), "main.go")
gen(path.Join(prefix, "chapter05"), "main.go")
gen(path.Join(prefix, "chapter06"), "main.go")
gen(path.Join(prefix, "chapter07"), "main.go")
gen(path.Join(prefix, "chapter08"), "main.go")
gen(path.Join(prefix, "chapter09"), "main.go")
gen(path.Join(prefix, "chapter10"), "main.go")
gen(path.Join(prefix, "chapter11"), "main.go")
gen(path.Join(prefix, "chapter12"), "main.go")
gen(path.Join(prefix, "chapter13"), "main.go")
gen(path.Join(prefix, "chapter14"), "main.go")
}
// gennetxlite generates the netxlite chapters.
func gennetxlite() {
prefix := path.Join(".", "netxlite")
@ -102,7 +121,9 @@ func gennetxlite() {
gen(path.Join(prefix, "chapter07"), "main.go")
gen(path.Join(prefix, "chapter08"), "main.go")
}
func main() {
gentorsf()
genmeasurex()
gennetxlite()
}

104
internal/tutorial/measurex/README.md Normal file
View File

@ -0,0 +1,104 @@
# Using the measurex package to write network experiments
This tutorial teaches you how to write OONI network
experiments using the primitives in the `./internal/measurex`
package. The name of this package means either "measure
extensions" or "measure crossover".
The measure extension interpretation of the name explains
what this package does. It contains extensions to our
basic networking code (`./internal/netxlite`) that allow
us to perform OONI measurements.
The measure crossover interpretation explains its history. Since
OONI has been written in Go, we've mostly performed measurements
using "tracing". That is, by registering hooks that run when
specific operations happen (e.g., TCP connect or TLS handshake)
and then making sense of the network trace. This is the approach
with which most experiments are written as of 2021-09-24. But
we have also seen that in several cases a step-by-step approach
is preferrable. Under this approach, you perform individual
operations and record their result right away. So, for example,
you have a connection and a TLS config, you perform a TLS
handshake, and immediately after you create and store somewhere
a data structure containing the result. This package is at the
crossover of these two approaches, because basically it contains
enough primitives to support both.
What we are going to do in this tutorial is the following:
- we will start from very simple-step-by-step measurements such
as TCP connect, DNS lookup, and TLS handshake;
- we will see how `measurex` provides support for composing
these primitives in larger steps, which will lead us to
eventually perform all the measurements that matter for a
given input URL (including discovering QUIC endpoints
and following redirections);
- finally, as an exercise, we will use the knowledge
acquired in the rest of the tutorial to rewrite a
subset of the Web Connectivity experiment (as of 2021-09-24
the flagship OONI experiment). This will be an opportunity
to explore more low level aspects of `measurex`.
As part of the process, we'll introduce you to the data
format used by OONI and there will be proposed exercises
where we simulate censorship conditions and we see how
that impacts the generated measurements.
Every chapter will show how to write a simple `main.go`
program that explains how to use some primitives. The
chapter text itself is autogenerated from comments inside
the actual `main.go` the we describe in the chapter.
For this reason, if you need to change a chapter, you
need to change the corresponding `main.go` file and then
follow the instructions at `./internal/tutorial/generate`
to regenerate the markdown text of the chapter.
More in detail, here's the index:
- [chapter01](chapter01) explains how to use the "system" resolver
- [chapter02](chapter02) deals with establishing TCP connections
- [chapter03](chapter03) is about using custom DNS-over-UDP resolvers
- [chapter04](chapter04) shows how to measure TLS handshakes
- [chapter05](chapter05) is about the QUIC handshake
- [chapter06](chapter06) shows how to get a webpage knowing its
URL and the endpoint (i.e., IP address and TCP/UDP port)
- [chapter07](chapter07) shows how to extend what we did in
chapter06 to cover _all_ the IP addresses in the URL's domain
- [chapter08](chapter08) is about HTTPSSvc DNS queries and
how they can be used to discover and test QUIC endpoints, thus
extending the work done in chapter07
- [chapter09](chapter09) improves upon chapter08 showing
how to run endpoints measurements in parallel
- [chapter10](chapter10) improves upon chapter09 by
also running DNS queries in parallel
- [chapter11](chapter11) tells you that all the code we
have been writing so far, and specifically the code we have
in chapter10, is actually the implementation of an API
of `measurex` called `MeasureURL`, and then shows you how
you can simplify the code in chapter10 by using this API.
- [chapter12](chapter12) extends the work done in
chapter11 by teaching you about a more high-level API
that discovers and follows all redirections, calling
`MeasureURL` for each redirection.
- [chapter13](chapter13) contains the exercise regarding
rewriting WebConnectivity using all the tools you have
learned so far and pointing you at additional `measurex`
API that could be useful to solve the problem.
- [chapter14](chapter14) contains our solution to the exercise.

366
internal/tutorial/measurex/chapter01/README.md Normal file
View File

@ -0,0 +1,366 @@
# Chapter I: using the system resolver
In this chapter we explain how to measure DNS resolutions performed
using the system resolver. *En passant*, we will also introduce you to
the `Measurer`, which we will use for the rest of the tutorial.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter01/main.go`.)
## The system resolver
We define "system resolver" as the DNS resolver implemented by the C
library. On Unix, the most popular interface to such a resolver is
the `getaddrinfo(3)` C library function.
Most OONI experiments (also known as nettests) use the system
resolver to map domain names to IP addresses. The advantage of
the system resolver is that it's provided by the system. So,
it should _generally_ work. Also, it is the resolver that the
user of the system will use every day, therefore its results
should be representative (even though the rise of DNS over
HTTPS embedded in browsers may make this statement less solid
than it were ten years ago).
The disadvantage of the system resolver is that we do not
know how it is configured. Say the user has configured a
DNS over TLS resolver; then the measurements may miss censorship
that we would otherwise see if using a custom DNS resolver.
Now that we have justified why the system resolver is
important for OONI, let us perform some measurements with it.
We will first write a simple `main.go` file that shows how to use
this functionality. Then, we will show some runs of this file, and
we will comment the output that we see.
## main.go
We declare the package and import useful packages. The most
important package we're importing here is, of course, `internal/measurex`.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
```
### Setup
We define command line flags useful to test this program. We use
the `flags` package for that. We want the user to be able to configure
both the domain name to resolve and the resolution timeout.
```Go
domain := flag.String("domain", "example.com", "domain to resolve")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
```
We call `flag.Parse` to parse the CLI flags.
```Go
flag.Parse()
```
We create a context and we attach a timeout to it. (This is a pretty
standard way of configuring a timeout in Go.)
```Go
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
```
### Creating a Measurer
Now we create a `Measurer`.
```Go
mx := measurex.NewMeasurerWithDefaultSettings()
```
The `Measurer` is a concrete type that contains many fields
requiring initialization. For this reason, we provide a factory
that creates one with default settings. The expected usage
pattern is that you do not modify a `Measurer`'s field after
initialization. Modifying them while the `Measurer` is in
use could, in fact, lead to races.
Let's now invoke the system resolver to resolve `*domain`!
### Invoking the system resolver
We call the `LookupHostSystem` method of the `Measurer`. The
arguments are the Context, that in this case carries the timeout
we configured above, and the domain to resolve.
The call itself is named `LookupHost` because this is the name
used by the Go function that performs a domain lookup.
Under the hood, `mx.LookupHostSystem` will eventually call
`(*net.Resolver).LookupHost`. In turn, in the common case on
Unix, this function will eventually call `getaddrinfo(3)`.
```Go
m := mx.LookupHostSystem(ctx, *domain)
```
The return value of `(*net.Resolver).LookupHost` is either a
list of IP addresses or an error. Our `LookupHostSystem` method,
instead, returns a `*measurex.DNSMeasurement` type.
This is probably a good moment to remind you of Go's
built in help system. We could include a definition of the
`DNSMeasurement` structure, but since this definition is
just a comment in the main.go file, it might age badly.
Instead, if you run
```
go doc ./internal/measurex.DNSMeasurement
```
You get the current definition. As you can see, this type
is basically just a wrapper around `Measurement`. Now,
checking the docs of `Measurement` with
```
go doc ./internal/measurex.Measurement
```
we can see a container of events
classified by event type. In our case, because we're
doing a `LookupHost`, we should have at least one entry
inside of the `Measurement.LookupHost` field.
This entry is of type `DNSLookupEvent`. Let us check
together the definition of this type:
```
go doc ./internal/measurex.DNSLookupEvent
```
If you are familiar with [the OONI data format specs](
https://github.com/ooni/spec/tree/master/data-formats), you
should probably recognize that this structure is the Go
representation of the `df-002-dnst` data format.
In fact, every event field inside of a `Measurement`
should serialize nicely to JSON to one of the OONI data
formats.
### Printing the measurement
Because there is a close relationship between the
events inside a `Measurement` and the JSON OONI data
format, in the remainder of this program we're
going to serialize the `Measurement` to JSON and
print it to the standard output.
```Go
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
```
As a final note, the `PanicOnError` is here because the
message `m` *can* be marshalled to JSON. It still feels a
bit better having an assertion for our assumptions than
outrightly ignoring the error code. (We tend to use such
a convention quite frequently in the OONI codebase.)
```Go
}
```
## Running the example program
Let us run the program with default arguments first. You can do
this operation by running:
```bash
go run -race ./internal/tutorial/measurex/chapter01
```
If you do that you obtain some logging messages, which are out of
the scope of this tutorial, and the following JSON:
```JSON
{
"domain": "example.com",
"lookup_host": [
{
"answers": [
{
"answer_type": "A",
"ipv4": "93.184.216.34"
}
],
"engine": "system",
"failure": null,
"hostname": "example.com",
"query_type": "A",
"resolver_address": "",
"t": 0.002996459,
"started": 9.8e-05,
"oddity": ""
},
{
"answers": [
{
"answer_type": "AAAA",
"ivp6": "2606:2800:220:1:248:1893:25c8:1946"
}
],
"engine": "system",
"failure": null,
"hostname": "example.com",
"query_type": "AAAA",
"resolver_address": "",
"t": 0.002996459,
"started": 9.8e-05,
"oddity": ""
}
]
}
```
You see that we have two messages here. OONI splits a DNS
resolution performed using the system resolver into two "fake"
DNS resolutions for A and AAAA. (Under the hood, this is
what the system resolver would most likely do.)
The most important fields are:
- _engine_, indicating that we are using the "system" resolver;
- _hostname_, meaning that we wanted to resolve the "example.com" domain;
- _answers_, which contains a list of answers;
- _t_, which is the time when the LookupHost operation completed.
### NXDOMAIN measurement
Let us now change the domain to resolve to be `antani.ooni.org` (a
nonexisting domain), which we can do by running this command:
```bash
go run -race ./internal/tutorial/measurex/chapter01 -domain antani.ooni.org
```
This is the output JSON:
```JSON
{
"domain": "antani.ooni.org",
"lookup_host": [
{
"answers": null,
"engine": "system",
"failure": "dns_nxdomain_error",
"hostname": "antani.ooni.org",
"query_type": "A",
"resolver_address": "",
"t": 0.072963834,
"started": 0.000125417,
"oddity": "dns.lookup.nxdomain"
},
{
"answers": null,
"engine": "system",
"failure": "dns_nxdomain_error",
"hostname": "antani.ooni.org",
"query_type": "AAAA",
"resolver_address": "",
"t": 0.072963834,
"started": 0.000125417,
"oddity": "dns.lookup.nxdomain"
}
]
}
```
So we see a failure that says there was indeed an NXDOMAIN
error and we also see a field named `oddity`.
What is an oddity? We define oddity something unexpected thay
may be explained by censorship as well as by a transient failure
or other normal network conditions. (In this case, the result
is perfectly normal since we're looking up a nonexistent domain.)
The difference between failure and oddity is that the failure
indicates the error that occurred, while the oddity classifies
the error in the context of the operation during which it
occurred. (In this case the difference is subtle, but we'll
have a better example later, when we'll see what happens on timeout.)
Failures are specified in
[df-007-errors](https://github.com/ooni/spec/blob/master/data-formats/df-007-errors.md).
Inside the `internal/netxlite/errorsx`
package, there is code that maps Go errors to failures. (The
`netxlite` package is the fundamental network package we use, on
top of which `measurex` is written.)
### Measurement with timeout
Let us now try with an insanely low timeout:
```bash
go run -race ./internal/tutorial/measurex/chapter01 -timeout 250us
```
To get this JSON:
```JSON
{
"domain": "example.com",
"lookup_host": [
{
"answers": null,
"engine": "system",
"failure": "generic_timeout_error",
"hostname": "example.com",
"query_type": "A",
"resolver_address": "",
"t": 0.000489167,
"started": 9.2583e-05,
"oddity": "dns.lookup.timeout"
},
{
"answers": null,
"engine": "system",
"failure": "generic_timeout_error",
"hostname": "example.com",
"query_type": "AAAA",
"resolver_address": "",
"t": 0.000489167,
"started": 9.2583e-05,
"oddity": "dns.lookup.timeout"
}
]
}
```
You should now better see the difference between a failure and
an oddity. The context timeout maps to a `generic_timeout_error` while
the oddity clearly indicates the timeout happens during a DNS
lookup. As we mentioned above, the failure is just an error while
an oddity is an error put in context.
## Conclusions
This is it. We have seen how to measure with the system resolver and we have
also seen which easy-to-provoke errors we can get.

368
internal/tutorial/measurex/chapter01/main.go Normal file
View File

@ -0,0 +1,368 @@
// -=-=- StartHere -=-=-
//
// # Chapter I: using the system resolver
//
// In this chapter we explain how to measure DNS resolutions performed
// using the system resolver. *En passant*, we will also introduce you to
// the `Measurer`, which we will use for the rest of the tutorial.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter01/main.go`.)
//
// ## The system resolver
//
// We define "system resolver" as the DNS resolver implemented by the C
// library. On Unix, the most popular interface to such a resolver is
// the `getaddrinfo(3)` C library function.
//
// Most OONI experiments (also known as nettests) use the system
// resolver to map domain names to IP addresses. The advantage of
// the system resolver is that it's provided by the system. So,
// it should _generally_ work. Also, it is the resolver that the
// user of the system will use every day, therefore its results
// should be representative (even though the rise of DNS over
// HTTPS embedded in browsers may make this statement less solid
// than it were ten years ago).
//
// The disadvantage of the system resolver is that we do not
// know how it is configured. Say the user has configured a
// DNS over TLS resolver; then the measurements may miss censorship
// that we would otherwise see if using a custom DNS resolver.
//
// Now that we have justified why the system resolver is
// important for OONI, let us perform some measurements with it.
//
// We will first write a simple `main.go` file that shows how to use
// this functionality. Then, we will show some runs of this file, and
// we will comment the output that we see.
//
// ## main.go
//
// We declare the package and import useful packages. The most
// important package we're importing here is, of course, `internal/measurex`.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
// ```
// ### Setup
//
// We define command line flags useful to test this program. We use
// the `flags` package for that. We want the user to be able to configure
// both the domain name to resolve and the resolution timeout.
//
// ```Go
domain := flag.String("domain", "example.com", "domain to resolve")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
// ```
//
// We call `flag.Parse` to parse the CLI flags.
//
// ```Go
flag.Parse()
// ```
//
// We create a context and we attach a timeout to it. (This is a pretty
// standard way of configuring a timeout in Go.)
//
// ```Go
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
// ```
//
// ### Creating a Measurer
//
// Now we create a `Measurer`.
//
// ```Go
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// The `Measurer` is a concrete type that contains many fields
// requiring initialization. For this reason, we provide a factory
// that creates one with default settings. The expected usage
// pattern is that you do not modify a `Measurer`'s field after
// initialization. Modifying them while the `Measurer` is in
// use could, in fact, lead to races.
//
// Let's now invoke the system resolver to resolve `*domain`!
//
// ### Invoking the system resolver
//
// We call the `LookupHostSystem` method of the `Measurer`. The
// arguments are the Context, that in this case carries the timeout
// we configured above, and the domain to resolve.
//
// The call itself is named `LookupHost` because this is the name
// used by the Go function that performs a domain lookup.
//
// Under the hood, `mx.LookupHostSystem` will eventually call
// `(*net.Resolver).LookupHost`. In turn, in the common case on
// Unix, this function will eventually call `getaddrinfo(3)`.
//
// ```Go
m := mx.LookupHostSystem(ctx, *domain)
// ```
//
// The return value of `(*net.Resolver).LookupHost` is either a
// list of IP addresses or an error. Our `LookupHostSystem` method,
// instead, returns a `*measurex.DNSMeasurement` type.
//
// This is probably a good moment to remind you of Go's
// built in help system. We could include a definition of the
// `DNSMeasurement` structure, but since this definition is
// just a comment in the main.go file, it might age badly.
//
// Instead, if you run
//
// ```
// go doc ./internal/measurex.DNSMeasurement
// ```
//
// You get the current definition. As you can see, this type
// is basically just a wrapper around `Measurement`. Now,
// checking the docs of `Measurement` with
//
// ```
// go doc ./internal/measurex.Measurement
// ```
//
// we can see a container of events
// classified by event type. In our case, because we're
// doing a `LookupHost`, we should have at least one entry
// inside of the `Measurement.LookupHost` field.
//
// This entry is of type `DNSLookupEvent`. Let us check
// together the definition of this type:
//
// ```
// go doc ./internal/measurex.DNSLookupEvent
// ```
//
// If you are familiar with [the OONI data format specs](
// https://github.com/ooni/spec/tree/master/data-formats), you
// should probably recognize that this structure is the Go
// representation of the `df-002-dnst` data format.
//
// In fact, every event field inside of a `Measurement`
// should serialize nicely to JSON to one of the OONI data
// formats.
//
// ### Printing the measurement
//
// Because there is a close relationship between the
// events inside a `Measurement` and the JSON OONI data
// format, in the remainder of this program we're
// going to serialize the `Measurement` to JSON and
// print it to the standard output.
//
// ```Go
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
// ```
//
// As a final note, the `PanicOnError` is here because the
// message `m` *can* be marshalled to JSON. It still feels a
// bit better having an assertion for our assumptions than
// outrightly ignoring the error code. (We tend to use such
// a convention quite frequently in the OONI codebase.)
//
// ```Go
}
// ```
//
// ## Running the example program
//
// Let us run the program with default arguments first. You can do
// this operation by running:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter01
// ```
//
// If you do that you obtain some logging messages, which are out of
// the scope of this tutorial, and the following JSON:
//
// ```JSON
// {
// "domain": "example.com",
// "lookup_host": [
// {
// "answers": [
// {
// "answer_type": "A",
// "ipv4": "93.184.216.34"
// }
// ],
// "engine": "system",
// "failure": null,
// "hostname": "example.com",
// "query_type": "A",
// "resolver_address": "",
// "t": 0.002996459,
// "started": 9.8e-05,
// "oddity": ""
// },
// {
// "answers": [
// {
// "answer_type": "AAAA",
// "ivp6": "2606:2800:220:1:248:1893:25c8:1946"
// }
// ],
// "engine": "system",
// "failure": null,
// "hostname": "example.com",
// "query_type": "AAAA",
// "resolver_address": "",
// "t": 0.002996459,
// "started": 9.8e-05,
// "oddity": ""
// }
// ]
// }
// ```
//
// You see that we have two messages here. OONI splits a DNS
// resolution performed using the system resolver into two "fake"
// DNS resolutions for A and AAAA. (Under the hood, this is
// what the system resolver would most likely do.)
//
// The most important fields are:
//
// - _engine_, indicating that we are using the "system" resolver;
//
// - _hostname_, meaning that we wanted to resolve the "example.com" domain;
//
// - _answers_, which contains a list of answers;
//
// - _t_, which is the time when the LookupHost operation completed.
//
// ### NXDOMAIN measurement
//
// Let us now change the domain to resolve to be `antani.ooni.org` (a
// nonexisting domain), which we can do by running this command:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter01 -domain antani.ooni.org
// ```
//
// This is the output JSON:
//
// ```JSON
// {
// "domain": "antani.ooni.org",
// "lookup_host": [
// {
// "answers": null,
// "engine": "system",
// "failure": "dns_nxdomain_error",
// "hostname": "antani.ooni.org",
// "query_type": "A",
// "resolver_address": "",
// "t": 0.072963834,
// "started": 0.000125417,
// "oddity": "dns.lookup.nxdomain"
// },
// {
// "answers": null,
// "engine": "system",
// "failure": "dns_nxdomain_error",
// "hostname": "antani.ooni.org",
// "query_type": "AAAA",
// "resolver_address": "",
// "t": 0.072963834,
// "started": 0.000125417,
// "oddity": "dns.lookup.nxdomain"
// }
// ]
// }
// ```
//
// So we see a failure that says there was indeed an NXDOMAIN
// error and we also see a field named `oddity`.
//
// What is an oddity? We define oddity something unexpected thay
// may be explained by censorship as well as by a transient failure
// or other normal network conditions. (In this case, the result
// is perfectly normal since we're looking up a nonexistent domain.)
//
// The difference between failure and oddity is that the failure
// indicates the error that occurred, while the oddity classifies
// the error in the context of the operation during which it
// occurred. (In this case the difference is subtle, but we'll
// have a better example later, when we'll see what happens on timeout.)
//
// Failures are specified in
// [df-007-errors](https://github.com/ooni/spec/blob/master/data-formats/df-007-errors.md).
// Inside the `internal/netxlite/errorsx`
// package, there is code that maps Go errors to failures. (The
// `netxlite` package is the fundamental network package we use, on
// top of which `measurex` is written.)
//
// ### Measurement with timeout
//
// Let us now try with an insanely low timeout:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter01 -timeout 250us
// ```
//
// To get this JSON:
//
// ```JSON
// {
// "domain": "example.com",
// "lookup_host": [
// {
// "answers": null,
// "engine": "system",
// "failure": "generic_timeout_error",
// "hostname": "example.com",
// "query_type": "A",
// "resolver_address": "",
// "t": 0.000489167,
// "started": 9.2583e-05,
// "oddity": "dns.lookup.timeout"
// },
// {
// "answers": null,
// "engine": "system",
// "failure": "generic_timeout_error",
// "hostname": "example.com",
// "query_type": "AAAA",
// "resolver_address": "",
// "t": 0.000489167,
// "started": 9.2583e-05,
// "oddity": "dns.lookup.timeout"
// }
// ]
// }
// ```
//
// You should now better see the difference between a failure and
// an oddity. The context timeout maps to a `generic_timeout_error` while
// the oddity clearly indicates the timeout happens during a DNS
// lookup. As we mentioned above, the failure is just an error while
// an oddity is an error put in context.
//
// ## Conclusions
//
// This is it. We have seen how to measure with the system resolver and we have
// also seen which easy-to-provoke errors we can get.
//
// -=-=- StopHere -=-=-

228
internal/tutorial/measurex/chapter02/README.md Normal file
View File

@ -0,0 +1,228 @@
# Chapter II: establishing TCP connections
In this chapter we explain how to measure establishing TCP connections.
We will first write a simple `main.go` file that shows how to use
this functionality. Then, we will show some runs of this file, and
we will comment the output that we see.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter02/main.go`.)
## main.go
We declare the package and import useful packages. The most
important package we're importing here is, of course, `internal/measurex`.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
```
### Setup
This first part of `main.go` is really similar to the previous
chapter, so there is not much new to say here.
```Go
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
```
### Creating a Measurer
We create a `Measurer` like we did in the previous chapter.
```Go
mx := measurex.NewMeasurerWithDefaultSettings()
```
### Establishing a TCP connection
We then call `TCPConnect`, which establishes a connection
and returns the corresponding measurement.
The arguments are the context (for timeouts), and the address
of the endpoint to which we want to connect. (Here and in
most of this tutorial with "endpoint" we mean an IP address
and a port, serialized as "ADDRESS:PORT", where the
address is quoted with "[" and "]" if IPv6, e.g., `[::1]:53`.)
```Go
m := mx.TCPConnect(ctx, *address)
```
### Printing the measurement
The rest of the main function is just like in the previous chapter.
```Go
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
```
## Running the example program
Let us run the program with default arguments first. You can do
this operation by running:
```bash
go run -race ./internal/tutorial/measurex/chapter02
```
Here is the JSON we obtain in output:
```JavaScript
{
// These two fields identify the endpoint
"network": "tcp",
"address": "8.8.4.4:443",
// This block contains the results of the connect syscall
// using the df-008-netevents data format.
"connect": [
{
"address": "8.8.4.4:443",
"failure": null,
"operation": "connect",
"proto": "tcp",
"t": 0.026879041,
"started": 8.8625e-05,
"oddity": ""
}
]
}
```
This is what it says:
- we are connecting a "tcp" socket;
- the destination endpoint address is "8.8.4.4:443";
- connect terminated ~0.027 seconds into the program's life;
- the operation succeeded (`failure` is `nil`).
Let us now see if we can provoke some errors and timeouts.
### Measurement with connection refused
Let us start with an IP address where there's no listening socket:
```bash
go run -race ./internal/tutorial/measurex/chapter02 -address 127.0.0.1:1
```
We get this JSON:
```JSON
{
"network": "tcp",
"address": "127.0.0.1:1",
"connect": [
{
"address": "127.0.0.1:1",
"failure": "connection_refused",
"operation": "connect",
"proto": "tcp",
"t": 0.000372167,
"started": 8.4917e-05,
"oddity": "tcp.connect.refused"
}
]
}
```
And here's an error telling us the connection was refused and
the oddity that classifies the error.
### Measurement with timeouts
Let us now try to obtain a timeout:
```bash
go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1
```
We get this JSON:
```JSON
{
"network": "tcp",
"address": "8.8.4.4:1",
"connect": [
{
"address": "8.8.4.4:1",
"failure": "generic_timeout_error",
"operation": "connect",
"proto": "tcp",
"t": 10.005494583,
"started": 8.4833e-05,
"oddity": "tcp.connect.timeout"
}
]
}
```
So, we clearly see from the value of `t` that our 60 seconds
default timeout did not hit, because there is a lower watchdog
timeout (10 s). We also see again how the oddity is more
precise than just the error alone.
Let us now use a very small timeout:
```bash
go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1 -timeout 100ms
```
To get this JSON:
```JSON
{
"network": "tcp",
"address": "8.8.4.4:1",
"connect": [
{
"address": "8.8.4.4:1",
"failure": "generic_timeout_error",
"operation": "connect",
"proto": "tcp",
"t": 0.10148025,
"started": 0.000122375,
"oddity": "tcp.connect.timeout"
}
]
}
```
We see a timeout after ~0.1s. We enforce a reasonably small
timeout for connecting, equal to 10 s, because we want to
guarantee that measurements eventually terminate. Also, since
often censorship is implemented by timing out, we don't want
to spend to much time waiting for a timeout to expire.
## Conclusions
We have seen how to measure the operation of connecting
to a specific TCP endpoint.

230
internal/tutorial/measurex/chapter02/main.go Normal file
View File

@ -0,0 +1,230 @@
// -=-=- StartHere -=-=-
//
// # Chapter II: establishing TCP connections
//
// In this chapter we explain how to measure establishing TCP connections.
//
// We will first write a simple `main.go` file that shows how to use
// this functionality. Then, we will show some runs of this file, and
// we will comment the output that we see.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter02/main.go`.)
//
// ## main.go
//
// We declare the package and import useful packages. The most
// important package we're importing here is, of course, `internal/measurex`.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
// ```
// ### Setup
//
// This first part of `main.go` is really similar to the previous
// chapter, so there is not much new to say here.
//
// ```Go
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
// ```
//
// ### Creating a Measurer
//
// We create a `Measurer` like we did in the previous chapter.
//
// ```Go
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// ### Establishing a TCP connection
//
// We then call `TCPConnect`, which establishes a connection
// and returns the corresponding measurement.
//
// The arguments are the context (for timeouts), and the address
// of the endpoint to which we want to connect. (Here and in
// most of this tutorial with "endpoint" we mean an IP address
// and a port, serialized as "ADDRESS:PORT", where the
// address is quoted with "[" and "]" if IPv6, e.g., `[::1]:53`.)
//
// ```Go
m := mx.TCPConnect(ctx, *address)
// ```
//
// ### Printing the measurement
//
// The rest of the main function is just like in the previous chapter.
//
// ```Go
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
// ```
//
// ## Running the example program
//
// Let us run the program with default arguments first. You can do
// this operation by running:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter02
// ```
//
// Here is the JSON we obtain in output:
//
// ```JavaScript
// {
// // These two fields identify the endpoint
// "network": "tcp",
// "address": "8.8.4.4:443",
//
// // This block contains the results of the connect syscall
// // using the df-008-netevents data format.
// "connect": [
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "operation": "connect",
// "proto": "tcp",
// "t": 0.026879041,
// "started": 8.8625e-05,
// "oddity": ""
// }
// ]
// }
// ```
//
// This is what it says:
//
// - we are connecting a "tcp" socket;
//
// - the destination endpoint address is "8.8.4.4:443";
//
// - connect terminated ~0.027 seconds into the program's life;
//
// - the operation succeeded (`failure` is `nil`).
//
// Let us now see if we can provoke some errors and timeouts.
//
// ### Measurement with connection refused
//
// Let us start with an IP address where there's no listening socket:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter02 -address 127.0.0.1:1
// ```
//
// We get this JSON:
//
// ```JSON
// {
// "network": "tcp",
// "address": "127.0.0.1:1",
// "connect": [
// {
// "address": "127.0.0.1:1",
// "failure": "connection_refused",
// "operation": "connect",
// "proto": "tcp",
// "t": 0.000372167,
// "started": 8.4917e-05,
// "oddity": "tcp.connect.refused"
// }
// ]
// }
//
// ```
//
// And here's an error telling us the connection was refused and
// the oddity that classifies the error.
//
// ### Measurement with timeouts
//
// Let us now try to obtain a timeout:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1
// ```
//
// We get this JSON:
//
// ```JSON
// {
// "network": "tcp",
// "address": "8.8.4.4:1",
// "connect": [
// {
// "address": "8.8.4.4:1",
// "failure": "generic_timeout_error",
// "operation": "connect",
// "proto": "tcp",
// "t": 10.005494583,
// "started": 8.4833e-05,
// "oddity": "tcp.connect.timeout"
// }
// ]
// }
// ```
//
// So, we clearly see from the value of `t` that our 60 seconds
// default timeout did not hit, because there is a lower watchdog
// timeout (10 s). We also see again how the oddity is more
// precise than just the error alone.
//
// Let us now use a very small timeout:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1 -timeout 100ms
// ```
//
// To get this JSON:
//
// ```JSON
// {
// "network": "tcp",
// "address": "8.8.4.4:1",
// "connect": [
// {
// "address": "8.8.4.4:1",
// "failure": "generic_timeout_error",
// "operation": "connect",
// "proto": "tcp",
// "t": 0.10148025,
// "started": 0.000122375,
// "oddity": "tcp.connect.timeout"
// }
// ]
// }
// ```
//
// We see a timeout after ~0.1s. We enforce a reasonably small
// timeout for connecting, equal to 10 s, because we want to
// guarantee that measurements eventually terminate. Also, since
// often censorship is implemented by timing out, we don't want
// to spend to much time waiting for a timeout to expire.
//
// ## Conclusions
//
// We have seen how to measure the operation of connecting
// to a specific TCP endpoint.
//
// -=-=- StopHere -=-=-

567
internal/tutorial/measurex/chapter03/README.md Normal file
View File

@ -0,0 +1,567 @@
# Chapter III: using a custom DNS-over-UDP resolver
In this chapter we learn how to measure sending DNS queries to
a DNS server speaking the DNS-over-UDP protocol.
Without further ado, let's describe our example `main.go` program
and let's use it to better understand this flow.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter03/main.go`.)
## main.go
The initial part of the program is pretty much the same as the one
used in previous chapters, so I will not add further comments.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
query := flag.String("query", "example.com", "domain to resolver")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
```
### Using a custom UDP resolver
We now invoke `LookupHostUDP`. We specify:
- a context for timeout information;
- the domain to query for;
- the address of the DNS-over-UDP server endpoint.
```Go
m := mx.LookupHostUDP(ctx, *query, *address)
```
Also this operation returns a measurement, which
we print using the usual three-liner.
```Go
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
```
## Running the example program
As before, let us start off with a vanilla run:
```bash
go run -race ./internal/tutorial/measurex/chapter03
```
This time we get a much larger JSON, so I will pretend it is
actually JavaScript and add comments to explain it inline.
(This is the first case in which we see how a single
method call for measurer causes several events to
be generated and inserted into a `Measurement`.)
```JavaScript
{
"domain": "example.com",
// This block tells us about the UDP connect events
// where we bind to the server's endpoint
"connect": [
{
"address": "8.8.4.4:53",
"failure": null,
"operation": "connect",
"proto": "udp",
"t": 0.00043175,
"started": 0.000191958,
"oddity": ""
},
{
"address": "8.8.4.4:53",
"failure": null,
"operation": "connect",
"proto": "udp",
"t": 0.042198458,
"started": 0.042113208,
"oddity": ""
}
],
// This block shows the read and write events
// occurred on the sockets (because we control
// in full the implementation of this DNS
// over UDP resolver, we can see these events)
"read_write": [
{
"address": "8.8.4.4:53",
"failure": null,
"num_bytes": 29,
"operation": "write",
"proto": "udp",
"t": 0.000459583,
"started": 0.00043825,
"oddity": ""
},
{
"address": "8.8.4.4:53",
"failure": null,
"num_bytes": 45,
"operation": "read",
"proto": "udp",
"t": 0.041955792,
"started": 0.000471833,
"oddity": ""
},
{
"address": "8.8.4.4:53",
"failure": null,
"num_bytes": 29,
"operation": "write",
"proto": "udp",
"t": 0.042218917,
"started": 0.042203,
"oddity": ""
},
{
"address": "8.8.4.4:53",
"failure": null,
"num_bytes": 57,
"operation": "read",
"proto": "udp",
"t": 0.196646583,
"started": 0.042233167,
"oddity": ""
}
],
// This is the same kind of result as before, we
// show the emitted queries and the resolved addrs.
//
// Also note how here the resolver_address is the
// correct endpoint address and the engine tells us
// that we're using DNS over UDP.
"lookup_host": [
{
"answers": [
{
"answer_type": "A",
"ipv4": "93.184.216.34"
}
],
"engine": "udp",
"failure": null,
"hostname": "example.com",
"query_type": "A",
"resolver_address": "8.8.4.4:53",
"t": 0.196777042,
"started": 0.000118542,
"oddity": ""
},
{
"answers": [
{
"answer_type": "AAAA",
"ivp6": "2606:2800:220:1:248:1893:25c8:1946"
}
],
"engine": "udp",
"failure": null,
"hostname": "example.com",
"query_type": "AAAA",
"resolver_address": "8.8.4.4:53",
"t": 0.196777042,
"started": 0.000118542,
"oddity": ""
}
],
// This block shows the query we sent (encoded as base64)
// and the response we received. Here we clearly see
// that we perform two DNS "round trip" (i.e., send request
// and receive response) to resolve a domain: one for
// A and the other for AAAA.
"dns_round_trip": [
{
"engine": "udp",
"resolver_address": "8.8.4.4:53",
"raw_query": {
"data": "PrcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
"format": "base64"
},
"started": 0.000191625,
"t": 0.041998667,
"failure": null,
"raw_reply": {
"data": "PreBgAABAAEAAAAAB2V4YW1wbGUDY29tAAABAAHADAABAAEAAE8BAARduNgi",
"format": "base64"
}
},
{
"engine": "udp",
"resolver_address": "8.8.4.4:53",
"raw_query": {
"data": "LAwBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=",
"format": "base64"
},
"started": 0.04210775,
"t": 0.196701333,
"failure": null,
"raw_reply": {
"data": "LAyBgAABAAEAAAAAB2V4YW1wbGUDY29tAAAcAAHADAAcAAEAAE6nABAmBigAAiAAAQJIGJMlyBlG",
"format": "base64"
}
}
]
}
```
This data format is really an extension of the `LookupHostSystem`
one. It just adds more fields that clarify what happened at low
level in terms of socket I/O and queries sent and received.
Let us now try to provoke some errors and see how the
output JSON changes because of them.
### Measurement with NXDOMAIN
Let us try to get a NXDOMAIN error.
```bash
go run -race ./internal/tutorial/measurex/chapter03 -query antani.ooni.org
```
This produces the following JSON:
```JavaScript
{
"domain": "antani.ooni.org",
"connect": [ /* snip */ ],
"read_write": [ /* snip */ ],
"lookup_host": [
{
"answers": null,
"engine": "udp",
"failure": "dns_nxdomain_error",
"hostname": "antani.ooni.org",
"query_type": "A",
"resolver_address": "8.8.4.4:53",
"t": 0.098208709,
"started": 8.95e-05,
"oddity": "dns.lookup.nxdomain"
},
{
"answers": null,
"engine": "udp",
"failure": "dns_nxdomain_error",
"hostname": "antani.ooni.org",
"query_type": "AAAA",
"resolver_address": "8.8.4.4:53",
"t": 0.098208709,
"started": 8.95e-05,
"oddity": "dns.lookup.nxdomain"
}
],
"dns_round_trip": [
{
"engine": "udp",
"resolver_address": "8.8.4.4:53",
"raw_query": {
"data": "jLIBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAAQAB",
"format": "base64"
},
"started": 0.000141542,
"t": 0.034689417,
"failure": null,
"raw_reply": {
"data": "jLKBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAAQABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==",
"format": "base64"
}
},
{
"engine": "udp",
"resolver_address": "8.8.4.4:53",
"raw_query": {
"data": "azEBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAHAAB",
"format": "base64"
},
"started": 0.034776709,
"t": 0.098170542,
"failure": null,
"raw_reply": {
"data": "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==",
"format": "base64"
}
}
]
}
```
We indeed get a NXDOMAIN error as the failure in `lookup_host`.
Let us now decode one of the replies by using this program:
```
package main
import (
"fmt"
"encoding/base64"
"github.com/miekg/dns"
)
func main() {
const query = "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ=="
data, _ := base64.StdEncoding.DecodeString(query)
msg := new(dns.Msg)
_ = msg.Unpack(data)
fmt.Printf("%s\n", msg)
}
```
where `query` is one of the replies. If we run this program
we get as the output:
```
;; opcode: QUERY, status: NXDOMAIN, id: 27441
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;antani.ooni.org. IN AAAA
;; AUTHORITY SECTION:
ooni.org. 1800 IN SOA dns1.registrar-servers.com. hostmaster.registrar-servers.com. 1627397372 43200 3600 604800 3601
```
### Measurement with timeout
Let us now query an IP address known for not responding
to DNS queries, to get a timeout.
```bash
go run -race ./internal/tutorial/measurex/chapter03 -address 182.92.22.222:53
```
Here's the corresponding JSON:
```JavaScript
{
"domain": "example.com",
"connect": [ /* snip */ ],
"read_write": [
{
"address": "182.92.22.222:53",
"failure": null,
"num_bytes": 29,
"operation": "write",
"proto": "udp",
"t": 0.0005275,
"started": 0.000500209,
"oddity": ""
},
{
"address": "182.92.22.222:53",
"failure": "generic_timeout_error", /* <--- */
"operation": "read",
"proto": "udp",
"t": 5.001140125,
"started": 0.000544042,
"oddity": ""
}
],
"lookup_host": [
{
"answers": null,
"engine": "udp",
"failure": "generic_timeout_error", /* <--- */
"hostname": "example.com",
"query_type": "A",
"resolver_address": "182.92.22.222:53",
"t": 5.001462084,
"started": 0.000127917,
"oddity": "dns.lookup.timeout" /* <--- */
},
{
"answers": null,
"engine": "udp",
"failure": "generic_timeout_error",
"hostname": "example.com",
"query_type": "AAAA",
"resolver_address": "182.92.22.222:53",
"t": 5.001462084,
"started": 0.000127917,
"oddity": "dns.lookup.timeout"
}
],
"dns_round_trip": [
{
"engine": "udp",
"resolver_address": "182.92.22.222:53",
"raw_query": {
"data": "ej8BAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
"format": "base64"
},
"started": 0.000220584,
"t": 5.001317417,
"failure": "generic_timeout_error",
"raw_reply": null
}
]
}
```
We see that we do fail with a timeout (I have marked some of them
with comments inside the JSON). We see the timeout at three different
level of abstractions (from lower to higher abstraction): at the socket layer,
during the DNS round trip, during the DNS lookup.
What we also see is that `t`'s value is ~5s when the `read` event
fails, which tells us about the socket's read timeout.
### Measurement with REFUSED error
Let us now try to get a REFUSED DNS Rcode, again from servers
that are, let's say, kind enough to easily help.
```bash
go run -race ./internal/tutorial/measurex/chapter03 -address 180.97.36.63:53
```
Here's the answer I get:
```JavaScript
{
"domain": "example.com",
"connect": [ /* snip */ ],
// The I/O events look normal this time
"read_write": [
{
"address": "180.97.36.63:53",
"failure": null,
"num_bytes": 29,
"operation": "write",
"proto": "udp",
"t": 0.000333583,
"started": 0.000312125,
"oddity": ""
},
{
"address": "180.97.36.63:53",
"failure": null,
"num_bytes": 29,
"operation": "read",
"proto": "udp",
"t": 0.334948125,
"started": 0.000366625,
"oddity": ""
},
{
"address": "180.97.36.63:53",
"failure": null,
"num_bytes": 29,
"operation": "write",
"proto": "udp",
"t": 0.3358025,
"started": 0.335725958,
"oddity": ""
},
{
"address": "180.97.36.63:53",
"failure": null,
"num_bytes": 29,
"operation": "read",
"proto": "udp",
"t": 0.739987666,
"started": 0.335863875,
"oddity": ""
}
],
// But we see both in the error and in the oddity
// that the response was "REFUSED"
"lookup_host": [
{
"answers": null,
"engine": "udp",
"failure": "dns_refused_error",
"hostname": "example.com",
"query_type": "A",
"resolver_address": "180.97.36.63:53",
"t": 0.7402975,
"started": 7.2291e-05,
"oddity": "dns.lookup.refused"
},
{
"answers": null,
"engine": "udp",
"failure": "dns_refused_error",
"hostname": "example.com",
"query_type": "AAAA",
"resolver_address": "180.97.36.63:53",
"t": 0.7402975,
"started": 7.2291e-05,
"oddity": "dns.lookup.refused"
}
],
// Exercise: do like I did before and decode the messages
"dns_round_trip": [
{
"engine": "udp",
"resolver_address": "180.97.36.63:53",
"raw_query": {
"data": "crkBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
"format": "base64"
},
"started": 0.000130666,
"t": 0.33509925,
"failure": null,
"raw_reply": {
"data": "crmBBQABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
"format": "base64"
}
},
{
"engine": "udp",
"resolver_address": "180.97.36.63:53",
"raw_query": {
"data": "ywcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=",
"format": "base64"
},
"started": 0.335321333,
"t": 0.740152375,
"failure": null,
"raw_reply": {
"data": "yweBBQABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=",
"format": "base64"
}
}
]
}
```
## Conclusion
We have seen how we sending DNS queries over UDP, measure the
results, and what happens on common error conditions.

569
internal/tutorial/measurex/chapter03/main.go Normal file
View File

@ -0,0 +1,569 @@
// -=-=- StartHere -=-=-
//
// # Chapter III: using a custom DNS-over-UDP resolver
//
// In this chapter we learn how to measure sending DNS queries to
// a DNS server speaking the DNS-over-UDP protocol.
//
// Without further ado, let's describe our example `main.go` program
// and let's use it to better understand this flow.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter03/main.go`.)
//
// ## main.go
//
// The initial part of the program is pretty much the same as the one
// used in previous chapters, so I will not add further comments.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
query := flag.String("query", "example.com", "domain to resolver")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// ### Using a custom UDP resolver
//
// We now invoke `LookupHostUDP`. We specify:
//
// - a context for timeout information;
//
// - the domain to query for;
//
// - the address of the DNS-over-UDP server endpoint.
//
// ```Go
m := mx.LookupHostUDP(ctx, *query, *address)
// ```
//
// Also this operation returns a measurement, which
// we print using the usual three-liner.
//
// ```Go
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
// ```
//
// ## Running the example program
//
// As before, let us start off with a vanilla run:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter03
// ```
//
// This time we get a much larger JSON, so I will pretend it is
// actually JavaScript and add comments to explain it inline.
//
// (This is the first case in which we see how a single
// method call for measurer causes several events to
// be generated and inserted into a `Measurement`.)
//
// ```JavaScript
// {
// "domain": "example.com",
//
// // This block tells us about the UDP connect events
// // where we bind to the server's endpoint
// "connect": [
// {
// "address": "8.8.4.4:53",
// "failure": null,
// "operation": "connect",
// "proto": "udp",
// "t": 0.00043175,
// "started": 0.000191958,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:53",
// "failure": null,
// "operation": "connect",
// "proto": "udp",
// "t": 0.042198458,
// "started": 0.042113208,
// "oddity": ""
// }
// ],
//
// // This block shows the read and write events
// // occurred on the sockets (because we control
// // in full the implementation of this DNS
// // over UDP resolver, we can see these events)
// "read_write": [
// {
// "address": "8.8.4.4:53",
// "failure": null,
// "num_bytes": 29,
// "operation": "write",
// "proto": "udp",
// "t": 0.000459583,
// "started": 0.00043825,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:53",
// "failure": null,
// "num_bytes": 45,
// "operation": "read",
// "proto": "udp",
// "t": 0.041955792,
// "started": 0.000471833,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:53",
// "failure": null,
// "num_bytes": 29,
// "operation": "write",
// "proto": "udp",
// "t": 0.042218917,
// "started": 0.042203,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:53",
// "failure": null,
// "num_bytes": 57,
// "operation": "read",
// "proto": "udp",
// "t": 0.196646583,
// "started": 0.042233167,
// "oddity": ""
// }
// ],
//
// // This is the same kind of result as before, we
// // show the emitted queries and the resolved addrs.
// //
// // Also note how here the resolver_address is the
// // correct endpoint address and the engine tells us
// // that we're using DNS over UDP.
// "lookup_host": [
// {
// "answers": [
// {
// "answer_type": "A",
// "ipv4": "93.184.216.34"
// }
// ],
// "engine": "udp",
// "failure": null,
// "hostname": "example.com",
// "query_type": "A",
// "resolver_address": "8.8.4.4:53",
// "t": 0.196777042,
// "started": 0.000118542,
// "oddity": ""
// },
// {
// "answers": [
// {
// "answer_type": "AAAA",
// "ivp6": "2606:2800:220:1:248:1893:25c8:1946"
// }
// ],
// "engine": "udp",
// "failure": null,
// "hostname": "example.com",
// "query_type": "AAAA",
// "resolver_address": "8.8.4.4:53",
// "t": 0.196777042,
// "started": 0.000118542,
// "oddity": ""
// }
// ],
//
// // This block shows the query we sent (encoded as base64)
// // and the response we received. Here we clearly see
// // that we perform two DNS "round trip" (i.e., send request
// // and receive response) to resolve a domain: one for
// // A and the other for AAAA.
// "dns_round_trip": [
// {
// "engine": "udp",
// "resolver_address": "8.8.4.4:53",
// "raw_query": {
// "data": "PrcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
// "format": "base64"
// },
// "started": 0.000191625,
// "t": 0.041998667,
// "failure": null,
// "raw_reply": {
// "data": "PreBgAABAAEAAAAAB2V4YW1wbGUDY29tAAABAAHADAABAAEAAE8BAARduNgi",
// "format": "base64"
// }
// },
// {
// "engine": "udp",
// "resolver_address": "8.8.4.4:53",
// "raw_query": {
// "data": "LAwBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=",
// "format": "base64"
// },
// "started": 0.04210775,
// "t": 0.196701333,
// "failure": null,
// "raw_reply": {
// "data": "LAyBgAABAAEAAAAAB2V4YW1wbGUDY29tAAAcAAHADAAcAAEAAE6nABAmBigAAiAAAQJIGJMlyBlG",
// "format": "base64"
// }
// }
// ]
// }
// ```
//
// This data format is really an extension of the `LookupHostSystem`
// one. It just adds more fields that clarify what happened at low
// level in terms of socket I/O and queries sent and received.
//
// Let us now try to provoke some errors and see how the
// output JSON changes because of them.
//
// ### Measurement with NXDOMAIN
//
// Let us try to get a NXDOMAIN error.
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter03 -query antani.ooni.org
// ```
//
// This produces the following JSON:
//
// ```JavaScript
// {
// "domain": "antani.ooni.org",
// "connect": [ /* snip */ ],
// "read_write": [ /* snip */ ],
// "lookup_host": [
// {
// "answers": null,
// "engine": "udp",
// "failure": "dns_nxdomain_error",
// "hostname": "antani.ooni.org",
// "query_type": "A",
// "resolver_address": "8.8.4.4:53",
// "t": 0.098208709,
// "started": 8.95e-05,
// "oddity": "dns.lookup.nxdomain"
// },
// {
// "answers": null,
// "engine": "udp",
// "failure": "dns_nxdomain_error",
// "hostname": "antani.ooni.org",
// "query_type": "AAAA",
// "resolver_address": "8.8.4.4:53",
// "t": 0.098208709,
// "started": 8.95e-05,
// "oddity": "dns.lookup.nxdomain"
// }
// ],
// "dns_round_trip": [
// {
// "engine": "udp",
// "resolver_address": "8.8.4.4:53",
// "raw_query": {
// "data": "jLIBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAAQAB",
// "format": "base64"
// },
// "started": 0.000141542,
// "t": 0.034689417,
// "failure": null,
// "raw_reply": {
// "data": "jLKBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAAQABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==",
// "format": "base64"
// }
// },
// {
// "engine": "udp",
// "resolver_address": "8.8.4.4:53",
// "raw_query": {
// "data": "azEBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAHAAB",
// "format": "base64"
// },
// "started": 0.034776709,
// "t": 0.098170542,
// "failure": null,
// "raw_reply": {
// "data": "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==",
// "format": "base64"
// }
// }
// ]
// }
// ```
//
// We indeed get a NXDOMAIN error as the failure in `lookup_host`.
//
// Let us now decode one of the replies by using this program:
//
// ```
// package main
//
// import (
// "fmt"
// "encoding/base64"
//
// "github.com/miekg/dns"
// )
//
// func main() {
// const query = "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ=="
// data, _ := base64.StdEncoding.DecodeString(query)
// msg := new(dns.Msg)
// _ = msg.Unpack(data)
// fmt.Printf("%s\n", msg)
//}
// ```
//
// where `query` is one of the replies. If we run this program
// we get as the output:
//
// ```
// ;; opcode: QUERY, status: NXDOMAIN, id: 27441
// ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
//
// ;; QUESTION SECTION:
// ;antani.ooni.org. IN AAAA
//
// ;; AUTHORITY SECTION:
// ooni.org. 1800 IN SOA dns1.registrar-servers.com. hostmaster.registrar-servers.com. 1627397372 43200 3600 604800 3601
// ```
//
// ### Measurement with timeout
//
// Let us now query an IP address known for not responding
// to DNS queries, to get a timeout.
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter03 -address 182.92.22.222:53
// ```
//
// Here's the corresponding JSON:
//
// ```JavaScript
// {
// "domain": "example.com",
// "connect": [ /* snip */ ],
// "read_write": [
// {
// "address": "182.92.22.222:53",
// "failure": null,
// "num_bytes": 29,
// "operation": "write",
// "proto": "udp",
// "t": 0.0005275,
// "started": 0.000500209,
// "oddity": ""
// },
// {
// "address": "182.92.22.222:53",
// "failure": "generic_timeout_error", /* <--- */
// "operation": "read",
// "proto": "udp",
// "t": 5.001140125,
// "started": 0.000544042,
// "oddity": ""
// }
// ],
// "lookup_host": [
// {
// "answers": null,
// "engine": "udp",
// "failure": "generic_timeout_error", /* <--- */
// "hostname": "example.com",
// "query_type": "A",
// "resolver_address": "182.92.22.222:53",
// "t": 5.001462084,
// "started": 0.000127917,
// "oddity": "dns.lookup.timeout" /* <--- */
// },
// {
// "answers": null,
// "engine": "udp",
// "failure": "generic_timeout_error",
// "hostname": "example.com",
// "query_type": "AAAA",
// "resolver_address": "182.92.22.222:53",
// "t": 5.001462084,
// "started": 0.000127917,
// "oddity": "dns.lookup.timeout"
// }
// ],
// "dns_round_trip": [
// {
// "engine": "udp",
// "resolver_address": "182.92.22.222:53",
// "raw_query": {
// "data": "ej8BAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
// "format": "base64"
// },
// "started": 0.000220584,
// "t": 5.001317417,
// "failure": "generic_timeout_error",
// "raw_reply": null
// }
// ]
// }
// ```
//
// We see that we do fail with a timeout (I have marked some of them
// with comments inside the JSON). We see the timeout at three different
// level of abstractions (from lower to higher abstraction): at the socket layer,
// during the DNS round trip, during the DNS lookup.
//
// What we also see is that `t`'s value is ~5s when the `read` event
// fails, which tells us about the socket's read timeout.
//
// ### Measurement with REFUSED error
//
// Let us now try to get a REFUSED DNS Rcode, again from servers
// that are, let's say, kind enough to easily help.
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter03 -address 180.97.36.63:53
// ```
//
// Here's the answer I get:
//
// ```JavaScript
// {
// "domain": "example.com",
// "connect": [ /* snip */ ],
//
// // The I/O events look normal this time
// "read_write": [
// {
// "address": "180.97.36.63:53",
// "failure": null,
// "num_bytes": 29,
// "operation": "write",
// "proto": "udp",
// "t": 0.000333583,
// "started": 0.000312125,
// "oddity": ""
// },
// {
// "address": "180.97.36.63:53",
// "failure": null,
// "num_bytes": 29,
// "operation": "read",
// "proto": "udp",
// "t": 0.334948125,
// "started": 0.000366625,
// "oddity": ""
// },
// {
// "address": "180.97.36.63:53",
// "failure": null,
// "num_bytes": 29,
// "operation": "write",
// "proto": "udp",
// "t": 0.3358025,
// "started": 0.335725958,
// "oddity": ""
// },
// {
// "address": "180.97.36.63:53",
// "failure": null,
// "num_bytes": 29,
// "operation": "read",
// "proto": "udp",
// "t": 0.739987666,
// "started": 0.335863875,
// "oddity": ""
// }
// ],
//
// // But we see both in the error and in the oddity
// // that the response was "REFUSED"
// "lookup_host": [
// {
// "answers": null,
// "engine": "udp",
// "failure": "dns_refused_error",
// "hostname": "example.com",
// "query_type": "A",
// "resolver_address": "180.97.36.63:53",
// "t": 0.7402975,
// "started": 7.2291e-05,
// "oddity": "dns.lookup.refused"
// },
// {
// "answers": null,
// "engine": "udp",
// "failure": "dns_refused_error",
// "hostname": "example.com",
// "query_type": "AAAA",
// "resolver_address": "180.97.36.63:53",
// "t": 0.7402975,
// "started": 7.2291e-05,
// "oddity": "dns.lookup.refused"
// }
// ],
//
// // Exercise: do like I did before and decode the messages
// "dns_round_trip": [
// {
// "engine": "udp",
// "resolver_address": "180.97.36.63:53",
// "raw_query": {
// "data": "crkBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
// "format": "base64"
// },
// "started": 0.000130666,
// "t": 0.33509925,
// "failure": null,
// "raw_reply": {
// "data": "crmBBQABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=",
// "format": "base64"
// }
// },
// {
// "engine": "udp",
// "resolver_address": "180.97.36.63:53",
// "raw_query": {
// "data": "ywcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=",
// "format": "base64"
// },
// "started": 0.335321333,
// "t": 0.740152375,
// "failure": null,
// "raw_reply": {
// "data": "yweBBQABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=",
// "format": "base64"
// }
// }
// ]
// }
// ```
//
// ## Conclusion
//
// We have seen how we sending DNS queries over UDP, measure the
// results, and what happens on common error conditions.
//
// -=-=- StopHere -=-=-

247
internal/tutorial/measurex/chapter04/README.md Normal file
View File

@ -0,0 +1,247 @@
# Chapter IV: TLS handshaking
This chapter describes measuring TLS handshakes.
Without further ado, let's describe our example `main.go` program
and let's use it to better understand how to measure that.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter04/main.go`.)
## main.go
The initial part of the program is pretty much the same as the one
used in previous chapters, so I will not add further comments.
```Go
package main
import (
"context"
"crypto/tls"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/netxlite"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
sni := flag.String("sni", "dns.google", "domain to resolver")
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
```
### Connecting and handshaking.
We call the `ConnectAndHandshake` method. The arguments
are the context, the address, and a TLS config.
Under the hood, the code will call the TCP connect functionality
we have seen in chapter02, using the address argument. Then, if
successful, it will TLS handshake using the given TLS config.
```Go
m := mx.TLSConnectAndHandshake(ctx, *address, &tls.Config{
ServerName: *sni,
NextProtos: []string{"h2", "http/1.1"},
RootCAs: netxlite.NewDefaultCertPool(),
})
```
Regarding the TLS config, in particular,
the three fields above are the files you should always set
in a TLS config when doing handshakes manually. The `ServerName`
field forces the SNI, the NextProtos field forces the ALPN,
and the `RootCAs` field is overridden so that we use the
CA bundle that we bundle with OONI. (This CA bundle is the
same you can find at https://curl.haxx.se/ca/.)
As usual, the method to perform a measurement returns
the measurement itself, which we print below.
```
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
```
## Running the example program
As before, let us start off with a vanilla run:
```bash
go run -race ./internal/tutorial/measurex/chapter04
```
Let us comment the JSON in detail:
```JavaScript
{
"network": "tcp",
"address": "8.8.4.4:443",
// This block is generated when connecting to a TCP
// socket, as we've already seen in chapter02
"connect": [
{
"address": "8.8.4.4:443",
"failure": null,
"operation": "connect",
"proto": "tcp",
"t": 0.046959084,
"started": 0.022998875,
"oddity": ""
}
],
// These are the I/O events during the handshake
"read_write": [
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 280,
"operation": "write",
"proto": "tcp",
"t": 0.048752875,
"started": 0.04874125,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 517,
"operation": "read",
"proto": "tcp",
"t": 0.087221334,
"started": 0.048760417,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 4301,
"operation": "read",
"proto": "tcp",
"t": 0.088843584,
"started": 0.088830959,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 64,
"operation": "write",
"proto": "tcp",
"t": 0.092078042,
"started": 0.092064042,
"oddity": ""
}
],
// This block contains information about the handshake
"tls_handshake": [
{
"cipher_suite": "TLS_AES_128_GCM_SHA256",
"failure": null,
"negotiated_proto": "h2",
"tls_version": "TLSv1.3",
"peer_certificates": [
{
"data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc",
"format": "base64"
},
{
"data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd",
"format": "base64"
},
{
"data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=",
"format": "base64"
}
],
"t": 0.092117709,
"address": "8.8.4.4:443",
"server_name": "dns.google",
"alpn": [
"h2",
"http/1.1"
],
"no_tls_verify": false,
"oddity": "",
"proto": "tcp",
"started": 0.047288542
}
]
}
```
All the data formats we're using here are, by the way,
compatible with the data formats specified at
https://github.com/ooni/spec/tree/master/data-formats.
### Suggested follow-up experiments
Try to run experiments in the following scenarios, and
check the output JSON to familiarize with what changes in
different error conditions.
1. measurement that causes timeout
2. measurement with wrong SNI
3. measurement with self-signed certificate
4. measurement with expired certificate
5. measurement with connection reset during handshake
6. measurement with timeout during handshake
Here are the commands I used for each proposed exercise:
1. go run -race ./internal/tutorial/measurex/chapter04 -address 8.8.4.4:1
2. go run -race ./internal/tutorial/measurex/chapter04 -sni example.org
3. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni self-signed.badssl.com
4. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni expire.badssl.com
To emulate the two last scenario, if you're on Linux, a
possibility is building Jafar with this command:
```
go build -v ./internal/cmd/jafar
```
Then, for example, to provoke a connection reset you
can run in a terminal:
```
sudo ./jafar -iptables-reset-keyword dns.google
```
and you can run this tutorial with `dns.google` as
the SNI in another terminal.
Likewise, you can obtain a timeout using the
`-iptables-drop-keyword` flag instead.
(Jafar runs forever and censors. You need to use
`^C` to terminate it from running.)
## Conclusion
We have seen how to measure TLS handshakes. We have seen how
this flow produces different output on different error conditions.

249
internal/tutorial/measurex/chapter04/main.go Normal file
View File

@ -0,0 +1,249 @@
// -=-=- StartHere -=-=-
//
// # Chapter IV: TLS handshaking
//
// This chapter describes measuring TLS handshakes.
//
// Without further ado, let's describe our example `main.go` program
// and let's use it to better understand how to measure that.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter04/main.go`.)
//
// ## main.go
//
// The initial part of the program is pretty much the same as the one
// used in previous chapters, so I will not add further comments.
//
// ```Go
package main
import (
"context"
"crypto/tls"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/netxlite"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
sni := flag.String("sni", "dns.google", "domain to resolver")
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// ### Connecting and handshaking.
//
// We call the `ConnectAndHandshake` method. The arguments
// are the context, the address, and a TLS config.
//
// Under the hood, the code will call the TCP connect functionality
// we have seen in chapter02, using the address argument. Then, if
// successful, it will TLS handshake using the given TLS config.
//
// ```Go
m := mx.TLSConnectAndHandshake(ctx, *address, &tls.Config{
ServerName: *sni,
NextProtos: []string{"h2", "http/1.1"},
RootCAs: netxlite.NewDefaultCertPool(),
})
// ```
//
// Regarding the TLS config, in particular,
// the three fields above are the files you should always set
// in a TLS config when doing handshakes manually. The `ServerName`
// field forces the SNI, the NextProtos field forces the ALPN,
// and the `RootCAs` field is overridden so that we use the
// CA bundle that we bundle with OONI. (This CA bundle is the
// same you can find at https://curl.haxx.se/ca/.)
//
// As usual, the method to perform a measurement returns
// the measurement itself, which we print below.
//
// ```
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
// ```
// ## Running the example program
//
// As before, let us start off with a vanilla run:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter04
// ```
//
// Let us comment the JSON in detail:
//
// ```JavaScript
// {
// "network": "tcp",
// "address": "8.8.4.4:443",
//
// // This block is generated when connecting to a TCP
// // socket, as we've already seen in chapter02
// "connect": [
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "operation": "connect",
// "proto": "tcp",
// "t": 0.046959084,
// "started": 0.022998875,
// "oddity": ""
// }
// ],
//
// // These are the I/O events during the handshake
// "read_write": [
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 280,
// "operation": "write",
// "proto": "tcp",
// "t": 0.048752875,
// "started": 0.04874125,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 517,
// "operation": "read",
// "proto": "tcp",
// "t": 0.087221334,
// "started": 0.048760417,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 4301,
// "operation": "read",
// "proto": "tcp",
// "t": 0.088843584,
// "started": 0.088830959,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 64,
// "operation": "write",
// "proto": "tcp",
// "t": 0.092078042,
// "started": 0.092064042,
// "oddity": ""
// }
// ],
//
// // This block contains information about the handshake
// "tls_handshake": [
// {
// "cipher_suite": "TLS_AES_128_GCM_SHA256",
// "failure": null,
// "negotiated_proto": "h2",
// "tls_version": "TLSv1.3",
// "peer_certificates": [
// {
// "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc",
// "format": "base64"
// },
// {
// "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd",
// "format": "base64"
// },
// {
// "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=",
// "format": "base64"
// }
// ],
// "t": 0.092117709,
// "address": "8.8.4.4:443",
// "server_name": "dns.google",
// "alpn": [
// "h2",
// "http/1.1"
// ],
// "no_tls_verify": false,
// "oddity": "",
// "proto": "tcp",
// "started": 0.047288542
// }
// ]
// }
// ```
//
// All the data formats we're using here are, by the way,
// compatible with the data formats specified at
// https://github.com/ooni/spec/tree/master/data-formats.
//
// ### Suggested follow-up experiments
//
// Try to run experiments in the following scenarios, and
// check the output JSON to familiarize with what changes in
// different error conditions.
//
// 1. measurement that causes timeout
//
// 2. measurement with wrong SNI
//
// 3. measurement with self-signed certificate
//
// 4. measurement with expired certificate
//
// 5. measurement with connection reset during handshake
//
// 6. measurement with timeout during handshake
//
// Here are the commands I used for each proposed exercise:
//
// 1. go run -race ./internal/tutorial/measurex/chapter04 -address 8.8.4.4:1
//
// 2. go run -race ./internal/tutorial/measurex/chapter04 -sni example.org
//
// 3. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni self-signed.badssl.com
//
// 4. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni expire.badssl.com
//
// To emulate the two last scenario, if you're on Linux, a
// possibility is building Jafar with this command:
//
// ```
// go build -v ./internal/cmd/jafar
// ```
//
// Then, for example, to provoke a connection reset you
// can run in a terminal:
//
// ```
// sudo ./jafar -iptables-reset-keyword dns.google
// ```
//
// and you can run this tutorial with `dns.google` as
// the SNI in another terminal.
//
// Likewise, you can obtain a timeout using the
// `-iptables-drop-keyword` flag instead.
//
// (Jafar runs forever and censors. You need to use
// `^C` to terminate it from running.)
//
// ## Conclusion
//
// We have seen how to measure TLS handshakes. We have seen how
// this flow produces different output on different error conditions.
//
// -=-=- StopHere -=-=-

280
internal/tutorial/measurex/chapter05/README.md Normal file
View File

@ -0,0 +1,280 @@
# Chapter V: QUIC handshaking
This chapter describes measuring QUIC handshakes. Conceptually,
and code wise, this is very similar to the previous chapter.
The API call, in fact, has exactly the same structure, though
under the hood QUIC is different because there are no
separate connection establishment and handshake primitives.
For this reason, we will not see a connect event, but we
will only see a "QUIC handshake event".
Having said that, let us now move on and see the code of
the simple program that shows this functionality.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measure/chapter05/main.go`.)
## main.go
The initial part of the program is pretty much the same as the one
used in previous chapters, so I will not add further comments.
```Go
package main
import (
"context"
"crypto/tls"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/netxlite"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
sni := flag.String("sni", "dns.google", "value for SNI extension")
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
```
### Handshaking with QUIC
The API signature is indeed the same as the previous chapter,
except that here we call the `QUICHandshake` function.
```Go
m := mx.QUICHandshake(ctx, *address, &tls.Config{
ServerName: *sni,
NextProtos: []string{"h3"},
RootCAs: netxlite.NewDefaultCertPool(),
})
```
The same remarks mentioned in the previous chapter regarding
the arguments for the TLS config also apply here. We need
to specify the SNI (`ServerName`), the ALPN (`NextProtos`),
and the CA pool we want to use. Here, again, we're using
the CA pool from cURL that we bundle with ooniprobe.
As we did in the previous chapters, here's the usual three
lines of code for printing the resulting measurement.
```
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
```
## Running the example program
As before, let us start off with a vanilla run:
```bash
go run -race ./internal/tutorial/measurex/chapter05
```
Produces this JSON:
```JavaScript
{
// In chapter02 these two fields were similar but
// the network was "tcp" as opposed to "quic"
"network": "quic",
"address": "8.8.4.4:443",
// This block contains I/O operations. Note that
// the protocol is "quic" and that the syscalls
// are "read_from" and "write_to" because QUIC does
// not bind/connect sockets. (The real syscalls
// are actually `recvfrom` and `sendto` but here
// we follow the Go convention of using read/write
// more frequently than send/recv.)
"read_write": [
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 1252,
"operation": "write_to",
"proto": "quic",
"t": 0.003903167,
"started": 0.0037395,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 1252,
"operation": "read_from",
"proto": "quic",
"t": 0.029389125,
"started": 0.002954792,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 1252,
"operation": "write_to",
"proto": "quic",
"t": 0.029757584,
"started": 0.02972325,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 1252,
"operation": "read_from",
"proto": "quic",
"t": 0.045039875,
"started": 0.029424792,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 1252,
"operation": "read_from",
"proto": "quic",
"t": 0.045055334,
"started": 0.045049625,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 1252,
"operation": "read_from",
"proto": "quic",
"t": 0.045073917,
"started": 0.045069667,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 1233,
"operation": "read_from",
"proto": "quic",
"t": 0.04508,
"started": 0.045075292,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 64,
"operation": "read_from",
"proto": "quic",
"t": 0.045088167,
"started": 0.045081167,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 44,
"operation": "write_to",
"proto": "quic",
"t": 0.045370417,
"started": 0.045338667,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 44,
"operation": "write_to",
"proto": "quic",
"t": 0.045392125,
"started": 0.045380959,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 83,
"operation": "write_to",
"proto": "quic",
"t": 0.047042542,
"started": 0.047001917,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 33,
"operation": "write_to",
"proto": "quic",
"t": 0.047060834,
"started": 0.047046875,
"oddity": ""
}
],
// This section describes the QUIC handshake and it has
// basically the same fields of the TLS handshake.
"quic_handshake": [
{
"cipher_suite": "TLS_CHACHA20_POLY1305_SHA256",
"failure": null,
"negotiated_proto": "h3",
"tls_version": "TLSv1.3",
"peer_certificates": [
{
"data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc",
"format": "base64"
},
{
"data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd",
"format": "base64"
},
{
"data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=",
"format": "base64"
}
],
"t": 0.047042459,
"address": "8.8.4.4:443",
"server_name": "dns.google",
"alpn": [
"h3"
],
"no_tls_verify": false,
"oddity": "",
"proto": "quic",
"started": 0.002154834
}
]
}
```
Here are some suggestions on other experiments to run:
1. obtain a timeout by connecting on a port that is not
actually listening for QUIC;
2. obtain a certificate validation error by forcing
a different SNI;
3. use a different ALPN (by changing the code), and see
how the error and the oddity are handled. Can we do
anything about this by changing `./internal/netxlite/errorx`
to better support for this specific error condition?
## Conclusion
We have seen how to perform QUIC handshake and
collect measurements.

282
internal/tutorial/measurex/chapter05/main.go Normal file
View File

@ -0,0 +1,282 @@
// -=-=- StartHere -=-=-
//
// # Chapter V: QUIC handshaking
//
// This chapter describes measuring QUIC handshakes. Conceptually,
// and code wise, this is very similar to the previous chapter.
// The API call, in fact, has exactly the same structure, though
// under the hood QUIC is different because there are no
// separate connection establishment and handshake primitives.
// For this reason, we will not see a connect event, but we
// will only see a "QUIC handshake event".
//
// Having said that, let us now move on and see the code of
// the simple program that shows this functionality.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measure/chapter05/main.go`.)
//
// ## main.go
//
// The initial part of the program is pretty much the same as the one
// used in previous chapters, so I will not add further comments.
//
// ```Go
package main
import (
"context"
"crypto/tls"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/netxlite"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func main() {
sni := flag.String("sni", "dns.google", "value for SNI extension")
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// ### Handshaking with QUIC
//
// The API signature is indeed the same as the previous chapter,
// except that here we call the `QUICHandshake` function.
//
// ```Go
m := mx.QUICHandshake(ctx, *address, &tls.Config{
ServerName: *sni,
NextProtos: []string{"h3"},
RootCAs: netxlite.NewDefaultCertPool(),
})
// ```
//
// The same remarks mentioned in the previous chapter regarding
// the arguments for the TLS config also apply here. We need
// to specify the SNI (`ServerName`), the ALPN (`NextProtos`),
// and the CA pool we want to use. Here, again, we're using
// the CA pool from cURL that we bundle with ooniprobe.
//
// As we did in the previous chapters, here's the usual three
// lines of code for printing the resulting measurement.
//
// ```
data, err := json.Marshal(m)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
// ```
//
// ## Running the example program
//
// As before, let us start off with a vanilla run:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter05
// ```
//
// Produces this JSON:
//
// ```JavaScript
// {
// // In chapter02 these two fields were similar but
// // the network was "tcp" as opposed to "quic"
// "network": "quic",
// "address": "8.8.4.4:443",
//
// // This block contains I/O operations. Note that
// // the protocol is "quic" and that the syscalls
// // are "read_from" and "write_to" because QUIC does
// // not bind/connect sockets. (The real syscalls
// // are actually `recvfrom` and `sendto` but here
// // we follow the Go convention of using read/write
// // more frequently than send/recv.)
// "read_write": [
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 1252,
// "operation": "write_to",
// "proto": "quic",
// "t": 0.003903167,
// "started": 0.0037395,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 1252,
// "operation": "read_from",
// "proto": "quic",
// "t": 0.029389125,
// "started": 0.002954792,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 1252,
// "operation": "write_to",
// "proto": "quic",
// "t": 0.029757584,
// "started": 0.02972325,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 1252,
// "operation": "read_from",
// "proto": "quic",
// "t": 0.045039875,
// "started": 0.029424792,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 1252,
// "operation": "read_from",
// "proto": "quic",
// "t": 0.045055334,
// "started": 0.045049625,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 1252,
// "operation": "read_from",
// "proto": "quic",
// "t": 0.045073917,
// "started": 0.045069667,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 1233,
// "operation": "read_from",
// "proto": "quic",
// "t": 0.04508,
// "started": 0.045075292,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 64,
// "operation": "read_from",
// "proto": "quic",
// "t": 0.045088167,
// "started": 0.045081167,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 44,
// "operation": "write_to",
// "proto": "quic",
// "t": 0.045370417,
// "started": 0.045338667,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 44,
// "operation": "write_to",
// "proto": "quic",
// "t": 0.045392125,
// "started": 0.045380959,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 83,
// "operation": "write_to",
// "proto": "quic",
// "t": 0.047042542,
// "started": 0.047001917,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 33,
// "operation": "write_to",
// "proto": "quic",
// "t": 0.047060834,
// "started": 0.047046875,
// "oddity": ""
// }
// ],
//
// // This section describes the QUIC handshake and it has
// // basically the same fields of the TLS handshake.
// "quic_handshake": [
// {
// "cipher_suite": "TLS_CHACHA20_POLY1305_SHA256",
// "failure": null,
// "negotiated_proto": "h3",
// "tls_version": "TLSv1.3",
// "peer_certificates": [
// {
// "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc",
// "format": "base64"
// },
// {
// "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd",
// "format": "base64"
// },
// {
// "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=",
// "format": "base64"
// }
// ],
// "t": 0.047042459,
// "address": "8.8.4.4:443",
// "server_name": "dns.google",
// "alpn": [
// "h3"
// ],
// "no_tls_verify": false,
// "oddity": "",
// "proto": "quic",
// "started": 0.002154834
// }
// ]
// }
// ```
//
// Here are some suggestions on other experiments to run:
//
// 1. obtain a timeout by connecting on a port that is not
// actually listening for QUIC;
//
// 2. obtain a certificate validation error by forcing
// a different SNI;
//
// 3. use a different ALPN (by changing the code), and see
// how the error and the oddity are handled. Can we do
// anything about this by changing `./internal/netxlite/errorx`
// to better support for this specific error condition?
//
// ## Conclusion
//
// We have seen how to perform QUIC handshake and
// collect measurements.
//
// -=-=- StopHere -=-=-

486
internal/tutorial/measurex/chapter06/README.md Normal file
View File

@ -0,0 +1,486 @@
# Chapter VI: Getting a webpage from an HTTP/HTTPS/HTTP3 endpoint.
This chapter describes measuring getting a webpage from an
HTTPS endpoint. We have seen how to TCP connect, we have
seen how to TLS handshake, now it's time to see how we can
combine these operations with fetching a webpage from a
given TCP endpoint speaking HTTP and TLS. (As well as to
provide you with information on how to otherwise fetch
from HTTP and HTTP/3 endpoints.)
The program we're going to write, `main.go`, will show a
high-level operation to perform this measurement in a
single API call. The code implementing this API call will
combine the operations we have seen in previous chapter
with the "give me the webpage" operation. We are still
quite far away from the ability of "measuring a URL" but
we are increasingly moving towards more complex operations.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter06/main.go`.)
## main.go
We have package declaration and imports as usual.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
```
We have factored the three lines to print a measurement
into the following utility function called `print`.
```Go
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
```
The initial part of the program is pretty much the same as the one
used in previous chapters, expect that we have a few more command line
flags now, so I will not add further comments.
```Go
func main() {
sni := flag.String("sni", "dns.google", "value for SNI extension")
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
```
### HTTPEndpoint: a description of what to measure
First of all, let us create a description of the endpoint
for `measurex`. Up to this point we have used endpoint
to describe a level-4-like address. Therefore, we have seen
TCP endpoints being used for the TCP connect and the TLS
handshake, and we have seen QUIC endpoints being used
for the QUIC handshake. Now, however, we are going
to need more information to characterize the endpoint.
```Go
epnt := &measurex.HTTPEndpoint{
Domain: *sni,
Network: "tcp",
Address: *address,
SNI: *sni,
ALPN: []string{"h2", "http/1.1"},
URL: &url.URL{
Scheme: "https",
Host: *sni,
Path: "/",
},
Header: measurex.NewHTTPRequestHeaderForMeasuring(),
}
```
In fact, in the above definition we recognize fields
we have already discussed, such as:
- `Network`, describing whether to use "tcp" or "quic";
- `Address`, which is the endpoint address.
But we also need to combine into this view of the
endpoint additional fields for TLS:
- `SNI`, to set the SNI;
- `ALPN`, to set the ALPN;
But then we also need to specify:
- the URL to use;
- the headers to use (for which we're using a handy
factory for creating reasonable defaults for measuring).
This API is not the highest level API with which to do
the job, but it's still handy to introduce the
`measurex.HTTPEndpoint` data structure since it's
used by higher level APIs.
(You may also be wondering about the CA pool. It turns
out that for APIs such as this one and for higher
level APIs, the default is to always use the bundled
Mozilla CA pool, because this is what we use in
most cases for performing measurements.)
### HTTPEndpointGetWithoutCookies
When used with an HTTP URL, the `HTTPEndpointGetWithoutCookies`
method combines two operations:
- TCP connect
- HTTP GET
When the URL is HTTPS, we do:
- TCP connect
- TLS handshake
- HTTP GET (or HTTP/2 GET depending on the ALPN)
When the `HTTPEndpoint.Network` field value
is QUIC, instead we do:
- QUIC handshake
- HTTP/3 GET
```Go
m := mx.HTTPEndpointGetWithoutCookies(ctx, epnt)
```
The arguments for `HTTPEndpointGetWithDBWithoutCookies` are:
- the context for deadline/timeout
- the HTTPEndpoint descriptor
The result is an `HTTPEndpointMeasurement` which
you can inspect with
```
go doc ./internal/measurex.HTTPEndpointMeasurement
```
### Printing the measurement
Let us now print the resulting measurement.
```Go
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter06
```
This is the JSON output. Let us comment it in detail:
```Javascript
{
// The returned type is called HTTPEndpointMeasurement
// and you see that here on top we indeed have the
// information on the endpoint and the URL.
"url": "https://dns.google/",
"network": "tcp",
"address": "8.8.4.4:443",
// Internally, HTTPEndpointGetWithoutCookies calls
// TCPConnect and here we see the corresponding event
"connect": [
{
"address": "8.8.4.4:443",
"failure": null,
"operation": "connect",
"proto": "tcp",
"t": 0.02422375,
"started": 0.002269291,
"oddity": ""
}
],
// These are the I/O operations we have already seen
// in previous chapters
"read_write": [
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 280,
"operation": "write",
"proto": "tcp",
"t": 0.024931791,
"started": 0.024910416,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 517,
"operation": "read",
"proto": "tcp",
"t": 0.063629791,
"started": 0.024935666,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 4301,
"operation": "read",
"proto": "tcp",
"t": 0.064183,
"started": 0.064144208,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 64,
"operation": "write",
"proto": "tcp",
"t": 0.065464041,
"started": 0.065441333,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 86,
"operation": "write",
"proto": "tcp",
"t": 0.067256083,
"started": 0.067224375,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 201,
"operation": "write",
"proto": "tcp",
"t": 0.067674416,
"started": 0.067652375,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 93,
"operation": "read",
"proto": "tcp",
"t": 0.086618708,
"started": 0.067599208,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 31,
"operation": "write",
"proto": "tcp",
"t": 0.086703625,
"started": 0.0866745,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 2028,
"operation": "read",
"proto": "tcp",
"t": 0.337785916,
"started": 0.086717333,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 39,
"operation": "write",
"proto": "tcp",
"t": 0.338514916,
"started": 0.338485375,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": null,
"num_bytes": 24,
"operation": "write",
"proto": "tcp",
"t": 0.338800833,
"started": 0.338788625,
"oddity": ""
},
{
"address": "8.8.4.4:443",
"failure": "connection_already_closed",
"operation": "read",
"proto": "tcp",
"t": 0.338888041,
"started": 0.338523291,
"oddity": ""
}
],
// Internally, HTTPEndpointGetWithoutCookies calls TLSConnectAndHandshake,
// and here's the resulting handshake event. Of course, if we
// specified a QUIC endpoint we would instead see here a
// QUIC handshake event. And, we would not see any handshake
// if the URL was instead an HTTP URL.
"tls_handshake": [
{
"cipher_suite": "TLS_AES_128_GCM_SHA256",
"failure": null,
"negotiated_proto": "h2",
"tls_version": "TLSv1.3",
"peer_certificates": [
{
"data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc",
"format": "base64"
},
{
"data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd",
"format": "base64"
},
{
"data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=",
"format": "base64"
}
],
"t": 0.065514708,
"address": "8.8.4.4:443",
"server_name": "dns.google",
"alpn": [
"h2",
"http/1.1"
],
"no_tls_verify": false,
"oddity": "",
"proto": "tcp",
"started": 0.024404083
}
],
// Finally here we see information about the round trip, which
// is formatted according the df-001-httpt data format:
"http_round_trip": [
{
// This field indicates whether there was an error during
// the HTTP round trip:
"failure": null,
// This field contains the request method, URL, and HTTP headers
"request": {
"method": "GET",
"url": "https://dns.google/",
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"accept-language": "en-US;q=0.8,en;q=0.5",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
}
},
// This field contains the response status code, body,
// and headers.
"response": {
"code": 200,
"headers": {
"accept-ranges": "none",
"alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
"cache-control": "private",
"content-security-policy": "object-src 'none';base-uri 'self';script-src 'nonce-bSLcJjaotppZl3Y2moIaxg==' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/honest_dns/1_0;frame-ancestors 'none'",
"content-type": "text/html; charset=UTF-8",
"date": "Fri, 24 Sep 2021 08:51:01 GMT",
"server": "scaffolding on HTTPServer2",
"strict-transport-security": "max-age=31536000; includeSubDomains; preload",
"vary": "Accept-Encoding",
"x-content-type-options": "nosniff",
"x-frame-options": "SAMEORIGIN",
"x-xss-protection": "0"
},
// The body in particular is a snapshot of the response
// body: we don't want to read and submit to the OONI
// collector large bodies.
"body": {
"data": "PCFET0NUWVBFIGh0bWw+CjxodG1sIGxhbmc9ImVuIj4gPGhlYWQ+IDx0aXRsZT5Hb29nbGUgUHVibGljIEROUzwvdGl0bGU+ICA8bWV0YSBjaGFyc2V0PSJVVEYtOCI+IDxsaW5rIGhyZWY9Ii9zdGF0aWMvOTNkZDU5NTQvZmF2aWNvbi5wbmciIHJlbD0ic2hvcnRjdXQgaWNvbiIgdHlwZT0iaW1hZ2UvcG5nIj4gPGxpbmsgaHJlZj0iL3N0YXRpYy84MzZhZWJjNi9tYXR0ZXIubWluLmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPGxpbmsgaHJlZj0iL3N0YXRpYy9iODUzNmMzNy9zaGFyZWQuY3NzIiByZWw9InN0eWxlc2hlZXQiPiA8bWV0YSBuYW1lPSJ2aWV3cG9ydCIgY29udGVudD0id2lkdGg9ZGV2aWNlLXdpZHRoLCBpbml0aWFsLXNjYWxlPTEiPiAgPGxpbmsgaHJlZj0iL3N0YXRpYy9kMDVjZDZiYS9yb290LmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPC9oZWFkPiA8Ym9keT4gPHNwYW4gY2xhc3M9ImZpbGxlciB0b3AiPjwvc3Bhbj4gICA8ZGl2IGNsYXNzPSJsb2dvIiB0aXRsZT0iR29vZ2xlIFB1YmxpYyBETlMiPiA8ZGl2IGNsYXNzPSJsb2dvLXRleHQiPjxzcGFuPlB1YmxpYyBETlM8L3NwYW4+PC9kaXY+IDwvZGl2PiAgPGZvcm0gYWN0aW9uPSIvcXVlcnkiIG1ldGhvZD0iR0VUIj4gIDxkaXYgY2xhc3M9InJvdyI+IDxsYWJlbCBjbGFzcz0ibWF0dGVyLXRleHRmaWVsZC1vdXRsaW5lZCI+IDxpbnB1dCB0eXBlPSJ0ZXh0IiBuYW1lPSJuYW1lIiBwbGFjZWhvbGRlcj0iJm5ic3A7Ij4gPHNwYW4+RE5TIE5hbWU8L3NwYW4+IDxwIGNsYXNzPSJoZWxwIj4gRW50ZXIgYSBkb21haW4gKGxpa2UgZXhhbXBsZS5jb20pIG9yIElQIGFkZHJlc3MgKGxpa2UgOC44LjguOCBvciAyMDAxOjQ4NjA6NDg2MDo6ODg0NCkgaGVyZS4gPC9wPiA8L2xhYmVsPiA8YnV0dG9uIGNsYXNzPSJtYXR0ZXItYnV0dG9uLWNvbnRhaW5lZCBtYXR0ZXItcHJpbWFyeSIgdHlwZT0ic3VibWl0Ij5SZXNvbHZlPC9idXR0b24+IDwvZGl2PiA8L2Zvcm0+ICA8c3BhbiBjbGFzcz0iZmlsbGVyIGJvdHRvbSI+PC9zcGFuPiA8Zm9vdGVyIGNsYXNzPSJyb3ciPiA8YSBocmVmPSJodHRwczovL2RldmVsb3BlcnMuZ29vZ2xlLmNvbS9zcGVlZC9wdWJsaWMtZG5zIj5IZWxwPC9hPiA8YSBocmVmPSIvY2FjaGUiPkNhY2hlIEZsdXNoPC9hPiA8c3BhbiBjbGFzcz0iZmlsbGVyIj48L3NwYW4+IDxhIGhyZWY9Imh0dHBzOi8vZGV2ZWxvcGVycy5nb29nbGUuY29tL3NwZWVkL3B1YmxpYy1kbnMvZG9jcy91c2luZyI+IEdldCBTdGFydGVkIHdpdGggR29vZ2xlIFB1YmxpYyBETlMgPC9hPiA8L2Zvb3Rlcj4gICA8c2NyaXB0IG5vbmNlPSJiU0xjSmphb3RwcFpsM1kybW9JYXhnPT0iPmRvY3VtZW50LmZvcm1zWzBdLm5hbWUuZm9jdXMoKTs8L3NjcmlwdD4gPC9ib2R5PiA8L2h0bWw+",
"format": "base64"
},
// This field tells us whether the size of the read
// snapshot was smaller than the snapshot size. If
// not, then the body has been truncated.
"body_is_truncated": false,
// These extra fields are not part of the spec and
// hence we prefix them with `x_`. They tell us
// the length of the body and whether the content
// of the body is valid UTF8.
"x_body_length": 1383,
"x_body_is_utf8": true
},
// The t field is the moment where we finished the
// round trip and saved the event. The started field
// is instead when we started the round trip.
// You may notice that the start of the round trip
// if after the `t` of the handshake. This tells us
// that the code first connects, then handshakes, and
// finally creates HTTP code for performing the
// round trip.
"t": 0.338674625,
"started": 0.065926625,
// As usual we also compute an oddity value related
// in this case to the HTTP round trip.
"oddity": ""
}
]
}
```
Here are some suggestions for follow up measurements:
1. provoke a connect error by using:
```
go run -race ./internal/tutorial/measurex/chapter06 -address 127.0.0.1:1
```
2. provoke a TLS handshake error by using:
```
go run -race ./internal/tutorial/measurex/chapter06 -sni example.com
```
3. provoke an HTTP round trip error by using:
```
go run -race ./internal/tutorial/measurex/chapter06 -address 8.8.8.8:853
```
4. modify the code to fetch an HTTP endpoint instead (hint: you
need to change the HTTPEndpoint's URL scheme);
5. modify the code to use QUIC and HTTP/3 instead (hint: you need to
change the HTTPEndpoint's network and... is this enough?).
## Conclusion
We have seen how to measure the flow of fetching a
specific webpage from an HTTPEndpoint.

488
internal/tutorial/measurex/chapter06/main.go Normal file
View File

@ -0,0 +1,488 @@
// -=-=- StartHere -=-=-
//
// # Chapter VI: Getting a webpage from an HTTP/HTTPS/HTTP3 endpoint.
//
// This chapter describes measuring getting a webpage from an
// HTTPS endpoint. We have seen how to TCP connect, we have
// seen how to TLS handshake, now it's time to see how we can
// combine these operations with fetching a webpage from a
// given TCP endpoint speaking HTTP and TLS. (As well as to
// provide you with information on how to otherwise fetch
// from HTTP and HTTP/3 endpoints.)
//
// The program we're going to write, `main.go`, will show a
// high-level operation to perform this measurement in a
// single API call. The code implementing this API call will
// combine the operations we have seen in previous chapter
// with the "give me the webpage" operation. We are still
// quite far away from the ability of "measuring a URL" but
// we are increasingly moving towards more complex operations.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter06/main.go`.)
//
// ## main.go
//
// We have package declaration and imports as usual.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
// ```
//
// We have factored the three lines to print a measurement
// into the following utility function called `print`.
//
// ```Go
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
// ```
//
// The initial part of the program is pretty much the same as the one
// used in previous chapters, expect that we have a few more command line
// flags now, so I will not add further comments.
//
// ```Go
func main() {
sni := flag.String("sni", "dns.google", "value for SNI extension")
address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// ### HTTPEndpoint: a description of what to measure
//
// First of all, let us create a description of the endpoint
// for `measurex`. Up to this point we have used endpoint
// to describe a level-4-like address. Therefore, we have seen
// TCP endpoints being used for the TCP connect and the TLS
// handshake, and we have seen QUIC endpoints being used
// for the QUIC handshake. Now, however, we are going
// to need more information to characterize the endpoint.
//
// ```Go
epnt := &measurex.HTTPEndpoint{
Domain: *sni,
Network: "tcp",
Address: *address,
SNI: *sni,
ALPN: []string{"h2", "http/1.1"},
URL: &url.URL{
Scheme: "https",
Host: *sni,
Path: "/",
},
Header: measurex.NewHTTPRequestHeaderForMeasuring(),
}
// ```
//
// In fact, in the above definition we recognize fields
// we have already discussed, such as:
//
// - `Network`, describing whether to use "tcp" or "quic";
//
// - `Address`, which is the endpoint address.
//
// But we also need to combine into this view of the
// endpoint additional fields for TLS:
//
// - `SNI`, to set the SNI;
//
// - `ALPN`, to set the ALPN;
//
// But then we also need to specify:
//
// - the URL to use;
//
// - the headers to use (for which we're using a handy
// factory for creating reasonable defaults for measuring).
//
// This API is not the highest level API with which to do
// the job, but it's still handy to introduce the
// `measurex.HTTPEndpoint` data structure since it's
// used by higher level APIs.
//
// (You may also be wondering about the CA pool. It turns
// out that for APIs such as this one and for higher
// level APIs, the default is to always use the bundled
// Mozilla CA pool, because this is what we use in
// most cases for performing measurements.)
//
// ### HTTPEndpointGetWithoutCookies
//
// When used with an HTTP URL, the `HTTPEndpointGetWithoutCookies`
// method combines two operations:
//
// - TCP connect
//
// - HTTP GET
//
// When the URL is HTTPS, we do:
//
// - TCP connect
//
// - TLS handshake
//
// - HTTP GET (or HTTP/2 GET depending on the ALPN)
//
// When the `HTTPEndpoint.Network` field value
// is QUIC, instead we do:
//
// - QUIC handshake
//
// - HTTP/3 GET
//
// ```Go
m := mx.HTTPEndpointGetWithoutCookies(ctx, epnt)
// ```
//
// The arguments for `HTTPEndpointGetWithDBWithoutCookies` are:
//
// - the context for deadline/timeout
//
// - the HTTPEndpoint descriptor
//
// The result is an `HTTPEndpointMeasurement` which
// you can inspect with
//
// ```
// go doc ./internal/measurex.HTTPEndpointMeasurement
// ```
//
// ### Printing the measurement
//
// Let us now print the resulting measurement.
//
// ```Go
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter06
// ```
//
// This is the JSON output. Let us comment it in detail:
//
// ```Javascript
// {
// // The returned type is called HTTPEndpointMeasurement
// // and you see that here on top we indeed have the
// // information on the endpoint and the URL.
// "url": "https://dns.google/",
// "network": "tcp",
// "address": "8.8.4.4:443",
//
// // Internally, HTTPEndpointGetWithoutCookies calls
// // TCPConnect and here we see the corresponding event
// "connect": [
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "operation": "connect",
// "proto": "tcp",
// "t": 0.02422375,
// "started": 0.002269291,
// "oddity": ""
// }
// ],
//
// // These are the I/O operations we have already seen
// // in previous chapters
// "read_write": [
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 280,
// "operation": "write",
// "proto": "tcp",
// "t": 0.024931791,
// "started": 0.024910416,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 517,
// "operation": "read",
// "proto": "tcp",
// "t": 0.063629791,
// "started": 0.024935666,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 4301,
// "operation": "read",
// "proto": "tcp",
// "t": 0.064183,
// "started": 0.064144208,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 64,
// "operation": "write",
// "proto": "tcp",
// "t": 0.065464041,
// "started": 0.065441333,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 86,
// "operation": "write",
// "proto": "tcp",
// "t": 0.067256083,
// "started": 0.067224375,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 201,
// "operation": "write",
// "proto": "tcp",
// "t": 0.067674416,
// "started": 0.067652375,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 93,
// "operation": "read",
// "proto": "tcp",
// "t": 0.086618708,
// "started": 0.067599208,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 31,
// "operation": "write",
// "proto": "tcp",
// "t": 0.086703625,
// "started": 0.0866745,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 2028,
// "operation": "read",
// "proto": "tcp",
// "t": 0.337785916,
// "started": 0.086717333,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 39,
// "operation": "write",
// "proto": "tcp",
// "t": 0.338514916,
// "started": 0.338485375,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": null,
// "num_bytes": 24,
// "operation": "write",
// "proto": "tcp",
// "t": 0.338800833,
// "started": 0.338788625,
// "oddity": ""
// },
// {
// "address": "8.8.4.4:443",
// "failure": "connection_already_closed",
// "operation": "read",
// "proto": "tcp",
// "t": 0.338888041,
// "started": 0.338523291,
// "oddity": ""
// }
// ],
//
// // Internally, HTTPEndpointGetWithoutCookies calls TLSConnectAndHandshake,
// // and here's the resulting handshake event. Of course, if we
// // specified a QUIC endpoint we would instead see here a
// // QUIC handshake event. And, we would not see any handshake
// // if the URL was instead an HTTP URL.
// "tls_handshake": [
// {
// "cipher_suite": "TLS_AES_128_GCM_SHA256",
// "failure": null,
// "negotiated_proto": "h2",
// "tls_version": "TLSv1.3",
// "peer_certificates": [
// {
// "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc",
// "format": "base64"
// },
// {
// "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd",
// "format": "base64"
// },
// {
// "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=",
// "format": "base64"
// }
// ],
// "t": 0.065514708,
// "address": "8.8.4.4:443",
// "server_name": "dns.google",
// "alpn": [
// "h2",
// "http/1.1"
// ],
// "no_tls_verify": false,
// "oddity": "",
// "proto": "tcp",
// "started": 0.024404083
// }
// ],
//
// // Finally here we see information about the round trip, which
// // is formatted according the df-001-httpt data format:
// "http_round_trip": [
// {
//
// // This field indicates whether there was an error during
// // the HTTP round trip:
// "failure": null,
//
// // This field contains the request method, URL, and HTTP headers
// "request": {
// "method": "GET",
// "url": "https://dns.google/",
// "headers": {
// "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
// "accept-language": "en-US;q=0.8,en;q=0.5",
// "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
// }
// },
//
// // This field contains the response status code, body,
// // and headers.
// "response": {
// "code": 200,
// "headers": {
// "accept-ranges": "none",
// "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
// "cache-control": "private",
// "content-security-policy": "object-src 'none';base-uri 'self';script-src 'nonce-bSLcJjaotppZl3Y2moIaxg==' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/honest_dns/1_0;frame-ancestors 'none'",
// "content-type": "text/html; charset=UTF-8",
// "date": "Fri, 24 Sep 2021 08:51:01 GMT",
// "server": "scaffolding on HTTPServer2",
// "strict-transport-security": "max-age=31536000; includeSubDomains; preload",
// "vary": "Accept-Encoding",
// "x-content-type-options": "nosniff",
// "x-frame-options": "SAMEORIGIN",
// "x-xss-protection": "0"
// },
//
// // The body in particular is a snapshot of the response
// // body: we don't want to read and submit to the OONI
// // collector large bodies.
// "body": {
// "data": "PCFET0NUWVBFIGh0bWw+CjxodG1sIGxhbmc9ImVuIj4gPGhlYWQ+IDx0aXRsZT5Hb29nbGUgUHVibGljIEROUzwvdGl0bGU+ICA8bWV0YSBjaGFyc2V0PSJVVEYtOCI+IDxsaW5rIGhyZWY9Ii9zdGF0aWMvOTNkZDU5NTQvZmF2aWNvbi5wbmciIHJlbD0ic2hvcnRjdXQgaWNvbiIgdHlwZT0iaW1hZ2UvcG5nIj4gPGxpbmsgaHJlZj0iL3N0YXRpYy84MzZhZWJjNi9tYXR0ZXIubWluLmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPGxpbmsgaHJlZj0iL3N0YXRpYy9iODUzNmMzNy9zaGFyZWQuY3NzIiByZWw9InN0eWxlc2hlZXQiPiA8bWV0YSBuYW1lPSJ2aWV3cG9ydCIgY29udGVudD0id2lkdGg9ZGV2aWNlLXdpZHRoLCBpbml0aWFsLXNjYWxlPTEiPiAgPGxpbmsgaHJlZj0iL3N0YXRpYy9kMDVjZDZiYS9yb290LmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPC9oZWFkPiA8Ym9keT4gPHNwYW4gY2xhc3M9ImZpbGxlciB0b3AiPjwvc3Bhbj4gICA8ZGl2IGNsYXNzPSJsb2dvIiB0aXRsZT0iR29vZ2xlIFB1YmxpYyBETlMiPiA8ZGl2IGNsYXNzPSJsb2dvLXRleHQiPjxzcGFuPlB1YmxpYyBETlM8L3NwYW4+PC9kaXY+IDwvZGl2PiAgPGZvcm0gYWN0aW9uPSIvcXVlcnkiIG1ldGhvZD0iR0VUIj4gIDxkaXYgY2xhc3M9InJvdyI+IDxsYWJlbCBjbGFzcz0ibWF0dGVyLXRleHRmaWVsZC1vdXRsaW5lZCI+IDxpbnB1dCB0eXBlPSJ0ZXh0IiBuYW1lPSJuYW1lIiBwbGFjZWhvbGRlcj0iJm5ic3A7Ij4gPHNwYW4+RE5TIE5hbWU8L3NwYW4+IDxwIGNsYXNzPSJoZWxwIj4gRW50ZXIgYSBkb21haW4gKGxpa2UgZXhhbXBsZS5jb20pIG9yIElQIGFkZHJlc3MgKGxpa2UgOC44LjguOCBvciAyMDAxOjQ4NjA6NDg2MDo6ODg0NCkgaGVyZS4gPC9wPiA8L2xhYmVsPiA8YnV0dG9uIGNsYXNzPSJtYXR0ZXItYnV0dG9uLWNvbnRhaW5lZCBtYXR0ZXItcHJpbWFyeSIgdHlwZT0ic3VibWl0Ij5SZXNvbHZlPC9idXR0b24+IDwvZGl2PiA8L2Zvcm0+ICA8c3BhbiBjbGFzcz0iZmlsbGVyIGJvdHRvbSI+PC9zcGFuPiA8Zm9vdGVyIGNsYXNzPSJyb3ciPiA8YSBocmVmPSJodHRwczovL2RldmVsb3BlcnMuZ29vZ2xlLmNvbS9zcGVlZC9wdWJsaWMtZG5zIj5IZWxwPC9hPiA8YSBocmVmPSIvY2FjaGUiPkNhY2hlIEZsdXNoPC9hPiA8c3BhbiBjbGFzcz0iZmlsbGVyIj48L3NwYW4+IDxhIGhyZWY9Imh0dHBzOi8vZGV2ZWxvcGVycy5nb29nbGUuY29tL3NwZWVkL3B1YmxpYy1kbnMvZG9jcy91c2luZyI+IEdldCBTdGFydGVkIHdpdGggR29vZ2xlIFB1YmxpYyBETlMgPC9hPiA8L2Zvb3Rlcj4gICA8c2NyaXB0IG5vbmNlPSJiU0xjSmphb3RwcFpsM1kybW9JYXhnPT0iPmRvY3VtZW50LmZvcm1zWzBdLm5hbWUuZm9jdXMoKTs8L3NjcmlwdD4gPC9ib2R5PiA8L2h0bWw+",
// "format": "base64"
// },
//
// // This field tells us whether the size of the read
// // snapshot was smaller than the snapshot size. If
// // not, then the body has been truncated.
// "body_is_truncated": false,
//
// // These extra fields are not part of the spec and
// // hence we prefix them with `x_`. They tell us
// // the length of the body and whether the content
// // of the body is valid UTF8.
// "x_body_length": 1383,
// "x_body_is_utf8": true
// },
//
// // The t field is the moment where we finished the
// // round trip and saved the event. The started field
// // is instead when we started the round trip.
//
// // You may notice that the start of the round trip
// // if after the `t` of the handshake. This tells us
// // that the code first connects, then handshakes, and
// // finally creates HTTP code for performing the
// // round trip.
// "t": 0.338674625,
// "started": 0.065926625,
//
// // As usual we also compute an oddity value related
// // in this case to the HTTP round trip.
// "oddity": ""
// }
// ]
// }
// ```
//
// Here are some suggestions for follow up measurements:
//
// 1. provoke a connect error by using:
//
// ```
// go run -race ./internal/tutorial/measurex/chapter06 -address 127.0.0.1:1
// ```
//
// 2. provoke a TLS handshake error by using:
//
// ```
// go run -race ./internal/tutorial/measurex/chapter06 -sni example.com
// ```
//
// 3. provoke an HTTP round trip error by using:
//
// ```
// go run -race ./internal/tutorial/measurex/chapter06 -address 8.8.8.8:853
// ```
//
// 4. modify the code to fetch an HTTP endpoint instead (hint: you
// need to change the HTTPEndpoint's URL scheme);
//
// 5. modify the code to use QUIC and HTTP/3 instead (hint: you need to
// change the HTTPEndpoint's network and... is this enough?).
//
// ## Conclusion
//
// We have seen how to measure the flow of fetching a
// specific webpage from an HTTPEndpoint.
//
// -=-=- StopHere -=-=-

142
internal/tutorial/measurex/chapter07/README.md Normal file
View File

@ -0,0 +1,142 @@
# Chapter VII: Measuring all the HTTPEndpoints for a domain
We are now going to combine DNS resolutions with getting
HTTPEndpoints. Conceptually, the DNS resolution yields
us a list of IP addresses. For each address, we build the
HTTPEndpoint and fetch it like we did in chapter06.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter07/main.go`.)
## main.go
We have package declaration and imports as usual.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
```
Here we define an helper type for containing the DNS
measurement and the subsequent endpoints measurements.
```Go
type measurement struct {
DNS *measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
```
The rest of the program is quite similar to what we had before.
```Go
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://google.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
```
This is where the main.go file starts to diverge. We create an
instance of our measurement type to hold the results.
```Go
m := &measurement{}
```
Then we perform a DNS lookup using UDP like we saw in chapter03.
```Go
m.DNS = mx.LookupHostUDP(ctx, parsed.Hostname(), *address)
```
Like we did in the previous chapter, we create suitable HTTP
headers for performing an HTTP measurement.
```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
```
The following is an entirely new function we're learning
about just now. `AllHTTPEndpointsForURL` is a free function
in `measurex` that given:
- an already parsed HTTP/HTTPS URL
- headers we want to use
- the result of one or more DNS queries
builds us a list of HTTPEndpoint data structures.
```Go
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
```
This function may fail if, for example, the URL is not HTTP/HTTPS. We
handle the error panicking, because this is an example program.
We are almost done now: we loop over all the endpoints and apply the
`HTTPEndpointGetWithoutCookies` method we have seen in chapter06.
```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
```
Finally, we print the results.
```Go
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter07
```
Please, check the JSON output. Do you recognize the fields
we have described in previous chapters?
Can you provoke common errors such as DNS resolution
errors, TCP connect errors, TLS handshake errors, and
HTTP round trip errors? How does the JSON change?
## Conclusion
We have seen how to combine DNS resolutions (chapter01 and
chapter03) with HTTPEndpoint GET (chapter06) to measure
all the HTTP endpoints for a given domain.

144
internal/tutorial/measurex/chapter07/main.go Normal file
View File

@ -0,0 +1,144 @@
// -=-=- StartHere -=-=-
//
// # Chapter VII: Measuring all the HTTPEndpoints for a domain
//
// We are now going to combine DNS resolutions with getting
// HTTPEndpoints. Conceptually, the DNS resolution yields
// us a list of IP addresses. For each address, we build the
// HTTPEndpoint and fetch it like we did in chapter06.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter07/main.go`.)
//
// ## main.go
//
// We have package declaration and imports as usual.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
// ```
//
// Here we define an helper type for containing the DNS
// measurement and the subsequent endpoints measurements.
//
// ```Go
type measurement struct {
DNS *measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
// ```
//
// The rest of the program is quite similar to what we had before.
//
// ```Go
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://google.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// This is where the main.go file starts to diverge. We create an
// instance of our measurement type to hold the results.
//
// ```Go
m := &measurement{}
// ```
//
// Then we perform a DNS lookup using UDP like we saw in chapter03.
//
// ```Go
m.DNS = mx.LookupHostUDP(ctx, parsed.Hostname(), *address)
// ```
//
// Like we did in the previous chapter, we create suitable HTTP
// headers for performing an HTTP measurement.
//
// ```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
// ```
//
// The following is an entirely new function we're learning
// about just now. `AllHTTPEndpointsForURL` is a free function
// in `measurex` that given:
//
// - an already parsed HTTP/HTTPS URL
//
// - headers we want to use
//
// - the result of one or more DNS queries
//
// builds us a list of HTTPEndpoint data structures.
//
// ```Go
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
// ```
//
// This function may fail if, for example, the URL is not HTTP/HTTPS. We
// handle the error panicking, because this is an example program.
//
// We are almost done now: we loop over all the endpoints and apply the
// `HTTPEndpointGetWithoutCookies` method we have seen in chapter06.
//
// ```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
// ```
//
// Finally, we print the results.
//
// ```Go
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter07
// ```
//
// Please, check the JSON output. Do you recognize the fields
// we have described in previous chapters?
//
// Can you provoke common errors such as DNS resolution
// errors, TCP connect errors, TLS handshake errors, and
// HTTP round trip errors? How does the JSON change?
//
// ## Conclusion
//
// We have seen how to combine DNS resolutions (chapter01 and
// chapter03) with HTTPEndpoint GET (chapter06) to measure
// all the HTTP endpoints for a given domain.
//
// -=-=- StopHere -=-=-

132
internal/tutorial/measurex/chapter08/README.md Normal file
View File

@ -0,0 +1,132 @@
# Chapter VII: HTTPSSvc DNS queries
The program we see here is _really_ similar to the one we
discussed in the previous chapter. The main difference
is the following: now we also issue HTTPSSvc DNS queries
to discover HTTP/3 endpoints. (Because HTTPSSvc is
still a draft and is mostly implemented by Cloudflare
at this point, we are going to use as the example
input URL a Cloudflare URL.)
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter08/main.go`.)
## main.go
The beginning of the program is pretty much the same. We
have just amended our `measurement` type to contain multiple
`DNSMeasurement` results.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
```
### Call LookupHTTPSSvc
Here we perform the `LookupHostUDP` we performed in the
previous chapter and then we call `LookupHTTPSvcUDP`.
```Go
m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address))
m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address))
```
The `LookupHTTPSSvcUDP` function has the same signature
of `LookupHostUDP` _but_ it behaves differently. Rather than
querying for `A` and `AAAA`, it performs an `HTTPS` DNS
lookup. This query returns:
1. a list of ALPNs for the domain;
2. a list of IPv4 addresses;
3. a list of IPv6 addresses.
### Build an []HTTPEndpoint and run serial measurements
Here we call `AllHTTPEndpointsForURL` like we did in the
previous chapter. However, note that we pass to it the
whole content of `m.DNS`, which now contains not only the
A/AAAA lookups results but also the HTTPS lookup results.
The `AllHTTPEndpointsForURL` function will recognize that
we also have HTTPS lookups and, if the "h3" ALPN is
present, will _also_ build HTTP/3 endpoints using "quic"
as the `HTTPEndpoint.Network`.
```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
```
This is it. The rest of the program is exactly the same.
```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter08
```
Please, check the JSON output. Do you recognize the fields
we have described in previous chapters? You should see
that, compared to previous chapters, now we're also testing
QUIC/HTTP3 endpoints.
Can you provoke common errors such as DNS resolution
errors, TCP connect errors, TLS handshake errors, and
HTTP round trip errors? What is a good way to cause
timeout and SNI mismatch errors for QUIC?
## Conclusion
We have seen how to extend fetching all the HTTPS
endpoints to include the QUIC/HTTP3 endpoints discovered
using HTTPSSvc.

134
internal/tutorial/measurex/chapter08/main.go Normal file
View File

@ -0,0 +1,134 @@
// -=-=- StartHere -=-=-
//
// # Chapter VII: HTTPSSvc DNS queries
//
// The program we see here is _really_ similar to the one we
// discussed in the previous chapter. The main difference
// is the following: now we also issue HTTPSSvc DNS queries
// to discover HTTP/3 endpoints. (Because HTTPSSvc is
// still a draft and is mostly implemented by Cloudflare
// at this point, we are going to use as the example
// input URL a Cloudflare URL.)
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter08/main.go`.)
//
// ## main.go
//
// The beginning of the program is pretty much the same. We
// have just amended our `measurement` type to contain multiple
// `DNSMeasurement` results.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
// ```
// ### Call LookupHTTPSSvc
//
// Here we perform the `LookupHostUDP` we performed in the
// previous chapter and then we call `LookupHTTPSvcUDP`.
//
// ```Go
m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address))
m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address))
// ```
//
// The `LookupHTTPSSvcUDP` function has the same signature
// of `LookupHostUDP` _but_ it behaves differently. Rather than
// querying for `A` and `AAAA`, it performs an `HTTPS` DNS
// lookup. This query returns:
//
// 1. a list of ALPNs for the domain;
//
// 2. a list of IPv4 addresses;
//
// 3. a list of IPv6 addresses.
//
// ### Build an []HTTPEndpoint and run serial measurements
//
// Here we call `AllHTTPEndpointsForURL` like we did in the
// previous chapter. However, note that we pass to it the
// whole content of `m.DNS`, which now contains not only the
// A/AAAA lookups results but also the HTTPS lookup results.
//
// The `AllHTTPEndpointsForURL` function will recognize that
// we also have HTTPS lookups and, if the "h3" ALPN is
// present, will _also_ build HTTP/3 endpoints using "quic"
// as the `HTTPEndpoint.Network`.
//
// ```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
// ```
//
// This is it. The rest of the program is exactly the same.
//
// ```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter08
// ```
//
// Please, check the JSON output. Do you recognize the fields
// we have described in previous chapters? You should see
// that, compared to previous chapters, now we're also testing
// QUIC/HTTP3 endpoints.
//
// Can you provoke common errors such as DNS resolution
// errors, TCP connect errors, TLS handshake errors, and
// HTTP round trip errors? What is a good way to cause
// timeout and SNI mismatch errors for QUIC?
//
// ## Conclusion
//
// We have seen how to extend fetching all the HTTPS
// endpoints to include the QUIC/HTTP3 endpoints discovered
// using HTTPSSvc.
//
// -=-=- StopHere -=-=-

124
internal/tutorial/measurex/chapter09/README.md Normal file
View File

@ -0,0 +1,124 @@
# Chapter IX: Parallel HTTPEndpoint measurements
The program we see here is _really_ similar to the one we
discussed in the previous chapter. The main difference
is the following: rather than looping through the list of
HTTPEndpoint, we call a function that runs through the
list of endpoints using a small pool of background workers.
There is a trade off between quick measurements and
false positives. A timeout is one of the most common
ways of censoring HTTPS and HTTP3 endpoints. So, if
we run measurements sequentially, a whole scan could
in principle take a long time. On the other hand,
if we run too many parallel measurements, we may cause
our own congestion and maybe some measurements will
fail because of that. Our solution to this problem is
to have low parallelism: at the moment of writing
this note, we have three workers. If you submit
more than three HTTPEndpoint at a a time, we will
service the first three immediately and all the
other endpoints will be queued for later measurement.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter09/main.go`.)
## main.go
The beginning of the program is pretty much the same.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address))
m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address))
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
```
This is where the program changes. First, we need to create a jar
for cookies because the API we're about to call requires a
cookie jar. (We mostly use this API with redirects and we want
to have cookies with redirects because a small portion of the
URLs we typically test require cookies to properly redirect,
see https://github.com/ooni/probe/issues/1727 for more information).
Then, we call `HTTPEndpointGetParallel`. The arguments are:
- as usual, the context
- the cookie jar
- all the endpoints to measure
```Go
cookies := measurex.NewCookieJar()
for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) {
m.Endpoints = append(m.Endpoints, epnt)
}
```
The `HTTPEndpointGetParallel` method returns a channel where it
posts `HTTPEndpointMeasurements`. Once the input list has been
fully measured, this method closes the returned channel.
Like we did before, we append the resulting measurements to
our `m` container and we print it.
```Go
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter09
```
Take a look at the JSON output. Can you spot that
endpoints measurements are run in parallel?
## Conclusion
We have seen how to run HTTPEndpoint measurements in parallel.

126
internal/tutorial/measurex/chapter09/main.go Normal file
View File

@ -0,0 +1,126 @@
// -=-=- StartHere -=-=-
//
// # Chapter IX: Parallel HTTPEndpoint measurements
//
// The program we see here is _really_ similar to the one we
// discussed in the previous chapter. The main difference
// is the following: rather than looping through the list of
// HTTPEndpoint, we call a function that runs through the
// list of endpoints using a small pool of background workers.
//
// There is a trade off between quick measurements and
// false positives. A timeout is one of the most common
// ways of censoring HTTPS and HTTP3 endpoints. So, if
// we run measurements sequentially, a whole scan could
// in principle take a long time. On the other hand,
// if we run too many parallel measurements, we may cause
// our own congestion and maybe some measurements will
// fail because of that. Our solution to this problem is
// to have low parallelism: at the moment of writing
// this note, we have three workers. If you submit
// more than three HTTPEndpoint at a a time, we will
// service the first three immediately and all the
// other endpoints will be queued for later measurement.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter09/main.go`.)
//
// ## main.go
//
// The beginning of the program is pretty much the same.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address))
m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address))
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
// ```
//
// This is where the program changes. First, we need to create a jar
// for cookies because the API we're about to call requires a
// cookie jar. (We mostly use this API with redirects and we want
// to have cookies with redirects because a small portion of the
// URLs we typically test require cookies to properly redirect,
// see https://github.com/ooni/probe/issues/1727 for more information).
//
// Then, we call `HTTPEndpointGetParallel`. The arguments are:
//
// - as usual, the context
//
// - the cookie jar
//
// - all the endpoints to measure
//
// ```Go
cookies := measurex.NewCookieJar()
for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) {
m.Endpoints = append(m.Endpoints, epnt)
}
// ```
//
// The `HTTPEndpointGetParallel` method returns a channel where it
// posts `HTTPEndpointMeasurements`. Once the input list has been
// fully measured, this method closes the returned channel.
//
// Like we did before, we append the resulting measurements to
// our `m` container and we print it.
//
// ```Go
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter09
// ```
//
// Take a look at the JSON output. Can you spot that
// endpoints measurements are run in parallel?
//
// ## Conclusion
//
// We have seen how to run HTTPEndpoint measurements in parallel.
//
// -=-=- StopHere -=-=-

127
internal/tutorial/measurex/chapter10/README.md Normal file
View File

@ -0,0 +1,127 @@
# Chapter IX: Parallel DNS lookups
The program we see here is _really_ similar to the one we
discussed in the previous chapter. The main difference
is the following: rather than performing DNS lookups
sequentially, we call a function that runs through the
list of resolvers and run them in parallel.
Again, we are going to use low parallelism for the same
rationale mentioned in chapter09.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter10/main.go`.)
## main.go
The beginning of the program is pretty much the same.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
```
The bulk of the difference is here. We create
a list of DNS resolvers. For each of them, we specify
the type and the endpoint address. (There is no
endpoint address for the system resolver, therefore
we leave its address empty.)
```Go
resolvers := []*measurex.ResolverInfo{{
Network: measurex.ResolverUDP,
Address: "8.8.8.8:53",
}, {
Network: measurex.ResolverUDP,
Address: "8.8.4.4:53",
}, {
Network: measurex.ResolverUDP,
Address: "1.1.1.1:53",
}, {
Network: measurex.ResolverUDP,
Address: "1.0.0.1:53",
}, {
Network: measurex.ResolverSystem,
Address: "",
}}
```
Then we call `LookupURLHostParallel`. This function runs
the queries that make sense given the input URL using a
pool of (currently three) background goroutines.
When I say "queries that make sense", I mostly mean
that we only query for HTTPSSvc when the input URL
scheme is "https". Otherwise, if it's just "http", it
does not make sense to send this query.
```Go
for dns := range mx.LookupURLHostParallel(ctx, parsed, resolvers...) {
m.DNS = append(m.DNS, dns)
}
```
The rest of the program is exactly like in chapter09.
```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
cookies := measurex.NewCookieJar()
for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) {
m.Endpoints = append(m.Endpoints, epnt)
}
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter10
```
Take a look at the JSON output. Can you spot that
DNS queries are run in parallel?
## Conclusion
We have seen how to run parallel DNS queries.

129
internal/tutorial/measurex/chapter10/main.go Normal file
View File

@ -0,0 +1,129 @@
// -=-=- StartHere -=-=-
//
// # Chapter IX: Parallel DNS lookups
//
// The program we see here is _really_ similar to the one we
// discussed in the previous chapter. The main difference
// is the following: rather than performing DNS lookups
// sequentially, we call a function that runs through the
// list of resolvers and run them in parallel.
//
// Again, we are going to use low parallelism for the same
// rationale mentioned in chapter09.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter10/main.go`.)
//
// ## main.go
//
// The beginning of the program is pretty much the same.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
// ```
//
// The bulk of the difference is here. We create
// a list of DNS resolvers. For each of them, we specify
// the type and the endpoint address. (There is no
// endpoint address for the system resolver, therefore
// we leave its address empty.)
//
// ```Go
resolvers := []*measurex.ResolverInfo{{
Network: measurex.ResolverUDP,
Address: "8.8.8.8:53",
}, {
Network: measurex.ResolverUDP,
Address: "8.8.4.4:53",
}, {
Network: measurex.ResolverUDP,
Address: "1.1.1.1:53",
}, {
Network: measurex.ResolverUDP,
Address: "1.0.0.1:53",
}, {
Network: measurex.ResolverSystem,
Address: "",
}}
// ```
//
// Then we call `LookupURLHostParallel`. This function runs
// the queries that make sense given the input URL using a
// pool of (currently three) background goroutines.
//
// When I say "queries that make sense", I mostly mean
// that we only query for HTTPSSvc when the input URL
// scheme is "https". Otherwise, if it's just "http", it
// does not make sense to send this query.
//
// ```Go
for dns := range mx.LookupURLHostParallel(ctx, parsed, resolvers...) {
m.DNS = append(m.DNS, dns)
}
// ```
//
// The rest of the program is exactly like in chapter09.
//
// ```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
cookies := measurex.NewCookieJar()
for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) {
m.Endpoints = append(m.Endpoints, epnt)
}
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter10
// ```
//
// Take a look at the JSON output. Can you spot that
// DNS queries are run in parallel?
//
// ## Conclusion
//
// We have seen how to run parallel DNS queries.
//
// -=-=- StopHere -=-=-

126
internal/tutorial/measurex/chapter11/README.md Normal file
View File

@ -0,0 +1,126 @@
# Chapter XI: Measuring a URL
This program shows how to measure an HTTP/HTTPS URL. We
are going to call an API whose implementation is
basically the same code we have seen in the previous
chapter, to obtain an URL measurement in a more compact
way. (As an historical note, the API we are going to
call has indeed been written as a refactoring of
the code we introduced in the previous chapter.)
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter11/main.go`.)
## main.go
The beginning of the program is much simpler. We have removed
out custom measurement type. We are now going to use the
`URLMeasurement` type (`go doc ./internal/measurex.URLMeasurement`),
which as the same fields of `measurement` in chapter10 _plus_
some extra fields that we'll examine in a later chapter.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://www.google.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
```
We create a measurer, cookies, and headers like we
saw in the previous chapter.
```Go
mx := measurex.NewMeasurerWithDefaultSettings()
cookies := measurex.NewCookieJar()
headers := measurex.NewHTTPRequestHeaderForMeasuring()
```
Then we call `MeasureURL`. This function's implementation
is in `./internal/measurex/measurer.go` and is pretty
much a refactoring of the code in chapter10.
The arguments are:
- the context as usual
- the unparsed URL to measure
- the headers we want to use
- a jar for cookies
```Go
m, err := mx.MeasureURL(ctx, *URL, headers, cookies)
```
The return value is either an `URLMeasurement`
or an error. The error happens, for example, if
the input URL scheme is not "http" or "https" (which
we handled by panicking in chapter11).
Now, rather than panicking inside `MeasureURL`, we
return the error to the caller and we `panic`
here on `main` using the `PanicOnError` function.
```Go
runtimex.PanicOnError(err, "mx.MeasureURL failed")
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter11
```
Take a look at the JSON output and compare it with:
```bash
go run -race ./internal/tutorial/measurex/chapter10 -url https://www.google.com
```
(which is basically forcing chapter10 to run with the
the default URL we use in this chapter).
Can you explain why we are able to measure more endpoints
in this chapter by checking the implementation of `MeasureURL`
and compare it to the code written in chapter10?
Now run:
```bash
go run -race ./internal/tutorial/measurex/chapter11 -url https://google.com
```
Do you see the opportunity there for following redirections? :^).
## Conclusion
We have introduced `MeasureURL`, the top-level API for
measuring a single URL.

128
internal/tutorial/measurex/chapter11/main.go Normal file
View File

@ -0,0 +1,128 @@
// -=-=- StartHere -=-=-
//
// # Chapter XI: Measuring a URL
//
// This program shows how to measure an HTTP/HTTPS URL. We
// are going to call an API whose implementation is
// basically the same code we have seen in the previous
// chapter, to obtain an URL measurement in a more compact
// way. (As an historical note, the API we are going to
// call has indeed been written as a refactoring of
// the code we introduced in the previous chapter.)
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter11/main.go`.)
//
// ## main.go
//
// The beginning of the program is much simpler. We have removed
// out custom measurement type. We are now going to use the
// `URLMeasurement` type (`go doc ./internal/measurex.URLMeasurement`),
// which as the same fields of `measurement` in chapter10 _plus_
// some extra fields that we'll examine in a later chapter.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://www.google.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
// ```
//
// We create a measurer, cookies, and headers like we
// saw in the previous chapter.
//
// ```Go
mx := measurex.NewMeasurerWithDefaultSettings()
cookies := measurex.NewCookieJar()
headers := measurex.NewHTTPRequestHeaderForMeasuring()
// ```
//
// Then we call `MeasureURL`. This function's implementation
// is in `./internal/measurex/measurer.go` and is pretty
// much a refactoring of the code in chapter10.
//
// The arguments are:
//
// - the context as usual
//
// - the unparsed URL to measure
//
// - the headers we want to use
//
// - a jar for cookies
//
// ```Go
m, err := mx.MeasureURL(ctx, *URL, headers, cookies)
// ```
// The return value is either an `URLMeasurement`
// or an error. The error happens, for example, if
// the input URL scheme is not "http" or "https" (which
// we handled by panicking in chapter11).
//
// Now, rather than panicking inside `MeasureURL`, we
// return the error to the caller and we `panic`
// here on `main` using the `PanicOnError` function.
//
// ```Go
runtimex.PanicOnError(err, "mx.MeasureURL failed")
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter11
// ```
//
// Take a look at the JSON output and compare it with:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter10 -url https://www.google.com
// ```
//
// (which is basically forcing chapter10 to run with the
// the default URL we use in this chapter).
//
// Can you explain why we are able to measure more endpoints
// in this chapter by checking the implementation of `MeasureURL`
// and compare it to the code written in chapter10?
//
// Now run:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter11 -url https://google.com
// ```
//
// Do you see the opportunity there for following redirections? :^).
//
// ## Conclusion
//
// We have introduced `MeasureURL`, the top-level API for
// measuring a single URL.
//
// -=-=- StopHere -=-=-

94
internal/tutorial/measurex/chapter12/README.md Normal file
View File

@ -0,0 +1,94 @@
# Chapter XII: Following redirections.
This program shows how to combine the URL measurement
"step" introduced in the previous chapter with
following redirections. If we say that the previous
chapter performed a "web step", then we can say
that here we're performing multiple "web steps".
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter12/main.go`.)
## main.go
The beginning of the program is pretty much the
same, except that here we need to define a
`measurement` container type that will contain
the result of each "web step".
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
URLs []*measurex.URLMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "http://facebook.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
all := &measurement{}
mx := measurex.NewMeasurerWithDefaultSettings()
cookies := measurex.NewCookieJar()
headers := measurex.NewHTTPRequestHeaderForMeasuring()
```
Everything above this line is like in chapter11. What changes
now is that we're calling `MeasureURLAndFollowRedirections`
instead of `MeasureURL`.
Rather than returning a single measurement, this function
returns a channel where it posts the result of measuring
the original URL along with all its redirections. Internally,
`MeasureURLAndFollowRedirections` calls `MeasureURL`.
We accumulate the results in `URLs` and print `m`. The channel
is closed when done by `MeasureURLAndFollowRedirections`, so we leave the loop.
```Go
for m := range mx.MeasureURLAndFollowRedirections(ctx, *URL, headers, cookies) {
all.URLs = append(all.URLs, m)
}
print(all)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter12
```
Take a look at the JSON. You should see several redirects
and that we measure each endpoint of each redirect, including
QUIC endpoints that we discover on the way.
## Conclusion
We have introduced `MeasureURLAndFollowRedirect`, the
top-level API for fully measuring a URL and all the URLs
that derive from such an URL via redirection.

96
internal/tutorial/measurex/chapter12/main.go Normal file
View File

@ -0,0 +1,96 @@
// -=-=- StartHere -=-=-
//
// # Chapter XII: Following redirections.
//
// This program shows how to combine the URL measurement
// "step" introduced in the previous chapter with
// following redirections. If we say that the previous
// chapter performed a "web step", then we can say
// that here we're performing multiple "web steps".
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter12/main.go`.)
//
// ## main.go
//
// The beginning of the program is pretty much the
// same, except that here we need to define a
// `measurement` container type that will contain
// the result of each "web step".
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
URLs []*measurex.URLMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "http://facebook.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
all := &measurement{}
mx := measurex.NewMeasurerWithDefaultSettings()
cookies := measurex.NewCookieJar()
headers := measurex.NewHTTPRequestHeaderForMeasuring()
// ```
//
// Everything above this line is like in chapter11. What changes
// now is that we're calling `MeasureURLAndFollowRedirections`
// instead of `MeasureURL`.
//
// Rather than returning a single measurement, this function
// returns a channel where it posts the result of measuring
// the original URL along with all its redirections. Internally,
// `MeasureURLAndFollowRedirections` calls `MeasureURL`.
//
// We accumulate the results in `URLs` and print `m`. The channel
// is closed when done by `MeasureURLAndFollowRedirections`, so we leave the loop.
//
// ```Go
for m := range mx.MeasureURLAndFollowRedirections(ctx, *URL, headers, cookies) {
all.URLs = append(all.URLs, m)
}
print(all)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter12
// ```
//
// Take a look at the JSON. You should see several redirects
// and that we measure each endpoint of each redirect, including
// QUIC endpoints that we discover on the way.
//
// ## Conclusion
//
// We have introduced `MeasureURLAndFollowRedirect`, the
// top-level API for fully measuring a URL and all the URLs
// that derive from such an URL via redirection.
//
// -=-=- StopHere -=-=-

97
internal/tutorial/measurex/chapter13/README.md Normal file
View File

@ -0,0 +1,97 @@
# Chapter XIII: Rewriting Web Connectivity
This chapter contains an exercise. We are going to
use the `measurex` API to rewrite part of the
Web Connectivity network experiment.
(This is probably the right place to prod you
to go to the [ooni/spec](https://github.com/ooni/spec)
repository, locate the ts-017-web-connectivity.md
spec, and read it.)
Read the spec? Good, so
what we are more precisely going to do here
is implement the network measurement part of
Web Connectivity where we:
1. enumerate all the IP addresses of the target
URL using the system resolver;
2. build endpoints with such IPs with a suitable
port, thus obtaining a list of HTTP endpoints;
3. TCP connect each of the endpoints and save the
results into a measurement object compatible
with Web Connectivity's data format;
4. TLS handshake each endpoint (only if this
makes sense, of course);
5. HTTP GET the URL and follow redirects until
we reach a webpage, fetch the body, and store it
for later analysis (which we'll not implement
as part of this exercise).
Let us now provide extra context that should
help you figure out how to solve this exercise.
## Regarding points 3-4
You already know all the primitives.
## Regarding point 5
Historically this point has always been
performed by a separate HTTP client. This
means that any implementation:
- will not include any TCP or TLS event
generated during point 5 in the measurement;
- most likely will resolve the URL's domain
again (even though the probe-cli implementation
uses a fake Resolver to avoid that);
- tries every available IP address and stops
at the first one to which it can connect to (which
is what a naive HTTP client does, whereas a more
advanced one likely tries a couple of addrs in
parallel, especially when both IPv4 and IPv6
are supported - this is also known as happy eyeballs).
In terms of `measurex`, the best API to do what
you're required to do in point 5 is probably
`NewTracingHTTPTransportWithDefaultSettings`, which
allows you to trace only the HTTP round trip and
ignores any other event.
Once you have such a transport, the best `Measurer`
API for the task is probably `HTTPClientGET`.
## Other remarks
You also need to learn about how to measure
events at low level, which entails creating an
instance of `MeasurementDB`, passing it to
the relevant networking code, and then calling
its `AsMeasurement` method to get back a
measurement. (You can probably get an idea
of how this is done in general by checking the
implementation of `Measurer.TCPConnect`.)
Hopefully, this should be enough information
to help you tackle this task. As you see
below, the main function is there empty waiting
for your implementation. We will provide our
own solution to this problem in the next chapter.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter13/main.go`.)
## The main.go file
```Go
package main
func main() {
}

98
internal/tutorial/measurex/chapter13/main.go Normal file
View File

@ -0,0 +1,98 @@
// -=-=- StartHere -=-=-
//
// # Chapter XIII: Rewriting Web Connectivity
//
// This chapter contains an exercise. We are going to
// use the `measurex` API to rewrite part of the
// Web Connectivity network experiment.
// (This is probably the right place to prod you
// to go to the [ooni/spec](https://github.com/ooni/spec)
// repository, locate the ts-017-web-connectivity.md
// spec, and read it.)
//
// Read the spec? Good, so
// what we are more precisely going to do here
// is implement the network measurement part of
// Web Connectivity where we:
//
// 1. enumerate all the IP addresses of the target
// URL using the system resolver;
//
// 2. build endpoints with such IPs with a suitable
// port, thus obtaining a list of HTTP endpoints;
//
// 3. TCP connect each of the endpoints and save the
// results into a measurement object compatible
// with Web Connectivity's data format;
//
// 4. TLS handshake each endpoint (only if this
// makes sense, of course);
//
// 5. HTTP GET the URL and follow redirects until
// we reach a webpage, fetch the body, and store it
// for later analysis (which we'll not implement
// as part of this exercise).
//
// Let us now provide extra context that should
// help you figure out how to solve this exercise.
//
// ## Regarding points 3-4
//
// You already know all the primitives.
//
// ## Regarding point 5
//
// Historically this point has always been
// performed by a separate HTTP client. This
// means that any implementation:
//
// - will not include any TCP or TLS event
// generated during point 5 in the measurement;
//
// - most likely will resolve the URL's domain
// again (even though the probe-cli implementation
// uses a fake Resolver to avoid that);
//
// - tries every available IP address and stops
// at the first one to which it can connect to (which
// is what a naive HTTP client does, whereas a more
// advanced one likely tries a couple of addrs in
// parallel, especially when both IPv4 and IPv6
// are supported - this is also known as happy eyeballs).
//
// In terms of `measurex`, the best API to do what
// you're required to do in point 5 is probably
// `NewTracingHTTPTransportWithDefaultSettings`, which
// allows you to trace only the HTTP round trip and
// ignores any other event.
//
// Once you have such a transport, the best `Measurer`
// API for the task is probably `HTTPClientGET`.
//
// ## Other remarks
//
// You also need to learn about how to measure
// events at low level, which entails creating an
// instance of `MeasurementDB`, passing it to
// the relevant networking code, and then calling
// its `AsMeasurement` method to get back a
// measurement. (You can probably get an idea
// of how this is done in general by checking the
// implementation of `Measurer.TCPConnect`.)
//
// Hopefully, this should be enough information
// to help you tackle this task. As you see
// below, the main function is there empty waiting
// for your implementation. We will provide our
// own solution to this problem in the next chapter.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter13/main.go`.)
//
// ## The main.go file
//
// ```Go
package main
func main() {
}

318
internal/tutorial/measurex/chapter14/README.md Normal file
View File

@ -0,0 +1,318 @@
# Chapter XIV: A possible rewrite of Web Connectivity
In this chapter we try to solve the exercise laid out in
the previous chapter, using `measurex` primitives.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter14/main.go`.)
## main.go
The beginning of the file is always pretty much the same.
```Go
package main
import (
"context"
"crypto/tls"
"encoding/json"
"flag"
"fmt"
"net/http"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/netxlite"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
```
## measurement type
We define a measurement type with the fields
that a Web Connectivity measurement should have.
```Go
type measurement struct {
Queries []*measurex.DNSLookupEvent `json:"queries"`
TCPConnect []*measurex.NetworkEvent `json:"tcp_connect"`
TLSHandshakes []*measurex.TLSHandshakeEvent `json:"tls_handshakes"`
Requests []*measurex.HTTPRoundTripEvent `json:"requests"`
}
```
## WebConnectivity implementation
We define a function that takes in input a context and a URL to
measure and returns a measurement or an error.
We will only error out in case the input does not allow us to
proceed (i.e., invalid input URL).
```Go
func webConnectivity(ctx context.Context, URL string) (*measurement, error) {
```
We start by parsing the input URL. If we cannot parse it, of
course this is an hard error and we cannot continue.
```Go
parsedURL, err := url.Parse(URL)
if err != nil {
return nil, err
}
```
We create an empty measurement and a measurer with
default settings like we did in the previous chapters.
```Go
m := &measurement{}
mx := measurex.NewMeasurerWithDefaultSettings()
```
Now it's time to start measuring. We will address all
the points laid out in the previous chapter.
### 1. Enumerating IP addrs
Let us enumerate all the IP addresses for
the input URL's domain using the system resolver.
```Go
dns := mx.LookupHostSystem(ctx, parsedURL.Hostname())
m.Queries = append(m.Queries, dns.LookupHost...)
```
This is code we have already seen in previous chapter.
### 2. Building a list of endpoints
```Go
epnts, err := measurex.AllHTTPEndpointsForURL(parsedURL, http.Header{}, dns)
if err != nil {
return nil, err
}
```
This is also code we have seen in previous chapters. The only
difference is that we supply empty headers since we're not going
to actually use the headers inside the endpoints.
### 3 and 4. Measure each endpoint
We will loop through the endpoints in the previous point
and issue the correct TCP or TLS primitive depending on
whether the input URL is HTTP or HTTPS.
```Go
for _, epnt := range epnts {
switch parsedURL.Scheme {
case "http":
tcp := mx.TCPConnect(ctx, epnt.Address)
m.TCPConnect = append(m.TCPConnect, tcp.Connect...)
case "https":
config := &tls.Config{
ServerName: parsedURL.Hostname(),
NextProtos: []string{"h2", "http/1.1"},
RootCAs: netxlite.NewDefaultCertPool(),
}
tls := mx.TLSConnectAndHandshake(ctx, epnt.Address, config)
m.TCPConnect = append(m.TCPConnect, tls.Connect...)
m.TLSHandshakes = append(m.TLSHandshakes, tls.TLSHandshake...)
}
}
```
At this point we've addressed points 1-4. So let's
now focus on the last point:
### 5. HTTP measurement
We need to manually build a `MeasurementDB`. This is a
"database" where networking code will store events.
```Go
db := &measurex.MeasurementDB{}
```
Following the hint from the previous chapter we use the
`NewTracingHTTPTransportWithDefaultSettings` factory
to create an `http.Transport`-like object that will trace
HTTP round trip events writing them into `db`.
```Go
txp := measurex.NewTracingHTTPTransportWithDefaultSettings(mx.Begin, mx.Logger, db)
```
We now build an `http.Client` using the transport
we've just created and a cookie jar (which we
use because otherwise some redirects will lead
to a redirect loop, as mentioned in previous chapters).
```Go
clnt := &http.Client{
Transport: txp,
Jar: measurex.NewCookieJar(),
}
```
Now we use a method of the measurer that allows us to
perform an HTTP GET with an existing HTTP client
and a URL. This method will set a timeout and perform
the round trip. Reading a snapshot of the response
body is not implemented by this function but rather
is a property of the "tracing" HTTP transport we
created above (this type of transport is the one we
have been internally using in all the examples
presented so far.)
```Go
resp, _ := mx.HTTPClientGET(ctx, clnt, parsedURL)
```
To be tidy, we also close the response body in case
we have a response. We don't really need to read
the body here. As mentioned previously, we're already
using an HTTP transport reading a body snapshot.
```Go
if resp != nil {
resp.Body.Close() // tidy
}
```
Finally, we append the round trips we performed into
the right field and return the measurement.
To this end, we're using the `db.AsMeasurement` method that
takes the current set of events into `db` and assembles
them into the `Measurement` struct we've been using in all
the chapters we have seen so far.
```Go
m.Requests = append(m.Requests, db.AsMeasurement().HTTPRoundTrip...)
return m, nil
}
```
The rest of the program is pretty straightforward.
```Go
func main() {
URL := flag.String("url", "https://www.google.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
m, err := webConnectivity(ctx, *URL)
runtimex.PanicOnError(err, "invalid arguments to webConnectivity (wrong URL?)")
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter14
```
Take a look at the JSON.
Now try running the program with `http://gmail.com` as
input. Take note of the redirect chain. See how the
domain changes during the redirect. Take note of the
fact that we are not measuring any TLS handshake. See
how we're not trying QUIC endpoints. These are, in
fact, some of the limitations of Web Connectivity that
we were trying to address when we wrote `measurex`.
Also, build the miniooni research client:
```
go build -v ./internal/cmd/miniooni
```
Run Web Connectivity with:
```
./miniooni -ni http://gmail.com web_connectivity
```
This writes the report in a file named `report.jsonl`.
Check the content of the file and match it with the
output of this chapter. Are there other notable
differences between the two outputs?
### Bonus question
The solution we presented is true to the original
spirit of Web Connectivity, where we first perform
separate DNS, TCP/TLS steps, and then we also
perform a separate HTTP step. Is there in `measurex`
an API allowing you to invert the order of the
operations, that is:
1. build a full-fledged HTTP client where we can
trace _any_ operation;
2. use such client to measure the URL;
3. figure out what TCP endpoints we did not
test for TCP/TLS during this process and run
TCP/TLS testing only for them?
If such an API exist, can you write a simple
main.go client that implements points 1-3 above?
## Conclusion
We have presented the solution to the exercise
proposed in the previous chapter, i.e., how
to rewrite Web Connectivity using `measurex` API.
You have now been exposed to some complexity and
APIs to perform OONI measurements. So you should now
be read to help us write new and maitain existing
network experiments.
If you have further questions, please [contact us](
https://ooni.org/about/).

320
internal/tutorial/measurex/chapter14/main.go Normal file
View File

@ -0,0 +1,320 @@
// -=-=- StartHere -=-=-
//
// # Chapter XIV: A possible rewrite of Web Connectivity
//
// In this chapter we try to solve the exercise laid out in
// the previous chapter, using `measurex` primitives.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter14/main.go`.)
//
// ## main.go
//
// The beginning of the file is always pretty much the same.
//
// ```Go
package main
import (
"context"
"crypto/tls"
"encoding/json"
"flag"
"fmt"
"net/http"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/netxlite"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
// ```
//
// ## measurement type
//
// We define a measurement type with the fields
// that a Web Connectivity measurement should have.
//
// ```Go
type measurement struct {
Queries []*measurex.DNSLookupEvent `json:"queries"`
TCPConnect []*measurex.NetworkEvent `json:"tcp_connect"`
TLSHandshakes []*measurex.TLSHandshakeEvent `json:"tls_handshakes"`
Requests []*measurex.HTTPRoundTripEvent `json:"requests"`
}
// ```
//
// ## WebConnectivity implementation
//
// We define a function that takes in input a context and a URL to
// measure and returns a measurement or an error.
//
// We will only error out in case the input does not allow us to
// proceed (i.e., invalid input URL).
//
// ```Go
func webConnectivity(ctx context.Context, URL string) (*measurement, error) {
// ```
//
// We start by parsing the input URL. If we cannot parse it, of
// course this is an hard error and we cannot continue.
//
// ```Go
parsedURL, err := url.Parse(URL)
if err != nil {
return nil, err
}
// ```
//
// We create an empty measurement and a measurer with
// default settings like we did in the previous chapters.
//
// ```Go
m := &measurement{}
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// Now it's time to start measuring. We will address all
// the points laid out in the previous chapter.
//
// ### 1. Enumerating IP addrs
//
// Let us enumerate all the IP addresses for
// the input URL's domain using the system resolver.
//
// ```Go
dns := mx.LookupHostSystem(ctx, parsedURL.Hostname())
m.Queries = append(m.Queries, dns.LookupHost...)
// ```
//
// This is code we have already seen in previous chapter.
//
//
// ### 2. Building a list of endpoints
//
// ```Go
epnts, err := measurex.AllHTTPEndpointsForURL(parsedURL, http.Header{}, dns)
if err != nil {
return nil, err
}
// ```
//
// This is also code we have seen in previous chapters. The only
// difference is that we supply empty headers since we're not going
// to actually use the headers inside the endpoints.
//
// ### 3 and 4. Measure each endpoint
//
// We will loop through the endpoints in the previous point
// and issue the correct TCP or TLS primitive depending on
// whether the input URL is HTTP or HTTPS.
//
// ```Go
for _, epnt := range epnts {
switch parsedURL.Scheme {
case "http":
tcp := mx.TCPConnect(ctx, epnt.Address)
m.TCPConnect = append(m.TCPConnect, tcp.Connect...)
case "https":
config := &tls.Config{
ServerName: parsedURL.Hostname(),
NextProtos: []string{"h2", "http/1.1"},
RootCAs: netxlite.NewDefaultCertPool(),
}
tls := mx.TLSConnectAndHandshake(ctx, epnt.Address, config)
m.TCPConnect = append(m.TCPConnect, tls.Connect...)
m.TLSHandshakes = append(m.TLSHandshakes, tls.TLSHandshake...)
}
}
// ```
//
// At this point we've addressed points 1-4. So let's
// now focus on the last point:
//
// ### 5. HTTP measurement
//
// We need to manually build a `MeasurementDB`. This is a
// "database" where networking code will store events.
//
// ```Go
db := &measurex.MeasurementDB{}
// ```
//
// Following the hint from the previous chapter we use the
// `NewTracingHTTPTransportWithDefaultSettings` factory
// to create an `http.Transport`-like object that will trace
// HTTP round trip events writing them into `db`.
//
//
// ```Go
txp := measurex.NewTracingHTTPTransportWithDefaultSettings(mx.Begin, mx.Logger, db)
// ```
//
// We now build an `http.Client` using the transport
// we've just created and a cookie jar (which we
// use because otherwise some redirects will lead
// to a redirect loop, as mentioned in previous chapters).
//
// ```Go
clnt := &http.Client{
Transport: txp,
Jar: measurex.NewCookieJar(),
}
// ```
//
// Now we use a method of the measurer that allows us to
// perform an HTTP GET with an existing HTTP client
// and a URL. This method will set a timeout and perform
// the round trip. Reading a snapshot of the response
// body is not implemented by this function but rather
// is a property of the "tracing" HTTP transport we
// created above (this type of transport is the one we
// have been internally using in all the examples
// presented so far.)
//
// ```Go
resp, _ := mx.HTTPClientGET(ctx, clnt, parsedURL)
// ```
//
// To be tidy, we also close the response body in case
// we have a response. We don't really need to read
// the body here. As mentioned previously, we're already
// using an HTTP transport reading a body snapshot.
//
// ```Go
if resp != nil {
resp.Body.Close() // tidy
}
// ```
//
// Finally, we append the round trips we performed into
// the right field and return the measurement.
//
// To this end, we're using the `db.AsMeasurement` method that
// takes the current set of events into `db` and assembles
// them into the `Measurement` struct we've been using in all
// the chapters we have seen so far.
//
// ```Go
m.Requests = append(m.Requests, db.AsMeasurement().HTTPRoundTrip...)
return m, nil
}
// ```
//
// The rest of the program is pretty straightforward.
//
// ```Go
func main() {
URL := flag.String("url", "https://www.google.com/", "URL to fetch")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
m, err := webConnectivity(ctx, *URL)
runtimex.PanicOnError(err, "invalid arguments to webConnectivity (wrong URL?)")
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter14
// ```
//
// Take a look at the JSON.
//
// Now try running the program with `http://gmail.com` as
// input. Take note of the redirect chain. See how the
// domain changes during the redirect. Take note of the
// fact that we are not measuring any TLS handshake. See
// how we're not trying QUIC endpoints. These are, in
// fact, some of the limitations of Web Connectivity that
// we were trying to address when we wrote `measurex`.
//
// Also, build the miniooni research client:
//
// ```
// go build -v ./internal/cmd/miniooni
// ```
//
// Run Web Connectivity with:
//
// ```
// ./miniooni -ni http://gmail.com web_connectivity
// ```
//
// This writes the report in a file named `report.jsonl`.
//
// Check the content of the file and match it with the
// output of this chapter. Are there other notable
// differences between the two outputs?
//
// ### Bonus question
//
// The solution we presented is true to the original
// spirit of Web Connectivity, where we first perform
// separate DNS, TCP/TLS steps, and then we also
// perform a separate HTTP step. Is there in `measurex`
// an API allowing you to invert the order of the
// operations, that is:
//
// 1. build a full-fledged HTTP client where we can
// trace _any_ operation;
//
// 2. use such client to measure the URL;
//
// 3. figure out what TCP endpoints we did not
// test for TCP/TLS during this process and run
// TCP/TLS testing only for them?
//
// If such an API exist, can you write a simple
// main.go client that implements points 1-3 above?
//
// ## Conclusion
//
// We have presented the solution to the exercise
// proposed in the previous chapter, i.e., how
// to rewrite Web Connectivity using `measurex` API.
//
// You have now been exposed to some complexity and
// APIs to perform OONI measurements. So you should now
// be read to help us write new and maitain existing
// network experiments.
//
// If you have further questions, please [contact us](
// https://ooni.org/about/).
//
// -=-=- StopHere -=-=-