diff --git a/internal/tutorial/README.md b/internal/tutorial/README.md index 22f1070..a66c0a0 100644 --- a/internal/tutorial/README.md +++ b/internal/tutorial/README.md @@ -10,6 +10,8 @@ real OONI code, it should always be up to date. - [Rewriting the torsf experiment](experiment/torsf/) +- [Using the measurex package to write network experiments](measurex) + ## Regenerating the tutorials diff --git a/internal/tutorial/generator/main.go b/internal/tutorial/generator/main.go index a7b02ab..b35316f 100644 --- a/internal/tutorial/generator/main.go +++ b/internal/tutorial/generator/main.go @@ -90,6 +90,25 @@ func gentorsf() { gen(path.Join(prefix, "chapter04"), "torsf.go") } +// genmeasurex generates the measurex chapters. +func genmeasurex() { + prefix := path.Join(".", "measurex") + gen(path.Join(prefix, "chapter01"), "main.go") + gen(path.Join(prefix, "chapter02"), "main.go") + gen(path.Join(prefix, "chapter03"), "main.go") + gen(path.Join(prefix, "chapter04"), "main.go") + gen(path.Join(prefix, "chapter05"), "main.go") + gen(path.Join(prefix, "chapter06"), "main.go") + gen(path.Join(prefix, "chapter07"), "main.go") + gen(path.Join(prefix, "chapter08"), "main.go") + gen(path.Join(prefix, "chapter09"), "main.go") + gen(path.Join(prefix, "chapter10"), "main.go") + gen(path.Join(prefix, "chapter11"), "main.go") + gen(path.Join(prefix, "chapter12"), "main.go") + gen(path.Join(prefix, "chapter13"), "main.go") + gen(path.Join(prefix, "chapter14"), "main.go") +} + // gennetxlite generates the netxlite chapters. func gennetxlite() { prefix := path.Join(".", "netxlite") @@ -102,7 +121,9 @@ func gennetxlite() { gen(path.Join(prefix, "chapter07"), "main.go") gen(path.Join(prefix, "chapter08"), "main.go") } + func main() { gentorsf() + genmeasurex() gennetxlite() } diff --git a/internal/tutorial/measurex/README.md b/internal/tutorial/measurex/README.md new file mode 100644 index 0000000..f89f9aa --- /dev/null +++ b/internal/tutorial/measurex/README.md @@ -0,0 +1,104 @@ +# Using the measurex package to write network experiments + +This tutorial teaches you how to write OONI network +experiments using the primitives in the `./internal/measurex` +package. The name of this package means either "measure +extensions" or "measure crossover". + +The measure extension interpretation of the name explains +what this package does. It contains extensions to our +basic networking code (`./internal/netxlite`) that allow +us to perform OONI measurements. + +The measure crossover interpretation explains its history. Since +OONI has been written in Go, we've mostly performed measurements +using "tracing". That is, by registering hooks that run when +specific operations happen (e.g., TCP connect or TLS handshake) +and then making sense of the network trace. This is the approach +with which most experiments are written as of 2021-09-24. But +we have also seen that in several cases a step-by-step approach +is preferrable. Under this approach, you perform individual +operations and record their result right away. So, for example, +you have a connection and a TLS config, you perform a TLS +handshake, and immediately after you create and store somewhere +a data structure containing the result. This package is at the +crossover of these two approaches, because basically it contains +enough primitives to support both. + +What we are going to do in this tutorial is the following: + +- we will start from very simple-step-by-step measurements such +as TCP connect, DNS lookup, and TLS handshake; + +- we will see how `measurex` provides support for composing +these primitives in larger steps, which will lead us to +eventually perform all the measurements that matter for a +given input URL (including discovering QUIC endpoints +and following redirections); + +- finally, as an exercise, we will use the knowledge +acquired in the rest of the tutorial to rewrite a +subset of the Web Connectivity experiment (as of 2021-09-24 +the flagship OONI experiment). This will be an opportunity +to explore more low level aspects of `measurex`. + +As part of the process, we'll introduce you to the data +format used by OONI and there will be proposed exercises +where we simulate censorship conditions and we see how +that impacts the generated measurements. + +Every chapter will show how to write a simple `main.go` +program that explains how to use some primitives. The +chapter text itself is autogenerated from comments inside +the actual `main.go` the we describe in the chapter. + +For this reason, if you need to change a chapter, you +need to change the corresponding `main.go` file and then +follow the instructions at `./internal/tutorial/generate` +to regenerate the markdown text of the chapter. + +More in detail, here's the index: + +- [chapter01](chapter01) explains how to use the "system" resolver + +- [chapter02](chapter02) deals with establishing TCP connections + +- [chapter03](chapter03) is about using custom DNS-over-UDP resolvers + +- [chapter04](chapter04) shows how to measure TLS handshakes + +- [chapter05](chapter05) is about the QUIC handshake + +- [chapter06](chapter06) shows how to get a webpage knowing its +URL and the endpoint (i.e., IP address and TCP/UDP port) + +- [chapter07](chapter07) shows how to extend what we did in +chapter06 to cover _all_ the IP addresses in the URL's domain + +- [chapter08](chapter08) is about HTTPSSvc DNS queries and +how they can be used to discover and test QUIC endpoints, thus +extending the work done in chapter07 + +- [chapter09](chapter09) improves upon chapter08 showing +how to run endpoints measurements in parallel + +- [chapter10](chapter10) improves upon chapter09 by +also running DNS queries in parallel + +- [chapter11](chapter11) tells you that all the code we +have been writing so far, and specifically the code we have +in chapter10, is actually the implementation of an API +of `measurex` called `MeasureURL`, and then shows you how +you can simplify the code in chapter10 by using this API. + +- [chapter12](chapter12) extends the work done in +chapter11 by teaching you about a more high-level API +that discovers and follows all redirections, calling +`MeasureURL` for each redirection. + +- [chapter13](chapter13) contains the exercise regarding +rewriting WebConnectivity using all the tools you have +learned so far and pointing you at additional `measurex` +API that could be useful to solve the problem. + +- [chapter14](chapter14) contains our solution to the exercise. diff --git a/internal/tutorial/measurex/chapter01/README.md b/internal/tutorial/measurex/chapter01/README.md new file mode 100644 index 0000000..2b5e8b3 --- /dev/null +++ b/internal/tutorial/measurex/chapter01/README.md @@ -0,0 +1,366 @@ + +# Chapter I: using the system resolver + +In this chapter we explain how to measure DNS resolutions performed +using the system resolver. *En passant*, we will also introduce you to +the `Measurer`, which we will use for the rest of the tutorial. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter01/main.go`.) + +## The system resolver + +We define "system resolver" as the DNS resolver implemented by the C +library. On Unix, the most popular interface to such a resolver is +the `getaddrinfo(3)` C library function. + +Most OONI experiments (also known as nettests) use the system +resolver to map domain names to IP addresses. The advantage of +the system resolver is that it's provided by the system. So, +it should _generally_ work. Also, it is the resolver that the +user of the system will use every day, therefore its results +should be representative (even though the rise of DNS over +HTTPS embedded in browsers may make this statement less solid +than it were ten years ago). + +The disadvantage of the system resolver is that we do not +know how it is configured. Say the user has configured a +DNS over TLS resolver; then the measurements may miss censorship +that we would otherwise see if using a custom DNS resolver. + +Now that we have justified why the system resolver is +important for OONI, let us perform some measurements with it. + +We will first write a simple `main.go` file that shows how to use +this functionality. Then, we will show some runs of this file, and +we will comment the output that we see. + +## main.go + +We declare the package and import useful packages. The most +important package we're importing here is, of course, `internal/measurex`. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { +``` +### Setup + +We define command line flags useful to test this program. We use +the `flags` package for that. We want the user to be able to configure +both the domain name to resolve and the resolution timeout. + +```Go + domain := flag.String("domain", "example.com", "domain to resolve") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") +``` + +We call `flag.Parse` to parse the CLI flags. + +```Go + flag.Parse() +``` + +We create a context and we attach a timeout to it. (This is a pretty +standard way of configuring a timeout in Go.) + +```Go + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() +``` + +### Creating a Measurer + +Now we create a `Measurer`. + +```Go + mx := measurex.NewMeasurerWithDefaultSettings() +``` + +The `Measurer` is a concrete type that contains many fields +requiring initialization. For this reason, we provide a factory +that creates one with default settings. The expected usage +pattern is that you do not modify a `Measurer`'s field after +initialization. Modifying them while the `Measurer` is in +use could, in fact, lead to races. + +Let's now invoke the system resolver to resolve `*domain`! + +### Invoking the system resolver + +We call the `LookupHostSystem` method of the `Measurer`. The +arguments are the Context, that in this case carries the timeout +we configured above, and the domain to resolve. + +The call itself is named `LookupHost` because this is the name +used by the Go function that performs a domain lookup. + +Under the hood, `mx.LookupHostSystem` will eventually call +`(*net.Resolver).LookupHost`. In turn, in the common case on +Unix, this function will eventually call `getaddrinfo(3)`. + +```Go + m := mx.LookupHostSystem(ctx, *domain) +``` + +The return value of `(*net.Resolver).LookupHost` is either a +list of IP addresses or an error. Our `LookupHostSystem` method, +instead, returns a `*measurex.DNSMeasurement` type. + +This is probably a good moment to remind you of Go's +built in help system. We could include a definition of the +`DNSMeasurement` structure, but since this definition is +just a comment in the main.go file, it might age badly. + +Instead, if you run + +``` +go doc ./internal/measurex.DNSMeasurement +``` + +You get the current definition. As you can see, this type +is basically just a wrapper around `Measurement`. Now, +checking the docs of `Measurement` with + +``` +go doc ./internal/measurex.Measurement +``` + +we can see a container of events +classified by event type. In our case, because we're +doing a `LookupHost`, we should have at least one entry +inside of the `Measurement.LookupHost` field. + +This entry is of type `DNSLookupEvent`. Let us check +together the definition of this type: + +``` +go doc ./internal/measurex.DNSLookupEvent +``` + +If you are familiar with [the OONI data format specs]( +https://github.com/ooni/spec/tree/master/data-formats), you +should probably recognize that this structure is the Go +representation of the `df-002-dnst` data format. + +In fact, every event field inside of a `Measurement` +should serialize nicely to JSON to one of the OONI data +formats. + +### Printing the measurement + +Because there is a close relationship between the +events inside a `Measurement` and the JSON OONI data +format, in the remainder of this program we're +going to serialize the `Measurement` to JSON and +print it to the standard output. + +```Go + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +``` + +As a final note, the `PanicOnError` is here because the +message `m` *can* be marshalled to JSON. It still feels a +bit better having an assertion for our assumptions than +outrightly ignoring the error code. (We tend to use such +a convention quite frequently in the OONI codebase.) + +```Go +} + +``` + +## Running the example program + +Let us run the program with default arguments first. You can do +this operation by running: + +```bash +go run -race ./internal/tutorial/measurex/chapter01 +``` + +If you do that you obtain some logging messages, which are out of +the scope of this tutorial, and the following JSON: + +```JSON +{ + "domain": "example.com", + "lookup_host": [ + { + "answers": [ + { + "answer_type": "A", + "ipv4": "93.184.216.34" + } + ], + "engine": "system", + "failure": null, + "hostname": "example.com", + "query_type": "A", + "resolver_address": "", + "t": 0.002996459, + "started": 9.8e-05, + "oddity": "" + }, + { + "answers": [ + { + "answer_type": "AAAA", + "ivp6": "2606:2800:220:1:248:1893:25c8:1946" + } + ], + "engine": "system", + "failure": null, + "hostname": "example.com", + "query_type": "AAAA", + "resolver_address": "", + "t": 0.002996459, + "started": 9.8e-05, + "oddity": "" + } + ] +} +``` + +You see that we have two messages here. OONI splits a DNS +resolution performed using the system resolver into two "fake" +DNS resolutions for A and AAAA. (Under the hood, this is +what the system resolver would most likely do.) + +The most important fields are: + +- _engine_, indicating that we are using the "system" resolver; + +- _hostname_, meaning that we wanted to resolve the "example.com" domain; + +- _answers_, which contains a list of answers; + +- _t_, which is the time when the LookupHost operation completed. + +### NXDOMAIN measurement + +Let us now change the domain to resolve to be `antani.ooni.org` (a +nonexisting domain), which we can do by running this command: + +```bash +go run -race ./internal/tutorial/measurex/chapter01 -domain antani.ooni.org +``` + +This is the output JSON: + +```JSON +{ + "domain": "antani.ooni.org", + "lookup_host": [ + { + "answers": null, + "engine": "system", + "failure": "dns_nxdomain_error", + "hostname": "antani.ooni.org", + "query_type": "A", + "resolver_address": "", + "t": 0.072963834, + "started": 0.000125417, + "oddity": "dns.lookup.nxdomain" + }, + { + "answers": null, + "engine": "system", + "failure": "dns_nxdomain_error", + "hostname": "antani.ooni.org", + "query_type": "AAAA", + "resolver_address": "", + "t": 0.072963834, + "started": 0.000125417, + "oddity": "dns.lookup.nxdomain" + } + ] +} +``` + +So we see a failure that says there was indeed an NXDOMAIN +error and we also see a field named `oddity`. + +What is an oddity? We define oddity something unexpected thay +may be explained by censorship as well as by a transient failure +or other normal network conditions. (In this case, the result +is perfectly normal since we're looking up a nonexistent domain.) + +The difference between failure and oddity is that the failure +indicates the error that occurred, while the oddity classifies +the error in the context of the operation during which it +occurred. (In this case the difference is subtle, but we'll +have a better example later, when we'll see what happens on timeout.) + +Failures are specified in +[df-007-errors](https://github.com/ooni/spec/blob/master/data-formats/df-007-errors.md). +Inside the `internal/netxlite/errorsx` +package, there is code that maps Go errors to failures. (The +`netxlite` package is the fundamental network package we use, on +top of which `measurex` is written.) + +### Measurement with timeout + +Let us now try with an insanely low timeout: + +```bash +go run -race ./internal/tutorial/measurex/chapter01 -timeout 250us +``` + +To get this JSON: + +```JSON +{ + "domain": "example.com", + "lookup_host": [ + { + "answers": null, + "engine": "system", + "failure": "generic_timeout_error", + "hostname": "example.com", + "query_type": "A", + "resolver_address": "", + "t": 0.000489167, + "started": 9.2583e-05, + "oddity": "dns.lookup.timeout" + }, + { + "answers": null, + "engine": "system", + "failure": "generic_timeout_error", + "hostname": "example.com", + "query_type": "AAAA", + "resolver_address": "", + "t": 0.000489167, + "started": 9.2583e-05, + "oddity": "dns.lookup.timeout" + } + ] +} +``` + +You should now better see the difference between a failure and +an oddity. The context timeout maps to a `generic_timeout_error` while +the oddity clearly indicates the timeout happens during a DNS +lookup. As we mentioned above, the failure is just an error while +an oddity is an error put in context. + +## Conclusions + +This is it. We have seen how to measure with the system resolver and we have +also seen which easy-to-provoke errors we can get. + diff --git a/internal/tutorial/measurex/chapter01/main.go b/internal/tutorial/measurex/chapter01/main.go new file mode 100644 index 0000000..1c99f9c --- /dev/null +++ b/internal/tutorial/measurex/chapter01/main.go @@ -0,0 +1,368 @@ +// -=-=- StartHere -=-=- +// +// # Chapter I: using the system resolver +// +// In this chapter we explain how to measure DNS resolutions performed +// using the system resolver. *En passant*, we will also introduce you to +// the `Measurer`, which we will use for the rest of the tutorial. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter01/main.go`.) +// +// ## The system resolver +// +// We define "system resolver" as the DNS resolver implemented by the C +// library. On Unix, the most popular interface to such a resolver is +// the `getaddrinfo(3)` C library function. +// +// Most OONI experiments (also known as nettests) use the system +// resolver to map domain names to IP addresses. The advantage of +// the system resolver is that it's provided by the system. So, +// it should _generally_ work. Also, it is the resolver that the +// user of the system will use every day, therefore its results +// should be representative (even though the rise of DNS over +// HTTPS embedded in browsers may make this statement less solid +// than it were ten years ago). +// +// The disadvantage of the system resolver is that we do not +// know how it is configured. Say the user has configured a +// DNS over TLS resolver; then the measurements may miss censorship +// that we would otherwise see if using a custom DNS resolver. +// +// Now that we have justified why the system resolver is +// important for OONI, let us perform some measurements with it. +// +// We will first write a simple `main.go` file that shows how to use +// this functionality. Then, we will show some runs of this file, and +// we will comment the output that we see. +// +// ## main.go +// +// We declare the package and import useful packages. The most +// important package we're importing here is, of course, `internal/measurex`. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + // ``` + // ### Setup + // + // We define command line flags useful to test this program. We use + // the `flags` package for that. We want the user to be able to configure + // both the domain name to resolve and the resolution timeout. + // + // ```Go + domain := flag.String("domain", "example.com", "domain to resolve") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + // ``` + // + // We call `flag.Parse` to parse the CLI flags. + // + // ```Go + flag.Parse() + // ``` + // + // We create a context and we attach a timeout to it. (This is a pretty + // standard way of configuring a timeout in Go.) + // + // ```Go + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + // ``` + // + // ### Creating a Measurer + // + // Now we create a `Measurer`. + // + // ```Go + mx := measurex.NewMeasurerWithDefaultSettings() + // ``` + // + // The `Measurer` is a concrete type that contains many fields + // requiring initialization. For this reason, we provide a factory + // that creates one with default settings. The expected usage + // pattern is that you do not modify a `Measurer`'s field after + // initialization. Modifying them while the `Measurer` is in + // use could, in fact, lead to races. + // + // Let's now invoke the system resolver to resolve `*domain`! + // + // ### Invoking the system resolver + // + // We call the `LookupHostSystem` method of the `Measurer`. The + // arguments are the Context, that in this case carries the timeout + // we configured above, and the domain to resolve. + // + // The call itself is named `LookupHost` because this is the name + // used by the Go function that performs a domain lookup. + // + // Under the hood, `mx.LookupHostSystem` will eventually call + // `(*net.Resolver).LookupHost`. In turn, in the common case on + // Unix, this function will eventually call `getaddrinfo(3)`. + // + // ```Go + m := mx.LookupHostSystem(ctx, *domain) + // ``` + // + // The return value of `(*net.Resolver).LookupHost` is either a + // list of IP addresses or an error. Our `LookupHostSystem` method, + // instead, returns a `*measurex.DNSMeasurement` type. + // + // This is probably a good moment to remind you of Go's + // built in help system. We could include a definition of the + // `DNSMeasurement` structure, but since this definition is + // just a comment in the main.go file, it might age badly. + // + // Instead, if you run + // + // ``` + // go doc ./internal/measurex.DNSMeasurement + // ``` + // + // You get the current definition. As you can see, this type + // is basically just a wrapper around `Measurement`. Now, + // checking the docs of `Measurement` with + // + // ``` + // go doc ./internal/measurex.Measurement + // ``` + // + // we can see a container of events + // classified by event type. In our case, because we're + // doing a `LookupHost`, we should have at least one entry + // inside of the `Measurement.LookupHost` field. + // + // This entry is of type `DNSLookupEvent`. Let us check + // together the definition of this type: + // + // ``` + // go doc ./internal/measurex.DNSLookupEvent + // ``` + // + // If you are familiar with [the OONI data format specs]( + // https://github.com/ooni/spec/tree/master/data-formats), you + // should probably recognize that this structure is the Go + // representation of the `df-002-dnst` data format. + // + // In fact, every event field inside of a `Measurement` + // should serialize nicely to JSON to one of the OONI data + // formats. + // + // ### Printing the measurement + // + // Because there is a close relationship between the + // events inside a `Measurement` and the JSON OONI data + // format, in the remainder of this program we're + // going to serialize the `Measurement` to JSON and + // print it to the standard output. + // + // ```Go + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) + // ``` + // + // As a final note, the `PanicOnError` is here because the + // message `m` *can* be marshalled to JSON. It still feels a + // bit better having an assertion for our assumptions than + // outrightly ignoring the error code. (We tend to use such + // a convention quite frequently in the OONI codebase.) + // + // ```Go +} + +// ``` +// +// ## Running the example program +// +// Let us run the program with default arguments first. You can do +// this operation by running: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter01 +// ``` +// +// If you do that you obtain some logging messages, which are out of +// the scope of this tutorial, and the following JSON: +// +// ```JSON +// { +// "domain": "example.com", +// "lookup_host": [ +// { +// "answers": [ +// { +// "answer_type": "A", +// "ipv4": "93.184.216.34" +// } +// ], +// "engine": "system", +// "failure": null, +// "hostname": "example.com", +// "query_type": "A", +// "resolver_address": "", +// "t": 0.002996459, +// "started": 9.8e-05, +// "oddity": "" +// }, +// { +// "answers": [ +// { +// "answer_type": "AAAA", +// "ivp6": "2606:2800:220:1:248:1893:25c8:1946" +// } +// ], +// "engine": "system", +// "failure": null, +// "hostname": "example.com", +// "query_type": "AAAA", +// "resolver_address": "", +// "t": 0.002996459, +// "started": 9.8e-05, +// "oddity": "" +// } +// ] +// } +// ``` +// +// You see that we have two messages here. OONI splits a DNS +// resolution performed using the system resolver into two "fake" +// DNS resolutions for A and AAAA. (Under the hood, this is +// what the system resolver would most likely do.) +// +// The most important fields are: +// +// - _engine_, indicating that we are using the "system" resolver; +// +// - _hostname_, meaning that we wanted to resolve the "example.com" domain; +// +// - _answers_, which contains a list of answers; +// +// - _t_, which is the time when the LookupHost operation completed. +// +// ### NXDOMAIN measurement +// +// Let us now change the domain to resolve to be `antani.ooni.org` (a +// nonexisting domain), which we can do by running this command: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter01 -domain antani.ooni.org +// ``` +// +// This is the output JSON: +// +// ```JSON +// { +// "domain": "antani.ooni.org", +// "lookup_host": [ +// { +// "answers": null, +// "engine": "system", +// "failure": "dns_nxdomain_error", +// "hostname": "antani.ooni.org", +// "query_type": "A", +// "resolver_address": "", +// "t": 0.072963834, +// "started": 0.000125417, +// "oddity": "dns.lookup.nxdomain" +// }, +// { +// "answers": null, +// "engine": "system", +// "failure": "dns_nxdomain_error", +// "hostname": "antani.ooni.org", +// "query_type": "AAAA", +// "resolver_address": "", +// "t": 0.072963834, +// "started": 0.000125417, +// "oddity": "dns.lookup.nxdomain" +// } +// ] +// } +// ``` +// +// So we see a failure that says there was indeed an NXDOMAIN +// error and we also see a field named `oddity`. +// +// What is an oddity? We define oddity something unexpected thay +// may be explained by censorship as well as by a transient failure +// or other normal network conditions. (In this case, the result +// is perfectly normal since we're looking up a nonexistent domain.) +// +// The difference between failure and oddity is that the failure +// indicates the error that occurred, while the oddity classifies +// the error in the context of the operation during which it +// occurred. (In this case the difference is subtle, but we'll +// have a better example later, when we'll see what happens on timeout.) +// +// Failures are specified in +// [df-007-errors](https://github.com/ooni/spec/blob/master/data-formats/df-007-errors.md). +// Inside the `internal/netxlite/errorsx` +// package, there is code that maps Go errors to failures. (The +// `netxlite` package is the fundamental network package we use, on +// top of which `measurex` is written.) +// +// ### Measurement with timeout +// +// Let us now try with an insanely low timeout: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter01 -timeout 250us +// ``` +// +// To get this JSON: +// +// ```JSON +// { +// "domain": "example.com", +// "lookup_host": [ +// { +// "answers": null, +// "engine": "system", +// "failure": "generic_timeout_error", +// "hostname": "example.com", +// "query_type": "A", +// "resolver_address": "", +// "t": 0.000489167, +// "started": 9.2583e-05, +// "oddity": "dns.lookup.timeout" +// }, +// { +// "answers": null, +// "engine": "system", +// "failure": "generic_timeout_error", +// "hostname": "example.com", +// "query_type": "AAAA", +// "resolver_address": "", +// "t": 0.000489167, +// "started": 9.2583e-05, +// "oddity": "dns.lookup.timeout" +// } +// ] +// } +// ``` +// +// You should now better see the difference between a failure and +// an oddity. The context timeout maps to a `generic_timeout_error` while +// the oddity clearly indicates the timeout happens during a DNS +// lookup. As we mentioned above, the failure is just an error while +// an oddity is an error put in context. +// +// ## Conclusions +// +// This is it. We have seen how to measure with the system resolver and we have +// also seen which easy-to-provoke errors we can get. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter02/README.md b/internal/tutorial/measurex/chapter02/README.md new file mode 100644 index 0000000..ed661bf --- /dev/null +++ b/internal/tutorial/measurex/chapter02/README.md @@ -0,0 +1,228 @@ + +# Chapter II: establishing TCP connections + +In this chapter we explain how to measure establishing TCP connections. + +We will first write a simple `main.go` file that shows how to use +this functionality. Then, we will show some runs of this file, and +we will comment the output that we see. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter02/main.go`.) + +## main.go + +We declare the package and import useful packages. The most +important package we're importing here is, of course, `internal/measurex`. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { +``` +### Setup + +This first part of `main.go` is really similar to the previous +chapter, so there is not much new to say here. + +```Go + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() +``` + +### Creating a Measurer + +We create a `Measurer` like we did in the previous chapter. + +```Go + mx := measurex.NewMeasurerWithDefaultSettings() +``` + +### Establishing a TCP connection + +We then call `TCPConnect`, which establishes a connection +and returns the corresponding measurement. + +The arguments are the context (for timeouts), and the address +of the endpoint to which we want to connect. (Here and in +most of this tutorial with "endpoint" we mean an IP address +and a port, serialized as "ADDRESS:PORT", where the +address is quoted with "[" and "]" if IPv6, e.g., `[::1]:53`.) + +```Go + m := mx.TCPConnect(ctx, *address) +``` + +### Printing the measurement + +The rest of the main function is just like in the previous chapter. + +```Go + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +``` + +## Running the example program + +Let us run the program with default arguments first. You can do +this operation by running: + +```bash +go run -race ./internal/tutorial/measurex/chapter02 +``` + +Here is the JSON we obtain in output: + +```JavaScript +{ + // These two fields identify the endpoint + "network": "tcp", + "address": "8.8.4.4:443", + + // This block contains the results of the connect syscall + // using the df-008-netevents data format. + "connect": [ + { + "address": "8.8.4.4:443", + "failure": null, + "operation": "connect", + "proto": "tcp", + "t": 0.026879041, + "started": 8.8625e-05, + "oddity": "" + } + ] +} +``` + +This is what it says: + +- we are connecting a "tcp" socket; + +- the destination endpoint address is "8.8.4.4:443"; + +- connect terminated ~0.027 seconds into the program's life; + +- the operation succeeded (`failure` is `nil`). + +Let us now see if we can provoke some errors and timeouts. + +### Measurement with connection refused + +Let us start with an IP address where there's no listening socket: + +```bash +go run -race ./internal/tutorial/measurex/chapter02 -address 127.0.0.1:1 +``` + +We get this JSON: + +```JSON +{ + "network": "tcp", + "address": "127.0.0.1:1", + "connect": [ + { + "address": "127.0.0.1:1", + "failure": "connection_refused", + "operation": "connect", + "proto": "tcp", + "t": 0.000372167, + "started": 8.4917e-05, + "oddity": "tcp.connect.refused" + } + ] +} + +``` + +And here's an error telling us the connection was refused and +the oddity that classifies the error. + +### Measurement with timeouts + +Let us now try to obtain a timeout: + +```bash +go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1 +``` + +We get this JSON: + +```JSON +{ + "network": "tcp", + "address": "8.8.4.4:1", + "connect": [ + { + "address": "8.8.4.4:1", + "failure": "generic_timeout_error", + "operation": "connect", + "proto": "tcp", + "t": 10.005494583, + "started": 8.4833e-05, + "oddity": "tcp.connect.timeout" + } + ] +} +``` + +So, we clearly see from the value of `t` that our 60 seconds +default timeout did not hit, because there is a lower watchdog +timeout (10 s). We also see again how the oddity is more +precise than just the error alone. + +Let us now use a very small timeout: + +```bash +go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1 -timeout 100ms +``` + +To get this JSON: + +```JSON +{ + "network": "tcp", + "address": "8.8.4.4:1", + "connect": [ + { + "address": "8.8.4.4:1", + "failure": "generic_timeout_error", + "operation": "connect", + "proto": "tcp", + "t": 0.10148025, + "started": 0.000122375, + "oddity": "tcp.connect.timeout" + } + ] +} +``` + +We see a timeout after ~0.1s. We enforce a reasonably small +timeout for connecting, equal to 10 s, because we want to +guarantee that measurements eventually terminate. Also, since +often censorship is implemented by timing out, we don't want +to spend to much time waiting for a timeout to expire. + +## Conclusions + +We have seen how to measure the operation of connecting +to a specific TCP endpoint. + diff --git a/internal/tutorial/measurex/chapter02/main.go b/internal/tutorial/measurex/chapter02/main.go new file mode 100644 index 0000000..0d19971 --- /dev/null +++ b/internal/tutorial/measurex/chapter02/main.go @@ -0,0 +1,230 @@ +// -=-=- StartHere -=-=- +// +// # Chapter II: establishing TCP connections +// +// In this chapter we explain how to measure establishing TCP connections. +// +// We will first write a simple `main.go` file that shows how to use +// this functionality. Then, we will show some runs of this file, and +// we will comment the output that we see. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter02/main.go`.) +// +// ## main.go +// +// We declare the package and import useful packages. The most +// important package we're importing here is, of course, `internal/measurex`. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + // ``` + // ### Setup + // + // This first part of `main.go` is really similar to the previous + // chapter, so there is not much new to say here. + // + // ```Go + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + // ``` + // + // ### Creating a Measurer + // + // We create a `Measurer` like we did in the previous chapter. + // + // ```Go + mx := measurex.NewMeasurerWithDefaultSettings() + // ``` + // + // ### Establishing a TCP connection + // + // We then call `TCPConnect`, which establishes a connection + // and returns the corresponding measurement. + // + // The arguments are the context (for timeouts), and the address + // of the endpoint to which we want to connect. (Here and in + // most of this tutorial with "endpoint" we mean an IP address + // and a port, serialized as "ADDRESS:PORT", where the + // address is quoted with "[" and "]" if IPv6, e.g., `[::1]:53`.) + // + // ```Go + m := mx.TCPConnect(ctx, *address) + // ``` + // + // ### Printing the measurement + // + // The rest of the main function is just like in the previous chapter. + // + // ```Go + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +// ``` +// +// ## Running the example program +// +// Let us run the program with default arguments first. You can do +// this operation by running: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter02 +// ``` +// +// Here is the JSON we obtain in output: +// +// ```JavaScript +// { +// // These two fields identify the endpoint +// "network": "tcp", +// "address": "8.8.4.4:443", +// +// // This block contains the results of the connect syscall +// // using the df-008-netevents data format. +// "connect": [ +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "operation": "connect", +// "proto": "tcp", +// "t": 0.026879041, +// "started": 8.8625e-05, +// "oddity": "" +// } +// ] +// } +// ``` +// +// This is what it says: +// +// - we are connecting a "tcp" socket; +// +// - the destination endpoint address is "8.8.4.4:443"; +// +// - connect terminated ~0.027 seconds into the program's life; +// +// - the operation succeeded (`failure` is `nil`). +// +// Let us now see if we can provoke some errors and timeouts. +// +// ### Measurement with connection refused +// +// Let us start with an IP address where there's no listening socket: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter02 -address 127.0.0.1:1 +// ``` +// +// We get this JSON: +// +// ```JSON +// { +// "network": "tcp", +// "address": "127.0.0.1:1", +// "connect": [ +// { +// "address": "127.0.0.1:1", +// "failure": "connection_refused", +// "operation": "connect", +// "proto": "tcp", +// "t": 0.000372167, +// "started": 8.4917e-05, +// "oddity": "tcp.connect.refused" +// } +// ] +// } +// +// ``` +// +// And here's an error telling us the connection was refused and +// the oddity that classifies the error. +// +// ### Measurement with timeouts +// +// Let us now try to obtain a timeout: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1 +// ``` +// +// We get this JSON: +// +// ```JSON +// { +// "network": "tcp", +// "address": "8.8.4.4:1", +// "connect": [ +// { +// "address": "8.8.4.4:1", +// "failure": "generic_timeout_error", +// "operation": "connect", +// "proto": "tcp", +// "t": 10.005494583, +// "started": 8.4833e-05, +// "oddity": "tcp.connect.timeout" +// } +// ] +// } +// ``` +// +// So, we clearly see from the value of `t` that our 60 seconds +// default timeout did not hit, because there is a lower watchdog +// timeout (10 s). We also see again how the oddity is more +// precise than just the error alone. +// +// Let us now use a very small timeout: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter02 -address 8.8.4.4:1 -timeout 100ms +// ``` +// +// To get this JSON: +// +// ```JSON +// { +// "network": "tcp", +// "address": "8.8.4.4:1", +// "connect": [ +// { +// "address": "8.8.4.4:1", +// "failure": "generic_timeout_error", +// "operation": "connect", +// "proto": "tcp", +// "t": 0.10148025, +// "started": 0.000122375, +// "oddity": "tcp.connect.timeout" +// } +// ] +// } +// ``` +// +// We see a timeout after ~0.1s. We enforce a reasonably small +// timeout for connecting, equal to 10 s, because we want to +// guarantee that measurements eventually terminate. Also, since +// often censorship is implemented by timing out, we don't want +// to spend to much time waiting for a timeout to expire. +// +// ## Conclusions +// +// We have seen how to measure the operation of connecting +// to a specific TCP endpoint. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter03/README.md b/internal/tutorial/measurex/chapter03/README.md new file mode 100644 index 0000000..1c7e4f3 --- /dev/null +++ b/internal/tutorial/measurex/chapter03/README.md @@ -0,0 +1,567 @@ + +# Chapter III: using a custom DNS-over-UDP resolver + +In this chapter we learn how to measure sending DNS queries to +a DNS server speaking the DNS-over-UDP protocol. + +Without further ado, let's describe our example `main.go` program +and let's use it to better understand this flow. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter03/main.go`.) + +## main.go + +The initial part of the program is pretty much the same as the one +used in previous chapters, so I will not add further comments. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + query := flag.String("query", "example.com", "domain to resolver") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() +``` + +### Using a custom UDP resolver + +We now invoke `LookupHostUDP`. We specify: + +- a context for timeout information; + +- the domain to query for; + +- the address of the DNS-over-UDP server endpoint. + +```Go + m := mx.LookupHostUDP(ctx, *query, *address) +``` + +Also this operation returns a measurement, which +we print using the usual three-liner. + +```Go + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +``` + +## Running the example program + +As before, let us start off with a vanilla run: + +```bash +go run -race ./internal/tutorial/measurex/chapter03 +``` + +This time we get a much larger JSON, so I will pretend it is +actually JavaScript and add comments to explain it inline. + +(This is the first case in which we see how a single +method call for measurer causes several events to +be generated and inserted into a `Measurement`.) + +```JavaScript +{ + "domain": "example.com", + + // This block tells us about the UDP connect events + // where we bind to the server's endpoint + "connect": [ + { + "address": "8.8.4.4:53", + "failure": null, + "operation": "connect", + "proto": "udp", + "t": 0.00043175, + "started": 0.000191958, + "oddity": "" + }, + { + "address": "8.8.4.4:53", + "failure": null, + "operation": "connect", + "proto": "udp", + "t": 0.042198458, + "started": 0.042113208, + "oddity": "" + } + ], + + // This block shows the read and write events + // occurred on the sockets (because we control + // in full the implementation of this DNS + // over UDP resolver, we can see these events) + "read_write": [ + { + "address": "8.8.4.4:53", + "failure": null, + "num_bytes": 29, + "operation": "write", + "proto": "udp", + "t": 0.000459583, + "started": 0.00043825, + "oddity": "" + }, + { + "address": "8.8.4.4:53", + "failure": null, + "num_bytes": 45, + "operation": "read", + "proto": "udp", + "t": 0.041955792, + "started": 0.000471833, + "oddity": "" + }, + { + "address": "8.8.4.4:53", + "failure": null, + "num_bytes": 29, + "operation": "write", + "proto": "udp", + "t": 0.042218917, + "started": 0.042203, + "oddity": "" + }, + { + "address": "8.8.4.4:53", + "failure": null, + "num_bytes": 57, + "operation": "read", + "proto": "udp", + "t": 0.196646583, + "started": 0.042233167, + "oddity": "" + } + ], + + // This is the same kind of result as before, we + // show the emitted queries and the resolved addrs. + // + // Also note how here the resolver_address is the + // correct endpoint address and the engine tells us + // that we're using DNS over UDP. + "lookup_host": [ + { + "answers": [ + { + "answer_type": "A", + "ipv4": "93.184.216.34" + } + ], + "engine": "udp", + "failure": null, + "hostname": "example.com", + "query_type": "A", + "resolver_address": "8.8.4.4:53", + "t": 0.196777042, + "started": 0.000118542, + "oddity": "" + }, + { + "answers": [ + { + "answer_type": "AAAA", + "ivp6": "2606:2800:220:1:248:1893:25c8:1946" + } + ], + "engine": "udp", + "failure": null, + "hostname": "example.com", + "query_type": "AAAA", + "resolver_address": "8.8.4.4:53", + "t": 0.196777042, + "started": 0.000118542, + "oddity": "" + } + ], + + // This block shows the query we sent (encoded as base64) + // and the response we received. Here we clearly see + // that we perform two DNS "round trip" (i.e., send request + // and receive response) to resolve a domain: one for + // A and the other for AAAA. + "dns_round_trip": [ + { + "engine": "udp", + "resolver_address": "8.8.4.4:53", + "raw_query": { + "data": "PrcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", + "format": "base64" + }, + "started": 0.000191625, + "t": 0.041998667, + "failure": null, + "raw_reply": { + "data": "PreBgAABAAEAAAAAB2V4YW1wbGUDY29tAAABAAHADAABAAEAAE8BAARduNgi", + "format": "base64" + } + }, + { + "engine": "udp", + "resolver_address": "8.8.4.4:53", + "raw_query": { + "data": "LAwBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=", + "format": "base64" + }, + "started": 0.04210775, + "t": 0.196701333, + "failure": null, + "raw_reply": { + "data": "LAyBgAABAAEAAAAAB2V4YW1wbGUDY29tAAAcAAHADAAcAAEAAE6nABAmBigAAiAAAQJIGJMlyBlG", + "format": "base64" + } + } + ] +} +``` + +This data format is really an extension of the `LookupHostSystem` +one. It just adds more fields that clarify what happened at low +level in terms of socket I/O and queries sent and received. + +Let us now try to provoke some errors and see how the +output JSON changes because of them. + +### Measurement with NXDOMAIN + +Let us try to get a NXDOMAIN error. + +```bash +go run -race ./internal/tutorial/measurex/chapter03 -query antani.ooni.org +``` + +This produces the following JSON: + +```JavaScript +{ + "domain": "antani.ooni.org", + "connect": [ /* snip */ ], + "read_write": [ /* snip */ ], + "lookup_host": [ + { + "answers": null, + "engine": "udp", + "failure": "dns_nxdomain_error", + "hostname": "antani.ooni.org", + "query_type": "A", + "resolver_address": "8.8.4.4:53", + "t": 0.098208709, + "started": 8.95e-05, + "oddity": "dns.lookup.nxdomain" + }, + { + "answers": null, + "engine": "udp", + "failure": "dns_nxdomain_error", + "hostname": "antani.ooni.org", + "query_type": "AAAA", + "resolver_address": "8.8.4.4:53", + "t": 0.098208709, + "started": 8.95e-05, + "oddity": "dns.lookup.nxdomain" + } + ], + "dns_round_trip": [ + { + "engine": "udp", + "resolver_address": "8.8.4.4:53", + "raw_query": { + "data": "jLIBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAAQAB", + "format": "base64" + }, + "started": 0.000141542, + "t": 0.034689417, + "failure": null, + "raw_reply": { + "data": "jLKBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAAQABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==", + "format": "base64" + } + }, + { + "engine": "udp", + "resolver_address": "8.8.4.4:53", + "raw_query": { + "data": "azEBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAHAAB", + "format": "base64" + }, + "started": 0.034776709, + "t": 0.098170542, + "failure": null, + "raw_reply": { + "data": "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==", + "format": "base64" + } + } + ] +} +``` + +We indeed get a NXDOMAIN error as the failure in `lookup_host`. + +Let us now decode one of the replies by using this program: + +``` +package main + +import ( +"fmt" +"encoding/base64" + +"github.com/miekg/dns" +) + +func main() { + const query = "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==" + data, _ := base64.StdEncoding.DecodeString(query) + msg := new(dns.Msg) + _ = msg.Unpack(data) + fmt.Printf("%s\n", msg) +} +``` + +where `query` is one of the replies. If we run this program +we get as the output: + +``` +;; opcode: QUERY, status: NXDOMAIN, id: 27441 +;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 + +;; QUESTION SECTION: +;antani.ooni.org. IN AAAA + +;; AUTHORITY SECTION: +ooni.org. 1800 IN SOA dns1.registrar-servers.com. hostmaster.registrar-servers.com. 1627397372 43200 3600 604800 3601 +``` + +### Measurement with timeout + +Let us now query an IP address known for not responding +to DNS queries, to get a timeout. + +```bash +go run -race ./internal/tutorial/measurex/chapter03 -address 182.92.22.222:53 +``` + +Here's the corresponding JSON: + +```JavaScript +{ + "domain": "example.com", + "connect": [ /* snip */ ], + "read_write": [ + { + "address": "182.92.22.222:53", + "failure": null, + "num_bytes": 29, + "operation": "write", + "proto": "udp", + "t": 0.0005275, + "started": 0.000500209, + "oddity": "" + }, + { + "address": "182.92.22.222:53", + "failure": "generic_timeout_error", /* <--- */ + "operation": "read", + "proto": "udp", + "t": 5.001140125, + "started": 0.000544042, + "oddity": "" + } + ], + "lookup_host": [ + { + "answers": null, + "engine": "udp", + "failure": "generic_timeout_error", /* <--- */ + "hostname": "example.com", + "query_type": "A", + "resolver_address": "182.92.22.222:53", + "t": 5.001462084, + "started": 0.000127917, + "oddity": "dns.lookup.timeout" /* <--- */ + }, + { + "answers": null, + "engine": "udp", + "failure": "generic_timeout_error", + "hostname": "example.com", + "query_type": "AAAA", + "resolver_address": "182.92.22.222:53", + "t": 5.001462084, + "started": 0.000127917, + "oddity": "dns.lookup.timeout" + } + ], + "dns_round_trip": [ + { + "engine": "udp", + "resolver_address": "182.92.22.222:53", + "raw_query": { + "data": "ej8BAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", + "format": "base64" + }, + "started": 0.000220584, + "t": 5.001317417, + "failure": "generic_timeout_error", + "raw_reply": null + } + ] +} +``` + +We see that we do fail with a timeout (I have marked some of them +with comments inside the JSON). We see the timeout at three different +level of abstractions (from lower to higher abstraction): at the socket layer, +during the DNS round trip, during the DNS lookup. + +What we also see is that `t`'s value is ~5s when the `read` event +fails, which tells us about the socket's read timeout. + +### Measurement with REFUSED error + +Let us now try to get a REFUSED DNS Rcode, again from servers +that are, let's say, kind enough to easily help. + +```bash +go run -race ./internal/tutorial/measurex/chapter03 -address 180.97.36.63:53 +``` + +Here's the answer I get: + +```JavaScript +{ + "domain": "example.com", + "connect": [ /* snip */ ], + + // The I/O events look normal this time + "read_write": [ + { + "address": "180.97.36.63:53", + "failure": null, + "num_bytes": 29, + "operation": "write", + "proto": "udp", + "t": 0.000333583, + "started": 0.000312125, + "oddity": "" + }, + { + "address": "180.97.36.63:53", + "failure": null, + "num_bytes": 29, + "operation": "read", + "proto": "udp", + "t": 0.334948125, + "started": 0.000366625, + "oddity": "" + }, + { + "address": "180.97.36.63:53", + "failure": null, + "num_bytes": 29, + "operation": "write", + "proto": "udp", + "t": 0.3358025, + "started": 0.335725958, + "oddity": "" + }, + { + "address": "180.97.36.63:53", + "failure": null, + "num_bytes": 29, + "operation": "read", + "proto": "udp", + "t": 0.739987666, + "started": 0.335863875, + "oddity": "" + } + ], + + // But we see both in the error and in the oddity + // that the response was "REFUSED" + "lookup_host": [ + { + "answers": null, + "engine": "udp", + "failure": "dns_refused_error", + "hostname": "example.com", + "query_type": "A", + "resolver_address": "180.97.36.63:53", + "t": 0.7402975, + "started": 7.2291e-05, + "oddity": "dns.lookup.refused" + }, + { + "answers": null, + "engine": "udp", + "failure": "dns_refused_error", + "hostname": "example.com", + "query_type": "AAAA", + "resolver_address": "180.97.36.63:53", + "t": 0.7402975, + "started": 7.2291e-05, + "oddity": "dns.lookup.refused" + } + ], + + // Exercise: do like I did before and decode the messages + "dns_round_trip": [ + { + "engine": "udp", + "resolver_address": "180.97.36.63:53", + "raw_query": { + "data": "crkBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", + "format": "base64" + }, + "started": 0.000130666, + "t": 0.33509925, + "failure": null, + "raw_reply": { + "data": "crmBBQABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", + "format": "base64" + } + }, + { + "engine": "udp", + "resolver_address": "180.97.36.63:53", + "raw_query": { + "data": "ywcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=", + "format": "base64" + }, + "started": 0.335321333, + "t": 0.740152375, + "failure": null, + "raw_reply": { + "data": "yweBBQABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=", + "format": "base64" + } + } + ] +} +``` + +## Conclusion + +We have seen how we sending DNS queries over UDP, measure the +results, and what happens on common error conditions. + diff --git a/internal/tutorial/measurex/chapter03/main.go b/internal/tutorial/measurex/chapter03/main.go new file mode 100644 index 0000000..8d000c4 --- /dev/null +++ b/internal/tutorial/measurex/chapter03/main.go @@ -0,0 +1,569 @@ +// -=-=- StartHere -=-=- +// +// # Chapter III: using a custom DNS-over-UDP resolver +// +// In this chapter we learn how to measure sending DNS queries to +// a DNS server speaking the DNS-over-UDP protocol. +// +// Without further ado, let's describe our example `main.go` program +// and let's use it to better understand this flow. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter03/main.go`.) +// +// ## main.go +// +// The initial part of the program is pretty much the same as the one +// used in previous chapters, so I will not add further comments. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + query := flag.String("query", "example.com", "domain to resolver") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() + // ``` + // + // ### Using a custom UDP resolver + // + // We now invoke `LookupHostUDP`. We specify: + // + // - a context for timeout information; + // + // - the domain to query for; + // + // - the address of the DNS-over-UDP server endpoint. + // + // ```Go + m := mx.LookupHostUDP(ctx, *query, *address) + // ``` + // + // Also this operation returns a measurement, which + // we print using the usual three-liner. + // + // ```Go + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +// ``` +// +// ## Running the example program +// +// As before, let us start off with a vanilla run: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter03 +// ``` +// +// This time we get a much larger JSON, so I will pretend it is +// actually JavaScript and add comments to explain it inline. +// +// (This is the first case in which we see how a single +// method call for measurer causes several events to +// be generated and inserted into a `Measurement`.) +// +// ```JavaScript +// { +// "domain": "example.com", +// +// // This block tells us about the UDP connect events +// // where we bind to the server's endpoint +// "connect": [ +// { +// "address": "8.8.4.4:53", +// "failure": null, +// "operation": "connect", +// "proto": "udp", +// "t": 0.00043175, +// "started": 0.000191958, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:53", +// "failure": null, +// "operation": "connect", +// "proto": "udp", +// "t": 0.042198458, +// "started": 0.042113208, +// "oddity": "" +// } +// ], +// +// // This block shows the read and write events +// // occurred on the sockets (because we control +// // in full the implementation of this DNS +// // over UDP resolver, we can see these events) +// "read_write": [ +// { +// "address": "8.8.4.4:53", +// "failure": null, +// "num_bytes": 29, +// "operation": "write", +// "proto": "udp", +// "t": 0.000459583, +// "started": 0.00043825, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:53", +// "failure": null, +// "num_bytes": 45, +// "operation": "read", +// "proto": "udp", +// "t": 0.041955792, +// "started": 0.000471833, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:53", +// "failure": null, +// "num_bytes": 29, +// "operation": "write", +// "proto": "udp", +// "t": 0.042218917, +// "started": 0.042203, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:53", +// "failure": null, +// "num_bytes": 57, +// "operation": "read", +// "proto": "udp", +// "t": 0.196646583, +// "started": 0.042233167, +// "oddity": "" +// } +// ], +// +// // This is the same kind of result as before, we +// // show the emitted queries and the resolved addrs. +// // +// // Also note how here the resolver_address is the +// // correct endpoint address and the engine tells us +// // that we're using DNS over UDP. +// "lookup_host": [ +// { +// "answers": [ +// { +// "answer_type": "A", +// "ipv4": "93.184.216.34" +// } +// ], +// "engine": "udp", +// "failure": null, +// "hostname": "example.com", +// "query_type": "A", +// "resolver_address": "8.8.4.4:53", +// "t": 0.196777042, +// "started": 0.000118542, +// "oddity": "" +// }, +// { +// "answers": [ +// { +// "answer_type": "AAAA", +// "ivp6": "2606:2800:220:1:248:1893:25c8:1946" +// } +// ], +// "engine": "udp", +// "failure": null, +// "hostname": "example.com", +// "query_type": "AAAA", +// "resolver_address": "8.8.4.4:53", +// "t": 0.196777042, +// "started": 0.000118542, +// "oddity": "" +// } +// ], +// +// // This block shows the query we sent (encoded as base64) +// // and the response we received. Here we clearly see +// // that we perform two DNS "round trip" (i.e., send request +// // and receive response) to resolve a domain: one for +// // A and the other for AAAA. +// "dns_round_trip": [ +// { +// "engine": "udp", +// "resolver_address": "8.8.4.4:53", +// "raw_query": { +// "data": "PrcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", +// "format": "base64" +// }, +// "started": 0.000191625, +// "t": 0.041998667, +// "failure": null, +// "raw_reply": { +// "data": "PreBgAABAAEAAAAAB2V4YW1wbGUDY29tAAABAAHADAABAAEAAE8BAARduNgi", +// "format": "base64" +// } +// }, +// { +// "engine": "udp", +// "resolver_address": "8.8.4.4:53", +// "raw_query": { +// "data": "LAwBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=", +// "format": "base64" +// }, +// "started": 0.04210775, +// "t": 0.196701333, +// "failure": null, +// "raw_reply": { +// "data": "LAyBgAABAAEAAAAAB2V4YW1wbGUDY29tAAAcAAHADAAcAAEAAE6nABAmBigAAiAAAQJIGJMlyBlG", +// "format": "base64" +// } +// } +// ] +// } +// ``` +// +// This data format is really an extension of the `LookupHostSystem` +// one. It just adds more fields that clarify what happened at low +// level in terms of socket I/O and queries sent and received. +// +// Let us now try to provoke some errors and see how the +// output JSON changes because of them. +// +// ### Measurement with NXDOMAIN +// +// Let us try to get a NXDOMAIN error. +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter03 -query antani.ooni.org +// ``` +// +// This produces the following JSON: +// +// ```JavaScript +// { +// "domain": "antani.ooni.org", +// "connect": [ /* snip */ ], +// "read_write": [ /* snip */ ], +// "lookup_host": [ +// { +// "answers": null, +// "engine": "udp", +// "failure": "dns_nxdomain_error", +// "hostname": "antani.ooni.org", +// "query_type": "A", +// "resolver_address": "8.8.4.4:53", +// "t": 0.098208709, +// "started": 8.95e-05, +// "oddity": "dns.lookup.nxdomain" +// }, +// { +// "answers": null, +// "engine": "udp", +// "failure": "dns_nxdomain_error", +// "hostname": "antani.ooni.org", +// "query_type": "AAAA", +// "resolver_address": "8.8.4.4:53", +// "t": 0.098208709, +// "started": 8.95e-05, +// "oddity": "dns.lookup.nxdomain" +// } +// ], +// "dns_round_trip": [ +// { +// "engine": "udp", +// "resolver_address": "8.8.4.4:53", +// "raw_query": { +// "data": "jLIBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAAQAB", +// "format": "base64" +// }, +// "started": 0.000141542, +// "t": 0.034689417, +// "failure": null, +// "raw_reply": { +// "data": "jLKBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAAQABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==", +// "format": "base64" +// } +// }, +// { +// "engine": "udp", +// "resolver_address": "8.8.4.4:53", +// "raw_query": { +// "data": "azEBAAABAAAAAAAABmFudGFuaQRvb25pA29yZwAAHAAB", +// "format": "base64" +// }, +// "started": 0.034776709, +// "t": 0.098170542, +// "failure": null, +// "raw_reply": { +// "data": "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==", +// "format": "base64" +// } +// } +// ] +// } +// ``` +// +// We indeed get a NXDOMAIN error as the failure in `lookup_host`. +// +// Let us now decode one of the replies by using this program: +// +// ``` +// package main +// +// import ( +// "fmt" +// "encoding/base64" +// +// "github.com/miekg/dns" +// ) +// +// func main() { +// const query = "azGBgwABAAAAAQAABmFudGFuaQRvb25pA29yZwAAHAABwBMABgABAAAHCAA9BGRuczERcmVnaXN0cmFyLXNlcnZlcnMDY29tAApob3N0bWFzdGVywDJhABz8AACowAAADhAACTqAAAAOEQ==" +// data, _ := base64.StdEncoding.DecodeString(query) +// msg := new(dns.Msg) +// _ = msg.Unpack(data) +// fmt.Printf("%s\n", msg) +//} +// ``` +// +// where `query` is one of the replies. If we run this program +// we get as the output: +// +// ``` +// ;; opcode: QUERY, status: NXDOMAIN, id: 27441 +// ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 +// +// ;; QUESTION SECTION: +// ;antani.ooni.org. IN AAAA +// +// ;; AUTHORITY SECTION: +// ooni.org. 1800 IN SOA dns1.registrar-servers.com. hostmaster.registrar-servers.com. 1627397372 43200 3600 604800 3601 +// ``` +// +// ### Measurement with timeout +// +// Let us now query an IP address known for not responding +// to DNS queries, to get a timeout. +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter03 -address 182.92.22.222:53 +// ``` +// +// Here's the corresponding JSON: +// +// ```JavaScript +// { +// "domain": "example.com", +// "connect": [ /* snip */ ], +// "read_write": [ +// { +// "address": "182.92.22.222:53", +// "failure": null, +// "num_bytes": 29, +// "operation": "write", +// "proto": "udp", +// "t": 0.0005275, +// "started": 0.000500209, +// "oddity": "" +// }, +// { +// "address": "182.92.22.222:53", +// "failure": "generic_timeout_error", /* <--- */ +// "operation": "read", +// "proto": "udp", +// "t": 5.001140125, +// "started": 0.000544042, +// "oddity": "" +// } +// ], +// "lookup_host": [ +// { +// "answers": null, +// "engine": "udp", +// "failure": "generic_timeout_error", /* <--- */ +// "hostname": "example.com", +// "query_type": "A", +// "resolver_address": "182.92.22.222:53", +// "t": 5.001462084, +// "started": 0.000127917, +// "oddity": "dns.lookup.timeout" /* <--- */ +// }, +// { +// "answers": null, +// "engine": "udp", +// "failure": "generic_timeout_error", +// "hostname": "example.com", +// "query_type": "AAAA", +// "resolver_address": "182.92.22.222:53", +// "t": 5.001462084, +// "started": 0.000127917, +// "oddity": "dns.lookup.timeout" +// } +// ], +// "dns_round_trip": [ +// { +// "engine": "udp", +// "resolver_address": "182.92.22.222:53", +// "raw_query": { +// "data": "ej8BAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", +// "format": "base64" +// }, +// "started": 0.000220584, +// "t": 5.001317417, +// "failure": "generic_timeout_error", +// "raw_reply": null +// } +// ] +// } +// ``` +// +// We see that we do fail with a timeout (I have marked some of them +// with comments inside the JSON). We see the timeout at three different +// level of abstractions (from lower to higher abstraction): at the socket layer, +// during the DNS round trip, during the DNS lookup. +// +// What we also see is that `t`'s value is ~5s when the `read` event +// fails, which tells us about the socket's read timeout. +// +// ### Measurement with REFUSED error +// +// Let us now try to get a REFUSED DNS Rcode, again from servers +// that are, let's say, kind enough to easily help. +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter03 -address 180.97.36.63:53 +// ``` +// +// Here's the answer I get: +// +// ```JavaScript +// { +// "domain": "example.com", +// "connect": [ /* snip */ ], +// +// // The I/O events look normal this time +// "read_write": [ +// { +// "address": "180.97.36.63:53", +// "failure": null, +// "num_bytes": 29, +// "operation": "write", +// "proto": "udp", +// "t": 0.000333583, +// "started": 0.000312125, +// "oddity": "" +// }, +// { +// "address": "180.97.36.63:53", +// "failure": null, +// "num_bytes": 29, +// "operation": "read", +// "proto": "udp", +// "t": 0.334948125, +// "started": 0.000366625, +// "oddity": "" +// }, +// { +// "address": "180.97.36.63:53", +// "failure": null, +// "num_bytes": 29, +// "operation": "write", +// "proto": "udp", +// "t": 0.3358025, +// "started": 0.335725958, +// "oddity": "" +// }, +// { +// "address": "180.97.36.63:53", +// "failure": null, +// "num_bytes": 29, +// "operation": "read", +// "proto": "udp", +// "t": 0.739987666, +// "started": 0.335863875, +// "oddity": "" +// } +// ], +// +// // But we see both in the error and in the oddity +// // that the response was "REFUSED" +// "lookup_host": [ +// { +// "answers": null, +// "engine": "udp", +// "failure": "dns_refused_error", +// "hostname": "example.com", +// "query_type": "A", +// "resolver_address": "180.97.36.63:53", +// "t": 0.7402975, +// "started": 7.2291e-05, +// "oddity": "dns.lookup.refused" +// }, +// { +// "answers": null, +// "engine": "udp", +// "failure": "dns_refused_error", +// "hostname": "example.com", +// "query_type": "AAAA", +// "resolver_address": "180.97.36.63:53", +// "t": 0.7402975, +// "started": 7.2291e-05, +// "oddity": "dns.lookup.refused" +// } +// ], +// +// // Exercise: do like I did before and decode the messages +// "dns_round_trip": [ +// { +// "engine": "udp", +// "resolver_address": "180.97.36.63:53", +// "raw_query": { +// "data": "crkBAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", +// "format": "base64" +// }, +// "started": 0.000130666, +// "t": 0.33509925, +// "failure": null, +// "raw_reply": { +// "data": "crmBBQABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=", +// "format": "base64" +// } +// }, +// { +// "engine": "udp", +// "resolver_address": "180.97.36.63:53", +// "raw_query": { +// "data": "ywcBAAABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=", +// "format": "base64" +// }, +// "started": 0.335321333, +// "t": 0.740152375, +// "failure": null, +// "raw_reply": { +// "data": "yweBBQABAAAAAAAAB2V4YW1wbGUDY29tAAAcAAE=", +// "format": "base64" +// } +// } +// ] +// } +// ``` +// +// ## Conclusion +// +// We have seen how we sending DNS queries over UDP, measure the +// results, and what happens on common error conditions. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter04/README.md b/internal/tutorial/measurex/chapter04/README.md new file mode 100644 index 0000000..970d860 --- /dev/null +++ b/internal/tutorial/measurex/chapter04/README.md @@ -0,0 +1,247 @@ + +# Chapter IV: TLS handshaking + +This chapter describes measuring TLS handshakes. + +Without further ado, let's describe our example `main.go` program +and let's use it to better understand how to measure that. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter04/main.go`.) + +## main.go + +The initial part of the program is pretty much the same as the one +used in previous chapters, so I will not add further comments. + +```Go +package main + +import ( + "context" + "crypto/tls" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/netxlite" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + sni := flag.String("sni", "dns.google", "domain to resolver") + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() +``` + +### Connecting and handshaking. + +We call the `ConnectAndHandshake` method. The arguments +are the context, the address, and a TLS config. + +Under the hood, the code will call the TCP connect functionality +we have seen in chapter02, using the address argument. Then, if +successful, it will TLS handshake using the given TLS config. + +```Go + m := mx.TLSConnectAndHandshake(ctx, *address, &tls.Config{ + ServerName: *sni, + NextProtos: []string{"h2", "http/1.1"}, + RootCAs: netxlite.NewDefaultCertPool(), + }) +``` + +Regarding the TLS config, in particular, +the three fields above are the files you should always set +in a TLS config when doing handshakes manually. The `ServerName` +field forces the SNI, the NextProtos field forces the ALPN, +and the `RootCAs` field is overridden so that we use the +CA bundle that we bundle with OONI. (This CA bundle is the +same you can find at https://curl.haxx.se/ca/.) + +As usual, the method to perform a measurement returns +the measurement itself, which we print below. + +``` + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +``` +## Running the example program + +As before, let us start off with a vanilla run: + +```bash +go run -race ./internal/tutorial/measurex/chapter04 +``` + +Let us comment the JSON in detail: + +```JavaScript +{ + "network": "tcp", + "address": "8.8.4.4:443", + + // This block is generated when connecting to a TCP + // socket, as we've already seen in chapter02 + "connect": [ + { + "address": "8.8.4.4:443", + "failure": null, + "operation": "connect", + "proto": "tcp", + "t": 0.046959084, + "started": 0.022998875, + "oddity": "" + } + ], + + // These are the I/O events during the handshake + "read_write": [ + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 280, + "operation": "write", + "proto": "tcp", + "t": 0.048752875, + "started": 0.04874125, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 517, + "operation": "read", + "proto": "tcp", + "t": 0.087221334, + "started": 0.048760417, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 4301, + "operation": "read", + "proto": "tcp", + "t": 0.088843584, + "started": 0.088830959, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 64, + "operation": "write", + "proto": "tcp", + "t": 0.092078042, + "started": 0.092064042, + "oddity": "" + } + ], + + // This block contains information about the handshake + "tls_handshake": [ + { + "cipher_suite": "TLS_AES_128_GCM_SHA256", + "failure": null, + "negotiated_proto": "h2", + "tls_version": "TLSv1.3", + "peer_certificates": [ + { + "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc", + "format": "base64" + }, + { + "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd", + "format": "base64" + }, + { + "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=", + "format": "base64" + } + ], + "t": 0.092117709, + "address": "8.8.4.4:443", + "server_name": "dns.google", + "alpn": [ + "h2", + "http/1.1" + ], + "no_tls_verify": false, + "oddity": "", + "proto": "tcp", + "started": 0.047288542 + } + ] +} +``` + +All the data formats we're using here are, by the way, +compatible with the data formats specified at +https://github.com/ooni/spec/tree/master/data-formats. + +### Suggested follow-up experiments + +Try to run experiments in the following scenarios, and +check the output JSON to familiarize with what changes in +different error conditions. + +1. measurement that causes timeout + +2. measurement with wrong SNI + +3. measurement with self-signed certificate + +4. measurement with expired certificate + +5. measurement with connection reset during handshake + +6. measurement with timeout during handshake + +Here are the commands I used for each proposed exercise: + +1. go run -race ./internal/tutorial/measurex/chapter04 -address 8.8.4.4:1 + +2. go run -race ./internal/tutorial/measurex/chapter04 -sni example.org + +3. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni self-signed.badssl.com + +4. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni expire.badssl.com + +To emulate the two last scenario, if you're on Linux, a +possibility is building Jafar with this command: + +``` +go build -v ./internal/cmd/jafar +``` + +Then, for example, to provoke a connection reset you +can run in a terminal: + +``` +sudo ./jafar -iptables-reset-keyword dns.google +``` + +and you can run this tutorial with `dns.google` as +the SNI in another terminal. + +Likewise, you can obtain a timeout using the +`-iptables-drop-keyword` flag instead. + +(Jafar runs forever and censors. You need to use +`^C` to terminate it from running.) + +## Conclusion + +We have seen how to measure TLS handshakes. We have seen how +this flow produces different output on different error conditions. + diff --git a/internal/tutorial/measurex/chapter04/main.go b/internal/tutorial/measurex/chapter04/main.go new file mode 100644 index 0000000..3ca7bfc --- /dev/null +++ b/internal/tutorial/measurex/chapter04/main.go @@ -0,0 +1,249 @@ +// -=-=- StartHere -=-=- +// +// # Chapter IV: TLS handshaking +// +// This chapter describes measuring TLS handshakes. +// +// Without further ado, let's describe our example `main.go` program +// and let's use it to better understand how to measure that. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter04/main.go`.) +// +// ## main.go +// +// The initial part of the program is pretty much the same as the one +// used in previous chapters, so I will not add further comments. +// +// ```Go +package main + +import ( + "context" + "crypto/tls" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/netxlite" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + sni := flag.String("sni", "dns.google", "domain to resolver") + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() + // ``` + // + // ### Connecting and handshaking. + // + // We call the `ConnectAndHandshake` method. The arguments + // are the context, the address, and a TLS config. + // + // Under the hood, the code will call the TCP connect functionality + // we have seen in chapter02, using the address argument. Then, if + // successful, it will TLS handshake using the given TLS config. + // + // ```Go + m := mx.TLSConnectAndHandshake(ctx, *address, &tls.Config{ + ServerName: *sni, + NextProtos: []string{"h2", "http/1.1"}, + RootCAs: netxlite.NewDefaultCertPool(), + }) + // ``` + // + // Regarding the TLS config, in particular, + // the three fields above are the files you should always set + // in a TLS config when doing handshakes manually. The `ServerName` + // field forces the SNI, the NextProtos field forces the ALPN, + // and the `RootCAs` field is overridden so that we use the + // CA bundle that we bundle with OONI. (This CA bundle is the + // same you can find at https://curl.haxx.se/ca/.) + // + // As usual, the method to perform a measurement returns + // the measurement itself, which we print below. + // + // ``` + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +// ``` +// ## Running the example program +// +// As before, let us start off with a vanilla run: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter04 +// ``` +// +// Let us comment the JSON in detail: +// +// ```JavaScript +// { +// "network": "tcp", +// "address": "8.8.4.4:443", +// +// // This block is generated when connecting to a TCP +// // socket, as we've already seen in chapter02 +// "connect": [ +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "operation": "connect", +// "proto": "tcp", +// "t": 0.046959084, +// "started": 0.022998875, +// "oddity": "" +// } +// ], +// +// // These are the I/O events during the handshake +// "read_write": [ +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 280, +// "operation": "write", +// "proto": "tcp", +// "t": 0.048752875, +// "started": 0.04874125, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 517, +// "operation": "read", +// "proto": "tcp", +// "t": 0.087221334, +// "started": 0.048760417, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 4301, +// "operation": "read", +// "proto": "tcp", +// "t": 0.088843584, +// "started": 0.088830959, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 64, +// "operation": "write", +// "proto": "tcp", +// "t": 0.092078042, +// "started": 0.092064042, +// "oddity": "" +// } +// ], +// +// // This block contains information about the handshake +// "tls_handshake": [ +// { +// "cipher_suite": "TLS_AES_128_GCM_SHA256", +// "failure": null, +// "negotiated_proto": "h2", +// "tls_version": "TLSv1.3", +// "peer_certificates": [ +// { +// "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc", +// "format": "base64" +// }, +// { +// "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd", +// "format": "base64" +// }, +// { +// "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=", +// "format": "base64" +// } +// ], +// "t": 0.092117709, +// "address": "8.8.4.4:443", +// "server_name": "dns.google", +// "alpn": [ +// "h2", +// "http/1.1" +// ], +// "no_tls_verify": false, +// "oddity": "", +// "proto": "tcp", +// "started": 0.047288542 +// } +// ] +// } +// ``` +// +// All the data formats we're using here are, by the way, +// compatible with the data formats specified at +// https://github.com/ooni/spec/tree/master/data-formats. +// +// ### Suggested follow-up experiments +// +// Try to run experiments in the following scenarios, and +// check the output JSON to familiarize with what changes in +// different error conditions. +// +// 1. measurement that causes timeout +// +// 2. measurement with wrong SNI +// +// 3. measurement with self-signed certificate +// +// 4. measurement with expired certificate +// +// 5. measurement with connection reset during handshake +// +// 6. measurement with timeout during handshake +// +// Here are the commands I used for each proposed exercise: +// +// 1. go run -race ./internal/tutorial/measurex/chapter04 -address 8.8.4.4:1 +// +// 2. go run -race ./internal/tutorial/measurex/chapter04 -sni example.org +// +// 3. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni self-signed.badssl.com +// +// 4. go run -race ./internal/tutorial/measurex/chapter04 -address 104.154.89.105:443 -sni expire.badssl.com +// +// To emulate the two last scenario, if you're on Linux, a +// possibility is building Jafar with this command: +// +// ``` +// go build -v ./internal/cmd/jafar +// ``` +// +// Then, for example, to provoke a connection reset you +// can run in a terminal: +// +// ``` +// sudo ./jafar -iptables-reset-keyword dns.google +// ``` +// +// and you can run this tutorial with `dns.google` as +// the SNI in another terminal. +// +// Likewise, you can obtain a timeout using the +// `-iptables-drop-keyword` flag instead. +// +// (Jafar runs forever and censors. You need to use +// `^C` to terminate it from running.) +// +// ## Conclusion +// +// We have seen how to measure TLS handshakes. We have seen how +// this flow produces different output on different error conditions. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter05/README.md b/internal/tutorial/measurex/chapter05/README.md new file mode 100644 index 0000000..9898dfc --- /dev/null +++ b/internal/tutorial/measurex/chapter05/README.md @@ -0,0 +1,280 @@ + +# Chapter V: QUIC handshaking + +This chapter describes measuring QUIC handshakes. Conceptually, +and code wise, this is very similar to the previous chapter. +The API call, in fact, has exactly the same structure, though +under the hood QUIC is different because there are no +separate connection establishment and handshake primitives. +For this reason, we will not see a connect event, but we +will only see a "QUIC handshake event". + +Having said that, let us now move on and see the code of +the simple program that shows this functionality. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measure/chapter05/main.go`.) + +## main.go + +The initial part of the program is pretty much the same as the one +used in previous chapters, so I will not add further comments. + +```Go +package main + +import ( + "context" + "crypto/tls" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/netxlite" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + sni := flag.String("sni", "dns.google", "value for SNI extension") + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() +``` + +### Handshaking with QUIC + +The API signature is indeed the same as the previous chapter, +except that here we call the `QUICHandshake` function. + +```Go + m := mx.QUICHandshake(ctx, *address, &tls.Config{ + ServerName: *sni, + NextProtos: []string{"h3"}, + RootCAs: netxlite.NewDefaultCertPool(), + }) +``` + +The same remarks mentioned in the previous chapter regarding +the arguments for the TLS config also apply here. We need +to specify the SNI (`ServerName`), the ALPN (`NextProtos`), +and the CA pool we want to use. Here, again, we're using +the CA pool from cURL that we bundle with ooniprobe. + +As we did in the previous chapters, here's the usual three +lines of code for printing the resulting measurement. + +``` + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +``` + +## Running the example program + +As before, let us start off with a vanilla run: + +```bash +go run -race ./internal/tutorial/measurex/chapter05 +``` + +Produces this JSON: + +```JavaScript +{ + // In chapter02 these two fields were similar but + // the network was "tcp" as opposed to "quic" + "network": "quic", + "address": "8.8.4.4:443", + + // This block contains I/O operations. Note that + // the protocol is "quic" and that the syscalls + // are "read_from" and "write_to" because QUIC does + // not bind/connect sockets. (The real syscalls + // are actually `recvfrom` and `sendto` but here + // we follow the Go convention of using read/write + // more frequently than send/recv.) + "read_write": [ + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 1252, + "operation": "write_to", + "proto": "quic", + "t": 0.003903167, + "started": 0.0037395, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 1252, + "operation": "read_from", + "proto": "quic", + "t": 0.029389125, + "started": 0.002954792, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 1252, + "operation": "write_to", + "proto": "quic", + "t": 0.029757584, + "started": 0.02972325, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 1252, + "operation": "read_from", + "proto": "quic", + "t": 0.045039875, + "started": 0.029424792, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 1252, + "operation": "read_from", + "proto": "quic", + "t": 0.045055334, + "started": 0.045049625, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 1252, + "operation": "read_from", + "proto": "quic", + "t": 0.045073917, + "started": 0.045069667, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 1233, + "operation": "read_from", + "proto": "quic", + "t": 0.04508, + "started": 0.045075292, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 64, + "operation": "read_from", + "proto": "quic", + "t": 0.045088167, + "started": 0.045081167, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 44, + "operation": "write_to", + "proto": "quic", + "t": 0.045370417, + "started": 0.045338667, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 44, + "operation": "write_to", + "proto": "quic", + "t": 0.045392125, + "started": 0.045380959, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 83, + "operation": "write_to", + "proto": "quic", + "t": 0.047042542, + "started": 0.047001917, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 33, + "operation": "write_to", + "proto": "quic", + "t": 0.047060834, + "started": 0.047046875, + "oddity": "" + } + ], + + // This section describes the QUIC handshake and it has + // basically the same fields of the TLS handshake. + "quic_handshake": [ + { + "cipher_suite": "TLS_CHACHA20_POLY1305_SHA256", + "failure": null, + "negotiated_proto": "h3", + "tls_version": "TLSv1.3", + "peer_certificates": [ + { + "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc", + "format": "base64" + }, + { + "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd", + "format": "base64" + }, + { + "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=", + "format": "base64" + } + ], + "t": 0.047042459, + "address": "8.8.4.4:443", + "server_name": "dns.google", + "alpn": [ + "h3" + ], + "no_tls_verify": false, + "oddity": "", + "proto": "quic", + "started": 0.002154834 + } + ] +} +``` + +Here are some suggestions on other experiments to run: + +1. obtain a timeout by connecting on a port that is not +actually listening for QUIC; + +2. obtain a certificate validation error by forcing +a different SNI; + +3. use a different ALPN (by changing the code), and see +how the error and the oddity are handled. Can we do +anything about this by changing `./internal/netxlite/errorx` +to better support for this specific error condition? + +## Conclusion + +We have seen how to perform QUIC handshake and +collect measurements. + diff --git a/internal/tutorial/measurex/chapter05/main.go b/internal/tutorial/measurex/chapter05/main.go new file mode 100644 index 0000000..8d92d6b --- /dev/null +++ b/internal/tutorial/measurex/chapter05/main.go @@ -0,0 +1,282 @@ +// -=-=- StartHere -=-=- +// +// # Chapter V: QUIC handshaking +// +// This chapter describes measuring QUIC handshakes. Conceptually, +// and code wise, this is very similar to the previous chapter. +// The API call, in fact, has exactly the same structure, though +// under the hood QUIC is different because there are no +// separate connection establishment and handshake primitives. +// For this reason, we will not see a connect event, but we +// will only see a "QUIC handshake event". +// +// Having said that, let us now move on and see the code of +// the simple program that shows this functionality. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measure/chapter05/main.go`.) +// +// ## main.go +// +// The initial part of the program is pretty much the same as the one +// used in previous chapters, so I will not add further comments. +// +// ```Go +package main + +import ( + "context" + "crypto/tls" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/netxlite" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func main() { + sni := flag.String("sni", "dns.google", "value for SNI extension") + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() + // ``` + // + // ### Handshaking with QUIC + // + // The API signature is indeed the same as the previous chapter, + // except that here we call the `QUICHandshake` function. + // + // ```Go + m := mx.QUICHandshake(ctx, *address, &tls.Config{ + ServerName: *sni, + NextProtos: []string{"h3"}, + RootCAs: netxlite.NewDefaultCertPool(), + }) + // ``` + // + // The same remarks mentioned in the previous chapter regarding + // the arguments for the TLS config also apply here. We need + // to specify the SNI (`ServerName`), the ALPN (`NextProtos`), + // and the CA pool we want to use. Here, again, we're using + // the CA pool from cURL that we bundle with ooniprobe. + // + // As we did in the previous chapters, here's the usual three + // lines of code for printing the resulting measurement. + // + // ``` + data, err := json.Marshal(m) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +// ``` +// +// ## Running the example program +// +// As before, let us start off with a vanilla run: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter05 +// ``` +// +// Produces this JSON: +// +// ```JavaScript +// { +// // In chapter02 these two fields were similar but +// // the network was "tcp" as opposed to "quic" +// "network": "quic", +// "address": "8.8.4.4:443", +// +// // This block contains I/O operations. Note that +// // the protocol is "quic" and that the syscalls +// // are "read_from" and "write_to" because QUIC does +// // not bind/connect sockets. (The real syscalls +// // are actually `recvfrom` and `sendto` but here +// // we follow the Go convention of using read/write +// // more frequently than send/recv.) +// "read_write": [ +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 1252, +// "operation": "write_to", +// "proto": "quic", +// "t": 0.003903167, +// "started": 0.0037395, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 1252, +// "operation": "read_from", +// "proto": "quic", +// "t": 0.029389125, +// "started": 0.002954792, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 1252, +// "operation": "write_to", +// "proto": "quic", +// "t": 0.029757584, +// "started": 0.02972325, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 1252, +// "operation": "read_from", +// "proto": "quic", +// "t": 0.045039875, +// "started": 0.029424792, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 1252, +// "operation": "read_from", +// "proto": "quic", +// "t": 0.045055334, +// "started": 0.045049625, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 1252, +// "operation": "read_from", +// "proto": "quic", +// "t": 0.045073917, +// "started": 0.045069667, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 1233, +// "operation": "read_from", +// "proto": "quic", +// "t": 0.04508, +// "started": 0.045075292, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 64, +// "operation": "read_from", +// "proto": "quic", +// "t": 0.045088167, +// "started": 0.045081167, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 44, +// "operation": "write_to", +// "proto": "quic", +// "t": 0.045370417, +// "started": 0.045338667, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 44, +// "operation": "write_to", +// "proto": "quic", +// "t": 0.045392125, +// "started": 0.045380959, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 83, +// "operation": "write_to", +// "proto": "quic", +// "t": 0.047042542, +// "started": 0.047001917, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 33, +// "operation": "write_to", +// "proto": "quic", +// "t": 0.047060834, +// "started": 0.047046875, +// "oddity": "" +// } +// ], +// +// // This section describes the QUIC handshake and it has +// // basically the same fields of the TLS handshake. +// "quic_handshake": [ +// { +// "cipher_suite": "TLS_CHACHA20_POLY1305_SHA256", +// "failure": null, +// "negotiated_proto": "h3", +// "tls_version": "TLSv1.3", +// "peer_certificates": [ +// { +// "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc", +// "format": "base64" +// }, +// { +// "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd", +// "format": "base64" +// }, +// { +// "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=", +// "format": "base64" +// } +// ], +// "t": 0.047042459, +// "address": "8.8.4.4:443", +// "server_name": "dns.google", +// "alpn": [ +// "h3" +// ], +// "no_tls_verify": false, +// "oddity": "", +// "proto": "quic", +// "started": 0.002154834 +// } +// ] +// } +// ``` +// +// Here are some suggestions on other experiments to run: +// +// 1. obtain a timeout by connecting on a port that is not +// actually listening for QUIC; +// +// 2. obtain a certificate validation error by forcing +// a different SNI; +// +// 3. use a different ALPN (by changing the code), and see +// how the error and the oddity are handled. Can we do +// anything about this by changing `./internal/netxlite/errorx` +// to better support for this specific error condition? +// +// ## Conclusion +// +// We have seen how to perform QUIC handshake and +// collect measurements. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter06/README.md b/internal/tutorial/measurex/chapter06/README.md new file mode 100644 index 0000000..f37b8cc --- /dev/null +++ b/internal/tutorial/measurex/chapter06/README.md @@ -0,0 +1,486 @@ + +# Chapter VI: Getting a webpage from an HTTP/HTTPS/HTTP3 endpoint. + +This chapter describes measuring getting a webpage from an +HTTPS endpoint. We have seen how to TCP connect, we have +seen how to TLS handshake, now it's time to see how we can +combine these operations with fetching a webpage from a +given TCP endpoint speaking HTTP and TLS. (As well as to +provide you with information on how to otherwise fetch +from HTTP and HTTP/3 endpoints.) + +The program we're going to write, `main.go`, will show a +high-level operation to perform this measurement in a +single API call. The code implementing this API call will +combine the operations we have seen in previous chapter +with the "give me the webpage" operation. We are still +quite far away from the ability of "measuring a URL" but +we are increasingly moving towards more complex operations. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter06/main.go`.) + +## main.go + +We have package declaration and imports as usual. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +``` + +We have factored the three lines to print a measurement +into the following utility function called `print`. + +```Go +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +``` + +The initial part of the program is pretty much the same as the one +used in previous chapters, expect that we have a few more command line +flags now, so I will not add further comments. + +```Go +func main() { + sni := flag.String("sni", "dns.google", "value for SNI extension") + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() +``` + +### HTTPEndpoint: a description of what to measure + +First of all, let us create a description of the endpoint +for `measurex`. Up to this point we have used endpoint +to describe a level-4-like address. Therefore, we have seen +TCP endpoints being used for the TCP connect and the TLS +handshake, and we have seen QUIC endpoints being used +for the QUIC handshake. Now, however, we are going +to need more information to characterize the endpoint. + +```Go + epnt := &measurex.HTTPEndpoint{ + Domain: *sni, + Network: "tcp", + Address: *address, + SNI: *sni, + ALPN: []string{"h2", "http/1.1"}, + URL: &url.URL{ + Scheme: "https", + Host: *sni, + Path: "/", + }, + Header: measurex.NewHTTPRequestHeaderForMeasuring(), + } +``` + +In fact, in the above definition we recognize fields +we have already discussed, such as: + +- `Network`, describing whether to use "tcp" or "quic"; + +- `Address`, which is the endpoint address. + +But we also need to combine into this view of the +endpoint additional fields for TLS: + +- `SNI`, to set the SNI; + +- `ALPN`, to set the ALPN; + +But then we also need to specify: + +- the URL to use; + +- the headers to use (for which we're using a handy +factory for creating reasonable defaults for measuring). + +This API is not the highest level API with which to do +the job, but it's still handy to introduce the +`measurex.HTTPEndpoint` data structure since it's +used by higher level APIs. + +(You may also be wondering about the CA pool. It turns +out that for APIs such as this one and for higher +level APIs, the default is to always use the bundled +Mozilla CA pool, because this is what we use in +most cases for performing measurements.) + +### HTTPEndpointGetWithoutCookies + +When used with an HTTP URL, the `HTTPEndpointGetWithoutCookies` +method combines two operations: + +- TCP connect + +- HTTP GET + +When the URL is HTTPS, we do: + +- TCP connect + +- TLS handshake + +- HTTP GET (or HTTP/2 GET depending on the ALPN) + +When the `HTTPEndpoint.Network` field value +is QUIC, instead we do: + +- QUIC handshake + +- HTTP/3 GET + +```Go + m := mx.HTTPEndpointGetWithoutCookies(ctx, epnt) +``` + +The arguments for `HTTPEndpointGetWithDBWithoutCookies` are: + +- the context for deadline/timeout + +- the HTTPEndpoint descriptor + +The result is an `HTTPEndpointMeasurement` which +you can inspect with + +``` +go doc ./internal/measurex.HTTPEndpointMeasurement +``` + +### Printing the measurement + +Let us now print the resulting measurement. + +```Go + print(m) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter06 +``` + +This is the JSON output. Let us comment it in detail: + +```Javascript +{ + // The returned type is called HTTPEndpointMeasurement + // and you see that here on top we indeed have the + // information on the endpoint and the URL. + "url": "https://dns.google/", + "network": "tcp", + "address": "8.8.4.4:443", + + // Internally, HTTPEndpointGetWithoutCookies calls + // TCPConnect and here we see the corresponding event + "connect": [ + { + "address": "8.8.4.4:443", + "failure": null, + "operation": "connect", + "proto": "tcp", + "t": 0.02422375, + "started": 0.002269291, + "oddity": "" + } + ], + + // These are the I/O operations we have already seen + // in previous chapters + "read_write": [ + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 280, + "operation": "write", + "proto": "tcp", + "t": 0.024931791, + "started": 0.024910416, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 517, + "operation": "read", + "proto": "tcp", + "t": 0.063629791, + "started": 0.024935666, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 4301, + "operation": "read", + "proto": "tcp", + "t": 0.064183, + "started": 0.064144208, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 64, + "operation": "write", + "proto": "tcp", + "t": 0.065464041, + "started": 0.065441333, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 86, + "operation": "write", + "proto": "tcp", + "t": 0.067256083, + "started": 0.067224375, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 201, + "operation": "write", + "proto": "tcp", + "t": 0.067674416, + "started": 0.067652375, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 93, + "operation": "read", + "proto": "tcp", + "t": 0.086618708, + "started": 0.067599208, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 31, + "operation": "write", + "proto": "tcp", + "t": 0.086703625, + "started": 0.0866745, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 2028, + "operation": "read", + "proto": "tcp", + "t": 0.337785916, + "started": 0.086717333, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 39, + "operation": "write", + "proto": "tcp", + "t": 0.338514916, + "started": 0.338485375, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": null, + "num_bytes": 24, + "operation": "write", + "proto": "tcp", + "t": 0.338800833, + "started": 0.338788625, + "oddity": "" + }, + { + "address": "8.8.4.4:443", + "failure": "connection_already_closed", + "operation": "read", + "proto": "tcp", + "t": 0.338888041, + "started": 0.338523291, + "oddity": "" + } + ], + + // Internally, HTTPEndpointGetWithoutCookies calls TLSConnectAndHandshake, + // and here's the resulting handshake event. Of course, if we + // specified a QUIC endpoint we would instead see here a + // QUIC handshake event. And, we would not see any handshake + // if the URL was instead an HTTP URL. + "tls_handshake": [ + { + "cipher_suite": "TLS_AES_128_GCM_SHA256", + "failure": null, + "negotiated_proto": "h2", + "tls_version": "TLSv1.3", + "peer_certificates": [ + { + "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc", + "format": "base64" + }, + { + "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd", + "format": "base64" + }, + { + "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=", + "format": "base64" + } + ], + "t": 0.065514708, + "address": "8.8.4.4:443", + "server_name": "dns.google", + "alpn": [ + "h2", + "http/1.1" + ], + "no_tls_verify": false, + "oddity": "", + "proto": "tcp", + "started": 0.024404083 + } + ], + + // Finally here we see information about the round trip, which + // is formatted according the df-001-httpt data format: + "http_round_trip": [ + { + + // This field indicates whether there was an error during + // the HTTP round trip: + "failure": null, + + // This field contains the request method, URL, and HTTP headers + "request": { + "method": "GET", + "url": "https://dns.google/", + "headers": { + "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", + "accept-language": "en-US;q=0.8,en;q=0.5", + "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" + } + }, + + // This field contains the response status code, body, + // and headers. + "response": { + "code": 200, + "headers": { + "accept-ranges": "none", + "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"", + "cache-control": "private", + "content-security-policy": "object-src 'none';base-uri 'self';script-src 'nonce-bSLcJjaotppZl3Y2moIaxg==' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/honest_dns/1_0;frame-ancestors 'none'", + "content-type": "text/html; charset=UTF-8", + "date": "Fri, 24 Sep 2021 08:51:01 GMT", + "server": "scaffolding on HTTPServer2", + "strict-transport-security": "max-age=31536000; includeSubDomains; preload", + "vary": "Accept-Encoding", + "x-content-type-options": "nosniff", + "x-frame-options": "SAMEORIGIN", + "x-xss-protection": "0" + }, + + // The body in particular is a snapshot of the response + // body: we don't want to read and submit to the OONI + // collector large bodies. + "body": { + "data": "PCFET0NUWVBFIGh0bWw+CjxodG1sIGxhbmc9ImVuIj4gPGhlYWQ+IDx0aXRsZT5Hb29nbGUgUHVibGljIEROUzwvdGl0bGU+ICA8bWV0YSBjaGFyc2V0PSJVVEYtOCI+IDxsaW5rIGhyZWY9Ii9zdGF0aWMvOTNkZDU5NTQvZmF2aWNvbi5wbmciIHJlbD0ic2hvcnRjdXQgaWNvbiIgdHlwZT0iaW1hZ2UvcG5nIj4gPGxpbmsgaHJlZj0iL3N0YXRpYy84MzZhZWJjNi9tYXR0ZXIubWluLmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPGxpbmsgaHJlZj0iL3N0YXRpYy9iODUzNmMzNy9zaGFyZWQuY3NzIiByZWw9InN0eWxlc2hlZXQiPiA8bWV0YSBuYW1lPSJ2aWV3cG9ydCIgY29udGVudD0id2lkdGg9ZGV2aWNlLXdpZHRoLCBpbml0aWFsLXNjYWxlPTEiPiAgPGxpbmsgaHJlZj0iL3N0YXRpYy9kMDVjZDZiYS9yb290LmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPC9oZWFkPiA8Ym9keT4gPHNwYW4gY2xhc3M9ImZpbGxlciB0b3AiPjwvc3Bhbj4gICA8ZGl2IGNsYXNzPSJsb2dvIiB0aXRsZT0iR29vZ2xlIFB1YmxpYyBETlMiPiA8ZGl2IGNsYXNzPSJsb2dvLXRleHQiPjxzcGFuPlB1YmxpYyBETlM8L3NwYW4+PC9kaXY+IDwvZGl2PiAgPGZvcm0gYWN0aW9uPSIvcXVlcnkiIG1ldGhvZD0iR0VUIj4gIDxkaXYgY2xhc3M9InJvdyI+IDxsYWJlbCBjbGFzcz0ibWF0dGVyLXRleHRmaWVsZC1vdXRsaW5lZCI+IDxpbnB1dCB0eXBlPSJ0ZXh0IiBuYW1lPSJuYW1lIiBwbGFjZWhvbGRlcj0iJm5ic3A7Ij4gPHNwYW4+RE5TIE5hbWU8L3NwYW4+IDxwIGNsYXNzPSJoZWxwIj4gRW50ZXIgYSBkb21haW4gKGxpa2UgZXhhbXBsZS5jb20pIG9yIElQIGFkZHJlc3MgKGxpa2UgOC44LjguOCBvciAyMDAxOjQ4NjA6NDg2MDo6ODg0NCkgaGVyZS4gPC9wPiA8L2xhYmVsPiA8YnV0dG9uIGNsYXNzPSJtYXR0ZXItYnV0dG9uLWNvbnRhaW5lZCBtYXR0ZXItcHJpbWFyeSIgdHlwZT0ic3VibWl0Ij5SZXNvbHZlPC9idXR0b24+IDwvZGl2PiA8L2Zvcm0+ICA8c3BhbiBjbGFzcz0iZmlsbGVyIGJvdHRvbSI+PC9zcGFuPiA8Zm9vdGVyIGNsYXNzPSJyb3ciPiA8YSBocmVmPSJodHRwczovL2RldmVsb3BlcnMuZ29vZ2xlLmNvbS9zcGVlZC9wdWJsaWMtZG5zIj5IZWxwPC9hPiA8YSBocmVmPSIvY2FjaGUiPkNhY2hlIEZsdXNoPC9hPiA8c3BhbiBjbGFzcz0iZmlsbGVyIj48L3NwYW4+IDxhIGhyZWY9Imh0dHBzOi8vZGV2ZWxvcGVycy5nb29nbGUuY29tL3NwZWVkL3B1YmxpYy1kbnMvZG9jcy91c2luZyI+IEdldCBTdGFydGVkIHdpdGggR29vZ2xlIFB1YmxpYyBETlMgPC9hPiA8L2Zvb3Rlcj4gICA8c2NyaXB0IG5vbmNlPSJiU0xjSmphb3RwcFpsM1kybW9JYXhnPT0iPmRvY3VtZW50LmZvcm1zWzBdLm5hbWUuZm9jdXMoKTs8L3NjcmlwdD4gPC9ib2R5PiA8L2h0bWw+", + "format": "base64" + }, + + // This field tells us whether the size of the read + // snapshot was smaller than the snapshot size. If + // not, then the body has been truncated. + "body_is_truncated": false, + + // These extra fields are not part of the spec and + // hence we prefix them with `x_`. They tell us + // the length of the body and whether the content + // of the body is valid UTF8. + "x_body_length": 1383, + "x_body_is_utf8": true + }, + + // The t field is the moment where we finished the + // round trip and saved the event. The started field + // is instead when we started the round trip. + + // You may notice that the start of the round trip + // if after the `t` of the handshake. This tells us + // that the code first connects, then handshakes, and + // finally creates HTTP code for performing the + // round trip. + "t": 0.338674625, + "started": 0.065926625, + + // As usual we also compute an oddity value related + // in this case to the HTTP round trip. + "oddity": "" + } + ] +} +``` + +Here are some suggestions for follow up measurements: + +1. provoke a connect error by using: + +``` +go run -race ./internal/tutorial/measurex/chapter06 -address 127.0.0.1:1 +``` + +2. provoke a TLS handshake error by using: + +``` +go run -race ./internal/tutorial/measurex/chapter06 -sni example.com +``` + +3. provoke an HTTP round trip error by using: + +``` +go run -race ./internal/tutorial/measurex/chapter06 -address 8.8.8.8:853 +``` + +4. modify the code to fetch an HTTP endpoint instead (hint: you +need to change the HTTPEndpoint's URL scheme); + +5. modify the code to use QUIC and HTTP/3 instead (hint: you need to +change the HTTPEndpoint's network and... is this enough?). + +## Conclusion + +We have seen how to measure the flow of fetching a +specific webpage from an HTTPEndpoint. + diff --git a/internal/tutorial/measurex/chapter06/main.go b/internal/tutorial/measurex/chapter06/main.go new file mode 100644 index 0000000..42ff264 --- /dev/null +++ b/internal/tutorial/measurex/chapter06/main.go @@ -0,0 +1,488 @@ +// -=-=- StartHere -=-=- +// +// # Chapter VI: Getting a webpage from an HTTP/HTTPS/HTTP3 endpoint. +// +// This chapter describes measuring getting a webpage from an +// HTTPS endpoint. We have seen how to TCP connect, we have +// seen how to TLS handshake, now it's time to see how we can +// combine these operations with fetching a webpage from a +// given TCP endpoint speaking HTTP and TLS. (As well as to +// provide you with information on how to otherwise fetch +// from HTTP and HTTP/3 endpoints.) +// +// The program we're going to write, `main.go`, will show a +// high-level operation to perform this measurement in a +// single API call. The code implementing this API call will +// combine the operations we have seen in previous chapter +// with the "give me the webpage" operation. We are still +// quite far away from the ability of "measuring a URL" but +// we are increasingly moving towards more complex operations. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter06/main.go`.) +// +// ## main.go +// +// We have package declaration and imports as usual. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +// ``` +// +// We have factored the three lines to print a measurement +// into the following utility function called `print`. +// +// ```Go +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +// ``` +// +// The initial part of the program is pretty much the same as the one +// used in previous chapters, expect that we have a few more command line +// flags now, so I will not add further comments. +// +// ```Go +func main() { + sni := flag.String("sni", "dns.google", "value for SNI extension") + address := flag.String("address", "8.8.4.4:443", "remote endpoint address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + mx := measurex.NewMeasurerWithDefaultSettings() + // ``` + // + // ### HTTPEndpoint: a description of what to measure + // + // First of all, let us create a description of the endpoint + // for `measurex`. Up to this point we have used endpoint + // to describe a level-4-like address. Therefore, we have seen + // TCP endpoints being used for the TCP connect and the TLS + // handshake, and we have seen QUIC endpoints being used + // for the QUIC handshake. Now, however, we are going + // to need more information to characterize the endpoint. + // + // ```Go + epnt := &measurex.HTTPEndpoint{ + Domain: *sni, + Network: "tcp", + Address: *address, + SNI: *sni, + ALPN: []string{"h2", "http/1.1"}, + URL: &url.URL{ + Scheme: "https", + Host: *sni, + Path: "/", + }, + Header: measurex.NewHTTPRequestHeaderForMeasuring(), + } + // ``` + // + // In fact, in the above definition we recognize fields + // we have already discussed, such as: + // + // - `Network`, describing whether to use "tcp" or "quic"; + // + // - `Address`, which is the endpoint address. + // + // But we also need to combine into this view of the + // endpoint additional fields for TLS: + // + // - `SNI`, to set the SNI; + // + // - `ALPN`, to set the ALPN; + // + // But then we also need to specify: + // + // - the URL to use; + // + // - the headers to use (for which we're using a handy + // factory for creating reasonable defaults for measuring). + // + // This API is not the highest level API with which to do + // the job, but it's still handy to introduce the + // `measurex.HTTPEndpoint` data structure since it's + // used by higher level APIs. + // + // (You may also be wondering about the CA pool. It turns + // out that for APIs such as this one and for higher + // level APIs, the default is to always use the bundled + // Mozilla CA pool, because this is what we use in + // most cases for performing measurements.) + // + // ### HTTPEndpointGetWithoutCookies + // + // When used with an HTTP URL, the `HTTPEndpointGetWithoutCookies` + // method combines two operations: + // + // - TCP connect + // + // - HTTP GET + // + // When the URL is HTTPS, we do: + // + // - TCP connect + // + // - TLS handshake + // + // - HTTP GET (or HTTP/2 GET depending on the ALPN) + // + // When the `HTTPEndpoint.Network` field value + // is QUIC, instead we do: + // + // - QUIC handshake + // + // - HTTP/3 GET + // + // ```Go + m := mx.HTTPEndpointGetWithoutCookies(ctx, epnt) + // ``` + // + // The arguments for `HTTPEndpointGetWithDBWithoutCookies` are: + // + // - the context for deadline/timeout + // + // - the HTTPEndpoint descriptor + // + // The result is an `HTTPEndpointMeasurement` which + // you can inspect with + // + // ``` + // go doc ./internal/measurex.HTTPEndpointMeasurement + // ``` + // + // ### Printing the measurement + // + // Let us now print the resulting measurement. + // + // ```Go + print(m) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter06 +// ``` +// +// This is the JSON output. Let us comment it in detail: +// +// ```Javascript +// { +// // The returned type is called HTTPEndpointMeasurement +// // and you see that here on top we indeed have the +// // information on the endpoint and the URL. +// "url": "https://dns.google/", +// "network": "tcp", +// "address": "8.8.4.4:443", +// +// // Internally, HTTPEndpointGetWithoutCookies calls +// // TCPConnect and here we see the corresponding event +// "connect": [ +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "operation": "connect", +// "proto": "tcp", +// "t": 0.02422375, +// "started": 0.002269291, +// "oddity": "" +// } +// ], +// +// // These are the I/O operations we have already seen +// // in previous chapters +// "read_write": [ +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 280, +// "operation": "write", +// "proto": "tcp", +// "t": 0.024931791, +// "started": 0.024910416, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 517, +// "operation": "read", +// "proto": "tcp", +// "t": 0.063629791, +// "started": 0.024935666, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 4301, +// "operation": "read", +// "proto": "tcp", +// "t": 0.064183, +// "started": 0.064144208, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 64, +// "operation": "write", +// "proto": "tcp", +// "t": 0.065464041, +// "started": 0.065441333, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 86, +// "operation": "write", +// "proto": "tcp", +// "t": 0.067256083, +// "started": 0.067224375, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 201, +// "operation": "write", +// "proto": "tcp", +// "t": 0.067674416, +// "started": 0.067652375, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 93, +// "operation": "read", +// "proto": "tcp", +// "t": 0.086618708, +// "started": 0.067599208, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 31, +// "operation": "write", +// "proto": "tcp", +// "t": 0.086703625, +// "started": 0.0866745, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 2028, +// "operation": "read", +// "proto": "tcp", +// "t": 0.337785916, +// "started": 0.086717333, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 39, +// "operation": "write", +// "proto": "tcp", +// "t": 0.338514916, +// "started": 0.338485375, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": null, +// "num_bytes": 24, +// "operation": "write", +// "proto": "tcp", +// "t": 0.338800833, +// "started": 0.338788625, +// "oddity": "" +// }, +// { +// "address": "8.8.4.4:443", +// "failure": "connection_already_closed", +// "operation": "read", +// "proto": "tcp", +// "t": 0.338888041, +// "started": 0.338523291, +// "oddity": "" +// } +// ], +// +// // Internally, HTTPEndpointGetWithoutCookies calls TLSConnectAndHandshake, +// // and here's the resulting handshake event. Of course, if we +// // specified a QUIC endpoint we would instead see here a +// // QUIC handshake event. And, we would not see any handshake +// // if the URL was instead an HTTP URL. +// "tls_handshake": [ +// { +// "cipher_suite": "TLS_AES_128_GCM_SHA256", +// "failure": null, +// "negotiated_proto": "h2", +// "tls_version": "TLSv1.3", +// "peer_certificates": [ +// { +// "data": "MIIF4TCCBMmgAwIBAgIQGa7QSAXLo6sKAAAAAPz4cjANBgkqhkiG9w0BAQsFADBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMTA4MzAwNDAwMDBaFw0yMTExMjIwMzU5NTlaMBUxEzARBgNVBAMTCmRucy5nb29nbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8cttrGHp3SS9YGYgsNLXt43dhW4d8FPULk0n6WYWC+EbMLkLnYXHLZHXJEz1Tor5hrCfHEVyX4xmhY2LCt0jprP6Gfo+gkKyjSV3LO65aWx6ezejvIdQBiLhSo/R5E3NwjMUAbm9PoNfSZSLiP3RjC3Px1vXFVmlcap4bUHnv9OvcPvwV1wmw5IMVzCuGBjCzJ4c4fxgyyggES1mbXZpYcDO4YKhSqIJx2D0gop9wzBQevI/kb35miN1pAvIKK2lgf7kZvYa7HH5vJ+vtn3Vkr34dKUAc/cO62t+NVufADPwn2/Tx8y8fPxlnCmoJeI+MPsw+StTYDawxajkjvZfdAgMBAAGjggL6MIIC9jAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUooaIxGAth6+bJh0JHYVWccyuoUcwHwYDVR0jBBgwFoAUinR/r4XN7pXNPZzQ4kYU83E1HScwagYIKwYBBQUHAQEEXjBcMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxYzMwMQYIKwYBBQUHMAKGJWh0dHA6Ly9wa2kuZ29vZy9yZXBvL2NlcnRzL2d0czFjMy5kZXIwgawGA1UdEQSBpDCBoYIKZG5zLmdvb2dsZYIOZG5zLmdvb2dsZS5jb22CECouZG5zLmdvb2dsZS5jb22CCzg4ODguZ29vZ2xlghBkbnM2NC5kbnMuZ29vZ2xlhwQICAgIhwQICAQEhxAgAUhgSGAAAAAAAAAAAIiIhxAgAUhgSGAAAAAAAAAAAIhEhxAgAUhgSGAAAAAAAAAAAGRkhxAgAUhgSGAAAAAAAAAAAABkMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3RzMWMzL2ZWSnhiVi1LdG1rLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABe5VtuiwAAAQDAEYwRAIgAwzr02ayTnNk/G+HDP50WTZUls3g+9P1fTGR9PEywpYCIAIOIQJ7nJTlcJdSyyOvgzX4BxJDr18mOKJPHlJs1naIAHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF7lW26IQAABAMARzBFAiAtlIkbCH+QgiO6T6Y/+UAf+eqHB2wdzMNfOoo4SnUhVgIhALPiRtyPMo8fPPxN3VgiXBqVF7tzLWTJUjprOe4kQUCgMA0GCSqGSIb3DQEBCwUAA4IBAQDVq3WWgg6eYSpFLfNgo2KzLKDPkWZx42gW2Tum6JZd6O/Nj+mjYGOyXyryTslUwmONxiq2Ip3PLA/qlbPdYic1F1mDwMHSzRteSe7axwEP6RkoxhMy5zuI4hfijhSrfhVUZF299PesDf2gI+Vh30s6muHVfQjbXOl/AkAqIPLSetv2mS9MHQLeHcCCXpwsXQJwusZ3+ILrgCRAGv6NLXwbfE0t3OjXV0gnNRp3DWEaF+yrfjE0oU1myeYDNtugsw8VRwTzCM53Nqf/BJffnuShmBBZfZ2jlsPnLys0UqCZo2dg5wdwj3DaKtHO5Pofq6P8r4w6W/aUZCTLUi1jZ3Gc", +// "format": "base64" +// }, +// { +// "data": "MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAwMDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzpkgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsXlOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcmBA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKAgOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwLtmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYDVR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYGCCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcwAoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcGA1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3BraS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcNAQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQcSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrLRklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2YrPxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IERlQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGsYye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjOz23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJGAJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKwjuDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd", +// "format": "base64" +// }, +// { +// "data": "MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYxOTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwSiV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351kKSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZDrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zkj5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esWCruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35EiEua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbapsZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAfBgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUHMAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAyMAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIFAwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvid0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=", +// "format": "base64" +// } +// ], +// "t": 0.065514708, +// "address": "8.8.4.4:443", +// "server_name": "dns.google", +// "alpn": [ +// "h2", +// "http/1.1" +// ], +// "no_tls_verify": false, +// "oddity": "", +// "proto": "tcp", +// "started": 0.024404083 +// } +// ], +// +// // Finally here we see information about the round trip, which +// // is formatted according the df-001-httpt data format: +// "http_round_trip": [ +// { +// +// // This field indicates whether there was an error during +// // the HTTP round trip: +// "failure": null, +// +// // This field contains the request method, URL, and HTTP headers +// "request": { +// "method": "GET", +// "url": "https://dns.google/", +// "headers": { +// "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", +// "accept-language": "en-US;q=0.8,en;q=0.5", +// "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" +// } +// }, +// +// // This field contains the response status code, body, +// // and headers. +// "response": { +// "code": 200, +// "headers": { +// "accept-ranges": "none", +// "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"", +// "cache-control": "private", +// "content-security-policy": "object-src 'none';base-uri 'self';script-src 'nonce-bSLcJjaotppZl3Y2moIaxg==' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/honest_dns/1_0;frame-ancestors 'none'", +// "content-type": "text/html; charset=UTF-8", +// "date": "Fri, 24 Sep 2021 08:51:01 GMT", +// "server": "scaffolding on HTTPServer2", +// "strict-transport-security": "max-age=31536000; includeSubDomains; preload", +// "vary": "Accept-Encoding", +// "x-content-type-options": "nosniff", +// "x-frame-options": "SAMEORIGIN", +// "x-xss-protection": "0" +// }, +// +// // The body in particular is a snapshot of the response +// // body: we don't want to read and submit to the OONI +// // collector large bodies. +// "body": { +// "data": "PCFET0NUWVBFIGh0bWw+CjxodG1sIGxhbmc9ImVuIj4gPGhlYWQ+IDx0aXRsZT5Hb29nbGUgUHVibGljIEROUzwvdGl0bGU+ICA8bWV0YSBjaGFyc2V0PSJVVEYtOCI+IDxsaW5rIGhyZWY9Ii9zdGF0aWMvOTNkZDU5NTQvZmF2aWNvbi5wbmciIHJlbD0ic2hvcnRjdXQgaWNvbiIgdHlwZT0iaW1hZ2UvcG5nIj4gPGxpbmsgaHJlZj0iL3N0YXRpYy84MzZhZWJjNi9tYXR0ZXIubWluLmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPGxpbmsgaHJlZj0iL3N0YXRpYy9iODUzNmMzNy9zaGFyZWQuY3NzIiByZWw9InN0eWxlc2hlZXQiPiA8bWV0YSBuYW1lPSJ2aWV3cG9ydCIgY29udGVudD0id2lkdGg9ZGV2aWNlLXdpZHRoLCBpbml0aWFsLXNjYWxlPTEiPiAgPGxpbmsgaHJlZj0iL3N0YXRpYy9kMDVjZDZiYS9yb290LmNzcyIgcmVsPSJzdHlsZXNoZWV0Ij4gPC9oZWFkPiA8Ym9keT4gPHNwYW4gY2xhc3M9ImZpbGxlciB0b3AiPjwvc3Bhbj4gICA8ZGl2IGNsYXNzPSJsb2dvIiB0aXRsZT0iR29vZ2xlIFB1YmxpYyBETlMiPiA8ZGl2IGNsYXNzPSJsb2dvLXRleHQiPjxzcGFuPlB1YmxpYyBETlM8L3NwYW4+PC9kaXY+IDwvZGl2PiAgPGZvcm0gYWN0aW9uPSIvcXVlcnkiIG1ldGhvZD0iR0VUIj4gIDxkaXYgY2xhc3M9InJvdyI+IDxsYWJlbCBjbGFzcz0ibWF0dGVyLXRleHRmaWVsZC1vdXRsaW5lZCI+IDxpbnB1dCB0eXBlPSJ0ZXh0IiBuYW1lPSJuYW1lIiBwbGFjZWhvbGRlcj0iJm5ic3A7Ij4gPHNwYW4+RE5TIE5hbWU8L3NwYW4+IDxwIGNsYXNzPSJoZWxwIj4gRW50ZXIgYSBkb21haW4gKGxpa2UgZXhhbXBsZS5jb20pIG9yIElQIGFkZHJlc3MgKGxpa2UgOC44LjguOCBvciAyMDAxOjQ4NjA6NDg2MDo6ODg0NCkgaGVyZS4gPC9wPiA8L2xhYmVsPiA8YnV0dG9uIGNsYXNzPSJtYXR0ZXItYnV0dG9uLWNvbnRhaW5lZCBtYXR0ZXItcHJpbWFyeSIgdHlwZT0ic3VibWl0Ij5SZXNvbHZlPC9idXR0b24+IDwvZGl2PiA8L2Zvcm0+ICA8c3BhbiBjbGFzcz0iZmlsbGVyIGJvdHRvbSI+PC9zcGFuPiA8Zm9vdGVyIGNsYXNzPSJyb3ciPiA8YSBocmVmPSJodHRwczovL2RldmVsb3BlcnMuZ29vZ2xlLmNvbS9zcGVlZC9wdWJsaWMtZG5zIj5IZWxwPC9hPiA8YSBocmVmPSIvY2FjaGUiPkNhY2hlIEZsdXNoPC9hPiA8c3BhbiBjbGFzcz0iZmlsbGVyIj48L3NwYW4+IDxhIGhyZWY9Imh0dHBzOi8vZGV2ZWxvcGVycy5nb29nbGUuY29tL3NwZWVkL3B1YmxpYy1kbnMvZG9jcy91c2luZyI+IEdldCBTdGFydGVkIHdpdGggR29vZ2xlIFB1YmxpYyBETlMgPC9hPiA8L2Zvb3Rlcj4gICA8c2NyaXB0IG5vbmNlPSJiU0xjSmphb3RwcFpsM1kybW9JYXhnPT0iPmRvY3VtZW50LmZvcm1zWzBdLm5hbWUuZm9jdXMoKTs8L3NjcmlwdD4gPC9ib2R5PiA8L2h0bWw+", +// "format": "base64" +// }, +// +// // This field tells us whether the size of the read +// // snapshot was smaller than the snapshot size. If +// // not, then the body has been truncated. +// "body_is_truncated": false, +// +// // These extra fields are not part of the spec and +// // hence we prefix them with `x_`. They tell us +// // the length of the body and whether the content +// // of the body is valid UTF8. +// "x_body_length": 1383, +// "x_body_is_utf8": true +// }, +// +// // The t field is the moment where we finished the +// // round trip and saved the event. The started field +// // is instead when we started the round trip. +// +// // You may notice that the start of the round trip +// // if after the `t` of the handshake. This tells us +// // that the code first connects, then handshakes, and +// // finally creates HTTP code for performing the +// // round trip. +// "t": 0.338674625, +// "started": 0.065926625, +// +// // As usual we also compute an oddity value related +// // in this case to the HTTP round trip. +// "oddity": "" +// } +// ] +// } +// ``` +// +// Here are some suggestions for follow up measurements: +// +// 1. provoke a connect error by using: +// +// ``` +// go run -race ./internal/tutorial/measurex/chapter06 -address 127.0.0.1:1 +// ``` +// +// 2. provoke a TLS handshake error by using: +// +// ``` +// go run -race ./internal/tutorial/measurex/chapter06 -sni example.com +// ``` +// +// 3. provoke an HTTP round trip error by using: +// +// ``` +// go run -race ./internal/tutorial/measurex/chapter06 -address 8.8.8.8:853 +// ``` +// +// 4. modify the code to fetch an HTTP endpoint instead (hint: you +// need to change the HTTPEndpoint's URL scheme); +// +// 5. modify the code to use QUIC and HTTP/3 instead (hint: you need to +// change the HTTPEndpoint's network and... is this enough?). +// +// ## Conclusion +// +// We have seen how to measure the flow of fetching a +// specific webpage from an HTTPEndpoint. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter07/README.md b/internal/tutorial/measurex/chapter07/README.md new file mode 100644 index 0000000..7879009 --- /dev/null +++ b/internal/tutorial/measurex/chapter07/README.md @@ -0,0 +1,142 @@ + +# Chapter VII: Measuring all the HTTPEndpoints for a domain + +We are now going to combine DNS resolutions with getting +HTTPEndpoints. Conceptually, the DNS resolution yields +us a list of IP addresses. For each address, we build the +HTTPEndpoint and fetch it like we did in chapter06. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter07/main.go`.) + +## main.go + +We have package declaration and imports as usual. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +``` + +Here we define an helper type for containing the DNS +measurement and the subsequent endpoints measurements. + +```Go +type measurement struct { + DNS *measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +``` + +The rest of the program is quite similar to what we had before. + +```Go +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://google.com/", "URL to fetch") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() +``` + +This is where the main.go file starts to diverge. We create an +instance of our measurement type to hold the results. + +```Go + m := &measurement{} +``` + +Then we perform a DNS lookup using UDP like we saw in chapter03. + +```Go + m.DNS = mx.LookupHostUDP(ctx, parsed.Hostname(), *address) +``` + +Like we did in the previous chapter, we create suitable HTTP +headers for performing an HTTP measurement. + +```Go + headers := measurex.NewHTTPRequestHeaderForMeasuring() +``` + +The following is an entirely new function we're learning +about just now. `AllHTTPEndpointsForURL` is a free function +in `measurex` that given: + +- an already parsed HTTP/HTTPS URL + +- headers we want to use + +- the result of one or more DNS queries + +builds us a list of HTTPEndpoint data structures. + +```Go + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") +``` + +This function may fail if, for example, the URL is not HTTP/HTTPS. We +handle the error panicking, because this is an example program. + +We are almost done now: we loop over all the endpoints and apply the +`HTTPEndpointGetWithoutCookies` method we have seen in chapter06. + +```Go + for _, epnt := range httpEndpoints { + m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt)) + } +``` + +Finally, we print the results. + +```Go + print(m) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter07 +``` + +Please, check the JSON output. Do you recognize the fields +we have described in previous chapters? + +Can you provoke common errors such as DNS resolution +errors, TCP connect errors, TLS handshake errors, and +HTTP round trip errors? How does the JSON change? + +## Conclusion + +We have seen how to combine DNS resolutions (chapter01 and +chapter03) with HTTPEndpoint GET (chapter06) to measure +all the HTTP endpoints for a given domain. + diff --git a/internal/tutorial/measurex/chapter07/main.go b/internal/tutorial/measurex/chapter07/main.go new file mode 100644 index 0000000..851df2d --- /dev/null +++ b/internal/tutorial/measurex/chapter07/main.go @@ -0,0 +1,144 @@ +// -=-=- StartHere -=-=- +// +// # Chapter VII: Measuring all the HTTPEndpoints for a domain +// +// We are now going to combine DNS resolutions with getting +// HTTPEndpoints. Conceptually, the DNS resolution yields +// us a list of IP addresses. For each address, we build the +// HTTPEndpoint and fetch it like we did in chapter06. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter07/main.go`.) +// +// ## main.go +// +// We have package declaration and imports as usual. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +// ``` +// +// Here we define an helper type for containing the DNS +// measurement and the subsequent endpoints measurements. +// +// ```Go +type measurement struct { + DNS *measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +// ``` +// +// The rest of the program is quite similar to what we had before. +// +// ```Go +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://google.com/", "URL to fetch") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() + // ``` + // + // This is where the main.go file starts to diverge. We create an + // instance of our measurement type to hold the results. + // + // ```Go + m := &measurement{} + // ``` + // + // Then we perform a DNS lookup using UDP like we saw in chapter03. + // + // ```Go + m.DNS = mx.LookupHostUDP(ctx, parsed.Hostname(), *address) + // ``` + // + // Like we did in the previous chapter, we create suitable HTTP + // headers for performing an HTTP measurement. + // + // ```Go + headers := measurex.NewHTTPRequestHeaderForMeasuring() + // ``` + // + // The following is an entirely new function we're learning + // about just now. `AllHTTPEndpointsForURL` is a free function + // in `measurex` that given: + // + // - an already parsed HTTP/HTTPS URL + // + // - headers we want to use + // + // - the result of one or more DNS queries + // + // builds us a list of HTTPEndpoint data structures. + // + // ```Go + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") + // ``` + // + // This function may fail if, for example, the URL is not HTTP/HTTPS. We + // handle the error panicking, because this is an example program. + // + // We are almost done now: we loop over all the endpoints and apply the + // `HTTPEndpointGetWithoutCookies` method we have seen in chapter06. + // + // ```Go + for _, epnt := range httpEndpoints { + m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt)) + } + // ``` + // + // Finally, we print the results. + // + // ```Go + print(m) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter07 +// ``` +// +// Please, check the JSON output. Do you recognize the fields +// we have described in previous chapters? +// +// Can you provoke common errors such as DNS resolution +// errors, TCP connect errors, TLS handshake errors, and +// HTTP round trip errors? How does the JSON change? +// +// ## Conclusion +// +// We have seen how to combine DNS resolutions (chapter01 and +// chapter03) with HTTPEndpoint GET (chapter06) to measure +// all the HTTP endpoints for a given domain. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter08/README.md b/internal/tutorial/measurex/chapter08/README.md new file mode 100644 index 0000000..5baba85 --- /dev/null +++ b/internal/tutorial/measurex/chapter08/README.md @@ -0,0 +1,132 @@ + +# Chapter VII: HTTPSSvc DNS queries + +The program we see here is _really_ similar to the one we +discussed in the previous chapter. The main difference +is the following: now we also issue HTTPSSvc DNS queries +to discover HTTP/3 endpoints. (Because HTTPSSvc is +still a draft and is mostly implemented by Cloudflare +at this point, we are going to use as the example +input URL a Cloudflare URL.) + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter08/main.go`.) + +## main.go + +The beginning of the program is pretty much the same. We +have just amended our `measurement` type to contain multiple +`DNSMeasurement` results. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + DNS []*measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() + m := &measurement{} +``` +### Call LookupHTTPSSvc + +Here we perform the `LookupHostUDP` we performed in the +previous chapter and then we call `LookupHTTPSvcUDP`. + +```Go + m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address)) + m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address)) +``` + +The `LookupHTTPSSvcUDP` function has the same signature +of `LookupHostUDP` _but_ it behaves differently. Rather than +querying for `A` and `AAAA`, it performs an `HTTPS` DNS +lookup. This query returns: + +1. a list of ALPNs for the domain; + +2. a list of IPv4 addresses; + +3. a list of IPv6 addresses. + +### Build an []HTTPEndpoint and run serial measurements + +Here we call `AllHTTPEndpointsForURL` like we did in the +previous chapter. However, note that we pass to it the +whole content of `m.DNS`, which now contains not only the +A/AAAA lookups results but also the HTTPS lookup results. + +The `AllHTTPEndpointsForURL` function will recognize that +we also have HTTPS lookups and, if the "h3" ALPN is +present, will _also_ build HTTP/3 endpoints using "quic" +as the `HTTPEndpoint.Network`. + +```Go + headers := measurex.NewHTTPRequestHeaderForMeasuring() + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") +``` + +This is it. The rest of the program is exactly the same. + +```Go + for _, epnt := range httpEndpoints { + m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt)) + } + print(m) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter08 +``` + +Please, check the JSON output. Do you recognize the fields +we have described in previous chapters? You should see +that, compared to previous chapters, now we're also testing +QUIC/HTTP3 endpoints. + +Can you provoke common errors such as DNS resolution +errors, TCP connect errors, TLS handshake errors, and +HTTP round trip errors? What is a good way to cause +timeout and SNI mismatch errors for QUIC? + +## Conclusion + +We have seen how to extend fetching all the HTTPS +endpoints to include the QUIC/HTTP3 endpoints discovered +using HTTPSSvc. + diff --git a/internal/tutorial/measurex/chapter08/main.go b/internal/tutorial/measurex/chapter08/main.go new file mode 100644 index 0000000..eb1c6cd --- /dev/null +++ b/internal/tutorial/measurex/chapter08/main.go @@ -0,0 +1,134 @@ +// -=-=- StartHere -=-=- +// +// # Chapter VII: HTTPSSvc DNS queries +// +// The program we see here is _really_ similar to the one we +// discussed in the previous chapter. The main difference +// is the following: now we also issue HTTPSSvc DNS queries +// to discover HTTP/3 endpoints. (Because HTTPSSvc is +// still a draft and is mostly implemented by Cloudflare +// at this point, we are going to use as the example +// input URL a Cloudflare URL.) +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter08/main.go`.) +// +// ## main.go +// +// The beginning of the program is pretty much the same. We +// have just amended our `measurement` type to contain multiple +// `DNSMeasurement` results. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + DNS []*measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() + m := &measurement{} + // ``` + // ### Call LookupHTTPSSvc + // + // Here we perform the `LookupHostUDP` we performed in the + // previous chapter and then we call `LookupHTTPSvcUDP`. + // + // ```Go + m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address)) + m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address)) + // ``` + // + // The `LookupHTTPSSvcUDP` function has the same signature + // of `LookupHostUDP` _but_ it behaves differently. Rather than + // querying for `A` and `AAAA`, it performs an `HTTPS` DNS + // lookup. This query returns: + // + // 1. a list of ALPNs for the domain; + // + // 2. a list of IPv4 addresses; + // + // 3. a list of IPv6 addresses. + // + // ### Build an []HTTPEndpoint and run serial measurements + // + // Here we call `AllHTTPEndpointsForURL` like we did in the + // previous chapter. However, note that we pass to it the + // whole content of `m.DNS`, which now contains not only the + // A/AAAA lookups results but also the HTTPS lookup results. + // + // The `AllHTTPEndpointsForURL` function will recognize that + // we also have HTTPS lookups and, if the "h3" ALPN is + // present, will _also_ build HTTP/3 endpoints using "quic" + // as the `HTTPEndpoint.Network`. + // + // ```Go + headers := measurex.NewHTTPRequestHeaderForMeasuring() + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") + // ``` + // + // This is it. The rest of the program is exactly the same. + // + // ```Go + for _, epnt := range httpEndpoints { + m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt)) + } + print(m) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter08 +// ``` +// +// Please, check the JSON output. Do you recognize the fields +// we have described in previous chapters? You should see +// that, compared to previous chapters, now we're also testing +// QUIC/HTTP3 endpoints. +// +// Can you provoke common errors such as DNS resolution +// errors, TCP connect errors, TLS handshake errors, and +// HTTP round trip errors? What is a good way to cause +// timeout and SNI mismatch errors for QUIC? +// +// ## Conclusion +// +// We have seen how to extend fetching all the HTTPS +// endpoints to include the QUIC/HTTP3 endpoints discovered +// using HTTPSSvc. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter09/README.md b/internal/tutorial/measurex/chapter09/README.md new file mode 100644 index 0000000..fe20445 --- /dev/null +++ b/internal/tutorial/measurex/chapter09/README.md @@ -0,0 +1,124 @@ + +# Chapter IX: Parallel HTTPEndpoint measurements + +The program we see here is _really_ similar to the one we +discussed in the previous chapter. The main difference +is the following: rather than looping through the list of +HTTPEndpoint, we call a function that runs through the +list of endpoints using a small pool of background workers. + +There is a trade off between quick measurements and +false positives. A timeout is one of the most common +ways of censoring HTTPS and HTTP3 endpoints. So, if +we run measurements sequentially, a whole scan could +in principle take a long time. On the other hand, +if we run too many parallel measurements, we may cause +our own congestion and maybe some measurements will +fail because of that. Our solution to this problem is +to have low parallelism: at the moment of writing +this note, we have three workers. If you submit +more than three HTTPEndpoint at a a time, we will +service the first three immediately and all the +other endpoints will be queued for later measurement. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter09/main.go`.) + +## main.go + +The beginning of the program is pretty much the same. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + DNS []*measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() + m := &measurement{} + m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address)) + m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address)) + headers := measurex.NewHTTPRequestHeaderForMeasuring() + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") +``` + +This is where the program changes. First, we need to create a jar +for cookies because the API we're about to call requires a +cookie jar. (We mostly use this API with redirects and we want +to have cookies with redirects because a small portion of the +URLs we typically test require cookies to properly redirect, +see https://github.com/ooni/probe/issues/1727 for more information). + +Then, we call `HTTPEndpointGetParallel`. The arguments are: + +- as usual, the context + +- the cookie jar + +- all the endpoints to measure + +```Go + cookies := measurex.NewCookieJar() + for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) { + m.Endpoints = append(m.Endpoints, epnt) + } +``` + +The `HTTPEndpointGetParallel` method returns a channel where it +posts `HTTPEndpointMeasurements`. Once the input list has been +fully measured, this method closes the returned channel. + +Like we did before, we append the resulting measurements to +our `m` container and we print it. + +```Go + print(m) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter09 +``` + +Take a look at the JSON output. Can you spot that +endpoints measurements are run in parallel? + +## Conclusion + +We have seen how to run HTTPEndpoint measurements in parallel. + diff --git a/internal/tutorial/measurex/chapter09/main.go b/internal/tutorial/measurex/chapter09/main.go new file mode 100644 index 0000000..f431ff3 --- /dev/null +++ b/internal/tutorial/measurex/chapter09/main.go @@ -0,0 +1,126 @@ +// -=-=- StartHere -=-=- +// +// # Chapter IX: Parallel HTTPEndpoint measurements +// +// The program we see here is _really_ similar to the one we +// discussed in the previous chapter. The main difference +// is the following: rather than looping through the list of +// HTTPEndpoint, we call a function that runs through the +// list of endpoints using a small pool of background workers. +// +// There is a trade off between quick measurements and +// false positives. A timeout is one of the most common +// ways of censoring HTTPS and HTTP3 endpoints. So, if +// we run measurements sequentially, a whole scan could +// in principle take a long time. On the other hand, +// if we run too many parallel measurements, we may cause +// our own congestion and maybe some measurements will +// fail because of that. Our solution to this problem is +// to have low parallelism: at the moment of writing +// this note, we have three workers. If you submit +// more than three HTTPEndpoint at a a time, we will +// service the first three immediately and all the +// other endpoints will be queued for later measurement. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter09/main.go`.) +// +// ## main.go +// +// The beginning of the program is pretty much the same. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + DNS []*measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch") + address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() + m := &measurement{} + m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address)) + m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address)) + headers := measurex.NewHTTPRequestHeaderForMeasuring() + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") + // ``` + // + // This is where the program changes. First, we need to create a jar + // for cookies because the API we're about to call requires a + // cookie jar. (We mostly use this API with redirects and we want + // to have cookies with redirects because a small portion of the + // URLs we typically test require cookies to properly redirect, + // see https://github.com/ooni/probe/issues/1727 for more information). + // + // Then, we call `HTTPEndpointGetParallel`. The arguments are: + // + // - as usual, the context + // + // - the cookie jar + // + // - all the endpoints to measure + // + // ```Go + cookies := measurex.NewCookieJar() + for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) { + m.Endpoints = append(m.Endpoints, epnt) + } + // ``` + // + // The `HTTPEndpointGetParallel` method returns a channel where it + // posts `HTTPEndpointMeasurements`. Once the input list has been + // fully measured, this method closes the returned channel. + // + // Like we did before, we append the resulting measurements to + // our `m` container and we print it. + // + // ```Go + print(m) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter09 +// ``` +// +// Take a look at the JSON output. Can you spot that +// endpoints measurements are run in parallel? +// +// ## Conclusion +// +// We have seen how to run HTTPEndpoint measurements in parallel. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter10/README.md b/internal/tutorial/measurex/chapter10/README.md new file mode 100644 index 0000000..d851a06 --- /dev/null +++ b/internal/tutorial/measurex/chapter10/README.md @@ -0,0 +1,127 @@ + +# Chapter IX: Parallel DNS lookups + +The program we see here is _really_ similar to the one we +discussed in the previous chapter. The main difference +is the following: rather than performing DNS lookups +sequentially, we call a function that runs through the +list of resolvers and run them in parallel. + +Again, we are going to use low parallelism for the same +rationale mentioned in chapter09. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter10/main.go`.) + +## main.go + +The beginning of the program is pretty much the same. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + DNS []*measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() + m := &measurement{} +``` + +The bulk of the difference is here. We create +a list of DNS resolvers. For each of them, we specify +the type and the endpoint address. (There is no +endpoint address for the system resolver, therefore +we leave its address empty.) + +```Go + resolvers := []*measurex.ResolverInfo{{ + Network: measurex.ResolverUDP, + Address: "8.8.8.8:53", + }, { + Network: measurex.ResolverUDP, + Address: "8.8.4.4:53", + }, { + Network: measurex.ResolverUDP, + Address: "1.1.1.1:53", + }, { + Network: measurex.ResolverUDP, + Address: "1.0.0.1:53", + }, { + Network: measurex.ResolverSystem, + Address: "", + }} +``` + +Then we call `LookupURLHostParallel`. This function runs +the queries that make sense given the input URL using a +pool of (currently three) background goroutines. + +When I say "queries that make sense", I mostly mean +that we only query for HTTPSSvc when the input URL +scheme is "https". Otherwise, if it's just "http", it +does not make sense to send this query. + +```Go + for dns := range mx.LookupURLHostParallel(ctx, parsed, resolvers...) { + m.DNS = append(m.DNS, dns) + } +``` + +The rest of the program is exactly like in chapter09. + +```Go + headers := measurex.NewHTTPRequestHeaderForMeasuring() + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") + cookies := measurex.NewCookieJar() + for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) { + m.Endpoints = append(m.Endpoints, epnt) + } + print(m) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter10 +``` + +Take a look at the JSON output. Can you spot that +DNS queries are run in parallel? + +## Conclusion + +We have seen how to run parallel DNS queries. + diff --git a/internal/tutorial/measurex/chapter10/main.go b/internal/tutorial/measurex/chapter10/main.go new file mode 100644 index 0000000..566100b --- /dev/null +++ b/internal/tutorial/measurex/chapter10/main.go @@ -0,0 +1,129 @@ +// -=-=- StartHere -=-=- +// +// # Chapter IX: Parallel DNS lookups +// +// The program we see here is _really_ similar to the one we +// discussed in the previous chapter. The main difference +// is the following: rather than performing DNS lookups +// sequentially, we call a function that runs through the +// list of resolvers and run them in parallel. +// +// Again, we are going to use low parallelism for the same +// rationale mentioned in chapter09. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter10/main.go`.) +// +// ## main.go +// +// The beginning of the program is pretty much the same. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + DNS []*measurex.DNSMeasurement + Endpoints []*measurex.HTTPEndpointMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + parsed, err := url.Parse(*URL) + runtimex.PanicOnError(err, "url.Parse failed") + mx := measurex.NewMeasurerWithDefaultSettings() + m := &measurement{} + // ``` + // + // The bulk of the difference is here. We create + // a list of DNS resolvers. For each of them, we specify + // the type and the endpoint address. (There is no + // endpoint address for the system resolver, therefore + // we leave its address empty.) + // + // ```Go + resolvers := []*measurex.ResolverInfo{{ + Network: measurex.ResolverUDP, + Address: "8.8.8.8:53", + }, { + Network: measurex.ResolverUDP, + Address: "8.8.4.4:53", + }, { + Network: measurex.ResolverUDP, + Address: "1.1.1.1:53", + }, { + Network: measurex.ResolverUDP, + Address: "1.0.0.1:53", + }, { + Network: measurex.ResolverSystem, + Address: "", + }} + // ``` + // + // Then we call `LookupURLHostParallel`. This function runs + // the queries that make sense given the input URL using a + // pool of (currently three) background goroutines. + // + // When I say "queries that make sense", I mostly mean + // that we only query for HTTPSSvc when the input URL + // scheme is "https". Otherwise, if it's just "http", it + // does not make sense to send this query. + // + // ```Go + for dns := range mx.LookupURLHostParallel(ctx, parsed, resolvers...) { + m.DNS = append(m.DNS, dns) + } + // ``` + // + // The rest of the program is exactly like in chapter09. + // + // ```Go + headers := measurex.NewHTTPRequestHeaderForMeasuring() + httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...) + runtimex.PanicOnError(err, "cannot get all the HTTP endpoints") + cookies := measurex.NewCookieJar() + for epnt := range mx.HTTPEndpointGetParallel(ctx, cookies, httpEndpoints...) { + m.Endpoints = append(m.Endpoints, epnt) + } + print(m) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter10 +// ``` +// +// Take a look at the JSON output. Can you spot that +// DNS queries are run in parallel? +// +// ## Conclusion +// +// We have seen how to run parallel DNS queries. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter11/README.md b/internal/tutorial/measurex/chapter11/README.md new file mode 100644 index 0000000..482e7dd --- /dev/null +++ b/internal/tutorial/measurex/chapter11/README.md @@ -0,0 +1,126 @@ + +# Chapter XI: Measuring a URL + +This program shows how to measure an HTTP/HTTPS URL. We +are going to call an API whose implementation is +basically the same code we have seen in the previous +chapter, to obtain an URL measurement in a more compact +way. (As an historical note, the API we are going to +call has indeed been written as a refactoring of +the code we introduced in the previous chapter.) + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter11/main.go`.) + +## main.go + +The beginning of the program is much simpler. We have removed +out custom measurement type. We are now going to use the +`URLMeasurement` type (`go doc ./internal/measurex.URLMeasurement`), +which as the same fields of `measurement` in chapter10 _plus_ +some extra fields that we'll examine in a later chapter. + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://www.google.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() +``` + +We create a measurer, cookies, and headers like we +saw in the previous chapter. + +```Go + mx := measurex.NewMeasurerWithDefaultSettings() + cookies := measurex.NewCookieJar() + headers := measurex.NewHTTPRequestHeaderForMeasuring() +``` + +Then we call `MeasureURL`. This function's implementation +is in `./internal/measurex/measurer.go` and is pretty +much a refactoring of the code in chapter10. + +The arguments are: + +- the context as usual + +- the unparsed URL to measure + +- the headers we want to use + +- a jar for cookies + +```Go + m, err := mx.MeasureURL(ctx, *URL, headers, cookies) +``` +The return value is either an `URLMeasurement` +or an error. The error happens, for example, if +the input URL scheme is not "http" or "https" (which +we handled by panicking in chapter11). + +Now, rather than panicking inside `MeasureURL`, we +return the error to the caller and we `panic` +here on `main` using the `PanicOnError` function. + +```Go + runtimex.PanicOnError(err, "mx.MeasureURL failed") + print(m) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter11 +``` + +Take a look at the JSON output and compare it with: + +```bash +go run -race ./internal/tutorial/measurex/chapter10 -url https://www.google.com +``` + +(which is basically forcing chapter10 to run with the +the default URL we use in this chapter). + +Can you explain why we are able to measure more endpoints +in this chapter by checking the implementation of `MeasureURL` +and compare it to the code written in chapter10? + +Now run: + +```bash +go run -race ./internal/tutorial/measurex/chapter11 -url https://google.com +``` + +Do you see the opportunity there for following redirections? :^). + +## Conclusion + +We have introduced `MeasureURL`, the top-level API for +measuring a single URL. + diff --git a/internal/tutorial/measurex/chapter11/main.go b/internal/tutorial/measurex/chapter11/main.go new file mode 100644 index 0000000..6f37a17 --- /dev/null +++ b/internal/tutorial/measurex/chapter11/main.go @@ -0,0 +1,128 @@ +// -=-=- StartHere -=-=- +// +// # Chapter XI: Measuring a URL +// +// This program shows how to measure an HTTP/HTTPS URL. We +// are going to call an API whose implementation is +// basically the same code we have seen in the previous +// chapter, to obtain an URL measurement in a more compact +// way. (As an historical note, the API we are going to +// call has indeed been written as a refactoring of +// the code we introduced in the previous chapter.) +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter11/main.go`.) +// +// ## main.go +// +// The beginning of the program is much simpler. We have removed +// out custom measurement type. We are now going to use the +// `URLMeasurement` type (`go doc ./internal/measurex.URLMeasurement`), +// which as the same fields of `measurement` in chapter10 _plus_ +// some extra fields that we'll examine in a later chapter. +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "https://www.google.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + // ``` + // + // We create a measurer, cookies, and headers like we + // saw in the previous chapter. + // + // ```Go + mx := measurex.NewMeasurerWithDefaultSettings() + cookies := measurex.NewCookieJar() + headers := measurex.NewHTTPRequestHeaderForMeasuring() + // ``` + // + // Then we call `MeasureURL`. This function's implementation + // is in `./internal/measurex/measurer.go` and is pretty + // much a refactoring of the code in chapter10. + // + // The arguments are: + // + // - the context as usual + // + // - the unparsed URL to measure + // + // - the headers we want to use + // + // - a jar for cookies + // + // ```Go + m, err := mx.MeasureURL(ctx, *URL, headers, cookies) + // ``` + // The return value is either an `URLMeasurement` + // or an error. The error happens, for example, if + // the input URL scheme is not "http" or "https" (which + // we handled by panicking in chapter11). + // + // Now, rather than panicking inside `MeasureURL`, we + // return the error to the caller and we `panic` + // here on `main` using the `PanicOnError` function. + // + // ```Go + runtimex.PanicOnError(err, "mx.MeasureURL failed") + print(m) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter11 +// ``` +// +// Take a look at the JSON output and compare it with: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter10 -url https://www.google.com +// ``` +// +// (which is basically forcing chapter10 to run with the +// the default URL we use in this chapter). +// +// Can you explain why we are able to measure more endpoints +// in this chapter by checking the implementation of `MeasureURL` +// and compare it to the code written in chapter10? +// +// Now run: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter11 -url https://google.com +// ``` +// +// Do you see the opportunity there for following redirections? :^). +// +// ## Conclusion +// +// We have introduced `MeasureURL`, the top-level API for +// measuring a single URL. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter12/README.md b/internal/tutorial/measurex/chapter12/README.md new file mode 100644 index 0000000..c5051ff --- /dev/null +++ b/internal/tutorial/measurex/chapter12/README.md @@ -0,0 +1,94 @@ + +# Chapter XII: Following redirections. + +This program shows how to combine the URL measurement +"step" introduced in the previous chapter with +following redirections. If we say that the previous +chapter performed a "web step", then we can say +that here we're performing multiple "web steps". + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter12/main.go`.) + +## main.go + +The beginning of the program is pretty much the +same, except that here we need to define a +`measurement` container type that will contain +the result of each "web step". + +```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + URLs []*measurex.URLMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "http://facebook.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + all := &measurement{} + mx := measurex.NewMeasurerWithDefaultSettings() + cookies := measurex.NewCookieJar() + headers := measurex.NewHTTPRequestHeaderForMeasuring() +``` + +Everything above this line is like in chapter11. What changes +now is that we're calling `MeasureURLAndFollowRedirections` +instead of `MeasureURL`. + +Rather than returning a single measurement, this function +returns a channel where it posts the result of measuring +the original URL along with all its redirections. Internally, +`MeasureURLAndFollowRedirections` calls `MeasureURL`. + +We accumulate the results in `URLs` and print `m`. The channel +is closed when done by `MeasureURLAndFollowRedirections`, so we leave the loop. + +```Go + for m := range mx.MeasureURLAndFollowRedirections(ctx, *URL, headers, cookies) { + all.URLs = append(all.URLs, m) + } + print(all) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter12 +``` + +Take a look at the JSON. You should see several redirects +and that we measure each endpoint of each redirect, including +QUIC endpoints that we discover on the way. + +## Conclusion + +We have introduced `MeasureURLAndFollowRedirect`, the +top-level API for fully measuring a URL and all the URLs +that derive from such an URL via redirection. + diff --git a/internal/tutorial/measurex/chapter12/main.go b/internal/tutorial/measurex/chapter12/main.go new file mode 100644 index 0000000..42a3638 --- /dev/null +++ b/internal/tutorial/measurex/chapter12/main.go @@ -0,0 +1,96 @@ +// -=-=- StartHere -=-=- +// +// # Chapter XII: Following redirections. +// +// This program shows how to combine the URL measurement +// "step" introduced in the previous chapter with +// following redirections. If we say that the previous +// chapter performed a "web step", then we can say +// that here we're performing multiple "web steps". +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter12/main.go`.) +// +// ## main.go +// +// The beginning of the program is pretty much the +// same, except that here we need to define a +// `measurement` container type that will contain +// the result of each "web step". +// +// ```Go +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +type measurement struct { + URLs []*measurex.URLMeasurement +} + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +func main() { + URL := flag.String("url", "http://facebook.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + all := &measurement{} + mx := measurex.NewMeasurerWithDefaultSettings() + cookies := measurex.NewCookieJar() + headers := measurex.NewHTTPRequestHeaderForMeasuring() + // ``` + // + // Everything above this line is like in chapter11. What changes + // now is that we're calling `MeasureURLAndFollowRedirections` + // instead of `MeasureURL`. + // + // Rather than returning a single measurement, this function + // returns a channel where it posts the result of measuring + // the original URL along with all its redirections. Internally, + // `MeasureURLAndFollowRedirections` calls `MeasureURL`. + // + // We accumulate the results in `URLs` and print `m`. The channel + // is closed when done by `MeasureURLAndFollowRedirections`, so we leave the loop. + // + // ```Go + for m := range mx.MeasureURLAndFollowRedirections(ctx, *URL, headers, cookies) { + all.URLs = append(all.URLs, m) + } + print(all) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter12 +// ``` +// +// Take a look at the JSON. You should see several redirects +// and that we measure each endpoint of each redirect, including +// QUIC endpoints that we discover on the way. +// +// ## Conclusion +// +// We have introduced `MeasureURLAndFollowRedirect`, the +// top-level API for fully measuring a URL and all the URLs +// that derive from such an URL via redirection. +// +// -=-=- StopHere -=-=- diff --git a/internal/tutorial/measurex/chapter13/README.md b/internal/tutorial/measurex/chapter13/README.md new file mode 100644 index 0000000..d31604e --- /dev/null +++ b/internal/tutorial/measurex/chapter13/README.md @@ -0,0 +1,97 @@ + +# Chapter XIII: Rewriting Web Connectivity + +This chapter contains an exercise. We are going to +use the `measurex` API to rewrite part of the +Web Connectivity network experiment. +(This is probably the right place to prod you +to go to the [ooni/spec](https://github.com/ooni/spec) +repository, locate the ts-017-web-connectivity.md +spec, and read it.) + +Read the spec? Good, so +what we are more precisely going to do here +is implement the network measurement part of +Web Connectivity where we: + +1. enumerate all the IP addresses of the target +URL using the system resolver; + +2. build endpoints with such IPs with a suitable +port, thus obtaining a list of HTTP endpoints; + +3. TCP connect each of the endpoints and save the +results into a measurement object compatible +with Web Connectivity's data format; + +4. TLS handshake each endpoint (only if this +makes sense, of course); + +5. HTTP GET the URL and follow redirects until +we reach a webpage, fetch the body, and store it +for later analysis (which we'll not implement +as part of this exercise). + +Let us now provide extra context that should +help you figure out how to solve this exercise. + +## Regarding points 3-4 + +You already know all the primitives. + +## Regarding point 5 + +Historically this point has always been +performed by a separate HTTP client. This +means that any implementation: + +- will not include any TCP or TLS event +generated during point 5 in the measurement; + +- most likely will resolve the URL's domain +again (even though the probe-cli implementation +uses a fake Resolver to avoid that); + +- tries every available IP address and stops +at the first one to which it can connect to (which +is what a naive HTTP client does, whereas a more +advanced one likely tries a couple of addrs in +parallel, especially when both IPv4 and IPv6 +are supported - this is also known as happy eyeballs). + +In terms of `measurex`, the best API to do what +you're required to do in point 5 is probably +`NewTracingHTTPTransportWithDefaultSettings`, which +allows you to trace only the HTTP round trip and +ignores any other event. + +Once you have such a transport, the best `Measurer` +API for the task is probably `HTTPClientGET`. + +## Other remarks + +You also need to learn about how to measure +events at low level, which entails creating an +instance of `MeasurementDB`, passing it to +the relevant networking code, and then calling +its `AsMeasurement` method to get back a +measurement. (You can probably get an idea +of how this is done in general by checking the +implementation of `Measurer.TCPConnect`.) + +Hopefully, this should be enough information +to help you tackle this task. As you see +below, the main function is there empty waiting +for your implementation. We will provide our +own solution to this problem in the next chapter. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter13/main.go`.) + +## The main.go file + +```Go +package main + +func main() { +} diff --git a/internal/tutorial/measurex/chapter13/main.go b/internal/tutorial/measurex/chapter13/main.go new file mode 100644 index 0000000..1aff64c --- /dev/null +++ b/internal/tutorial/measurex/chapter13/main.go @@ -0,0 +1,98 @@ +// -=-=- StartHere -=-=- +// +// # Chapter XIII: Rewriting Web Connectivity +// +// This chapter contains an exercise. We are going to +// use the `measurex` API to rewrite part of the +// Web Connectivity network experiment. +// (This is probably the right place to prod you +// to go to the [ooni/spec](https://github.com/ooni/spec) +// repository, locate the ts-017-web-connectivity.md +// spec, and read it.) +// +// Read the spec? Good, so +// what we are more precisely going to do here +// is implement the network measurement part of +// Web Connectivity where we: +// +// 1. enumerate all the IP addresses of the target +// URL using the system resolver; +// +// 2. build endpoints with such IPs with a suitable +// port, thus obtaining a list of HTTP endpoints; +// +// 3. TCP connect each of the endpoints and save the +// results into a measurement object compatible +// with Web Connectivity's data format; +// +// 4. TLS handshake each endpoint (only if this +// makes sense, of course); +// +// 5. HTTP GET the URL and follow redirects until +// we reach a webpage, fetch the body, and store it +// for later analysis (which we'll not implement +// as part of this exercise). +// +// Let us now provide extra context that should +// help you figure out how to solve this exercise. +// +// ## Regarding points 3-4 +// +// You already know all the primitives. +// +// ## Regarding point 5 +// +// Historically this point has always been +// performed by a separate HTTP client. This +// means that any implementation: +// +// - will not include any TCP or TLS event +// generated during point 5 in the measurement; +// +// - most likely will resolve the URL's domain +// again (even though the probe-cli implementation +// uses a fake Resolver to avoid that); +// +// - tries every available IP address and stops +// at the first one to which it can connect to (which +// is what a naive HTTP client does, whereas a more +// advanced one likely tries a couple of addrs in +// parallel, especially when both IPv4 and IPv6 +// are supported - this is also known as happy eyeballs). +// +// In terms of `measurex`, the best API to do what +// you're required to do in point 5 is probably +// `NewTracingHTTPTransportWithDefaultSettings`, which +// allows you to trace only the HTTP round trip and +// ignores any other event. +// +// Once you have such a transport, the best `Measurer` +// API for the task is probably `HTTPClientGET`. +// +// ## Other remarks +// +// You also need to learn about how to measure +// events at low level, which entails creating an +// instance of `MeasurementDB`, passing it to +// the relevant networking code, and then calling +// its `AsMeasurement` method to get back a +// measurement. (You can probably get an idea +// of how this is done in general by checking the +// implementation of `Measurer.TCPConnect`.) +// +// Hopefully, this should be enough information +// to help you tackle this task. As you see +// below, the main function is there empty waiting +// for your implementation. We will provide our +// own solution to this problem in the next chapter. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter13/main.go`.) +// +// ## The main.go file +// +// ```Go +package main + +func main() { +} diff --git a/internal/tutorial/measurex/chapter14/README.md b/internal/tutorial/measurex/chapter14/README.md new file mode 100644 index 0000000..348a4d4 --- /dev/null +++ b/internal/tutorial/measurex/chapter14/README.md @@ -0,0 +1,318 @@ + +# Chapter XIV: A possible rewrite of Web Connectivity + +In this chapter we try to solve the exercise laid out in +the previous chapter, using `measurex` primitives. + +(This file is auto-generated. Do not edit it directly! To apply +changes you need to modify `./internal/tutorial/measurex/chapter14/main.go`.) + +## main.go + +The beginning of the file is always pretty much the same. + +```Go +package main + +import ( + "context" + "crypto/tls" + "encoding/json" + "flag" + "fmt" + "net/http" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/netxlite" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +``` + +## measurement type + +We define a measurement type with the fields +that a Web Connectivity measurement should have. + +```Go + +type measurement struct { + Queries []*measurex.DNSLookupEvent `json:"queries"` + TCPConnect []*measurex.NetworkEvent `json:"tcp_connect"` + TLSHandshakes []*measurex.TLSHandshakeEvent `json:"tls_handshakes"` + Requests []*measurex.HTTPRoundTripEvent `json:"requests"` +} + +``` + +## WebConnectivity implementation + +We define a function that takes in input a context and a URL to +measure and returns a measurement or an error. + +We will only error out in case the input does not allow us to +proceed (i.e., invalid input URL). + +```Go + +func webConnectivity(ctx context.Context, URL string) (*measurement, error) { +``` + +We start by parsing the input URL. If we cannot parse it, of +course this is an hard error and we cannot continue. + +```Go + parsedURL, err := url.Parse(URL) + if err != nil { + return nil, err + } + +``` + +We create an empty measurement and a measurer with +default settings like we did in the previous chapters. + +```Go + m := &measurement{} + mx := measurex.NewMeasurerWithDefaultSettings() + +``` + +Now it's time to start measuring. We will address all +the points laid out in the previous chapter. + +### 1. Enumerating IP addrs + +Let us enumerate all the IP addresses for +the input URL's domain using the system resolver. + +```Go + dns := mx.LookupHostSystem(ctx, parsedURL.Hostname()) + m.Queries = append(m.Queries, dns.LookupHost...) + +``` + +This is code we have already seen in previous chapter. + + +### 2. Building a list of endpoints + +```Go + epnts, err := measurex.AllHTTPEndpointsForURL(parsedURL, http.Header{}, dns) + if err != nil { + return nil, err + } + +``` + +This is also code we have seen in previous chapters. The only +difference is that we supply empty headers since we're not going +to actually use the headers inside the endpoints. + +### 3 and 4. Measure each endpoint + +We will loop through the endpoints in the previous point +and issue the correct TCP or TLS primitive depending on +whether the input URL is HTTP or HTTPS. + +```Go + for _, epnt := range epnts { + switch parsedURL.Scheme { + case "http": + tcp := mx.TCPConnect(ctx, epnt.Address) + m.TCPConnect = append(m.TCPConnect, tcp.Connect...) + case "https": + config := &tls.Config{ + ServerName: parsedURL.Hostname(), + NextProtos: []string{"h2", "http/1.1"}, + RootCAs: netxlite.NewDefaultCertPool(), + } + tls := mx.TLSConnectAndHandshake(ctx, epnt.Address, config) + m.TCPConnect = append(m.TCPConnect, tls.Connect...) + m.TLSHandshakes = append(m.TLSHandshakes, tls.TLSHandshake...) + } + } + +``` + +At this point we've addressed points 1-4. So let's +now focus on the last point: + +### 5. HTTP measurement + +We need to manually build a `MeasurementDB`. This is a +"database" where networking code will store events. + +```Go + + db := &measurex.MeasurementDB{} + +``` + +Following the hint from the previous chapter we use the +`NewTracingHTTPTransportWithDefaultSettings` factory +to create an `http.Transport`-like object that will trace +HTTP round trip events writing them into `db`. + + +```Go + + txp := measurex.NewTracingHTTPTransportWithDefaultSettings(mx.Begin, mx.Logger, db) + +``` + +We now build an `http.Client` using the transport +we've just created and a cookie jar (which we +use because otherwise some redirects will lead +to a redirect loop, as mentioned in previous chapters). + +```Go + + clnt := &http.Client{ + Transport: txp, + Jar: measurex.NewCookieJar(), + } + +``` + +Now we use a method of the measurer that allows us to +perform an HTTP GET with an existing HTTP client +and a URL. This method will set a timeout and perform +the round trip. Reading a snapshot of the response +body is not implemented by this function but rather +is a property of the "tracing" HTTP transport we +created above (this type of transport is the one we +have been internally using in all the examples +presented so far.) + +```Go + + resp, _ := mx.HTTPClientGET(ctx, clnt, parsedURL) + +``` + +To be tidy, we also close the response body in case +we have a response. We don't really need to read +the body here. As mentioned previously, we're already +using an HTTP transport reading a body snapshot. + +```Go + + if resp != nil { + resp.Body.Close() // tidy + } + +``` + +Finally, we append the round trips we performed into +the right field and return the measurement. + +To this end, we're using the `db.AsMeasurement` method that +takes the current set of events into `db` and assembles +them into the `Measurement` struct we've been using in all +the chapters we have seen so far. + +```Go + + m.Requests = append(m.Requests, db.AsMeasurement().HTTPRoundTrip...) + return m, nil +} + +``` + +The rest of the program is pretty straightforward. + +```Go + +func main() { + URL := flag.String("url", "https://www.google.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + m, err := webConnectivity(ctx, *URL) + runtimex.PanicOnError(err, "invalid arguments to webConnectivity (wrong URL?)") + print(m) +} + +``` + +## Running the example program + +Let us perform a vanilla run first: + +```bash +go run -race ./internal/tutorial/measurex/chapter14 +``` + +Take a look at the JSON. + +Now try running the program with `http://gmail.com` as +input. Take note of the redirect chain. See how the +domain changes during the redirect. Take note of the +fact that we are not measuring any TLS handshake. See +how we're not trying QUIC endpoints. These are, in +fact, some of the limitations of Web Connectivity that +we were trying to address when we wrote `measurex`. + +Also, build the miniooni research client: + +``` +go build -v ./internal/cmd/miniooni +``` + +Run Web Connectivity with: + +``` +./miniooni -ni http://gmail.com web_connectivity +``` + +This writes the report in a file named `report.jsonl`. + +Check the content of the file and match it with the +output of this chapter. Are there other notable +differences between the two outputs? + +### Bonus question + +The solution we presented is true to the original +spirit of Web Connectivity, where we first perform +separate DNS, TCP/TLS steps, and then we also +perform a separate HTTP step. Is there in `measurex` +an API allowing you to invert the order of the +operations, that is: + +1. build a full-fledged HTTP client where we can +trace _any_ operation; + +2. use such client to measure the URL; + +3. figure out what TCP endpoints we did not +test for TCP/TLS during this process and run +TCP/TLS testing only for them? + +If such an API exist, can you write a simple +main.go client that implements points 1-3 above? + +## Conclusion + +We have presented the solution to the exercise +proposed in the previous chapter, i.e., how +to rewrite Web Connectivity using `measurex` API. + +You have now been exposed to some complexity and +APIs to perform OONI measurements. So you should now +be read to help us write new and maitain existing +network experiments. + +If you have further questions, please [contact us]( +https://ooni.org/about/). + diff --git a/internal/tutorial/measurex/chapter14/main.go b/internal/tutorial/measurex/chapter14/main.go new file mode 100644 index 0000000..0c07301 --- /dev/null +++ b/internal/tutorial/measurex/chapter14/main.go @@ -0,0 +1,320 @@ +// -=-=- StartHere -=-=- +// +// # Chapter XIV: A possible rewrite of Web Connectivity +// +// In this chapter we try to solve the exercise laid out in +// the previous chapter, using `measurex` primitives. +// +// (This file is auto-generated. Do not edit it directly! To apply +// changes you need to modify `./internal/tutorial/measurex/chapter14/main.go`.) +// +// ## main.go +// +// The beginning of the file is always pretty much the same. +// +// ```Go +package main + +import ( + "context" + "crypto/tls" + "encoding/json" + "flag" + "fmt" + "net/http" + "net/url" + "time" + + "github.com/ooni/probe-cli/v3/internal/measurex" + "github.com/ooni/probe-cli/v3/internal/netxlite" + "github.com/ooni/probe-cli/v3/internal/runtimex" +) + +func print(v interface{}) { + data, err := json.Marshal(v) + runtimex.PanicOnError(err, "json.Marshal failed") + fmt.Printf("%s\n", string(data)) +} + +// ``` +// +// ## measurement type +// +// We define a measurement type with the fields +// that a Web Connectivity measurement should have. +// +// ```Go + +type measurement struct { + Queries []*measurex.DNSLookupEvent `json:"queries"` + TCPConnect []*measurex.NetworkEvent `json:"tcp_connect"` + TLSHandshakes []*measurex.TLSHandshakeEvent `json:"tls_handshakes"` + Requests []*measurex.HTTPRoundTripEvent `json:"requests"` +} + +// ``` +// +// ## WebConnectivity implementation +// +// We define a function that takes in input a context and a URL to +// measure and returns a measurement or an error. +// +// We will only error out in case the input does not allow us to +// proceed (i.e., invalid input URL). +// +// ```Go + +func webConnectivity(ctx context.Context, URL string) (*measurement, error) { + // ``` + // + // We start by parsing the input URL. If we cannot parse it, of + // course this is an hard error and we cannot continue. + // + // ```Go + parsedURL, err := url.Parse(URL) + if err != nil { + return nil, err + } + + // ``` + // + // We create an empty measurement and a measurer with + // default settings like we did in the previous chapters. + // + // ```Go + m := &measurement{} + mx := measurex.NewMeasurerWithDefaultSettings() + + // ``` + // + // Now it's time to start measuring. We will address all + // the points laid out in the previous chapter. + // + // ### 1. Enumerating IP addrs + // + // Let us enumerate all the IP addresses for + // the input URL's domain using the system resolver. + // + // ```Go + dns := mx.LookupHostSystem(ctx, parsedURL.Hostname()) + m.Queries = append(m.Queries, dns.LookupHost...) + + // ``` + // + // This is code we have already seen in previous chapter. + // + // + // ### 2. Building a list of endpoints + // + // ```Go + epnts, err := measurex.AllHTTPEndpointsForURL(parsedURL, http.Header{}, dns) + if err != nil { + return nil, err + } + + // ``` + // + // This is also code we have seen in previous chapters. The only + // difference is that we supply empty headers since we're not going + // to actually use the headers inside the endpoints. + // + // ### 3 and 4. Measure each endpoint + // + // We will loop through the endpoints in the previous point + // and issue the correct TCP or TLS primitive depending on + // whether the input URL is HTTP or HTTPS. + // + // ```Go + for _, epnt := range epnts { + switch parsedURL.Scheme { + case "http": + tcp := mx.TCPConnect(ctx, epnt.Address) + m.TCPConnect = append(m.TCPConnect, tcp.Connect...) + case "https": + config := &tls.Config{ + ServerName: parsedURL.Hostname(), + NextProtos: []string{"h2", "http/1.1"}, + RootCAs: netxlite.NewDefaultCertPool(), + } + tls := mx.TLSConnectAndHandshake(ctx, epnt.Address, config) + m.TCPConnect = append(m.TCPConnect, tls.Connect...) + m.TLSHandshakes = append(m.TLSHandshakes, tls.TLSHandshake...) + } + } + + // ``` + // + // At this point we've addressed points 1-4. So let's + // now focus on the last point: + // + // ### 5. HTTP measurement + // + // We need to manually build a `MeasurementDB`. This is a + // "database" where networking code will store events. + // + // ```Go + + db := &measurex.MeasurementDB{} + + // ``` + // + // Following the hint from the previous chapter we use the + // `NewTracingHTTPTransportWithDefaultSettings` factory + // to create an `http.Transport`-like object that will trace + // HTTP round trip events writing them into `db`. + // + // + // ```Go + + txp := measurex.NewTracingHTTPTransportWithDefaultSettings(mx.Begin, mx.Logger, db) + + // ``` + // + // We now build an `http.Client` using the transport + // we've just created and a cookie jar (which we + // use because otherwise some redirects will lead + // to a redirect loop, as mentioned in previous chapters). + // + // ```Go + + clnt := &http.Client{ + Transport: txp, + Jar: measurex.NewCookieJar(), + } + + // ``` + // + // Now we use a method of the measurer that allows us to + // perform an HTTP GET with an existing HTTP client + // and a URL. This method will set a timeout and perform + // the round trip. Reading a snapshot of the response + // body is not implemented by this function but rather + // is a property of the "tracing" HTTP transport we + // created above (this type of transport is the one we + // have been internally using in all the examples + // presented so far.) + // + // ```Go + + resp, _ := mx.HTTPClientGET(ctx, clnt, parsedURL) + + // ``` + // + // To be tidy, we also close the response body in case + // we have a response. We don't really need to read + // the body here. As mentioned previously, we're already + // using an HTTP transport reading a body snapshot. + // + // ```Go + + if resp != nil { + resp.Body.Close() // tidy + } + + // ``` + // + // Finally, we append the round trips we performed into + // the right field and return the measurement. + // + // To this end, we're using the `db.AsMeasurement` method that + // takes the current set of events into `db` and assembles + // them into the `Measurement` struct we've been using in all + // the chapters we have seen so far. + // + // ```Go + + m.Requests = append(m.Requests, db.AsMeasurement().HTTPRoundTrip...) + return m, nil +} + +// ``` +// +// The rest of the program is pretty straightforward. +// +// ```Go + +func main() { + URL := flag.String("url", "https://www.google.com/", "URL to fetch") + timeout := flag.Duration("timeout", 60*time.Second, "timeout to use") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), *timeout) + defer cancel() + m, err := webConnectivity(ctx, *URL) + runtimex.PanicOnError(err, "invalid arguments to webConnectivity (wrong URL?)") + print(m) +} + +// ``` +// +// ## Running the example program +// +// Let us perform a vanilla run first: +// +// ```bash +// go run -race ./internal/tutorial/measurex/chapter14 +// ``` +// +// Take a look at the JSON. +// +// Now try running the program with `http://gmail.com` as +// input. Take note of the redirect chain. See how the +// domain changes during the redirect. Take note of the +// fact that we are not measuring any TLS handshake. See +// how we're not trying QUIC endpoints. These are, in +// fact, some of the limitations of Web Connectivity that +// we were trying to address when we wrote `measurex`. +// +// Also, build the miniooni research client: +// +// ``` +// go build -v ./internal/cmd/miniooni +// ``` +// +// Run Web Connectivity with: +// +// ``` +// ./miniooni -ni http://gmail.com web_connectivity +// ``` +// +// This writes the report in a file named `report.jsonl`. +// +// Check the content of the file and match it with the +// output of this chapter. Are there other notable +// differences between the two outputs? +// +// ### Bonus question +// +// The solution we presented is true to the original +// spirit of Web Connectivity, where we first perform +// separate DNS, TCP/TLS steps, and then we also +// perform a separate HTTP step. Is there in `measurex` +// an API allowing you to invert the order of the +// operations, that is: +// +// 1. build a full-fledged HTTP client where we can +// trace _any_ operation; +// +// 2. use such client to measure the URL; +// +// 3. figure out what TCP endpoints we did not +// test for TCP/TLS during this process and run +// TCP/TLS testing only for them? +// +// If such an API exist, can you write a simple +// main.go client that implements points 1-3 above? +// +// ## Conclusion +// +// We have presented the solution to the exercise +// proposed in the previous chapter, i.e., how +// to rewrite Web Connectivity using `measurex` API. +// +// You have now been exposed to some complexity and +// APIs to perform OONI measurements. So you should now +// be read to help us write new and maitain existing +// network experiments. +// +// If you have further questions, please [contact us]( +// https://ooni.org/about/). +// +// -=-=- StopHere -=-=-