c/ooni-probe-cli
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

doc(measurex): explain how to write experiments (#529)

Browse Source 
Part of https://github.com/ooni/ooni.org/issues/361

Co-authored-by: Arturo Filastò <arturo@openobservatory.org>
This commit is contained in:
Simone Basso
2021-09-30 01:36:03 +02:00
committed by GitHub
parent 399d2f65da
commit d45e58c14f
31 changed files with 6822 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/tutorial/measurex/chapter08/README.md
+132
View File
@@ -0,0 +1,132 @@
# Chapter VII: HTTPSSvc DNS queries
The program we see here is _really_ similar to the one we
discussed in the previous chapter. The main difference
is the following: now we also issue HTTPSSvc DNS queries
to discover HTTP/3 endpoints. (Because HTTPSSvc is
still a draft and is mostly implemented by Cloudflare
at this point, we are going to use as the example
input URL a Cloudflare URL.)
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter08/main.go`.)
## main.go
The beginning of the program is pretty much the same. We
have just amended our `measurement` type to contain multiple
`DNSMeasurement` results.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
```
### Call LookupHTTPSSvc
Here we perform the `LookupHostUDP` we performed in the
previous chapter and then we call `LookupHTTPSvcUDP`.
```Go
m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address))
m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address))
```
The `LookupHTTPSSvcUDP` function has the same signature
of `LookupHostUDP` _but_ it behaves differently. Rather than
querying for `A` and `AAAA`, it performs an `HTTPS` DNS
lookup. This query returns:
1. a list of ALPNs for the domain;
2. a list of IPv4 addresses;
3. a list of IPv6 addresses.
### Build an []HTTPEndpoint and run serial measurements
Here we call `AllHTTPEndpointsForURL` like we did in the
previous chapter. However, note that we pass to it the
whole content of `m.DNS`, which now contains not only the
A/AAAA lookups results but also the HTTPS lookup results.
The `AllHTTPEndpointsForURL` function will recognize that
we also have HTTPS lookups and, if the "h3" ALPN is
present, will _also_ build HTTP/3 endpoints using "quic"
as the `HTTPEndpoint.Network`.
```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
```
This is it. The rest of the program is exactly the same.
```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter08
```
Please, check the JSON output. Do you recognize the fields
we have described in previous chapters? You should see
that, compared to previous chapters, now we're also testing
QUIC/HTTP3 endpoints.
Can you provoke common errors such as DNS resolution
errors, TCP connect errors, TLS handshake errors, and
HTTP round trip errors? What is a good way to cause
timeout and SNI mismatch errors for QUIC?
## Conclusion
We have seen how to extend fetching all the HTTPS
endpoints to include the QUIC/HTTP3 endpoints discovered
using HTTPSSvc.
internal/tutorial/measurex/chapter08/main.go
+134
View File
@@ -0,0 +1,134 @@
// -=-=- StartHere -=-=-
//
// # Chapter VII: HTTPSSvc DNS queries
//
// The program we see here is _really_ similar to the one we
// discussed in the previous chapter. The main difference
// is the following: now we also issue HTTPSSvc DNS queries
// to discover HTTP/3 endpoints. (Because HTTPSSvc is
// still a draft and is mostly implemented by Cloudflare
// at this point, we are going to use as the example
// input URL a Cloudflare URL.)
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter08/main.go`.)
//
// ## main.go
//
// The beginning of the program is pretty much the same. We
// have just amended our `measurement` type to contain multiple
// `DNSMeasurement` results.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
type measurement struct {
DNS []*measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
m := &measurement{}
// ```
// ### Call LookupHTTPSSvc
//
// Here we perform the `LookupHostUDP` we performed in the
// previous chapter and then we call `LookupHTTPSvcUDP`.
//
// ```Go
m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address))
m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address))
// ```
//
// The `LookupHTTPSSvcUDP` function has the same signature
// of `LookupHostUDP` _but_ it behaves differently. Rather than
// querying for `A` and `AAAA`, it performs an `HTTPS` DNS
// lookup. This query returns:
//
// 1. a list of ALPNs for the domain;
//
// 2. a list of IPv4 addresses;
//
// 3. a list of IPv6 addresses.
//
// ### Build an []HTTPEndpoint and run serial measurements
//
// Here we call `AllHTTPEndpointsForURL` like we did in the
// previous chapter. However, note that we pass to it the
// whole content of `m.DNS`, which now contains not only the
// A/AAAA lookups results but also the HTTPS lookup results.
//
// The `AllHTTPEndpointsForURL` function will recognize that
// we also have HTTPS lookups and, if the "h3" ALPN is
// present, will _also_ build HTTP/3 endpoints using "quic"
// as the `HTTPEndpoint.Network`.
//
// ```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
// ```
//
// This is it. The rest of the program is exactly the same.
//
// ```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter08
// ```
//
// Please, check the JSON output. Do you recognize the fields
// we have described in previous chapters? You should see
// that, compared to previous chapters, now we're also testing
// QUIC/HTTP3 endpoints.
//
// Can you provoke common errors such as DNS resolution
// errors, TCP connect errors, TLS handshake errors, and
// HTTP round trip errors? What is a good way to cause
// timeout and SNI mismatch errors for QUIC?
//
// ## Conclusion
//
// We have seen how to extend fetching all the HTTPS
// endpoints to include the QUIC/HTTP3 endpoints discovered
// using HTTPSSvc.
//
// -=-=- StopHere -=-=-