c/ooni-probe-cli
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

doc(measurex): explain how to write experiments (#529)

Browse Source 
Part of https://github.com/ooni/ooni.org/issues/361

Co-authored-by: Arturo Filastò <arturo@openobservatory.org>
This commit is contained in:
Simone Basso
2021-09-30 01:36:03 +02:00
committed by GitHub
parent 399d2f65da
commit d45e58c14f
31 changed files with 6822 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/tutorial/measurex/chapter07/README.md
+142
View File
@@ -0,0 +1,142 @@
# Chapter VII: Measuring all the HTTPEndpoints for a domain
We are now going to combine DNS resolutions with getting
HTTPEndpoints. Conceptually, the DNS resolution yields
us a list of IP addresses. For each address, we build the
HTTPEndpoint and fetch it like we did in chapter06.
(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter07/main.go`.)
## main.go
We have package declaration and imports as usual.
```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
```
Here we define an helper type for containing the DNS
measurement and the subsequent endpoints measurements.
```Go
type measurement struct {
DNS *measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
```
The rest of the program is quite similar to what we had before.
```Go
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://google.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
```
This is where the main.go file starts to diverge. We create an
instance of our measurement type to hold the results.
```Go
m := &measurement{}
```
Then we perform a DNS lookup using UDP like we saw in chapter03.
```Go
m.DNS = mx.LookupHostUDP(ctx, parsed.Hostname(), *address)
```
Like we did in the previous chapter, we create suitable HTTP
headers for performing an HTTP measurement.
```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
```
The following is an entirely new function we're learning
about just now. `AllHTTPEndpointsForURL` is a free function
in `measurex` that given:
- an already parsed HTTP/HTTPS URL
- headers we want to use
- the result of one or more DNS queries
builds us a list of HTTPEndpoint data structures.
```Go
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
```
This function may fail if, for example, the URL is not HTTP/HTTPS. We
handle the error panicking, because this is an example program.
We are almost done now: we loop over all the endpoints and apply the
`HTTPEndpointGetWithoutCookies` method we have seen in chapter06.
```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
```
Finally, we print the results.
```Go
print(m)
}
```
## Running the example program
Let us perform a vanilla run first:
```bash
go run -race ./internal/tutorial/measurex/chapter07
```
Please, check the JSON output. Do you recognize the fields
we have described in previous chapters?
Can you provoke common errors such as DNS resolution
errors, TCP connect errors, TLS handshake errors, and
HTTP round trip errors? How does the JSON change?
## Conclusion
We have seen how to combine DNS resolutions (chapter01 and
chapter03) with HTTPEndpoint GET (chapter06) to measure
all the HTTP endpoints for a given domain.
internal/tutorial/measurex/chapter07/main.go
+144
View File
@@ -0,0 +1,144 @@
// -=-=- StartHere -=-=-
//
// # Chapter VII: Measuring all the HTTPEndpoints for a domain
//
// We are now going to combine DNS resolutions with getting
// HTTPEndpoints. Conceptually, the DNS resolution yields
// us a list of IP addresses. For each address, we build the
// HTTPEndpoint and fetch it like we did in chapter06.
//
// (This file is auto-generated. Do not edit it directly! To apply
// changes you need to modify `./internal/tutorial/measurex/chapter07/main.go`.)
//
// ## main.go
//
// We have package declaration and imports as usual.
//
// ```Go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"net/url"
"time"
"github.com/ooni/probe-cli/v3/internal/measurex"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
// ```
//
// Here we define an helper type for containing the DNS
// measurement and the subsequent endpoints measurements.
//
// ```Go
type measurement struct {
DNS *measurex.DNSMeasurement
Endpoints []*measurex.HTTPEndpointMeasurement
}
// ```
//
// The rest of the program is quite similar to what we had before.
//
// ```Go
func print(v interface{}) {
data, err := json.Marshal(v)
runtimex.PanicOnError(err, "json.Marshal failed")
fmt.Printf("%s\n", string(data))
}
func main() {
URL := flag.String("url", "https://google.com/", "URL to fetch")
address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
defer cancel()
parsed, err := url.Parse(*URL)
runtimex.PanicOnError(err, "url.Parse failed")
mx := measurex.NewMeasurerWithDefaultSettings()
// ```
//
// This is where the main.go file starts to diverge. We create an
// instance of our measurement type to hold the results.
//
// ```Go
m := &measurement{}
// ```
//
// Then we perform a DNS lookup using UDP like we saw in chapter03.
//
// ```Go
m.DNS = mx.LookupHostUDP(ctx, parsed.Hostname(), *address)
// ```
//
// Like we did in the previous chapter, we create suitable HTTP
// headers for performing an HTTP measurement.
//
// ```Go
headers := measurex.NewHTTPRequestHeaderForMeasuring()
// ```
//
// The following is an entirely new function we're learning
// about just now. `AllHTTPEndpointsForURL` is a free function
// in `measurex` that given:
//
// - an already parsed HTTP/HTTPS URL
//
// - headers we want to use
//
// - the result of one or more DNS queries
//
// builds us a list of HTTPEndpoint data structures.
//
// ```Go
httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS)
runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")
// ```
//
// This function may fail if, for example, the URL is not HTTP/HTTPS. We
// handle the error panicking, because this is an example program.
//
// We are almost done now: we loop over all the endpoints and apply the
// `HTTPEndpointGetWithoutCookies` method we have seen in chapter06.
//
// ```Go
for _, epnt := range httpEndpoints {
m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
}
// ```
//
// Finally, we print the results.
//
// ```Go
print(m)
}
// ```
//
// ## Running the example program
//
// Let us perform a vanilla run first:
//
// ```bash
// go run -race ./internal/tutorial/measurex/chapter07
// ```
//
// Please, check the JSON output. Do you recognize the fields
// we have described in previous chapters?
//
// Can you provoke common errors such as DNS resolution
// errors, TCP connect errors, TLS handshake errors, and
// HTTP round trip errors? How does the JSON change?
//
// ## Conclusion
//
// We have seen how to combine DNS resolutions (chapter01 and
// chapter03) with HTTPEndpoint GET (chapter06) to measure
// all the HTTP endpoints for a given domain.
//
// -=-=- StopHere -=-=-