ooni-probe-cli/internal/tutorial/measurex/chapter08

Chapter VII: HTTPSSvc DNS queries

The program we see here is really similar to the one we discussed in the previous chapter. The main difference is the following: now we also issue HTTPSSvc DNS queries to discover HTTP/3 endpoints. (Because HTTPSSvc is still a draft and is mostly implemented by Cloudflare at this point, we are going to use as the example input URL a Cloudflare URL.)

(This file is auto-generated. Do not edit it directly! To apply changes you need to modify ./internal/tutorial/measurex/chapter08/main.go.)

main.go

The beginning of the program is pretty much the same. We have just amended our measurement type to contain multiple DNSMeasurement results.

package main

import (
	"context"
	"encoding/json"
	"flag"
	"fmt"
	"net/url"
	"time"

	"github.com/ooni/probe-cli/v3/internal/measurex"
	"github.com/ooni/probe-cli/v3/internal/runtimex"
)

type measurement struct {
	DNS       []*measurex.DNSMeasurement
	Endpoints []*measurex.HTTPEndpointMeasurement
}

func print(v interface{}) {
	data, err := json.Marshal(v)
	runtimex.PanicOnError(err, "json.Marshal failed")
	fmt.Printf("%s\n", string(data))
}

func main() {
	URL := flag.String("url", "https://blog.cloudflare.com/", "URL to fetch")
	address := flag.String("address", "8.8.4.4:53", "DNS-over-UDP server address")
	timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
	flag.Parse()
	ctx, cancel := context.WithTimeout(context.Background(), *timeout)
	defer cancel()
	parsed, err := url.Parse(*URL)
	runtimex.PanicOnError(err, "url.Parse failed")
	mx := measurex.NewMeasurerWithDefaultSettings()
	m := &measurement{}

Call LookupHTTPSSvc

Here we perform the LookupHostUDP we used in the previous chapter and then we call LookupHTTPSvcUDP.

	m.DNS = append(m.DNS, mx.LookupHostUDP(ctx, parsed.Hostname(), *address))
	m.DNS = append(m.DNS, mx.LookupHTTPSSvcUDP(ctx, parsed.Hostname(), *address))

The LookupHTTPSSvcUDP function has the same signature as LookupHostUDP but it behaves differently. Rather than querying for A and AAAA, it performs an HTTPS DNS lookup. This query returns:

1. a list of ALPNs for the domain;

2. a list of IPv4 addresses;

3. a list of IPv6 addresses.

Build an []HTTPEndpoint and run serial measurements

Here we call AllHTTPEndpointsForURL like we did in the previous chapter. However, note that we pass it the whole content of m.DNS, which now contains not only the A/AAAA lookups results but also the HTTPS lookup results.

The AllHTTPEndpointsForURL function will recognize that we also have HTTPS lookups and, if the "h3" ALPN is present, will also build HTTP/3 endpoints using "udp" as the HTTPEndpoint.Network.

	headers := measurex.NewHTTPRequestHeaderForMeasuring()
	httpEndpoints, err := measurex.AllHTTPEndpointsForURL(parsed, headers, m.DNS...)
	runtimex.PanicOnError(err, "cannot get all the HTTP endpoints")

This is it. The rest of the program is exactly the same.

	for _, epnt := range httpEndpoints {
		m.Endpoints = append(m.Endpoints, mx.HTTPEndpointGetWithoutCookies(ctx, epnt))
	}

(Note that here, like in the previous chapter, we are not converting to the OONI data format. Rather, we're just dumping the internally used data structures. Exercise: can you modify this program to emit a JSON compliant with the OONI data format by using the proper] conversion functions exported by measurex?)

	print(m)
}

Running the example program

Let us perform a vanilla run first:

go run -race ./internal/tutorial/measurex/chapter08 | jq

Please, check the JSON output. Do you recognize the fields we have described in previous chapters? You should see that, compared to previous chapters, now we're also testing QUIC/HTTP3 endpoints.

Can you provoke common errors such as DNS resolution errors, TCP connect errors, TLS handshake errors, and HTTP round trip errors? What is a good way to cause timeout and SNI mismatch errors for QUIC?

Conclusion

We have seen how to extend fetching all the HTTPS endpoints to include the QUIC/HTTP3 endpoints discovered using HTTPSSvc.