
# Chapter VI: Getting a webpage from an HTTP/HTTPS/HTTP3 endpoint.

This chapter describes measuring the retrieval of a webpage from an
HTTPS endpoint. We have seen how to TCP connect, we have
seen how to TLS handshake, now it's time to see how we can
combine these operations with that of fetching a webpage from a
given TCP endpoint speaking HTTP and TLS. (As well as
providing you with information on how to otherwise fetch
from HTTP and HTTP/3 endpoints.)

The program we're going to write, `main.go`, will use a
high-level operation to perform this measurement in a
single API call. The code implementing this API call will
combine the operations we have seen in previous chapter
with the "give me the webpage" operation. We are still
quite far away from the ability of "measuring a URL" but
we are increasingly moving towards more complex operations.

(This file is auto-generated. Do not edit it directly! To apply
changes you need to modify `./internal/tutorial/measurex/chapter06/main.go`.)

## main.go

We have package declaration and imports as usual.

```Go
package main

import (
	"context"
	"encoding/json"
	"flag"
	"fmt"
	"net/url"
	"time"

	"github.com/ooni/probe-cli/v3/internal/measurex"
	"github.com/ooni/probe-cli/v3/internal/runtimex"
)

```

We have factored the three lines to print a measurement
into the following utility function called `print`.

```Go
func print(v interface{}) {
	data, err := json.Marshal(v)
	runtimex.PanicOnError(err, "json.Marshal failed")
	fmt.Printf("%s\n", string(data))
}

```

The initial part of the program is pretty much the same as the one
used in previous chapters, expect that we have a few more command line
flags now, so I will not add further comments.

```Go
func main() {
	sni := flag.String("sni", "dns.google", "value for SNI extension")
	address := flag.String("address", "8.8.4.4:443", "remote endpoint address")
	timeout := flag.Duration("timeout", 60*time.Second, "timeout to use")
	flag.Parse()
	ctx, cancel := context.WithTimeout(context.Background(), *timeout)
	defer cancel()
	mx := measurex.NewMeasurerWithDefaultSettings()
```

### HTTPEndpoint: a description of what to measure

First of all, let us create a description of the endpoint
for `measurex`. Up to this point we have used endpoint
to describe a level-4-like address. Therefore, we have seen
TCP endpoints being used for the TCP connect and the TLS
handshake, and we have seen QUIC endpoints being used
for the QUIC handshake. Now, however, we are going
to need more information to characterize the endpoint.

```Go
	epnt := &measurex.HTTPEndpoint{
		Domain:  *sni,
		Network: "tcp",
		Address: *address,
		SNI:     *sni,
		ALPN:    []string{"h2", "http/1.1"},
		URL: &url.URL{
			Scheme: "https",
			Host:   *sni,
			Path:   "/",
		},
		Header: measurex.NewHTTPRequestHeaderForMeasuring(),
	}
```

In fact, in the above definition we recognize fields
we have already discussed, such as:

- `Network`, describing whether to use "tcp" or "udp";

- `Address`, which is the endpoint address.

But we also need to combine into this view of the
endpoint additional fields for TLS:

- `SNI`, to set the SNI;

- `ALPN`, to set the ALPN;

But then we also need to specify:

- the URL to use;

- the headers to use (for which we're using a handy
factory for creating reasonable defaults for measuring).

This API is not the highest level API with which to do
the job, but it's still handy to introduce the
`measurex.HTTPEndpoint` data structure since it's
used by higher level APIs.

(You may also be wondering about the CA pool. It turns
out that for APIs such as this one and for higher
level APIs, the default is to always use the bundled
Mozilla CA pool, because this is what we use in
most cases for performing measurements.)

### HTTPEndpointGetWithoutCookies

When used with an HTTP URL, the `HTTPEndpointGetWithoutCookies`
method combines two operations:

- TCP connect

- HTTP GET

When the URL is HTTPS, we do:

- TCP connect

- TLS handshake

- HTTP GET (or HTTP/2 GET depending on the ALPN)

When the `HTTPEndpoint.Network` field value
is QUIC, instead we do:

- QUIC handshake

- HTTP/3 GET

```Go
	m := mx.HTTPEndpointGetWithoutCookies(ctx, epnt)
```

The arguments for `HTTPEndpointGetWithDBWithoutCookies` are:

- the context for deadline/timeout

- the HTTPEndpoint descriptor

The result is an `HTTPEndpointMeasurement` which
you can inspect with

```
go doc ./internal/measurex.HTTPEndpointMeasurement
```

### Printing the measurement

Let us now print the resulting measurement.

```Go
	print(measurex.NewArchivalHTTPEndpointMeasurement(m))
}

```

## Running the example program

Let us perform a vanilla run first:

```bash
go run -race ./internal/tutorial/measurex/chapter06 | jq
```

This is the JSON output. Let us comment it in detail:

```Javascript
{
  // The returned type is called HTTPEndpointMeasurement
  // and you see that here on top we indeed have the
  // information on the endpoint and the URL.
  "url": "https://dns.google/",
  "network": "tcp",
  "address": "8.8.4.4:443",

  // These are the I/O operations we have already seen
  // in previous chapters
  "network_events": [
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 280,
      "operation": "write",
      "proto": "tcp",
      "t": 0.045800292,
      "started": 0.045782167,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 517,
      "operation": "read",
      "proto": "tcp",
      "t": 0.082571,
      "started": 0.045805458,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 4303,
      "operation": "read",
      "proto": "tcp",
      "t": 0.084400542,
      "started": 0.084372667,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 64,
      "operation": "write",
      "proto": "tcp",
      "t": 0.086762625,
      "started": 0.086748292,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 86,
      "operation": "write",
      "proto": "tcp",
      "t": 0.087851,
      "started": 0.087837625,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 201,
      "operation": "write",
      "proto": "tcp",
      "t": 0.089527292,
      "started": 0.089507958,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 93,
      "operation": "read",
      "proto": "tcp",
      "t": 0.168585625,
      "started": 0.088068375,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 31,
      "operation": "write",
      "proto": "tcp",
      "t": 0.168713542,
      "started": 0.168671417,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 2000,
      "operation": "read",
      "proto": "tcp",
      "t": 0.468671417,
      "started": 0.168759333,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 39,
      "operation": "read",
      "proto": "tcp",
      "t": 0.47118175,
      "started": 0.471169667,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 39,
      "operation": "write",
      "proto": "tcp",
      "t": 0.471335458,
      "started": 0.471268583,
      "oddity": ""
    },
    {
      "address": "8.8.4.4:443",
      "failure": null,
      "num_bytes": 24,
      "operation": "write",
      "proto": "tcp",
      "t": 0.471865,
      "started": 0.471836292,
      "oddity": ""
    }
  ],

  // Internally, HTTPEndpointGetWithoutCookies calls
  // TCPConnect and here we see the corresponding event
  "tcp_connect": [
    {
      "ip": "8.8.4.4",
      "port": 443,
      "t": 0.043644958,
      "status": {
        "blocked": false,
        "failure": null,
        "success": true
      },
      "started": 0.022849458,
      "oddity": ""
    }
  ],

  // Internally, HTTPEndpointGetWithoutCookies calls TLSConnectAndHandshake,
  // and here's the resulting handshake event. Of course, if we
  // specified a QUIC endpoint we would instead see here a
  // QUIC handshake event. And, we would not see any handshake
  // if the URL was instead an HTTP URL.
  "tls_handshakes": [
    {
      "cipher_suite": "TLS_AES_128_GCM_SHA256",
      "failure": null,
      "negotiated_proto": "h2",
      "tls_version": "TLSv1.3",
      "peer_certificates": [
        {
          "data": " ",
          "format": "base64"
        },
        {
          "data": " ",
          "format": "base64"
        },
        {
          "data": " ",
          "format": "base64"
        }
      ],
      "t": 0.086816667,
      "address": "8.8.4.4:443",
      "server_name": "dns.google",
      "alpn": [
        "h2",
        "http/1.1"
      ],
      "no_tls_verify": false,
      "oddity": "",
      "proto": "tcp",
      "started": 0.043971083
    }
  ],

  // Finally here we see information about the round trip, which
  // is formatted according to https://github.com/ooni/spec/blob/master/data-formats/df-001-httpt.md:
  "requests": [
    {

      // This field indicates whether there was an error during
      // the HTTP round trip:
      "failure": null,

      // This field contains the request method, URL, and HTTP headers
      "request": {
        "method": "GET",
        "url": "https://dns.google/",
        "headers": {
          "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
          "accept-language": "en-US;q=0.8,en;q=0.5",
          "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
        }
      },

      // This field contains the response status code, body, and headers.
      "response": {
        "code": 200,
        "headers": {
          "accept-ranges": "none",
          "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
          "cache-control": "private",
          "content-security-policy": "object-src 'none';base-uri 'self';script-src 'nonce-y/OGliLR2gbEfTG2i4MAaw==' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/honest_dns/1_0;frame-ancestors 'none'",
          "content-type": "text/html; charset=UTF-8",
          "date": "Fri, 05 Nov 2021 08:59:37 GMT",
          "server": "scaffolding on HTTPServer2",
          "strict-transport-security": "max-age=31536000; includeSubDomains; preload",
          "vary": "Accept-Encoding",
          "x-content-type-options": "nosniff",
          "x-frame-options": "SAMEORIGIN",
          "x-xss-protection": "0"
        },

        // The body in particular is a snapshot of the response
        // body: we don't want to read and submit to the OONI
        // collector large bodies.
        "body": {
          "data": " ",
          "format": "base64"
        },

        // This field tells us whether the size of the read
        // snapshot was smaller than the snapshot size. If
        // not, then the body has been truncated.
        "body_is_truncated": false,

        // These extra fields are not part of the spec and
        // hence we prefix them with `x_`. They tell us
        // the length of the body and whether the content
        // of the body is valid UTF8.
        "x_body_length": 1383,
        "x_body_is_utf8": true
      },

      // The t field is the moment where we finished the
      // round trip and saved the event. The started field
      // is instead when we started the round trip.
      //
      // You may notice that the start of the round trip
      // if after the `t` of the handshake. This tells us
      // that the code first connects, then handshakes, and
      // finally creates HTTP code for performing the
      // round trip.
      "t": 0.471535167,
      "started": 0.087176458,

      // As usual we also compute an oddity value related
      // in this case to the HTTP round trip.
      "oddity": ""
    }
  ]
}
```

Here are some suggestions for follow up measurements:

1. provoke a connect error by using:

```
go run -race ./internal/tutorial/measurex/chapter06 -address 127.0.0.1:1 | jq
```

2. provoke a TLS handshake error by using:

```
go run -race ./internal/tutorial/measurex/chapter06 -sni example.com | jq
```

3. provoke an HTTP round trip error by using:

```
go run -race ./internal/tutorial/measurex/chapter06 -address 8.8.8.8:853 | jq
```

4. modify the code to fetch an HTTP endpoint instead (hint: you
need to change the HTTPEndpoint's URL scheme);

5. modify the code to use QUIC and HTTP/3 instead (hint: you need to
change the HTTPEndpoint's network and... is this enough?).

## Conclusion

We have seen how to measure the flow of fetching a
specific webpage from an HTTPEndpoint.

