feat(webconnectivity): long-term-evolution prototype (#882)

See https://github.com/ooni/probe/issues/2237

internal/experiment/webconnectivity/README.md

@@ -0,0 +1,109 @@
+# webconnectivity
+
+This directory contains a new implementation of [Web Connectivity](
+https://github.com/ooni/spec/blob/master/nettests/ts-017-web-connectivity.md).
+
+As of 2022-08-26, this code is experimental and is not selected
+by default when you run the `websites` group. You can select this
+implementation with `miniooni` using `miniooni web_connectivity@v0.5`
+from the command line.
+
+Issue [#2237](https://github.com/ooni/probe/issues/2237) explains the rationale
+behind writing this new implementation.
+
+## Implementation overview
+
+The experiment measures a single URL at a time. The OONI Engine invokes the
+`Run` method inside the [measurer.go](measurer.go) file.
+
+This code starts a number of background tasks, waits for them to complete, and
+finally calls `TestKeys.finalize` to finalize the content of the JSON measurement.
+
+The first task that is started deals with DNS and lives in the
+[dnsresolvers.go](dnsresolvers.go) file. This task is responsible for
+resolving the domain inside the URL into `0..N` IP addresses.
+
+The domain resolution includes the system resolver and a DNS-over-UDP
+resolver. The implementaion _may_ do more than that, but this is the
+bare minimum we're feeling like documenting right now. (We need to
+experiment a bit more to understand what else we can do there, hence
+the code is _probably_ doing more than just that.)
+
+Once we know the `0..N` IP addresses for the domain we do the following:
+
+1. start a background task to communicate with the Web Connectivity
+test helper, using code inside [control.go](control.go);
+
+2. start an endpoint measurement task for each IP adddress (which of
+course only happens when we know _at least_ one addr).
+
+Regarding starting endpoint measurements, we follow this policy:
+
+1. if the original URL is `http://...` then we start a cleartext task
+and an encrypted task for each address using ports `80` and `443`
+respectively.
+
+2. if it's `https://...`, then we only start encrypted tasks.
+
+Cleartext tasks are implemented by [cleartextflow.go](cleartextflow.go) while
+the encrypted tasks live in [secureflow.go](secureflow.go).
+
+A cleartext task does the following:
+
+1. TCP connect;
+
+2. additionally, the first task to establish a connection also performs
+a GET request to fetch a webpage (we cannot GET for all connections, because
+that would be `websteps` and would require a different data format).
+
+An encrypted task does the following:
+
+1. TCP connect;
+
+2. TLS handshake;
+
+3. additionally, the first task to handshake also performs
+a GET request to fetch a webpage _iff_ the input URL was `https://...` (we cannot GET
+for all connections, because that would be `websteps` and would require a
+different data format).
+
+If fetching the webpage returns a redirect, we start a new DNS task passing it
+the redirect URL as the new URL to measure. We do not call the test helper again
+when this happens, though. The Web Connectivity test helper already follows the whole
+redirect chain, so we would need to change the test helper to get information on
+each flow. When this will happen, this experiment will probably not be Web Connectivity
+anymore, but rather some form of [websteps](https://github.com/bassosimone/websteps-illustrated/).
+
+Additionally, when the test helper terminates, we run TCP connect and TLS handshake
+(when applicable) for new IP addresses discovered using the test helper that were
+previously unknown to the probe, thus collecting extra information. This logic lives
+inside the [control.go](control.go) file.
+
+As previously mentioned, when all tasks complete, we call `TestKeys.finalize`.
+
+In turn, this function analyzes the collected data by calling code implemented
+inside the following files:
+
+- [analysiscore.go](analysiscore.go) contains the core analysis algorithm;
+
+- [analysisdns.go](analysisdns.go) contains DNS specific analysis;
+
+- [analysishttpcore.go](analysishttpcore.go) contains the bulk of the HTTP
+analysis, where we mainly determine TLS blocking;
+
+- [analysishttpdiff.go](analysishttpdiff.go) contains the HTTP diff algorithm;
+
+- [analysistcpip.go](analysistcpip.go) checks for TCP/IP blocking.
+
+We emit the `blocking` and `accessible` keys we emitted before as well as new
+keys, prefixed by `x_` to indicate that they're experimental.
+
+## Limitations and next steps
+
+We need to extend the Web Connectivity test helper to return us information
+about TLS handshakes with IP addresses discovered by the probe. This information
+would allow us to make more precise TLS blocking statements.
+
+Further changes are probably possible. Departing too radically from the Web
+Connectivity model, though, will lead us to have a `websteps` implementation (but
+then the data model would most likely be different).