Commit Graph

8 Commits

Author SHA1 Message Date
Simone Basso
c2ea0b4704
feat(webconnectivity): try all the available THs (#980)
We introduce a fork of internal/httpx, named internal/httpapi, where there is a clear split between the concept of an API endpoint (such as https://0.th.ooni.org/) and of an API descriptor (such as using `GET` to access /api/v1/test-list/url).

Additionally, httpapi allows to create a SequenceCaller that tries to call a given API descriptor using multiple API endpoints. The SequenceCaller will stop once an endpoint works or when all the available endpoints have been tried unsuccessfully.

The definition of "success" is the following: we consider "failure" any error that occurs during the HTTP round trip or when reading the response body. We DO NOT consider "failure" errors (1) when parsing the input URL; (2) when the server returns >= 400; (3) when the server returns a string that does not parse as valid JSON. The idea of this classification of failures is that we ONLY want to retry when we see what looks like a network error that may be caused by (collateral or targeted) censorship.

We take advantage of the availability of this new package and we refactor web_connectivity@v0.4 and web_connectivity@v0.5 to use a SequenceCaller for calling the web connectivity TH API. This means that we will now try all the available THs advertised by the backend rather than just selecting and using the first one provided by the backend.

Because this diff is designed to be backported to the `release/3.16` branch, we have omitted additional changes to always use httpapi where we are currently using httpx. Yet, to remind ourselves about the need to do that, we have deprecated the httpx package. We will rewrite all the code currently using httpx to use httpapi as part of future work.

It is also worth noting that httpapi will allow us to refactor the backend code such that (1) we remove code to select a backend URL endpoint at the beginning and (2) we try several endpoints. The design of the code is such that we can add to the mix some endpoints using as `http.Client` a special client using a tunnel. This will allow us to automatically fallback backend queries.

Closes https://github.com/ooni/probe/issues/2353.

Related to https://github.com/ooni/probe/issues/1519.
2022-11-21 16:28:53 +01:00
Simone Basso
ad01856beb
fix(web_connectivity@v0.5): limit number of redirects (#965)
Part of https://github.com/ooni/probe/issues/2237
2022-09-15 08:46:53 +02:00
Simone Basso
5e76c6ec92
refactor(webconnectivity@v0.5): improve logging clarity (#964)
We're bumping the experiment's version number because we changed the name of the field used to contain late/duplicate DNS responses. We have also changed the algorithm to determine `#dnsDiff`. However, the change should only impact how we log this information. Overall, here the idea is to provide users with a reasonably clear explanation of how the probe maps observations to blocking and accessible using expected/unexpected as the conceptual framework.

Part of https://github.com/ooni/probe/issues/2237
2022-09-15 07:03:53 +02:00
Simone Basso
b10eea47e7
feat(webconnectivity@v0.5): flag case where noone resolved any address (#953)
See https://github.com/ooni/probe/issues/2290

While there, notice that in such a case the priority selector would hang because of the WaitGroup, so get rid of the WaitGroup and accept that the priority selector is going to hang around for the whole duration of the measurement in some cases. The cancellable `measurer.go`'s context will cause the priority selector to eventually exit when we return from `measurer.go`'s `Run` method.
2022-09-12 07:33:34 +02:00
Simone Basso
5e75512396
feat(webconnectivity@v0.5): get a webpage whenever possible (#950)
Implements https://github.com/ooni/probe/issues/2276 and supersedes https://github.com/ooni/probe-cli/pull/949.
2022-09-11 22:12:48 +02:00
Simone Basso
3b24b1196d
fix(webconnectivity@v0.5): fetch HTTP only using system-resolver addrs (#935)
While there, change the emoji logger to emit whitespace on info logs. This makes warnings stand out even more.

Closes https://github.com/ooni/probe/issues/2258
2022-09-05 13:33:59 +02:00
Simone Basso
c3964e43b3
feat(webconnectivity@v0.5): record late DNS replies (#883)
Part of https://github.com/ooni/probe/issues/2237
2022-08-26 18:32:36 +02:00
Simone Basso
1a1d3126ae
feat(webconnectivity): long-term-evolution prototype (#882)
See https://github.com/ooni/probe/issues/2237
2022-08-26 16:42:48 +02:00