ooni-probe-cli/internal/httpapi/endpoint.go

77 lines
2.3 KiB
Go
Raw Normal View History

feat(webconnectivity): try all the available THs (#980) We introduce a fork of internal/httpx, named internal/httpapi, where there is a clear split between the concept of an API endpoint (such as https://0.th.ooni.org/) and of an API descriptor (such as using `GET` to access /api/v1/test-list/url). Additionally, httpapi allows to create a SequenceCaller that tries to call a given API descriptor using multiple API endpoints. The SequenceCaller will stop once an endpoint works or when all the available endpoints have been tried unsuccessfully. The definition of "success" is the following: we consider "failure" any error that occurs during the HTTP round trip or when reading the response body. We DO NOT consider "failure" errors (1) when parsing the input URL; (2) when the server returns >= 400; (3) when the server returns a string that does not parse as valid JSON. The idea of this classification of failures is that we ONLY want to retry when we see what looks like a network error that may be caused by (collateral or targeted) censorship. We take advantage of the availability of this new package and we refactor web_connectivity@v0.4 and web_connectivity@v0.5 to use a SequenceCaller for calling the web connectivity TH API. This means that we will now try all the available THs advertised by the backend rather than just selecting and using the first one provided by the backend. Because this diff is designed to be backported to the `release/3.16` branch, we have omitted additional changes to always use httpapi where we are currently using httpx. Yet, to remind ourselves about the need to do that, we have deprecated the httpx package. We will rewrite all the code currently using httpx to use httpapi as part of future work. It is also worth noting that httpapi will allow us to refactor the backend code such that (1) we remove code to select a backend URL endpoint at the beginning and (2) we try several endpoints. The design of the code is such that we can add to the mix some endpoints using as `http.Client` a special client using a tunnel. This will allow us to automatically fallback backend queries. Closes https://github.com/ooni/probe/issues/2353. Related to https://github.com/ooni/probe/issues/1519.
2022-11-21 16:28:53 +01:00
package httpapi
//
// HTTP API Endpoint (e.g., https://api.ooni.io)
//
import "github.com/ooni/probe-cli/v3/internal/model"
// Endpoint models an HTTP endpoint on which you can call
// several HTTP APIs (e.g., https://api.ooni.io) using a
// given HTTP client potentially using a circumvention tunnel
// mechanism such as psiphon or torsf.
//
// The zero value of this struct is invalid. Please, fill all the
// fields marked as MANDATORY for correct initialization.
type Endpoint struct {
// BaseURL is the MANDATORY endpoint base URL. We will honour the
// path of this URL and prepend it to the actual path specified inside
// a |Descriptor.URLPath|. However, we will always discard any query
// that may have been set inside the BaseURL. The only query string
// will be composed from the |Descriptor.URLQuery| values.
//
// For example, https://api.ooni.io.
BaseURL string
// HTTPClient is the MANDATORY HTTP client to use.
//
// For example, http.DefaultClient. You can introduce circumvention
// here by using an HTTPClient bound to a specific tunnel.
HTTPClient model.HTTPClient
// Host is the OPTIONAL host header to use.
//
// If this field is empty we use the BaseURL's hostname. A specific
// host header may be needed when using cloudfronting.
Host string
// User-Agent is the OPTIONAL user-agent to use. If empty,
// we'll use the stdlib's default user-agent string.
UserAgent string
}
// NewEndpointList constructs a list of API endpoints from |services|
// returned by the OONI backend (or known in advance).
//
// Arguments:
//
// - httpClient is the HTTP client to use for accessing the endpoints;
//
// - userAgent is the user agent you would like to use;
//
// - service is the list of services gathered from the backend.
func NewEndpointList(httpClient model.HTTPClient,
userAgent string, services ...model.OOAPIService) (out []*Endpoint) {
for _, svc := range services {
switch svc.Type {
case "https":
out = append(out, &Endpoint{
BaseURL: svc.Address,
HTTPClient: httpClient,
Host: "",
UserAgent: userAgent,
})
case "cloudfront":
out = append(out, &Endpoint{
BaseURL: svc.Address,
HTTPClient: httpClient,
Host: svc.Front,
UserAgent: userAgent,
})
default:
// nothing!
}
}
return
}