c/ooni-probe-cli
Watch
1
0
Fork
You've already forked ooni-probe-cli
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

feat(webconnectivity): try all the available THs (#980)

Browse source 
We introduce a fork of internal/httpx, named internal/httpapi, where there is a clear split between the concept of an API endpoint (such as https://0.th.ooni.org/) and of an API descriptor (such as using `GET` to access /api/v1/test-list/url).

Additionally, httpapi allows to create a SequenceCaller that tries to call a given API descriptor using multiple API endpoints. The SequenceCaller will stop once an endpoint works or when all the available endpoints have been tried unsuccessfully.

The definition of "success" is the following: we consider "failure" any error that occurs during the HTTP round trip or when reading the response body. We DO NOT consider "failure" errors (1) when parsing the input URL; (2) when the server returns >= 400; (3) when the server returns a string that does not parse as valid JSON. The idea of this classification of failures is that we ONLY want to retry when we see what looks like a network error that may be caused by (collateral or targeted) censorship.

We take advantage of the availability of this new package and we refactor web_connectivity@v0.4 and web_connectivity@v0.5 to use a SequenceCaller for calling the web connectivity TH API. This means that we will now try all the available THs advertised by the backend rather than just selecting and using the first one provided by the backend.

Because this diff is designed to be backported to the `release/3.16` branch, we have omitted additional changes to always use httpapi where we are currently using httpx. Yet, to remind ourselves about the need to do that, we have deprecated the httpx package. We will rewrite all the code currently using httpx to use httpapi as part of future work.

It is also worth noting that httpapi will allow us to refactor the backend code such that (1) we remove code to select a backend URL endpoint at the beginning and (2) we try several endpoints. The design of the code is such that we can add to the mix some endpoints using as `http.Client` a special client using a tunnel. This will allow us to automatically fallback backend queries.

Closes https://github.com/ooni/probe/issues/2353.

Related to https://github.com/ooni/probe/issues/1519.
This commit is contained in:
Simone Basso 2022-11-21 16:28:53 +01:00 • committed by GitHub
commit c2ea0b4704
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
19 changed files with 2444 additions and 65 deletions
Download patch file Download diff file Expand all files Collapse all files

2
internal/experiment/webconnectivity/cleartextflow.go
View file

@ -285,7 +285,7 @@ func (t *CleartextFlow) maybeFollowRedirects(ctx context.Context, resp *http.Res
WaitGroup: t.WaitGroup,
Referer: resp.Request.URL.String(),
Session: nil, // no need to issue another control request
THAddr: "", // ditto
TestHelpers: nil, // ditto
UDPAddress: t.UDPAddress,
}
resolvers.Start(ctx)

33
internal/experiment/webconnectivity/control.go
View file

@ -8,10 +8,11 @@ import (
"time"
"github.com/ooni/probe-cli/v3/internal/engine/experiment/webconnectivity"
"github.com/ooni/probe-cli/v3/internal/httpx"
"github.com/ooni/probe-cli/v3/internal/httpapi"
"github.com/ooni/probe-cli/v3/internal/measurexlite"
"github.com/ooni/probe-cli/v3/internal/model"
"github.com/ooni/probe-cli/v3/internal/netxlite"
"github.com/ooni/probe-cli/v3/internal/runtimex"
)
// EndpointMeasurementsStarter is used by Control to start extra
@ -51,8 +52,8 @@ type Control struct {
// Session is the MANDATORY session to use.
Session model.ExperimentSession
// THAddr is the MANDATORY TH's URL.
THAddr string
// TestHelpers is the MANDATORY list of test helpers.
TestHelpers []model.OOAPIService
// URL is the MANDATORY URL we are measuring.
URL *url.URL
@ -102,26 +103,20 @@ func (c *Control) Run(parentCtx context.Context) {
// create logger for this operation
ol := measurexlite.NewOperationLogger(
c.Logger,
"control for %s using %s",
"control for %s using %+v",
creq.HTTPRequest,
c.THAddr,
c.TestHelpers,
)
// create an API client
clnt := (&httpx.APIClientTemplate{
Accept: "",
Authorization: "",
BaseURL: c.THAddr,
HTTPClient: c.Session.DefaultHTTPClient(),
Host: "", // use the one inside the URL
LogBody: true,
Logger: c.Logger,
UserAgent: c.Session.UserAgent(),
}).Build()
// create an httpapi sequence caller
seqCaller := httpapi.NewSequenceCaller(
httpapi.MustNewPOSTJSONWithJSONResponseDescriptor(c.Logger, "/", creq).WithBodyLogging(true),
httpapi.NewEndpointList(c.Session.DefaultHTTPClient(), c.Session.UserAgent(), c.TestHelpers...)...,
)
// issue the control request and wait for the response
var cresp webconnectivity.ControlResponse
err := clnt.PostJSON(opCtx, "/", creq, &cresp)
idx, err := seqCaller.CallWithJSONResponse(opCtx, &cresp)
if err != nil {
// make sure error is wrapped
err = netxlite.NewTopLevelGenericErrWrapper(err)
@ -134,6 +129,10 @@ func (c *Control) Run(parentCtx context.Context) {
c.TestKeys.SetControl(&cresp)
ol.Stop(nil)
// record the specific TH that worked
runtimex.Assert(idx >= 0 && idx < len(c.TestHelpers), "idx out of bounds")
c.TestKeys.setTestHelper(&c.TestHelpers[idx])
// if the TH returned us addresses we did not previously were
// aware of, make sure we also measure them
c.maybeStartExtraMeasurements(parentCtx, cresp.DNS.Addrs)

13
internal/experiment/webconnectivity/dnsresolvers.go
View file

@ -67,8 +67,9 @@ type DNSResolvers struct {
// always follow the redirect chain caused by the provided URL.
Session model.ExperimentSession
// THAddr is the OPTIONAL test helper address.
THAddr string
// TestHelpers is the OPTIONAL list of test helpers. If the list is
// empty, we are not going to try to contact any test helper.
TestHelpers []model.OOAPIService
// UDPAddress is the OPTIONAL address of the UDP resolver to use. If this
// field is not set we use a default one (e.g., `8.8.8.8:53`).
@ -498,15 +499,15 @@ func (t *DNSResolvers) startSecureFlows(
}
}
// maybeStartControlFlow starts the control flow iff .Session and .THAddr are set.
// maybeStartControlFlow starts the control flow iff .Session and .TestHelpers are set.
func (t *DNSResolvers) maybeStartControlFlow(
ctx context.Context,
ps *prioritySelector,
addresses []DNSEntry,
) {
// note: for subsequent requests we don't set .Session and .THAddr hence
// note: for subsequent requests we don't set .Session and .TestHelpers hence
// we are not going to query the test helper more than once
if t.Session != nil && t.THAddr != "" {
if t.Session != nil && len(t.TestHelpers) > 0 {
var addrs []string
for _, addr := range addresses {
addrs = append(addrs, addr.Addr)
@ -518,7 +519,7 @@ func (t *DNSResolvers) maybeStartControlFlow(
PrioSelector: ps,
TestKeys: t.TestKeys,
Session: t.Session,
THAddr: t.THAddr,
TestHelpers: t.TestHelpers,
URL: t.URL,
WaitGroup: t.WaitGroup,
}

26
internal/experiment/webconnectivity/measurer.go
View file

@ -36,7 +36,7 @@ func (m *Measurer) ExperimentName() string {
// ExperimentVersion implements model.ExperimentMeasurer.
func (m *Measurer) ExperimentVersion() string {
return "0.5.18"
return "0.5.19"
}
// Run implements model.ExperimentMeasurer.
@ -89,17 +89,7 @@ func (m *Measurer) Run(ctx context.Context, sess model.ExperimentSession,
// obtain the test helper's address
testhelpers, _ := sess.GetTestHelpersByName("web-connectivity")
var thAddr string
for _, th := range testhelpers {
if th.Type == "https" {
thAddr = th.Address
measurement.TestHelpers = map[string]any{
"backend": &th,
}
break
}
}
if thAddr == "" {
if len(testhelpers) < 1 {
sess.Logger().Warnf("continuing without a valid TH address")
tk.SetControlFailure(webconnectivity.ErrNoAvailableTestHelpers)
}
@ -120,7 +110,7 @@ func (m *Measurer) Run(ctx context.Context, sess model.ExperimentSession,
CookieJar: jar,
Referer: "",
Session: sess,
THAddr: thAddr,
TestHelpers: testhelpers,
UDPAddress: "",
}
resos.Start(ctx)
@ -137,6 +127,16 @@ func (m *Measurer) Run(ctx context.Context, sess model.ExperimentSession,
// perform any deferred computation on the test keys
tk.Finalize(sess.Logger())
// set the test helper we used
// TODO(bassosimone): it may be more informative to know about all the
// test helpers we _tried_ to use, however the data format does not have
// support for that as far as I can tell...
if th := tk.getTestHelper(); th != nil {
measurement.TestHelpers = map[string]interface{}{
"backend": th,
}
}
// return whether there was a fundamental failure, which would prevent
// the measurement from being submitted to the OONI collector.
return tk.fundamentalFailure

2
internal/experiment/webconnectivity/secureflow.go
View file

@ -337,7 +337,7 @@ func (t *SecureFlow) maybeFollowRedirects(ctx context.Context, resp *http.Respon
WaitGroup: t.WaitGroup,
Referer: resp.Request.URL.String(),
Session: nil, // no need to issue another control request
THAddr: "", // ditto
TestHelpers: nil, // ditto
UDPAddress: t.UDPAddress,
}
resolvers.Start(ctx)

20
internal/experiment/webconnectivity/testkeys.go
View file

@ -134,6 +134,10 @@ type TestKeys struct {
// mu provides mutual exclusion for accessing the test keys.
mu *sync.Mutex
// testHelper is used to communicate the TH that worked to the main
// goroutine such that we can fill measurement.TestHelpers.
testHelper *model.OOAPIService
}
// ConnPriorityLogEntry is an entry in the TestKeys.ConnPriorityLog slice.
@ -302,6 +306,21 @@ func (tk *TestKeys) AppendConnPriorityLogEntry(entry *ConnPriorityLogEntry) {
tk.mu.Unlock()
}
// setTestHelper sets .testHelper in a thread safe way
func (tk *TestKeys) setTestHelper(th *model.OOAPIService) {
tk.mu.Lock()
tk.testHelper = th
tk.mu.Unlock()
}
// getTestHelper gets .testHelper in a thread safe way
func (tk *TestKeys) getTestHelper() (th *model.OOAPIService) {
tk.mu.Lock()
th = tk.testHelper
tk.mu.Unlock()
return
}
// NewTestKeys creates a new instance of TestKeys.
func NewTestKeys() *TestKeys {
return &TestKeys{
@ -348,6 +367,7 @@ func NewTestKeys() *TestKeys {
ControlRequest: nil,
fundamentalFailure: nil,
mu: &sync.Mutex{},
testHelper: nil,
}
}