c2ea0b4704
We introduce a fork of internal/httpx, named internal/httpapi, where there is a clear split between the concept of an API endpoint (such as https://0.th.ooni.org/) and of an API descriptor (such as using `GET` to access /api/v1/test-list/url). Additionally, httpapi allows to create a SequenceCaller that tries to call a given API descriptor using multiple API endpoints. The SequenceCaller will stop once an endpoint works or when all the available endpoints have been tried unsuccessfully. The definition of "success" is the following: we consider "failure" any error that occurs during the HTTP round trip or when reading the response body. We DO NOT consider "failure" errors (1) when parsing the input URL; (2) when the server returns >= 400; (3) when the server returns a string that does not parse as valid JSON. The idea of this classification of failures is that we ONLY want to retry when we see what looks like a network error that may be caused by (collateral or targeted) censorship. We take advantage of the availability of this new package and we refactor web_connectivity@v0.4 and web_connectivity@v0.5 to use a SequenceCaller for calling the web connectivity TH API. This means that we will now try all the available THs advertised by the backend rather than just selecting and using the first one provided by the backend. Because this diff is designed to be backported to the `release/3.16` branch, we have omitted additional changes to always use httpapi where we are currently using httpx. Yet, to remind ourselves about the need to do that, we have deprecated the httpx package. We will rewrite all the code currently using httpx to use httpapi as part of future work. It is also worth noting that httpapi will allow us to refactor the backend code such that (1) we remove code to select a backend URL endpoint at the beginning and (2) we try several endpoints. The design of the code is such that we can add to the mix some endpoints using as `http.Client` a special client using a tunnel. This will allow us to automatically fallback backend queries. Closes https://github.com/ooni/probe/issues/2353. Related to https://github.com/ooni/probe/issues/1519.
OONI Probe Client Library and CLI
The Open Observatory of Network Interference (OONI) is a non-profit free software project that aims to empower decentralized efforts in documenting Internet censorship around the world.
This repository contains core OONI tools written in Go:
-
the CLI client (cmd/ooniprobe);
-
the test helper server (internal/cmd/oohelperd);
-
the mobile library (pkg/oonimkall);
-
the OONI Probe engine (inside internal).
Every top-level directory in this repository contains an explanatory README file. You
may also notice that some internal packages live under internal/engine
while most others are top-level. This is part of a long-standing refactoring started when we merged
https://github.com/ooni/probe-engine into this repository. We'll slowly
ensure that all packages inside engine are moved out of it and inside internal.
Semantic versioning policy
The mobile library is a public package for technical reasons. Go mobile tools require
a public package to build from. Yet, we don't consider API breakages happening in
such a package to be sufficient to bump our major version number. For us, the mobile
library is just a mean to implement OONI Probe Android and OONI Probe iOS. We'll
only bump the major version number if we change ./cmd/ooniprobe's CLI.
License
SPDX-License-Identifier: GPL-3.0-or-later
User setup
Please, follow the instructions at ooni.org/install/cli
to install ooniprobe. If we do not support your use case, please let us know. Once
ooniprobe is installed, try ooniprobe help to get interactive help.
Reporting issues
Report issues at github.com/ooni/probe.
Please, make sure you add the ooni/probe-cli label.
Build instructions
Be sure you have:
-
the golang version mentioned inside the GOVERSION file;
-
a C compiler (Mingw-w64 for Windows).
Caveats
As of 2022-08-22, building with go1.19 will not include Psiphon as a dependency. Fixing this issue is TODO(https://github.com/ooni/probe/issues/2222).
ooniprobe
Ooniprobe is the official CLI client. Compile using:
go build -v -ldflags "-s -w" ./cmd/ooniprobe
This will generate a binary called ooniprobe in the current directory.
miniooni
Miniooni is the experimental OONI client used for research. Compile using:
go build -v -ldflags "-s -w" ./internal/cmd/miniooni
This will generate a binary called miniooni in the current directory.
oohelperd
Oohelperd is the test helper server. Compile using:
go build -v -ldflags "-s -w" ./internal/cmd/oohelperd
This will generate a binary called oohelperd in the current directory.
Specifications
Every nettest (aka experiment) implemented in this repository has a companion spec in ooni/spec.
Contributing
Please, see CONTRIBUTING.md.
Updating dependencies
go get -t -u -v ./... && go mod tidy
Releasing
Create an issue according to the routine release template and perform any item inside the check-list.
We build releases using Makefile, which requires GNU make. Run
make help for detailed usage.