We are working on ooniprobe for Debian. Before starting to apply
changes to the codebase, I'd like to apply some refactoring steps
that I've been thinking about for quite some time.
The general concept here is that the purpose of this repository
changed since it was designed and now there is probe-engine which
is a library, therefore, this repo can be mostly private.
* Update go-bindata and regenerate binary data
* Pin to ooni/probe-engine 0.17.0 and update dependencies
* Set version to 3.0.7
* Readme.md: better release instructions
* Use ooni/probe-engine 0.16.0
* Update all the other dependencies
* Use GitHub Actions rather than Travis CI
* Automatically build and test binaries on the target OS (for Windows, macOS, Linux on amd64)
* Make sure we correctly measure coverage
* Make sure we use `-race` when running tests
* Remove unnecessary scripts
* Make sure the README is up-to-date
* Write small script to update binary data and add GitHub Actions checks for it
* Notice that we needed to run ./updatebindata.sh and run it
* Self documenting instructions regarding cross compiling
* Set version number to v3.0.7-beta
Part of https://github.com/ooni/probe-engine/issues/748
* nettests/groups.go: remove redundant struct names
* go.mod go.sum: update deps except probe-engine
* Update to ooni/probe-engine@e768161f91
The API has changed. Methods that used to change bits of the session have
been removed. Now the session is more immutable than before.
As such, we need to completely fill the config before using it.
* Set IncludeCountry to always true
Co-authored-by: Arturo Filastò <arturo@filasto.net>
* Optionally treat EOF on stdin just like SIGTERM
On Unix, Node.js allows us to gracefully kill a process. On Windows
this is more compex. You certainly cannot rely on the default `kill()`
function, which calls `TerminateProcess`.
There is a bunch of C/C++ extensions that in principle allow you to
attempt to gracefully shutdown a Windows process.
But, hey, here's a reality check. Node.js controls our stdin. Node.js
does IPC easy. Controlling uv_spawn flags and using the right not well maintained
C/C++ Node.js extension to kill a process is fragile.
So, treat EOF and any other error on stdin as equivalent to SIGTERM.
However, systemd.
The sane thing to do with systemd is `StandardInput=null`. With such
configuration, stdin immediately returns EOF.
Then, introduce the `OONI_STDIN_EOF_IMPLIES_SIGTERM` environment
variable. When it is `true`, this behaviour is enabled, e.g.:
```bash
export OONI_STDIN_EOF_IMPLIES_SIGTERM=true # behaviour enabled
ooniprobe run
```
I want the default to be disabled because:
1. in the future we may find a better way to solve this problem and I
don't want the _default behaviour_ to change in such case
2. we know we need this knob for ooniprobe-desktop, and we will not
fail to provide it, so it won't suprise/damage us
3. a person trying to write a systemd unit for ooniprobe would be very
surprised to find out they need to disable this behaviour, if it was
enabled by default by this PR
Hence, I believe this design is consistent with designing for the
future and for trying to minimize surprises.
Also, why an environment variable and not a command line flag? Because:
1. we don't want such hypothetical flag to be available where it does not
make sense, e.g., for all subcommands but `run`
2. we don't want the ooni/probe-desktop app to write conditional
code because it needs to check the command we're using and then decide
whether to add such hypothetical flag
Also, why not enabling this only on Windows? Because again we don't
want the ooni/probe-desktop app to write conditional code.
To summarize: we want ooni/probe-desktop app to see the same behaviour
everywhere and we want others to be the least surprised.
Related to https://github.com/ooni/probe/issues/1005
* Update ooni.go
We were writing to the same measurement_file_path for a given test
group, because we were using a different filename only in the case of a
many input test, but not in the case of many test_names inside of a
given test group.
* Use ~/.ooniprobe as the home directory
Remove all probe-legacy related to code since there is no more conflict
between the two
Fixes: ooni/probe#972
* Update .gitignore
Co-authored-by: Simone Basso <bassosimone@gmail.com>
1. only print time left if ETA is positive
2. skip ETA calculation with a single input
3. don't compute ETA for first entry[*]
[*] this is actually what avoids emitting infinite but the other
parts of this diff felt useful yak shaving as well.
Closes#91
1. the description of the command and the helper function are
clear hints that the command is intended to show a single JSON
measurement at a time (also the use case seems clear) [*]
2. the function used to read lines was failing for all my
measurements that take input. Since that was not the optimal
pattern anyway, use a better pattern to fix it.
3. some changes are automatically applied by my editor (VSCode
with the Go plugin) and I am fine with them.
4. while reading code, I also applied my preferred pattern
wrt whitespaces, i.e.: no whitespace inside functions, if a
function feels too long in this way, just break it.
Closes#57
[*] Even if we want to show many measurements at a time, which
does not seem needed, given the UI patterns, this functionality
won't be P0. What is P0 is to bless a new beta and give to
@sarathms binaries for all archs that support a basic `show`.
* utils/geoip.go: use github.com/ooni/probe-engine
Let's start using the engine by rewriting utils/geoip.go to
be just a thin wrapper around the engine functionality.
* Ready for review
* Checkpoint: the im tests are converted
Still have some doubts with respect to the variables that
are passed to MK via probe-engine. Will double check.
* fix(i/c/r/run.go): write the correct logic
* nettests: one more comment and also fix a format string
* Tweak previous
* progress
* Fix doofus
* better comment
* XXX => actionable comment
* Add glue to simplify test keys management
Making the concept of measurement more abstract in the engine is
not feasible because, when submitting a measurement, we need to
modify it to update the report ID and the measurement ID. Therefore,
returning a serialized measurement is not a good idea. We will
keep using a model.Measurement in the engine.
Changing model.Measurement.TestKeys's type from a `interface{}`
pointing to a well defined data structure to `map[string]interface{}`
is a regression because means that we are moving from code that
has a clear and defined structure to code that is more complicated
to parse and validate. Since we're already suffering havily from
the lack of a good schema, I'm not going to make the situation
worst by worsening the engine. At least for ndt7 and psiphon, we
now have a good schema and I don't want to lose that.
However, the current code in this repository is expecting the
test keys to be a `map[string]interface{}`. This choice was
dictated by the fact that we receive a JSON from Measurement Kit
and by the fact that there's not a clear schema.
To solve this tension, in this commit I am going to write glue
adapter code that makes sure that the TestKeys of a Measurement
are converted to `map[string]interface{}`. This will be done
using a type cast where possible and JSON serialization and parsing
otherwise. In a perfect world, glue is not a good idea, but in a
real world it may actually be useful.
When all tests in the engine will have a clear Go data structure,
we'll then remove the glue and just cast to the proper data
structure from `interface{}` where required.
* nettests/performance: use probe-engine
* go.{mod,sum}: upgrade to latest probe-engine
* nettests/middlebox: use ooni/probe-engine
* Update to the latest probe-engine
* web_connectivity: rewrite to use probe-engine
* Cosmetic change suggested by @hellais
* nettests/nettests.go: remove unused code
* nettests/nettests.go: fix progress
* nettests/nettests.go: remove go-measurement-kit code
* We don't depend on go-measurement-kit anymore
* Improve non-verbose output where possible
See also: https://github.com/measurement-kit/measurement-kit/issues/1856
* Make web_connectivity output pleasant
* Update to the latest probe-engine
* nettests/nettests.go: honour sharing settings
* Update to the latest probe-engine
* Use log.WithFields for probe-engine
* Update go.mod go.sum
* Revert "Update go.mod go.sum"
This reverts commit 5ecd38d8236f4a4e9b77ddb8e8a0d1e3cdd4b818.
* Revert "Revert "Update go.mod go.sum""
This reverts commit 6114b31eca98826112032776bd0feff02d763ecd.
* Upgrade ooni/probe-engine
* Unset GOPATH before running go build commands
* Dockefile: fix linux build by using latest
* Update to the latest ooni/probe-engine
```
go get -u github.com/ooni/probe-engine
go mod tidy
```
* Repair build
* Gopkg.lock: use MK v0.10.3
* ooni: stop using legacy GeoIP database files
* Some yak shaving of Makefile
1. remove now broken commands to download deps
2. also define the CXX cross compiler
* chore(dep): migrate from dep to go 1.11+ modules
See https://blog.callr.tech/migrating-from-dep-to-go-1.11-modules/
I need this to simplify my life in building for Travis.
* Introduce build.sh and repair build
In going forward, I believe we don't actually need a Makefile but I
didn't want to make such a radical change now.
* Another strategy wrt gopath
* travis: run regress tests on macOS
Closes#30
