Commit Graph

85 Commits

Author SHA1 Message Date
Arturo Filastò
e3d68457b3 Implement writing using the new single measurement pattern 2020-01-28 11:53:00 +01:00
Simone Basso
03e7d2ccac
circumvention: add the tor experiment (#100)
* go.mod go.sum: update all non-probe-engine deps

For each line in the go.mod, run `go get -u -v $package` if the
line is not an indirect dependency and is not probe-engine.

Upgrading probe-engine is going to require the same spell that
is used in probe-engine to update psiphon.

* go get -v github.com/ooni/probe-engine@v0.5.0

This just pins to the latest probe-engine but we've not manually
pinned all the other dependencieds yet.

Take care of the trivial API changes in probe-engine as well, such
that we can have a working build after this commit.

* go.mod go.sum: pin to probe-engine dependencies

Basically: remove all indirect dependencies. Merge this go.mod with
the one of probe-engine, to pin dependencies. Run `go mod tidy`.

* circumvention: add basic implementation of tor

This needs to be polished further, of course. But at least we have
now added support for running tor in the circumvention group.

* Readme.md: document how to update dependencies

* go get -v github.com/ooni/probe-engine@fcc9ee0a7afb

* go get -v github.com/ooni/probe-engine@4d254f5b2

* nettests/tor.go: implement summary test keys
2020-01-28 10:05:54 +01:00
Arturo Filastò
427ca18985 Refactor pattern of ok !ok 2020-01-07 16:44:11 +02:00
Arturo Filastò
2cb6661340 Fix typo 2020-01-07 16:42:04 +02:00
Arturo Filastò
24316728b9 Improve the output and extraction of psiphon metrics 2020-01-07 16:13:13 +02:00
Simone Basso
1e8b482c23
Allow to specify custom software name and version (#94)
In turn, this allows us to identify as ooniprobe-cli v3.0.0-rc.5.

Closes #41.
2019-12-29 14:07:57 +01:00
Simone Basso
955509e6f8 nettests: don't say ETA is infinite
1. only print time left if ETA is positive

2. skip ETA calculation with a single input

3. don't compute ETA for first entry[*]

[*] this is actually what avoids emitting infinite but the other
parts of this diff felt useful yak shaving as well.

Closes #91
2019-12-28 18:28:08 +01:00
Simone Basso
991b875bd9 fix(nettests/telegram.go): remove MK option
This option was meaningful for an MK implementation of Telegram but
is not meaningful for the new Go based implementation.

Closes #90
2019-12-28 17:56:47 +01:00
Simone Basso
259ab41b15 Implement the circumvention nettest group
Only Psiphon for now. Closes #87 #77.
2019-12-28 17:48:07 +01:00
Arturo Filastò
7bbbab8774 Handle the SIGINT and SIGTERM signals to support stopping a test cleanly (#84) 2019-12-27 11:32:08 +01:00
Simone Basso
8a1622d0c1 Update to ooni/probe-engine@v0.3.0
1. `go get -u github.com/ooni/probe-engine@v0.3.0`

2. `go mod tidy`

3. fix build failures
2019-12-26 18:09:23 +01:00
Arturo Filastò
b7deb6c88f Use a faster nettest for testing 2019-12-02 18:00:12 +02:00
Arturo Filastò
ccdff3ba24 Move IM tests into top level tree too 2019-12-02 17:59:37 +02:00
Arturo Filastò
b20af107dc Refactor nettests into top tree 2019-12-02 17:57:55 +02:00
Arturo Filastò
8a27de6c7d
Remove all settings which are not implemented (#73)
* Remove all settings which are not implemented

* Add support for migrating config files from version 0 -> 1

Add basic unittests for the config file migration

* Update the config file used by travis

* Fix typos in console log messages

* Improve the comment

* Fix type of limit

* Set informed consent to true in the travis config
2019-11-14 17:58:31 +01:00
Arturo Filastò
bc1314ca61
Add support for calculating the estimate time remaining (#71) 2019-11-14 12:01:40 +01:00
Simone Basso
946289d6fe
Use more general ooni/probe-engine API (#67)
No functional change, just importing less stuff and meddling
much less with the internals of ooni/probe-engine.
2019-10-28 14:05:05 +01:00
Arturo Filastò
f425d3f007 Implement the show command (#53) 2019-10-02 18:23:14 +02:00
Simone Basso
b9b555ba68 Integrate further with ooni/probe-engine: episode two (#46)
* utils/geoip.go: use github.com/ooni/probe-engine

Let's start using the engine by rewriting utils/geoip.go to
be just a thin wrapper around the engine functionality.

* Ready for review

* Checkpoint: the im tests are converted

Still have some doubts with respect to the variables that
are passed to MK via probe-engine. Will double check.

* fix(i/c/r/run.go): write the correct logic

* nettests: one more comment and also fix a format string

* Tweak previous

* progress

* Fix doofus

* better comment

* XXX => actionable comment

* Add glue to simplify test keys management

Making the concept of measurement more abstract in the engine is
not feasible because, when submitting a measurement, we need to
modify it to update the report ID and the measurement ID. Therefore,
returning a serialized measurement is not a good idea. We will
keep using a model.Measurement in the engine.

Changing model.Measurement.TestKeys's type from a `interface{}`
pointing to a well defined data structure to `map[string]interface{}`
is a regression because means that we are moving from code that
has a clear and defined structure to code that is more complicated
to parse and validate. Since we're already suffering havily from
the lack of a good schema, I'm not going to make the situation
worst by worsening the engine. At least for ndt7 and psiphon, we
now have a good schema and I don't want to lose that.

However, the current code in this repository is expecting the
test keys to be a `map[string]interface{}`. This choice was
dictated by the fact that we receive a JSON from Measurement Kit
and by the fact that there's not a clear schema.

To solve this tension, in this commit I am going to write glue
adapter code that makes sure that the TestKeys of a Measurement
are converted to `map[string]interface{}`. This will be done
using a type cast where possible and JSON serialization and parsing
otherwise. In a perfect world, glue is not a good idea, but in a
real world it may actually be useful.

When all tests in the engine will have a clear Go data structure,
we'll then remove the glue and just cast to the proper data
structure from `interface{}` where required.

* nettests/performance: use probe-engine

* go.{mod,sum}: upgrade to latest probe-engine

* nettests/middlebox: use ooni/probe-engine

* Update to the latest probe-engine

* web_connectivity: rewrite to use probe-engine

* Cosmetic change suggested by @hellais

* nettests/nettests.go: remove unused code

* nettests/nettests.go: fix progress

* nettests/nettests.go: remove go-measurement-kit code

* We don't depend on go-measurement-kit anymore

* Improve non-verbose output where possible

See also: https://github.com/measurement-kit/measurement-kit/issues/1856

* Make web_connectivity output pleasant

* Update to the latest probe-engine

* nettests/nettests.go: honour sharing settings

* Update to the latest probe-engine

* Use log.WithFields for probe-engine

* Update go.mod go.sum

* Revert "Update go.mod go.sum"

This reverts commit 5ecd38d8236f4a4e9b77ddb8e8a0d1e3cdd4b818.

* Revert "Revert "Update go.mod go.sum""

This reverts commit 6114b31eca98826112032776bd0feff02d763ecd.

* Upgrade ooni/probe-engine

* Unset GOPATH before running go build commands

* Dockefile: fix linux build by using latest

* Update to the latest ooni/probe-engine

```
go get -u github.com/ooni/probe-engine
go mod tidy
```

* Repair build
2019-08-15 18:08:43 +02:00
Simone Basso
df629237be
Use ooni/probe-engine: episode 1 (#42) 2019-05-23 16:38:46 +02:00
Arturo Filastò
23423cd74f Put MK loglevel to info 2019-05-16 10:22:29 +02:00
Arturo Filastò
6afc8bcc94 Remove reference to caBundlePath 2019-05-15 19:30:10 +02:00
Arturo Filastò
4ead3802a4 Bump MK verbosity and print SSL_CERT value 2019-05-15 19:29:34 +02:00
Arturo Filastò
9155ec95ae Remove getCaBundlePath 2019-05-15 19:25:23 +02:00
Arturo Filastò
5d0e6e6c8c Use info level MK debug 2019-05-15 16:14:01 +02:00
Simone Basso
9992690f8f Use MK v0.10.x, zap legacy GeoIP, build also on Linux (#29)
* Gopkg.lock: use MK v0.10.3

* ooni: stop using legacy GeoIP database files

* Some yak shaving of Makefile

1. remove now broken commands to download deps

2. also define the CXX cross compiler

* chore(dep): migrate from dep to go 1.11+ modules

See https://blog.callr.tech/migrating-from-dep-to-go-1.11-modules/

I need this to simplify my life in building for Travis.

* Introduce build.sh and repair build

In going forward, I believe we don't actually need a Makefile but I
didn't want to make such a radical change now.

* Another strategy wrt gopath

* travis: run regress tests on macOS

Closes #30
2019-05-15 15:48:06 +03:00
Arturo Filastò
1879f8e961 Make the testKeys extractions methods more robust 2018-09-27 11:40:57 +02:00
Arturo Filastò
bc5fe7d8bf Fix bug in report_create event 2018-09-27 11:12:22 +02:00
Arturo Filastò
a60f376846 Adjust the progress bars depending on the test type 2018-09-24 15:57:28 +02:00
Arturo Filastò
0d21186492 Add support for custom collectors and bouncers 2018-09-21 18:54:44 +02:00
Arturo Filastò
cd04ba8bf9 Remove sketchy workaround to UTF-8 characters in homedir
We now implement a workaround to this inside of probe-desktop.
See: d97189f312
2018-09-18 12:10:11 +02:00
Arturo Filastò
4738f7f32d Workaround to keep track of upload state 2018-09-17 13:00:00 +02:00
Arturo Filastò
bd6a5acf5b Sent ooniprobe-desktop as the software name 2018-09-17 11:52:10 +02:00
Arturo Filastò
5cae6b0b83 Make it possible to disable uploading of results 2018-09-17 11:51:54 +02:00
Arturo Filastò
4ed94dfc53 Improve the presentation of the measurement listing from the CLI 2018-09-13 14:54:56 +02:00
Arturo Filastò
7f5df07782 Fix a very annoying logic bug in URL insertion
Write test case for it
2018-09-12 18:47:57 +02:00
Arturo Filastò
2b1fb662fb Fix showing data usage indicators 2018-09-12 14:03:07 +02:00
Arturo Filastò
4880d31120 Fix data usage count 2018-09-12 13:48:18 +02:00
Arturo Filastò
54af7170d4 Add support for data_usage event 2018-09-12 13:42:16 +02:00
Arturo Filastò
1637fdbf93 Structured output of measurements 2018-09-10 18:03:32 +02:00
Arturo Filastò
53791076c9 Disable input list randomization 2018-09-10 17:37:47 +02:00
Arturo Filastò
e0c0acffeb Make error reporting more robust to panic 2018-09-10 16:29:14 +02:00
Arturo Filastò
b29071f37b Replace summary with test_keys 2018-09-10 12:41:28 +02:00
Arturo Filastò
6f0defc672 Refactor URL table row creation into the actions file 2018-09-07 15:23:29 +02:00
Arturo Filastò
35bd334cfc Update the measurement, network and url creation to the new schema 2018-09-07 12:55:27 +02:00
Arturo Filastò
b75cee5e84 Progress on schema and fixing unittests 2018-09-06 16:13:04 +02:00
Arturo Filastò
ff2f973523 Start integrating upper/db as a new ORM 2018-09-05 18:40:37 +02:00
Arturo Filastò
f343a2a1bd Fix bug in checks for keys in middlebox tests 2018-07-18 18:42:54 +02:00
Arturo Filastò
45aea93e6b Fix default config and version 2018-07-14 16:58:08 +02:00
Arturo Filastò
b2faf3fd34 Make some improvements to the homedir ghetto hax
This avoid breakage when inside a dev environment
2018-07-09 19:17:59 +02:00