* refactor: move more commands to internal/cmd
Part of https://github.com/ooni/probe/issues/1335.
We would like all commands to be at the same level of engine
rather than inside engine (now that we can do it).
* fix: update .gitignore
* refactor: also move jafar outside engine
* We should be good now?
* refactor: miniooni should be outside of the engine
This is part of https://github.com/ooni/probe/issues/1335. We also need
to think whether we wanna keep libminiooni and miniooni separated.
The previous use case for having a top-level libminiooni was that of
enabling others to integrate miniooni into other binaries.
This was usegul when studying internet censorship in Spain in May 2020.
I am wondering whether we should be keeping this complexity. I am not
sure about this and probably we should be killing it.
(In any case, reducing complexity is not the objective of this diff,
since I would like instead to move things around with minimal changes
and make sure we have a ~good repository organization here.)
* fix: import in libminiooni
* refactor: start building an Android package
Part of https://github.com/ooni/probe/issues/1335.
This seems also a good moment to move some packages out of the
engine, e.g., oonimkall. This package, for example, is a consumer
of the engine, so it makes sense it's not _inside_ it.
* fix: committed some stuff I didn't need to commit
* fix: oonimkall needs to be public to build
The side effect is that we will probably need to bump the major
version number every time we change one of these APIs.
(We can also of course choose to violate the basic guidelines of Go
software, but I believe this is bad form.)
I have no problem in bumping the major quite frequently and in
any case this monorepo solution is convinving me more than continuing
to keep a split between engine and cli. The need to embed assets to
make the probe more reliable trumps the negative effects of having to
~frequently bump major because we expose a public API.
* fix: let's not forget about libooniffi
Honestly, I don't know what to do with this library. I added it
to provide a drop in replacement for MK but I have no idea whether
it's used and useful. I would not feel comfortable exposing it,
unlike oonimkall, since we're not using it.
It may be that the right thing to do here is just to delete the
package and reduce the amount of code we're maintaining?
* woops, we're still missing the publish android script
* fix(publish-android.bash): add proper API key
* ouch fix another place where the name changed
This diff is part of https://github.com/ooni/probe/issues/1335.
We are moving more probe-engine workflows to toplevel.
The general idea here is to migrate all possible workflows and to
delete the ones that we cannot use in this repo (if any).
* refactor: build miniooni from toplevel
Of course, also move the specific test checking whether we are
still able of building miniooni.
Part of https://github.com/ooni/probe/issues/1335
* build for current branch just to confirm
* fix: correct the path where linux/arm binary is
* okay, it works, we can remove the special rule
This is how I did it:
1. `git clone https://github.com/ooni/probe-engine internal/engine`
2. ```
(cd internal/engine && git describe --tags)
v0.23.0
```
3. `nvim go.mod` (merging `go.mod` with `internal/engine/go.mod`
4. `rm -rf internal/.git internal/engine/go.{mod,sum}`
5. `git add internal/engine`
6. `find . -type f -name \*.go -exec sed -i 's@/ooni/probe-engine@/ooni/probe-cli/v3/internal/engine@g' {} \;`
7. `go build ./...` (passes)
8. `go test -race ./...` (temporary failure on RiseupVPN)
9. `go mod tidy`
10. this commit message
Once this piece of work is done, we can build a new version of `ooniprobe` that
is using `internal/engine` directly. We need to do more work to ensure all the
other functionality in `probe-engine` (e.g. making mobile packages) are still WAI.
Part of https://github.com/ooni/probe/issues/1335
This release only contains updates in debian packaging. All other platforms
could safely continue to use 3.2.0 or 3.3.0.
I didn't want to make a path release, though, because I didn't want to convey
the meaning that something was fixed.
Related to https://github.com/ooni/ooni.org/issues/677
* feat: sketch out periodic command
* feat: sketch out periodic command for macOS
* feat: implement darwin's launch agent
* refactor: better way to run on darwin
Make sure we have code that builds on all platforms.
* fix(run): max 10 URLs with darwin in unattended mode
* feat: add support for seeing/streaming logs
* feat: implement the status command and add usage hints
* feat(periodic): run onboarding if needed
* fix: no too confusing function names
* fix: s/periodic/autorun/
Discussed earlier this morning with @hellais.
* fix: we cannot show logs before Big Sur
Bug reported by @hellais.
The CloseReport method is gone. We don't need to close reports
anymore with the new OONI backend.
The InputsRequired flags now is InputsOrQueryTestLists.
* Implement support for not writing to disk and fetching measurements from the API
* Handle case of input not being set
* Comment about exposing raw_measurement in probe-engine
* Add basic test for GetMeasurementJSON
* Update internal/database/actions.go
Co-authored-by: Simone Basso <bassosimone@gmail.com>
We don't want to run performance in the background because this
causes too much traffic towards m-lab servers.
When we'll have the check-in API, this will be the entry point we'll
use to contact such an API and get things to do.
Part of https://github.com/ooni/probe/issues/1289.
* feat: implement syslog logging
With this functionality in tree, macOS users could easily access
ooniprobe logs by filtering by process name in the console app.
Part of https://github.com/ooni/probe/issues/1289
* fix: build on windows
* fix: all build issues
This diff introduces the possibility of specifying --input-file file
multiple times to force ooniprobe to read inputs from file.
Like we do for miniooni, the file shall contain a single entry per
line and this entry should be a URL for websites.
Likewise, one can specify --input URL multiple times.
This implementation is a very simple, initial implementation and there
is a bunch of changes I'd like to add on top of it.
And also perhaps a bunch of cleanups.
I've chosen to expose these flag _only_ for websites for now.
Part of https://github.com/ooni/probe/issues/1283.
This diff pins to ooni/probe-engine@3049779878
and starts using the recently introduced probe-engine APIs.
Namely, here, we use the InputLoader for loading URLs.
I've confirmed manually everything is still working as intended.
Part of https://github.com/ooni/probe/issues/1283.
(In particular, the InputLoader is the abstraction allowing us to load
input from several sources, including command line flags and external
files.)
* feat: use ooni/probe-engine@286613b74e and cleanup
1. zap unused configuration settings from the config file but do not
bump the version number because doing that _may_ interact in unexpected
ways with probe-desktop (hence https://github.com/ooni/probe/issues/1297)
and also because we've just _removed_ stuff for now, therefore any
previous configuration file will continue to work, except that we'll
be ignoring a bunch of options. In a future version of probe-cli I'll
spend some time to further improve config file management.
2. accordingly, make sure all current configuration files that are around
in the tree are current and only feature supported options.
3. update to ooni/probe-engine@286613b74e, which contains a bunch of
APIs that should allow us to simplify the interaction between the cli and
the engine, by sharing code more cleverly.
4. zap GetTestKeys because now we use code in probe-engine instead.
5. zap LogSummary because it was not being used.
6. the main change related to cleaning up the config and to the update
to the latest probe-engine is that include_{cc,asn,ip} settings are
gone and we now share the CC and the ASN and we never share the IP addr.
Reference issue: https://github.com/ooni/probe/issues/1283.
After this change is landed, there's a bunch more work to do to further
unify cli and engine. The final state will be that the cli uses ~the code
used by miniooni, so it will have a bunch of desirable options.
* fix: bindata after recent changes
When the input is /dev/null, every read returns EOF. In general, it
may also happen that read doesn't work as intended. So, the robust thing
to do here is to ensure that we check the return values. By doing that
we notice of io.EOF errors and we don't proceed with the onboarding.
This diff fixes the issue described by https://github.com/ooni/probe/issues/1281
however it may be that we also want (in the near or not-so-near future)
to stop onboarding if the input terminal is not a tty. This is however a
possible future evolution that should not prevent us for committing and
merging this simple fix that unblocks creating a Debian package.
* chore: set version to 3.0.12-alpha
I need to bless 3.0.11 now to pin to ooni/probe-engine v0.20.2.
* chore: update all dependencies
Most notably, pin to ooni/probe-engine 0.20.2.
We are working on ooniprobe for Debian. Before starting to apply
changes to the codebase, I'd like to apply some refactoring steps
that I've been thinking about for quite some time.
The general concept here is that the purpose of this repository
changed since it was designed and now there is probe-engine which
is a library, therefore, this repo can be mostly private.
* Update go-bindata and regenerate binary data
* Pin to ooni/probe-engine 0.17.0 and update dependencies
* Set version to 3.0.7
* Readme.md: better release instructions
* Use ooni/probe-engine 0.16.0
* Update all the other dependencies
* Use GitHub Actions rather than Travis CI
* Automatically build and test binaries on the target OS (for Windows, macOS, Linux on amd64)
* Make sure we correctly measure coverage
* Make sure we use `-race` when running tests
* Remove unnecessary scripts
* Make sure the README is up-to-date
* Write small script to update binary data and add GitHub Actions checks for it
* Notice that we needed to run ./updatebindata.sh and run it
* Self documenting instructions regarding cross compiling
* Set version number to v3.0.7-beta
Part of https://github.com/ooni/probe-engine/issues/748
* nettests/groups.go: remove redundant struct names
* go.mod go.sum: update deps except probe-engine
* Update to ooni/probe-engine@e768161f91
The API has changed. Methods that used to change bits of the session have
been removed. Now the session is more immutable than before.
As such, we need to completely fill the config before using it.
* Set IncludeCountry to always true
Co-authored-by: Arturo Filastò <arturo@filasto.net>
