* fix(webconnectivity): expose network events
By not exposing network events in webconnectivity, we are missing
several interesting, explanatory data points.
This diff fixes the issue by:
1. enriching the definition of network events to include extra
data useful for performing (manual) data analysis;
2. adding a tags field to network events such that we can add
tags to specific events and understand where they come from;
3. exposing all the (tagged) network events that happen when running
a webconnectivity experiment.
See https://github.com/ooni/probe-engine/issues/1157.
* progress
* more work towards landing this diff
* Apply suggestions from code review
* Add signal to the im test group
* fix(ipconfig_test.go): disable when running in CI
Reference issue: https://github.com/ooni/probe/issues/1418
* fix(geolocate): remove unused variable
Came across this while looking into this issue with the CI that
is now failing. Guess fixing it here comes across as leaving the
camp slightly less in a bad shape than how I found it.
Co-authored-by: Simone Basso <bassosimone@gmail.com>
* fix(geolocate): no proxy when discovering our IP address
The use case of --proxy is that you cannot contact the OONI
backend otherwise. It is wrong, though, using the proxy when
discovering our IP address. The measurement won't use the
proxy anyway. Therefore, we need to use the IP address that
is performing the measurement. Not the one of the proxy.
What's more, stun is not using a proxy. Therefore, it does
not make much sense that http IP resolvers use a proxy. This
leads to inconsistencies. So, here's anothe reason why this
patch is a good thing (TM).
Finally, because knowing the IP address enables us to sanitize
the data, it's important we discover the correct IP.
Now, up until this point, the `--proxy` option has mostly
been a developers toy. But, users have asked us to have the
possibility of configuring a proxy.
This explains why I have been looking into making `--proxy`
right for a couple of hours now.
See https://github.com/ooni/probe/issues/1382
* fix(session): properly configure the IP lookupper
In reality, we are not going to use the sessionresolver when we're
using a proxy (I just tested). But, it nonetheless feels a lot more
robust to write a correct sessionresolver that handles the proxy
in the most correct way. That is, the sessionresolver will now skip
all the entries that cannot use a socks5 proxy (including among them
also the system resolver). What's more, it will construct a child
resolver that propagates the proxy.
We have confidence that this holds true because we have added a test
ensuring that we are really using the configured proxy.
See https://github.com/ooni/probe/issues/1381
This comes just a few days after 3.6.0. It contains small
improvements required by ooni/probe-desktop.
For this reason, I am going to skeep the normal release
process and I am just bumping the version number.
* fix(webconnectivity): allow measuring https://1.1.1.1
There were two issues preventing us from doing so:
1. in netx, the address resolver was too later in the resolver
chain. Therefore, its result wasn't added to the events.
2. when building the DNSCache (in httpget.go), we didn't consider
the case where the input is an address. We need to treat this
case specially to make sure there is no DNSCache.
See https://github.com/ooni/probe/issues/1376.
* fix: add unit tests for code making the dnscache
* fix(netx): make sure all tests pass
* chore: bump webconnectivity version
* internal/engine/ooapi: auto-generated API client
* feat: introduce the callers abstraction
* feat: implement API caching on disk
* feat: implement cloneWithToken when we require login
* feat: implement login
* fix: do not cache all APIs
* feat: start making space for more tests
* feat: implement caching policy
* feat: write tests for caching layer
* feat: add integration tests and fix some minor issues
* feat: write much more unit tests
* feat: add some more easy unit tests
* feat: add tests that use a local server
While there, make sure many fields we care about are OK.
* doc: write basic documentation
* fix: tweak sentence
* doc: improve ooapi documentation
* doc(ooapi): other documentation improvements
* fix(ooapi): remove caching for most APIs
We discussed this topic yesterday with @FedericoCeratto. The only
place where we want LRU caching is MeasurementMeta.
* feat(ooapi): improve handling of errors during login
This was also discussed yesterday with @FedericoCeratto
* fix(swaggerdiff_test.go): temporarily disable
Before I work on this, I need to tend onto other tasks.
* fix(ootest): add one more test case
We're going towards 100% coverage of this package, as it ought to be.
* feat(ooapi): test cases for when the probe clock is off
* fix(ooapi): change test to have 100% unittest coverage
* feat: sync server and client APIs definition
Companion PR: https://github.com/ooni/api/pull/218
* fix(ooapi): start testing again against API
* fix(ooapi): only generate each file once
* chore: set version to 3.7.0-alpha
While there, make sure we don't always skip a currently failing
riseupvpn test, and slightly clarify the readme.
* fix(kvstore): less scoped error message
* chore: update dependencies
* chore: update user agent for measurements
* chore: we're now at v3.6.0
* chore: update assets
* chore: update bundled CA
* fix: address some goreportcard.com warnings
* fix(debian/changelog): zap release that breaks out build scripts
We're forcing the content of changelog with `dch`, so it's fine to
not have any specific new release in there.
* fix: make sure tests are passing locally
Notably, I removed a chunk of code where we were checking for network
activity. Now we don't fetch the databases and it's not important. Before,
it was important because the databases are ~large.
* fix: temporarily comment out riseupvn integration tests
See https://github.com/ooni/probe/issues/1354 for work aimed at
reducing the rate of false positives (thanks @cyBerta!)
* feat(sessionresolver): try many and use what works
* fix(sessionresolver): make sure we can use quic
* fix: the config struct is unnecessary
* fix: make kvstore optional
* feat: write simple integration test
* feat: start adding tests
* feat: continue writing tests
* fix(sessionresolver): add more unit tests
* fix(sessionresolver): finish adding tests
* refactor(sessionresolver): changes after code review
* feat: use go1.16 embedding for resources
We want to embed everything that can be easily embedded. We should, at a
minimum, replace the downloading of resources and bindata.
Ref: https://github.com/ooni/probe/issues/1367.
* fix: get rid of bindata and use go embed instead
* fix: start unbreaking some automatic tests
* fix: fetch resources as part of the mobile build
* fix: convert more stuff to go1.16
I still expect many breakages, but we'll fix them.
* fix: make the windows CI green
* fix: get resources before running QA
* fix: go1.16 uses modules by default
* hopefully fix all other outstanding issues
* fix(QA/telegram.py): add another DC IP address
* Apply suggestions from code review
* MVP of a signal messenger test
* Add minimal signal test unit tests
* Add Signal test to the im nettest group
* Add test for https://sfu.voip.signal.org/
* Fix bug in client-side determination of blocking status
* Add uptime.signal.org to the test targets
* Add more tests
* Check for invalid CA being passed
* Check that the update function works as expected
* Update internal/engine/experiment/signal/signal_test.go
Co-authored-by: Simone Basso <bassosimone@gmail.com>
* fix: back out URL we shouldn't have changed
When merging probe-engine into probe-cli, we changed too many URLs
and some of them should not have been changed.
I noticed this during the review of Signal and I choose to add
this commit to revert such changes.
While there, make sure the URL of the experiment is OK.
* fix(signal): reach 100% of coverage
Just so that we can focus on areas of the codebase where we need
more coverage, let us avoid missing an easy line to test.
Co-authored-by: Simone Basso <bassosimone@gmail.com>
This is useful when someone is running manual measurements and is
sharing their measurements with us, because it means we don't need
manually keep track of this bit of information.
Closes https://github.com/ooni/probe-engine/issues/1194
* feat: add end-to-end testing to this repository
Part of https://github.com/ooni/probe-engine/issues/1181
Motivation: we want to run this check from the repository where
we work the most, such that it's unlikely it pauses due to inactivity,
as it may happen for less frequently touched upon repositories.
Code adapted from https://github.com/ooni/e2etesting/
* fix: correct name for main branch
* doc: ensure all top dirs have an explanatory README
This makes the repository a lil bit nicer to newcomers.
Part of https://github.com/ooni/probe/issues/1335
* fix: re-run bindata to embed the README
The readme is small, so we can pay the price of adding it.
On a related note, I am very pleased the Go team implemented the
`//go:embed` feature, so we can get rid of this bindata thing.
* chore: remove duplicate code of conduct
* chore: remove AUTHORS file
I doubt this actually has any value in the era of GitHub.
* chore: move CODEOWNERS to toplevel
* chore: move CONTRIBUTING.md to toplevel and adapt it
* chore: remove duplicated LICENSE file
* chore(engine): remove now-obsolete design document
* chore: remove the testusing test
We're not going to make this code importable from third parties
like we did for probe-engine. It seems this feature was only used
for the experiment in Spain so it makes sense to drop it.
* chore: enable code generation tests
See https://github.com/ooni/probe/issues/1335
* chore: enable code-ql checks
* cleanup: remove libooniffi code and tests
It seems this code is not used. We are not aware of anyone using it. And we
don't want to expose it publicly as an API. So, what to do?
I guess it's fine to delete it. If there is anyone that needs it, we have
in the history a reference to it and we can always reinstate it.
* chore: move issue templates to ooni/probe
* refactor: move more commands to internal/cmd
Part of https://github.com/ooni/probe/issues/1335.
We would like all commands to be at the same level of engine
rather than inside engine (now that we can do it).
* fix: update .gitignore
* refactor: also move jafar outside engine
* We should be good now?
* refactor: miniooni should be outside of the engine
This is part of https://github.com/ooni/probe/issues/1335. We also need
to think whether we wanna keep libminiooni and miniooni separated.
The previous use case for having a top-level libminiooni was that of
enabling others to integrate miniooni into other binaries.
This was usegul when studying internet censorship in Spain in May 2020.
I am wondering whether we should be keeping this complexity. I am not
sure about this and probably we should be killing it.
(In any case, reducing complexity is not the objective of this diff,
since I would like instead to move things around with minimal changes
and make sure we have a ~good repository organization here.)
* fix: import in libminiooni
* refactor: start building an Android package
Part of https://github.com/ooni/probe/issues/1335.
This seems also a good moment to move some packages out of the
engine, e.g., oonimkall. This package, for example, is a consumer
of the engine, so it makes sense it's not _inside_ it.
* fix: committed some stuff I didn't need to commit
* fix: oonimkall needs to be public to build
The side effect is that we will probably need to bump the major
version number every time we change one of these APIs.
(We can also of course choose to violate the basic guidelines of Go
software, but I believe this is bad form.)
I have no problem in bumping the major quite frequently and in
any case this monorepo solution is convinving me more than continuing
to keep a split between engine and cli. The need to embed assets to
make the probe more reliable trumps the negative effects of having to
~frequently bump major because we expose a public API.
* fix: let's not forget about libooniffi
Honestly, I don't know what to do with this library. I added it
to provide a drop in replacement for MK but I have no idea whether
it's used and useful. I would not feel comfortable exposing it,
unlike oonimkall, since we're not using it.
It may be that the right thing to do here is just to delete the
package and reduce the amount of code we're maintaining?
* woops, we're still missing the publish android script
* fix(publish-android.bash): add proper API key
* ouch fix another place where the name changed
This diff is part of https://github.com/ooni/probe/issues/1335.
We are moving more probe-engine workflows to toplevel.
The general idea here is to migrate all possible workflows and to
delete the ones that we cannot use in this repo (if any).
* refactor: build miniooni from toplevel
Of course, also move the specific test checking whether we are
still able of building miniooni.
Part of https://github.com/ooni/probe/issues/1335
* build for current branch just to confirm
* fix: correct the path where linux/arm binary is
* okay, it works, we can remove the special rule
This is how I did it:
1. `git clone https://github.com/ooni/probe-engine internal/engine`
2. ```
(cd internal/engine && git describe --tags)
v0.23.0
```
3. `nvim go.mod` (merging `go.mod` with `internal/engine/go.mod`
4. `rm -rf internal/.git internal/engine/go.{mod,sum}`
5. `git add internal/engine`
6. `find . -type f -name \*.go -exec sed -i 's@/ooni/probe-engine@/ooni/probe-cli/v3/internal/engine@g' {} \;`
7. `go build ./...` (passes)
8. `go test -race ./...` (temporary failure on RiseupVPN)
9. `go mod tidy`
10. this commit message
Once this piece of work is done, we can build a new version of `ooniprobe` that
is using `internal/engine` directly. We need to do more work to ensure all the
other functionality in `probe-engine` (e.g. making mobile packages) are still WAI.
Part of https://github.com/ooni/probe/issues/1335
This release only contains updates in debian packaging. All other platforms
could safely continue to use 3.2.0 or 3.3.0.
I didn't want to make a path release, though, because I didn't want to convey
the meaning that something was fixed.
Related to https://github.com/ooni/ooni.org/issues/677
* feat: sketch out periodic command
* feat: sketch out periodic command for macOS
* feat: implement darwin's launch agent
* refactor: better way to run on darwin
Make sure we have code that builds on all platforms.
* fix(run): max 10 URLs with darwin in unattended mode
* feat: add support for seeing/streaming logs
* feat: implement the status command and add usage hints
* feat(periodic): run onboarding if needed
* fix: no too confusing function names
* fix: s/periodic/autorun/
Discussed earlier this morning with @hellais.
* fix: we cannot show logs before Big Sur
Bug reported by @hellais.
