* ongoing
* while there, make sure we test everything
* reorganize previous commit
* ensure we have reasonable coverage in session
The code in here would be better with unit tests. We have too many
integration tests and the tests overall are too slow. But it's also
true that I should not write a giant diff as part of this PR.
* fix(pkg.go.dev): import a subpackage containing the assets
We're trying to fix this issue that pkg.go.dev does not build.
Thanks to @hellais for this very neat idea! Let's keep our
fingers crossed and see whether it fixes!
* feat: use embedded geoip databases
Closes https://github.com/ooni/probe/issues/1372.
Work done as part of https://github.com/ooni/probe/issues/1369.
* fix(assetsx): add tests
* feat: simplify and just vendor uncompressed DBs
* remove tests that seems not necessary anymore
* fix: run go mod tidy
* Address https://github.com/ooni/probe-cli/pull/260/files#r605181364
* rewrite a test in a better way
* fix: gently cleanup the legacy assetsdir
Do not remove the whole directory with brute force. Just zap the
files whose name we know. Then attempt to delete the legacy directory
as well. If not empty, just fail. This is fine because it means the
user has stored other files inside the directory.
* fix: create .miniooni if missing
There was a face-palming error in the implementation causing the proxy
check to be implemented also without a proxy.
This meant that we were ALWAYS skipping http3 and system resolvers.
The bug has been introduced in 3.8.0. So, the currently released
version of the probe, sadly, has this beheavior :-(.
Reference issue https://github.com/ooni/probe/issues/1426.
We use ExperimentOrchestraClient in several places to help us
calling probe-services APIs. We need to call CheckIn because we
want to use CheckIn in InputLoader.
(We also want to remove the URLs API, but that is not something
doable now, since the mobile app is still using this API via
the wrappers at pkg/oonimkall.)
Work part of https://github.com/ooni/probe/issues/1299.
In reality, we are not going to use the sessionresolver when we're
using a proxy (I just tested). But, it nonetheless feels a lot more
robust to write a correct sessionresolver that handles the proxy
in the most correct way. That is, the sessionresolver will now skip
all the entries that cannot use a socks5 proxy (including among them
also the system resolver). What's more, it will construct a child
resolver that propagates the proxy.
We have confidence that this holds true because we have added a test
ensuring that we are really using the configured proxy.
See https://github.com/ooni/probe/issues/1381
* feat(sessionresolver): try many and use what works
* fix(sessionresolver): make sure we can use quic
* fix: the config struct is unnecessary
* fix: make kvstore optional
* feat: write simple integration test
* feat: start adding tests
* feat: continue writing tests
* fix(sessionresolver): add more unit tests
* fix(sessionresolver): finish adding tests
* refactor(sessionresolver): changes after code review
* refactor: move more commands to internal/cmd
Part of https://github.com/ooni/probe/issues/1335.
We would like all commands to be at the same level of engine
rather than inside engine (now that we can do it).
* fix: update .gitignore
* refactor: also move jafar outside engine
* We should be good now?
* refactor: miniooni should be outside of the engine
This is part of https://github.com/ooni/probe/issues/1335. We also need
to think whether we wanna keep libminiooni and miniooni separated.
The previous use case for having a top-level libminiooni was that of
enabling others to integrate miniooni into other binaries.
This was usegul when studying internet censorship in Spain in May 2020.
I am wondering whether we should be keeping this complexity. I am not
sure about this and probably we should be killing it.
(In any case, reducing complexity is not the objective of this diff,
since I would like instead to move things around with minimal changes
and make sure we have a ~good repository organization here.)
* fix: import in libminiooni
* refactor: start building an Android package
Part of https://github.com/ooni/probe/issues/1335.
This seems also a good moment to move some packages out of the
engine, e.g., oonimkall. This package, for example, is a consumer
of the engine, so it makes sense it's not _inside_ it.
* fix: committed some stuff I didn't need to commit
* fix: oonimkall needs to be public to build
The side effect is that we will probably need to bump the major
version number every time we change one of these APIs.
(We can also of course choose to violate the basic guidelines of Go
software, but I believe this is bad form.)
I have no problem in bumping the major quite frequently and in
any case this monorepo solution is convinving me more than continuing
to keep a split between engine and cli. The need to embed assets to
make the probe more reliable trumps the negative effects of having to
~frequently bump major because we expose a public API.
* fix: let's not forget about libooniffi
Honestly, I don't know what to do with this library. I added it
to provide a drop in replacement for MK but I have no idea whether
it's used and useful. I would not feel comfortable exposing it,
unlike oonimkall, since we're not using it.
It may be that the right thing to do here is just to delete the
package and reduce the amount of code we're maintaining?
* woops, we're still missing the publish android script
* fix(publish-android.bash): add proper API key
* ouch fix another place where the name changed
This is how I did it:
1. `git clone https://github.com/ooni/probe-engine internal/engine`
2. ```
(cd internal/engine && git describe --tags)
v0.23.0
```
3. `nvim go.mod` (merging `go.mod` with `internal/engine/go.mod`
4. `rm -rf internal/.git internal/engine/go.{mod,sum}`
5. `git add internal/engine`
6. `find . -type f -name \*.go -exec sed -i 's@/ooni/probe-engine@/ooni/probe-cli/v3/internal/engine@g' {} \;`
7. `go build ./...` (passes)
8. `go test -race ./...` (temporary failure on RiseupVPN)
9. `go mod tidy`
10. this commit message
Once this piece of work is done, we can build a new version of `ooniprobe` that
is using `internal/engine` directly. We need to do more work to ensure all the
other functionality in `probe-engine` (e.g. making mobile packages) are still WAI.
Part of https://github.com/ooni/probe/issues/1335
