We use an optional build tag to hide this configuration. When you
choose this configuration, you need to provide the encrypted config
as well as the corresponding decryption key.
This is not the final design. This is an interim design to start
working and experimenting with this functionality. The general
idea here is to support psiphon in the binaries we build without
committing the psiphon config to the repository itself.
Part of https://github.com/ooni/probe/issues/985
* ongoing
* while there, make sure we test everything
* reorganize previous commit
* ensure we have reasonable coverage in session
The code in here would be better with unit tests. We have too many
integration tests and the tests overall are too slow. But it's also
true that I should not write a giant diff as part of this PR.
* chore(oohelper): increase tests verbosity
Hopefully this helps with https://github.com/ooni/probe/issues/1409.
* fix(oohelper): use a nonstandard resolver
* fix previous
* make the diff pleasant/committable/correct
* chore: update the user-agent we use
Part of the check-list at https://github.com/ooni/probe/issues/1369.
* chore: set version to 3.9.0
See https://github.com/ooni/probe/issues/1369
* chore: run go generate ./...
This is meant to update the bundled CA. We have heard of issues with
our bundled CA, but it seems there have been no changes upstream.
The website https://curl.se/docs/caextract.html still lists as the
last change the one done on Jan 19, 2021, which is the version of
the CA that we're currently bundling.
For the sake of continuing with the release process, I am going
to further investigate the CA once the release is done.
This chore is part of https://github.com/ooni/probe/issues/1369.
* fix(pkg.go.dev): import a subpackage containing the assets
We're trying to fix this issue that pkg.go.dev does not build.
Thanks to @hellais for this very neat idea! Let's keep our
fingers crossed and see whether it fixes!
* feat: use embedded geoip databases
Closes https://github.com/ooni/probe/issues/1372.
Work done as part of https://github.com/ooni/probe/issues/1369.
* fix(assetsx): add tests
* feat: simplify and just vendor uncompressed DBs
* remove tests that seems not necessary anymore
* fix: run go mod tidy
* Address https://github.com/ooni/probe-cli/pull/260/files#r605181364
* rewrite a test in a better way
* fix: gently cleanup the legacy assetsdir
Do not remove the whole directory with brute force. Just zap the
files whose name we know. Then attempt to delete the legacy directory
as well. If not empty, just fail. This is fine because it means the
user has stored other files inside the directory.
* fix: create .miniooni if missing
* feat(ooniprobe): discard lists not in selected categories
One day we may make an integration mistake and for any reason
we may end up with URLs that do not belong to the categories
originally selected by the user. If that happens, it's nice to
have a safety net where we remove URLs that do not belong to
the right category before proceeding with testing.
This diff was conceived while discussing the robustness of
https://github.com/ooni/probe/issues/1299 with @hellais.
* fix behavior and add unit test
* more robust
* fetch RiseupVPN CA cert with MultiGetter. It allows us to write better tests and ensures this test step is added in the logs
* Implement TransportStatus for RiseupVPN tests. It indicates if a whole transport is blocked, which is considered as a test anomaly
* Redesign unit tests for RiseupVPN. Instead of a real backend, mocked server responses are used. Tests for invalid CA certs and for TransportStatus are added.
* Update internal/engine/experiment/riseupvpn/riseupvpn.go
Co-authored-by: Simone Basso <bassosimone@gmail.com>
We cannot control anymore the maximum number of URLs using the API
because now we are using check-in, that has no such limit.
We could theoretically clamp the number of URLs to measure after
the call to check-in, and still honour the setting.
Yet, the right thing to do seems to introduce a max runtime variable
because that is what desktop and mobile do.
Thus, introduce code that warns the user about the change in the
settings, should they have set the URL limit to nonzero.
We are going to do a best effort conversion from the URL limit to
the maximum runtime for the rest of 2021.
Since then, we will silently ignore the URL limit.
This work is part of https://github.com/ooni/probe/issues/1299.
This change allows us to have all fasts tests together. They are
mostly unit tests or integration tests that do not require the
network. The advantage of this strategy is the following. We can
now run all these tests with a single click in VSCode. In turn,
doing that tells us which lines of code we are not covering.
The tests requiring the network are in a separate file, so we can
easily see which lines of code are testing without using the network
and which ones instead depend on that. (Currently, 100% of the
inputloader.go file is tested without using the network.)
While there, rename the other file such that is clear that it
contains tests requiring the network. We now have some tests in
inputloader_test.go that are not strictly unit tests.
This refactoring was identified as useful while working
on https://github.com/ooni/probe/issues/1299.
We used to have an external package called libminiooni so that
third parties could use it. We wrote this such that we could
support github.com/bassosimone/aladdin.
That was actually a not-so-good idea because it added to the APIs
we needed to maintain.
Since the merge of engine into cli, such an API is not public
anymore and aladdin has been deprecated and archived.
Therefore, we can now cleanup the situation and merge libminiooni
into miniooni again, thus making the codebase more local.
This cleanup has been identified while working on
https://github.com/ooni/probe/issues/1299.
* ongoing work
* reduce diff with master
* feat(inputloader): use the check-in API
Part of https://github.com/ooni/probe/issues/1299
* fix: better naming for a variable
* chore: add more tests
* fix: add one more TODO
* feat(session): expose CheckIn method
It seems to me the right thing to do is to query the CheckIn API
from the Session rather than querying it from InputLoader.
Then, InputLoader could just take a reference to a Session-like
interface that allows this functionality.
So, this diff exposes the Session.CheckIn method.
Doing that, in turn, required some refactoring to allow for
more and better unit tests.
While doing that, I also noticed that Session required a mutex
to be a well-behaving type, so I did that.
While doing that, I also tried to cover all the lines in session.go
and, as part of that, I have removed unused code.
Reference issue: https://github.com/ooni/probe/issues/1299.
* fix: reinstate comment I shan't have removed
* fix: repair broken test
* fix: a bit more coverage, annotations, etc.
* Update internal/engine/session.go
* Update internal/engine/session_integration_test.go
* Update internal/engine/session_internal_test.go
There was a face-palming error in the implementation causing the proxy
check to be implemented also without a proxy.
This meant that we were ALWAYS skipping http3 and system resolvers.
The bug has been introduced in 3.8.0. So, the currently released
version of the probe, sadly, has this beheavior :-(.
Reference issue https://github.com/ooni/probe/issues/1426.
We use ExperimentOrchestraClient in several places to help us
calling probe-services APIs. We need to call CheckIn because we
want to use CheckIn in InputLoader.
(We also want to remove the URLs API, but that is not something
doable now, since the mobile app is still using this API via
the wrappers at pkg/oonimkall.)
Work part of https://github.com/ooni/probe/issues/1299.
* fix(webconnectivity): expose network events
By not exposing network events in webconnectivity, we are missing
several interesting, explanatory data points.
This diff fixes the issue by:
1. enriching the definition of network events to include extra
data useful for performing (manual) data analysis;
2. adding a tags field to network events such that we can add
tags to specific events and understand where they come from;
3. exposing all the (tagged) network events that happen when running
a webconnectivity experiment.
See https://github.com/ooni/probe-engine/issues/1157.
* progress
* more work towards landing this diff
* Apply suggestions from code review
* Add signal to the im test group
* fix(ipconfig_test.go): disable when running in CI
Reference issue: https://github.com/ooni/probe/issues/1418
* fix(geolocate): remove unused variable
Came across this while looking into this issue with the CI that
is now failing. Guess fixing it here comes across as leaving the
camp slightly less in a bad shape than how I found it.
Co-authored-by: Simone Basso <bassosimone@gmail.com>
* poc: mobile api for running WebConnectivity
Yesterday we had a team meeting where we discussed the importance
of stopping using MK wrappers for running experiments.
Here's a PoC showing how we can do that for WebConnectivity. It
seems to me we probably want to add more tests and merge this code
such that we can experiment with it quite soon.
There seems to be opportunities for auto-generating code, BTW.
While working on this PoC, I've also took a chance to revamp the
external documentation of pkg/oonimkall.
* feat: start making the code more abstract
* chore: write unit tests for new code
* fix(oonimkall): improve naming
* refactor: cosmetic changes
* fix: explain
* fix(geolocate): no proxy when discovering our IP address
The use case of --proxy is that you cannot contact the OONI
backend otherwise. It is wrong, though, using the proxy when
discovering our IP address. The measurement won't use the
proxy anyway. Therefore, we need to use the IP address that
is performing the measurement. Not the one of the proxy.
What's more, stun is not using a proxy. Therefore, it does
not make much sense that http IP resolvers use a proxy. This
leads to inconsistencies. So, here's anothe reason why this
patch is a good thing (TM).
Finally, because knowing the IP address enables us to sanitize
the data, it's important we discover the correct IP.
Now, up until this point, the `--proxy` option has mostly
been a developers toy. But, users have asked us to have the
possibility of configuring a proxy.
This explains why I have been looking into making `--proxy`
right for a couple of hours now.
See https://github.com/ooni/probe/issues/1382
* fix(session): properly configure the IP lookupper
In reality, we are not going to use the sessionresolver when we're
using a proxy (I just tested). But, it nonetheless feels a lot more
robust to write a correct sessionresolver that handles the proxy
in the most correct way. That is, the sessionresolver will now skip
all the entries that cannot use a socks5 proxy (including among them
also the system resolver). What's more, it will construct a child
resolver that propagates the proxy.
We have confidence that this holds true because we have added a test
ensuring that we are really using the configured proxy.
See https://github.com/ooni/probe/issues/1381
This comes just a few days after 3.6.0. It contains small
improvements required by ooni/probe-desktop.
For this reason, I am going to skeep the normal release
process and I am just bumping the version number.
* refactor: signal et al. are now experimental nettests
We move signal into the experimental nettests group. While there,
also start adding dnscheck and stunreachability as well.
It seems there's more work to be done to correctly represent
the results of dnscheck, but this is fine!
The experimental section is here exactly for this reason!
In terms of UI, the new command is `ooniprobe run experimental`.
We will most likely move signal out of experimental soon, since it's
already working quite well. We need to keep it here for one more
cycle because the desktop app is not ready for it.
See the following issues:
1. https://github.com/ooni/probe/issues/1378
2. https://github.com/ooni/probe/issues/1262
* fix(dnscheck): spell check
* fix: improve documentation
* fix(webconnectivity): allow measuring https://1.1.1.1
There were two issues preventing us from doing so:
1. in netx, the address resolver was too later in the resolver
chain. Therefore, its result wasn't added to the events.
2. when building the DNSCache (in httpget.go), we didn't consider
the case where the input is an address. We need to treat this
case specially to make sure there is no DNSCache.
See https://github.com/ooni/probe/issues/1376.
* fix: add unit tests for code making the dnscache
* fix(netx): make sure all tests pass
* chore: bump webconnectivity version
