1. add eof channel to event emitter and use this channel as signal
that we shouldn't be sending anymore instead of using a pattern where
we use a timer to decide sending has timed out (because we're using
a buffered channel, it is still possible for some evetns to end up
in the channel if there is space, but this is not a concern, because
the events will be deleted when the channel itself is gone);
2. refactor all tests where we assumed the output channel was closed
to actually use a parallel "eof" channel and use it as signal we
should not be sending anymore (not strictly required but still the
right thing to do in terms of using consistent patterns);
3. modify how we construct a runner so that it passes to the
event emitter an "eof" channel and closes this channel when the
main goroutine running the task is terminating;
4. modify the task to signal events such as "task goroutine started"
and "task goroutine stopped" using channels, which helps to write
much more correct tests;
5. take advantage of the previous change to improve the test that
ensures we're not blocking for a small number of events and also
improve the name of such a test to reflect what it's testing.
The related issue in term of fixing the channel usage pattern is
https://github.com/ooni/probe/issues/1438.
Regarding improving testability, instead, the correct reference
issue is https://github.com/ooni/probe/issues/1903.
There are possibly more changes to apply here to improve this package
and its testability, but let's land this diff first and then see
how much time is left for further improvements.
I've run unit and integration tests with `-race` locally.
This diff will need to be backported to `release/3.11`.
This diff forward ports 90bf0629b957c912a5a6f3bb6c98ad3abb5a2ff6 to `master`.
If we close the channel to signal the end of a task we may panic
when some background goroutine tries to post on the channel.
This bug is rare but may happen.
See for example https://github.com/ooni/probe/issues/1438.
How can we improve?
First, let us add a timeout when sending to the channel. Given that
the channel is buffered and we have a generous timeout (1/4 of a
second), it's unlikely we will really block. But, in the event in
which a late message appears, we'll eventually _unblock_ when
sending with a timeout. So, now we don't have to worry anymore
about leaking forever a goroutine.
Then, let us change the protocol with which we signal that a task
is done. We used to close the channel. Now, instead we just
synchronously post a nil on the channel when done.
In turn, we interpret this nil to mean that the task is done when
we receive messages.
The _main_ different with respect to before is that now we are
asking the consumer of our API to drain the channel. Because
before we had a blocking channel, it seems to me we were already
requiring the consumer of the API to do that. Which means, I think
in practical terms it did not change much.
Finally, acknowledge that we don't need a specific state variable
to tell us we're done and simplify a little bit the API by
just making isRunning private and using the "we're done" signal
to determine whether we've stopped running the task.
All these changes should be enough to close https://github.com/ooni/probe/issues/1438.
* fix(pkg.go.dev): import a subpackage containing the assets
We're trying to fix this issue that pkg.go.dev does not build.
Thanks to @hellais for this very neat idea! Let's keep our
fingers crossed and see whether it fixes!
* feat: use embedded geoip databases
Closes https://github.com/ooni/probe/issues/1372.
Work done as part of https://github.com/ooni/probe/issues/1369.
* fix(assetsx): add tests
* feat: simplify and just vendor uncompressed DBs
* remove tests that seems not necessary anymore
* fix: run go mod tidy
* Address https://github.com/ooni/probe-cli/pull/260/files#r605181364
* rewrite a test in a better way
* fix: gently cleanup the legacy assetsdir
Do not remove the whole directory with brute force. Just zap the
files whose name we know. Then attempt to delete the legacy directory
as well. If not empty, just fail. This is fine because it means the
user has stored other files inside the directory.
* fix: create .miniooni if missing
* feat(session): expose CheckIn method
It seems to me the right thing to do is to query the CheckIn API
from the Session rather than querying it from InputLoader.
Then, InputLoader could just take a reference to a Session-like
interface that allows this functionality.
So, this diff exposes the Session.CheckIn method.
Doing that, in turn, required some refactoring to allow for
more and better unit tests.
While doing that, I also noticed that Session required a mutex
to be a well-behaving type, so I did that.
While doing that, I also tried to cover all the lines in session.go
and, as part of that, I have removed unused code.
Reference issue: https://github.com/ooni/probe/issues/1299.
* fix: reinstate comment I shan't have removed
* fix: repair broken test
* fix: a bit more coverage, annotations, etc.
* Update internal/engine/session.go
* Update internal/engine/session_integration_test.go
* Update internal/engine/session_internal_test.go
We use ExperimentOrchestraClient in several places to help us
calling probe-services APIs. We need to call CheckIn because we
want to use CheckIn in InputLoader.
(We also want to remove the URLs API, but that is not something
doable now, since the mobile app is still using this API via
the wrappers at pkg/oonimkall.)
Work part of https://github.com/ooni/probe/issues/1299.
* MVP of a signal messenger test
* Add minimal signal test unit tests
* Add Signal test to the im nettest group
* Add test for https://sfu.voip.signal.org/
* Fix bug in client-side determination of blocking status
* Add uptime.signal.org to the test targets
* Add more tests
* Check for invalid CA being passed
* Check that the update function works as expected
* Update internal/engine/experiment/signal/signal_test.go
Co-authored-by: Simone Basso <bassosimone@gmail.com>
* fix: back out URL we shouldn't have changed
When merging probe-engine into probe-cli, we changed too many URLs
and some of them should not have been changed.
I noticed this during the review of Signal and I choose to add
this commit to revert such changes.
While there, make sure the URL of the experiment is OK.
* fix(signal): reach 100% of coverage
Just so that we can focus on areas of the codebase where we need
more coverage, let us avoid missing an easy line to test.
Co-authored-by: Simone Basso <bassosimone@gmail.com>
* refactor: start building an Android package
Part of https://github.com/ooni/probe/issues/1335.
This seems also a good moment to move some packages out of the
engine, e.g., oonimkall. This package, for example, is a consumer
of the engine, so it makes sense it's not _inside_ it.
* fix: committed some stuff I didn't need to commit
* fix: oonimkall needs to be public to build
The side effect is that we will probably need to bump the major
version number every time we change one of these APIs.
(We can also of course choose to violate the basic guidelines of Go
software, but I believe this is bad form.)
I have no problem in bumping the major quite frequently and in
any case this monorepo solution is convinving me more than continuing
to keep a split between engine and cli. The need to embed assets to
make the probe more reliable trumps the negative effects of having to
~frequently bump major because we expose a public API.
* fix: let's not forget about libooniffi
Honestly, I don't know what to do with this library. I added it
to provide a drop in replacement for MK but I have no idea whether
it's used and useful. I would not feel comfortable exposing it,
unlike oonimkall, since we're not using it.
It may be that the right thing to do here is just to delete the
package and reduce the amount of code we're maintaining?
* woops, we're still missing the publish android script
* fix(publish-android.bash): add proper API key
* ouch fix another place where the name changed
