This diff includes some final changes to be ready for blessing
a cli release. These changes are:
1. run `go generate ./...` to update the bundled CA
2. update the header we use for measuring
3. ensure `mk` uses the latest version of several tools
Reference issue: https://github.com/ooni/probe/issues/1845
This commit message is the same across probe-cli, probe-desktop,
and probe-android. With the changes contained in the enclosed
diff, I'm starting to add support for torsf for android and for
desktop.
When smoke testing that torsf was WAI, I also noticed that its
progress messages in output are too frequent. We may want to do
better in a future version when we'll be able to read `tor`'s
output. In the meanwhile, make the progress messages less
frequent and indicated the maximum runtime inside of the messages
themselves. This improved message, albeit not so nice from the
UX PoV, should at least provide a clue that we're not stuck.
Reference issue: https://github.com/ooni/probe/issues/1917
Reference issue: https://github.com/ooni/probe/issues/1917.
I needed to change the summary key type returned by `torsf` to be a value. It seems the DB layer assumes that. If we pass it a pointer, it panics because it's experiment a value rather than a pointer 🤷.
I have experimented with a new approach for embedding psiphon in
7fc0bcd97c.
It seems the build is still building and the experiment is still
running. With the new approach, we're now vendoring less dependencies,
which hopefully puts us in the right track to, one day, import
Psiphon as a normal Go dependency.
I'll make sure to report to the Psiphon team what is currently
preventing us from importing their ClientLibrary directly.
This work is part of https://github.com/ooni/probe/issues/1894.
As part of running the update, I run `go get -u -v ./...`, which
led to go-cmp also being updated in the process.
This diff contains the following changes and enhancements:
1. upgrade snowflake to v2
2. observe that we were not changing defaults from outside of snowflake.go, so remove code allowing to do that;
3. bump the timeout to 600 seconds (it seems 300 was not always enough based on my testing);
4. add useful knob to disable `torsf` progress (it's really annoying on console, we should do something about this);
5. ptx.go: avoid printing an error when the connection has just been closed;
6. snowflake: test AMP cache, see that it's not working currently, so leave it disabled.
Related issues: https://github.com/ooni/probe/issues/1845, https://github.com/ooni/probe/issues/1894, and https://github.com/ooni/probe/issues/1917.
This diff updates all the dependencies using the aforementioned
go command. What remains to be done is to check whether any
dependency that we're using has bumped their major verson number,
and I'll do that in a subsequent pull request.
Reference issue: https://github.com/ooni/probe/issues/1894.
We're starting to prepare a new release. The first step is to use
go1.17.6 in the following places:
1. everywhere we define the version of Go in this tree;
2. when we're building for Android (using ooni/go);
3. in our ooni/oohttp fork of Go net/http standard library.
Reference issue: https://github.com/ooni/probe/issues/1845
This diff introduces a new package called `./internal/archival`. This package collects data from `./internal/model` network interfaces (e.g., `Dialer`, `QUICDialer`, `HTTPTransport`), saves such data into an internal tabular data format suitable for on-line processing and analysis, and allows exporting data into the OONI data format.
The code for collecting and the internal tabular data formats are adapted from `measurex`. The code for formatting and exporting OONI data-format-compliant structures is adapted from `netx/archival`.
My original objective was to _also_ (1) fully replace `netx/archival` with this package and (2) adapt `measurex` to use this package rather than its own code. Both operations seem easily feasible because: (a) this code is `measurex` code without extensions that are `measurex` related, which will need to be added back as part of the process; (b) the API provided by this code allows for trivially converting from using `netx/archival` to using this code.
Yet, both changes should not be taken lightly. After implementing them, there's need to spend some time doing QA and ensuring all nettests work as intended. However, I am planning a release in the next two weeks, and this QA task is likely going to defer the release. For this reason, I have chosen to commit the work done so far into the tree and defer the second part of this refactoring for a later moment in time. (This explains why the title mentions "1/N").
On a more high-level perspective, it would also be beneficial, I guess, to explain _why_ I am doing these changes. There are two intertwined reasons. The first reason is that `netx/archival` has shortcomings deriving from its original https://github.com/ooni/netx legacy. The most relevant shortcoming is that it saves all kind of data into the same tabular structure named `Event`. This design choice is unfortunate because it does not allow one to apply data-type specific logic when processing the results. In turn, this choice results in complex processing code. Therefore, I believe that replacing the code with event-specific data structures is clearly an improvement in terms of code maintainability and would quite likely lead us to more confidently change and evolve the codebase.
The second reason why I would like to move forward these changes is to unify the codepaths used for measuring. At this point in time, we basically have two codepaths: `./internal/engine/netx` and `./internal/measurex`. They both have pros and cons and I don't think we want to rewrite whole experiments using `netx`. Rather, what we probably want is to gradually merge these two codepaths such that `netx` is a set of abstractions on top of `measurex` (which is more low-level and has a more-easily-testable design). Because saving events and generating an archival data format out of them consists of at least 50% of the complexity of both `netx` and `measurex`, it seems reasonable to unify this archival-related part of the two codebases as the first step.
At the highest level of abstraction, these changes are part of the train of changes which will eventually lead us to bless `websteps` as a first class citizen in OONI land. Because `websteps` requires different underlying primitives, I chose to develop these primitives from scratch rather than wrestling with `netx`, which used another model. The model used by `websteps` is that we perform each operation in isolation and immediately we save the results, while `netx` creates whole data structures and collects all the events happening via tracing. We believe the model used by `websteps` to be better because it does not require your code to figure out everything that happened after the measurement, which is a source of subtle bugs in the current implementation. So, when I started implementing websteps I extracted the bits of `netx` that could also be beneficial to `websteps` into a separate library, thus `netxlite` was born.
The reference issue describing merging the archival of `netx` and `measurex` is https://github.com/ooni/probe/issues/1957. As of this writing the issue still references the original plan, which I could not complete by the end of this Sprint, so I am going to adapt the text of the issue to only refer to what was done in here next. Of course, I also need follow-up issues.
* Refactor the list measurements function to make use of nested queries
With a dataset of 1489 test, the ooniprobe list command went from
taking 17.27s to run, to requiring 0.17s or a 100x speed boost
See https://github.com/ooni/probe/issues/1966
* Remove dead code from actions
* Improve the tests for the ListResults function
* Add test for AnomalyCount
* Add more documentation about the merging of the test_keys
* chore(netxlite): add currently failing test case
This diff introduces a test cases that will fail because of the reason
explained in https://github.com/ooni/probe/issues/1965.
* chore(netxlite/iox_test.go): add failing unit tests
These tests directly show how the Go implementation of ReadAll
and Copy has the issue of checking for io.EOF equality.
* fix(netxlite): make {ReadAll,Copy}Context robust to wrapped io.EOF
The fix is simple: we just need to check for `errors.Is(err, io.EOF)`
after either io.ReadAll or io.Copy has returned. When this condition is
true, we need to convert the error back to `nil` as it ought to be.
While there, observe that the unit tests I committed in the previous
commit are wrongly asserting that the error must be wrapped. This
assertion is not correct, because in both cases we have just ensured
that the returned error is `nil` (i.e., success).
See https://github.com/ooni/probe/issues/1965.
* cleanup: remove previous workaround for wrapped io.EOF
These workarounds were partial, meaning that they would cover some
cases in which the issue occurred but not all of them.
Handling the problem in `netxlite.{ReadAll,Copy}Context` is the
right thing to do _as long as_ we always use these functions instead
of `io.{ReadAll,Copy}`.
This is why it's now important to ensure we clearly mention that
inside of the `CONTRIBUTING.md` guide and to also ensure that we're
not using these functions in the code base.
* fix(urlgetter): repair tests who assumed to see EOF error
Now that we have established that we should normalize EOF when
reading bodies like the stdlib does and now that it's clear why
our behavior diverged from the stdlib, we also need to repair
all the tests that assumed this incorrect behavior.
* fix(all): don't use io{,util}.{Copy,ReadAll}
* feat: add checks to ensure we don't use io.{Copy,ReadAll}
* doc(netxlite): document we know how to deal w/ wrapped io.EOF
* fix(nocopyreadall.bash): add exception for i/n/iox.go
The DNSClient type existed because the Resolver type did not
include CloseIdleConnections in its signature.
Now that Resolver includes CloseIdleConnections, the DNSClient
type has become unnecessary and can be safely removed.
See https://github.com/ooni/probe/issues/1956.
We recently started moving core data structures inside of the
internal/model package as detailed in https://github.com/ooni/probe/issues/1885.
The chief reason to do that is to have a set of fundamental
shared data types to help us rationalize the codebase.
This specific diff moves internal/netx/archival's core data types
inside the internal/model package. While there, it also refactors the
existing tests to improve their quality. Additionally, we also added
an extra test to ensure `ArchivalHTTPBody` is an alias for
`ArchivalMaybeBinaryData`, which is required to ensure the
custom JSON serialization process works for it.
We're doing that because both internal/netx/archival and
internal/measurex define their own archival data structures.
We developed measurex using its own structures because it
allowed to iterate more quickly. Now that we have sketched
out measurex, the time has come to consolidate.
My overall aim is to spend a few more hours this week on
engineering measurex. This work is preliminary work before
we finish up both measurex and websteps.
We described this cleanup in https://github.com/ooni/probe/issues/1957.
This diff addresses two items of https://github.com/ooni/probe/issues/1956:
> - [ ] we can remove legacy names from `./internal/engine/netx/resolver/legacy.go`
>
> - [ ] we can remove `DialTLSContext` from `./internal/engine/netx/resolver/tls_test.go`
More cleanups may follow.
This is another cleanup point mentioned by https://github.com/ooni/probe/issues/1956.
While there, fix a bunch of comments in jafar that were incorrectly
referring to the netx package name.
This diff addresses another point of https://github.com/ooni/probe/issues/1956:
> - [ ] observe that we're still using a bunch of private interfaces for common interfaces such as the `Dialer`, so we can get rid of these private interfaces and always use the ones in `model`, which allows us to remove a bunch of legacy wrappers
Additional cleanups may still be possible. The more I cleanup, the more I see
there's extra legacy code we can dispose of (which seems good?).
This diff implements the first two cleanups defined at https://github.com/ooni/probe/issues/1956:
> - [ ] observe that `netxlite` and `netx` differ in error wrapping only in the way in which we set `ErrWrapper.Operation`. Observe that the code using `netxlite` does not care about such a field. Therefore, we can modify `netxlite` to set such a field using the code of `netx` and we can remove `netx` specific code for errors (which currently lives inside of the `./internal/engine/legacy/errorsx` package
>
> - [ ] after we've done the previous cleanup, we can make all the classifiers code private, since there's no code outside `netxlite` that needs them
A subsequent diff will address the remaining cleanup.
While there, notice that there are failing, unrelated obfs4 tests, so disable them in short mode. (I am confident these tests are unrelated because they fail for me when running test locally from the `master` branch.)
Since https://github.com/ooni/probe-cli/pull/527, if an experiment
returns an error, the corresponding measurement is not submitted since
the semantics of returning an error is that something fundamental
went wrong (e.g., we could not parse the input URL).
This diff ensures that all experiments only return and error when
something fundamental was wrong and return nil otherwise.
Reference issue: https://github.com/ooni/probe/issues/1808.
This commit completely removes the original netx implementation,
which was only used by `tor`, since this has changed in
https://github.com/ooni/probe-cli/pull/652.
The original netx implementation was my first attempt at performing
network measurements using Go. It started its life inside of the
https://github.com/ooni/netx repository. It was later merged into
the https://github.com/ooni/probe-engine repository. It finally
ended up into this repository when we merged probe-engine with it.
The main issue with the original implementation is that it was
a bit too complex and used channels where they were probably not
necessary. Because of that, later I introduced a second netx
implementation, which currently lives in ./internal/engine/netx.
The current netx implementation, the third one, lives in the
./internal/netxlite package. We are currently working to replace
the second implementation with the third one, but this is happening
at a slow pace. Also, the second implementation does not have big
maintenance concerns but it's just a bit too bureaucratic to use
since it involves creating lots of `Config` structures.
The reference issue is probably https://github.com/ooni/probe/issues/1688,
since this diff has been enabled by rewriting Tor to use `measurex`
(a library living on top of `netxlite`).
This diff rewrites the tor experiment to use measurex "easy" API.
To this end, we need to introduce an "easy" measurex API, which basically
performs easy measurements returning two pieces of data:
1. the resulting measurement, which is already using the OONI
archival data format and is always non-nil
2. a failure (i.e., the pointer to an error string), which
is nil on success and points to a string on failure
With this change, we should now be able to completely dispose of
the original netx API, which was only used by tor.
Reference issue: https://github.com/ooni/probe/issues/1688.
1. we want optionally to log the body (we don't want to log the body
when we're fetching psiphon secrets or tor targets)
2. we want body logging to _also_ happen on error since this is quite
useful to debug possible errors when accessing the API
This diff adds the above functionality, which were previously
described in https://github.com/ooni/probe/issues/1951.
This diff also adds comprehensive testing.
* refactor(httpx): use mocks to implement tests
While there, make sure no test depends on external services by
replacing such tests with httptest.
See https://github.com/ooni/probe/issues/1951.
* fix(httpx): ensure we honour the context
This diff extracts the fakefiller inside of internal/ooapi (a
currently unused package) into its own package.
The fakefiller knows how to fill many fields that are typically
shared as data structures across processes.
It is not perfect in that it cannot fill logger or http client
fields, but still helps with better filling and testing.
So, here we're using the fakefiller to improve testing of httpx
and, nicely enough, we've already catched a bug in the way in
which APIClientTemplate.Build misses to forward Authorization from
the original template. Yay!
Work part of https://github.com/ooni/probe/issues/1951
As mentioned in https://github.com/ooni/probe/issues/1951, one of
the main issues I did see with httpx.APIClient is that in some cases
it's used in a very fragile way by probeservices.Client.
This happens in psiphon.go and tor.go, where we create a copy of
the APIClient and then modify it's Authorization field.
If we ever refactor probeservices.Client to take a pointer to
httpx.Client, we are now mutating the httpx.Client.
Of course, we don't want that to happen.
This diff attempts to address such a problem as follows:
1. we create a new APIClientTemplate type that holds the same
fields of an APIClient and allows to build an APIClient
2. we modify every user of APIClient to use APIClientTemplate
3. when we need an APIClient, we build it from the corresponding
template and, when we need to use a specific Authorization, we
use a build factory that sets APIClient.Authorization
4. we hide APIClient by renaming it apiClient and by defining
an interface called APIClient that allows to use it
So, now the codebase always uses the opaque APIClient interface to
issue API calls and always uses the APIClientTemplate to build an
opaque APIClient.
Boom! We have separated construction from usage and we are not
mutating in weird ways the APIClient anymore.
This PR starts to implement the refactoring described at https://github.com/ooni/probe/issues/1951. I originally wrote more patches than the ones in this PR, but overall they were not readable. Since I want to squash and merge, here's a reasonable subset of the original patches that will still be readable and understandable in the future.
This diff lightly refactors the code in measurex to allow a user
to configure all possible timeouts and the max-snapshot-size.
There is currently a little bit of tension between setting timeouts
inside of measurex and the watchdog timeouts inside of netxlite.
This tension has been documented.
Let us repeat the issue also in this commit message. If you are
using a masurex.Measurer configured with very large timeouts and
the underlying netxlite implementation uses shorter whatchdog
timeouts, then you are going to see shorter than expected timeouts.
Ideally, we would like to have just a single timeout but there is
no way to ask the context "hey, can you tell me if you already have
a configured timeout?".
It may be that the right solution is to modify netxlite to have
some sort of root/library object with this configuration.
If that's the case, then a Measurer could be refactored as follows:
- create the underlying netxlite "library"
- initialize the timeouts desired by the Measurer
- create a Dialer, of whatever is needed
- use it
Now this is not possible because netxlite timeouts are internal
static settings rather than attributes of a structure.
Anyway, for now I'm happy with this just being documented.
(I suspect this issue will need to be addresses when we'll write
unit tests for measurex; at that time a proper solution should
come out naturally due to the unit tests constraints.)
I'm working on this refactoring, BTW, to facilitate rewriting `tor`
using measurex (see https://github.com/ooni/probe/issues/1688).
## Checklist
- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/1885
- [x] related ooni/spec pull request: N/A
Location of the issue tracker: https://github.com/ooni/probe
## Description
This PR contains a set of changes to move important interfaces and data types into the `./internal/model` package.
The criteria for including an interface or data type in here is roughly that the type should be important and used by several packages. We are especially interested to move more interfaces here to increase modularity.
An additional side effect is that, by reading this package, one should be able to understand more quickly how different parts of the codebase interact with each other.
This is what I want to move in `internal/model`:
- [x] most important interfaces from `internal/netxlite`
- [x] everything that was previously part of `internal/engine/model`
- [x] mocks from `internal/netxlite/mocks` should also be moved in here as a subpackage
This branch adds support for running:
1. `go-libtor` on mobile.
2. the tor provided by the desktop app via the `OONI_TOR_BINARY` environment variable.
See https://github.com/ooni/ooni.org/issues/761.
Co-authored-by: Simone Basso <bassosimone@gmail.com>
This diff is part of https://github.com/ooni/probe/issues/1814 and
teaches `ooniprobe` to run dnscheck and stunreachability by using the
default static input feature of the `InputLoader`.
I've manually tested that we can still run `websites` like
we did before (including category filtering).
I've also manually tested that now we can run `experimental` and
get parseable results for dnscheck and stunreachability.
With this diff in, we have fixed the original problem highlighted in
the https://github.com/ooni/probe/issues/1814 issue.
Yet, because of the way in which I solved the problem, there is
more work to do. My changes have broken stunreachability for
mobile and now it's time I apply fixes to make it work again.
This diff was extracted from https://github.com/ooni/probe-cli/pull/539,
which at this point only basically contains the remaining fixes to
ensure we can run stunreachability on mobile.
Co-authored-by: Arturo Filastò <arturo@filasto.net>
Co-authored-by: Arturo Filastò <arturo@filasto.net>
This commit introduces a new `InputLoader` policy by which, if no
input is provided, we use a static default input list.
We also modify the code to use this policy for dnscheck and
stunreachability, with proper input.
We also modify `miniooni` to pass the new `ExperimentName` field to
the `InputLoader` to indicate which default input list to use.
This diff is part of a set of diffs aiming at fixing
https://github.com/ooni/probe/issues/1814 and has been
extracted from https://github.com/ooni/probe-cli/pull/539.
What remains to be done, after this diff has landed is to ensure
things also work for ooniprobe and oonimkall.
We want stunreachability to use the same STUN servers used by
snowflake, so let's start by making a common package holding the
servers. Let's also use this new package in Snowflake.
We're currently not using this package in stunreachability, but
I am going to apply this as a subsequent diff.
Reference issue: https://github.com/ooni/probe/issues/1814. This
issue is a bit complex to address in a single PR, so we are going
to proceed incremntally.
This diff was extracted from https://github.com/ooni/probe-cli/pull/539.
Here we're refactoring stunreachability to not provide internally a
default input and to take in input an URL rather than a string.
The related ooni/spec change is https://github.com/ooni/spec/pull/227.
This diff has been extracted from https://github.com/ooni/probe-cli/pull/539.
Because the original diff was large, I'm splitting it in a set of
more easily manageable diffs.
The reference issue is https://github.com/ooni/probe/issues/1814, which
is complex enough to require us to proceed incrementally.
This diff WILL need to be backported to release/3.11.
* [forwardport] fix(oonimkall): make logger used by tasks unit testable (#623)
This diff forward ports e4b04642c51e7461728b25941624e1b97ef0ec83.
Reference issue: https://github.com/ooni/probe/issues/1903
* [forwardport] feat(oonimkall): improve taskEmitter testability (#624)
This diff forward ports 3e0f01a389c1f4cdd7878ec151aff91870a0bdff.
1. rename eventemitter{,_test}.go => taskemitter{,_test}.go because
the new name is more proper after we merged the internal/task package
inside of the oonimkall package;
2. rename runner.go's `run` function to `runTask`;
3. modify `runTask` to use the new `taskEmitterUsingChan` abstraction
on which we will spend more works in a later point of this list;
4. introduce `runTaskWithEmitter` factory that is called by `runTask`
and allows us to more easily write unit tests;
5. acknowledge that `runner` was not using its `out` field;
6. use the new `taskEmitterWrapper` in `newRunner`;
7. acknowledge that `runnerCallbacks` could use a generic
`taskEmitter` as field type rather than a specific type;
8. rewrite tests to use `runTaskWithEmitter` which leads to
simpler code that does not require a goroutine;
9. acknowledge that the code has been ignoring the `DisabledEvents`
settings for quite some time, so stop supporting it;
10. refactor the `taskEmitter` implementation to be like:
1. we still have the `taskEmitter` interface;
2. `taskEmitterUsingChan` wraps the channel and allows for
emitting events using the channel;
3. `taskEmitterUsingChan` owns an `eof` channel that is
closed by `Close` (which is idempotent) and signals we
should be stop emitting;
4. make sure `runTask` creates a `taskEmitterUsingChan`
and calls its `Close` method when done;
5. completely remove the code for disabling events
since the code was actually ignoring the stting;
6. add a `taskEmitterWrapper` that adds common functions
for emitting events to _any_ `taskWrapper`;
7. write unit tests for `taskEmitterUsingChan` and
for `taskEmitterWrapper`;
11. acknowledge that the abstraction we need for testing is
actually a thread-safe thing that collects events into a
vector containing events and refactor all tests accordingly.
See https://github.com/ooni/probe/issues/1903
* [forwardport] refactor(oonimkall): make the runner unit-testable (#625)
This diff forward ports 9423947faf6980d92d2fe67efe3829e8fef76586.
See https://github.com/ooni/probe/issues/1903
* [forwardport] feat(oonimkall): write unit tests for the runner component (#626)
This diff forward ports 35dd0e3788b8fa99c541452bbb5e0ae4871239e1.
Forward porting note: compared to 35dd0e3788b8fa99c541452bbb5e0ae4871239e1,
the diff I'm committing here is slightly different. In `master` we do not
have the case where a measurement fails and a measurement is returned, thus
I needed to adapt the test to become like this:
```diff
diff --git a/pkg/oonimkall/runner_internal_test.go b/pkg/oonimkall/runner_internal_test.go
index 334b574..84c7436 100644
--- a/pkg/oonimkall/runner_internal_test.go
+++ b/pkg/oonimkall/runner_internal_test.go
@@ -568,15 +568,6 @@ func TestTaskRunnerRun(t *testing.T) {
}, {
Key: failureMeasurement,
Count: 1,
- }, {
- Key: measurement,
- Count: 1,
- }, {
- Key: statusMeasurementSubmission,
- Count: 1,
- }, {
- Key: statusMeasurementDone,
- Count: 1,
}, {
Key: statusEnd,
Count: 1,
```
I still need to write more assertions for each emitted event
but the code we've here is already a great starting point.
See https://github.com/ooni/probe/issues/1903
* [forwardport] refactor(oonimkall): merge files, use proper names, zap unneeded integration tests (#627)
This diff forward ports f894427d24edc9a03fc78306d0093e7b51c46c25.
Forward porting note: this diff is slightly different from the original
mentioned above because it carries forward changes mentioned in the
previous diff caused by a different way of handling a failed measurement
in the master branch compared to the release/3.11 branch.
Move everything that looked like "task's model" inside of the
taskmodel.go file, for consistency.
Make sure it's clear some variables are event types.
Rename the concrete `runner` as `runnerForTask`.
Also, remove now-unnecessary (and flaky!) integration tests
for the `runnerForTask` type.
While there, notice there were wrong URLs that were generated
during the probe-engine => probe-cli move and fix them.
See https://github.com/ooni/probe/issues/1903
* [forwardport] refactor(oonimkall): we can simplify StartTask tests (#628)
This diff forward ports dcf2986c2032d8185d58d24130a7f2c2d61ef2fb.
* refactor(oonimkall): we can simplify StartTask tests
We have enough checks for runnerForTask. So we do not need to
duplicate them when checking for StartTask.
While there, refactor how we start tasks to remove the need for
extra runner functions.
This is the objective I wanted to achieve for oonimkall:
1. less duplicate tests, and
2. more unit tests (which are less flaky)
At this point, we're basically done (pending forwardporting to
master) with https://github.com/ooni/probe/issues/1903.
* fix(oonimkall): TestStartTaskGood shouldn't cancel the test
This creates a race condition where the test may fail if we cannot
complete the whole "Example" test in less than one second.
This should explain the build failures I've seen so far and why
I didn't see those failures when running locally.
1. add eof channel to event emitter and use this channel as signal
that we shouldn't be sending anymore instead of using a pattern where
we use a timer to decide sending has timed out (because we're using
a buffered channel, it is still possible for some evetns to end up
in the channel if there is space, but this is not a concern, because
the events will be deleted when the channel itself is gone);
2. refactor all tests where we assumed the output channel was closed
to actually use a parallel "eof" channel and use it as signal we
should not be sending anymore (not strictly required but still the
right thing to do in terms of using consistent patterns);
3. modify how we construct a runner so that it passes to the
event emitter an "eof" channel and closes this channel when the
main goroutine running the task is terminating;
4. modify the task to signal events such as "task goroutine started"
and "task goroutine stopped" using channels, which helps to write
much more correct tests;
5. take advantage of the previous change to improve the test that
ensures we're not blocking for a small number of events and also
improve the name of such a test to reflect what it's testing.
The related issue in term of fixing the channel usage pattern is
https://github.com/ooni/probe/issues/1438.
Regarding improving testability, instead, the correct reference
issue is https://github.com/ooni/probe/issues/1903.
There are possibly more changes to apply here to improve this package
and its testability, but let's land this diff first and then see
how much time is left for further improvements.
I've run unit and integration tests with `-race` locally.
This diff will need to be backported to `release/3.11`.
This diff forward ports 90bf0629b957c912a5a6f3bb6c98ad3abb5a2ff6 to `master`.
If we close the channel to signal the end of a task we may panic
when some background goroutine tries to post on the channel.
This bug is rare but may happen.
See for example https://github.com/ooni/probe/issues/1438.
How can we improve?
First, let us add a timeout when sending to the channel. Given that
the channel is buffered and we have a generous timeout (1/4 of a
second), it's unlikely we will really block. But, in the event in
which a late message appears, we'll eventually _unblock_ when
sending with a timeout. So, now we don't have to worry anymore
about leaking forever a goroutine.
Then, let us change the protocol with which we signal that a task
is done. We used to close the channel. Now, instead we just
synchronously post a nil on the channel when done.
In turn, we interpret this nil to mean that the task is done when
we receive messages.
The _main_ different with respect to before is that now we are
asking the consumer of our API to drain the channel. Because
before we had a blocking channel, it seems to me we were already
requiring the consumer of the API to do that. Which means, I think
in practical terms it did not change much.
Finally, acknowledge that we don't need a specific state variable
to tell us we're done and simplify a little bit the API by
just making isRunning private and using the "we're done" signal
to determine whether we've stopped running the task.
All these changes should be enough to close https://github.com/ooni/probe/issues/1438.
