This diff lightly refactors the code in measurex to allow a user
to configure all possible timeouts and the max-snapshot-size.
There is currently a little bit of tension between setting timeouts
inside of measurex and the watchdog timeouts inside of netxlite.
This tension has been documented.
Let us repeat the issue also in this commit message. If you are
using a masurex.Measurer configured with very large timeouts and
the underlying netxlite implementation uses shorter whatchdog
timeouts, then you are going to see shorter than expected timeouts.
Ideally, we would like to have just a single timeout but there is
no way to ask the context "hey, can you tell me if you already have
a configured timeout?".
It may be that the right solution is to modify netxlite to have
some sort of root/library object with this configuration.
If that's the case, then a Measurer could be refactored as follows:
- create the underlying netxlite "library"
- initialize the timeouts desired by the Measurer
- create a Dialer, of whatever is needed
- use it
Now this is not possible because netxlite timeouts are internal
static settings rather than attributes of a structure.
Anyway, for now I'm happy with this just being documented.
(I suspect this issue will need to be addresses when we'll write
unit tests for measurex; at that time a proper solution should
come out naturally due to the unit tests constraints.)
I'm working on this refactoring, BTW, to facilitate rewriting `tor`
using measurex (see https://github.com/ooni/probe/issues/1688).
## Checklist
- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/1885
- [x] related ooni/spec pull request: N/A
Location of the issue tracker: https://github.com/ooni/probe
## Description
This PR contains a set of changes to move important interfaces and data types into the `./internal/model` package.
The criteria for including an interface or data type in here is roughly that the type should be important and used by several packages. We are especially interested to move more interfaces here to increase modularity.
An additional side effect is that, by reading this package, one should be able to understand more quickly how different parts of the codebase interact with each other.
This is what I want to move in `internal/model`:
- [x] most important interfaces from `internal/netxlite`
- [x] everything that was previously part of `internal/engine/model`
- [x] mocks from `internal/netxlite/mocks` should also be moved in here as a subpackage
This change should simplify the pipeline's job.
Reference issue: https://github.com/ooni/probe/issues/1817.
I previously dismissed this possibility, but now it seems clear it
is simpler to have a very tabular data format internally and to
convert such a format to OONI's data format when serializing.
The OONI data format is what the pipeline expects, but processing
is easier with a more linear/tabular format.
This is the most immediate fix to the issue described by
https://github.com/ooni/probe/issues/1792.
So, the logic was actually miss the increment, which
would have been noticed with proper unit testing.
Anyway, I am not sure why the loop ensues in the first
time. By looking at the headers, it seems we're passing
the headers correctly.
So, even though this fix interrupts the loop, it still
remains the question of whether the loop is legit or
whether we're missing extra logic to properly redirect.
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
