The T0 field is the moment when we started collecting data, while T
is the moment when we finished collecting data.
The TransactionID field will be repurposed for step-by-step measurements
to indicate related observations collected as part of the same flow
(e.g., TCP+TLS+HTTP).
Note that, for now, this change will only affect measurexlite and we're
not planning on changing other libraries for measuring.
Part of https://github.com/ooni/probe/issues/2137
* feat: save CNAME into archival data format
When a DNSResponse contains a non-empty CNAME, we include it
into the related list of answers.
Closes https://github.com/ooni/probe/issues/2227
* doc: add design note
While there, make code more compact and robust to a case where
we're going to extract additional answers.
* doc: document the expected growth of extraction function
Based on feedback by @DecFox
* feat(dnsovergetaddrinfo): collect the CNAME
This diff modifies how dnsovergetaddrinfo.go works such that the
returned DNSResponse includes the CNAME.
Closes https://github.com/ooni/probe/issues/2226.
While there, recognize that we can remove getaddrinfoLookupHost and
always call getaddrinfoLookupANY everywhere. (This simplification is
why we did https://github.com/ooni/probe-cli/pull/874.)
* fix: extra debugging because of failing CI
Everything is OK locally (on macOS). However, maybe things are a bit
different on GNU/Linux perhaps?
Here's the error:
```
--- FAIL: TestPass (0.11s)
resolver_test.go:113: unexpected rcode
FAIL
coverage: 95.7% of statements
FAIL github.com/ooni/probe-cli/v3/internal/cmd/jafar/resolver 0.242s
```
I'm a bit confused because jafar's resolver is _unrelated_. But actually this
error never occurred again after a committed the debugging diff.
* feat(netxlite): support extracting the CNAME
Closes https://github.com/ooni/probe/issues/2225
* fix(netxlite): attempt to increase coverage and improve tests
1. dnsovergetaddrinfo: specify the behavior of a DNSResponse returned
by this file to make it line with normal responses and write unit tests
to make sure we adhere to expectations;
2. dnsoverudp: make sure we wait to deferred responses also w/o a
custom context and post on a private channel and test that;
3. utls: recognize that we can actually write a test for NetConn and
what needs to change when we'll use go1.19 by default will just be
a cast that at that point can be removed.
* cleanup: remove UnderlyingNetworkLibrary and TProxy
While there, replace mixture of mocking and real connections inside
quicping with pure mocking of network connections.
Closes https://github.com/ooni/probe/issues/2224
* cleanup: we don't need a SimpleResolver now
This type was only used by UnderlyingNetworkLibrary and all the
rest of the code uses Resolver. So, let's avoid complexity by zapping
the SimpleResolver type and merging it inside Resolver.
* upgrade to our go.mod enabled of psiphon-tunnel-core such that
we're now using v2.0.24 of the tunnel-core;
* upgrade to the latest lucas-clemente/quic-go release;
* upgrade to the latest ooni/oohttp release (which is based on go1.19
but the diff seems good enough to continue using go1.18.x as well);
* upgrade to the latest ooni/oocrypto release (for which we can make the
same remarks regarding using go1.18.x);
* deal with changes in lucas-clemente/quic-go API as well as changes
in what a go1.19 *tls.Conn compatible type should look like.
Unfortunately, we cannot switch to go1.19 because psiphon forks quic-go
and their fork's still not building using such a version of go.
Part of ooni/probe#2211.
Skip options that begin with the `Safe` prefix from appearing in the
serialization of a Measurement that will be submitted to the OONI
backend.
Fixes https://github.com/ooni/probe/issues/2214
While there, document what we need to document without mentioning too
many details about release builds, for which we have the Makefile.
See https://github.com/ooni/probe/issues/2218.
I made a mistake while adapting code from an experimental branch thus
breaking these two experiments because of interface conversion.
This diff fixes it.
While there, remove the panic trap for miniooni. Because miniooni is
an experimental tool, we want to see the full panic text, which definitely
leads to a more pleasant and effective debugging experience.
See https://github.com/ooni/probe/issues/2216 for context on why we
were trying to change how we register experiments.
The broken commit is 6a0ae5c70b.
* feat: add support for system resolver in measurexlite
* more tests for coverage
* Apply suggestions from code review
Co-authored-by: decfox <decfox@github.com>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
Until OONI Run v2 has support for repeating the measurement with a schedule, introduce a command line flag requested by users to repeat a measurement every given number of seconds.
Part of https://github.com/ooni/probe/issues/2184
This diff adds support for running OONIRun v1 links.
Run with `miniooni` using:
```
./miniooni -i LINK oonirun
```
Part of https://github.com/ooni/probe/issues/2184
This diff refactors the ./internal/cmd/miniooni pkg and moves the code
for running experiments inside of the ./internal/oonirun pkg.
It's the first concrete step towards https://github.com/ooni/probe/issues/2184.
The integration test that was broken was:
```
--- FAIL: TestCreateInvalidExperiment (0.35s)
experiment_integration_test.go:192: expected a nil builder here
```
While there improve the documentation of the ExperimentSession
and see there's a method that we are not using.
This diff is a cleanup that I come up with while working
on https://github.com/ooni/probe/issues/2184.
This option has been disabled for a long time and we said in the
codebase we were going to remove it after 2021-11-01.
So, it feels okay to remove it.
This diff is a cleanup in preparation for https://github.com/ooni/probe/issues/2184.
This diff modifies the engine package to make Experiment and
ExperimentBuilder interfaces rather than structs.
The previosuly existing structs are now named experiment{,Builder}.
This diff helps https://github.com/ooni/probe/issues/2184
because it allows us to write unit tests more easily.
There should be no functional change.
While there, I removed a bunch of deprecated functions, which were
unnecessarily complicate the implementation and could be easily
replaced by passing them a context.Context or context.Background().
This diff refactors how we set options for experiments to accept
in input an any value or a map[string]any, depending on which method
we choose to actually set options.
There should be no functional change, except that now we're not
guessing the type and then attempting to set the value of the selected
field: now, instead, we match the provided type and the field's type
as part of the same function (i.e., SetOptionAny).
This diff is functional to https://github.com/ooni/probe/issues/2184,
because it will allow us to load options from a map[string]any,
which will be part of the OONI Run v2 JSON descriptor.
If we didn't apply this change, we would only have been to set options
from a map[string]string, which is good enough as a solution for the
CLI but is definitely clumsy when you have to write stuff like:
```JSON
{
"options": {
"HTTP3Enabled": "true"
}
}
```
when you could instead more naturally write:
```JSON
{
"options": {
"HTTP3Enabled": true
}
}
```
This diff makes the implementation of the engine package more
abstract by changing HTTPClient() to return a model.HTTPClient
as opposed to returning an *http.Client.
Part of https://github.com/ooni/probe/issues/2184
In https://github.com/ooni/probe-cli/pull/832's initial diff, I
mentioned it would be cool to flatten oohelperd's hier.
I'm doing this now, and just for the master branch.
This diff is mostly a mechanical refactoring with very light
and apparently rather safe manual changes.
This diff modifies the implementation of oohelperd in the master branch
to always use throw-away HTTPClient, Dialer, and Resolver.
The rationale of this change is to ensure we're not hitting limits of the
HTTPClient regarding the max number of connections per host.
This issue is described at https://github.com/ooni/probe/issues/2182.
While there, it feels more correct to use throw-away Dialer and Resolver.
We have a different patch for the release/3.15 branch because of
netx-related refactorings: https://github.com/ooni/probe-cli/pull/832.
## Checklist
- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/2158
- [x] if you changed anything related how experiments work and you need to reflect these changes in the ooni/spec repository, please link to the related ooni/spec pull request: https://github.com/ooni/spec/pull/250
## Description
This diff refactors the codebase to reimplement tlsping and tcpping
to use the step-by-step measurements style.
See docs/design/dd-003-step-by-step.md for more information on the
step-by-step measurement style.
I am not 100% sure I was able to fix all the cases in which we
need higher permissions than the strict default.
At least, I tried.
It may be reasonable to make an interim release to check whether I
successfully fixed all the cases.
Ref issue: https://github.com/ooni/probe/issues/2154
This bug is one of these bugs that definitely help one to stay
humble and focused on improving the codebase.
Of course I `<facepalmed>` when I understood the root cause.
We did not move the annotations below the `if` which is checking
whether the measurement was successful when we refactored the
codebase to support returning multiple measurements per run, which
happened in https://github.com/ooni/probe-cli/pull/527.
While I am not going to whip myself too much because of this, it's
clearly a bummer that we didn't notice this bug back then. On top
of this, it's also quite sad it took us so much time to notice that
there was this bug inside the tree.
The lesson (hopefully) learned is probably that we need to be more
careful when we refactor and we should always ask the question of
whether, not only we have tests, but whether these tests could maybe
be improved to give us even more confidence about correctness.
The reference issue is https://github.com/ooni/probe/issues/2173.
This pull request publishes the step-by-step design document that I have been discussing with @hellais and @DecFox recently. Compared to the document that was approved, this one has been edited for readability.
While there, I figured it was also be beneficial to publish the few ooni/probe-cli related design documents we produced in the past, because they probably help someone to get acquainted with the codebase.
Reference issue for this pull request: https://github.com/ooni/probe/issues/2148
