* fix(model/archival.go): more optional keys
Basically, `t0` and `transaction_id` should be optional. Version 0.4.x
of web_connectivity should not include them, version 0.5.x should.
There is a technical reason why v0.4.x should not include them. The code
it is based on, tracex, does not record these two fields.
Whereas, v0.5.x, uses measurexlite, which records these two fields.
Part of https://github.com/ooni/probe/issues/2238
* fix(webconnectivity@v0.5): add more fields
This diff adds the following fields to webconnectivity@v0.5:
1. agent, always set to "redirect" (legacy field);
2. client_resolver, properly initialized w/ the resolver's IPv4 address;
3. retries, legacy field always set to null;
4. socksproxy, legacy field always set to null.
Part of https://github.com/ooni/probe/issues/2238
* fix(webconnectivity@v0.5): register extensions
The general idea behind this field is that we would be able
in the future to tweak the data model for some fields, by declaring
we're using a later version, so it seems useful to add it.
See https://github.com/ooni/probe/issues/2238
* fix(measurexlite): use tcp or quic for tls handshake network
This diff fixes a bug where measurexlite was using "tls" as the
protocol for the TLS handshake when using TCP.
While this choice _could_ make sense, the rest of the code we have
written so far uses "tcp" instead.
Using "tcp" makes more sense because it allows you to search for
the same endpoint across different events by checking for the same
network and for the same endpoint rather than special casing TLS
handshakes for using "tls" when the endpoint is "tcp".
See https://github.com/ooni/probe/issues/2238
* chore: run alltests.yml for "alltestsbuild" branches
Part of https://github.com/ooni/probe/issues/2238
In a pure step-by-step model, we don't need to trace HTTP round trips like we did before. We _may_ want in the future to also have some form of HTTP tracing (see https://github.com/ooni/probe-cli/pull/868 for a prototype) but doing that is currently not in scope for moving forward the step-by-step design. For this reason, I only added a public convenience function for formatting an OONI spec compatible request. I also added new fields, which should be documented inside the ooni/spec repository (see https://github.com/ooni/probe/issues/2238).
Required by https://github.com/ooni/probe/issues/2237
The T0 field is the moment when we started collecting data, while T
is the moment when we finished collecting data.
The TransactionID field will be repurposed for step-by-step measurements
to indicate related observations collected as part of the same flow
(e.g., TCP+TLS+HTTP).
Note that, for now, this change will only affect measurexlite and we're
not planning on changing other libraries for measuring.
Part of https://github.com/ooni/probe/issues/2137
* tls_handshakes: add IP addresses
* tls_handshakes: extract ip from tcp-connect
* tls_handshake: switched to trace event
* saver.go: get remoteAddr before handshake
Not sure whether this is strictly necessary, but I'd rather take the
remoteAddr before calling Handshake, just in case a future version
of the handshake closes the `conn`. In such a case, `conn.RemoteAddr`
would return `nil` and we would crash here.
This occurred to me while reading once again the diff before merging.
Co-authored-by: decfox <decfox>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
We recently started moving core data structures inside of the
internal/model package as detailed in https://github.com/ooni/probe/issues/1885.
The chief reason to do that is to have a set of fundamental
shared data types to help us rationalize the codebase.
This specific diff moves internal/netx/archival's core data types
inside the internal/model package. While there, it also refactors the
existing tests to improve their quality. Additionally, we also added
an extra test to ensure `ArchivalHTTPBody` is an alias for
`ArchivalMaybeBinaryData`, which is required to ensure the
custom JSON serialization process works for it.
We're doing that because both internal/netx/archival and
internal/measurex define their own archival data structures.
We developed measurex using its own structures because it
allowed to iterate more quickly. Now that we have sketched
out measurex, the time has come to consolidate.
My overall aim is to spend a few more hours this week on
engineering measurex. This work is preliminary work before
we finish up both measurex and websteps.
We described this cleanup in https://github.com/ooni/probe/issues/1957.
