These two small packages could easily be merged into the model
package, since they're clearly model-like packages.
Part of https://github.com/ooni/probe/issues/2115
The objective is to make PR checks run much faster.
See https://github.com/ooni/probe/issues/2113 for context.
Regarding netxlite's tests:
Checking for every commit on master or on a release branch is
good enough and makes pull requests faster than one minute since
netxlite for windows is now 1m slower than coverage.
We're losing some coverage but coverage from integration tests
is not so good anyway, so I'm not super sad about this loss.
Rather than building for Android using ooni/go, we're now using
ooni/oocryto as the TLS dependency. Such a repository only forks
crypto/tls and some minor crypto packages and includes the
same set of patches that we have been using in ooni/go.
This new strategy should be better than the previous one in
terms of building for Android, because we can use the vanilla
go1.18.2 build. It also seems that it is easier to track and
merge from upstream with ooni/oocrypto than it is with ooni/go.
Should this assessment be wrong, we can revert back to the
previous scenario where we used ooni/go.
See https://github.com/ooni/probe/issues/2106 for extra context.
Previously, the DNS decoder did not check whether it was parsing
a DNS query or a DNS response, which was wrong.
As a side note, it seems I am using "reply" in the codebase instead
of "response". The latter seems correct DNS terminology.
This diff has been extracted from 9249d14f80
See https://github.com/ooni/probe/issues/2096.
This diff has been extracted and adapted from 8848c8c516
The reason to prefer composition over embedding is that we want the
build to break if we add new methods to interfaces we define. If the build
does not break, we may forget about wrapping methods we should
actually be wrapping. I noticed this issue inside netxlite when I was working
on websteps-illustrated and I added support for NS and PTR queries.
See https://github.com/ooni/probe/issues/2096
While there, perform comprehensive netxlite code review
and apply minor changes and improve the docs.
This diff has been extracted from c2f7ccab0e
See https://github.com/ooni/probe/issues/2096
While there, export DecodeReply to decode a raw reply without
interpreting the Rcode or parsing the results, which seems a
nice extra feature to have to more flexibly parse DNS replies
in other parts of the codebase.
This error occurred for example when querying kazemjalali.com
in websteps measurements run from Iran.
This error is relatively uncommon, but it still makes sense to
create a specific mapping rule for it.
Originally: 4269e82fbd
See https://github.com/ooni/probe/issues/2096
I've seen some measurements returning some IP addresses for HTTPSSvc
queries but not returning any ALPN value.
For example:
```
% d4
decoding DNS round trip 0:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57768
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;psiphon.ca. IN HTTPS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57768
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;psiphon.ca. IN HTTPS
;; ANSWER SECTION:
psiphon.ca. 121 IN A 31.13.85.53
```
Now, the response is clearly bogus. At the time of this writing that
IP address belongs to Facebook. This measurement has been collected in
China, so it's expected for the GFW to behave like this.
Yet, I don't feel like it's accurate to report this measurement as a
"no answer" response. Rather, this response is a valid one containing
a clearly invalid IP address and should be flagged as such.
Originally: 57a023bcf4
See https://github.com/ooni/probe/issues/2096
This diff fixes the short-circuit-IP-addr resolver to
correctly handle IP addrs during LookupHTTPS.
The original diff was: 2b51d144bf
See https://github.com/ooni/probe/issues/2096
While there, add unit tests for IPv6.
* quic-go upgrade: replaced Session/EarlySession with Connection/EarlyConnection
* quic-go upgrade: added context to RoundTripper.Dial
* quic-go upgrade: made corresponding changes to tutorial
* quic-go upgrade: changed sess variable instances to qconn
* quic-go upgrade: made corresponding changes to tutorial
* cleanup: remove unnecessary comments
Those comments made sense in terms of illustrating the changes
but they're going to be less useful once we merge.
* fix(go.mod): apparently we needed `go1.18.1 mod tidy`
VSCode just warned me about this. It seems fine to apply this
change as part of the pull request at hand.
* cleanup(netxlite): http3dialer can be removed
We used to use http3dialer to glue a QUIC dialer, which had a
context as its first argument, to the Dial function used by the
HTTP3 transport, which did not have a context as its first
argument.
Now that HTTP3 transport has a Dial function taking a context as
its first argument, we don't need http3dialer
anymore, since we can use the QUIC dialer directly.
Cc: @DecFox
* Revert "cleanup(netxlite): http3dialer can be removed"
This reverts commit c62244c620cee5fadcc2ca89d8228c8db0b96add
to investigate the build failure mentioned at
https://github.com/ooni/probe-cli/pull/715#issuecomment-1119450484
* chore(netx): show that test was already broken
We didn't see the breakage before because we were not using
the created transport, but the issue of using a nil dialer was
already present before, we just didn't see it.
Now we understand why removing the http3transport in
c62244c620cee5fadcc2ca89d8228c8db0b96add did cause the
breakage mentioned at
https://github.com/ooni/probe-cli/pull/715#issuecomment-1119450484
* fix(netx): convert broken integration test to working unit test
There's no point in using the network here. Add a fake dialer that
breaks and ensure we're getting the expected error.
We've now improved upon the original test because the original test was
not doing anything while now we're testing whether we get back a QUIC
dialer that _can be used_.
After this commit, I can then readd the cleanup commit
c62244c620cee5fadcc2ca89d8228c8db0b96add and it won't be
broken anymore (at least, this is what I expected to happen).
* Revert "Revert "cleanup(netxlite): http3dialer can be removed""
This reverts commit 0e254bfc6ba3bfd65365ce3d8de2c8ec51b925ff
because now we should have fixed the broken test.
Co-authored-by: decfox <decfox>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
Here's the squash of the following patches that enable support
for go1.18 and update our dependencies.
This diff WILL need to be backported to the release/3.14 branch.
* chore: use go1.17.8
See https://github.com/ooni/probe/issues/2067
* chore: upgrade to probe-assets@v0.8.0
See https://github.com/ooni/probe/issues/2067.
* chore: update dependencies and enable go1.18
As mentioned in 7a0d17ea91,
the tree won't build with `go1.18` unless we say it does.
So, not only here we need to update dependencies but also we
need to explicitly say `go1.18` in the `go.mod`.
This work is part of https://github.com/ooni/probe/issues/2067.
* chore(coverage.yml): run with go1.18
This change will give us a bare minimum confidence that we're
going to build our tree using version 1.18 of golang.
See https://github.com/ooni/probe/issues/2067.
* chore: update user agent used for measuring
See https://github.com/ooni/probe/issues/2067
* chore: run `go generate ./...`
See https://github.com/ooni/probe/issues/2067
* fix(dialer_test.go): make test work with go1.17 and go1.18
1. the original test wanted the dial to fail, so ensure we're not
passing any domain name to exercise dialing not resolving;
2. match the end of the error rather than the whole error string.
Tested locally with both go1.17 and go1.18.
See https://github.com/ooni/probe-cli/pull/708#issuecomment-1096447186
This diff includes some final changes to be ready for blessing
a cli release. These changes are:
1. run `go generate ./...` to update the bundled CA
2. update the header we use for measuring
3. ensure `mk` uses the latest version of several tools
Reference issue: https://github.com/ooni/probe/issues/1845
This diff introduces a new package called `./internal/archival`. This package collects data from `./internal/model` network interfaces (e.g., `Dialer`, `QUICDialer`, `HTTPTransport`), saves such data into an internal tabular data format suitable for on-line processing and analysis, and allows exporting data into the OONI data format.
The code for collecting and the internal tabular data formats are adapted from `measurex`. The code for formatting and exporting OONI data-format-compliant structures is adapted from `netx/archival`.
My original objective was to _also_ (1) fully replace `netx/archival` with this package and (2) adapt `measurex` to use this package rather than its own code. Both operations seem easily feasible because: (a) this code is `measurex` code without extensions that are `measurex` related, which will need to be added back as part of the process; (b) the API provided by this code allows for trivially converting from using `netx/archival` to using this code.
Yet, both changes should not be taken lightly. After implementing them, there's need to spend some time doing QA and ensuring all nettests work as intended. However, I am planning a release in the next two weeks, and this QA task is likely going to defer the release. For this reason, I have chosen to commit the work done so far into the tree and defer the second part of this refactoring for a later moment in time. (This explains why the title mentions "1/N").
On a more high-level perspective, it would also be beneficial, I guess, to explain _why_ I am doing these changes. There are two intertwined reasons. The first reason is that `netx/archival` has shortcomings deriving from its original https://github.com/ooni/netx legacy. The most relevant shortcoming is that it saves all kind of data into the same tabular structure named `Event`. This design choice is unfortunate because it does not allow one to apply data-type specific logic when processing the results. In turn, this choice results in complex processing code. Therefore, I believe that replacing the code with event-specific data structures is clearly an improvement in terms of code maintainability and would quite likely lead us to more confidently change and evolve the codebase.
The second reason why I would like to move forward these changes is to unify the codepaths used for measuring. At this point in time, we basically have two codepaths: `./internal/engine/netx` and `./internal/measurex`. They both have pros and cons and I don't think we want to rewrite whole experiments using `netx`. Rather, what we probably want is to gradually merge these two codepaths such that `netx` is a set of abstractions on top of `measurex` (which is more low-level and has a more-easily-testable design). Because saving events and generating an archival data format out of them consists of at least 50% of the complexity of both `netx` and `measurex`, it seems reasonable to unify this archival-related part of the two codebases as the first step.
At the highest level of abstraction, these changes are part of the train of changes which will eventually lead us to bless `websteps` as a first class citizen in OONI land. Because `websteps` requires different underlying primitives, I chose to develop these primitives from scratch rather than wrestling with `netx`, which used another model. The model used by `websteps` is that we perform each operation in isolation and immediately we save the results, while `netx` creates whole data structures and collects all the events happening via tracing. We believe the model used by `websteps` to be better because it does not require your code to figure out everything that happened after the measurement, which is a source of subtle bugs in the current implementation. So, when I started implementing websteps I extracted the bits of `netx` that could also be beneficial to `websteps` into a separate library, thus `netxlite` was born.
The reference issue describing merging the archival of `netx` and `measurex` is https://github.com/ooni/probe/issues/1957. As of this writing the issue still references the original plan, which I could not complete by the end of this Sprint, so I am going to adapt the text of the issue to only refer to what was done in here next. Of course, I also need follow-up issues.
* chore(netxlite): add currently failing test case
This diff introduces a test cases that will fail because of the reason
explained in https://github.com/ooni/probe/issues/1965.
* chore(netxlite/iox_test.go): add failing unit tests
These tests directly show how the Go implementation of ReadAll
and Copy has the issue of checking for io.EOF equality.
* fix(netxlite): make {ReadAll,Copy}Context robust to wrapped io.EOF
The fix is simple: we just need to check for `errors.Is(err, io.EOF)`
after either io.ReadAll or io.Copy has returned. When this condition is
true, we need to convert the error back to `nil` as it ought to be.
While there, observe that the unit tests I committed in the previous
commit are wrongly asserting that the error must be wrapped. This
assertion is not correct, because in both cases we have just ensured
that the returned error is `nil` (i.e., success).
See https://github.com/ooni/probe/issues/1965.
* cleanup: remove previous workaround for wrapped io.EOF
These workarounds were partial, meaning that they would cover some
cases in which the issue occurred but not all of them.
Handling the problem in `netxlite.{ReadAll,Copy}Context` is the
right thing to do _as long as_ we always use these functions instead
of `io.{ReadAll,Copy}`.
This is why it's now important to ensure we clearly mention that
inside of the `CONTRIBUTING.md` guide and to also ensure that we're
not using these functions in the code base.
* fix(urlgetter): repair tests who assumed to see EOF error
Now that we have established that we should normalize EOF when
reading bodies like the stdlib does and now that it's clear why
our behavior diverged from the stdlib, we also need to repair
all the tests that assumed this incorrect behavior.
* fix(all): don't use io{,util}.{Copy,ReadAll}
* feat: add checks to ensure we don't use io.{Copy,ReadAll}
* doc(netxlite): document we know how to deal w/ wrapped io.EOF
* fix(nocopyreadall.bash): add exception for i/n/iox.go
This diff addresses another point of https://github.com/ooni/probe/issues/1956:
> - [ ] observe that we're still using a bunch of private interfaces for common interfaces such as the `Dialer`, so we can get rid of these private interfaces and always use the ones in `model`, which allows us to remove a bunch of legacy wrappers
Additional cleanups may still be possible. The more I cleanup, the more I see
there's extra legacy code we can dispose of (which seems good?).
This diff implements the first two cleanups defined at https://github.com/ooni/probe/issues/1956:
> - [ ] observe that `netxlite` and `netx` differ in error wrapping only in the way in which we set `ErrWrapper.Operation`. Observe that the code using `netxlite` does not care about such a field. Therefore, we can modify `netxlite` to set such a field using the code of `netx` and we can remove `netx` specific code for errors (which currently lives inside of the `./internal/engine/legacy/errorsx` package
>
> - [ ] after we've done the previous cleanup, we can make all the classifiers code private, since there's no code outside `netxlite` that needs them
A subsequent diff will address the remaining cleanup.
While there, notice that there are failing, unrelated obfs4 tests, so disable them in short mode. (I am confident these tests are unrelated because they fail for me when running test locally from the `master` branch.)
Since https://github.com/ooni/probe-cli/pull/527, if an experiment
returns an error, the corresponding measurement is not submitted since
the semantics of returning an error is that something fundamental
went wrong (e.g., we could not parse the input URL).
This diff ensures that all experiments only return and error when
something fundamental was wrong and return nil otherwise.
Reference issue: https://github.com/ooni/probe/issues/1808.
## Checklist
- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/1885
- [x] related ooni/spec pull request: N/A
Location of the issue tracker: https://github.com/ooni/probe
## Description
This PR contains a set of changes to move important interfaces and data types into the `./internal/model` package.
The criteria for including an interface or data type in here is roughly that the type should be important and used by several packages. We are especially interested to move more interfaces here to increase modularity.
An additional side effect is that, by reading this package, one should be able to understand more quickly how different parts of the codebase interact with each other.
This is what I want to move in `internal/model`:
- [x] most important interfaces from `internal/netxlite`
- [x] everything that was previously part of `internal/engine/model`
- [x] mocks from `internal/netxlite/mocks` should also be moved in here as a subpackage
This diff forward ports f47b0c6c16e0cd417e3591358eb85b45962f307d to master.
Original commit message:
- - -
1. we now need to name the framework `.xcframework` otherwise
gomobile refuses to build a new framework for us ¯\_(ツ)_/¯
2. remove duplicate errno definition for iOS (iOS and darwin
are considered the same, therefore we don't need iOS defs)
Reference issue for this PR: https://github.com/ooni/probe/issues/1876
This diff WILL need to be forwardported to master.
This commit forward ports dedd84fa7ecb09f718f6b1a9c83999cb37b34dfa.
Original commit message:
- - -
This diff changes code the release/3.11 branch to ensure we're not using dns.google and www.google.com over HTTP3. As documented in https://github.com/ooni/probe/issues/1873, since this morning (approx) these services do not support HTTP3 anymore. (I didn't bother with checking whether this issue affects _other_ Google services; I just limited my analysis to the services that we were using as part of testing.)
This patch WILL require forward porting to the master branch.
This commit forward ports 74947dbbd12266c12a38fad51a70fc78a21720fd from
the `release/3.11` branch to `master`. Here's the original commit message:
- - -
Android is also Linux. The Android build fails because both
errno_linux.go and errno_android.go are compiled.
There's no difference between the files except into a comment
that mentions "linux" or "android".
Therefore, it's safe to remove the android-specific file
and just keep and use the linux-specific one.
Part of https://github.com/ooni/probe/issues/1863, where we're
forward porting ooni/go patches to go1.17.
I'm still trying to figure out whether I can build oonimkall
using the forward ported patches and this error prevents me
from building, because the build fails.
"やれやれだぜ"
Note that this patch WILL need to be forward ported to master.
This bug was previosuly reported to me by @hellais.
Because I did run `go generate ./internal/netxlite/...` we also
get for free updated certificates, which is OK.
1. introduce implementations of HTTPTransport and HTTPClient
that apply an error wrapping policy using the constructor
for a generic top-level error wrapper
2. make sure we use the implementations in point 1 when we
are constructing HTTPTransport and HTTPClient
3. make sure we apply error wrapping using the constructor for
a generic top-level error wrapper when reading bodies
4. acknowledge that error wrapping would be broken if we do
not return the same classification _and_ operation when we wrap
an already wrapped error, so fix the to code to do that
5. acknowledge that the classifiers already deal with preserving
the error string and explain why this is a quirk and why we
cannot remove it right now and what needs to happen to safely
remove this quirk from the codebase
Closes https://github.com/ooni/probe/issues/1860
This reverts commit 851b9913fa because
it seems it's not enough to allow us to see certificate errors with
quic, plus it's complex code. So, we'd rather develop a better approach,
and perhaps a simpler one, that works with QUIC as well.
This is the policy we need to provoke certificate errors. We'll divert
from, say, `8.8.8.8:443/udp` to, say, `1.1.1.1:443/udp`.
We'll do something similar for `443/tcp`.
This will cause certificate validation errors.
With this change, we have now implemented the simple design described
by https://github.com/ooni/probe/issues/1803#issuecomment-957323297.
When we're testing multiple endpoints, it's quite important to control
the order with which they are returned to the code.
This feature is especially relevant to Web Connectivity, which will
check the endpoints to connect to in order.
Therefore, we need to force deterministic results to ensure that we can
have deterministic tests when doing Web Connectivity QA.
This diff gives us the guarantee that we can have determinism.
Part of https://github.com/ooni/probe/issues/1803#issuecomment-957323297.
1. in normal code is better to always do if err != nil so that
the ifs only contain error code (this is ~coding policy)
2. in tests we want to ensure we narrow down the error to the
real error that happened, to have greater confidence
Written while working on https://github.com/ooni/probe/issues/1803#issuecomment-957323297
This change will simplify follow-up work done as part of
https://github.com/ooni/probe/issues/1803#issuecomment-957323297 to
implement a comprehensive self-censoring solution.
While there, rename the "proxy" action to "pass" because what we
are effectively doing is passing traffic to the network (that's a
minor change but it seems a better analogy).
* feat: run ~always netxlite integration tests
This diff ensures that we check on windows, linux, macos that our
fundamental networking library (netxlite) works.
We combine unit and integration tests.
This work is part of https://github.com/ooni/probe/issues/1733, where
I want to have more strong guarantees about the foundations.
* fix(filtering/tls_test.go): make portable on Windows
The trick here is to use the wrapped error so to normalize the
different errors messages we see on Windows.
* fix(netxlite/quic_test.go): make portable on windows
Rather than using the zero port, use the `x` port which fails
when the stdlib is parsing the address.
The zero port seems to work on Windows while it does not on Unix.
* fix(serialresolver_test.go): make error more timeout than before
This seems enough to convince Go on Windows about this error
being really a timeout timeouty timeouted thingie.
On Windows, GetAddrInfoW is a syscall and the Go resolver does
not attempt to map errors beyond WSA_HOST_NOT_FOUND, which becomes
"no such host", which we map to "dns_nxdomain_error".
See https://github.com/golang/go/blob/go1.17.1/src/net/lookup_windows.go#L16.
To map more GetAddrInfoW errors, thus, we need to enhance our
error classifier to have system specific errors.
Then, we need to filter for the WSA errors that are most likely
to pop up and map them to OONI failures. Those are three:
- WSANO_DATA which we have from our own UDP resolver as well
and which we can map to `dns_no_answer`
- WSANO_RECOVERY which we don't have but existed for MK so
we will use `dns_non_recoverable_failure`, which was an MK error
- WSATRY_AGAIN which likewise we map to the error that MK
used to emit, so `dns_temporary_failure`
This diff should address https://github.com/ooni/probe/issues/1467.
I need to run test on Windows and I just discovered that:
1. the `errno_unix.go` filename does not mean anything because
`unix` is not a valid platform, so we need a filename for
each platform that we care about;
2. on Windows we need to use WSA prefixed names;
3. `i/e/session_psiphon.go` was not building because of the
migration from `netxlite/iox` to `netxlite`.
This diff attempts to fix all three issues.
The reference issue is https://github.com/ooni/probe/issues/1733,
because I was working on such an issue.
When preparing a tutorial for netxlite, I figured it is easier
to tell people "hey, this is the package you should use for all
low-level networking stuff" rather than introducing people to
a set of packages working together where some piece of functionality
is here and some other piece is there.
Part of https://github.com/ooni/probe/issues/1591
I have recently seen a data race related our way of
mutating the outgoing request to set the host header.
Unfortunately, I've lost track of the race output,
because I rebooted my Linux box before saving it.
Though, after inspecting why and and where we're mutating
outgoing requets, I've found that:
1. we add the host header when logging to have it logged,
which is not a big deal since we already emit the URL
rather than just the URL path when logging a request, and
so we can safely zap this piece of code;
2. as a result, in measurements we may omit the host header
but again this is pretty much obvious from the URL itself
and so it should not be very important (nonetheless,
avoid surprises and keep the existing behavior);
3. when the User-Agent header is not set, we default to
a `miniooni/0.1.0-dev` user agent, which is probably not
very useful anyway, so we can actually remove it.
Part of https://github.com/ooni/probe/issues/1733 (this diff
has been extracted from https://github.com/ooni/probe-cli/pull/506).
While there, modernize the way in which we run tests to avoid
depending on the fake files scattered around the tree and to
use some well defined mock structures instead.
Part of https://github.com/ooni/probe/issues/1591
I discovered which transport were used by apitool and made sure he gets the same transports now. While there, I discovered an issue with ooni/oohttp that has been fixed with cba9b1ce5e.
Part of https://github.com/ooni/probe/issues/1591
* netxlite: improve docs, tests, and code quality
* better documentation
* more strict testing of dialer (especially make sure we
document the quirk in https://github.com/ooni/probe/issues/1779
and we have tests to guarantee we don't screw up here)
* introduce NewErrWrapper factory for creating errors so we
have confidence we are creating them correctly
Part of https://github.com/ooni/probe/issues/1591
Adapt other places where it was not using a logger to either choose
a reasonable logger or disable logging for backwards compat.
See https://github.com/ooni/probe/issues/1591
This simplifies serializing errors to `*string`. It did not
occur to me before. It seems quite a nice improvement.
Part of https://github.com/ooni/probe/issues/1591
They are now more readable. I'll do another pass and start
separating integration testing from unit testing.
I think we need to have some always on integration testing
for netxlite that runs on macOS, linux, and windows.
See https://github.com/ooni/probe/issues/1591
No real functional change. A few are needed and they will come
next. With this diff I just wanted to do cosmetic changes and
documentation changes, to ensure this package is okay.
See https://github.com/ooni/probe/issues/1591
The legacy part for now is internal/errorsx. It will stay there until
I figure out whether it also needs some extra bug fixing.
The good part is now in internal/netxlite/errorsx and contains all the
logic for mapping errors. We need to further improve upon this logic
by writing more thorough integration tests for QUIC.
We also need to copy the various dialer, conn, etc adapters that set
errors. We will put them inside netxlite and we will generate errors in
a way that is less crazy with respect to the major operation. (The
idea is to always wrap, given that now we measure in an incremental way
and we don't measure every operation together.)
Part of https://github.com/ooni/probe/issues/1591
No functional change, as it's clearly obvious from the output.
While there, also rename the generator for certifi. We are planning
on merging errorsx into netxlite. The first step is to give different
names to the code generating programs.
See https://github.com/ooni/probe/issues/1591
This is the last bit of functionality we need before rewriting a
chunk of websteps to use netxlite.
To rewrite most of it, we still need to move over:
1. dnstransport code
2. errorsx code
With both done, netxlite is a good library for websteps as well
as for most other operations we perform outside of the experiments.
Part of https://github.com/ooni/probe/issues/1591
With this change, we are now able to change more dependent code to simplify
the way in which we create and manage resolvers.
See https://github.com/ooni/probe/issues/1591
Like before, do not touch the rest of the tree. Rather create
compatibility types declared as legacy.
We will soon be able to close idle connections for an HTTP3
transport using any kind of resolvers more easily.
See https://github.com/ooni/probe/issues/1591
This quirk really saddens me. It's a piece of tech debt we're
carrying over from the original netx implementation.
We cannot remove it _until_ we have legacy netx code around.
The second best thing we can do is to clearly move this code in
a place where it's clear it's a quirk and write and use some extra
code that makes sure the quirk's assumptions are always met.
Sigh.
See https://github.com/ooni/probe/issues/1591
To make this happen, we need to take as argument a TLSDialer rather than
a TLSHandshaker. Then, we need to arrange the code so that we always
enforce a timeout for both TCP and TLS connections.
Because a TLSDialer can be constructed with a custom TLSConfig, we cover
also the case where the users wants to provide such a config.
While there, make sure we have better unit tests of the HTTP code.
See https://github.com/ooni/probe/issues/1591
While there reorganize mocks' tls implementation to use a single file
called tls.go (and tls_test.go) just like netxlite does.
While there write tests ensuring we always add timeouts when we are
making TCP connections (be them TLS or cleartext).
See https://github.com/ooni/probe/issues/1591
This basically adapts already existing code inside websteps to
instead be into the netxlite package, where it belongs.
In the process, abstract the TLSDialer but keep a reference to the
previous name to avoid refactoring existing code (just for now).
While there, notice that the right name is CloseIdleConnections (i.e.,
plural not singular) and change the name.
While there, since we abstracted TLSDialer to be an interface, create
suitable factories for making a TLSDialer type from a Dialer and a
TLSHandshaker.
See https://github.com/ooni/probe/issues/1591
We are proceeding with this plan of every major type being able to
close idle connections, which will simplify making DNS resolvers.
See https://github.com/ooni/probe/issues/1591.
