This diff addresses another point of https://github.com/ooni/probe/issues/1956:
> - [ ] observe that we're still using a bunch of private interfaces for common interfaces such as the `Dialer`, so we can get rid of these private interfaces and always use the ones in `model`, which allows us to remove a bunch of legacy wrappers
Additional cleanups may still be possible. The more I cleanup, the more I see
there's extra legacy code we can dispose of (which seems good?).
This diff implements the first two cleanups defined at https://github.com/ooni/probe/issues/1956:
> - [ ] observe that `netxlite` and `netx` differ in error wrapping only in the way in which we set `ErrWrapper.Operation`. Observe that the code using `netxlite` does not care about such a field. Therefore, we can modify `netxlite` to set such a field using the code of `netx` and we can remove `netx` specific code for errors (which currently lives inside of the `./internal/engine/legacy/errorsx` package
>
> - [ ] after we've done the previous cleanup, we can make all the classifiers code private, since there's no code outside `netxlite` that needs them
A subsequent diff will address the remaining cleanup.
While there, notice that there are failing, unrelated obfs4 tests, so disable them in short mode. (I am confident these tests are unrelated because they fail for me when running test locally from the `master` branch.)
Since https://github.com/ooni/probe-cli/pull/527, if an experiment
returns an error, the corresponding measurement is not submitted since
the semantics of returning an error is that something fundamental
went wrong (e.g., we could not parse the input URL).
This diff ensures that all experiments only return and error when
something fundamental was wrong and return nil otherwise.
Reference issue: https://github.com/ooni/probe/issues/1808.
## Checklist
- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/1885
- [x] related ooni/spec pull request: N/A
Location of the issue tracker: https://github.com/ooni/probe
## Description
This PR contains a set of changes to move important interfaces and data types into the `./internal/model` package.
The criteria for including an interface or data type in here is roughly that the type should be important and used by several packages. We are especially interested to move more interfaces here to increase modularity.
An additional side effect is that, by reading this package, one should be able to understand more quickly how different parts of the codebase interact with each other.
This is what I want to move in `internal/model`:
- [x] most important interfaces from `internal/netxlite`
- [x] everything that was previously part of `internal/engine/model`
- [x] mocks from `internal/netxlite/mocks` should also be moved in here as a subpackage
This diff forward ports f47b0c6c16e0cd417e3591358eb85b45962f307d to master.
Original commit message:
- - -
1. we now need to name the framework `.xcframework` otherwise
gomobile refuses to build a new framework for us ¯\_(ツ)_/¯
2. remove duplicate errno definition for iOS (iOS and darwin
are considered the same, therefore we don't need iOS defs)
Reference issue for this PR: https://github.com/ooni/probe/issues/1876
This diff WILL need to be forwardported to master.
This commit forward ports dedd84fa7ecb09f718f6b1a9c83999cb37b34dfa.
Original commit message:
- - -
This diff changes code the release/3.11 branch to ensure we're not using dns.google and www.google.com over HTTP3. As documented in https://github.com/ooni/probe/issues/1873, since this morning (approx) these services do not support HTTP3 anymore. (I didn't bother with checking whether this issue affects _other_ Google services; I just limited my analysis to the services that we were using as part of testing.)
This patch WILL require forward porting to the master branch.
This commit forward ports 74947dbbd12266c12a38fad51a70fc78a21720fd from
the `release/3.11` branch to `master`. Here's the original commit message:
- - -
Android is also Linux. The Android build fails because both
errno_linux.go and errno_android.go are compiled.
There's no difference between the files except into a comment
that mentions "linux" or "android".
Therefore, it's safe to remove the android-specific file
and just keep and use the linux-specific one.
Part of https://github.com/ooni/probe/issues/1863, where we're
forward porting ooni/go patches to go1.17.
I'm still trying to figure out whether I can build oonimkall
using the forward ported patches and this error prevents me
from building, because the build fails.
"やれやれだぜ"
Note that this patch WILL need to be forward ported to master.
This bug was previosuly reported to me by @hellais.
Because I did run `go generate ./internal/netxlite/...` we also
get for free updated certificates, which is OK.
1. introduce implementations of HTTPTransport and HTTPClient
that apply an error wrapping policy using the constructor
for a generic top-level error wrapper
2. make sure we use the implementations in point 1 when we
are constructing HTTPTransport and HTTPClient
3. make sure we apply error wrapping using the constructor for
a generic top-level error wrapper when reading bodies
4. acknowledge that error wrapping would be broken if we do
not return the same classification _and_ operation when we wrap
an already wrapped error, so fix the to code to do that
5. acknowledge that the classifiers already deal with preserving
the error string and explain why this is a quirk and why we
cannot remove it right now and what needs to happen to safely
remove this quirk from the codebase
Closes https://github.com/ooni/probe/issues/1860
This reverts commit 851b9913fa because
it seems it's not enough to allow us to see certificate errors with
quic, plus it's complex code. So, we'd rather develop a better approach,
and perhaps a simpler one, that works with QUIC as well.
This is the policy we need to provoke certificate errors. We'll divert
from, say, `8.8.8.8:443/udp` to, say, `1.1.1.1:443/udp`.
We'll do something similar for `443/tcp`.
This will cause certificate validation errors.
With this change, we have now implemented the simple design described
by https://github.com/ooni/probe/issues/1803#issuecomment-957323297.
When we're testing multiple endpoints, it's quite important to control
the order with which they are returned to the code.
This feature is especially relevant to Web Connectivity, which will
check the endpoints to connect to in order.
Therefore, we need to force deterministic results to ensure that we can
have deterministic tests when doing Web Connectivity QA.
This diff gives us the guarantee that we can have determinism.
Part of https://github.com/ooni/probe/issues/1803#issuecomment-957323297.
1. in normal code is better to always do if err != nil so that
the ifs only contain error code (this is ~coding policy)
2. in tests we want to ensure we narrow down the error to the
real error that happened, to have greater confidence
Written while working on https://github.com/ooni/probe/issues/1803#issuecomment-957323297
This change will simplify follow-up work done as part of
https://github.com/ooni/probe/issues/1803#issuecomment-957323297 to
implement a comprehensive self-censoring solution.
While there, rename the "proxy" action to "pass" because what we
are effectively doing is passing traffic to the network (that's a
minor change but it seems a better analogy).
* feat: run ~always netxlite integration tests
This diff ensures that we check on windows, linux, macos that our
fundamental networking library (netxlite) works.
We combine unit and integration tests.
This work is part of https://github.com/ooni/probe/issues/1733, where
I want to have more strong guarantees about the foundations.
* fix(filtering/tls_test.go): make portable on Windows
The trick here is to use the wrapped error so to normalize the
different errors messages we see on Windows.
* fix(netxlite/quic_test.go): make portable on windows
Rather than using the zero port, use the `x` port which fails
when the stdlib is parsing the address.
The zero port seems to work on Windows while it does not on Unix.
* fix(serialresolver_test.go): make error more timeout than before
This seems enough to convince Go on Windows about this error
being really a timeout timeouty timeouted thingie.
On Windows, GetAddrInfoW is a syscall and the Go resolver does
not attempt to map errors beyond WSA_HOST_NOT_FOUND, which becomes
"no such host", which we map to "dns_nxdomain_error".
See https://github.com/golang/go/blob/go1.17.1/src/net/lookup_windows.go#L16.
To map more GetAddrInfoW errors, thus, we need to enhance our
error classifier to have system specific errors.
Then, we need to filter for the WSA errors that are most likely
to pop up and map them to OONI failures. Those are three:
- WSANO_DATA which we have from our own UDP resolver as well
and which we can map to `dns_no_answer`
- WSANO_RECOVERY which we don't have but existed for MK so
we will use `dns_non_recoverable_failure`, which was an MK error
- WSATRY_AGAIN which likewise we map to the error that MK
used to emit, so `dns_temporary_failure`
This diff should address https://github.com/ooni/probe/issues/1467.
I need to run test on Windows and I just discovered that:
1. the `errno_unix.go` filename does not mean anything because
`unix` is not a valid platform, so we need a filename for
each platform that we care about;
2. on Windows we need to use WSA prefixed names;
3. `i/e/session_psiphon.go` was not building because of the
migration from `netxlite/iox` to `netxlite`.
This diff attempts to fix all three issues.
The reference issue is https://github.com/ooni/probe/issues/1733,
because I was working on such an issue.
When preparing a tutorial for netxlite, I figured it is easier
to tell people "hey, this is the package you should use for all
low-level networking stuff" rather than introducing people to
a set of packages working together where some piece of functionality
is here and some other piece is there.
Part of https://github.com/ooni/probe/issues/1591
I have recently seen a data race related our way of
mutating the outgoing request to set the host header.
Unfortunately, I've lost track of the race output,
because I rebooted my Linux box before saving it.
Though, after inspecting why and and where we're mutating
outgoing requets, I've found that:
1. we add the host header when logging to have it logged,
which is not a big deal since we already emit the URL
rather than just the URL path when logging a request, and
so we can safely zap this piece of code;
2. as a result, in measurements we may omit the host header
but again this is pretty much obvious from the URL itself
and so it should not be very important (nonetheless,
avoid surprises and keep the existing behavior);
3. when the User-Agent header is not set, we default to
a `miniooni/0.1.0-dev` user agent, which is probably not
very useful anyway, so we can actually remove it.
Part of https://github.com/ooni/probe/issues/1733 (this diff
has been extracted from https://github.com/ooni/probe-cli/pull/506).
While there, modernize the way in which we run tests to avoid
depending on the fake files scattered around the tree and to
use some well defined mock structures instead.
Part of https://github.com/ooni/probe/issues/1591
I discovered which transport were used by apitool and made sure he gets the same transports now. While there, I discovered an issue with ooni/oohttp that has been fixed with cba9b1ce5e.
Part of https://github.com/ooni/probe/issues/1591
* netxlite: improve docs, tests, and code quality
* better documentation
* more strict testing of dialer (especially make sure we
document the quirk in https://github.com/ooni/probe/issues/1779
and we have tests to guarantee we don't screw up here)
* introduce NewErrWrapper factory for creating errors so we
have confidence we are creating them correctly
Part of https://github.com/ooni/probe/issues/1591
Adapt other places where it was not using a logger to either choose
a reasonable logger or disable logging for backwards compat.
See https://github.com/ooni/probe/issues/1591
This simplifies serializing errors to `*string`. It did not
occur to me before. It seems quite a nice improvement.
Part of https://github.com/ooni/probe/issues/1591
