Commit Graph

141 Commits

Author SHA1 Message Date
Simone Basso
69fd0c5119
refactor(netxlite): allow easy dialer chain customization (#770)
This diff modifies the construction of a dialer to allow one
to insert custom dialer wrappers into the dialers chain.

The point of the chain in which we allow custom wrappers is the
optimal one for connect, read, and write measurements.

This new design is better than the previous netx design since
we don't need to construct the whole chain manually now.

The work in this diff is part of the effort to make engine/netx
just a tiny wrapper around netxlite.

See https://github.com/ooni/probe/issues/2121.
2022-05-31 20:02:11 +02:00
Simone Basso
e4f10eeac2
refactor: continue to simplify engine/netx (#769)
The objective of this diff is to simplify the code inside engine/netx
while moving more bits of code inside netxlite.

See https://github.com/ooni/probe/issues/2121
2022-05-31 08:11:07 +02:00
Simone Basso
314c3c934d
refactor(session.go): replace engine/netx with netxlite (#767)
This diff replaces engine/netx code with netxlite code in
the engine/session.go file. To this end, we needed to move
some code from engine/netx to netxlite. While there, we
did review and improve the unit tests.

A notable change in this diff is (or seems to be) that in
engine/session.go we're not filtering for bogons anymore so
that, in principle, we could believe a resolver returning
to us bogon IP addresses for OONI services. However, I did
not bother with changing bogons filtering because the
sessionresolver package is already filtering for bogons,
so it is actually okay to avoid doing that again the
session.go code. See:

https://github.com/ooni/probe-cli/blob/v3.15.0-alpha.1/internal/engine/internal/sessionresolver/resolvermaker.go#L88

There are two reference issues for this cleanup:

1. https://github.com/ooni/probe/issues/2115

2. https://github.com/ooni/probe/issues/2121
2022-05-30 22:00:45 +02:00
Simone Basso
595d0744db
netxlite: do not call netgo the CGO_ENABLED=0 resolver (#766)
In https://github.com/ooni/probe/issues/2029#issuecomment-1140805266, we
explained why calling it "netgo" would be incorrect.

In other words, we can get the platform's `getaddrinfo` as long as
we're not cross compiling. We do cross compile `ooniprobe`, actually
it's not even possible to cross compile it.

For increased accuracy, we should stop cross compiling `miniooni`
as well, so it would also directly use `getaddrinfo`.

This diff fixes at the same time ooni/probe-cli and ooni/spec
and we'll open two pull requests in parallel.
2022-05-30 10:06:53 +02:00
Simone Basso
f3912188e1
getaddrinfo: fix CGO_ENABLED=0 and record resolver type (#765)
After https://github.com/ooni/probe-cli/pull/764, the build for
CGO_ENABLED=0 has been broken for miniooni:

https://github.com/ooni/probe-cli/runs/6636995859?check_suite_focus=true

Likewise, it's not possible to run tests with CGO_ENABLED=0.

To make tests work with `CGO_ENABLED=0`, I needed to sacrifice some
unit tests run for the CGO case. It is not fully clear to me what was happening
here, but basically `getaddrinfo_cgo_test.go` was compiled with CGO
being disabled, even though the ``//go:build cgo` flag was specified.

Additionally, @hellais previously raised a valid point in the review
of https://github.com/ooni/probe-cli/pull/698:

> Another issue we should consider is that, if I understand how
> this works correctly, depending on whether or not we have built
> with CGO_ENABLED=0 on or not, we are going to be measuring
> things in a different way (using our cgo inspired getaddrinfo
> implementation or using netgo). This might present issues when
> analyzing or interpreting the data.
>
> Do we perhaps want to add some field to the output data format that
> gives us an indication of which DNS resolution code was used to
> generate the the metric?

This comment is relevant to the current commit because
https://github.com/ooni/probe-cli/pull/698 is the previous
iteration of https://github.com/ooni/probe-cli/pull/764.

So, while fixing the build and test issues, let us also distinguish
between the CGO_ENABLED=1 and CGO_ENABLED=0 cases.

Before this commit, OONI used "system" to indicate the case where
we were using net.DefaultResolver. This behavior dates back to the
Measurement Kit days. While it is true that ooni/probe-engine and
ooni/probe-cli could have been using netgo in the past when we
said "system" as the resolver, it also seems reasonable to continue
to use "system" top indicate getaddrinfo.

So, the choice here is basically to use "netgo" from now on to
indicate the cases in which we were built with CGO_ENABLED=0.

This change will need to be documented into ooni/spec along with
the introduction of the `android_dns_cache_no_data` error.

## Checklist

- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/2029
- [x] if you changed anything related how experiments work and you need to reflect these changes in the ooni/spec repository, please link to the related ooni/spec pull request: https://github.com/ooni/spec/pull/242
2022-05-30 07:34:25 +02:00
Simone Basso
cf6dbe48e0
netxlite: call getaddrinfo and handle platform-specific oddities (#764)
This commit changes our system resolver to call getaddrinfo directly when CGO is enabled. This change allows us to:

1. obtain the CNAME easily

2. obtain the real getaddrinfo retval

3. handle platform specific oddities such as `EAI_NODATA`
returned on Android devices

See https://github.com/ooni/probe/issues/2029 and https://github.com/ooni/probe/issues/2029#issuecomment-1140258729 in particular.

See https://github.com/ooni/probe/issues/2033 for documentation regarding the desire to see `getaddrinfo`'s retval.

See https://github.com/ooni/probe/issues/2118 for possible follow-up changes.
2022-05-28 15:10:30 +02:00
Simone Basso
62bd62ece1
fix(dnsoverudp): allow to cancel async round trip immediately (#763)
To this end, we need to refactor the implementation to give the
DNSOverUDPChannel owenership over the net.Conn.

Once this happens, DNSOverUDPChannel.Close closes the conn.

When the conn is closed, the background goroutine will terminate
immediately because any blocking I/O operation will be immediately
unblocked and return net.ErrClosed.

See https://github.com/ooni/probe/issues/2099#issuecomment-1139066946
2022-05-26 23:49:14 +02:00
Simone Basso
16f7407b13
feat(netxlite): observe additional DNS-over-UDP responses (#762)
This diff introduces support for observing additional DNS-over-UDP
responses in some censored environments (e.g. China).

After some uncertainty around whether to use connected or unconnected
UDP sockets, I eventually settled for connected.

Here's a recap:

|                         | connected | unconnected |
| ----------------------- | --------- | ----------- |
| see ICMP errors         | ✔️         |           |
| responses from any server |         | ✔️           |

Because most if not all DNS resolvers expect answers from exactly
the same servers to which they sent the query, I would say that
it's more important to have some limited ability of observing the
effect of ICMP errors (e.g., host_unreachable when we set a low
TTL and send out a query to a server).

Therefore, my choice was to modify the existing DNS-over-UDP transport.

Here's an overview of the changes:

1. introduce a new API for performing an async round trip that returns
a channel wrapper where all responses are posted. The channel will not ever
be closed, so the reader needs to use select for safely reading. If the
reader users the wrapper's Next or TryNextResponses methods, these details
do not matter because they already implement a safe reading pattern.

2. the async round trip API performs the round trip in the background
and stops processing when it sees the first error.

3. the background running code will use an overall deadline derived
from the DNSTransport.IOTimeout field to know when to stop.

4. the background running code will additionally stop running if
noone is reading the channel and there are no empty slots in the
channel's buffer.

5. the RoundTrip method has been rewritten in terms of the async API.

The design I'm using here implements the proposal for async round
trips defined at https://github.com/ooni/probe/issues/2099. I have
chosen not to make all transports async because the DNS transport
seems the only transport that needs to also work in async mode.

While there, I noticed that we were not propagating CloseIdleConnection
to the underlying dialer, which was potentially wrong, so I did it.
2022-05-26 20:09:00 +02:00
Simone Basso
01a513a496
refactor: DNSTransport I/Os DNS messages (#760)
This diff refactors the DNSTransport model to receive in input a DNSQuery and return in output a DNSResponse.

The design of DNSQuery and DNSResponse takes into account the use case of a transport using getaddrinfo, meaning that we don't need to serialize and deserialize messages when using getaddrinfo.

The current codebase does not use a getaddrinfo transport, but I wrote one such a transport in the Websteps Winter 2021 prototype (https://github.com/bassosimone/websteps-illustrated/).

The design conversation that lead to producing this diff is https://github.com/ooni/probe/issues/2099
2022-05-25 17:03:58 +02:00
Simone Basso
7a0a156aec
Spring cleanup: remove unused/unneded code (#761)
* cleanup: remove the archival package

See https://github.com/ooni/probe/issues/2116

* cleanup: remove websteps fall 2021 edition

See https://github.com/ooni/probe/issues/2116

* cleanup: remove JavaScript based testing framework

https://github.com/ooni/probe/issues/2116

* cleanup: remove the unused ooapi package

See https://github.com/ooni/probe/issues/2116
2022-05-25 13:21:39 +02:00
Simone Basso
2d721baa91
cleanup: merge httpheader and httpfailure into model (#758)
These two small packages could easily be merged into the model
package, since they're clearly model-like packages.

Part of https://github.com/ooni/probe/issues/2115
2022-05-25 09:54:50 +02:00
Simone Basso
d922bd9afc
cleanup: mark more integration tests as !short mode (#755)
The objective is to make PR checks run much faster.

See https://github.com/ooni/probe/issues/2113 for context.

Regarding netxlite's tests:

Checking for every commit on master or on a release branch is
good enough and makes pull requests faster than one minute since
netxlite for windows is now 1m slower than coverage.

We're losing some coverage but coverage from integration tests
is not so good anyway, so I'm not super sad about this loss.
2022-05-24 21:01:15 +02:00
Simone Basso
6924d1ad81
refactor: only use shaping dialer for ndt7 and dash (#754)
See https://github.com/ooni/probe/issues/2112 for context.

While there, run `go fix -fix buildtag ./...`
2022-05-24 18:23:42 +02:00
Simone Basso
ebc00a95fe
refactor: use ooni/oocrypto instead of ooni/go (#751)
Rather than building for Android using ooni/go, we're now using
ooni/oocryto as the TLS dependency. Such a repository only forks
crypto/tls and some minor crypto packages and includes the
same set of patches that we have been using in ooni/go.

This new strategy should be better than the previous one in
terms of building for Android, because we can use the vanilla
go1.18.2 build. It also seems that it is easier to track and
merge from upstream with ooni/oocrypto than it is with ooni/go.

Should this assessment be wrong, we can revert back to the
previous scenario where we used ooni/go.

See https://github.com/ooni/probe/issues/2106 for extra context.
2022-05-22 19:53:37 +02:00
Simone Basso
1a7234e0f1
chore: run go generate ./... (#743)
See https://github.com/ooni/probe/issues/2100
2022-05-19 20:44:39 +02:00
Simone Basso
7c45f7b88c
fix(netxlite): ensure we only accept DNS responses (#735)
Previously, the DNS decoder did not check whether it was parsing
a DNS query or a DNS response, which was wrong.

As a side note, it seems I am using "reply" in the codebase instead
of "response". The latter seems correct DNS terminology.

This diff has been extracted from 9249d14f80

See https://github.com/ooni/probe/issues/2096.
2022-05-16 11:17:30 +02:00
Simone Basso
ce052b665e
feat(netxlite): implements NS queries (#734)
This diff has been extracted from eb0bf38957.

See https://github.com/ooni/probe/issues/2096.

While there, skip the broken tests caused by issue
https://github.com/ooni/probe/issues/2098.
2022-05-16 10:46:53 +02:00
Simone Basso
c1b06a2d09
fix(netxlite): prefer composition over embedding (#733)
This diff has been extracted and adapted from 8848c8c516

The reason to prefer composition over embedding is that we want the
build to break if we add new methods to interfaces we define. If the build
does not break, we may forget about wrapping methods we should
actually be wrapping. I noticed this issue inside netxlite when I was working
on websteps-illustrated and I added support for NS and PTR queries.

See https://github.com/ooni/probe/issues/2096

While there, perform comprehensive netxlite code review
and apply minor changes and improve the docs.
2022-05-15 19:25:27 +02:00
Simone Basso
9d2301cae2
fix(netxlite): reject replies with wrong queryID (#732)
This diff has been extracted from c2f7ccab0e

See https://github.com/ooni/probe/issues/2096

While there, export DecodeReply to decode a raw reply without
interpreting the Rcode or parsing the results, which seems a
nice extra feature to have to more flexibly parse DNS replies
in other parts of the codebase.
2022-05-14 19:38:46 +02:00
Simone Basso
f5b801ae95
refactor(netxlite): add Transport suffix to DNS transports (#731)
This diff has been extracted from c2f7ccab0e

See https://github.com/ooni/probe/issues/2096
2022-05-14 17:38:31 +02:00
Simone Basso
6c388d2c61
netxlite: factor common code for parsing UDPAddr (#730)
This diff has been extracted from c2f7ccab0e

See https://github.com/ooni/probe/issues/2096
2022-05-14 17:15:08 +02:00
Simone Basso
2238908afe
cleanup(quic): wait for handshake completion in netxlite (#729)
See https://github.com/ooni/probe/issues/2097
2022-05-14 16:32:32 +02:00
Simone Basso
5904e6988d
fix(netxlite): map servfail error (#728)
This error occurred for example when querying kazemjalali.com
in websteps measurements run from Iran.

This error is relatively uncommon, but it still makes sense to
create a specific mapping rule for it.

Originally: 4269e82fbd

See https://github.com/ooni/probe/issues/2096
2022-05-13 19:25:22 +02:00
Simone Basso
b872dd0e1e
fix(netxlite): HTTPSSvc: better no_answer checks (#727)
I've seen some measurements returning some IP addresses for HTTPSSvc
queries but not returning any ALPN value.

For example:

```
% d4
decoding DNS round trip 0:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57768
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;psiphon.ca.                    IN      HTTPS

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57768
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;psiphon.ca.                    IN      HTTPS
;; ANSWER SECTION:
psiphon.ca.             121     IN      A       31.13.85.53
```

Now, the response is clearly bogus. At the time of this writing that
IP address belongs to Facebook. This measurement has been collected in
China, so it's expected for the GFW to behave like this.

Yet, I don't feel like it's accurate to report this measurement as a
"no answer" response. Rather, this response is a valid one containing
a clearly invalid IP address and should be flagged as such.

Originally: 57a023bcf4

See https://github.com/ooni/probe/issues/2096
2022-05-13 19:00:51 +02:00
Simone Basso
192dfd49b4
fix(netxlite): consolidate IPv4/IPv6 checking code (#726)
Originally 966e7f7cdd

See https://github.com/ooni/probe/issues/2096
2022-05-13 18:49:18 +02:00
Simone Basso
e126e73de7
fix(netxlite): LookupHTTPS: short circuit IP addr (#725)
This diff fixes the short-circuit-IP-addr resolver to
correctly handle IP addrs during LookupHTTPS.

The original diff was: 2b51d144bf

See https://github.com/ooni/probe/issues/2096

While there, add unit tests for IPv6.
2022-05-13 18:26:15 +02:00
Simone Basso
ec0561ea8c
feat(netxlite): implement parallel resolver (#724)
This diff imports the parallel resolver from websteps winter 2022
edition, which was originally implemented here:

55231d73cd

See https://github.com/ooni/probe/issues/2096
2022-05-13 17:36:58 +02:00
Simone Basso
0efd4ff130
chore: import improved bogons handling code (#723)
This diff imports improved bogons handling code from websteps
winter 2022 edition's repository.

See https://github.com/ooni/probe/issues/2095

See a65f3e8579/internal/netxlite/bogon.go
2022-05-13 15:32:47 +02:00
DecFox
5d2afaade4
cli: upgrade to lucas-clemente/quic-go@v0.27.0 (#715)
* quic-go upgrade: replaced Session/EarlySession with Connection/EarlyConnection

* quic-go upgrade: added context to RoundTripper.Dial

* quic-go upgrade: made corresponding changes to tutorial

* quic-go upgrade: changed sess variable instances to qconn

* quic-go upgrade: made corresponding changes to tutorial

* cleanup: remove unnecessary comments

Those comments made sense in terms of illustrating the changes
but they're going to be less useful once we merge.

* fix(go.mod): apparently we needed `go1.18.1 mod tidy`

VSCode just warned me about this. It seems fine to apply this
change as part of the pull request at hand.

* cleanup(netxlite): http3dialer can be removed

We used to use http3dialer to glue a QUIC dialer, which had a
context as its first argument, to the Dial function used by the
HTTP3 transport, which did not have a context as its first
argument.

Now that HTTP3 transport has a Dial function taking a context as
its first argument, we don't need http3dialer
anymore, since we can use the QUIC dialer directly.

Cc: @DecFox

* Revert "cleanup(netxlite): http3dialer can be removed"

This reverts commit c62244c620cee5fadcc2ca89d8228c8db0b96add
to investigate the build failure mentioned at
https://github.com/ooni/probe-cli/pull/715#issuecomment-1119450484

* chore(netx): show that test was already broken

We didn't see the breakage before because we were not using
the created transport, but the issue of using a nil dialer was
already present before, we just didn't see it.

Now we understand why removing the http3transport in
c62244c620cee5fadcc2ca89d8228c8db0b96add did cause the
breakage mentioned at
https://github.com/ooni/probe-cli/pull/715#issuecomment-1119450484

* fix(netx): convert broken integration test to working unit test

There's no point in using the network here. Add a fake dialer that
breaks and ensure we're getting the expected error.

We've now improved upon the original test because the original test was
not doing anything while now we're testing whether we get back a QUIC
dialer that _can be used_.

After this commit, I can then readd the cleanup commit
c62244c620cee5fadcc2ca89d8228c8db0b96add and it won't be
broken anymore (at least, this is what I expected to happen).

* Revert "Revert "cleanup(netxlite): http3dialer can be removed""

This reverts commit 0e254bfc6ba3bfd65365ce3d8de2c8ec51b925ff
because now we should have fixed the broken test.

Co-authored-by: decfox <decfox>
Co-authored-by: Simone Basso <bassosimone@gmail.com>
2022-05-06 12:24:03 +02:00
Simone Basso
306d18e466
chore: support go1.18 and update dependencies (#708)
Here's the squash of the following patches that enable support
for go1.18 and update our dependencies.

This diff WILL need to be backported to the release/3.14 branch.

* chore: use go1.17.8

See https://github.com/ooni/probe/issues/2067

* chore: upgrade to probe-assets@v0.8.0

See https://github.com/ooni/probe/issues/2067.

* chore: update dependencies and enable go1.18

As mentioned in 7a0d17ea91,
the tree won't build with `go1.18` unless we say it does.

So, not only here we need to update dependencies but also we
need to explicitly say `go1.18` in the `go.mod`.

This work is part of https://github.com/ooni/probe/issues/2067.

* chore(coverage.yml): run with go1.18

This change will give us a bare minimum confidence that we're
going to build our tree using version 1.18 of golang.

See https://github.com/ooni/probe/issues/2067.

* chore: update user agent used for measuring

See https://github.com/ooni/probe/issues/2067

* chore: run `go generate ./...`

See https://github.com/ooni/probe/issues/2067

* fix(dialer_test.go): make test work with go1.17 and go1.18

1. the original test wanted the dial to fail, so ensure we're not
passing any domain name to exercise dialing not resolving;

2. match the end of the error rather than the whole error string.

Tested locally with both go1.17 and go1.18.

See https://github.com/ooni/probe-cli/pull/708#issuecomment-1096447186
2022-04-12 11:43:12 +02:00
Dionysis Grigoropoulos
07f8db9dc2
feat: add support for OpenBSD (#703)
Closes https://github.com/ooni/probe/issues/2052
2022-03-08 12:25:33 +01:00
Simone Basso
ce8ec5b391
fix(reduceErrors): return error when given an empty list (#675)
See https://github.com/ooni/probe/issues/1985 for context.

While there, ensure nextlite has 100% of coverage.
2022-01-26 12:18:36 +01:00
Simone Basso
2a566f2046
feat: start preparing for a cli release (#672)
This diff includes some final changes to be ready for blessing
a cli release. These changes are:

1. run `go generate ./...` to update the bundled CA

2. update the header we use for measuring

3. ensure `mk` uses the latest version of several tools

Reference issue: https://github.com/ooni/probe/issues/1845
2022-01-24 14:56:51 +01:00
Simone Basso
e904b90006
feature: merge measurex and netx archival layer (1/N) (#663)
This diff introduces a new package called `./internal/archival`. This package collects data from `./internal/model` network interfaces (e.g., `Dialer`, `QUICDialer`, `HTTPTransport`), saves such data into an internal tabular data format suitable for on-line processing and analysis, and allows exporting data into the OONI data format.

The code for collecting and the internal tabular data formats are adapted from `measurex`. The code for formatting and exporting OONI data-format-compliant structures is adapted from `netx/archival`.

My original objective was to _also_ (1) fully replace `netx/archival` with this package and (2) adapt `measurex` to use this package rather than its own code. Both operations seem easily feasible because: (a) this code is `measurex` code without extensions that are `measurex` related, which will need to be added back as part of the process; (b) the API provided by this code allows for trivially converting from using `netx/archival` to using this code.

Yet, both changes should not be taken lightly. After implementing them, there's need to spend some time doing QA and ensuring all nettests work as intended. However, I am planning a release in the next two weeks, and this QA task is likely going to defer the release. For this reason, I have chosen to commit the work done so far into the tree and defer the second part of this refactoring for a later moment in time. (This explains why the title mentions "1/N").

On a more high-level perspective, it would also be beneficial, I guess, to explain _why_ I am doing these changes. There are two intertwined reasons. The first reason is that `netx/archival` has shortcomings deriving from its original https://github.com/ooni/netx legacy. The most relevant shortcoming is that it saves all kind of data into the same tabular structure named `Event`. This design choice is unfortunate because it does not allow one to apply data-type specific logic when processing the results. In turn, this choice results in complex processing code. Therefore, I believe that replacing the code with event-specific data structures is clearly an improvement in terms of code maintainability and would quite likely lead us to more confidently change and evolve the codebase.

The second reason why I would like to move forward these changes is to unify the codepaths used for measuring. At this point in time, we basically have two codepaths: `./internal/engine/netx` and `./internal/measurex`. They both have pros and cons and I don't think we want to rewrite whole experiments using `netx`. Rather, what we probably want is to gradually merge these two codepaths such that `netx` is a set of abstractions on top of `measurex` (which is more low-level and has a more-easily-testable design). Because saving events and generating an archival data format out of them consists of at least 50% of the complexity of both `netx` and `measurex`, it seems reasonable to unify this archival-related part of the two codebases as the first step.

At the highest level of abstraction, these changes are part of the train of changes which will eventually lead us to bless `websteps` as a first class citizen in OONI land. Because `websteps` requires different underlying primitives, I chose to develop these primitives from scratch rather than wrestling with `netx`, which used another model. The model used by `websteps` is that we perform each operation in isolation and immediately we save the results, while `netx` creates whole data structures and collects all the events happening via tracing. We believe the model used by `websteps` to be better because it does not require your code to figure out everything that happened after the measurement, which is a source of subtle bugs in the current implementation. So, when I started implementing websteps I extracted the bits of `netx` that could also be beneficial to `websteps` into a separate library, thus `netxlite` was born.

The reference issue describing merging the archival of `netx` and `measurex` is https://github.com/ooni/probe/issues/1957. As of this writing the issue still references the original plan, which I could not complete by the end of this Sprint, so I am going to adapt the text of the issue to only refer to what was done in here next. Of course, I also need follow-up issues.
2022-01-14 12:13:10 +01:00
Simone Basso
b5da8be183
fix(netxlite): robust {ReadAll,Copy}Context with wrapped io.EOF (#661)
* chore(netxlite): add currently failing test case

This diff introduces a test cases that will fail because of the reason
explained in https://github.com/ooni/probe/issues/1965.

* chore(netxlite/iox_test.go): add failing unit tests

These tests directly show how the Go implementation of ReadAll
and Copy has the issue of checking for io.EOF equality.

* fix(netxlite): make {ReadAll,Copy}Context robust to wrapped io.EOF

The fix is simple: we just need to check for `errors.Is(err, io.EOF)`
after either io.ReadAll or io.Copy has returned. When this condition is
true, we need to convert the error back to `nil` as it ought to be.

While there, observe that the unit tests I committed in the previous
commit are wrongly asserting that the error must be wrapped. This
assertion is not correct, because in both cases we have just ensured
that the returned error is `nil` (i.e., success).

See https://github.com/ooni/probe/issues/1965.

* cleanup: remove previous workaround for wrapped io.EOF

These workarounds were partial, meaning that they would cover some
cases in which the issue occurred but not all of them.

Handling the problem in `netxlite.{ReadAll,Copy}Context` is the
right thing to do _as long as_ we always use these functions instead
of `io.{ReadAll,Copy}`.

This is why it's now important to ensure we clearly mention that
inside of the `CONTRIBUTING.md` guide and to also ensure that we're
not using these functions in the code base.

* fix(urlgetter): repair tests who assumed to see EOF error

Now that we have established that we should normalize EOF when
reading bodies like the stdlib does and now that it's clear why
our behavior diverged from the stdlib, we also need to repair
all the tests that assumed this incorrect behavior.

* fix(all): don't use io{,util}.{Copy,ReadAll}

* feat: add checks to ensure we don't use io.{Copy,ReadAll}

* doc(netxlite): document we know how to deal w/ wrapped io.EOF

* fix(nocopyreadall.bash): add exception for i/n/iox.go
2022-01-12 14:26:10 +01:00
Simone Basso
566c6b246a
cleanup: remove unnecessary legacy interfaces (#656)
This diff addresses another point of https://github.com/ooni/probe/issues/1956:

> - [ ] observe that we're still using a bunch of private interfaces for common interfaces such as the `Dialer`, so we can get rid of these private interfaces and always use the ones in `model`, which allows us to remove a bunch of legacy wrappers

Additional cleanups may still be possible. The more I cleanup, the more I see
there's extra legacy code we can dispose of (which seems good?).
2022-01-07 18:33:37 +01:00
Simone Basso
1c057d322d
cleanup: merge legacy errorsx in netxlite and hide classifiers (#655)
This diff implements the first two cleanups defined at https://github.com/ooni/probe/issues/1956:

> - [ ] observe that `netxlite` and `netx` differ in error wrapping only in the way in which we set `ErrWrapper.Operation`. Observe that the code using `netxlite` does not care about such a field. Therefore, we can modify `netxlite` to set such a field using the code of `netx` and we can remove `netx` specific code for errors (which currently lives inside of the `./internal/engine/legacy/errorsx` package
>
> - [ ] after we've done the previous cleanup, we can make all the classifiers code private, since there's no code outside `netxlite` that needs them

A subsequent diff will address the remaining cleanup.

While there, notice that there are failing, unrelated obfs4 tests, so disable them in short mode. (I am confident these tests are unrelated because they fail for me when running test locally from the `master` branch.)
2022-01-07 17:31:21 +01:00
Simone Basso
99ec7ffca9
fix: ensure experiments return nil when we want to submit (#654)
Since https://github.com/ooni/probe-cli/pull/527, if an experiment
returns an error, the corresponding measurement is not submitted since
the semantics of returning an error is that something fundamental
went wrong (e.g., we could not parse the input URL).

This diff ensures that all experiments only return and error when
something fundamental was wrong and return nil otherwise.

Reference issue: https://github.com/ooni/probe/issues/1808.
2022-01-07 13:17:20 +01:00
Simone Basso
f0181c432f
refactor: move httpx into the internal package (#646)
This concludes the TODO list at https://github.com/ooni/probe/issues/1951
2022-01-05 17:17:20 +01:00
Simone Basso
273b70bacc
refactor: interfaces and data types into the model package (#642)
## Checklist

- [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md)
- [x] reference issue for this pull request: https://github.com/ooni/probe/issues/1885
- [x] related ooni/spec pull request: N/A

Location of the issue tracker: https://github.com/ooni/probe

## Description

This PR contains a set of changes to move important interfaces and data types into the `./internal/model` package.

The criteria for including an interface or data type in here is roughly that the type should be important and used by several packages. We are especially interested to move more interfaces here to increase modularity.

An additional side effect is that, by reading this package, one should be able to understand more quickly how different parts of the codebase interact with each other.

This is what I want to move in `internal/model`:

- [x] most important interfaces from `internal/netxlite`
- [x] everything that was previously part of `internal/engine/model`
- [x] mocks from `internal/netxlite/mocks` should also be moved in here as a subpackage
2022-01-03 13:53:23 +01:00
Simone Basso
611fed05f4
[forwardport] release 3.11: update all the dependencies (#636) (#637)
* [forwardport] release 3.11: update all the dependencies (#636)

This diff forward ports e291e436b3c332300f5567796f9c48bb9bc1e652.

* chore: use go1.17.4 everywhere

Part of https://github.com/ooni/probe/issues/1843

* chore: update to the latest ooni/oohttp

Part of https://github.com/ooni/probe/issues/1843

* chore: update the dependencies

Note: I did an update and not an upgrade (i.e., I didn't check
whether we have next-major-versions of dependencies).

Part of https://github.com/ooni/probe/issues/1843

* chore: update the user-agent we use

Part of https://github.com/ooni/probe/issues/1843

* chore: update ooni/oohttp and ooni/probe-assets

Part of https://github.com/ooni/probe/issues/1843

* chore: run go generate again

Part of https://github.com/ooni/probe/issues/1843

* fix(jafar): skip currently broken test

Created issue for it here: https://github.com/ooni/probe/issues/1913
2021-12-06 17:46:13 +01:00
Simone Basso
50576daec8
[forwardport] fix(ios): make sure we can build an iOS release (#598) (#599)
This diff forward ports f47b0c6c16e0cd417e3591358eb85b45962f307d to master.

Original commit message:

- - -

1. we now need to name the framework `.xcframework` otherwise
gomobile refuses to build a new framework for us ¯\_(ツ)_/¯

2. remove duplicate errno definition for iOS (iOS and darwin
are considered the same, therefore we don't need iOS defs)

Reference issue for this PR: https://github.com/ooni/probe/issues/1876

This diff WILL need to be forwardported to master.
2021-11-15 14:24:20 +01:00
Simone Basso
0a322ebab0
[forwardport] fix: avoid http3 for dns.google and www.google.com (#593) (#594)
This commit forward ports dedd84fa7ecb09f718f6b1a9c83999cb37b34dfa.

Original commit message:

- - -

This diff changes code the release/3.11 branch to ensure we're not using dns.google and www.google.com over HTTP3. As documented in https://github.com/ooni/probe/issues/1873, since this morning (approx) these services do not support HTTP3 anymore. (I didn't bother with checking whether this issue affects _other_ Google services; I just limited my analysis to the services that we were using as part of testing.)

This patch WILL require forward porting to the master branch.
2021-11-12 14:43:28 +01:00
Simone Basso
0d65438ea1
[forwardport] fix(netxlite): remove android-specific errno (#586) (#587)
This commit forward ports 74947dbbd12266c12a38fad51a70fc78a21720fd from
the `release/3.11` branch to `master`. Here's the original commit message:

- - -

Android is also Linux. The Android build fails because both
errno_linux.go and errno_android.go are compiled.

There's no difference between the files except into a comment
that mentions "linux" or "android".

Therefore, it's safe to remove the android-specific file
and just keep and use the linux-specific one.

Part of https://github.com/ooni/probe/issues/1863, where we're
forward porting ooni/go patches to go1.17.

I'm still trying to figure out whether I can build oonimkall
using the forward ported patches and this error prevents me
from building, because the build fails.

"やれやれだぜ"

Note that this patch WILL need to be forward ported to master.

This bug was previosuly reported to me by @hellais.

Because I did run `go generate ./internal/netxlite/...` we also
get for free updated certificates, which is OK.
2021-11-10 13:00:41 +01:00
Simone Basso
6a935d5407
fix(netxlite): ensure HTTP errors are always wrapped (#584)
1. introduce implementations of HTTPTransport and HTTPClient
that apply an error wrapping policy using the constructor
for a generic top-level error wrapper

2. make sure we use the implementations in point 1 when we
are constructing HTTPTransport and HTTPClient

3. make sure we apply error wrapping using the constructor for
a generic top-level error wrapper when reading bodies

4. acknowledge that error wrapping would be broken if we do
not return the same classification _and_ operation when we wrap
an already wrapped error, so fix the to code to do that

5. acknowledge that the classifiers already deal with preserving
the error string and explain why this is a quirk and why we
cannot remove it right now and what needs to happen to safely
remove this quirk from the codebase

Closes https://github.com/ooni/probe/issues/1860
2021-11-06 17:49:58 +01:00
Simone Basso
100cc559a2
Revert "feat(filtering): implement the divert policy (#569)" (#570)
This reverts commit 851b9913fa because
it seems it's not enough to allow us to see certificate errors with
quic, plus it's complex code. So, we'd rather develop a better approach,
and perhaps a simpler one, that works with QUIC as well.
2021-11-03 14:01:32 +01:00
Simone Basso
851b9913fa
feat(filtering): implement the divert policy (#569)
This is the policy we need to provoke certificate errors. We'll divert
from, say, `8.8.8.8:443/udp` to, say, `1.1.1.1:443/udp`.

We'll do something similar for `443/tcp`.

This will cause certificate validation errors.

With this change, we have now implemented the simple design described
by https://github.com/ooni/probe/issues/1803#issuecomment-957323297.
2021-11-03 00:29:14 +01:00
Simone Basso
675e3a5ba5
feat(filtering): introduce DNS cache (#568)
When we're testing multiple endpoints, it's quite important to control
the order with which they are returned to the code.

This feature is especially relevant to Web Connectivity, which will
check the endpoints to connect to in order.

Therefore, we need to force deterministic results to ensure that we can
have deterministic tests when doing Web Connectivity QA.

This diff gives us the guarantee that we can have determinism.

Part of https://github.com/ooni/probe/issues/1803#issuecomment-957323297.
2021-11-02 23:35:09 +01:00
Simone Basso
11ccd16a0c
feat(filtering): add transparent proxy with censorship policies (#566)
This PR implements the core concept described at
https://github.com/ooni/probe/issues/1803#issuecomment-957323297
2021-11-02 21:52:32 +01:00
Simone Basso
560b1a9a97
fix(filtering): avoid the if err == nil pattern (#567)
1. in normal code is better to always do if err != nil so that
the ifs only contain error code (this is ~coding policy)

2. in tests we want to ensure we narrow down the error to the
real error that happened, to have greater confidence

Written while working on https://github.com/ooni/probe/issues/1803#issuecomment-957323297
2021-11-02 19:48:10 +01:00