* feat: introduce ptx package for pluggable transports dialers
Version 2 of the pluggable transports specification defines a function
that's like `Dial() (net.Conn, error`).
Because we use contexts as much as possible in `probe-cli`, we are
wrapping such an interface into a `DialContext` func.
The code for obfs4 is adapted from https://github.com/ooni/probe-cli/pull/341.
The code for snowflake is significantly easier than it is in
https://github.com/ooni/probe-cli/pull/341, because now Snowflake
supports the PTv2 spec (thanks @cohosh!).
The code for setting up a pluggable transport listener has also
been adapted from https://github.com/ooni/probe-cli/pull/341.
We cannot merge this code yet, because we need unit testing, yet the
newly added code already seems suitable for these use cases:
1. testing by dialing and seeing whether we can dial (which is not
very useful but still better than not doing it);
2. spawning tor+pluggable transports for circumvention (we need a
little more hammering like we did in https://github.com/ooni/probe-cli/pull/341,
which is basically https://github.com/ooni/probe/issues/1565, and then
we will be able to do that, as demonstrated by the new, simple client which
already allows us to use pluggable transports with tor);
3. testing by launching tor (when available) with a set of
pluggable transports (which depends on https://github.com/ooni/probe-engine/issues/897
and has not been assigned an issue yet).
* fix: tweaks after self code-review
* feat: write quick tests for ptx/obfs4
(They run in 0.4s, so I think it's fine for them to always run.)
* feat(ptx/snowflake): write unit and integration tests
* feat: create a fake PTDialer
The idea is that we'll use this simpler PTDialer for testing.
* feat: finish writing tests for new package
* Apply suggestions from code review
* Update internal/ptx/dependencies_test.go
Co-authored-by: Arturo Filastò <arturo@openobservatory.org>
* Update internal/ptx/dependencies_test.go
Co-authored-by: Arturo Filastò <arturo@openobservatory.org>
* chore: use as testing bridge one that's used by tor browser
The previous testing bridge used to be used by tor browser but
it was subsequently removed here:
e26e91bef8
See https://github.com/ooni/probe-cli/pull/373#discussion_r649820724
Co-authored-by: Arturo Filastò <arturo@openobservatory.org>
* refactor(netx/dialer): hide implementation complexity
This follows the blueprint of `module.Config` and `nodule.New`
described at https://github.com/ooni/probe/issues/1591.
* fix: ndt7 bug where we were not using the right resolver
* fix(legacy/netx): clarify irrelevant implementation change
* fix: improve comments
* fix(hhfm): do not use dialer.New b/c it breaks it
Unclear to me why this is happening. Still, improve upon the
previous situation by adding a timeout.
It does not seem a priority to look into this issue now.
The socks5 factory always returns a DialContext capable dialer. We just
need to cast to obtain such a dialer.
Also, the code will use the DialContext if passed a dialer that
implements DialContext.
Write a test that proves my point.
Part of https://github.com/ooni/probe/issues/1591.
We already configure a timeout in the underlying dialer, hence
there's no point in keeping the TimeoutDialer around.
Part of https://github.com/ooni/probe/issues/1507
We're currently use jafar for QA and jafar is a better mechanism,
even though it is not portable outside of Linux.
This self censorship mechanism was less cool and added a bunch
of (also cognitive) complexity to netx.
If we ever want to go down a self censorship like road, we probably
want to do as little work as possible in the problem and as much
work as possible inside a helper like jafar.
Part of https://github.com/ooni/probe/issues/1591.
I am currently working on https://github.com/ooni/probe-engine/issues/725 with an eye on https://github.com/ooni/probe/issues/1507. While approaching these issues, it seems that the `netx` codebase could use some consolidation (_coagula_) and some splitting (_solve_).
The general idea of these changes is that I want to arrive to the situation where we have (1) a `New` factory method for each package under `netx` for which it makes sense (e.g., `dialer.New`, `tlsdialer.New`, `httptransport.New`), (2) a separate `Config` structure per package (e.g., `dialer.Config`) rather than having all the possible config into the same structure (3) part of the `urlgetter` code (and namely the low-level part) moved into the `netx` package. (See https://github.com/ooni/probe/issues/1591.)
There is too much bureaucracy around writing a new experiment. Much of this bureaucracy will go (it seems) if we do what I have said above. After that, we will end up that you run tests by using the top-level `netx` package. (In any case, I am not of course 100% sure about all the changes that will come after, but this comment seems enough to set a direction.)
These are the changes in this pull request:
* refactor(netx): extract tlsdialer from dialer
We want these two packages to be separate. Dialer was doing too much before. Separating TLS dialing code into a separate package allows us to have a `tlsdialer.New` factory that clearly conveys the meaning. Also, this would allow us to much more clearly separate configuration and simplify reasoning on what each factory does.
* refactor(engine): move `tlsx` package inside `netx` and merge the `gocertifi` package inside it
It would be tempting to merge everything inside `tlsdialer` but the reality is that also the `quicdialer` package needs to use the same code, therefore, it seems more tidy to actually have some tls extensions in `netx`.
* cleanup(platform): we don't need CGO anymore
Since go1.16, we have the `ios` port. So we can easily
disambiguate between ios and darwin.
This means we don't need to rely on CGO to correctly guess
whether we are on ios or darwin anymore.
So, now miniooni does not depend on a C compiler even
when you are not cross compiling.
* Update internal/platform/platform.go
* refactor(atomicx): move outside the engine package
After merging probe-engine into probe-cli, my impression is that we have
too much unnecessary nesting of packages in this repository.
The idea of this commit and of a bunch of following commits will instead
be to reduce the nesting and simplify the structure.
While there, improve the documentation.
* fix: always use the atomicx package
For consistency, never use sync/atomic and always use ./internal/atomicx
so we can just grep and make sure we're not risking to crash if we make
a subtle mistake on a 32 bit platform.
While there, mention in the contributing guidelines that we want to
always prefer the ./internal/atomicx package over sync/atomic.
* fix(atomicx): remove unnecessary constructor
We don't need a constructor here. The default constructed `&Int64{}`
instance is already usable and the constructor does not add anything to
what we are doing, rather it just creates extra confusion.
* cleanup(atomicx): we are not using Float64
Because atomicx.Float64 is unused, we can safely zap it.
* cleanup(atomicx): simplify impl and improve tests
We can simplify the implementation by using defer and by letting
the Load() method call Add(0).
We can improve tests by making many goroutines updated the
atomic int64 value concurrently.
* refactor(fsx): can live in the ./internal pkg
Let us reduce the amount of nesting. While there, ensure that the
package only exports the bare minimum, and improve the documentation
of the tests, to ease reading the code.
* refactor: move runtimex to ./internal
* refactor: move shellx into the ./internal package
While there, remove unnecessary dependency between packages.
While there, specify in the contributing guidelines that
one should use x/sys/execabs instead of os/exec.
* refactor: move ooapi into the ./internal pkg
* refactor(humanize): move to ./internal and better docs
* refactor: move platform to ./internal
* refactor(randx): move to ./internal
* refactor(multierror): move into the ./internal pkg
* refactor(kvstore): all kvstores in ./internal
Rather than having part of the kvstore inside ./internal/engine/kvstore
and part in ./internal/engine/kvstore.go, let us put every piece of code
that is kvstore related into the ./internal/kvstore package.
* fix(kvstore): always return ErrNoSuchKey on Get() error
It should help to use the kvstore everywhere removing all the
copies that are lingering around the tree.
* sessionresolver: make KVStore mandatory
Simplifies implementation. While there, use the ./internal/kvstore
package rather than having our private implementation.
* fix(ooapi): use the ./internal/kvstore package
* fix(platform): better documentation
We are mostly good to declare a stable release. We still need to deal with https://github.com/ooni/probe/issues/1484.
In this PR, we fix the aforementioned issue. These are the changes:
1. we remove the vendored `debops-ci`, and we pull it directly from `ooni/sysadmin`
2. we introduce a new script, `./CLI/linux/pubdebian`, to publish packages
3. we modify `./mk` to allow for publishing debian packages built outside of CI
The latter point has been quite useful in debugging what was wrong.
1. we can merge the e2eminiooni.yml test into the miniooni.yml test
so to reduce the number of tests we run;
2. ideally we would like the smoketest.sh test to evolve and also
check whether we can fetch the measurements we submitted, so start
moving this script into the `./E2E` folder, add a note saying we
would like to do that, and direct all the tests to run this script
at its new location and with its new name (`ooniprobe.sh`).
With these two changes, it's fine to remove the ooniprobe2debian.yml
test in ooni/e2etesting because we're moving its functionality to this
repository. (We mentioned the need to do this move in a previous TODO
comment at the top of such a script.)
Work part of https://github.com/ooni/probe/issues/1468
This pull request fixes https://github.com/ooni/probe/issues/1471. We have replaced the original build script (`./make`) with the `./mk` makefile (executable using `#!/usr/bin/make -f`). We concluded supporting direct builds from Windows is not worth the effort and halving the code we need to maintain is probably a good plus. Both macOS and Linux install GNU make at `/usr/bin/make`, so we should be okay in the common use cases.
I significantly simplified the management of Go versioning by requiring the user to manage it and by enforcing that we are using the desired Go version. This speeds up builds and works in sane operating systems that use the last version of a specific package. Otherwise, it's possible to use the `go get golang.org/dl/go${version}` feature.
The remaining question mark was related to updating the Android SDK. I have determined that a good course of action is pinning to the latest CLI tools and always forcing the CLI tools to install the latest required packages (e.g., the NDK).
We're still working on https://github.com/ooni/probe/issues/1466. The idea here is to teach the GH action for Linux to publish the debian package for arm64. When this is done, we can cleanup legacy build scripts and GH actions, because there is no remaining use case for them: we now build everything using the `./make` tool.
Part of https://github.com/ooni/probe/issues/1466. We're building both `arm64` and `amd64`. We are still not publishing `arm64` packages, which is what is asked in the original issue, but we're really close to doing that.
This PR groups misc cleanup and changes from https://github.com/ooni/probe-cli/pull/331.
* CLI/linux/build: add documentation
* debian/.gitignore: ignore generated files
* debian/TODO: unnecessary at this point
* debian/ooniprobe-cli.service: remove commented out lines
* debian/rules: remove unnecessary actions
* make: reindent and fix spelling
* smoketest.sh: don't run in verbose mode
Part of https://github.com/ooni/probe/issues/1466
1. reduce the number of periodic builds
2. just build as part of the release process in most cases
3. shorttests duplicates coverage
Preliminary changes as part of https://github.com/ooni/probe/issues/1466
Just consistency. There was no clear semantic regarding why
sometimes I was using __name and sometimes _name.
So prefer _name everywhere.
Occurred to me while working on https://github.com/ooni/probe/issues/1466.
After all the refactoring done so far, we can run checks directly
inside of `make`, because we have auto-cleanup, temporary environments
and we don't need wrapper scripts anymore.
Part of https://github.com/ooni/probe/issues/1466.
The previous fix allowed us to return values directly from
implementations of Engine. Here we take advantage of this
possibility by immediately refactoring how backticks work.
Part of https://github.com/ooni/probe/issues/1466.
