Closes https://github.com/ooni/probe/issues/2334.
While there, reinstate integration tests, which were also lost in a previous refactoring. However, only run those tests for linux/amd64 because we can be confident that the Go compiler is WAI for all archs we support.
While there, always use bash for running end-to-end tests.
H/T @ainghazal for discovering and reporting this bug.
This work aims to make Linux builds faster to make https://github.com/ooni/probe/issues/2249 more convenient. Since those builds runs inside Docker, the problem to solve here is to save/restore the Go caches notwithstanding Docker. Because Docker runs as root, we need to modify the build a bit to run as a normal user. Otherwise, we will not be able to save the Go cache using actions/cache@v3. (Other approaches such as using `sudo` are possible but running the build as an unprivileged user actually looks cleaner, so I chose to do that.) While there, add a `.editorconfig`.
This diff modifies all the github actions that produce assets to
publish on a release called rolling when we are not building a tag.
If everything goes as planned, we should be able to provide
people with automatically generated fresh binaries for testing.
While there, introduce caching for all builds to make them
as fast as possible. I suspect gomobile based builds will not
see any speed up but other builds most likely will.
See https://github.com/ooni/probe/issues/2249
Part of https://github.com/ooni/probe/issues/1753.
While there, introduce a rule by which, if the branch is named `fullbuild` we run all possible builds. It helps to test all the builds without creating a release branch. Because release branches are protected, they cannot be deleted easily. On the contrary, the `fullbuild` branch can easily be disposed of.
While there, document what we need to document without mentioning too
many details about release builds, for which we have the Makefile.
See https://github.com/ooni/probe/issues/2218.
I am not 100% sure I was able to fix all the cases in which we
need higher permissions than the strict default.
At least, I tried.
It may be reasonable to make an interim release to check whether I
successfully fixed all the cases.
Ref issue: https://github.com/ooni/probe/issues/2154
This diff forwardports 856e436e20d511a4f0d618546da7921fa9f8c5f6 to the master branch
Original commit message:
- - -
This pull request changes `mk` and github workflows to build and publish binaries on tag. We also update the documentation to explain this new branching model. Basically, we have release branches where we produce binary packages and we add extra code, on tag, to publish such packages inside a release.
We discussed removing most secrets from builds in this repository and having a different tool/repository that takes in input also secrets for doing follow-up actions after publishing. As a consequence, this pull request also removes all pieces of code that require secrets. The next step is to reinstate this code in this new repository/tool.
The existing code in `mk` also implemented caching. This feature was useful when doing local builds because it reduced the time required to obtain binary releases. With builds running as part of GitHub actions, we don't need caching because we spawn parallel machines to build binaries. Therefore, let us also remove caching, which makes the code simpler. (Caching in itself is hard and in https://github.com/ooni/probe/issues/1875 I noted that, for example, caching of the `ooni/go` repository was leading to some unwanted behaviour when changing the branch. Without caching, this behaviour is gone and we always generally use fresh information to produce builds.) Of course, this means that local builds are now slower, but I do not think this is a problem _because_ we want to use GitHub actions for building in the common case.
Reference issues: https://github.com/ooni/probe/issues/1879 and https://github.com/ooni/probe/issues/1875.
The final aspect to mention to conclude this description is an implementation one:
```
gh release create -p $tag --target $GITHUB_SHA || true
```
The code above uses `|| true` because there could already be a release. So, basically, it means that, if a release does not already exist, then we're going to create one. Otherwise, it does not matter because there's already a release.
This diff forward ports ea44e99451f345474738b9010ff791759a1f1367.
Original commit message:
- - -
This change allows for producing cloud builds using the psiphon
config files. We will add those files as build secrets. Only people
in the organization and collaborators with at least "write"
access could trigger builds containing such secrets.
Before this change, `./mk` unconditionally attempted to clone
github.com/ooni/probe-private. Now, it only checks whether
we need to clone _if_ files are not already there.
This allows us to use GitHub actions and secrets to copy the
files in there _without_ needing to clone a private repo.
Cloning a private repo would require us to include as repository
secret an access token with full `repo` scope, which is a very
broad scope. Instead, by using secrets to include psiphon config,
we are narrowing down the secrets required to make a release build.
See https://github.com/ooni/probe/issues/1878
This diff WILL require forward porting to the master branch.
In https://github.com/ooni/probe/issues/1741, we observed that
every attempt to use `docker --platform` along with `debian` for
packaging ooniprobe fails with `SEGFAULT`, except when using
the `debian:oldstable` container.
To fix this issue, in this diff we fix Debian packaging to run on
any debian system (`debian:stable` in our case) provided that we
have `qemu-user-static` installed on the system and the system is
a Debian (or Debian-derived) system.
The trick here is to use `dpkg-buildpackage -a $deb_arch`. We
also need to disable a few `debian/rules` that we don't actually
need anyway.
Closes https://github.com/ooni/probe/issues/1741.
This cherry-picks 36a5bf34f99f382a081efd642dd472888a57602b
from the stable branch into the master branch.
We are mostly good to declare a stable release. We still need to deal with https://github.com/ooni/probe/issues/1484.
In this PR, we fix the aforementioned issue. These are the changes:
1. we remove the vendored `debops-ci`, and we pull it directly from `ooni/sysadmin`
2. we introduce a new script, `./CLI/linux/pubdebian`, to publish packages
3. we modify `./mk` to allow for publishing debian packages built outside of CI
The latter point has been quite useful in debugging what was wrong.
1. we can merge the e2eminiooni.yml test into the miniooni.yml test
so to reduce the number of tests we run;
2. ideally we would like the smoketest.sh test to evolve and also
check whether we can fetch the measurements we submitted, so start
moving this script into the `./E2E` folder, add a note saying we
would like to do that, and direct all the tests to run this script
at its new location and with its new name (`ooniprobe.sh`).
With these two changes, it's fine to remove the ooniprobe2debian.yml
test in ooni/e2etesting because we're moving its functionality to this
repository. (We mentioned the need to do this move in a previous TODO
comment at the top of such a script.)
Work part of https://github.com/ooni/probe/issues/1468
This pull request fixes https://github.com/ooni/probe/issues/1471. We have replaced the original build script (`./make`) with the `./mk` makefile (executable using `#!/usr/bin/make -f`). We concluded supporting direct builds from Windows is not worth the effort and halving the code we need to maintain is probably a good plus. Both macOS and Linux install GNU make at `/usr/bin/make`, so we should be okay in the common use cases.
I significantly simplified the management of Go versioning by requiring the user to manage it and by enforcing that we are using the desired Go version. This speeds up builds and works in sane operating systems that use the last version of a specific package. Otherwise, it's possible to use the `go get golang.org/dl/go${version}` feature.
The remaining question mark was related to updating the Android SDK. I have determined that a good course of action is pinning to the latest CLI tools and always forcing the CLI tools to install the latest required packages (e.g., the NDK).
We're still working on https://github.com/ooni/probe/issues/1466. The idea here is to teach the GH action for Linux to publish the debian package for arm64. When this is done, we can cleanup legacy build scripts and GH actions, because there is no remaining use case for them: we now build everything using the `./make` tool.
Part of https://github.com/ooni/probe/issues/1466. We're building both `arm64` and `amd64`. We are still not publishing `arm64` packages, which is what is asked in the original issue, but we're really close to doing that.
* feat: use go1.16 embedding for resources
We want to embed everything that can be easily embedded. We should, at a
minimum, replace the downloading of resources and bindata.
Ref: https://github.com/ooni/probe/issues/1367.
* fix: get rid of bindata and use go embed instead
* fix: start unbreaking some automatic tests
* fix: fetch resources as part of the mobile build
* fix: convert more stuff to go1.16
I still expect many breakages, but we'll fix them.
* fix: make the windows CI green
* fix: get resources before running QA
* fix: go1.16 uses modules by default
* hopefully fix all other outstanding issues
* fix(QA/telegram.py): add another DC IP address
* Apply suggestions from code review
* Use ooni/probe-engine 0.16.0
* Update all the other dependencies
* Use GitHub Actions rather than Travis CI
* Automatically build and test binaries on the target OS (for Windows, macOS, Linux on amd64)
* Make sure we correctly measure coverage
* Make sure we use `-race` when running tests
* Remove unnecessary scripts
* Make sure the README is up-to-date
* Write small script to update binary data and add GitHub Actions checks for it
* Notice that we needed to run ./updatebindata.sh and run it
* Self documenting instructions regarding cross compiling
* Set version number to v3.0.7-beta
Part of https://github.com/ooni/probe-engine/issues/748
