This diff forwardports 856e436e20d511a4f0d618546da7921fa9f8c5f6 to the master branch
Original commit message:
- - -
This pull request changes `mk` and github workflows to build and publish binaries on tag. We also update the documentation to explain this new branching model. Basically, we have release branches where we produce binary packages and we add extra code, on tag, to publish such packages inside a release.
We discussed removing most secrets from builds in this repository and having a different tool/repository that takes in input also secrets for doing follow-up actions after publishing. As a consequence, this pull request also removes all pieces of code that require secrets. The next step is to reinstate this code in this new repository/tool.
The existing code in `mk` also implemented caching. This feature was useful when doing local builds because it reduced the time required to obtain binary releases. With builds running as part of GitHub actions, we don't need caching because we spawn parallel machines to build binaries. Therefore, let us also remove caching, which makes the code simpler. (Caching in itself is hard and in https://github.com/ooni/probe/issues/1875 I noted that, for example, caching of the `ooni/go` repository was leading to some unwanted behaviour when changing the branch. Without caching, this behaviour is gone and we always generally use fresh information to produce builds.) Of course, this means that local builds are now slower, but I do not think this is a problem _because_ we want to use GitHub actions for building in the common case.
Reference issues: https://github.com/ooni/probe/issues/1879 and https://github.com/ooni/probe/issues/1875.
The final aspect to mention to conclude this description is an implementation one:
```
gh release create -p $tag --target $GITHUB_SHA || true
```
The code above uses `|| true` because there could already be a release. So, basically, it means that, if a release does not already exist, then we're going to create one. Otherwise, it does not matter because there's already a release.
This diff forward ports ea44e99451f345474738b9010ff791759a1f1367.
Original commit message:
- - -
This change allows for producing cloud builds using the psiphon
config files. We will add those files as build secrets. Only people
in the organization and collaborators with at least "write"
access could trigger builds containing such secrets.
Before this change, `./mk` unconditionally attempted to clone
github.com/ooni/probe-private. Now, it only checks whether
we need to clone _if_ files are not already there.
This allows us to use GitHub actions and secrets to copy the
files in there _without_ needing to clone a private repo.
Cloning a private repo would require us to include as repository
secret an access token with full `repo` scope, which is a very
broad scope. Instead, by using secrets to include psiphon config,
we are narrowing down the secrets required to make a release build.
See https://github.com/ooni/probe/issues/1878
This diff WILL require forward porting to the master branch.
This diff forward ports adcb0f9ae3b9e074c301d4f7f0e8f2d0ef6466b9.
Original commit message:
- - -
- ensure we use go1.17.3 in workflows
- update to a version of ooni/oohttp that uses go1.17.3
This change WILL need to be forward ported to master.
Closes https://github.com/ooni/probe/issues/1861
This pull request fixes https://github.com/ooni/probe/issues/1471. We have replaced the original build script (`./make`) with the `./mk` makefile (executable using `#!/usr/bin/make -f`). We concluded supporting direct builds from Windows is not worth the effort and halving the code we need to maintain is probably a good plus. Both macOS and Linux install GNU make at `/usr/bin/make`, so we should be okay in the common use cases.
I significantly simplified the management of Go versioning by requiring the user to manage it and by enforcing that we are using the desired Go version. This speeds up builds and works in sane operating systems that use the last version of a specific package. Otherwise, it's possible to use the `go get golang.org/dl/go${version}` feature.
The remaining question mark was related to updating the Android SDK. I have determined that a good course of action is pinning to the latest CLI tools and always forcing the CLI tools to install the latest required packages (e.g., the NDK).
1. reduce the number of periodic builds
2. just build as part of the release process in most cases
3. shorttests duplicates coverage
Preliminary changes as part of https://github.com/ooni/probe/issues/1466
Part of https://github.com/ooni/probe/issues/1440.
Basically, let us continue to update our build infrastructure so that we can release v3.10.0-beta.
Now, it's the turn of iOS.
* feat: use go1.16 embedding for resources
We want to embed everything that can be easily embedded. We should, at a
minimum, replace the downloading of resources and bindata.
Ref: https://github.com/ooni/probe/issues/1367.
* fix: get rid of bindata and use go embed instead
* fix: start unbreaking some automatic tests
* fix: fetch resources as part of the mobile build
* fix: convert more stuff to go1.16
I still expect many breakages, but we'll fix them.
* fix: make the windows CI green
* fix: get resources before running QA
* fix: go1.16 uses modules by default
* hopefully fix all other outstanding issues
* fix(QA/telegram.py): add another DC IP address
* Apply suggestions from code review
* refactor: start building an Android package
Part of https://github.com/ooni/probe/issues/1335.
This seems also a good moment to move some packages out of the
engine, e.g., oonimkall. This package, for example, is a consumer
of the engine, so it makes sense it's not _inside_ it.
* fix: committed some stuff I didn't need to commit
* fix: oonimkall needs to be public to build
The side effect is that we will probably need to bump the major
version number every time we change one of these APIs.
(We can also of course choose to violate the basic guidelines of Go
software, but I believe this is bad form.)
I have no problem in bumping the major quite frequently and in
any case this monorepo solution is convinving me more than continuing
to keep a split between engine and cli. The need to embed assets to
make the probe more reliable trumps the negative effects of having to
~frequently bump major because we expose a public API.
* fix: let's not forget about libooniffi
Honestly, I don't know what to do with this library. I added it
to provide a drop in replacement for MK but I have no idea whether
it's used and useful. I would not feel comfortable exposing it,
unlike oonimkall, since we're not using it.
It may be that the right thing to do here is just to delete the
package and reduce the amount of code we're maintaining?
* woops, we're still missing the publish android script
* fix(publish-android.bash): add proper API key
* ouch fix another place where the name changed
