2021-02-03 14:42:51 +01:00
# Contributing to ooni/probe-cli
2021-02-02 12:05:47 +01:00
This is an open source project, and contributions are welcome! You are welcome
to open pull requests. An open pull request will be reviewed by a core
developer. The review may request you to apply changes. Once the assigned
reviewer is satisfied, they will merge the pull request.
2021-08-19 18:55:49 +02:00
## OONI Software Development Guidelines
2021-02-02 12:05:47 +01:00
2021-08-19 18:55:49 +02:00
Please, make sure you read [OONI Software Development Guidelines](
https://ooni.org/post/ooni-software-development-guidelines/). We try in
general to follow these guidelines when working on ooni/probe-cli. In
2022-04-28 01:34:17 +02:00
the unlikely case where those guidelines conflict with this document, this
2021-08-19 18:55:49 +02:00
document will take precedence.
2021-02-02 12:05:47 +01:00
2022-02-18 21:09:51 +01:00
## Golang Resources
We use golang as our primary language for the development of OONI Probe CLI and do check out the resources below, quite useful to read before contributing.
- [Effective Go ](https://go.dev/doc/effective_go )
- [Go Code Review Comments ](https://github.com/golang/go/wiki/CodeReviewComments )
- [Concurrency ](https://go.dev/blog/pipelines ) and [Data races ](https://go.dev/ref/mem )
2022-07-08 19:49:20 +02:00
- [Channels Axioms ](https://dave.cheney.net/2014/03/19/channel-axioms )
2022-02-18 21:09:51 +01:00
2021-08-19 18:55:49 +02:00
## Opening issues
2021-02-03 14:42:51 +01:00
2021-08-19 18:55:49 +02:00
Please, before opening a new issue, check whether the issue or feature request
you want us to consider has not already been reported by someone else. The
issue tracker is at [github.com/ooni/probe/issues ](https://github.com/ooni/probe/issues ).
2021-02-03 14:42:51 +01:00
2021-02-02 12:05:47 +01:00
## PR requirements
Every pull request that introduces new functionality should feature
comprehensive test coverage. Any pull request that modifies existing
functionality should pass existing tests. What's more, any new pull
request that modifies existing functionality should not decrease the
existing code coverage.
Long-running tests should be skipped when running tests in short mode
2021-08-19 18:55:49 +02:00
using `go test -short` . We prefer internal testing to external
2021-02-02 12:05:47 +01:00
testing. We generally have a file called `foo_test.go` with tests
for every `foo.go` file. Sometimes we separate long running
integration tests in a `foo_integration_test.go` file.
If there is a top-level DESIGN.md document, make sure such document is
kept in sync with code changes you have applied.
Do not submit large PRs. A reviewer can best service your PR if the
code changes are around 200-600 lines. (It is okay to add more changes
afterwards, if the reviewer asks you to do more work; the key point
here is that the PR should be reasonably sized when the review starts.)
In this vein, we'd rather structure a complex issue as a sequence of
small PRs, than have a single large PR address it all.
As a general rule, a PR is reviewed by reading the whole diff. Let us
know if you want us to read each diff individually, if you think that's
functional to better understand your changes.
## Code style requirements
Please, use `go fmt` , `go vet` , and `golint` to check your code
contribution before submitting a pull request. Make sure your code
is documented. At the minimum document all the exported symbols.
Make sure you commit `go.mod` and `go.sum` changes. Make sure you
run `go mod tidy` to minimize such changes.
refactor: flatten and separate (#353)
* refactor(atomicx): move outside the engine package
After merging probe-engine into probe-cli, my impression is that we have
too much unnecessary nesting of packages in this repository.
The idea of this commit and of a bunch of following commits will instead
be to reduce the nesting and simplify the structure.
While there, improve the documentation.
* fix: always use the atomicx package
For consistency, never use sync/atomic and always use ./internal/atomicx
so we can just grep and make sure we're not risking to crash if we make
a subtle mistake on a 32 bit platform.
While there, mention in the contributing guidelines that we want to
always prefer the ./internal/atomicx package over sync/atomic.
* fix(atomicx): remove unnecessary constructor
We don't need a constructor here. The default constructed `&Int64{}`
instance is already usable and the constructor does not add anything to
what we are doing, rather it just creates extra confusion.
* cleanup(atomicx): we are not using Float64
Because atomicx.Float64 is unused, we can safely zap it.
* cleanup(atomicx): simplify impl and improve tests
We can simplify the implementation by using defer and by letting
the Load() method call Add(0).
We can improve tests by making many goroutines updated the
atomic int64 value concurrently.
* refactor(fsx): can live in the ./internal pkg
Let us reduce the amount of nesting. While there, ensure that the
package only exports the bare minimum, and improve the documentation
of the tests, to ease reading the code.
* refactor: move runtimex to ./internal
* refactor: move shellx into the ./internal package
While there, remove unnecessary dependency between packages.
While there, specify in the contributing guidelines that
one should use x/sys/execabs instead of os/exec.
* refactor: move ooapi into the ./internal pkg
* refactor(humanize): move to ./internal and better docs
* refactor: move platform to ./internal
* refactor(randx): move to ./internal
* refactor(multierror): move into the ./internal pkg
* refactor(kvstore): all kvstores in ./internal
Rather than having part of the kvstore inside ./internal/engine/kvstore
and part in ./internal/engine/kvstore.go, let us put every piece of code
that is kvstore related into the ./internal/kvstore package.
* fix(kvstore): always return ErrNoSuchKey on Get() error
It should help to use the kvstore everywhere removing all the
copies that are lingering around the tree.
* sessionresolver: make KVStore mandatory
Simplifies implementation. While there, use the ./internal/kvstore
package rather than having our private implementation.
* fix(ooapi): use the ./internal/kvstore package
* fix(platform): better documentation
2021-06-04 10:34:18 +02:00
## Implementation requirements
2021-06-04 11:39:00 +02:00
- use `./internal/atomicx` rather than `atomic/sync`
2021-06-04 14:02:18 +02:00
- do not use `os/exec` , use `x/sys/execabs` or `./internal/shellx`
refactor: flatten and separate (#353)
* refactor(atomicx): move outside the engine package
After merging probe-engine into probe-cli, my impression is that we have
too much unnecessary nesting of packages in this repository.
The idea of this commit and of a bunch of following commits will instead
be to reduce the nesting and simplify the structure.
While there, improve the documentation.
* fix: always use the atomicx package
For consistency, never use sync/atomic and always use ./internal/atomicx
so we can just grep and make sure we're not risking to crash if we make
a subtle mistake on a 32 bit platform.
While there, mention in the contributing guidelines that we want to
always prefer the ./internal/atomicx package over sync/atomic.
* fix(atomicx): remove unnecessary constructor
We don't need a constructor here. The default constructed `&Int64{}`
instance is already usable and the constructor does not add anything to
what we are doing, rather it just creates extra confusion.
* cleanup(atomicx): we are not using Float64
Because atomicx.Float64 is unused, we can safely zap it.
* cleanup(atomicx): simplify impl and improve tests
We can simplify the implementation by using defer and by letting
the Load() method call Add(0).
We can improve tests by making many goroutines updated the
atomic int64 value concurrently.
* refactor(fsx): can live in the ./internal pkg
Let us reduce the amount of nesting. While there, ensure that the
package only exports the bare minimum, and improve the documentation
of the tests, to ease reading the code.
* refactor: move runtimex to ./internal
* refactor: move shellx into the ./internal package
While there, remove unnecessary dependency between packages.
While there, specify in the contributing guidelines that
one should use x/sys/execabs instead of os/exec.
* refactor: move ooapi into the ./internal pkg
* refactor(humanize): move to ./internal and better docs
* refactor: move platform to ./internal
* refactor(randx): move to ./internal
* refactor(multierror): move into the ./internal pkg
* refactor(kvstore): all kvstores in ./internal
Rather than having part of the kvstore inside ./internal/engine/kvstore
and part in ./internal/engine/kvstore.go, let us put every piece of code
that is kvstore related into the ./internal/kvstore package.
* fix(kvstore): always return ErrNoSuchKey on Get() error
It should help to use the kvstore everywhere removing all the
copies that are lingering around the tree.
* sessionresolver: make KVStore mandatory
Simplifies implementation. While there, use the ./internal/kvstore
package rather than having our private implementation.
* fix(ooapi): use the ./internal/kvstore package
* fix(platform): better documentation
2021-06-04 10:34:18 +02:00
2021-06-04 14:02:18 +02:00
- use `./internal/fsx.OpenFile` when you need to open a file
refactor: flatten and separate (#353)
* refactor(atomicx): move outside the engine package
After merging probe-engine into probe-cli, my impression is that we have
too much unnecessary nesting of packages in this repository.
The idea of this commit and of a bunch of following commits will instead
be to reduce the nesting and simplify the structure.
While there, improve the documentation.
* fix: always use the atomicx package
For consistency, never use sync/atomic and always use ./internal/atomicx
so we can just grep and make sure we're not risking to crash if we make
a subtle mistake on a 32 bit platform.
While there, mention in the contributing guidelines that we want to
always prefer the ./internal/atomicx package over sync/atomic.
* fix(atomicx): remove unnecessary constructor
We don't need a constructor here. The default constructed `&Int64{}`
instance is already usable and the constructor does not add anything to
what we are doing, rather it just creates extra confusion.
* cleanup(atomicx): we are not using Float64
Because atomicx.Float64 is unused, we can safely zap it.
* cleanup(atomicx): simplify impl and improve tests
We can simplify the implementation by using defer and by letting
the Load() method call Add(0).
We can improve tests by making many goroutines updated the
atomic int64 value concurrently.
* refactor(fsx): can live in the ./internal pkg
Let us reduce the amount of nesting. While there, ensure that the
package only exports the bare minimum, and improve the documentation
of the tests, to ease reading the code.
* refactor: move runtimex to ./internal
* refactor: move shellx into the ./internal package
While there, remove unnecessary dependency between packages.
While there, specify in the contributing guidelines that
one should use x/sys/execabs instead of os/exec.
* refactor: move ooapi into the ./internal pkg
* refactor(humanize): move to ./internal and better docs
* refactor: move platform to ./internal
* refactor(randx): move to ./internal
* refactor(multierror): move into the ./internal pkg
* refactor(kvstore): all kvstores in ./internal
Rather than having part of the kvstore inside ./internal/engine/kvstore
and part in ./internal/engine/kvstore.go, let us put every piece of code
that is kvstore related into the ./internal/kvstore package.
* fix(kvstore): always return ErrNoSuchKey on Get() error
It should help to use the kvstore everywhere removing all the
copies that are lingering around the tree.
* sessionresolver: make KVStore mandatory
Simplifies implementation. While there, use the ./internal/kvstore
package rather than having our private implementation.
* fix(ooapi): use the ./internal/kvstore package
* fix(platform): better documentation
2021-06-04 10:34:18 +02:00
2021-09-28 12:42:01 +02:00
- use `./internal/netxlite.ReadAllContext` instead of `io.ReadAll`
and `./internal/netxlite.CopyContext` instead of `io.Copy`
2021-06-15 11:57:40 +02:00
2022-02-16 20:47:44 +01:00
- use `./internal/model.ErrorToStringOrOK` when
an experiment logs intermediate results
2021-02-02 12:05:47 +01:00
## Code testing requirements
Make sure all tests pass with `go test -race ./...` run from the
2022-02-05 12:21:22 +01:00
top-level directory of this repository. (Integration tests may be
flaky, so there may be some failures here and and there; we know
in particular that `./internal/cmd/jafar` is one of the usual
suspects and that it's not super pleasant to test it under Linux.)
2021-02-02 12:05:47 +01:00
## Writing a new OONI experiment
When you are implementing a new experiment (aka nettest), make sure
you have read the relevant spec from the [ooni/spec](
https://github.com/ooni/spec) repository. If the spec is missing,
please help the pull request reviewer to create it. If the spec is
not clear, please let us know during the review.
2021-10-22 15:41:53 +02:00
To get a sense of what we expect from an experiment, see the [internal/tutorial](
https://github.com/ooni/probe-cli/tree/master/internal/tutorial) tutorial
## Branching and releasing
The following diagram illustrates the overall branching and releasing
strategy loosely followed by the core team. If you are an external
contributor, you generally only care about the development part, which
is on the left-hand side of the diagram.
Development uses the `master` branch. When we need to implement a
feature or fix a bug, we branch off of the `master` branch. We squash
and merge to include a feature or fix branch back into `master` .
We periodically tag `-alpha` releases directly on `master` . The
semantics of such releases is that we reached a point where we have
features we would like to test using the `miniooni` research CLI
client. As part of these releases, we also update dependencies and
embedded assets. This process ensures that we perform better testing
of dependencies and assets as part of development.
The `master` branch and pull requests only run CI lightweight tests
that ensure the code still compiles, has good coverage, and we are
not introducing regressions in terms of the measurement engine.
To draft a release we branch off of `master` and create a `release/x.y`
branch where `x` is the major number and `y` is the minor number. For
release branches, we enable a very comprehensive set of tests that run
automatically with every commit. The purpose of a release branch is to
make sure all checks are green and hotfix bugs that we may discover
as part of more extensively testing a release candidate. Beta and stable
releases should occur on this branch. Subsequent patch releases should
also occur on this branch. We have one such branch for each `x.y`
release. If there are fixes on `master` that we want to backport, we
cherry-pick them into the release branch. Likewise, if we need to
2021-11-05 13:08:10 +01:00
forward port fixes, we cherry-pick them into `master` . When we backport,
the commit message should start with `[backport]` ; when we forward
port, the commit message should start with `[forwardport]` .
2021-10-22 15:41:53 +02:00
When we branch off release `x.y` from `master` , we also need to bump
the `alpha` version used by `master` .
[forwardport] ci/cd: publish binaries onto a release when we create a tag (#609) (#611)
This diff forwardports 856e436e20d511a4f0d618546da7921fa9f8c5f6 to the master branch
Original commit message:
- - -
This pull request changes `mk` and github workflows to build and publish binaries on tag. We also update the documentation to explain this new branching model. Basically, we have release branches where we produce binary packages and we add extra code, on tag, to publish such packages inside a release.
We discussed removing most secrets from builds in this repository and having a different tool/repository that takes in input also secrets for doing follow-up actions after publishing. As a consequence, this pull request also removes all pieces of code that require secrets. The next step is to reinstate this code in this new repository/tool.
The existing code in `mk` also implemented caching. This feature was useful when doing local builds because it reduced the time required to obtain binary releases. With builds running as part of GitHub actions, we don't need caching because we spawn parallel machines to build binaries. Therefore, let us also remove caching, which makes the code simpler. (Caching in itself is hard and in https://github.com/ooni/probe/issues/1875 I noted that, for example, caching of the `ooni/go` repository was leading to some unwanted behaviour when changing the branch. Without caching, this behaviour is gone and we always generally use fresh information to produce builds.) Of course, this means that local builds are now slower, but I do not think this is a problem _because_ we want to use GitHub actions for building in the common case.
Reference issues: https://github.com/ooni/probe/issues/1879 and https://github.com/ooni/probe/issues/1875.
The final aspect to mention to conclude this description is an implementation one:
```
gh release create -p $tag --target $GITHUB_SHA || true
```
The code above uses `|| true` because there could already be a release. So, basically, it means that, if a release does not already exist, then we're going to create one. Otherwise, it does not matter because there's already a release.
2021-11-23 15:56:25 +01:00
We build binary packages for each tagged release. We will use external
tools for publishing binaries to our Debian repository, Maven Central, etc.
2022-02-18 21:09:51 +01:00
## Community Channels
2022-04-28 01:34:17 +02:00
Stuck somewhere or Have any questions? please join our [Slack Channels ](https://slack.ooni.org/ ) or [IRC ](ircs://irc.oftc.net:6697/#ooni ). We're here to help and always available to discuss.
