chore: merge probe-engine into probe-cli (#201)

This is how I did it:

1. `git clone https://github.com/ooni/probe-engine internal/engine`

2. ```
(cd internal/engine && git describe --tags)
v0.23.0
```

3. `nvim go.mod` (merging `go.mod` with `internal/engine/go.mod`

4. `rm -rf internal/.git internal/engine/go.{mod,sum}`

5. `git add internal/engine`

6. `find . -type f -name \*.go -exec sed -i 's@/ooni/probe-engine@/ooni/probe-cli/v3/internal/engine@g' {} \;`

7. `go build ./...` (passes)

8. `go test -race ./...` (temporary failure on RiseupVPN)

9. `go mod tidy`

10. this commit message

Once this piece of work is done, we can build a new version of `ooniprobe` that
is using `internal/engine` directly. We need to do more work to ensure all the
other functionality in `probe-engine` (e.g. making mobile packages) are still WAI.

Part of https://github.com/ooni/probe/issues/1335

internal/engine/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,28 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug, triage
+assignees: bassosimone
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**System information (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

internal/engine/.github/ISSUE_TEMPLATE/routine-sprint-releases.md

@@ -0,0 +1,29 @@
+---
+name: Routine sprint releases
+about: Bi-weekly releases of probe-engine, etc.
+title: ''
+labels: effort/M, priority/medium
+assignees: bassosimone
+
+---
+
+- [ ] psiphon: run ./update.bash
+- [ ] engine: update dependencies
+- [ ] engine: update internal/httpheader/useragent.go
+- [ ] engine: update version/version.go
+- [ ] engine: update resources/assets.go
+- [ ] engine: update bundled certs (using `go generate ./...`)
+- [ ] engine: make sure all workflows are green
+- [ ] engine: tag a new version
+- [ ] engine: update again version.go to be alpha
+- [ ] engine: create release at GitHub
+- [ ] engine: update mobile-staging branch to create oonimkall
+- [ ] cli: pin to latest engine
+- [ ] cli: update internal/version/version.go
+- [ ] cli: tag a new version
+- [ ] cli: update internal/version/version.go again to be alpha
+- [ ] android: pin to latest oonimkall
+- [ ] ios: pin to latest oonimkall
+- [ ] desktop: pin to latest cli
+- [ ] engine: create issue for next routine release
+- [ ] e2etesting: see whether we can remove legacy checks

internal/engine/.github/workflows/alltests.yml

@@ -0,0 +1,16 @@
+name: alltests
+on:
+  push:
+    branches:
+      - 'release/**'
+  schedule:
+    - cron: "0 7 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: go test -race -tags integration,shaping -coverprofile=probe-engine.cov ./...

internal/engine/.github/workflows/android.yml

@@ -0,0 +1,23 @@
+name: android
+on:
+  push:
+    branches:
+      - mobile-staging
+      - 'release/**'
+jobs:
+  test:
+    runs-on: macos-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: brew install --cask android-sdk
+      - run: echo y | sdkmanager --install "platforms;android-29"
+      - run: echo y | sdkmanager --install "ndk;21.3.6528147"
+      - run: ./build-android.bash
+        env:
+          ANDROID_HOME: /usr/local/Caskroom/android-sdk/4333796
+      - run: ./publish-android.bash
+        env:
+          BINTRAY_API_KEY: ${{ secrets.BINTRAY_API_KEY }}

internal/engine/.github/workflows/codeql-analysis.yml

@@ -0,0 +1,58 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ master, mobile-staging ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ master ]
+  schedule:
+    - cron: '35 10 * * 1'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'go' ]
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v1
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 https://git.io/JvXDl
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

internal/engine/.github/workflows/coverage.yml

@@ -0,0 +1,29 @@
+name: coverage
+on:
+  pull_request:
+  push:
+    branches:
+      - "master"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        go: [ "1.14" ]
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "${{ matrix.go }}"
+      - uses: actions/checkout@v2
+      - run: go test -short -race -tags shaping -coverprofile=probe-engine.cov ./...
+      - uses: shogo82148/actions-goveralls@v1
+        with:
+          path-to-profile: probe-engine.cov
+          parallel: true
+  finish:
+    needs: test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: shogo82148/actions-goveralls@v1
+        with:
+          parallel-finished: true

internal/engine/.github/workflows/generate.yml

@@ -0,0 +1,16 @@
+name: generate
+on:
+  push:
+    branches:
+      - 'release/**'
+  schedule:
+    - cron: "0 0 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: go generate ./...

internal/engine/.github/workflows/ios.yml

@@ -0,0 +1,18 @@
+name: ios
+on:
+  push:
+    branches:
+      - mobile-staging
+      - 'release/**'
+jobs:
+  test:
+    runs-on: macos-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./build-ios.bash
+      - run: ./publish-ios.bash
+        env:
+          BINTRAY_API_KEY: ${{ secrets.BINTRAY_API_KEY }}

internal/engine/.github/workflows/jafar.yml

@@ -0,0 +1,14 @@
+name: jafar
+on:
+  schedule:
+    - cron: "0 5 * * 3"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: go build -v ./cmd/jafar
+      - run: sudo ./testjafar.bash

internal/engine/.github/workflows/libooniffi.yml

@@ -0,0 +1,29 @@
+name: libooniffi
+on:
+  schedule:
+    - cron: "0 5 * * 3"
+jobs:
+  darwin:
+    runs-on: macos-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./libooniffi/buildtest.bash darwin
+  linux:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./libooniffi/buildtest.bash linux
+  windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: bash.exe ./libooniffi/buildtest.bash windows

internal/engine/.github/workflows/miniooni.yml

@@ -0,0 +1,47 @@
+name: miniooni
+on:
+  push:
+    branches:
+      - 'release/**'
+  schedule:
+    - cron: "0 0 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+
+      - run: ./build-cli.sh linux
+      - run: ./CLI/linux/amd64/miniooni --yes -nNi https://example.com web_connectivity
+      - uses: actions/upload-artifact@v1
+        with:
+          name: miniooni-linux-386
+          path: ./CLI/linux/386/miniooni
+      - uses: actions/upload-artifact@v1
+        with:
+          name: miniooni-linux-amd64
+          path: ./CLI/linux/amd64/miniooni
+      - uses: actions/upload-artifact@v1
+        with:
+          name: miniooni-linux-armv7
+          path: ./CLI/linux/armv7/miniooni
+      - uses: actions/upload-artifact@v1
+        with:
+          name: miniooni-linux-arm64
+          path: ./CLI/linux/arm64/miniooni
+
+      - run: ./build-cli.sh darwin
+      - uses: actions/upload-artifact@v1
+        with:
+          name: miniooni-darwin-amd64
+          path: ./CLI/darwin/amd64/miniooni
+
+      - run: sudo apt install --yes mingw-w64
+      - run: ./build-cli.sh windows
+      - uses: actions/upload-artifact@v1
+        with:
+          name: miniooni-windows-amd64.exe
+          path: ./CLI/windows/amd64/miniooni.exe

internal/engine/.github/workflows/qafbmessenger.yml

@@ -0,0 +1,16 @@
+name: "qafbmessenger"
+on:
+  push:
+    branches:
+      - 'release/**'
+  schedule:
+    - cron: "0 3 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./QA/rundocker.bash "fbmessenger"

internal/engine/.github/workflows/qahhfm.yml

@@ -0,0 +1,13 @@
+name: "qahhfm"
+on:
+  schedule:
+    - cron: "5 3 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./QA/rundocker.bash "hhfm"

internal/engine/.github/workflows/qahirl.yml

@@ -0,0 +1,13 @@
+name: "qahirl"
+on:
+  schedule:
+    - cron: "10 3 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./QA/rundocker.bash "hirl"

internal/engine/.github/workflows/qatelegram.yml

@@ -0,0 +1,13 @@
+name: "qatelegram"
+on:
+  schedule:
+    - cron: "15 3 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./QA/rundocker.bash "telegram"

internal/engine/.github/workflows/qawebconnectivity.yml

@@ -0,0 +1,16 @@
+name: "qawebconnectivity"
+on:
+  push:
+    branches:
+      - 'release/**'
+  schedule:
+    - cron: "20 3 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./QA/rundocker.bash "webconnectivity"

internal/engine/.github/workflows/qawhatsapp.yml

@@ -0,0 +1,13 @@
+name: "qawhatsapp"
+on:
+  schedule:
+    - cron: "25 3 * * */1"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./QA/rundocker.bash "whatsapp"

internal/engine/.github/workflows/shorttests.yml

@@ -0,0 +1,18 @@
+name: shorttests
+on:
+  pull_request:
+  push:
+    branches:
+      - "master"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        go: [ "1.14", "1.15" ]
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "${{ matrix.go }}"
+      - uses: actions/checkout@v2
+      - run: go test -short -race -tags shaping ./...

internal/engine/.github/workflows/using.yml

@@ -0,0 +1,13 @@
+name: using
+on:
+  schedule:
+    - cron: "0 5 * * 3"
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v1
+        with:
+          go-version: "1.14"
+      - uses: actions/checkout@v2
+      - run: ./testusing.bash