summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorKevin Burke <kevin@burke.dev>2021-11-15 21:00:26 -0800
committerDaniel Stenberg <daniel@haxx.se>2021-11-23 09:38:44 +0100
commitcc2870e275666ef7e4a6c27fa8b99938c6d1ff2f (patch)
tree677ea94a2f65217b5286f2736ec9e1a0664ba26e
parent9108da2c26d18e927b91e33d3729d9cf0f3eb8fa (diff)
downloadcurl-cc2870e275666ef7e4a6c27fa8b99938c6d1ff2f.tar.gz
tests/CI.md: add more information on CI environments
Fixes #8012 Closes #8022
-rw-r--r--.circleci/config.yml2
-rw-r--r--docs/CONTRIBUTE.md38
-rw-r--r--tests/CI.md127
-rw-r--r--zuul.d/jobs.yaml5
4 files changed, 137 insertions, 35 deletions
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 92112c8ea..3c0ad67c8 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -20,6 +20,8 @@
#
###########################################################################
+# View these jobs in the browser: https://app.circleci.com/pipelines/github/curl/curl
+
# Use the latest 2.1 version of CircleCI pipeline process engine. See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1
diff --git a/docs/CONTRIBUTE.md b/docs/CONTRIBUTE.md
index b8254088f..3011ecd7b 100644
--- a/docs/CONTRIBUTE.md
+++ b/docs/CONTRIBUTE.md
@@ -154,41 +154,9 @@ git commit that is easy to merge and they are easy to track and not that easy
to lose in the flood of many emails, like they sometimes do on the mailing
lists.
-Every pull request submitted will automatically be tested in several different
-ways. Every pull request is verified for each of the following:
-
- - ... it still builds, warning-free, on Linux and macOS, with both
- clang and gcc
- - ... it still builds fine on Windows with several MSVC versions
- - ... it still builds with cmake on Linux, with gcc and clang
- - ... it follows rudimentary code style rules
- - ... the test suite still runs 100% fine
- - ... the release tarball (the "dist") still works
- - ... it builds fine in-tree as well as out-of-tree
- - ... code coverage does not shrink drastically
-
-If the pull-request fails one of these tests, it will show up as a red X and
-you are expected to fix the problem. If you do not understand when the issue is
-or have other problems to fix the complaint, just ask and other project
-members will likely be able to help out.
-
-Consider the following table while looking at pull request failures:
-
- | CI platform as shown in PR | State | What to look at next |
- | ----------------------------------- | ------ | -------------------------- |
- | CI / codeql | stable | quality check results |
- | CI / fuzzing | stable | fuzzing results |
- | CI / macos ... | stable | all errors and failures |
- | Code scanning results / CodeQL | stable | quality check results |
- | FreeBSD FreeBSD: ... | stable | all errors and failures |
- | LGTM analysis: Python | stable | new findings |
- | LGTM analysis: C/C++ | stable | new findings |
- | buildbot/curl_winssl_ ... | stable | all errors and failures |
- | continuous-integration/appveyor/pr | stable | all errors and failures |
- | curl.curl (linux ...) | stable | all errors and failures |
- | curl.curl (windows ...) | flaky | repetitive errors/failures |
- | deepcode-ci-bot | stable | new findings |
- | musedev | stable | new findings |
+Every pull request submitted will automatically be
+tested in several different ways. [See CI.md for more
+information](https://github.com/curl/curl/blob/master/tests/CI.md).
Sometimes the tests fail due to a dependency service temporarily being offline
or otherwise unavailable, eg. package downloads. In this case you can just
diff --git a/tests/CI.md b/tests/CI.md
new file mode 100644
index 000000000..5b2ab896c
--- /dev/null
+++ b/tests/CI.md
@@ -0,0 +1,127 @@
+# Continuous Integration for curl
+
+Curl runs in many different environments, so every change is run against a large
+number of test suites.
+
+Every pull request is verified for each of the following:
+
+ - ... it still builds, warning-free, on Linux and macOS, with both
+ clang and gcc
+ - ... it still builds fine on Windows with several MSVC versions
+ - ... it still builds with cmake on Linux, with gcc and clang
+ - ... it follows rudimentary code style rules
+ - ... the test suite still runs 100% fine
+ - ... the release tarball (the "dist") still works
+ - ... it builds fine in-tree as well as out-of-tree
+ - ... code coverage does not shrink drastically
+ - ... different TLS backends still compile and pass tests
+
+If the pull-request fails one of these tests, it will show up as a red X and
+you are expected to fix the problem. If you do not understand when the issue is
+or have other problems to fix the complaint, just ask and other project
+members will likely be able to help out.
+
+Consider the following table while looking at pull request failures:
+
+ | CI platform as shown in PR | State | What to look at next |
+ | ----------------------------------- | ------ | -------------------------- |
+ | CI / codeql | stable | quality check results |
+ | CI / fuzzing | stable | fuzzing results |
+ | CI / macos ... | stable | all errors and failures |
+ | Code scanning results / CodeQL | stable | quality check results |
+ | FreeBSD FreeBSD: ... | stable | all errors and failures |
+ | LGTM analysis: Python | stable | new findings |
+ | LGTM analysis: C/C++ | stable | new findings |
+ | buildbot/curl_winssl_ ... | stable | all errors and failures |
+ | continuous-integration/appveyor/pr | stable | all errors and failures |
+ | curl.curl (linux ...) | stable | all errors and failures |
+ | curl.curl (windows ...) | flaky | repetitive errors/failures |
+ | deepcode-ci-bot | stable | new findings |
+ | musedev | stable | new findings |
+
+Sometimes the tests fail due to a dependency service temporarily being offline
+or otherwise unavailable, eg. package downloads. In this case you can just
+try to update your pull requests to rerun the tests later as described below.
+
+## CI servers
+
+Here are the different CI environments that are currently in use, and how they
+are configured:
+
+### Github Actions
+
+Github Actions runs the following tests:
+
+- Mac OS tests with a variety of different compilation options
+- Fuzz tests ([see tests/fuzz/README for
+ more info](https://github.com/curl/curl/blob/master/tests/fuzz/README)).
+- Curl compiled using the Rust TLS backend with Hyper
+- CodeQL static analysis
+
+These are each configured in different files in `.github/workflows`.
+
+### Azure
+
+The following tests are run in Microsoft Azure CI environment:
+
+- Ubuntu tests with a variety of different compilation options.
+- Windows tests with a variety of different compilation options.
+
+These are all configured in `.azure-pipelines.yml`.
+
+As of November 2021 @bagder and @mback2k are the only people with administrator
+access to the Azure CI environment. Additional admins/group members can be added
+on request.
+
+### Appveyor
+
+Appveyor runs a variety of different Windows builds, with different compilation
+options.
+
+As of November 2021 @bagder, @mback2k, @jay, @vszakats, @dfandrich and
+@danielgustafsson have administrator access to the Appveyor CI environment.
+Additional admins/group members can be added on request.
+
+The tests are configured in appveyor.yml.
+
+### Zuul
+
+[Zuul](https://zuul-ci.org/) is an open source CI tool. A number of Curl tests
+are run at [curl.zuul.vexxhost.dev](https://curl.zuul.vexxhost.dev/builds):
+
+- Source code is formatted according to expectations (`make checksrc`).
+- Curl compiled with a number of different TLS configurations (WolfSSL, rustls,
+BoringSSL, etc).
+- Curl compiled with different C compilers.
+
+As of November 2021, the tests run (sometimes) but do not run consistently and
+do not report results to the Github checks runner - you need to manually check
+for failures. See [#7522](https://github.com/curl/curl/issues/7522) for more
+information.
+
+As of November 2021 Daniel Stenberg is the only person with administrator access
+to the Zuul CI environment.
+
+These are configured in `zuul.d` and have test runners in `scripts/zuul`.
+
+### CircleCI
+
+CircleCI runs a basic Linux test suite on Ubuntu for both x86 and ARM
+processors. This is configured in `.circleci/config.yml`.
+
+You can [view the full list of CI jobs on CircleCI's
+website](https://app.circleci.com/pipelines/github/curl/curl).
+
+@bagder has access to edit the "Project Settings" on that page.
+Additional admins/group members can be added on request.
+
+### Cirrus CI
+
+Cirrus CI runs a basic test suite on FreeBSD and Windows. This is configured in
+`.cirrus.yml`.
+
+You can [view the full list of CI jobs on Cirrus CI's
+website](https://cirrus-ci.com/github/curl/curl).
+
+@bagder has access to edit the "Project Settings" on that page.
+Additional admins/group members can be added on request.
diff --git a/zuul.d/jobs.yaml b/zuul.d/jobs.yaml
index 5fc36b9fe..d4c25426f 100644
--- a/zuul.d/jobs.yaml
+++ b/zuul.d/jobs.yaml
@@ -19,6 +19,11 @@
# KIND, either express or implied.
#
###########################################################################
+
+# The results of each of these jobs can be found at
+# https://curl.zuul.vexxhost.dev/builds. As of November 2021 they are not being
+# propagated to the Github "checks" UI, you need to check for them manually.
+
---
- job:
name: curl-base