diff options
Diffstat (limited to 'doc/development/pipelines.md')
| -rw-r--r-- | doc/development/pipelines.md | 216 |
1 files changed, 0 insertions, 216 deletions
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md deleted file mode 100644 index 448fb0f9f5a..00000000000 --- a/doc/development/pipelines.md +++ /dev/null @@ -1,216 +0,0 @@ -# Pipelines for the GitLab project - -Pipelines for `gitlab-org/gitlab` and `gitlab-org/gitlab-foss` (as well as the -`dev` instance's mirrors) are configured in the usual -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml) -which itself includes files under -[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/ci) -for easier maintenance. - -We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfooding) -GitLab [CI/CD features and best-practices](../ci/yaml/README.md) -as much as possible. - -## Stages - -The current stages are: - -- `prepare`: This stage includes jobs that prepare artifacts that are needed by - jobs in subsequent stages. -- `quick-test`: This stage includes test jobs that should run first and fail the - pipeline early (currently used to run Geo tests when the branch name starts - with `geo-`, `geo/`, or ends with `-geo`). -- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs. -- `review-prepare`: This stage includes a job that build the CNG images that are - later used by the (Helm) Review App deployment (see - [Review Apps](testing_guide/review_apps.md) for details). -- `review`: This stage includes jobs that deploy the GitLab and Docs Review Apps. -- `qa`: This stage includes jobs that perform QA tasks against the Review App - that is deployed in the previous stage. -- `post-test`: This stage includes jobs that build reports or gather data from - the previous stages' jobs (e.g. coverage, Knapsack metadata etc.). -- `pages`: This stage includes a job that deploys the various reports as - GitLab pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>, - <https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>, - <https://gitlab-org.gitlab.io/gitlab/webpack-report/>). - -## Default image - -The default image is currently -`dev.gitlab.org:5005/gitlab/gitlab-build-images:ruby-2.6.3-golang-1.11-git-2.22-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33`. -It includes Ruby 2.6.3, Go 1.11, Git 2.22, Chrome 73, Node 12, Yarn 1.16, -PostgreSQL 9.6, and Graphics Magick 1.3.33. - -The images used in our pipelines are configured in the -[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) -project, which is push-mirrored to <https://dev.gitlab.org/gitlab/gitlab-build-images> -for redundancy. - -The current version of the build images can be found in the -["Used by GitLab CE/EE section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml). - -## Default variables - -In addition to the [predefined variables](../ci/variables/predefined_variables.md), -each pipeline includes the following [variables](../ci/variables/README.md): - -- `RAILS_ENV: "test"` -- `NODE_ENV: "test"` -- `SIMPLECOV: "true"` -- `GIT_DEPTH: "20"` -- `GIT_SUBMODULE_STRATEGY: "none"` -- `GET_SOURCES_ATTEMPTS: "3"` -- `KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` -- `EE_KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/${CI_PROJECT_NAME}/rspec_report-master-ee.json` -- `FLAKY_RSPEC_SUITE_REPORT_PATH: rspec_flaky/report-suite.json` -- `BUILD_ASSETS_IMAGE: "false"` -- `ES_JAVA_OPTS: "-Xms256m -Xmx256m"` -- `ELASTIC_URL: "http://elastic:changeme@docker.elastic.co-elasticsearch-elasticsearch:9200"` - -## Common job definitions - -Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends) -that are scoped to a single -[configuration parameter](../ci/yaml/README.md#configuration-parameters). - -These common definitions are: - -- `.default-tags`: Ensures a job has the `gitlab-org` tag to ensure it's using - our dedicated runners. -- `.default-retry`: Allows a job to retry upon `unknown_failure`, `api_failure`, - `runner_system_failure`. -- `.default-before_script`: Allows a job to use a default `before_script` definition - suitable for Ruby/Rails tasks that may need a database running (e.g. tests). -- `.default-cache`: Allows a job to use a default `cache` definition suitable for - Ruby/Rails and frontend tasks. -- `.default-only`: Restricts the cases where a job is created. This currently - includes `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches), - `/^\d+-\d+-auto-deploy-\d+$/` (security branches), `merge_requests`, `tags`. - Note that jobs won't be created for branches with this default configuration. -- `.only-review`: Only creates a job for the `gitlab-org` namespace and if - Kubernetes integration is available. Also, prevents a job from being created - for `master` and auto-deploy branches. -- `.only-review-schedules`: Same as `.only-review` but also restrict a job to - only run for [schedules](../user/project/pipelines/schedules.md). -- `.use-pg`: Allows a job to use the `postgres:9.6.14` and `redis:alpine` services. -- `.use-pg-10`: Allows a job to use the `postgres:10.9` and `redis:alpine` services. -- `.only-ee`: Only creates a job for the `gitlab` project. - -## Changes detection - -If a job extends from `.default-only` (and most of the jobs should), it can restrict -the cases where it should be created -[based on the changes](../ci/yaml/README.md#onlychangesexceptchanges) -from a commit or MR by extending from the following CI definitions: - -- `.only-code-changes`: Allows a job to only be created upon code-related changes. -- `.only-qa-changes`: Allows a job to only be created upon QA-related changes. -- `.only-docs-changes`: Allows a job to only be created upon docs-related changes. -- `.only-code-qa-changes`: Allows a job to only be created upon code-related or QA-related changes. - -**See <https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml> -for the list of exact patterns.** - -## Directed acyclic graph - -We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to -execute jobs out of order for the following jobs: - -```mermaid -graph RL; - A[setup-test-env]; - B["gitlab:assets:compile<br/>(master only)"]; - C[gitlab:assets:compile pull-cache]; - D["cache gems<br/>(master and tags only)"]; - E[review-build-cng]; - F[build-qa-image]; - G[review-deploy]; - G2["schedule:review-deploy<br/>(master only)"]; - H[karma]; - I[jest]; - J["compile-assets<br/>(master only)"]; - K[compile-assets pull-cache]; - L[webpack-dev-server]; - M[coverage]; - N[pages]; - O[static-analysis]; - P["package-and-qa-manual:master<br/>(master schedule only)"]; - Q[package-and-qa]; - R[package-and-qa-manual]; - -subgraph "`prepare` stage" - A - F - J - K - end - -subgraph "`test` stage" - B --> |needs| A; - C --> |needs| A; - D --> |needs| A; - H -.-> |depends on| A; - H -.-> |depends on| J; - H -.-> |depends on| K; - I -.-> |depends on| A; - I -.-> |depends on| J; - I -.-> |depends on| K; - L -.-> |depends on| A; - L -.-> |depends on| J; - L -.-> |depends on| K; - downtime_check --> |needs and depends on| A; - db:* --> |needs| A; - gitlab:setup --> |needs| A; - O -.-> |depends on| A; - O -.-> |depends on| B; - O -.-> |depends on| C; - downtime_check --> |needs and depends on| A; - end - -subgraph "`review-prepare` stage" - E --> |needs| C; - X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| B; - end - -subgraph "`review` stage" - G --> |needs| E; - G2 --> |needs| E; - end - -subgraph "`qa` stage" - Q --> |needs| C; - Q --> |needs| F; - R --> |needs| C; - R --> |needs| F; - P --> |needs| B; - P --> |needs| F; - review-qa-smoke -.-> |depends on| G; - review-qa-all -.-> |depends on| G; - review-qa-performance -.-> |depends on| G; - X2["schedule:review-performance<br/>(master only)"] -.-> |depends on| G2; - dast -.-> |depends on| G; - end - -subgraph "`post-test` stage" - M - end - -subgraph "`pages` stage" - N -.-> |depends on| B; - N -.-> |depends on| H; - N -.-> |depends on| M; - end -``` - -## Test jobs - -Consult [GitLab tests in the Continuous Integration (CI) context](testing_guide/ci.md) -for more information. - -## Review app jobs - -Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. - ---- - -[Return to Development documentation](README.md) |