summaryrefslogtreecommitdiff
path: root/doc/development
diff options
context:
space:
mode:
Diffstat (limited to 'doc/development')
-rw-r--r--doc/development/README.md1
-rw-r--r--doc/development/api_styleguide.md13
-rw-r--r--doc/development/architecture.md8
-rw-r--r--doc/development/changelog.md10
-rw-r--r--doc/development/documentation/index.md133
-rw-r--r--doc/development/elasticsearch.md2
-rw-r--r--doc/development/packages.md6
-rw-r--r--doc/development/pipelines.md216
-rw-r--r--doc/development/polling.md4
-rw-r--r--doc/development/testing_guide/end_to_end/index.md18
-rw-r--r--doc/development/testing_guide/end_to_end/style_guide.md38
-rw-r--r--doc/development/testing_guide/img/k9s.pngbin364038 -> 0 bytes
-rw-r--r--doc/development/testing_guide/review_apps.md46
13 files changed, 161 insertions, 334 deletions
diff --git a/doc/development/README.md b/doc/development/README.md
index e8bebc79124..bbe73570f49 100644
--- a/doc/development/README.md
+++ b/doc/development/README.md
@@ -18,7 +18,6 @@ description: 'Learn how to contribute to GitLab.'
- [Generate a changelog entry with `bin/changelog`](changelog.md)
- [Code review guidelines](code_review.md) for reviewing code and having code reviewed
- [Database review guidelines](database_review.md) for reviewing database-related changes and complex SQL queries
-- [Pipelines for the GitLab project](pipelines.md)
- [Automatic CE->EE merge](automatic_ce_ee_merge.md)
- [Guidelines for implementing Enterprise Edition features](ee_features.md)
- [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer)
diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md
index 3f81440791e..61576236c96 100644
--- a/doc/development/api_styleguide.md
+++ b/doc/development/api_styleguide.md
@@ -92,18 +92,5 @@ For instance:
Model.create(foo: params[:foo])
```
-## Using API path helpers in GitLab Rails codebase
-
-Because we support [installing GitLab under a relative URL], one must take this
-into account when using API path helpers generated by Grape. Any such API path
-helper usage must be in wrapped into the `expose_path` helper call.
-
-For instance:
-
-```haml
-- endpoint = expose_path(api_v4_projects_issues_related_merge_requests_path(id: @project.id, issue_iid: @issue.iid))
-```
-
[Entity]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/api/entities.rb
[validation, and coercion of the parameters]: https://github.com/ruby-grape/grape#parameter-validation-and-coercion
-[installing GitLab under a relative URL]: https://docs.gitlab.com/ee/install/relative_url.html
diff --git a/doc/development/architecture.md b/doc/development/architecture.md
index e30b9d27f46..147bd21e6c7 100644
--- a/doc/development/architecture.md
+++ b/doc/development/architecture.md
@@ -134,7 +134,7 @@ Component statuses are linked to configuration documentation for each component.
| [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [✅][workhorse-omnibus] | [✅][workhorse-charts] | [✅][workhorse-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][workhorse-source] | ✅ | CE & EE |
| [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [✅][shell-omnibus] | [✅][shell-charts] | [✅][shell-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][shell-source] | [✅][gitlab-yml] | CE & EE |
| [GitLab Pages](#gitlab-pages) | Hosts static websites | [⚙][pages-omnibus] | [❌][pages-charts] | [❌][pages-charts] | [✅](../user/gitlab_com/index.md#gitlab-pages) | [⚙][pages-source] | [⚙][pages-gdk] | CE & EE |
-| [Registry](#registry) | Container registry, allows pushing and pulling of images | [⚙][registry-omnibus] | [✅][registry-charts] | [✅][registry-charts] | [✅](../user/packages/container_registry/index.md#build-and-push-images) | [⤓][registry-source] | [⚙][registry-gdk] | CE & EE |
+| [Registry](#registry) | Container registry, allows pushing and pulling of images | [⚙][registry-omnibus] | [✅][registry-charts] | [✅][registry-charts] | [✅](../user/project/container_registry.md#build-and-push-images) | [⤓][registry-source] | [⚙][registry-gdk] | CE & EE |
| [Redis](#redis) | Caching service | [✅][redis-omnibus] | [✅][redis-omnibus] | [✅][redis-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⤓][redis-source] | ✅ | CE & EE |
| [PostgreSQL](#postgresql) | Database | [✅][postgres-omnibus] | [✅][postgres-charts] | [✅][postgres-charts] | [✅](../user/gitlab_com/index.md#postgresql) | [⤓][postgres-source] | ✅ | CE & EE |
| [PgBouncer](#pgbouncer) | Database connection pooling, failover | [⚙][pgbouncer-omnibus] | [❌][pgbouncer-charts] | [❌][pgbouncer-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#database-architecture) | ❌ | ❌ | EE Only |
@@ -142,7 +142,7 @@ Component statuses are linked to configuration documentation for each component.
| [GitLab self-monitoring: Prometheus](#prometheus) | Time-series database, metrics collection, and query service | [✅][prometheus-omnibus] | [✅][prometheus-charts] | [⚙][prometheus-charts] | [✅](../user/gitlab_com/index.md#prometheus) | ❌ | ❌ | CE & EE |
| [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | [⚙][alertmanager-omnibus] | [✅][alertmanager-charts] | [⚙][alertmanager-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | [✅][grafana-omnibus] | [⤓][grafana-charts] | [⤓][grafana-charts] | [✅](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | ❌ | ❌ | CE & EE |
-| [GitLab self-monitoring: Sentry](#sentry) | Track errors generated by the GitLab instance | [⤓][sentry-omnibus] | [❌][sentry-charts] | [❌][sentry-charts] | [✅](https://about.gitlab.com/handbook/support/workflows/500_errors.html#searching-sentry) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE |
+| [GitLab self-monitoring: Sentry](#sentry) | Track errors generated by the GitLab instance | [⤓][sentry-omnibus] | [❌][sentry-charts] | [❌][sentry-charts] | [✅](https://about.gitlab.com/handbook/support/workflows/services/gitlab_com/500_errors.html#searching-sentry) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE |
| [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | [❌][jaeger-omnibus] | [❌][jaeger-charts] | [❌][jaeger-charts] | [❌](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | [⤓][jaeger-source] | [⚙][jaeger-gdk] | CE & EE |
| [Redis Exporter](#redis-exporter) | Prometheus endpoint with Redis metrics | [✅][redis-exporter-omnibus] | [✅][redis-exporter-charts] | [✅][redis-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
| [Postgres Exporter](#postgres-exporter) | Prometheus endpoint with PostgreSQL metrics | [✅][postgres-exporter-omnibus] | [✅][postgres-exporter-charts] | [✅][postgres-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
@@ -654,9 +654,9 @@ We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/ha
[pages-charts]: https://gitlab.com/gitlab-org/charts/gitlab/issues/37
[pages-source]: ../install/installation.md#install-gitlab-pages
[pages-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/pages.md
-[registry-omnibus]: ../administration/packages/container_registry.md#container-registry-domain-configuration
+[registry-omnibus]: ../administration/container_registry.md#container-registry-domain-configuration
[registry-charts]: https://docs.gitlab.com/charts/charts/registry/
-[registry-source]: ../administration/packages/container_registry.md#enable-the-container-registry
+[registry-source]: ../administration/container_registry.md#enable-the-container-registry
[registry-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/registry.md
[redis-omnibus]: https://docs.gitlab.com/omnibus/settings/redis.html
[redis-charts]: https://docs.gitlab.com/charts/charts/redis/
diff --git a/doc/development/changelog.md b/doc/development/changelog.md
index afc18885a80..5667f58b0c3 100644
--- a/doc/development/changelog.md
+++ b/doc/development/changelog.md
@@ -102,13 +102,6 @@ Its simplest usage is to provide the value for `title`:
bin/changelog 'Hey DZ, I added a feature to GitLab!'
```
-If you want to generate a changelog entry for GitLab EE, you will need to pass
-the `--ee` option:
-
-```text
-bin/changelog --ee 'Hey DZ, I added a feature to GitLab!'
-```
-
At this point the script would ask you to select the category of the change (mapped to the `type` field in the entry):
```text
@@ -138,6 +131,9 @@ author:
type:
```
+If you're working on the GitLab EE repository, the entry will be added to
+`ee/changelogs/unreleased/` instead.
+
### Arguments
| Argument | Shorthand | Purpose |
diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md
index 87892722d5e..c267669aa79 100644
--- a/doc/development/documentation/index.md
+++ b/doc/development/documentation/index.md
@@ -18,7 +18,7 @@ In addition to this page, the following resources can help you craft and contrib
## Source files and rendered web locations
-Documentation for GitLab, GitLab Runner, and Omnibus is published to [docs.gitlab.com](https://docs.gitlab.com). Documentation for GitLab is also published within the application at `/help` on the domain of the GitLab instance.
+Documentation for GitLab Community Edition (CE), Enterprise Edition (EE), GitLab Runner, and Omnibus is published to [docs.gitlab.com](https://docs.gitlab.com). Documentation for CE and EE is also published within the application at `/help` on the domain of the GitLab instance.
At `/help`, only help for your current edition and version is included. Help for other versions is available at docs.gitlab.com.
@@ -26,7 +26,8 @@ The source of the documentation exists within the codebase of each GitLab applic
| Project | Path |
| --- | --- |
-| [GitLab](https://gitlab.com/gitlab-org/gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab/tree/master/doc) |
+| [GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-ce/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc) |
+| [GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab-ee/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc) |
| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/) | [`/docs`](https://gitlab.com/gitlab-org/gitlab-runner/tree/master/docs) |
| [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc) |
@@ -51,6 +52,37 @@ Adhere to the [Documentation Style Guide](styleguide.md). If a style standard is
See the [Structure](styleguide.md#structure) section of the [Documentation Style Guide](styleguide.md).
+## Single codebase
+
+We maintain two sets of docs: one in the
+[gitlab-ce](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc) repo and
+one in [gitlab-ee](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc).
+These are identical, but they are different repositories.
+
+### CE first
+
+All merge requests for documentation must be submitted to CE. This means that:
+
+- For **EE-only docs changes**, you only have to submit an MR in the CE project.
+- For **EE-only features** that touch both the code and the docs, you have to submit
+ an EE MR containing all code changes, and a CE MR containing only the docs changes
+ and without a changelog entry.
+
+This might seem like a duplicate effort, but it's only for the short term.
+
+Since the CE and EE docs are combined, it's crucial to add the relevant
+[product badges](styleguide.md#product-badges) for all EE documentation, so that
+we can determine which features belong to which tier.
+
+### EE specific lines check
+
+There's a special test in place
+([`ee_specific_check.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/ee_specific_check/ee_specific_check.rb)),
+which checks and prevents creating or editing new files or directories
+in EE under `doc/`. This should fail when changes to anything in `/doc` are submitted
+in an EE MR. To pass the test, simply remove the docs changes from the EE MR, and
+[submit them in CE](#ce-first).
+
## Changing document location
Changing a document's location requires specific steps to ensure that
@@ -83,7 +115,7 @@ For example, if you move `doc/workflow/lfs/lfs_administration.md` to
1. Find and replace any occurrences of the old location with the new one.
A quick way to find them is to use `git grep`. First go to the root directory
- where you cloned the `gitlab` repository and then do:
+ where you cloned the `gitlab-ce` repository and then do:
```sh
git grep -n "workflow/lfs/lfs_administration"
@@ -152,13 +184,28 @@ disqus_identifier: 'https://docs.gitlab.com/my-old-location/README.html'
Note: it is necessary to include the file name in the `disqus_identifier` URL,
even if it's `index.html` or `README.html`.
+## Branch naming
+
+If your contribution contains **only** documentation changes, you can speed up
+the CI process by following these branch naming conventions:
+
+| Branch name | Valid example |
+|:----------------------|:-----------------------------|
+| Starting with `docs/` | `docs/update-api-issues` |
+| Starting with `docs-` | `docs-update-api-issues` |
+| Ending in `-docs` | `123-update-api-issues-docs` |
+
+If your branch name matches any of the above, it will run only the docs
+tests. If not, the whole application test suite will run (including docs tests).
+
## Merge requests for GitLab documentation
Before getting started, make sure you read the introductory section
"[contributing to docs](#contributing-to-docs)" above and the
[documentation workflow](workflow.md).
-- Use the current [merge request description template](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/merge_request_templates/Documentation.md)
+- Use the current [merge request description template](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab/merge_request_templates/Documentation.md)
+- Use the correct [branch name](#branch-naming)
- Label the MR `Documentation`
- Assign the correct milestone (see note below)
@@ -175,6 +222,30 @@ frozen or released, use the label `Pick into X.Y` to get it merged into
the correct release. Avoid picking into a past release as much as you can, as
it increases the work of the release managers.
+### Cherry-picking from CE to EE
+
+As we have the `master` branch of CE merged into EE once a day, it's common to
+run into merge conflicts. To avoid them, we [test for merge conflicts against EE](#testing)
+with the `ee-compat-check` job, and use the following method of creating equivalent
+branches for CE and EE.
+
+Follow this [method for cherry-picking from CE to EE](../automatic_ce_ee_merge.md#cherry-picking-from-ce-to-ee), with a few adjustments:
+
+- Create the [CE branch](#branch-naming) starting with `docs-`,
+ e.g.: `git checkout -b docs-example`
+- Create the EE-equivalent branch ending with `-ee`, e.g.,
+ `git checkout -b docs-example-ee`
+- Once all the jobs are passing in CE and EE, and you've addressed the
+ feedback from your own team, assign the CE MR to a technical writer for review
+- When both MRs are ready, the EE merge request will be merged first, and the
+ CE-equivalent will be merged next.
+- Note that the review will occur only in the CE MR, as the EE MR
+ contains the same commits as the CE MR.
+- If you have a few more changes that apply to the EE-version only, you can submit
+ a couple more commits to the EE branch, but ask the reviewer to review the EE merge request
+ additionally to the CE MR. If there are many EE-only changes though, start a new MR
+ to EE only.
+
## GitLab `/help`
Every GitLab instance includes the documentation, which is available at `/help`
@@ -184,7 +255,7 @@ There are [plans](https://gitlab.com/groups/gitlab-org/-/epics/693) to end this
practice and instead link out from the GitLab application to docs.gitlab.com URLs.
The documentation available online on docs.gitlab.com is continuously
-deployed every hour from the `master` branch of GitLab, Omnibus, and Runner. Therefore,
+deployed every hour from the `master` branch of CE, EE, Omnibus, and Runner. Therefore,
once a merge request gets merged, it will be available online on the same day.
However, they will be shipped (and available on `/help`) within the milestone assigned
to the MR.
@@ -264,10 +335,14 @@ You can combine one or more of the following:
### GitLab `/help` tests
-Several [rspec tests](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/features/help_pages_spec.rb)
+Several [rspec tests](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/features/help_pages_spec.rb)
are run to ensure GitLab documentation renders and works correctly. In particular, that [main docs landing page](../../README.md) will work correctly from `/help`.
For example, [GitLab.com's `/help`](https://gitlab.com/help).
+CAUTION: **Caution:**
+Because the rspec tests only run in a full pipeline, and not a special [docs-only pipeline](#branch-naming), it is possible
+to merge changes that will break `master` from a merge request with a successful docs-only pipeline run.
+
## Docs site architecture
See the [Docs site architecture](site_architecture/index.md) page to learn
@@ -287,12 +362,18 @@ To preview your changes to documentation locally, follow this
The live preview is currently enabled for the following projects:
-- <https://gitlab.com/gitlab-org/gitlab>
+- <https://gitlab.com/gitlab-org/gitlab-ce>
+- <https://gitlab.com/gitlab-org/gitlab-ee>
- <https://gitlab.com/gitlab-org/gitlab-runner>
-If your merge request has docs changes, you can use the manual `review-docs-deploy` job
-to deploy the docs review app for your merge request.
-You will need at least Maintainer permissions to be able to run it.
+If your branch contains only documentation changes, you can use
+[special branch names](#branch-naming) to avoid long-running pipelines.
+
+For [docs-only changes](#branch-naming), the review app is run automatically.
+For all other branches, you can use the manual `review-docs-deploy-manual` job
+in your merge request. You will need at least Maintainer permissions to be able
+to run it. In the mini pipeline graph, you should see a `>>` icon. Clicking it will
+reveal the `review-docs-deploy-manual` job. Click the play button to start the job.
![Manual trigger a docs build](img/manual_build_docs.png)
@@ -315,7 +396,7 @@ preview the changes. The docs URL can be found in two places:
triggered pipeline so that you can investigate whether something went wrong
TIP: **Tip:**
-Someone with no merge rights to the GitLab projects (think of forks from
+Someone with no merge rights to the CE/EE projects (think of forks from
contributors) cannot run the manual job. In that case, you can
ask someone from the GitLab team who has the permissions to do that for you.
@@ -341,8 +422,8 @@ In case the review app URL returns 404, follow these steps to debug:
If you want to know the in-depth details, here's what's really happening:
-1. You manually run the `review-docs-deploy` job in a merge request.
-1. The job runs the [`scripts/trigger-build-docs`](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/trigger-build-docs)
+1. You manually run the `review-docs-deploy` job in a CE/EE merge request.
+1. The job runs the [`scripts/trigger-build-docs`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/scripts/trigger-build-docs)
script with the `deploy` flag, which in turn:
1. Takes your branch name and applies the following:
- The slug of the branch name is used to avoid special characters since
@@ -384,6 +465,18 @@ The current tests are:
[`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) directory. In addition,
`docs-lint` also runs [`markdownlint`](#markdownlint) to ensure the
markdown is consistent across all documentation.
+1. [`ee_compat_check`](../automatic_ce_ee_merge.md#avoiding-ce-ee-merge-conflicts-beforehand) (runs on CE only):
+ When you submit a merge request to GitLab Community Edition (CE),
+ there is this additional job that runs against Enterprise Edition (EE)
+ and checks if your changes can apply cleanly to the EE codebase.
+ If that job fails, read the instructions in the job log for what to do next.
+ As CE is merged into EE once a day, it's important to avoid merge conflicts.
+ Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is
+ essential to avoid them.
+1. [`ee-files-location-check`/`ee-specific-lines-check`](#ee-specific-lines-check) (runs on EE only):
+ This test ensures that no new files/directories are created/changed in EE.
+ All docs should be submitted in CE instead, regardless the tier they are on.
+ This is for the [single codebase](#single-codebase) effort.
1. In a full pipeline, tests for [`/help`](#gitlab-help-tests).
### Linting
@@ -407,8 +500,8 @@ This list does not limit what other linters you can add to your local documentat
`proselint` can be used [on the command line](http://proselint.com/utility/), either on a single
Markdown file or on all Markdown files in a project. For example, to run `proselint` on all
- documentation in the [`gitlab` project](https://gitlab.com/gitlab-org/gitlab), run the
- following commands from within the `gitlab` project:
+ documentation in the [`gitlab-ce` project](https://gitlab.com/gitlab-org/gitlab-ce), run the
+ following commands from within the `gitlab-ce` project:
```sh
cd doc
@@ -450,8 +543,8 @@ documentation. This tool helps catch deviations from those guidelines.
`markdownlint` can be used [on the command line](https://github.com/igorshubovych/markdownlint-cli#markdownlint-cli--),
either on a single Markdown file or on all Markdown files in a project. For example, to run
-`markdownlint` on all documentation in the [`gitlab` project](https://gitlab.com/gitlab-org/gitlab),
-run the following commands from within your `gitlab` project root directory, which will
+`markdownlint` on all documentation in the [`gitlab-ce` project](https://gitlab.com/gitlab-org/gitlab-ce),
+run the following commands from within your `gitlab-ce` project root directory, which will
automatically detect the [`.markdownlint.json`](#markdownlint-configuration) config
file in the root of the project, and test all files in `/doc` and its subdirectories:
@@ -484,7 +577,7 @@ Each formatting issue that `markdownlint` checks has an associated
These rules are configured in the `.markdownlint.json` files located in the root of
four repos that are the sources for <https://docs.gitlab.com>:
-- <https://gitlab.com/gitlab-org/gitlab/blob/master/.markdownlint.json>
+- <https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.markdownlint.json>
- <https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.markdownlint.json>
- <https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.markdownlint.json>
- <https://gitlab.com/charts/gitlab/blob/master/.markdownlint.json>
@@ -497,5 +590,5 @@ rules, and also to configure optional parameters for enabled rules as needed.
GitLab uses [Danger](https://github.com/danger/danger) for some elements in
code review. For docs changes in merge requests, whenever a change to files under `/doc`
is made, Danger Bot leaves a comment with further instructions about the documentation
-process. This is configured in the Dangerfile in the GitLab repo under
-[/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/tree/master/danger/documentation).
+process. This is configured in the Dangerfile in the GitLab CE and EE repo under
+[/danger/documentation/](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/danger/documentation).
diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md
index f3ea55d3d5d..f2412c249c1 100644
--- a/doc/development/elasticsearch.md
+++ b/doc/development/elasticsearch.md
@@ -59,7 +59,7 @@ Additionally, if you need large repos or multiple forks for testing, please cons
## How does it work?
-The Elasticsearch integration depends on an external indexer. We ship an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task but, after this is done, GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb).
+The Elasticsearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb).
All indexing after the initial one is done via `ElasticIndexerWorker` (sidekiq jobs).
diff --git a/doc/development/packages.md b/doc/development/packages.md
index 3d209c4a93c..08aa0b08525 100644
--- a/doc/development/packages.md
+++ b/doc/development/packages.md
@@ -1,8 +1,8 @@
# Packages **(PREMIUM)**
-This document will guide you through adding another [package management system](../administration/packages/index.md) support to GitLab.
+This document will guide you through adding another [package management system](../administration/packages.md) support to GitLab.
-See already supported package types in [Packages documentation](../administration/packages/index.md)
+See already supported package types in [Packages documentation](../administration/packages.md)
Since GitLab packages' UI is pretty generic, it is possible to add new
package system support by solely backend changes. This guide is superficial and does
@@ -46,7 +46,7 @@ Group-level and instance-level endpoints are good to have but are optional.
NOTE: **Note:**
To avoid name conflict for instance-level endpoints we use
-[the package naming convention](../user/packages/npm_registry/index.md#package-naming-convention)
+[the package naming convention](../user/project/packages/npm_registry.md#package-naming-convention)
## Configuration
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md
deleted file mode 100644
index 448fb0f9f5a..00000000000
--- a/doc/development/pipelines.md
+++ /dev/null
@@ -1,216 +0,0 @@
-# Pipelines for the GitLab project
-
-Pipelines for `gitlab-org/gitlab` and `gitlab-org/gitlab-foss` (as well as the
-`dev` instance's mirrors) are configured in the usual
-[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml)
-which itself includes files under
-[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/ci)
-for easier maintenance.
-
-We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfooding)
-GitLab [CI/CD features and best-practices](../ci/yaml/README.md)
-as much as possible.
-
-## Stages
-
-The current stages are:
-
-- `prepare`: This stage includes jobs that prepare artifacts that are needed by
- jobs in subsequent stages.
-- `quick-test`: This stage includes test jobs that should run first and fail the
- pipeline early (currently used to run Geo tests when the branch name starts
- with `geo-`, `geo/`, or ends with `-geo`).
-- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs.
-- `review-prepare`: This stage includes a job that build the CNG images that are
- later used by the (Helm) Review App deployment (see
- [Review Apps](testing_guide/review_apps.md) for details).
-- `review`: This stage includes jobs that deploy the GitLab and Docs Review Apps.
-- `qa`: This stage includes jobs that perform QA tasks against the Review App
- that is deployed in the previous stage.
-- `post-test`: This stage includes jobs that build reports or gather data from
- the previous stages' jobs (e.g. coverage, Knapsack metadata etc.).
-- `pages`: This stage includes a job that deploys the various reports as
- GitLab pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>,
- <https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>,
- <https://gitlab-org.gitlab.io/gitlab/webpack-report/>).
-
-## Default image
-
-The default image is currently
-`dev.gitlab.org:5005/gitlab/gitlab-build-images:ruby-2.6.3-golang-1.11-git-2.22-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33`.
-It includes Ruby 2.6.3, Go 1.11, Git 2.22, Chrome 73, Node 12, Yarn 1.16,
-PostgreSQL 9.6, and Graphics Magick 1.3.33.
-
-The images used in our pipelines are configured in the
-[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images)
-project, which is push-mirrored to <https://dev.gitlab.org/gitlab/gitlab-build-images>
-for redundancy.
-
-The current version of the build images can be found in the
-["Used by GitLab CE/EE section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml).
-
-## Default variables
-
-In addition to the [predefined variables](../ci/variables/predefined_variables.md),
-each pipeline includes the following [variables](../ci/variables/README.md):
-
-- `RAILS_ENV: "test"`
-- `NODE_ENV: "test"`
-- `SIMPLECOV: "true"`
-- `GIT_DEPTH: "20"`
-- `GIT_SUBMODULE_STRATEGY: "none"`
-- `GET_SOURCES_ATTEMPTS: "3"`
-- `KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/${CI_PROJECT_NAME}/rspec_report-master.json`
-- `EE_KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/${CI_PROJECT_NAME}/rspec_report-master-ee.json`
-- `FLAKY_RSPEC_SUITE_REPORT_PATH: rspec_flaky/report-suite.json`
-- `BUILD_ASSETS_IMAGE: "false"`
-- `ES_JAVA_OPTS: "-Xms256m -Xmx256m"`
-- `ELASTIC_URL: "http://elastic:changeme@docker.elastic.co-elasticsearch-elasticsearch:9200"`
-
-## Common job definitions
-
-Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends)
-that are scoped to a single
-[configuration parameter](../ci/yaml/README.md#configuration-parameters).
-
-These common definitions are:
-
-- `.default-tags`: Ensures a job has the `gitlab-org` tag to ensure it's using
- our dedicated runners.
-- `.default-retry`: Allows a job to retry upon `unknown_failure`, `api_failure`,
- `runner_system_failure`.
-- `.default-before_script`: Allows a job to use a default `before_script` definition
- suitable for Ruby/Rails tasks that may need a database running (e.g. tests).
-- `.default-cache`: Allows a job to use a default `cache` definition suitable for
- Ruby/Rails and frontend tasks.
-- `.default-only`: Restricts the cases where a job is created. This currently
- includes `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches),
- `/^\d+-\d+-auto-deploy-\d+$/` (security branches), `merge_requests`, `tags`.
- Note that jobs won't be created for branches with this default configuration.
-- `.only-review`: Only creates a job for the `gitlab-org` namespace and if
- Kubernetes integration is available. Also, prevents a job from being created
- for `master` and auto-deploy branches.
-- `.only-review-schedules`: Same as `.only-review` but also restrict a job to
- only run for [schedules](../user/project/pipelines/schedules.md).
-- `.use-pg`: Allows a job to use the `postgres:9.6.14` and `redis:alpine` services.
-- `.use-pg-10`: Allows a job to use the `postgres:10.9` and `redis:alpine` services.
-- `.only-ee`: Only creates a job for the `gitlab` project.
-
-## Changes detection
-
-If a job extends from `.default-only` (and most of the jobs should), it can restrict
-the cases where it should be created
-[based on the changes](../ci/yaml/README.md#onlychangesexceptchanges)
-from a commit or MR by extending from the following CI definitions:
-
-- `.only-code-changes`: Allows a job to only be created upon code-related changes.
-- `.only-qa-changes`: Allows a job to only be created upon QA-related changes.
-- `.only-docs-changes`: Allows a job to only be created upon docs-related changes.
-- `.only-code-qa-changes`: Allows a job to only be created upon code-related or QA-related changes.
-
-**See <https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml>
-for the list of exact patterns.**
-
-## Directed acyclic graph
-
-We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to
-execute jobs out of order for the following jobs:
-
-```mermaid
-graph RL;
- A[setup-test-env];
- B["gitlab:assets:compile<br/>(master only)"];
- C[gitlab:assets:compile pull-cache];
- D["cache gems<br/>(master and tags only)"];
- E[review-build-cng];
- F[build-qa-image];
- G[review-deploy];
- G2["schedule:review-deploy<br/>(master only)"];
- H[karma];
- I[jest];
- J["compile-assets<br/>(master only)"];
- K[compile-assets pull-cache];
- L[webpack-dev-server];
- M[coverage];
- N[pages];
- O[static-analysis];
- P["package-and-qa-manual:master<br/>(master schedule only)"];
- Q[package-and-qa];
- R[package-and-qa-manual];
-
-subgraph "`prepare` stage"
- A
- F
- J
- K
- end
-
-subgraph "`test` stage"
- B --> |needs| A;
- C --> |needs| A;
- D --> |needs| A;
- H -.-> |depends on| A;
- H -.-> |depends on| J;
- H -.-> |depends on| K;
- I -.-> |depends on| A;
- I -.-> |depends on| J;
- I -.-> |depends on| K;
- L -.-> |depends on| A;
- L -.-> |depends on| J;
- L -.-> |depends on| K;
- downtime_check --> |needs and depends on| A;
- db:* --> |needs| A;
- gitlab:setup --> |needs| A;
- O -.-> |depends on| A;
- O -.-> |depends on| B;
- O -.-> |depends on| C;
- downtime_check --> |needs and depends on| A;
- end
-
-subgraph "`review-prepare` stage"
- E --> |needs| C;
- X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| B;
- end
-
-subgraph "`review` stage"
- G --> |needs| E;
- G2 --> |needs| E;
- end
-
-subgraph "`qa` stage"
- Q --> |needs| C;
- Q --> |needs| F;
- R --> |needs| C;
- R --> |needs| F;
- P --> |needs| B;
- P --> |needs| F;
- review-qa-smoke -.-> |depends on| G;
- review-qa-all -.-> |depends on| G;
- review-qa-performance -.-> |depends on| G;
- X2["schedule:review-performance<br/>(master only)"] -.-> |depends on| G2;
- dast -.-> |depends on| G;
- end
-
-subgraph "`post-test` stage"
- M
- end
-
-subgraph "`pages` stage"
- N -.-> |depends on| B;
- N -.-> |depends on| H;
- N -.-> |depends on| M;
- end
-```
-
-## Test jobs
-
-Consult [GitLab tests in the Continuous Integration (CI) context](testing_guide/ci.md)
-for more information.
-
-## Review app jobs
-
-Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information.
-
----
-
-[Return to Development documentation](README.md)
diff --git a/doc/development/polling.md b/doc/development/polling.md
index b664ddb9888..76bb5ae7819 100644
--- a/doc/development/polling.md
+++ b/doc/development/polling.md
@@ -11,9 +11,7 @@ Instead you should use polling mechanism with ETag caching in Redis.
## How to use it
1. Add the path of the endpoint which you want to poll to
- `Gitlab::EtagCaching::Router`.
-1. Set the polling interval header for the response with
- `Gitlab::PollingInterval.set_header`.
+ `Gitlab::EtagCaching::Middleware`.
1. Implement cache invalidation for the path of your endpoint using
`Gitlab::EtagCaching::Store`. Whenever a resource changes you
have to invalidate the ETag for the path that depends on this
diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md
index c00be77ce8c..f6a2f642274 100644
--- a/doc/development/testing_guide/end_to_end/index.md
+++ b/doc/development/testing_guide/end_to_end/index.md
@@ -7,6 +7,24 @@ as expected across the entire software stack and architecture, including
integration of all micro-services and components that are supposed to work
together.
+## Branch naming
+
+If your contribution contains **only** changes under the
+[`qa/` folder](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa), you can
+speed up the CI process by following some branch naming conventions. You have
+three choices:
+
+| Branch name | Valid example |
+|:----------------------|:-----------------------------|
+| Starting with `qa/` | `qa/new-oauth-login-test` |
+| Starting with `qa-` | `qa-new-oauth-login-test` |
+| Ending in `-qa` | `123-new-oauth-login-test-qa` |
+
+If your branch name matches any of the above, it will run only the QA-related
+jobs.
+If it does not, the whole application test suite will run (including QA-related
+jobs).
+
## How do we test GitLab?
We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we
diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md
index 9088e9e9bfb..54ed3f34c89 100644
--- a/doc/development/testing_guide/end_to_end/style_guide.md
+++ b/doc/development/testing_guide/end_to_end/style_guide.md
@@ -101,59 +101,43 @@ end
## Block argument naming
-To have a standard on what we call pages and resources when using the `.perform` method,
-we use the name of the page object in [snake_case](https://en.wikipedia.org/wiki/Snake_case)
-(all lowercase, with words separated by an underscore). See good and bad examples below.
-
-While we prefer to follow the standard in most cases, it is also acceptable to
-use common abbreviations (e.g., mr) or other alternatives, as long as
-the name is not ambiguous. This can include appending `_page` if it helps to
-avoid confusion or make the code more readable. For example, if a page object is
-named `New`, it could be confusing to name the block argument `new` because that
-is used to instantiate objects, so `new_page` would be acceptable.
-
-We chose not to simply use `page` because that would shadow the
-Capybara DSL, potentially leading to confusion and bugs.
+To have a standard on how we call pages when using the `.perform` method, we use the name of page object being called, all lowercased, and separated by underscore, if needed (see good and bad examples below.) This also applies to resources. We chose not to simply use `page` because that would shadow the Capybara DSL, potentially leading to confusion and bugs.
### Examples
**Good**
```ruby
+# qa/specs/features/browser_ui/1_manage/project/add_project_member_spec.rb
+
Page::Project::Settings::Members.perform do |members|
members.do_something
end
```
```ruby
+# qa/specs/features/ee/browser_ui/3_create/merge_request/add_batch_comments_in_merge_request_spec.rb
+
Resource::MergeRequest.fabricate! do |merge_request|
merge_request.do_something_else
end
```
-```ruby
-Resource::MergeRequest.fabricate! do |mr|
- mr.do_something_else
-end
-```
-
-```ruby
-Page::Project::New.peform do |new_page|
- new_page.do_something
-end
-```
-
**Bad**
```ruby
+# qa/specs/features/browser_ui/1_manage/project/add_project_member_spec.rb
+
Page::Project::Settings::Members.perform do |project_settings_members_page|
project_settings_members_page.do_something
end
```
```ruby
-Page::Project::New.peform do |page|
- page.do_something
+# qa/specs/features/ee/browser_ui/3_create/merge_request/add_batch_comments_in_merge_request_spec.rb
+
+Resource::MergeRequest.fabricate! do |merge_request_page|
+ merge_request_page.do_something_else
end
```
diff --git a/doc/development/testing_guide/img/k9s.png b/doc/development/testing_guide/img/k9s.png
deleted file mode 100644
index c4b222f0b64..00000000000
--- a/doc/development/testing_guide/img/k9s.png
+++ /dev/null
Binary files differ
diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md
index 8698a1e4c2d..13772cbe015 100644
--- a/doc/development/testing_guide/review_apps.md
+++ b/doc/development/testing_guide/review_apps.md
@@ -10,30 +10,24 @@ Review Apps are automatically deployed by each pipeline, both in
```mermaid
graph TD
- build-qa-image -->|once the `prepare` stage is done| gitlab:assets:compile
- gitlab:assets:compile -->|once the `gitlab:assets:compile` job is done| review-build-cng
- review-build-cng -.->|triggers a CNG-mirror pipeline and wait for it to be done| CNG-mirror
- CNG-mirror -.->|polls until completed| review-build-cng
- review-build-cng -->|once the `review-build-cng` job is done| review-deploy
- review-deploy -->|once the `review-deploy` job is done| review-qa-smoke
+ build-qa-image -.->|once the `prepare` stage is done| gitlab:assets:compile
+ review-build-cng -->|triggers a CNG-mirror pipeline and wait for it to be done| CNG-mirror
+ review-build-cng -.->|once the `test` stage is done| review-deploy
+ review-deploy -.->|once the `review` stage is done| review-qa-smoke
subgraph "1. gitlab-ce/ee `prepare` stage"
build-qa-image
end
subgraph "2. gitlab-ce/ee `test` stage"
- gitlab:assets:compile
+ gitlab:assets:compile -->|plays dependent job once done| review-build-cng
end
-subgraph "3. gitlab-ce/ee `review-prepare` stage"
- review-build-cng
- end
-
-subgraph "4. gitlab-ce/ee `review` stage"
+subgraph "3. gitlab-ce/ee `review` stage"
review-deploy["review-deploy<br><br>Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps-ce` or `review-apps-ee`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."]
end
-subgraph "5. gitlab-ce/ee `qa` stage"
+subgraph "4. gitlab-ce/ee `qa` stage"
review-qa-smoke[review-qa-smoke<br><br>gitlab-qa runs the smoke suite against the Review App.]
end
@@ -183,25 +177,6 @@ secure note named **gitlab-{ce,ee} Review App's root password**.
`review-qa-raise-e-12chm0-migrations.1-nqwtx`.
1. Click on the `Container logs` link.
-### Diagnosing unhealthy review-app releases
-
-If [Review App Stability](https://gitlab.com/gitlab-org/quality/team-tasks/issues/93) dips this may be a signal
-that the `review-apps-ce/ee` cluster is unhealthy. Leading indicators may be healthcheck failures leading to restarts or majority failure for Review App deployments.
-
-The following items may help diagnose this:
-
-- [Instance group CPU Utilization in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-a/gke-review-apps-ce-preemp-n1-standard-a4c9571c-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_CPU&duration=PT12H) - helpful to identify if nodes are problematic or the entire cluster is trending towards unhealthy
-- [Instance Group size in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-a/gke-review-apps-ce-preemp-n1-standard-a4c9571c-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_SIZE&duration=PT12H) - aids in identifying load spikes on the cluster. Kubernetes will add nodes up to 220 based on total resource requests.
-- `kubectl top nodes --sort-by=cpu` - can identify if node spikes are common or load on specific nodes which may get rebalanced by the Kubernetes scheduler.
-- `kubectl top pods --sort-by=cpu` -
-- [K9s] - K9s is a powerful command line dashboard which allows you to filter by labels. This can help identify trends with apps exceeding the [review-app resource requests](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/scripts/review_apps/base-config.yaml). Kubernetes will schedule pods to nodes based on resource requests and allow for CPU usage up to the limits.
- - In K9s you can sort or add filters by typing the `/` character
- - `-lrelease=<review-app-slug>` - filters down to all pods for a release. This aids in determining what is having issues in a single deployment
- - `-lapp=<app>` - filters down to all pods for a specific app. This aids in determining resource usage by app.
- - You can scroll to a Kubernetes resource and hit `d`(describe), `s`(shell), `l`(logs) for a deeper inspection
-
-![K9s](img/k9s.png)
-
### Troubleshoot a pending `dns-gitlab-review-app-external-dns` Deployment
#### Finding the problem
@@ -291,12 +266,6 @@ find a way to limit it to only us.**
## Other resources
- [Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing)
-- [Stability issues](https://gitlab.com/gitlab-org/quality/team-tasks/issues/212)
-
-### Helpful command line tools
-
-- [K9s] - enables CLI dashboard across pods and enabling filtering by labels
-- [Stern](https://github.com/wercker/stern) - enables cross pod log tailing based on label/field selectors
[charts-1068]: https://gitlab.com/gitlab-org/charts/gitlab/issues/1068
[gitlab-pipeline]: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/44362587
@@ -316,7 +285,6 @@ find a way to limit it to only us.**
[gitlab-ci-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml
[gitlab-k8s-integration]: ../../user/project/clusters/index.md
[password-bug]: https://gitlab.com/gitlab-org/gitlab-ce/issues/53621
-[K9s]: https://github.com/derailed/k9s
---
View Lorry Controller Status