From c93779accb4a302896485039805a3510de244374 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kamil=20Trzci=C5=84ski?= Date: Thu, 4 Apr 2019 17:31:56 +0200 Subject: Add usefull tips about big repositories --- doc/ci/README.md | 1 + doc/ci/large_repositories/index.md | 235 +++++++++++++++++++++++++++++++++++++ 2 files changed, 236 insertions(+) create mode 100644 doc/ci/large_repositories/index.md (limited to 'doc/ci') diff --git a/doc/ci/README.md b/doc/ci/README.md index 913e9d4d789..06c6b883909 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -62,6 +62,7 @@ into more features: | [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | | [Interactive web terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | | [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes in a per-branch basis. | +| [Optimising GitLab for large repositories](large_repositories/index.md) | Useful tips on how to optimise GitLab and GitLab Runner for big repositories. | | [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | | [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md new file mode 100644 index 00000000000..576914e9dc3 --- /dev/null +++ b/doc/ci/large_repositories/index.md @@ -0,0 +1,235 @@ +# Optimising GitLab for large repositories + +Large repositories consisting of more than 50k files in a worktree +often require special consideration because of +the time required to clone and check out. + +GitLab and GitLab Runner handle this scenario well +but require optimised configuration to efficiently perform its +set of operations. + +The general guidelines for handling big repositories are simple. +Each guideline is described in more detail in the sections below: + +- Always fetch incrementally. Do not clone in a way that results in recreating all of the worktree. +- Always use shallow clone to reduce data transfer. Be aware that this puts more burden + on GitLab instance due to higher CPU impact. +- Control the clone directory if you heavily use a fork-based workflow. +- Optimise `git clean` flags to ensure that you remove or keep data that might affect or speed-up your build. + +## Shallow cloning + +> Introduced in GitLab Runner 8.9. + +GitLab and GitLab Runner always perform a full clone by default. +While it means that all changes from GitLab are received, +it often results in receiving extra commit logs. + +Ideally, you should always use `GIT_DEPTH` with a small number +like 10. This will instruct GitLab Runner to perform shallow clones. +Shallow clones makes Git request only the latest set of changes for a given branch, +up to desired number of commits as defined by the `GIT_DEPTH` variable. + +This significantly speeds up fetching of changes from Git repositories, +especially if the repository has a very long backlog consisting of number +of big files as we effectively reduce amount of data transfer. + +The following example makes GitLab Runner shallow clone to fetch only a given branch, +it does not fetch any other branches nor tags. + +```yaml +variables: + GIT_DEPTH: 10 + +test: + script: + - ls -al +``` + +## Git strategy + +> Introduced in GitLab Runner 8.9. + +By default, GitLab is configured to always prefer the `GIT_STRATEGY: fetch` strategy. +The `GIT_STRATEGY: fetch` strategy will re-use existing worktrees if found +on disk. This is different to the `GIT_STRATEGY: clone` strategy +as in case of clones, if a worktree is found, it is removed before clone. + +Usage of `fetch` is preferred because it reduces the amount of data to transfer and +does not really impact the operations that you might do on a repository from CI. + +However, `fetch` does require access to the previous worktree. This works +well when using the `shell` or `docker` executor because these +try to preserve worktrees and try to re-use them by default. + +This does not work today for `kubernetes` executor and has limitations when using +`docker+machine`. `kubernetes` executor today always clones into ephemeral directory. + +GitLab also offers the `GIT_STRATEGY: none` strategy. This disables any `fetch` and `checkout` commands +done by GitLab, requiring you to do them. + +## Git clone path + +> Introduced in GitLab Runner 11.10. + +`GIT_CLONE_PATH` allows you to control where you clone your sources. +This can have implications if you heavily use big repositories with fork workflow. + +Fork workflow from GitLab Runner's perspective is stored as a separate repository +with separate worktree. That means that GitLab Runner cannot optimise the usage +of worktrees and you might have to instruct GitLab Runner to use that. + +In such cases, ideally you want to make the GitLab Runner executor be used only used only +for the given project and not shared across different projects to make this +process more efficient. + +The `GIT_CLONE_PATH` has to be within the `$CI_BUILDS_DIR`. Currently, +it is impossible to pick any path from disk. + +## Git clean flags + +> Introduced in GitLab Runner 11.10. + +`GIT_CLEAN_FLAGS` allows you to control whether or not you require +the `git clean` command to be executed for each CI job. +By default, GitLab ensures that you have your worktree on the given SHA, +and that your repository is clean. + +`GIT_CLEAN_FLAGS` is disabled when set to `none`. On very big repositories, this +might be desired because `git clean` is disk I/O intensive. Controlling that +with `GIT_CLEAN_FLAGS: -ffdx -e .build/`, for example, allows you to control and +disable removal of some directories within the worktree between subsequent runs, +which can speed-up the incremental builds. This has the biggest effect +if you re-use existing machines, and have an existing worktree that you can re-use +for builds. + +For exact parameters accepted by `GIT_CLEAN_FLAGS`, see the documentation +for [git clean](https://git-scm.com/docs/git-clean). The +available parameters are dependent on Git version. + +## Fork-based workflow + +> Introduced in GitLab Runner 11.10. + +Following the guidelines above, lets imagine that we want to: + +- Optimise for a big project (more than 50k files in directory). +- Use forks-based workflow for contributing. +- Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. +- Runner assigned only to project and all forks. + +Lets consider the following two examples, one using `shell` executor and +other using `docker` executor. + +### `shell` executor example + +Lets assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "shell" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.custom_build_dir] + enabled = true +``` + +This `config.toml`: + +- Uses the `shell` executor, +- Specifies a custom `/builds` directory where all clones will be stored. +- Enables the ability to specify `GIT_CLONE_PATH`, +- Runs at most 4 jobs at once. + +### `docker` executor example + +Lets assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This `config.toml`: + +- Uses the `docker` executor, +- Specifies a custom `/builds` directory on disk where all clones will be stored. + We host mount the `/builds` directory to make it reusable between subsequent runs + and be allowed to override the cloning strategy. +- Doesn't enable the ability to specify `GIT_CLONE_PATH` as it is enabled by default. +- Runs at most 4 jobs at once. + +### Our `.gitlab-ci.yml` + +Once we have the executor configured, we need to fine tune our `.gitlab-ci.yml`. + +Our pipeline will be most performant if we use the following `.gitlab-ci.yml`: + +```yaml +variables: + GIT_DEPTH: 10 + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME + +build: + script: ls -al +``` + +The above configures a: + +- Shallow clone of 10, to speed up subsequent `git fetch` commands. +- Custom clone path to make it possible to re-use worktrees between parent project and all forks + because we use the same clone path for all forks. + +Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting +between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor, +so as long as we use it to construct the path, it is guaranteed that this directory will not conflict +with other concurrent jobs running. + +### Store custom clone options in `config.toml` + +Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. +However, sometimes it is desirable to make these schemes part of Runner configuration. + +In the above example of Forks, making this configuration discoverable for users may be preferred, +but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. +In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it +a configuration of Runner. + +We can extend our [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +with the following specification that will be used by Runner if `.gitlab-ci.yml` will not override it: + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + environment = [ + "GIT_DEPTH=10", + "GIT_CLONE_PATH=$CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME" + ] + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This makes the cloning configuration to be part of given Runner, +and does not require us to update each `.gitlab-ci.yml`. -- cgit v1.2.1 From 2040649bbaf5bf5ca01c1d9fc25189f206a0bb50 Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Mon, 8 Apr 2019 12:32:38 +0000 Subject: Docs: Fix anchors related to variables doc --- doc/ci/chatops/README.md | 2 +- doc/ci/docker/using_docker_images.md | 3 +-- doc/ci/environments.md | 2 +- .../examples/laravel_with_gitlab_and_envoy/index.md | 2 +- doc/ci/examples/test-scala-application.md | 2 +- doc/ci/ssh_keys/README.md | 4 ++-- doc/ci/triggers/README.md | 2 +- doc/ci/variables/README.md | 19 ++++++------------- doc/ci/variables/predefined_variables.md | 2 +- doc/ci/variables/where_variables_can_be_used.md | 2 +- doc/ci/yaml/README.md | 4 ++-- 11 files changed, 18 insertions(+), 26 deletions(-) (limited to 'doc/ci') diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index a06fe6961a7..5ecdf0f8c54 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -24,7 +24,7 @@ Since ChatOps is built upon GitLab CI/CD, the job has all the same features and * It is strongly recommended to set `only: [chat]` so the job does not run as part of the standard CI pipeline. * If the job is set to `when: manual`, the pipeline will be created however the job will wait to be started. -* It is important to keep in mind that there is very limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](https://docs.gitlab.com/ce/ci/variables/README.html#priority-of-variables). +* It is important to keep in mind that there is very limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](../variables/README.html#priority-of-environment-variables). ### Controlling the ChatOps reply diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 3314c76d234..13c26bc5f47 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -500,7 +500,7 @@ To configure access for `registry.example.com:5000`, follow these steps: bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= ``` -1. Create a [variable] `DOCKER_AUTH_CONFIG` with the content of the +1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: ```json @@ -642,7 +642,6 @@ creation. [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ [runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry -[variable]: ../variables/README.md#variables [entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint [cmd]: https://docs.docker.com/engine/reference/builder/#cmd [register]: https://docs.gitlab.com/runner/register/ diff --git a/doc/ci/environments.md b/doc/ci/environments.md index eaafc7bc1c0..3e52cc786dd 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -224,7 +224,7 @@ The `name` and `url` parameters for dynamic environments can use most available including: - [Predefined environment variables](variables/README.md#predefined-environment-variables) -- [Project and group variables](variables/README.md#variables) +- [Project and group variables](variables/README.md#gitlab-cicd-environment-variables) - [`.gitlab-ci.yml` variables](yaml/README.md#variables) However, you cannot use variables defined: diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 0f809ed05ca..f56d5429fb7 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -116,7 +116,7 @@ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys cat ~/.ssh/id_rsa ``` -Now, let's add it to your GitLab project as a [variable](../../variables/README.md#variables). +Now, let's add it to your GitLab project as a [variable](../../variables/README.md#gitlab-cicd-environment-variables). Variables are user-defined variables and are stored out of `.gitlab-ci.yml`, for security purposes. They can be added per project by navigating to the project's **Settings** > **CI/CD**. diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index fa18cb22aed..e1164b8d03a 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -69,5 +69,5 @@ in the `.gitlab-ci.yml` file with your application's name. ## Heroku API key You can look up your Heroku API key in your -[account](https://dashboard.heroku.com/account). Add a [protected variable](../variables/README.md#protected-variables) with +[account](https://dashboard.heroku.com/account). Add a [protected variable](../variables/README.md#protected-environment-variables) with this value in **Project ➔ Variables** with key `HEROKU_API_KEY`. diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index ff63d0bd69f..9ed1ec5aa5c 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -49,7 +49,7 @@ to access it. This is where an SSH key pair comes in handy. **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. -1. Create a new [variable](../variables/README.md#variables). +1. Create a new [variable](../variables/README.md#gitlab-cicd-environment-variables). As **Key** enter the name `SSH_PRIVATE_KEY` and in the **Value** field paste the content of your _private_ key that you created earlier. @@ -157,7 +157,7 @@ ssh-keyscan example.com ssh-keyscan 1.2.3.4 ``` -Create a new [variable](../variables/README.md#variables) with +Create a new [variable](../variables/README.md#gitlab-cicd-environment-variables) with `SSH_KNOWN_HOSTS` as "Key", and as a "Value" add the output of `ssh-keyscan`. NOTE: **Note:** diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 398b017277f..0e72bdddee7 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -20,7 +20,7 @@ A unique trigger token can be obtained when [adding a new trigger](#adding-a-new DANGER: **Danger:** Passing plain text tokens in public projects is a security issue. Potential attackers can impersonate the user that exposed their trigger token publicly in -their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#variables) +their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#gitlab-cicd-environment-variables) to protect trigger tokens. ## Adding a new trigger diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 592fdfd2873..61d1a904f76 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -3,7 +3,6 @@ table_display_block: true --- # GitLab CI/CD environment variables -{: #variables} After a brief overview over the use of environment variables, this document teaches you how to use GitLab CI/CD's @@ -65,7 +64,7 @@ To get started, choose one of the existing to be output by the Runner. For example, let's say that you want a given job you're running through your script to output the stage that job is running for. In your `.gitlab-ci.yml` file, -call the variable from your script according to the [syntaxes](#syntax-of-variables-in-job-scripts) available. To +call the variable from your script according to the [syntaxes](#syntax-of-environment-variables-in-job-scripts) available. To output the job stage, use the predefined variable `CI_JOB_STAGE`: ```yaml @@ -145,7 +144,6 @@ settings](../../user/project/pipelines/settings.md#visibility-of-pipelines). Follow the discussion in issue [#13784][ce-13784] for masking the variables. ### Syntax of environment variables in job scripts -{: #syntax-of-variables-in-job-scripts} All variables are set as environment variables in the build environment, and they are accessible with normal methods that are used to access such variables. @@ -278,7 +276,6 @@ script: ``` ### Group-level environment variables -{: #group-level-variables} > Introduced in GitLab 9.4. @@ -297,19 +294,18 @@ Any variables of [subgroups](../../user/group/subgroups/index.md) will be inheri Once you set them, they will be available for all subsequent pipelines. ## Priority of environment variables -{: #priority-of-variables} Variables of different types can take precedence over other variables, depending on where they are defined. The order of precedence for variables is (from highest to lowest): -1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables) or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#making-use-of-scheduled-pipeline-variables). -1. Project-level [variables](#creating-a-custom-environment-variable) or [protected variables](#protected-variables). -1. Group-level [variables](#group-level-variables) or [protected variables](#protected-variables). +1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables) or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#using-variables). +1. Project-level [variables](#creating-a-custom-environment-variable) or [protected variables](#protected-environment-variables). +1. Group-level [variables](#group-level-environment-variables) or [protected variables](#protected-environment-variables). 1. YAML-defined [job-level variables](../yaml/README.md#variables). 1. YAML-defined [global variables](../yaml/README.md#variables). -1. [Deployment variables](#deployment-variables). +1. [Deployment variables](#deployment-environment-variables). 1. [Predefined environment variables](predefined_variables.md). For example, if you define: @@ -329,7 +325,6 @@ about which variables are [not supported](where_variables_can_be_used.md). ## Advanced use ### Protected environment variables -{: #protected-variables} > Introduced in GitLab 9.3. @@ -345,7 +340,6 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. ### Deployment environment variables -{: #deployment-variables} > Introduced in GitLab 8.15. @@ -382,7 +376,7 @@ limitations with the current Auto DevOps scripting environment. [Manually triggered pipelines](../pipelines.md#manually-executing-pipelines) allow you to override the value of a current variable. For instance, suppose you added a -[custom variable `$TEST`](#creating-a-custom-variable) +[custom variable `$TEST`](#creating-a-custom-environment-variable) as exemplified above and you want to override it in a manual pipeline. Navigate to your project's **CI/CD > Pipelines** and click **Run pipeline**. Choose the branch you want to run the pipeline for, then add a new variable @@ -396,7 +390,6 @@ value you set for this specific pipeline: ![Manually overridden variable output](img/override_value_via_manual_pipeline_output.png) ## Environment variables expressions -{: #variables-expressions} > Introduced in GitLab 10.7. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index b429dc8c8be..00884610e07 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -36,7 +36,7 @@ future GitLab releases.** | `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Present only when building tags. | | `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message | | `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | -| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | +| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug tracing](../README.md#debug-tracing) is enabled | | `CI_DEPLOY_PASSWORD` | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| | `CI_DEPLOY_USER` | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| | `CI_DISPOSABLE_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 1218ac0b071..091ddcb0bae 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -111,4 +111,4 @@ They are: - Script execution shell. - Not supported: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - - In the `only` and `except` [variables expressions](README.md#variables-expressions). + - In the `only` and `except` [variables expressions](README.md#environment-variables-expressions). diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 1d76f911943..5e44de13b51 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -518,7 +518,7 @@ end-to-end: - $CI_COMMIT_MESSAGE =~ /skip-end-to-end-tests/ ``` -Learn more about [variables expressions](../variables/README.md#variables-expressions). +Learn more about [variables expressions](../variables/README.md#environment-variables-expressions). #### `only:changes`/`except:changes` @@ -2249,7 +2249,7 @@ Runner itself](../variables/README.md#predefined-environment-variables). One example would be `CI_COMMIT_REF_NAME` which has the value of the branch or tag name for which project is built. Apart from the variables you can set in `.gitlab-ci.yml`, there are also the so called -[Variables](../variables/README.md#variables) +[Variables](../variables/README.md#gitlab-cicd-environment-variables) which can be set in GitLab's UI. Learn more about [variables and their priority][variables]. -- cgit v1.2.1 From a4a7344515ec67a5d24b27176ed5ef113730f714 Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Tue, 9 Apr 2019 03:01:30 +0000 Subject: Docs: Update predefined_variables.md to fix anchor --- doc/ci/variables/predefined_variables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/ci') diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 00884610e07..846c539daab 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -36,7 +36,7 @@ future GitLab releases.** | `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Present only when building tags. | | `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message | | `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | -| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug tracing](../README.md#debug-tracing) is enabled | +| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug tracing](README.md#debug-tracing) is enabled | | `CI_DEPLOY_PASSWORD` | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| | `CI_DEPLOY_USER` | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| | `CI_DISPOSABLE_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | -- cgit v1.2.1 From e47cc1086771dcd84f5ce216bfca23033cdf2b02 Mon Sep 17 00:00:00 2001 From: Evan Read Date: Tue, 9 Apr 2019 09:32:27 +0000 Subject: Update auth with registry docs --- doc/ci/docker/using_docker_build.md | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) (limited to 'doc/ci') diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 9266c4511be..5222cc45bc4 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -303,20 +303,19 @@ services: - docker:dind variables: - CONTAINER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH DOCKER_HOST: tcp://docker:2375 DOCKER_DRIVER: overlay2 before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build script: - - docker pull $CONTAINER_IMAGE:latest || true - - docker build --cache-from $CONTAINER_IMAGE:latest --tag $CONTAINER_IMAGE:$CI_COMMIT_SHA --tag $CONTAINER_IMAGE:latest . - - docker push $CONTAINER_IMAGE:$CI_COMMIT_SHA - - docker push $CONTAINER_IMAGE:latest + - docker pull $CI_REGISTRY_IMAGE:latest || true + - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest . + - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + - docker push $CI_REGISTRY_IMAGE:latest ``` The steps in the `script` section for the `build` stage can be summed up to: @@ -324,7 +323,7 @@ The steps in the `script` section for the `build` stage can be summed up to: 1. The first command tries to pull the image from the registry so that it can be used as a cache for the `docker build` command. 1. The second command builds a Docker image using the pulled image as a - cache (notice the `--cache-from $CONTAINER_IMAGE:latest` argument) if + cache (notice the `--cache-from $CI_REGISTRY_IMAGE:latest` argument) if available, and tags it. 1. The last two commands push the tagged Docker images to the container registry so that they may also be used as cache for subsequent builds. @@ -421,14 +420,14 @@ and depend on the visibility of your project. For all projects, mostly suitable for public ones: -- **Using the special `gitlab-ci-token` user**: This user is created for you in order to +- **Using the special `$CI_REGISTRY_USER` variable**: The user specified by this variable is created for you in order to push to the Registry connected to your project. Its password is automatically - set with the `$CI_JOB_TOKEN` variable. This allows you to automate building and deploying + set with the `$CI_REGISTRY_PASSWORD` variable. This allows you to automate building and deploying your Docker images and has read/write access to the Registry. This is ephemeral, so it's only valid for one job. You can use the following example as-is: ```sh - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY ``` For private and internal projects: @@ -436,8 +435,10 @@ For private and internal projects: - **Using a personal access token**: You can create and use a [personal access token](../../user/profile/personal_access_tokens.md) in case your project is private: - - For read (pull) access, the scope should be `read_registry`. - - For read/write (pull/push) access, use `api`. + + - For read (pull) access, the scope should be `read_registry`. + - For read/write (pull/push) access, use `api`. + Replace the `` and `` in the following example: ```sh @@ -469,9 +470,9 @@ could look like: DOCKER_DRIVER: overlay2 stage: build script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.example.com - - docker build -t registry.example.com/group/project/image:latest . - - docker push registry.example.com/group/project/image:latest + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY + - docker build -t $CI_REGISTRY/group/project/image:latest . + - docker push $CI_REGISTRY/group/project/image:latest ``` You can also make use of [other variables](../variables/README.md) to avoid hardcoding: @@ -486,7 +487,7 @@ variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build @@ -526,7 +527,7 @@ variables: CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build -- cgit v1.2.1 From 163b6efb6a5857bb67c89161be721467a9ac338b Mon Sep 17 00:00:00 2001 From: Evan Read Date: Tue, 9 Apr 2019 10:48:07 +0000 Subject: Use US English for content --- doc/ci/README.md | 2 +- doc/ci/large_repositories/index.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) (limited to 'doc/ci') diff --git a/doc/ci/README.md b/doc/ci/README.md index 06c6b883909..a7ad2018b09 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -62,7 +62,7 @@ into more features: | [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | | [Interactive web terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | | [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes in a per-branch basis. | -| [Optimising GitLab for large repositories](large_repositories/index.md) | Useful tips on how to optimise GitLab and GitLab Runner for big repositories. | +| [Optimizing GitLab for large repositories](large_repositories/index.md) | Useful tips on how to optimize GitLab and GitLab Runner for big repositories. | | [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | | [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 576914e9dc3..cfe638c0a22 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -1,11 +1,11 @@ -# Optimising GitLab for large repositories +# Optimizing GitLab for large repositories Large repositories consisting of more than 50k files in a worktree often require special consideration because of the time required to clone and check out. GitLab and GitLab Runner handle this scenario well -but require optimised configuration to efficiently perform its +but require optimized configuration to efficiently perform its set of operations. The general guidelines for handling big repositories are simple. @@ -15,7 +15,7 @@ Each guideline is described in more detail in the sections below: - Always use shallow clone to reduce data transfer. Be aware that this puts more burden on GitLab instance due to higher CPU impact. - Control the clone directory if you heavily use a fork-based workflow. -- Optimise `git clean` flags to ensure that you remove or keep data that might affect or speed-up your build. +- Optimize `git clean` flags to ensure that you remove or keep data that might affect or speed-up your build. ## Shallow cloning @@ -76,7 +76,7 @@ done by GitLab, requiring you to do them. This can have implications if you heavily use big repositories with fork workflow. Fork workflow from GitLab Runner's perspective is stored as a separate repository -with separate worktree. That means that GitLab Runner cannot optimise the usage +with separate worktree. That means that GitLab Runner cannot optimize the usage of worktrees and you might have to instruct GitLab Runner to use that. In such cases, ideally you want to make the GitLab Runner executor be used only used only @@ -113,7 +113,7 @@ available parameters are dependent on Git version. Following the guidelines above, lets imagine that we want to: -- Optimise for a big project (more than 50k files in directory). +- Optimize for a big project (more than 50k files in directory). - Use forks-based workflow for contributing. - Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. - Runner assigned only to project and all forks. -- cgit v1.2.1 From 98a48b41b9cd478843890e0e6dc29109746e4f7c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Coutable?= Date: Tue, 9 Apr 2019 17:25:44 +0200 Subject: Change mode to 644 for doc/ci/variables/img/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Rémy Coutable --- doc/ci/variables/img/ci_job_stage_output_example.png | Bin doc/ci/variables/img/custom_variable_output.png | Bin doc/ci/variables/img/new_custom_variable_example.png | Bin .../img/override_value_via_manual_pipeline_output.png | Bin doc/ci/variables/img/override_variable_manual_pipeline.png | Bin 5 files changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 doc/ci/variables/img/ci_job_stage_output_example.png mode change 100755 => 100644 doc/ci/variables/img/custom_variable_output.png mode change 100755 => 100644 doc/ci/variables/img/new_custom_variable_example.png mode change 100755 => 100644 doc/ci/variables/img/override_value_via_manual_pipeline_output.png mode change 100755 => 100644 doc/ci/variables/img/override_variable_manual_pipeline.png (limited to 'doc/ci') diff --git a/doc/ci/variables/img/ci_job_stage_output_example.png b/doc/ci/variables/img/ci_job_stage_output_example.png old mode 100755 new mode 100644 diff --git a/doc/ci/variables/img/custom_variable_output.png b/doc/ci/variables/img/custom_variable_output.png old mode 100755 new mode 100644 diff --git a/doc/ci/variables/img/new_custom_variable_example.png b/doc/ci/variables/img/new_custom_variable_example.png old mode 100755 new mode 100644 diff --git a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png old mode 100755 new mode 100644 diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png old mode 100755 new mode 100644 -- cgit v1.2.1 From ce042ef02a66cb380e33e07b71ce664178ddc07e Mon Sep 17 00:00:00 2001 From: Evan Read Date: Wed, 10 Apr 2019 06:29:36 +0000 Subject: Update broken links to new destinations --- doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) (limited to 'doc/ci') diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index 1a909e8892a..4a5fda661df 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -11,7 +11,7 @@ last_updated: 2019-03-06 [Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site], which is a functional language designed for productivity and maintainability that runs on the -[Erlang VM][erlang-site]. Erlang VM is really really fast and can handle very large numbers of +[Erlang VM](https://www.erlang.org). Erlang VM is really really fast and can handle very large numbers of simultaneous users. That's why we're hearing so much about Phoenix today. @@ -27,7 +27,7 @@ and GitLab UI._ ### What is Phoenix? [Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site] very useful - to build fast, reliable, and high-performance applications, as it uses [Erlang VM][erlang-site]. + to build fast, reliable, and high-performance applications, as it uses [Erlang VM](https://www.erlang.org). Many components and concepts are similar to Ruby on Rails or Python's Django. High developer productivity and high application performance are only a few advantages on learning how to use it. @@ -406,7 +406,6 @@ other reasons][ci-reasons] to keep using GitLab CI/CD. The benefits to our teams [elixir-site]: http://elixir-lang.org/ "Elixir" [elixir-mix]: http://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html "Introduction to mix" [elixir-docs]: http://elixir-lang.org/getting-started/introduction.html "Elixir Documentation" -[erlang-site]: http://erlang.org "Erlang" [elixir-install]: https://elixir-lang.org/install.html "Elixir Installation" [ecto]: http://hexdocs.pm/ecto "Ecto" [ecto-repo]: https://hexdocs.pm/ecto/Ecto.html#module-repositories "Ecto Repositories" -- cgit v1.2.1