diff options
| author | Tomasz Maczukin <tomasz@maczukin.pl> | 2015-12-21 13:27:34 +0100 |
|---|---|---|
| committer | Tomasz Maczukin <tomasz@maczukin.pl> | 2015-12-21 13:27:34 +0100 |
| commit | 3cfd892f382d3784f614fea75f929c44fe838559 (patch) | |
| tree | cb9aa9ab48ee02e8f00e8506fecc182d1e66b9ea /doc/ci | |
| parent | 85ad95be741848fbf15a01789f065e001326cefa (diff) | |
| parent | 4b4cbf0ce4925e22a635e4432e7ac8602199fa5b (diff) | |
| download | gitlab-ce-3cfd892f382d3784f614fea75f929c44fe838559.tar.gz | |
Merge branch 'master' into fix/visibility-level-setting-in-forked-projects
* master: (723 commits)
Bump Rack Attack to v4.3.1 for security fix
Remove duplicate entry in the changelog
Remove extra spaces after branchname
Fix merge-request-reopen button title
Add branch and tag operation to tree dropdown
Use gitlab-shell 2.6.9
Clarify Windows shell executor artifact upload support
Fix feature specs: we always show the build status if ci_commit is present
Do not display project group/name when issue and MR are in same project
Don't create CI status for refs that doesn't have .gitlab-ci.yml, even if the builds are enabled
Use gitlab-workhorse 0.5.1
Fix ci_projects migration by using the value only from latest row [ci skip]
Revert sidebar position for issue and merge request
Add info on using private Docker registries in CI [ci skip]
Upgrade Poltergeist to 1.8.1. #4131
Fix ux issue with "This issue will be closed automatically" message
Move MR Builds tab next to Commits
Api support for requesting starred projects for user
Fix Rubocop complain.
Fix merge widget JS for buttons
...
Conflicts:
app/models/project.rb
Diffstat (limited to 'doc/ci')
26 files changed, 1236 insertions, 282 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 97325069ceb..5d9d7a81db3 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -9,6 +9,18 @@ + [Using Docker Images](docker/using_docker_images.md) + [Using Docker Build](docker/using_docker_build.md) + [Using Variables](variables/README.md) ++ [Using SSH keys](ssh_keys/README.md) + +### Languages + ++ [Testing PHP](languages/php.md) + +### Services + ++ [Using MySQL](services/mysql.md) ++ [Using PostgreSQL](services/postgres.md) ++ [Using Redis](services/redis.md) ++ [Using Other Services](docker/using_docker_images.md#how-to-use-other-images-as-services) ### Examples diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index ef8a7ec1e86..31458d61674 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -1,19 +1,29 @@ # Using Docker Images -GitLab CI can use [Docker Engine](https://www.docker.com/) to build projects. -Docker is an open-source project that allows to use predefined images to run applications -in independent "containers" that are run within a single Linux instance. -[Docker Hub](https://registry.hub.docker.com/) have rich database of built images that can be used to build applications. +GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use +[Docker Engine](https://www.docker.com/) to test and build any application. -Docker when used with GitLab CI runs each build in separate and isolated container using predefined image and always from scratch. -It makes it easier to have simple and reproducible build environment that can also be run on your workstation. -This allows you to test all commands from your shell, rather than having to test them on a CI server. +Docker is an open-source project that allows you to use predefined images to +run applications in independent "containers" that are run within a single Linux +instance. [Docker Hub][hub] has a rich database of pre-built images that can be +used to test and build your applications. -### Register Docker runner -To use GitLab Runner with Docker you need to register new runner to use `docker` executor: +Docker, when used with GitLab CI, runs each build in a separate and isolated +container using the predefined image that is set up in +[`.gitlab-ci.yml`](../yaml/README.md). + +This makes it easier to have a simple and reproducible build environment that +can also run on your workstation. The added benefit is that you can test all +the commands that we will explore later from your shell, rather than having to +test them on a dedicated CI server. + +## Register docker runner + +To use GitLab Runner with docker you need to register a new runner to use the +`docker` executor: ```bash -gitlab-ci-multi-runner register \ +gitlab-runner register \ --url "https://gitlab.com/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "docker-ruby-2.1" \ @@ -23,101 +33,79 @@ gitlab-ci-multi-runner register \ --docker-mysql latest ``` -**The registered runner will use `ruby:2.1` image and will run two services (`postgres:latest` and `mysql:latest`) that will be accessible for time of the build.** +The registered runner will use the `ruby:2.1` docker image and will run two +services, `postgres:latest` and `mysql:latest`, both of which will be +accessible during the build process. -### What is image? -The image is the name of any repository that is present in local Docker Engine or any repository that can be found at [Docker Hub](https://registry.hub.docker.com/). -For more information about the image and Docker Hub please read the [Docker Fundamentals](https://docs.docker.com/introduction/understanding-docker/). +## What is image -### What is service? -Service is just another image that is run for time of your build and is linked to your build. This allows you to access the service image during build time. -The service image can run any application, but most common use case is to run some database container, ie.: `mysql`. -It's easier and faster to use existing image, run it as additional container than install `mysql` every time project is built. +The `image` keyword is the name of the docker image that is present in the +local Docker Engine (list all images with `docker images`) or any image that +can be found at [Docker Hub][hub]. For more information about images and Docker +Hub please read the [Docker Fundamentals][] documentation. -#### How is service linked to the build? -There's good document that describes how Docker linking works: [Linking containers together](https://docs.docker.com/userguide/dockerlinks/). -To summarize: if you add `mysql` as service to your application, the image will be used to create container that is linked to build container. -The service container for MySQL will be accessible under hostname `mysql`. -So, **to access your database service you have to connect to host: `mysql` instead of socket or `localhost`**. +In short, with `image` we refer to the docker image, which will be used to +create a container on which your build will run. -### How to use other images as services? -You are not limited to have only database services. -You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). -Look for `[runners.docker]` section: -``` -[runners.docker] - image = "ruby:2.1" - services = ["mysql:latest", "postgres:latest"] -``` +## What is service -For example you need `wordpress` instance to test some API integration with `Wordpress`. -You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). -This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in: -``` -[runners.docker] - image = "ruby:2.1" - services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"] -``` +The `services` keyword defines just another docker image that is run during +your build and is linked to the docker image that the `image` keyword defines. +This allows you to access the service image during build time. -Next time when you run your application the `tutum/wordpress` will be started -and you will have access to it from your build container under hostname: `tutum_wordpress`. +The service image can run any application, but the most common use case is to +run a database container, eg. `mysql`. It's easier and faster to use an +existing image and run it as an additional container than install `mysql` every +time the project is built. -Alias hostname for the service is made from the image name: -1. Everything after `:` is stripped, -2. '/' is replaced to `_`. +You can see some widely used services examples in the relevant documentation of +[CI services examples](../services/README.md). -### Configuring services -Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment. +### How is service linked to the build -GitLab Runner 0.5.0 and up passes all YAML-defined variables to created service containers. +To better understand how the container linking works, read +[Linking containers together](https://docs.docker.com/userguide/dockerlinks/). -1. To configure database name for [postgres](https://registry.hub.docker.com/u/library/postgres/) service, -you need to set POSTGRES_DB. +To summarize, if you add `mysql` as service to your application, the image will +then be used to create a container that is linked to the build container. - ```yaml - services: - - postgres - - variables: - POSTGRES_DB: gitlab - ``` +The service container for MySQL will be accessible under the hostname `mysql`. +So, in order to access your database service you have to connect to the host +named `mysql` instead of a socket or `localhost`. -1. To use [mysql](https://registry.hub.docker.com/u/library/mysql/) service with empty password for time of build, -you need to set MYSQL_ALLOW_EMPTY_PASSWORD. +## Overwrite image and services - ```yaml - services: - - mysql - - variables: - MYSQL_ALLOW_EMPTY_PASSWORD: "yes" - ``` +See [How to use other images as services](#how-to-use-other-images-as-services). -For other possible configuration variables check the -https://registry.hub.docker.com/u/library/mysql/ or https://registry.hub.docker.com/u/library/postgres/ -or README page for any other Docker image. +## How to use other images as services -**Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.** +You are not limited to have only database services. You can add as many +services you need to `.gitlab-ci.yml` or manually modify `config.toml`. +Any image found at [Docker Hub][hub] can be used as a service. -### Overwrite image and services -It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`. -If you add to your YAML the `image` and the `services` these parameters -be used instead of the ones that were specified during runner's registration. -``` +## Define image and services from `.gitlab-ci.yml` + +You can simply define an image that will be used for all jobs and a list of +services that you want to use during build time. + +```yaml image: ruby:2.2 + services: - postgres:9.3 -before_install: + +before_script: - bundle install - + test: script: - bundle exec rake spec ``` -It's possible to define image and service per-job: -``` -before_install: +It is also possible to define different images and services per job: + +```yaml +before_script: - bundle install test:2.1: @@ -135,34 +123,91 @@ test:2.2: - bundle exec rake spec ``` -#### How to enable overwriting? -To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). -You can do that by hand modifying runner configuration: `config.toml`. -Please go to section where is `[runners.docker]` definition for your runner. -Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`: +## Define image and services in `config.toml` + +Look for the `[runners.docker]` section: + ``` [runners.docker] image = "ruby:2.1" - allowed_images = ["ruby:*", "python:*"] - allowed_services = ["mysql:*", "redis:*"] + services = ["mysql:latest", "postgres:latest"] ``` -This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. -You will be able to pick only `ruby` and `python` images. -The same rule can be applied to limit services. -If you are courageous enough, you can make it fully open and accept everything: +The image and services defined this way will be added to all builds run by +that runner. + +## Define an image from a private Docker registry + +Starting with GitLab Runner 0.6.0, you are able to define images located to +private registries that could also require authentication. + +All you have to do is be explicit on the image definition in `.gitlab-ci.yml`. + +```yaml +image: my.registry.tld:5000/namepace/image:tag ``` -[runners.docker] - image = "ruby:2.1" - allowed_images = ["*", "*/*"] - allowed_services = ["*", "*/*"] + +In the example above, GitLab Runner will look at `my.registry.tld:5000` for the +image `namespace/image:tag`. + +If the repository is private you need to authenticate your GitLab Runner in the +registry. Learn how to do that on +[GitLab Runner's documentation][runner-priv-reg]. + +## Accessing the services + +Let's say that you need a Wordpress instance to test some API integration with +your application. + +You can then use for example the [tutum/wordpress][] image in your +`.gitlab-ci.yml`: + +```yaml +services: +- tutum/wordpress:latest ``` -**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.** +When the build is run, `tutum/wordpress` will be started and you will have +access to it from your build container under the hostname `tutum__wordpress`. + +The alias hostname for the service is made from the image name following these +rules: + +1. Everything after `:` is stripped +2. Backslash (`/`) is replaced with double underscores (`__`) + +## Configuring services + +Many services accept environment variables which allow you to easily change +database names or set account names depending on the environment. + +GitLab Runner 0.5.0 and up passes all YAML-defined variables to the created +service containers. + +For all possible configuration variables check the documentation of each image +provided in their corresponding Docker hub page. + +*Note: All variables will be passed to all services containers. It's not +designed to distinguish which variable should go where.* + +### PostgreSQL service example + +See the specific documentation for +[using PostgreSQL as a service](../services/postgres.md). + +### MySQL service example + +See the specific documentation for +[using MySQL as a service](../services/mysql.md). + +## How Docker integration works + +Below is a high level overview of the steps performed by docker during build +time. -### How Docker integration works 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. -1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example). +1. Create cache container to store all volumes as defined in `config.toml` and + `Dockerfile` of build image (`ruby:2.1` as in above example). 1. Create build container and link any service container to build container. 1. Start build container and send build script to the container. 1. Run build script. @@ -171,33 +216,64 @@ If you are courageous enough, you can make it fully open and accept everything: 1. Check exit status of build script. 1. Remove build container and all created service containers. -### How to debug a build locally -1. Create a file with build script: +## How to debug a build locally + +*Note: The following commands are run without root privileges. You should be +able to run docker with your regular user account.* + +First start with creating a file named `build script`: + ```bash -$ cat <<EOF > build_script +cat <<EOF > build_script git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner cd /builds/gitlab-org/gitlab-ci-multi-runner -make <- or any other build step +make EOF ``` -1. Create service containers: +Here we use as an example the GitLab Runner repository which contains a +Makefile, so running `make` will execute the commands defined in the Makefile. +Your mileage may vary, so instead of `make` you could run the command which +is specific to your project. + +Then create some service containers: + ``` -$ docker run -d -n service-mysql mysql:latest -$ docker run -d -n service-postgres postgres:latest +docker run -d -n service-mysql mysql:latest +docker run -d -n service-postgres postgres:latest ``` -This will create two service containers (MySQL and PostgreSQL). -1. Create a build container and execute script in its context: +This will create two service containers, named `service-mysql` and +`service-postgres` which use the latest MySQL and PostgreSQL images +respectively. They will both run in the background (`-d`). + +Finally, create a build container by executing the `build_script` file we +created earlier: + ``` -$ cat build_script | docker run -n build -i -l mysql:service-mysql -l postgres:service-postgres ruby:2.1 /bin/bash +docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script ``` -This will create build container that has two service containers linked. -The build_script is piped using STDIN to bash interpreter which executes the build script in container. -1. At the end remove all containers: +The above command will create a container named `build` that is spawned from +the `ruby:2.1` image and has two services linked to it. The `build_script` is +piped using STDIN to the bash interpreter which in turn executes the +`build_script` in the `build` container. + +When you finish testing and no longer need the containers, you can remove them +with: + ``` docker rm -f -v build service-mysql service-postgres ``` -This will forcefully (the `-f` switch) remove build container and service containers -and all volumes (the `-v` switch) that were created with the container creation. + +This will forcefully (`-f`) remove the `build` container, the two service +containers as well as all volumes (`-v`) that were created with the container +creation. + +[Docker Fundamentals]: https://docs.docker.com/engine/introduction/understanding-docker/ +[hub]: https://hub.docker.com/ +[linking-containers]: https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/ +[tutum/wordpress]: https://registry.hub.docker.com/u/tutum/wordpress/ +[postgres-hub]: https://registry.hub.docker.com/u/library/postgres/ +[mysql-hub]: https://registry.hub.docker.com/u/library/mysql/ +[runner-priv-reg]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry diff --git a/doc/ci/img/builds_tab.png b/doc/ci/img/builds_tab.png Binary files differnew file mode 100644 index 00000000000..d088b8b329d --- /dev/null +++ b/doc/ci/img/builds_tab.png diff --git a/doc/ci/languages/README.md b/doc/ci/languages/README.md new file mode 100644 index 00000000000..54b2343e08b --- /dev/null +++ b/doc/ci/languages/README.md @@ -0,0 +1,7 @@ +### Languages + +This is a list of languages you can test with GitLab CI. Each section has +comprehensive documentation and comes with a test repository hosted on +GitLab.com + ++ [Testing PHP](php.md) diff --git a/doc/ci/languages/php.md b/doc/ci/languages/php.md new file mode 100644 index 00000000000..dacb67fa3ff --- /dev/null +++ b/doc/ci/languages/php.md @@ -0,0 +1,284 @@ +# Testing PHP projects + +This guide covers basic building instructions for PHP projects. + +There are covered two cases: testing using the Docker executor and testing +using the Shell executor. + +## Test PHP projects using the Docker executor + +While it is possible to test PHP apps on any system, this would require manual +configuration from the developer. To overcome this we will be using the +official [PHP docker image][php-hub] that can be found in Docker Hub. + +This will allow us to test PHP projects against different versions of PHP. +However, not everything is plug 'n' play, you still need to onfigure some +things manually. + +As with every build, you need to create a valid `.gitlab-ci.yml` describing the +build environment. + +Let's first specify the PHP image that will be used for the build process +(you can read more about what an image means in the Runner's lingo reading +about [Using Docker images](../docker/using_docker_images.md#what-is-image)). + +Start by adding the image to your `.gitlab-ci.yml`: + +```yaml +image: php:5.6 +``` + +The official images are great, but they lack a few useful tools for testing. +We need to first prepare the build environment. A way to overcome this is to +create a script which installs all prerequisites prior the actual testing is +done. + +Let's create a `ci/docker_install.sh` file in the root directory of our +repository with the following content: + +```bash +#!/bin/bash + +# We need to install dependencies only for Docker +[[ ! -e /.dockerinit ]] && exit 0 + +set -xe + +# Install git (the php image doesn't have it) which is required by composer +apt-get update -yqq +apt-get install git -yqq + +# Install phpunit, the tool that we will use for testing +curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar +chmod +x /usr/local/bin/phpunit + +# Install mysql driver +# Here you can install any other extension that you need +docker-php-ext-install pdo_mysql +``` + +You might wonder what `docker-php-ext-install` is. In short, it is a script +provided by the official php docker image that you can use to easilly install +extensions. For more information read the the documentation at +<https://hub.docker.com/_/php/>. + +Now that we created the script that contains all prerequisites for our build +environment, let's add it in `.gitlab-ci.yml`: + +```yaml +... + +before_script: +- bash ci/docker_install.sh > /dev/null + +... +``` + +Last step, run the actual tests using `phpunit`: + +```yaml +... + +test:app: + script: + - phpunit --configuration phpunit_myapp.xml + +... +``` + +Finally, commit your files and push them to GitLab to see your build succeeding +(or failing). + +The final `.gitlab-ci.yml` should look similar to this: + +```yaml +# Select image from https://hub.docker.com/_/php/ +image: php:5.6 + +before_script: +# Install dependencies +- ci/docker_install.sh > /dev/null + +test:app: + script: + - phpunit --configuration phpunit_myapp.xml +``` + +### Test against different PHP versions in Docker builds + +Testing against multiple versions of PHP is super easy. Just add another job +with a different docker image version and the runner will do the rest: + +```yaml +before_script: +# Install dependencies +- ci/docker_install.sh > /dev/null + +# We test PHP5.6 +test:5.6: + image: php:5.6 + script: + - phpunit --configuration phpunit_myapp.xml + +# We test PHP7.0 (good luck with that) +test:7.0: + image: php:7.0 + script: + - phpunit --configuration phpunit_myapp.xml +``` + +### Custom PHP configuration in Docker builds + +There are times where you will need to customise your PHP environment by +putting your `.ini` file into `/usr/local/etc/php/conf.d/`. For that purpose +add a `before_script` action: + +```yaml +before_script: +- cp my_php.ini /usr/local/etc/php/conf.d/test.ini +``` + +Of course, `my_php.ini` must be present in the root directory of your repository. + +## Test PHP projects using the Shell executor + +The shell executor runs your builds in a terminal session on your server. +Thus, in order to test your projects you first need to make sure that all +dependencies are installed. + +For example, in a VM running Debian 8 we first update the cache, then we +install `phpunit` and `php5-mysql`: + +```bash +sudo apt-get update -y +sudo apt-get install -y phpunit php5-mysql +``` + +Next, add the following snippet to your `.gitlab-ci.yml`: + +```yaml +test:app: + script: + - phpunit --configuration phpunit_myapp.xml +``` + +Finally, push to GitLab and let the tests begin! + +### Test against different PHP versions in Shell builds + +The [phpenv][] project allows you to easily manage different versions of PHP +each with its own config. This is specially usefull when testing PHP projects +with the Shell executor. + +You will have to install it on your build machine under the `gitlab-runner` +user following [the upstream installation guide][phpenv-installation]. + +Using phpenv also allows to easily configure the PHP environment with: + +``` +phpenv config-add my_config.ini +``` + +*__Important note:__ It seems `phpenv/phpenv` + [is abandoned](https://github.com/phpenv/phpenv/issues/57). There is a fork + at [madumlao/phpenv](https://github.com/madumlao/phpenv) that tries to bring + the project back to life. [CHH/phpenv](https://github.com/CHH/phpenv) also + seems like a good alternative. Picking any of the mentioned tools will work + with the basic phpenv commands. Guiding you to choose the right phpenv is out + of the scope of this tutorial.* + +### Install custom extensions + +Since this is a pretty bare installation of the PHP environment, you may need +some extensions that are not currently present on the build machine. + +To install additional extensions simply execute: + +```bash +pecl install <extension> +``` + +It's not advised to add this to `.gitlab-ci.yml`. You should execute this +command once, only to setup the build environment. + +## Extend your tests + +### Using atoum + +Instead of PHPUnit, you can use any other tool to run unit tests. For example +you can use [atoum](https://github.com/atoum/atoum): + +```yaml +before_script: +- wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar + +test:atoum: + script: + - php mageekguy.atoum.phar +``` + +### Using Composer + +The majority of the PHP projects use Composer for managing their PHP packages. +In order to execute Composer before running your tests, simply add the +following in your `.gitlab-ci.yml`: + +```yaml +... + +# Composer stores all downloaded packages in the vendor/ directory. +# Do not use the following if the vendor/ directory is commited to +# your git repository. +cache: + paths: + - vendor/ + +before_script: +# Install composer dependencies +- curl -sS https://getcomposer.org/installer | php +- php composer.phar install + +... +``` + +## Access private packages / dependencies + +If your test suite needs to access a private repository, you need to configure +[the SSH keys](../ssh_keys/README.md) in order to be able to clone it. + +## Use databases or other services + +Most of the time you will need a running database in order for your tests to +run. If you are using the Docker executor you can leverage Docker's ability to +link to other containers. In GitLab Runner lingo, this can be achieved by +defining a `service`. + +This functionality is covered in [the CI services](../services/README.md) +documentation. + +## Testing things locally + +With GitLab Runner 1.0 you can also test any changes locally. From your +terminal execute: + +```bash +# Check using docker executor +gitlab-runner exec docker test:app + +# Check using shell executor +gitlab-runner exec shell test:app +``` + +## Example project + +We have set up an [Example PHP Project][php-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[php-hub]: https://hub.docker.com/_/php/ +[phpenv]: https://github.com/phpenv/phpenv +[phpenv-installation]: https://github.com/phpenv/phpenv#installation +[php-example-repo]: https://gitlab.com/gitlab-examples/php diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index d69064a91fd..a9b36139de9 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -1,44 +1,62 @@ # Quick Start -To start building projects with GitLab CI a few steps needs to be done. +Starting from version 8.0, GitLab Continuous Integration (CI) is fully +integrated into GitLab itself and is enabled by default on all projects. -## 1. Install GitLab and CI +This guide assumes that you: -First you need to have a working GitLab and GitLab CI instance. +- have a working GitLab instance of version 8.0 or higher or are using + [GitLab.com](https://gitlab.com/users/sign_in) +- have a project in GitLab that you would like to use CI for -You can omit this step if you use [GitLab.com](https://GitLab.com/). +In brief, the steps needed to have a working CI can be summed up to: -## 2. Create repository on GitLab +1. Create a new project +1. Add `.gitlab-ci.yml` to the git repository and push to GitLab +1. Configure a Runner -Once you login on your GitLab add a new repository where you will store your source code. -Push your application to that repository. +From there on, on every push to your git repository the build will be +automagically started by the Runner and will appear under the project's +`/builds` page. -## 3. Add project to CI +Now, let's break it down to pieces and work on solving the GitLab CI puzzle. -The next part is to login to GitLab CI. -Point your browser to the URL you have set GitLab or use [gitlab.com/ci](https://gitlab.com/ci/). +## Creating a `.gitlab-ci.yml` file -On the first screen you will see a list of GitLab's projects that you have access to: +Before you create `.gitlab-ci.yml` let's first explain in brief what this is +all about. - +### What is `.gitlab-ci.yml` -Click **Add Project to CI**. -This will create project in CI and authorize GitLab CI to fetch sources from GitLab. +The `.gitlab-ci.yml` file is where you configure what CI does with your project. +It lives in the root of your repository. -> GitLab CI creates unique token that is used to configure GitLab CI service in GitLab. -> This token allows to access GitLab's repository and configures GitLab to trigger GitLab CI webhook on **Push events** and **Tag push events**. -> You can see that token by going to Project's Settings > Services > GitLab CI. -> You will see there token, the same token is assigned in GitLab CI settings of project. +On any push to your repository, GitLab will look for the `.gitlab-ci.yml` +file and start builds on _Runners_ according to the contents of the file, +for that commit. -## 4. Create project's configuration - .gitlab-ci.yml +Because `.gitlab-ci.yml` is in the repository, it is version controlled, +old versions still build succesfully, forks can easily make use of CI, +branches can have separate builds and you have a single source of truth for CI. +You can read more about the reasons why we are using `.gitlab-ci.yml` +[in our blog about it][blog-ci]. -The next: You have to define how your project will be built. -GitLab CI uses [YAML](https://en.wikipedia.org/wiki/YAML) file to store build configuration. -You need to create `.gitlab-ci.yml` in root directory of your repository: +**Note:** `.gitlab-ci.yml` is a [YAML](https://en.wikipedia.org/wiki/YAML) file +so you have to pay extra attention to the identation. Always use spaces, not +tabs. + +### Creating a simple `.gitlab-ci.yml` file + +You need to create a file named `.gitlab-ci.yml` in the root directory of your +repository. Below is an example for a Ruby on Rails project. ```yaml before_script: - - bundle install + - apt-get update -qq && apt-get install -y -qq sqlite3 libsqlite3-dev nodejs + - ruby -v + - which ruby + - gem install bundler --no-ri --no-rdoc + - bundle install --jobs $(nproc) "${FLAGS[@]}" rspec: script: @@ -49,71 +67,131 @@ rubocop: - bundle exec rubocop ``` -This is the simplest possible build configuration that will work for most Ruby applications: -1. Define two jobs `rspec` and `rubocop` with two different commands to be executed. -1. Before every job execute commands defined by `before_script`. +This is the simplest possible build configuration that will work for most Ruby +applications: + +1. Define two jobs `rspec` and `rubocop` (the names are arbitrary) with + different commands to be executed. +1. Before every job, the commands defined by `before_script` are executed. -The `.gitlab-ci.yml` defines set of jobs with constrains how and when they should be run. -The jobs are defined as top-level elements with name and always have to contain the `script`. -Jobs are used to create builds, which are then picked by [runners](../runners/README.md) and executed within environment of the runner. -What is important that each job is run independently from each other. +The `.gitlab-ci.yml` file defines sets of jobs with constraints of how and when +they should be run. The jobs are defined as top-level elements with a name (in +our case `rspec` and `rubocop`) and always have to contain the `script` keyword. +Jobs are used to create builds, which are then picked by +[Runners](../runners/README.md) and executed within the environment of the Runner. -For more information and complete `.gitlab-ci.yml` syntax, please check the [Configuring project (.gitlab-ci.yml)](../yaml/README.md). +What is important is that each job is run independently from each other. -## 5. Add file and push .gitlab-ci.yml to repository +If you want to check whether your `.gitlab-ci.yml` file is valid, there is a +Lint tool under the page `/ci/lint` of your GitLab instance. You can also find +the link under **Settings > CI settings** in your project. -Once you created `.gitlab-ci.yml` you should add it to git repository and push it to GitLab. +For more information and a complete `.gitlab-ci.yml` syntax, please check +[the documentation on .gitlab-ci.yml](../yaml/README.md). + +### Push `.gitlab-ci.yml` to GitLab + +Once you've created `.gitlab-ci.yml`, you should add it to your git repository +and push it to GitLab. ```bash git add .gitlab-ci.yml -git commit +git commit -m "Add .gitlab-ci.yml" git push origin master ``` -If you refresh the project's page on GitLab CI you will notice a one new commit: +Now if you go to the **Builds** page you will see that the builds are pending. + +You can also go to the **Commits** page and notice the little clock icon next +to the commit SHA. + + + +Clicking on the clock icon you will be directed to the builds page for that +specific commit. + + + +Notice that there are two jobs pending which are named after what we wrote in +`.gitlab-ci.yml`. The red triangle indicates that there is no Runner configured +yet for these builds. + +The next step is to configure a Runner so that it picks the pending jobs. + +## Configuring a Runner + +In GitLab, Runners run the builds that you define in `.gitlab-ci.yml`. +A Runner can be a virtual machine, a VPS, a bare-metal machine, a docker +container or even a cluster of containers. GitLab and the Runners communicate +through an API, so the only needed requirement is that the machine on which the +Runner is configured to has Internet access. + +A Runner can be specific to a certain project or serve multiple projects in +GitLab. If it serves all projects it's called a _Shared Runner_. + +Find more information about different Runners in the +[Runners](../runners/README.md) documentation. + +You can find whether any Runners are assigned to your project by going to +**Settings > Runners**. Setting up a Runner is easy and straightforward. The +official Runner supported by GitLab is written in Go and can be found at +<https://gitlab.com/gitlab-org/gitlab-ci-multi-runner>. + +In order to have a functional Runner you need to follow two steps: + +1. [Install it][runner-install] +2. [Configure it](../runners/README.md#registering-a-specific-runner) + +Follow the links above to set up your own Runner or use a Shared Runner as +described in the next section. + +For other types of unofficial Runners written in other languages, see the +[instructions for the various GitLab Runners](https://about.gitlab.com/gitlab-ci/#gitlab-runner). + +Once the Runner has been set up, you should see it on the Runners page of your +project, following **Settings > Runners**. - + -However the commit has status **pending** which means that commit was not yet picked by runner. +### Shared Runners -## 6. Configure runner +If you use [GitLab.com](https://gitlab.com/) you can use **Shared Runners** +provided by GitLab Inc. -In GitLab CI, Runners run your builds. -A runner is a machine (can be virtual, bare-metal or VPS) that picks up builds through the coordinator API of GitLab CI. +These are special virtual machines that run on GitLab's infrastructure and can +build any project. -A runner can be specific to a certain project or serve any project in GitLab CI. -A runner that serves all projects is called a shared runner. -More information about different runner types can be found in [Configuring runner](../runners/README.md). +To enable **Shared Runners** you have to go to your project's +**Settings > Runners** and click **Enable shared runners**. -To check if you have runners assigned to your project go to **Runners**. You will find there information how to setup project specific runner: +[Read more on Shared Runners](../runners/README.md). -1. Install GitLab Runner software. Checkout the [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner) section to install it. -1. Specify following URL during runner setup: https://gitlab.com/ci/ -1. Use the following registration token during setup: TOKEN +## Seeing the status of your build -If you do it correctly your runner should be shown under **Runners activated for this project**: +After configuring the Runner succesfully, you should see the status of your +last commit change from _pending_ to either _running_, _success_ or _failed_. - +You can view all builds, by going to the **Builds** page in your project. -### Shared runners + -If you use [gitlab.com/ci](https://gitlab.com/ci/) you can use **Shared runners** provided by GitLab Inc. -These are special virtual machines that are run on GitLab's infrastructure that can build any project. -To enable **Shared runners** you have to go to **Runners** and click **Enable shared runners** for this project. +By clicking on a Build ID, you will be able to see the log of that build. +This is important to diagnose why a build failed or acted differently than +you expected. -## 7. Check status of commit + -If everything went OK and you go to commit, the status of the commit should change from **pending** to either **running**, **success** or **failed**. +You are also able to view the status of any commit in the various pages in +GitLab, such as **Commits** and **Merge Requests**. - +## Next steps -You can click **Build ID** to view build log for specific job. +Awesome! You started using CI in GitLab! -## 8. Congratulations! +Next you can look into doing more with the CI. Many people are using GitLab +to package, containerize, test and deploy software. -You managed to build your first project using GitLab CI. -You may need to tune your `.gitlab-ci.yml` file to implement build plan for your project. -A few examples how it can be done you can find on [Examples](../examples/README.md) page. +Visit our various languages examples at <https://gitlab.com/groups/gitlab-examples>. -GitLab CI also offers **the Lint** tool to verify validity of your `.gitlab-ci.yml` which can be useful to troubleshoot potential problems. -The Lint is available from project's settings or by adding `/lint` to GitLab CI url. +[runner-install]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/tree/master#installation +[blog-ci]: https://about.gitlab.com/2015/05/06/why-were-replacing-gitlab-ci-jobs-with-gitlab-ci-dot-yml/ diff --git a/doc/ci/quick_start/build_status.png b/doc/ci/quick_start/build_status.png Binary files differdeleted file mode 100644 index 333259e6acd..00000000000 --- a/doc/ci/quick_start/build_status.png +++ /dev/null diff --git a/doc/ci/quick_start/commit_status.png b/doc/ci/quick_start/commit_status.png Binary files differdeleted file mode 100644 index 725b79e6f91..00000000000 --- a/doc/ci/quick_start/commit_status.png +++ /dev/null diff --git a/doc/ci/quick_start/img/build_log.png b/doc/ci/quick_start/img/build_log.png Binary files differnew file mode 100644 index 00000000000..89e6cd40cb6 --- /dev/null +++ b/doc/ci/quick_start/img/build_log.png diff --git a/doc/ci/quick_start/img/builds_status.png b/doc/ci/quick_start/img/builds_status.png Binary files differnew file mode 100644 index 00000000000..b8e6c2a361a --- /dev/null +++ b/doc/ci/quick_start/img/builds_status.png diff --git a/doc/ci/quick_start/img/new_commit.png b/doc/ci/quick_start/img/new_commit.png Binary files differnew file mode 100644 index 00000000000..3d3c9d5c0bd --- /dev/null +++ b/doc/ci/quick_start/img/new_commit.png diff --git a/doc/ci/quick_start/img/runners_activated.png b/doc/ci/quick_start/img/runners_activated.png Binary files differnew file mode 100644 index 00000000000..eafcfd6ecd5 --- /dev/null +++ b/doc/ci/quick_start/img/runners_activated.png diff --git a/doc/ci/quick_start/img/single_commit_status_pending.png b/doc/ci/quick_start/img/single_commit_status_pending.png Binary files differnew file mode 100644 index 00000000000..23b3bb5acfc --- /dev/null +++ b/doc/ci/quick_start/img/single_commit_status_pending.png diff --git a/doc/ci/quick_start/img/status_pending.png b/doc/ci/quick_start/img/status_pending.png Binary files differnew file mode 100644 index 00000000000..a049ec2a5ba --- /dev/null +++ b/doc/ci/quick_start/img/status_pending.png diff --git a/doc/ci/quick_start/new_commit.png b/doc/ci/quick_start/new_commit.png Binary files differdeleted file mode 100644 index 3839e893c17..00000000000 --- a/doc/ci/quick_start/new_commit.png +++ /dev/null diff --git a/doc/ci/quick_start/projects.png b/doc/ci/quick_start/projects.png Binary files differdeleted file mode 100644 index 0b3430a69db..00000000000 --- a/doc/ci/quick_start/projects.png +++ /dev/null diff --git a/doc/ci/quick_start/runners.png b/doc/ci/quick_start/runners.png Binary files differdeleted file mode 100644 index 25b4046bc00..00000000000 --- a/doc/ci/quick_start/runners.png +++ /dev/null diff --git a/doc/ci/quick_start/runners_activated.png b/doc/ci/quick_start/runners_activated.png Binary files differdeleted file mode 100644 index c934bd12f41..00000000000 --- a/doc/ci/quick_start/runners_activated.png +++ /dev/null diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md new file mode 100644 index 00000000000..1ebb0a4a250 --- /dev/null +++ b/doc/ci/services/README.md @@ -0,0 +1,9 @@ +## GitLab CI Services + +GitLab CI uses the `services` keyword to define what docker containers should be +linked with your base image. Below is a list of examples you may use. + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) ++ [Using Other Services](../docker/using_docker_images.md#how-to-use-other-images-as-services) diff --git a/doc/ci/services/docker-services.md b/doc/ci/services/docker-services.md new file mode 100644 index 00000000000..df36ebaf7d4 --- /dev/null +++ b/doc/ci/services/docker-services.md @@ -0,0 +1,5 @@ +## GitLab CI Services + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md new file mode 100644 index 00000000000..c66d77122b2 --- /dev/null +++ b/doc/ci/services/mysql.md @@ -0,0 +1,118 @@ +# Using MySQL + +As many applications depend on MySQL as their database, you will eventually +need it in order for your tests to run. Below you are guided how to do this +with the Docker and Shell executors of GitLab Runner. + +## Use MySQL with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - mysql:latest + +variables: + # Configure mysql environment variables (https://hub.docker.com/_/mysql/) + MYSQL_DATABASE: el_duderino + MYSQL_ROOT_PASSWORD: mysql_strong_password +``` + +And then configure your application to use the database, for example: + +```yaml +Host: mysql +User: root +Password: mysql_strong_password +Database: el_duderino +``` + +If you are wondering why we used `mysql` for the `Host`, read more at +[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build). + +You can also use any other docker image available on [Docker Hub][hub-mysql]. +For example, to use MySQL 5.5 the service becomes `mysql:5.5`. + +The `mysql` image can accept some environment variables. For more details +check the documentation on [Docker Hub][hub-mysql]. + +## Use MySQL with the Shell executor + +You can also use MySQL on manually configured servers that are using +GitLab Runner with the Shell executor. + +First install the MySQL server: + +```bash +sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev +``` + +Pick a MySQL root password (can be anything), and type it twice when asked. + +*Note: As a security measure you can run `mysql_secure_installation` to +remove anonymous users, drop the test database and disable remote logins with +the root user.* + +The next step is to create a user, so login to MySQL as root: + +```bash +mysql -u root -p +``` + +Then create a user (in our case `runner`) which will be used by your +application. Change `$password` in the command below to a real strong password. + +*Note: Do not type `mysql>`, this is part of the MySQL prompt.* + +```bash +mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password'; +``` + +Create the database: + +```bash +mysql> CREATE DATABASE IF NOT EXISTS `el_duderino` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; +``` + +Grant the necessary permissions on the database: + +```bash +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `el_duderino`.* TO 'runner'@'localhost'; +``` + +If all went well you can now quit the database session: + +```bash +mysql> \q +``` + +Now, try to connect to the newly created database to check that everything is +in place: + +```bash +mysql -u runner -p -D el_duderino +``` + +As a final step, configure your application to use the database, for example: + +```bash +Host: localhost +User: runner +Password: $password +Database: el_duderino +``` + +## Example project + +We have set up an [Example MySQL Project][mysql-example-repo] for your +convenience that runs on [GitLab.com](https://gitlab.com) using our publicly +available [shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-mysql]: https://hub.docker.com/_/mysql/ +[mysql-example-repo]: https://gitlab.com/gitlab-examples/mysql diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md new file mode 100644 index 00000000000..17d21dbda1c --- /dev/null +++ b/doc/ci/services/postgres.md @@ -0,0 +1,114 @@ +# Using PostgreSQL + +As many applications depend on PostgreSQL as their database, you will +eventually need it in order for your tests to run. Below you are guided how to +do this with the Docker and Shell executors of GitLab Runner. + +## Use PostgreSQL with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - postgres:latest + +variables: + POSTGRES_DB: nice_marmot + POSTGRES_USER: runner + POSTGRES_PASSWORD: "" +``` + +And then configure your application to use the database, for example: + +```yaml +Host: postgres +User: runner +Password: +Database: nice_marmot +``` + +If you are wondering why we used `postgres` for the `Host`, read more at +[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build). + +You can also use any other docker image available on [Docker Hub][hub-pg]. +For example, to use PostgreSQL 9.3 the service becomes `postgres:9.3`. + +The `postgres` image can accept some environment variables. For more details +check the documentation on [Docker Hub][hub-pg]. + +## Use PostgreSQL with the Shell executor + +You can also use PostgreSQL on manually configured servers that are using +GitLab Runner with the Shell executor. + +First install the PostgreSQL server: + +```bash +sudo apt-get install -y postgresql postgresql-client libpq-dev +``` + +The next step is to create a user, so login to PostgreSQL: + +```bash +sudo -u postgres psql -d template1 +``` + +Then create a user (in our case `runner`) which will be used by your +application. Change `$password` in the command below to a real strong password. + +*__Note:__ Do not type `template1=#`, this is part of the PostgreSQL prompt.* + +```bash +template1=# CREATE USER runner WITH PASSWORD '$password' CREATEDB; +``` + +*__Note:__ Notice that we created the user with the privilege to be able to +create databases (`CREATEDB`). In the following steps we will create a database +explicitly for that user but having that privilege can be useful if in your +testing framework you have tools that drop and create databases.* + +Create the database and grant all privileges on it for the user `runner`: + +```bash +template1=# CREATE DATABASE nice_marmot OWNER runner; +``` + +If all went well you can now quit the database session: + +```bash +template1=# \q +``` + +Now, try to connect to the newly created database with the user `runner` to +check that everything is in place. + +```bash +psql -U runner -h localhost -d nice_marmot -W +``` + +*__Note:__ We are explicitly telling `psql` to connect to localhost in order +to use the md5 authentication. If you omit this step you will be denied access.* + +Finally, configure your application to use the database, for example: + +```yaml +Host: localhost +User: runner +Password: $password +Database: nice_marmot +``` + +## Example project + +We have set up an [Example PostgreSQL Project][postgres-example-repo] for your +convenience that runs on [GitLab.com](https://gitlab.com) using our publicly +available [shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-pg]: https://hub.docker.com/_/postgres/ +[postgres-example-repo]: https://gitlab.com/gitlab-examples/postgres diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md new file mode 100644 index 00000000000..b281e8f9f60 --- /dev/null +++ b/doc/ci/services/redis.md @@ -0,0 +1,69 @@ +# Using Redis + +As many applications depend on Redis as their key-value store, you will +eventually need it in order for your tests to run. Below you are guided how to +do this with the Docker and Shell executors of GitLab Runner. + +## Use Redis with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - redis:latest +``` + +Then you need to configure your application to use the Redis database, for +example: + +```yaml +Host: redis +``` + +And that's it. Redis will now be available to be used within your testing +framework. + +You can also use any other docker image available on [Docker Hub][hub-redis]. +For example, to use Redis 2.8 the service becomes `redis:2.8`. + +## Use Redis with the Shell executor + +Redis can also be used on manually configured servers that are using GitLab +Runner with the Shell executor. + +In your build machine install the Redis server: + +```bash +sudo apt-get install redis-server +``` + +Verify that you can connect to the server with the `gitlab-runner` user: + +```bash +# Try connecting the the Redis server +sudo -u gitlab-runner -H redis-cli + +# Quit the session +127.0.0.1:6379> quit +``` + +Finally, configure your application to use the database, for example: + +```yaml +Host: localhost +``` + +## Example project + +We have set up an [Example Redis Project][redis-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-redis]: https://hub.docker.com/_/redis/ +[redis-example-repo]: https://gitlab.com/gitlab-examples/redis diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md new file mode 100644 index 00000000000..210f9c3e849 --- /dev/null +++ b/doc/ci/ssh_keys/README.md @@ -0,0 +1,109 @@ +# Using SSH keys + +GitLab currently doesn't have built-in support for managing SSH keys in a build +environment. + +The SSH keys can be useful when: + +1. You want to checkout internal submodules +2. You want to download private packages using your package manager (eg. bundler) +3. You want to deploy your application to eg. Heroku or your own server +4. You want to execute SSH commands from the build server to the remote server +5. You want to rsync files from your build server to the remote server + +If anything of the above rings a bell, then you most likely need an SSH key. + +## Inject keys in your build server + +The most widely supported method is to inject an SSH key into your build +environment by extending your `.gitlab-ci.yml`. + +This is the universal solution which works with any type of executor +(docker, shell, etc.). + +### How it works + +1. Create a new SSH key pair with [ssh-keygen][] +2. Add the private key as a **Secret Variable** to the project +3. Run the [ssh-agent][] during build to load the private key. + +## SSH keys when using the Docker executor + +You will first need to create an SSH key pair. For more information, follow the +instructions to [generate an SSH key](../ssh/README.md). + +Then, create a new **Secret Variable** in your project settings on GitLab +following **Settings > Variables**. As **Key** add the name `SSH_PRIVATE_KEY` +and in the **Value** field paste the content of your _private_ key that you +created earlier. + +Next you need to modify your `.gitlab-ci.yml` with a `before_script` action. +Add it to the top: + +``` +before_script: + # Install ssh-agent if not already installed, it is required by Docker. + # (change apt-get to yum if you use a CentOS-based image) + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + + # Run ssh-agent (inside the build environment) + - eval $(ssh-agent -s) + + # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store + - ssh-add <(echo "$SSH_PRIVATE_KEY") + + # For Docker builds disable host key checking. Be aware that by adding that + # you are suspectible to man-in-the-middle attacks. + # WARNING: Use this only with the Docker executor, if you use it with shell + # you will overwrite your user's SSH config. + - mkdir -p ~/.ssh + - '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config` +``` + +As a final step, add the _public_ key from the one you created earlier to the +services that you want to have an access to from within the build environment. +If you are accessing a private GitLab repository you need to add it as a +[deploy key](../ssh/README.md#deploy-keys). + +That's it! You can now have access to private servers or repositories in your +build environment. + +## SSH keys when using the Shell executor + +If you are using the Shell executor and not Docker, it is easier to set up an +SSH key. + +You can generate the SSH key from the machine that GitLab Runner is installed +on, and use that key for all projects that are run on this machine. + +First, you need to login to the server that runs your builds. + +Then from the terminal login as the `gitlab-runner` user and generate the SSH +key pair as described in the [SSH keys documentation](../ssh/README.md). + +As a final step, add the _public_ key from the one you created earlier to the +services that you want to have an access to from within the build environment. +If you are accessing a private GitLab repository you need to add it as a +[deploy key](../ssh/README.md#deploy-keys). + +Once done, try to login to the remote server in order to accept the fingerprint: + +```bash +ssh <address-of-my-server> +``` + +For accessing repositories on GitLab.com, the `<address-of-my-server>` would be +`git@gitlab.com`. + +## Example project + +We have set up an [Example SSH Project][ssh-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[ssh-keygen]: http://linux.die.net/man/1/ssh-keygen +[ssh-agent]: http://linux.die.net/man/1/ssh-agent +[ssh-example-repo]: https://gitlab.com/gitlab-examples/ssh-private-key/ diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 022afb70042..b99ea25a3fe 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -27,7 +27,6 @@ The API_TOKEN will take the Secure Variable value: `SECURE`. | **CI_BUILD_TAG** | 0.5 | The commit tag name. Present only when building tags. | | **CI_BUILD_NAME** | 0.5 | The name of the build as defined in `.gitlab-ci.yml` | | **CI_BUILD_STAGE** | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| **CI_BUILD_BEFORE_SHA** | all | The first commit that were included in push request | | **CI_BUILD_REF_NAME** | all | The branch or tag name for which project is built | | **CI_BUILD_ID** | all | The unique id of the current build that GitLab CI uses internally | | **CI_BUILD_REPO** | all | The URL to clone the Git repository | @@ -40,7 +39,6 @@ The API_TOKEN will take the Secure Variable value: `SECURE`. Example values: ```bash -export CI_BUILD_BEFORE_SHA="9df57456fa9de2a6d335ca5edf9750ed812b9df0" export CI_BUILD_ID="50" export CI_BUILD_REF="1ecfd275763eff1d6b4844ea3168962458c9f27a" export CI_BUILD_REF_NAME="master" diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 3e6071a2ee7..fd0d49de4e4 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,9 +1,12 @@ # Configuration of your builds with .gitlab-ci.yml -From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (**.gitlab-ci.yml**) for the project configuration. -It is placed in the root of your repository and contains definitions of how your project should be built. -The YAML file defines a set of jobs with constraints stating when they should be run. -The jobs are defined as top-level elements with a name and always have to contain the `script` clause: +From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) +file (`.gitlab-ci.yml`) for the project configuration. It is placed in the root +of your repository and contains definitions of how your project should be built. + +The YAML file defines a set of jobs with constraints stating when they should +be run. The jobs are defined as top-level elements with a name and always have +to contain the `script` clause: ```yaml job1: @@ -13,15 +16,21 @@ job2: script: "execute-script-for-job2" ``` -The above example is the simplest possible CI configuration with two separate jobs, -where each of the jobs executes a different command. -Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. +The above example is the simplest possible CI configuration with two separate +jobs, where each of the jobs executes a different command. + +Of course a command can execute code directly (`./configure;make;make install`) +or run a script (`test.sh`) in the repository. -Jobs are used to create builds, which are then picked up by [runners](../runners/README.md) and executed within the environment of the runner. -What is important, is that each job is run independently from each other. +Jobs are used to create builds, which are then picked up by +[runners](../runners/README.md) and executed within the environment of the +runner. What is important, is that each job is run independently from each +other. ## .gitlab-ci.yml -The YAML syntax allows for using more complex job specifications than in the above example: + +The YAML syntax allows for using more complex job specifications than in the +above example: ```yaml image: ruby:2.1 @@ -46,26 +55,31 @@ job1: - docker ``` -There are a few `keywords` that can't be used as job names: +There are a few reserved `keywords` that **cannot** be used as job names: -| keyword | required | description | +| Keyword | Required | Description | |---------------|----------|-------------| -| image | optional | Use docker image, covered in [Use Docker](../docker/README.md) | -| services | optional | Use docker services, covered in [Use Docker](../docker/README.md) | -| stages | optional | Define build stages | -| types | optional | Alias for `stages` | -| before_script | optional | Define commands prepended for each job's script | -| variables | optional | Define build variables | -| cache | optional | Define list of files that should be cached between subsequent runs | +| image | no | Use docker image, covered in [Use Docker](../docker/README.md) | +| services | no | Use docker services, covered in [Use Docker](../docker/README.md) | +| stages | no | Define build stages | +| types | no | Alias for `stages` | +| before_script | no | Define commands that run before each job's script | +| variables | no | Define build variables | +| cache | no | Define list of files that should be cached between subsequent runs | ### image and services -This allows to specify a custom Docker image and a list of services that can be used for time of the build. -The configuration of this feature is covered in separate document: [Use Docker](../docker/README.md). + +This allows to specify a custom Docker image and a list of services that can be +used for time of the build. The configuration of this feature is covered in +separate document: [Use Docker](../docker/README.md). ### before_script -`before_script` is used to define the command that should be run before all builds, including deploy builds. This can be an array or a multiline string. + +`before_script` is used to define the command that should be run before all +builds, including deploy builds. This can be an array or a multi-line string. ### stages + `stages` is used to define build stages that can be used by jobs. The specification of `stages` allows for having flexible multi stage pipelines. @@ -75,7 +89,8 @@ The ordering of elements in `stages` defines the ordering of builds' execution: 1. Builds of next stage are run after success. Let's consider the following example, which defines 3 stages: -``` + +```yaml stages: - build - test @@ -86,21 +101,26 @@ stages: 1. If all jobs of `build` succeeds, the `test` jobs are executed in parallel. 1. If all jobs of `test` succeeds, the `deploy` jobs are executed in parallel. 1. If all jobs of `deploy` succeeds, the commit is marked as `success`. -1. If any of the previous jobs fails, the commit is marked as `failed` and no jobs of further stage are executed. +1. If any of the previous jobs fails, the commit is marked as `failed` and no + jobs of further stage are executed. There are also two edge cases worth mentioning: -1. If no `stages` is defined in `.gitlab-ci.yml`, then by default the `build`, `test` and `deploy` are allowed to be used as job's stage by default. +1. If no `stages` is defined in `.gitlab-ci.yml`, then by default the `build`, + `test` and `deploy` are allowed to be used as job's stage by default. 2. If a job doesn't specify `stage`, the job is assigned the `test` stage. ### types + Alias for [stages](#stages). ### variables -**This feature requires `gitlab-runner` with version equal or greater than 0.5.0.** -GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build environment. -The variables are stored in repository and are meant to store non-sensitive project configuration, ie. RAILS_ENV or DATABASE_URL. +_**Note:** Introduced in GitLab Runner v0.5.0._ + +GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build +environment. The variables are stored in the git repository and are meant to +store non-sensitive project configuration, for example: ```yaml variables: @@ -109,15 +129,23 @@ variables: These variables can be later used in all executed commands and scripts. -The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. +The YAML-defined variables are also set to all created service containers, +thus allowing to fine tune them. ### cache -`cache` is used to specify list of files and directories which should be cached between builds. -**The global setting allows to specify default cached files for all jobs.** +`cache` is used to specify a list of files and directories which should be +cached between builds. Caches are stored according to the branch/ref and the +job name. They are not currently shared between different job names or between +branches/refs, which means that caching will benefit you if you push subsequent +commits to an existing feature branch. + +If `cache` is defined outside the scope of the jobs, it means it is set +globally and all jobs will use its definition. To cache all git untracked files and files in `binaries`: -``` + +```yaml cache: untracked: true paths: @@ -125,9 +153,10 @@ cache: ``` ## Jobs -`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. -Each job has to have a unique `job_name`, which is not one of the keywords mentioned above. -A job is defined by a list of parameters that define the build behaviour. + +`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job +must have a unique name, which is not one of the Keywords mentioned above. +A job is defined by a list of parameters that define the build behavior. ```yaml job_name: @@ -145,21 +174,22 @@ job_name: allow_failure: true ``` -| keyword | required | description | +| Keyword | Required | Description | |---------------|----------|-------------| -| script | required | Defines a shell script which is executed by runner | -| stage | optional (default: test) | Defines a build stage | -| type | optional | Alias for `stage` | -| only | optional | Defines a list of git refs for which build is created | -| except | optional | Defines a list of git refs for which build is not created | -| tags | optional | Defines a list of tags which are used to select runner | -| allow_failure | optional | Allow build to fail. Failed build doesn't contribute to commit status | -| when | optional | Define when to run build. Can be `on_success`, `on_failure` or `always` | -| artifacts | optional | Define list build artifacts | -| cache | optional | Define list of files that should be cached between subsequent runs | +| script | yes | Defines a shell script which is executed by runner | +| stage | no (default: `test`) | Defines a build stage | +| type | no | Alias for `stage` | +| only | no | Defines a list of git refs for which build is created | +| except | no | Defines a list of git refs for which build is not created | +| tags | no | Defines a list of tags which are used to select runner | +| allow_failure | no | Allow build to fail. Failed build doesn't contribute to commit status | +| when | no | Define when to run build. Can be `on_success`, `on_failure` or `always` | +| artifacts | no | Define list build artifacts | +| cache | no | Define list of files that should be cached between subsequent runs | ### script -`script` is a shell script which is executed by runner. The shell script is prepended with `before_script`. + +`script` is a shell script which is executed by the runner. For example: ```yaml job: @@ -167,6 +197,7 @@ job: ``` This parameter can also contain several commands using an array: + ```yaml job: script: @@ -175,31 +206,45 @@ job: ``` ### stage -`stage` allows to group build into different stages. Builds of the same `stage` are executed in `parallel`. -For more info about the use of `stage` please check the [stages](#stages). + +`stage` allows to group build into different stages. Builds of the same `stage` +are executed in `parallel`. For more info about the use of `stage` please check +[stages](#stages). ### only and except -This are two parameters that allow for setting a refs policy to limit when jobs are built: -1. `only` defines the names of branches and tags for which job will be built. -2. `except` defines the names of branches and tags for which the job wil **not** be built. -There are a few rules that apply to usage of refs policy: +`only` and `except` are two parameters that set a refs policy to limit when +jobs are built: + +1. `only` defines the names of branches and tags for which the job will be + built. +2. `except` defines the names of branches and tags for which the job will + **not** be built. -1. `only` and `except` are inclusive. If both `only` and `except` are defined in job specification the ref is filtered by `only` and `except`. -1. `only` and `except` allow for using the regexp expressions. -1. `only` and `except` allow for using special keywords: `branches` and `tags`. -These names can be used for example to exclude all tags and all branches. +There are a few rules that apply to the usage of refs policy: + +* `only` and `except` are inclusive. If both `only` and `except` are defined + in a job specification, the ref is filtered by `only` and `except`. +* `only` and `except` allow the use of regular expressions. +* `only` and `except` allow the use of special keywords: `branches` and `tags`. +* `only` and `except` allow to specify a repository path to filter jobs for + forks. + +In the example below, `job` will run only for refs that start with `issue-`, +whereas all branches will be skipped. ```yaml job: + # use regexp only: - - /^issue-.*$/ # use regexp + - /^issue-.*$/ + # use special keyword except: - - branches # use special keyword + - branches ``` -1. `only` and `except` allow for specify repository path to filter jobs for forks. -The repository path can be used to have jobs executed only for parent repository. +The repository path can be used to have jobs executed only for the parent +repository and not forks: ```yaml job: @@ -208,33 +253,47 @@ job: except: - master@gitlab-org/gitlab-ce ``` -The above will run `job` for all branches on `gitlab-org/gitlab-ce`, except master . + +The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, +except master. ### tags -`tags` is used to select specific runners from the list of all runners that are allowed to run this project. -During registration of a runner, you can specify the runner's tags, ie.: `ruby`, `postgres`, `development`. -`tags` allow you to run builds with runners that have the specified tags assigned: +`tags` is used to select specific runners from the list of all runners that are +allowed to run this project. -``` +During the registration of a runner, you can specify the runner's tags, for +example `ruby`, `postgres`, `development`. + +`tags` allow you to run builds with runners that have the specified tags +assigned to them: + +```yaml job: tags: - ruby - postgres ``` -The above specification will make sure that `job` is built by a runner that have `ruby` AND `postgres` tags defined. +The specification above, will make sure that `job` is built by a runner that +has both `ruby` AND `postgres` tags defined. ### when -`when` is used to implement jobs that are run in case of failure or despite the failure. + +`when` is used to implement jobs that are run in case of failure or despite the +failure. `when` can be set to one of the following values: -1. `on_success` - execute build only when all builds from prior stages succeeded. This is the default. -1. `on_failure` - execute build only when at least one build from prior stages failed. +1. `on_success` - execute build only when all builds from prior stages + succeeded. This is the default. +1. `on_failure` - execute build only when at least one build from prior stages + failed. 1. `always` - execute build despite the status of builds from prior stages. -``` +For example: + +```yaml stages: - build - cleanup_build @@ -242,28 +301,28 @@ stages: - deploy - cleanup -build: +build_job: stage: build script: - make build -cleanup_build: +cleanup_build_job: stage: cleanup_build script: - cleanup build when failed when: on_failure -test: +test_job: stage: test script: - make test -deploy: +deploy_job: stage: deploy script: - make deploy -cleanup: +cleanup_job: stage: cleanup script: - cleanup after builds @@ -271,43 +330,56 @@ cleanup: ``` The above script will: -1. Execute `cleanup_build` only when the `build` failed, -2. Always execute `cleanup` as the last step in pipeline. + +1. Execute `cleanup_build_job` only when `build_job` fails +2. Always execute `cleanup_job` as the last step in pipeline. ### artifacts -`artifacts` is used to specify list of files and directories which should be attached to build after success. -1. Send all files in `binaries` and `.config`: -``` +_**Note:** Introduced in GitLab Runner v0.7.0. Also, the Windows shell executor + does not currently support artifact uploads._ + +`artifacts` is used to specify list of files and directories which should be +attached to build after success. Below are some examples. + +Send all files in `binaries` and `.config`: + +```yaml artifacts: paths: - binaries/ - .config ``` -2. Send all git untracked files: -``` +Send all git untracked files: + +```yaml artifacts: untracked: true ``` -3. Send all git untracked files and files in `binaries`: -``` +Send all git untracked files and files in `binaries`: + +```yaml artifacts: untracked: true paths: - binaries/ ``` -The artifacts will be send after the build success to GitLab and will be accessible in GitLab interface to download. - -This feature requires GitLab Runner v0.7.0 or higher. +The artifacts will be send after a successful build success to GitLab, and will +be accessible in the GitLab UI to download. ### cache -`cache` is used to specify list of files and directories which should be cached between builds. -1. Cache all files in `binaries` and `.config`: -``` +_**Note:** Introduced in GitLab Runner v0.7.0._ + +`cache` is used to specify list of files and directories which should be cached +between builds. Below are some examples: + +Cache all files in `binaries` and `.config`: + +```yaml rspec: script: test cache: @@ -316,16 +388,18 @@ rspec: - .config ``` -2. Cache all git untracked files: -``` +Cache all git untracked files: + +```yaml rspec: script: test cache: untracked: true ``` -3. Cache all git untracked files and files in `binaries`: -``` +Cache all git untracked files and files in `binaries`: + +```yaml rspec: script: test cache: @@ -334,9 +408,10 @@ rspec: - binaries/ ``` -4. Locally defined cache overwrites globally defined options. This will cache only `binaries/`: +Locally defined cache overwrites globally defined options. This will cache only +`binaries/`: -``` +```yaml cache: paths: - my/files @@ -348,15 +423,15 @@ rspec: - binaries/ ``` -The cache is provided on best effort basis, so don't expect that cache will be present. -For implementation details please check GitLab Runner. - -This feature requires GitLab Runner v0.7.0 or higher. - +The cache is provided on best effort basis, so don't expect that cache will be +always present. For implementation details please check GitLab Runner. ## Validate the .gitlab-ci.yml + Each instance of GitLab CI has an embedded debug tool called Lint. -You can find the link to the Lint in the project's settings page or use short url `/lint`. +You can find the link under `/ci/lint` of your gitlab instance. ## Skipping builds -There is one more way to skip all builds, if your commit message contains tag [ci skip]. In this case, commit will be created but builds will be skipped + +If your commit message contains `[ci skip]`, the commit will be created but the +builds will be skipped. |
