diff options
| author | Phil Hughes <me@iamphill.com> | 2018-03-21 09:09:48 +0000 |
|---|---|---|
| committer | Phil Hughes <me@iamphill.com> | 2018-03-21 09:09:48 +0000 |
| commit | c740047b50f3dc2c93a296a69a8ecdf70a8ef4fc (patch) | |
| tree | d3e0b1bb81ae1416859c8e95f09fc4bccacb81a9 /doc/articles | |
| parent | 5d400cc6d3249f4e2355f7bf3be1b39bf4ceaacf (diff) | |
| parent | 2025f361e7520e00962f6964aa77c4f4e6c3d56b (diff) | |
| download | gitlab-ce-c740047b50f3dc2c93a296a69a8ecdf70a8ef4fc.tar.gz | |
Merge branch 'master' into notebook-multiple-outputs
Diffstat (limited to 'doc/articles')
43 files changed, 20 insertions, 2821 deletions
diff --git a/doc/articles/artifactory_and_gitlab/index.md b/doc/articles/artifactory_and_gitlab/index.md index c64851bad2b..6a590b53727 100644 --- a/doc/articles/artifactory_and_gitlab/index.md +++ b/doc/articles/artifactory_and_gitlab/index.md @@ -1,278 +1 @@ -# How to deploy Maven projects to Artifactory with GitLab CI/CD - -> **Article [Type](../../development/writing_documentation.md#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Fabio Busatto](https://gitlab.com/bikebilly) || -> **Publication date:** 2017-08-15 - -## Introduction - -In this article, we will show how you can leverage the power of [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/) -to build a [Maven](https://maven.apache.org/) project, deploy it to [Artifactory](https://www.jfrog.com/artifactory/), and then use it from another Maven application as a dependency. - -You'll create two different projects: - -- `simple-maven-dep`: the app built and deployed to Artifactory (available at https://gitlab.com/gitlab-examples/maven/simple-maven-dep) -- `simple-maven-app`: the app using the previous one as a dependency (available at https://gitlab.com/gitlab-examples/maven/simple-maven-app) - -We assume that you already have a GitLab account on [GitLab.com](https://gitlab.com/), and that you know the basic usage of Git and [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/). -We also assume that an Artifactory instance is available and reachable from the internet, and that you have valid credentials to deploy on it. - -## Create the simple Maven dependency - -First of all, you need an application to work with: in this specific case we will -use a simple one, but it could be any Maven application. This will be the -dependency you want to package and deploy to Artifactory, in order to be -available to other projects. - -### Prepare the dependency application - -For this article you'll use a Maven app that can be cloned from our example -project: - -1. Log in to your GitLab account -1. Create a new project by selecting **Import project from ➔ Repo by URL** -1. Add the following URL: - - ``` - https://gitlab.com/gitlab-examples/maven/simple-maven-dep.git - ``` -1. Click **Create project** - -This application is nothing more than a basic class with a stub for a JUnit based test suite. -It exposes a method called `hello` that accepts a string as input, and prints a hello message on the screen. - -The project structure is really simple, and you should consider these two resources: - -- `pom.xml`: project object model (POM) configuration file -- `src/main/java/com/example/dep/Dep.java`: source of our application - -### Configure the Artifactory deployment - -The application is ready to use, but you need some additional steps to deploy it to Artifactory: - -1. Log in to Artifactory with your user's credentials. -1. From the main screen, click on the `libs-release-local` item in the **Set Me Up** panel. -1. Copy to clipboard the configuration snippet under the **Deploy** paragraph. -1. Change the `url` value in order to have it configurable via secret variables. -1. Copy the snippet in the `pom.xml` file for your project, just after the - `dependencies` section. The snippet should look like this: - - ```xml - <distributionManagement> - <repository> - <id>central</id> - <name>83d43b5afeb5-releases</name> - <url>${env.MAVEN_REPO_URL}/libs-release-local</url> - </repository> - </distributionManagement> - ``` - -Another step you need to do before you can deploy the dependency to Artifactory -is to configure the authentication data. It is a simple task, but Maven requires -it to stay in a file called `settings.xml` that has to be in the `.m2` subdirectory -in the user's homedir. - -Since you want to use GitLab Runner to automatically deploy the application, you -should create the file in the project's home directory and set a command line -parameter in `.gitlab-ci.yml` to use the custom location instead of the default one: - -1. Create a folder called `.m2` in the root of your repository -1. Create a file called `settings.xml` in the `.m2` folder -1. Copy the following content into a `settings.xml` file: - - ```xml - <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" - xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> - <servers> - <server> - <id>central</id> - <username>${env.MAVEN_REPO_USER}</username> - <password>${env.MAVEN_REPO_PASS}</password> - </server> - </servers> - </settings> - ``` - - Username and password will be replaced by the correct values using secret variables. - -### Configure GitLab CI/CD for `simple-maven-dep` - -Now it's time we set up [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/) to automatically build, test and deploy the dependency! - -GitLab CI/CD uses a file in the root of the repo, named `.gitlab-ci.yml`, to read the definitions for jobs -that will be executed by the configured GitLab Runners. You can read more about this file in the [GitLab Documentation](https://docs.gitlab.com/ee/ci/yaml/). - -First of all, remember to set up secret variables for your deployment. Navigate to your project's **Settings > CI/CD** page -and add the following secret variables (replace them with your current values, of course): - -- **MAVEN_REPO_URL**: `http://artifactory.example.com:8081/artifactory` (your Artifactory URL) -- **MAVEN_REPO_USER**: `gitlab` (your Artifactory username) -- **MAVEN_REPO_PASS**: `AKCp2WXr3G61Xjz1PLmYa3arm3yfBozPxSta4taP3SeNu2HPXYa7FhNYosnndFNNgoEds8BCS` (your Artifactory Encrypted Password) - -Now it's time to define jobs in `.gitlab-ci.yml` and push it to the repo: - -```yaml -image: maven:latest - -variables: - MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode" - MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository" - -cache: - paths: - - .m2/repository/ - - target/ - -build: - stage: build - script: - - mvn $MAVEN_CLI_OPTS compile - -test: - stage: test - script: - - mvn $MAVEN_CLI_OPTS test - -deploy: - stage: deploy - script: - - mvn $MAVEN_CLI_OPTS deploy - only: - - master -``` - -GitLab Runner will use the latest [Maven Docker image](https://hub.docker.com/_/maven/), which already contains all the tools and the dependencies you need to manage the project, -in order to run the jobs. - -Environment variables are set to instruct Maven to use the `homedir` of the repo instead of the user's home when searching for configuration and dependencies. - -Caching the `.m2/repository folder` (where all the Maven files are stored), and the `target` folder (where our application will be created), is useful for speeding up the process -by running all Maven phases in a sequential order, therefore, executing `mvn test` will automatically run `mvn compile` if necessary. - -Both `build` and `test` jobs leverage the `mvn` command to compile the application and to test it as defined in the test suite that is part of the application. - -Deploy to Artifactory is done as defined by the secret variables we have just set up. -The deployment occurs only if we're pushing or merging to `master` branch, so that the development versions are tested but not published. - -Done! Now you have all the changes in the GitLab repo, and a pipeline has already been started for this commit. In the **Pipelines** tab you can see what's happening. -If the deployment has been successful, the deploy job log will output: - -``` -[INFO] ------------------------------------------------------------------------ -[INFO] BUILD SUCCESS -[INFO] ------------------------------------------------------------------------ -[INFO] Total time: 1.983 s -``` - ->**Note**: -the `mvn` command downloads a lot of files from the internet, so you'll see a lot of extra activity in the log the first time you run it. - -Yay! You did it! Checking in Artifactory will confirm that you have a new artifact available in the `libs-release-local` repo. - -## Create the main Maven application - -Now that you have the dependency available on Artifactory, it's time to use it! -Let's see how we can have it as a dependency to our main application. - -### Prepare the main application - -We'll use again a Maven app that can be cloned from our example project: - -1. Create a new project by selecting **Import project from ➔ Repo by URL** -1. Add the following URL: - - ``` - https://gitlab.com/gitlab-examples/maven/simple-maven-app.git - ``` -1. Click **Create project** - -This one is a simple app as well. If you look at the `src/main/java/com/example/app/App.java` -file you can see that it imports the `com.example.dep.Dep` class and calls the `hello` method passing `GitLab` as a parameter. - -Since Maven doesn't know how to resolve the dependency, you need to modify the configuration: - -1. Go back to Artifactory -1. Browse the `libs-release-local` repository -1. Select the `simple-maven-dep-1.0.jar` file -1. Find the configuration snippet from the **Dependency Declaration** section of the main panel -1. Copy the snippet in the `dependencies` section of the `pom.xml` file. - The snippet should look like this: - - ```xml - <dependency> - <groupId>com.example.dep</groupId> - <artifactId>simple-maven-dep</artifactId> - <version>1.0</version> - </dependency> - ``` - -### Configure the Artifactory repository location - -At this point you defined the dependency for the application, but you still miss where you can find the required files. -You need to create a `.m2/settings.xml` file as you did for the dependency project, and let Maven know the location using environment variables. - -Here is how you can get the content of the file directly from Artifactory: - -1. From the main screen, click on the `libs-release-local` item in the **Set Me Up** panel -1. Click on **Generate Maven Settings** -1. Click on **Generate Settings** -1. Copy to clipboard the configuration file -1. Save the file as `.m2/settings.xml` in your repo - -Now you are ready to use the Artifactory repository to resolve dependencies and use `simple-maven-dep` in your main application! - -### Configure GitLab CI/CD for `simple-maven-app` - -You need a last step to have everything in place: configure the `.gitlab-ci.yml` file for this project, as you already did for `simple-maven-dep`. - -You want to leverage [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/) to automatically build, test and run your awesome application, -and see if you can get the greeting as expected! - -All you need to do is to add the following `.gitlab-ci.yml` to the repo: - -```yaml -image: maven:latest - -stages: - - build - - test - - run - -variables: - MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode" - MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository" - -cache: - paths: - - .m2/repository/ - - target/ - -build: - stage: build - script: - - mvn $MAVEN_CLI_OPTS compile - -test: - stage: test - script: - - mvn $MAVEN_CLI_OPTS test - -run: - stage: run - script: - - mvn $MAVEN_CLI_OPTS package - - mvn $MAVEN_CLI_OPTS exec:java -Dexec.mainClass="com.example.app.App" -``` - -It is very similar to the configuration used for `simple-maven-dep`, but instead of the `deploy` job there is a `run` job. -Probably something that you don't want to use in real projects, but here it is useful to see the application executed automatically. - -And that's it! In the `run` job output log you will find a friendly hello to GitLab! - -## Conclusion - -In this article we covered the basic steps to use an Artifactory Maven repository to automatically publish and consume artifacts. - -A similar approach could be used to interact with any other Maven compatible Binary Repository Manager. -Obviously, you can improve these examples, optimizing the `.gitlab-ci.yml` file to better suit your needs, and adapting to your workflow. +This document was moved to [another location](../../ci/examples/artifactory_and_gitlab/index.md) diff --git a/doc/articles/how_to_configure_ldap_gitlab_ce/img/gitlab_ou.png b/doc/articles/how_to_configure_ldap_gitlab_ce/img/gitlab_ou.png Binary files differdeleted file mode 100644 index 11ce324f938..00000000000 --- a/doc/articles/how_to_configure_ldap_gitlab_ce/img/gitlab_ou.png +++ /dev/null diff --git a/doc/articles/how_to_configure_ldap_gitlab_ce/img/ldap_ou.gif b/doc/articles/how_to_configure_ldap_gitlab_ce/img/ldap_ou.gif Binary files differdeleted file mode 100644 index a6727a3d85f..00000000000 --- a/doc/articles/how_to_configure_ldap_gitlab_ce/img/ldap_ou.gif +++ /dev/null diff --git a/doc/articles/how_to_configure_ldap_gitlab_ce/img/user_auth.gif b/doc/articles/how_to_configure_ldap_gitlab_ce/img/user_auth.gif Binary files differdeleted file mode 100644 index 36e6085259f..00000000000 --- a/doc/articles/how_to_configure_ldap_gitlab_ce/img/user_auth.gif +++ /dev/null diff --git a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md index 25a24bc1d32..a8320c12e6b 100644 --- a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md @@ -1,267 +1 @@ -# How to configure LDAP with GitLab CE - -> **Article [Type](../../development/writing_documentation.html#types-of-technical-articles):** admin guide || -> **Level:** intermediary || -> **Author:** [Chris Wilson](https://gitlab.com/MrChrisW) || -> **Publication date:** 2017-05-03 - -## Introduction - -Managing a large number of users in GitLab can become a burden for system administrators. As an organization grows so do user accounts. Keeping these user accounts in sync across multiple enterprise applications often becomes a time consuming task. - -In this guide we will focus on configuring GitLab with Active Directory. [Active Directory](https://en.wikipedia.org/wiki/Active_Directory) is a popular LDAP compatible directory service provided by Microsoft, included in all modern Windows Server operating systems. - -GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](#group-syncing-ee) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. - -## Getting started - -### Choosing an LDAP Server - -The main reason organizations choose to utilize a LDAP server is to keep the entire organization's user base consolidated into a central repository. Users can access multiple applications and systems across the IT environment using a single login. Because LDAP is an open, vendor-neutral, industry standard application protocol, the number of applications using LDAP authentication continues to increase. - -There are many commercial and open source [directory servers](https://en.wikipedia.org/wiki/Directory_service#LDAP_implementations) that support the LDAP protocol. Deciding on the right directory server highly depends on the existing IT environment in which the server will be integrated with. - -For example, [Active Directory](https://technet.microsoft.com/en-us/library/hh831484(v=ws.11).aspx) is generally favored in a primarily Windows environment, as this allows quick integration with existing services. Other popular directory services include: - -- [Oracle Internet Directory](http://www.oracle.com/technetwork/middleware/id-mgmt/overview/index-082035.html) -- [OpenLDAP](http://www.openldap.org/) -- [389 Directory](http://directory.fedoraproject.org/) -- [OpenDJ](https://forgerock.org/opendj/) -- [ApacheDS](https://directory.apache.org/) - -> GitLab uses the [Net::LDAP](https://rubygems.org/gems/net-ldap) library under the hood. This means it supports all [IETF](https://tools.ietf.org/html/rfc2251) compliant LDAPv3 servers. - -### Active Directory (AD) - -We won't cover the installation and configuration of Windows Server or Active Directory Domain Services in this tutorial. There are a number of resources online to guide you through this process: - -- Install Windows Server 2012 - (_technet.microsoft.com_) - [Installing Windows Server 2012 ](https://technet.microsoft.com/en-us/library/jj134246(v=ws.11).aspx) - -- Install Active Directory Domain Services (AD DS) (_technet.microsoft.com_)- [Install Active Directory Domain Services](https://technet.microsoft.com/windows-server-docs/identity/ad-ds/deploy/install-active-directory-domain-services--level-100-#BKMK_PS) - -> **Shortcut:** You can quickly install AD DS via PowerShell using -`Install-WindowsFeature AD-Domain-Services -IncludeManagementTools` - -### Creating an AD **OU** structure - -Configuring organizational units (**OU**s) is an important part of setting up Active Directory. **OU**s form the base for an entire organizational structure. Using GitLab as an example we have designed the **OU** structure below using the geographic **OU** model. In the Geographic Model we separate **OU**s for different geographic regions. - -| GitLab **OU** Design | GitLab AD Structure | -| :----------------------------: | :------------------------------: | -| ![GitLab OU Design][gitlab_ou] | ![GitLab AD Structure][ldap_ou] | - -[gitlab_ou]: img/gitlab_ou.png -[ldap_ou]: img/ldap_ou.gif - -Using PowerShell you can output the **OU** structure as a table (_all names are examples only_): - -```ps -Get-ADObject -LDAPFilter "(objectClass=*)" -SearchBase 'OU=GitLab INT,DC=GitLab,DC=org' -Properties CanonicalName | Format-Table Name,CanonicalName -A -``` - -``` -OU CanonicalName ----- ------------- -GitLab INT GitLab.org/GitLab INT -United States GitLab.org/GitLab INT/United States -Developers GitLab.org/GitLab INT/United States/Developers -Gary Johnson GitLab.org/GitLab INT/United States/Developers/Gary Johnson -Ellis Matthews GitLab.org/GitLab INT/United States/Developers/Ellis Matthews -William Collins GitLab.org/GitLab INT/United States/Developers/William Collins -People Ops GitLab.org/GitLab INT/United States/People Ops -Margaret Baker GitLab.org/GitLab INT/United States/People Ops/Margaret Baker -Libby Hartzler GitLab.org/GitLab INT/United States/People Ops/Libby Hartzler -Victoria Ryles GitLab.org/GitLab INT/United States/People Ops/Victoria Ryles -The Netherlands GitLab.org/GitLab INT/The Netherlands -Developers GitLab.org/GitLab INT/The Netherlands/Developers -John Doe GitLab.org/GitLab INT/The Netherlands/Developers/John Doe -Jon Mealy GitLab.org/GitLab INT/The Netherlands/Developers/Jon Mealy -Jane Weingarten GitLab.org/GitLab INT/The Netherlands/Developers/Jane Weingarten -Production GitLab.org/GitLab INT/The Netherlands/Production -Sarah Konopka GitLab.org/GitLab INT/The Netherlands/Production/Sarah Konopka -Cynthia Bruno GitLab.org/GitLab INT/The Netherlands/Production/Cynthia Bruno -David George GitLab.org/GitLab INT/The Netherlands/Production/David George -United Kingdom GitLab.org/GitLab INT/United Kingdom -Developers GitLab.org/GitLab INT/United Kingdom/Developers -Leroy Fox GitLab.org/GitLab INT/United Kingdom/Developers/Leroy Fox -Christopher Alley GitLab.org/GitLab INT/United Kingdom/Developers/Christopher Alley -Norris Morita GitLab.org/GitLab INT/United Kingdom/Developers/Norris Morita -Support GitLab.org/GitLab INT/United Kingdom/Support -Laura Stanley GitLab.org/GitLab INT/United Kingdom/Support/Laura Stanley -Nikki Schuman GitLab.org/GitLab INT/United Kingdom/Support/Nikki Schuman -Harriet Butcher GitLab.org/GitLab INT/United Kingdom/Support/Harriet Butcher -Global Groups GitLab.org/GitLab INT/Global Groups -DevelopersNL GitLab.org/GitLab INT/Global Groups/DevelopersNL -DevelopersUK GitLab.org/GitLab INT/Global Groups/DevelopersUK -DevelopersUS GitLab.org/GitLab INT/Global Groups/DevelopersUS -ProductionNL GitLab.org/GitLab INT/Global Groups/ProductionNL -SupportUK GitLab.org/GitLab INT/Global Groups/SupportUK -People Ops US GitLab.org/GitLab INT/Global Groups/People Ops US -Global Admins GitLab.org/GitLab INT/Global Groups/Global Admins -``` - -> See [more information](https://technet.microsoft.com/en-us/library/ff730967.aspx) on searching Active Directory with Windows PowerShell from [The Scripting Guys](https://technet.microsoft.com/en-us/scriptcenter/dd901334.aspx) - -## GitLab LDAP configuration - -The initial configuration of LDAP in GitLab requires changes to the `gitlab.rb` configuration file. Below is an example of a complete configuration using an Active Directory. - -The two Active Directory specific values are `active_directory: true` and `uid: 'sAMAccountName'`. `sAMAccountName` is an attribute returned by Active Directory used for GitLab usernames. See the example output from `ldapsearch` for a full list of attributes a "person" object (user) has in **AD** - [`ldapsearch` example](#using-ldapsearch-unix) - -> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](#gitlab-enterprise-edition---ldap-features) - -### Example `gitlab.rb` LDAP - -``` -gitlab_rails['ldap_enabled'] = true -gitlab_rails['ldap_servers'] = { -'main' => { - 'label' => 'GitLab AD', - 'host' => 'ad.example.org', - 'port' => 636, - 'uid' => 'sAMAccountName', - 'encryption' => 'simple_tls', - 'verify_certificates' => true, - 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=org', - 'password' => 'Password1', - 'active_directory' => true, - 'base' => 'OU=GitLab INT,DC=GitLab,DC=org', - 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org', - 'admin_group' => 'Global Admins' - } -} -``` - -> **Note:** Remember to run `gitlab-ctl reconfigure` after modifying `gitlab.rb` - -## Security improvements (LDAPS) - -Security is an important aspect when deploying an LDAP server. By default, LDAP traffic is transmitted unsecured. LDAP can be secured using SSL/TLS called LDAPS, or commonly "LDAP over SSL". - -Securing LDAP (enabling LDAPS) on Windows Server 2012 involves installing a valid SSL certificate. For full details see Microsoft's guide [How to enable LDAP over SSL with a third-party certification authority](https://support.microsoft.com/en-us/help/321051/how-to-enable-ldap-over-ssl-with-a-third-party-certification-authority) - -> By default a LDAP service listens for connections on TCP and UDP port 389. LDAPS (LDAP over SSL) listens on port 636 - -### Testing you AD server - -#### Using **AdFind** (Windows) - -You can use the [`AdFind`](https://social.technet.microsoft.com/wiki/contents/articles/7535.adfind-command-examples.aspx) utility (on Windows based systems) to test that your LDAP server is accessible and authentication is working correctly. This is a freeware utility built by [Joe Richards](http://www.joeware.net/freetools/tools/adfind/index.htm). - -**Return all objects** - -You can use the filter `objectclass=*` to return all directory objects. - -```sh -adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (objectClass=*) -``` - -**Return single object using filter** - -You can also retrieve a single object by **specifying** the object name or full **DN**. In this example we specify the object name only `CN=Leroy Fox`. - -```sh -adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (&(objectcategory=person)(CN=Leroy Fox))” -``` - -#### Using **ldapsearch** (Unix) - -You can use the `ldapsearch` utility (on Unix based systems) to test that your LDAP server is accessible and authentication is working correctly. This utility is included in the [`ldap-utils`](https://wiki.debian.org/LDAP/LDAPUtils) package. - -**Return all objects** - -You can use the filter `objectclass=*` to return all directory objects. - -```sh -ldapsearch -D "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" \ --w Password1 -p 636 -h ad.example.org \ --b "OU=GitLab INT,DC=GitLab,DC=org" -Z \ --s sub "(objectclass=*)" -``` - -**Return single object using filter** - -You can also retrieve a single object by **specifying** the object name or full **DN**. In this example we specify the object name only `CN=Leroy Fox`. - -```sh -ldapsearch -D "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -w Password1 -p 389 -h ad.example.org -b "OU=GitLab INT,DC=GitLab,DC=org" -Z -s sub "CN=Leroy Fox" -``` - -**Full output of `ldapsearch` command:** - Filtering for _CN=Leroy Fox_ - -``` -# LDAPv3 -# base <OU=GitLab INT,DC=GitLab,DC=org> with scope subtree -# filter: CN=Leroy Fox -# requesting: ALL -# - -# Leroy Fox, Developers, United Kingdom, GitLab INT, GitLab.org -dn: CN=Leroy Fox,OU=Developers,OU=United Kingdom,OU=GitLab INT,DC=GitLab,DC=or - g -objectClass: top -objectClass: person -objectClass: organizationalPerson -objectClass: user -cn: Leroy Fox -sn: Fox -givenName: Leroy -distinguishedName: CN=Leroy Fox,OU=Developers,OU=United Kingdom,OU=GitLab INT, - DC=GitLab,DC=org -instanceType: 4 -whenCreated: 20170210030500.0Z -whenChanged: 20170213050128.0Z -displayName: Leroy Fox -uSNCreated: 16790 -memberOf: CN=DevelopersUK,OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org -uSNChanged: 20812 -name: Leroy Fox -objectGUID:: rBCAo6NR6E6vfSKgzcUILg== -userAccountControl: 512 -badPwdCount: 0 -codePage: 0 -countryCode: 0 -badPasswordTime: 0 -lastLogoff: 0 -lastLogon: 0 -pwdLastSet: 131311695009850084 -primaryGroupID: 513 -objectSid:: AQUAAAAAAAUVAAAA9GMAb7tdJZvsATf7ZwQAAA== -accountExpires: 9223372036854775807 -logonCount: 0 -sAMAccountName: Leroyf -sAMAccountType: 805306368 -userPrincipalName: Leroyf@GitLab.org -objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=GitLab,DC=org -dSCorePropagationData: 16010101000000.0Z -lastLogonTimestamp: 131314356887754250 - -# search result -search: 2 -result: 0 Success - -# numResponses: 2 -# numEntries: 1 -``` - -## Basic user authentication - -After configuring LDAP, basic authentication will be available. Users can then login using their directory credentials. An extra tab is added to the GitLab login screen for the configured LDAP server (e.g "**GitLab AD**"). - - - -Users that are removed from the LDAP base group (e.g `OU=GitLab INT,DC=GitLab,DC=org`) will be **blocked** in GitLab. [More information](../../administration/auth/ldap.md#security) on LDAP security. - -If `allow_username_or_email_login` is enabled in the LDAP configuration, GitLab will ignore everything after the first '@' in the LDAP username used on login. Example: The username `jon.doe@example.com` is converted to `jon.doe` when authenticating with the LDAP server. Disable this setting if you use `userPrincipalName` as the `uid`. - -## LDAP extended features on GitLab EE - -With [GitLab Enterprise Edition (EE)](https://about.gitlab.com/gitlab-ee/), besides everything we just described, you'll -have extended functionalities with LDAP, such as: - -- Group sync -- Group permissions -- Updating user permissions -- Multiple LDAP servers - -Read through the article on [LDAP for GitLab EE](https://docs.gitlab.com/ee/articles/how_to_configure_ldap_gitlab_ee/) for an overview. +This document was moved to [another location](../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md). diff --git a/doc/articles/how_to_install_git/index.md b/doc/articles/how_to_install_git/index.md index 37b60501ce2..3e6003a33b7 100644 --- a/doc/articles/how_to_install_git/index.md +++ b/doc/articles/how_to_install_git/index.md @@ -1,66 +1 @@ -# Installing Git - -> **Article [Type](../../development/writing_documentation.html#types-of-technical-articles):** user guide || -> **Level:** beginner || -> **Author:** [Sean Packham](https://gitlab.com/SeanPackham) || -> **Publication date:** 2017-05-15 - -To begin contributing to GitLab projects -you will need to install the Git client on your computer. -This article will show you how to install Git on macOS, Ubuntu Linux and Windows. - -## Install Git on macOS using the Homebrew package manager - -Although it is easy to use the version of Git shipped with macOS -or install the latest version of Git on macOS by downloading it from the project website, -we recommend installing it via Homebrew to get access to -an extensive selection of dependancy managed libraries and applications. - -If you are sure you don't need access to any additional development libraries -or don't have approximately 15gb of available disk space for Xcode and Homebrew -use one of the the aforementioned methods. - -### Installing Xcode - -Xcode is needed by Homebrew to build dependencies. -You can install [XCode](https://developer.apple.com/xcode/) -through the macOS App Store. - -### Installing Homebrew - -Once Xcode is installed browse to the [Homebrew website](http://brew.sh/index.html) -for the official Homebrew installation instructions. - -### Installing Git via Homebrew - -With Homebrew installed you are now ready to install Git. -Open a Terminal and enter in the following command: - -```bash -brew install git -``` - -Congratulations you should now have Git installed via Homebrew. -Next read our article on [adding an SSH key to GitLab](../../ssh/README.md). - -## Install Git on Ubuntu Linux - -On Ubuntu and other Linux operating systems -it is recommended to use the built in package manager to install Git. - -Open a Terminal and enter in the following commands -to install the latest Git from the official Git maintained package archives: - -```bash -sudo apt-add-repository ppa:git-core/ppa -sudo apt-get update -sudo apt-get install git -``` - -Congratulations you should now have Git installed via the Ubuntu package manager. -Next read our article on [adding an SSH key to GitLab](../../ssh/README.md). - -## Installing Git on Windows from the Git website - -Browse to the [Git website](https://git-scm.com/) and download and install Git for Windows. -Next read our article on [adding an SSH key to GitLab](../../ssh/README.md). +This document was moved to [another location](../../topics/git/how_to_install_git/index.md). diff --git a/doc/articles/index.md b/doc/articles/index.md index 862fe0868a6..9f85533ea94 100644 --- a/doc/articles/index.md +++ b/doc/articles/index.md @@ -1,118 +1,18 @@ -# Technical Articles +--- +comments: false +--- -[Technical Articles](../development/writing_documentation.md#technical-articles) are +# Technical articles list (deprecated) + +[Technical articles](../development/writing_documentation.md#technical-articles) are topic-related documentation, written with an user-friendly approach and language, aiming to provide the community with guidance on specific processes to achieve certain objectives. -They are written by members of the GitLab Team and by -[Community Writers](https://about.gitlab.com/handbook/product/technical-writing/community-writers/). - -Part of the articles listed below link to the [GitLab Blog](https://about.gitlab.com/blog/), -where they were originally published. - -## Authentication - -Explore GitLab's supported [authentications methods](../topics/authentication/index.md): - -| Article title | Category | Publishing date | -| :------------ | :------: | --------------: | -| **LDAP** | -| [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md)| Admin guide | 2017-05-03 | -| [How to configure LDAP with GitLab EE](https://docs.gitlab.com/ee/articles/how_to_configure_ldap_gitlab_ee/) | Admin guide | 2017-05-03 | - -## Build, test, and deploy with GitLab CI/CD - -Build, test, and deploy the software you develop with [GitLab CI/CD](../ci/README.md): - -| Article title | Category | Publishing date | -| :------------ | :------: | --------------: | -| [Autoscaling GitLab Runners on AWS](runner_autoscale_aws/index.md) | Admin guide | 2017-11-24 | -| [How to test and deploy Laravel/PHP applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md) | Tutorial | 2017-08-31 | -| [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md) | Tutorial | 2017-08-15 | -| [Making CI Easier with GitLab](https://about.gitlab.com/2017/07/13/making-ci-easier-with-gitlab/) | Concepts | 2017-07-13 | -| [Dockerizing GitLab Review Apps](https://about.gitlab.com/2017/07/11/dockerizing-review-apps/) | Concepts | 2017-07-11 | -| [Continuous Integration: From Jenkins to GitLab Using Docker](https://about.gitlab.com/2017/07/27/docker-my-precious/) | Concepts | 2017-07-27 | -| [Continuous Delivery of a Spring Boot application with GitLab CI and Kubernetes](https://about.gitlab.com/2016/12/14/continuous-delivery-of-a-spring-boot-application-with-gitlab-ci-and-kubernetes/) | Tutorial | 2016-12-14 | -| [Setting up GitLab CI for Android projects](https://about.gitlab.com/2016/11/30/setting-up-gitlab-ci-for-android-projects/) | Tutorial | 2016-11-30 | -| [Automated Debian Package Build with GitLab CI](https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/) | Tutorial | 2016-10-12 | -| [Building an Elixir Release into a Docker image using GitLab CI](https://about.gitlab.com/2016/08/11/building-an-elixir-release-into-docker-image-using-gitlab-ci-part-1/) | Tutorial | 2016-08-11 | -| [Continuous Delivery with GitLab and Convox](https://about.gitlab.com/2016/06/09/continuous-delivery-with-gitlab-and-convox/) | Technical overview | 2016-06-09 | -| [GitLab Container Registry](https://about.gitlab.com/2016/05/23/gitlab-container-registry/) | Technical overview | 2016-05-23 | -| [How to use GitLab CI and MacStadium to build your macOS or iOS projects](https://about.gitlab.com/2017/05/15/how-to-use-macstadium-and-gitlab-ci-to-build-your-macos-or-ios-projects/) | Technical overview | 2017-05-15 | -| [Setting up GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) | Tutorial | 2016-03-10 | - -## Git - -Learn how to use [Git with GitLab](../topics/git/index.md): - -| Article title | Category | Publishing date | -| :------------ | :------: | --------------: | -| [Numerous _undo_ possibilities in Git](numerous_undo_possibilities_in_git/index.md) | Tutorial | 2017-08-17 | -| [Why Git is Worth the Learning Curve](https://about.gitlab.com/2017/05/17/learning-curve-is-the-biggest-challenge-developers-face-with-git/) | Concepts | 2017-05-17 | -| [How to install Git](how_to_install_git/index.md) | Tutorial | 2017-05-15 | -| [Getting Started with Git LFS](https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/) | Tutorial | 2017-01-30 | -| [Git Tips & Tricks](https://about.gitlab.com/2016/12/08/git-tips-and-tricks/) | Technical overview | 2016-12-08 | - -## GitLab Pages - -Learn how to deploy a static website with [GitLab Pages](../user/project/pages/index.md#getting-started): - -| Article title | Category | Publishing date | -| :------------ | :------: | --------------: | -| **Series: GitLab Pages from A to Z:** | -| [- Part 1: Static sites and GitLab Pages domains](../user/project/pages/getting_started_part_one.md)| User guide | 2017-02-22 | -| [- Part 2: Quick start guide - Setting up GitLab Pages](../user/project/pages/getting_started_part_two.md)| User guide | 2017-02-22 | -| [- Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](../user/project/pages/getting_started_part_three.md)| User guide | 2017-02-22 | -| [- Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages](../user/project/pages/getting_started_part_four.md)| User guide | 2017-02-22 | -| [Setting up GitLab Pages with CloudFlare Certificates](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) | Tutorial | 2017-02-07 | -| [Building a new GitLab Docs site with Nanoc, GitLab CI, and GitLab Pages](https://about.gitlab.com/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/) | Tutorial | 2016-12-07 | -| [Publish Code Coverage Report with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/) | Tutorial | 2016-11-03 | -| [GitLab CI: Deployment & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) | Tutorial | 2016-08-26 | -| [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/) | Tutorial | 2016-08-19 | -| **Series: Static Site Generator:** | -| [- Part 1: Dynamic vs Static Websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) | Tutorial | 2016-06-03 | -| [- Part 2: Modern Static Site Generators](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) | Tutorial | 2016-06-10 | -| [- Part 3: Build any SSG site with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) | Tutorial | 2016-06-17 | -| [Securing your GitLab Pages with TLS and Let's Encrypt](https://about.gitlab.com/2016/04/11/tutorial-securing-your-gitlab-pages-with-tls-and-letsencrypt/) | Tutorial | 2016-04-11 | -| [Hosting on GitLab.com with GitLab Pages](https://about.gitlab.com/2016/04/07/gitlab-pages-setup/) | Tutorial | 2016-04-07 | - -## Install and maintain GitLab - -[Admin](../README.md#administrator-documentation), [install](../install/README.md), -upgrade, integrate, migrate to GitLab: - -| Article title | Category | Publishing date | -| :------------ | :------: | --------------: | -| [Video Tutorial: Idea to Production on Google Container Engine (GKE)](https://about.gitlab.com/2017/01/23/video-tutorial-idea-to-production-on-google-container-engine-gke/) | Tutorial | 2017-01-23 | -| [How to Setup a GitLab Instance on Microsoft Azure](https://about.gitlab.com/2016/07/13/how-to-setup-a-gitlab-instance-on-microsoft-azure/) | Tutorial | 2016-07-13 | -| [Get started with OpenShift Origin 3 and GitLab](openshift_and_gitlab/index.md) | Tutorial | 2016-06-28 | -| [Getting started with GitLab and DigitalOcean](https://about.gitlab.com/2016/04/27/getting-started-with-gitlab-and-digitalocean/) | Tutorial | 2016-04-27 | - -## Software development - -Explore the best of GitLab's software development's capabilities: - -| Article title | Category | Publishing date | -| :------------ | :------: | --------------: | -| [Making CI Easier with GitLab](https://about.gitlab.com/2017/07/13/making-ci-easier-with-gitlab/) | Concepts | 2017-07-13 | -| [From 2/3 of the Self-Hosted Git Market, to the Next-Generation CI System, to Auto DevOps](https://about.gitlab.com/2017/06/29/whats-next-for-gitlab-ci/)| Concepts | 2017-06-29 | -| [Fast and Natural Continuous Integration with GitLab CI](https://about.gitlab.com/2017/05/22/fast-and-natural-continuous-integration-with-gitlab-ci/) | Concepts | 2017-05-22 | -| [Demo: Auto-Deploy from GitLab to an OpenShift Container Cluster](https://about.gitlab.com/2017/05/16/devops-containers-gitlab-openshift/) | Technical overview | 2017-05-16 | -| [Demo: GitLab Service Desk](https://about.gitlab.com/2017/05/09/demo-service-desk/) | Feature highlight | 2017-05-09 | -| [Demo: Mapping Work Versus Time, With Burndown Charts](https://about.gitlab.com/2017/04/25/mapping-work-to-do-versus-time-with-burndown-charts/) | Feature highlight | 2017-04-25 | -| [Demo: Cloud Native Development with GitLab](https://about.gitlab.com/2017/04/18/cloud-native-demo/) | Feature highlight | 2017-04-18 | -| [Demo: Mastering Code Review With GitLab](https://about.gitlab.com/2017/03/17/demo-mastering-code-review-with-gitlab/) | Feature highlight | 2017-03-17 | -| [In 13 minutes from Kubernetes to a complete application development tool](https://about.gitlab.com/2016/11/14/idea-to-production/) | Technical overview | 2016-11-14 | -| [GitLab Workflow, an Overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/) | Technical overview | 2016-10-25 | -| [Trends in Version Control Land: Microservices](https://about.gitlab.com/2016/08/16/trends-in-version-control-land-microservices/) | Concepts | 2016-08-16 | -| [Continuous Integration, Delivery, and Deployment with GitLab](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) | Concepts | 2016-08-05 | -| [Trends in Version Control Land: Innersourcing](https://about.gitlab.com/2016/07/07/trends-version-control-innersourcing/) | Concepts | 2016-07-07 | -| [Tutorial: It's all connected in GitLab](https://about.gitlab.com/2016/03/08/gitlab-tutorial-its-all-connected/) | Technical overview | 2016-03-08 | - -## Technologies +The list of technical articles was [deprecated](https://gitlab.com/gitlab-org/gitlab-ce/issues/41138) in favor of having them linked from their topic-related documentation: -| Article title | Category | Publishing date | -| :------------ | :------: | --------------: | -| [Why we are not leaving the cloud](https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-the-cloud/) | Concepts | 2017-03-02 | -| [Why We Chose Vue.js](https://about.gitlab.com/2016/10/20/why-we-chose-vue/) | Concepts | 2016-10-20 | -| [Markdown Kramdown Tips & Tricks](https://about.gitlab.com/2016/07/19/markdown-kramdown-tips-and-tricks/) | Technical overview | 2016-07-19 | +- [Git](../topics/git/index.md) +- [GitLab administrator](../administration/index.md) +- [GitLab CI/CD](../ci/README.md) +- [GitLab Pages](../user/project/pages/index.md) +- [GitLab user](../user/index.md) +- [Install GitLab](../install/README.md) diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_checkbox.png b/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_checkbox.png Binary files differdeleted file mode 100644 index a56c07a0da7..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_checkbox.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_page_empty_image.png b/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_page_empty_image.png Binary files differdeleted file mode 100644 index b1406fed6b8..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_page_empty_image.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_page_with_image.jpg b/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_page_with_image.jpg Binary files differdeleted file mode 100644 index d1f0cbc08ab..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/container_registry_page_with_image.jpg +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/deploy_keys_page.png b/doc/articles/laravel_with_gitlab_and_envoy/img/deploy_keys_page.png Binary files differdeleted file mode 100644 index 9aae11b8679..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/deploy_keys_page.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/environment_page.png b/doc/articles/laravel_with_gitlab_and_envoy/img/environment_page.png Binary files differdeleted file mode 100644 index a06b6d417cd..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/environment_page.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/environments_page.png b/doc/articles/laravel_with_gitlab_and_envoy/img/environments_page.png Binary files differdeleted file mode 100644 index d357ecda7d2..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/environments_page.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/laravel_welcome_page.png b/doc/articles/laravel_with_gitlab_and_envoy/img/laravel_welcome_page.png Binary files differdeleted file mode 100644 index 3bb21fd12b4..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/laravel_welcome_page.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/laravel_with_gitlab_and_envoy.png b/doc/articles/laravel_with_gitlab_and_envoy/img/laravel_with_gitlab_and_envoy.png Binary files differdeleted file mode 100644 index bc188f83fb1..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/laravel_with_gitlab_and_envoy.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/pipeline_page.png b/doc/articles/laravel_with_gitlab_and_envoy/img/pipeline_page.png Binary files differdeleted file mode 100644 index baf8dec499c..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/pipeline_page.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/pipelines_page.png b/doc/articles/laravel_with_gitlab_and_envoy/img/pipelines_page.png Binary files differdeleted file mode 100644 index d96c43bcf16..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/pipelines_page.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/pipelines_page_deploy_button.png b/doc/articles/laravel_with_gitlab_and_envoy/img/pipelines_page_deploy_button.png Binary files differdeleted file mode 100644 index 997db10189f..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/pipelines_page_deploy_button.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/production_server_app_directory.png b/doc/articles/laravel_with_gitlab_and_envoy/img/production_server_app_directory.png Binary files differdeleted file mode 100644 index 6dbc29fc25c..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/production_server_app_directory.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/production_server_current_directory.png b/doc/articles/laravel_with_gitlab_and_envoy/img/production_server_current_directory.png Binary files differdeleted file mode 100644 index 8a6dcccfa38..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/production_server_current_directory.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/img/secret_variables_page.png b/doc/articles/laravel_with_gitlab_and_envoy/img/secret_variables_page.png Binary files differdeleted file mode 100644 index 658c0b5bcac..00000000000 --- a/doc/articles/laravel_with_gitlab_and_envoy/img/secret_variables_page.png +++ /dev/null diff --git a/doc/articles/laravel_with_gitlab_and_envoy/index.md b/doc/articles/laravel_with_gitlab_and_envoy/index.md index b20bd8c247a..b092cdb0f7a 100644 --- a/doc/articles/laravel_with_gitlab_and_envoy/index.md +++ b/doc/articles/laravel_with_gitlab_and_envoy/index.md @@ -1,680 +1 @@ -# Test and deploy Laravel applications with GitLab CI/CD and Envoy - -> **[Article Type](../../development/writing_documentation.md#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Mehran Rasulian](https://gitlab.com/mehranrasulian) || -> **Publication date:** 2017-08-31 - -## Introduction - -GitLab features our applications with Continuous Integration, and it is possible to easily deploy the new code changes to the production server whenever we want. - -In this tutorial, we'll show you how to initialize a [Laravel](http://laravel.com/) application and setup our [Envoy](https://laravel.com/docs/envoy) tasks, then we'll jump into see how to test and deploy it with [GitLab CI/CD](../../ci/README.md) via [Continuous Delivery](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/). - -We assume you have a basic experience with Laravel, Linux servers, -and you know how to use GitLab. - -Laravel is a high quality web framework written in PHP. -It has a great community with a [fantastic documentation](https://laravel.com/docs). -Aside from the usual routing, controllers, requests, responses, views, and (blade) templates, out of the box Laravel provides plenty of additional services such as cache, events, localization, authentication and many others. - -We will use [Envoy](https://laravel.com/docs/master/envoy) as an SSH task runner based on PHP. -It uses a clean, minimal [Blade syntax](https://laravel.com/docs/blade) to setup tasks that can run on remote servers, such as, cloning your project from the repository, installing the Composer dependencies, and running [Artisan commands](https://laravel.com/docs/artisan). - -## Initialize our Laravel app on GitLab - -We assume [you have installed a new laravel project](https://laravel.com/docs/installation#installation), so let's start with a unit test, and initialize Git for the project. - -### Unit Test - -Every new installation of Laravel (currently 5.4) comes with two type of tests, 'Feature' and 'Unit', placed in the tests directory. -Here's a unit test from `test/Unit/ExampleTest.php`: - -```php -<?php - -namespace Tests\Unit; - -... - -class ExampleTest extends TestCase -{ - public function testBasicTest() - { - $this->assertTrue(true); - } -} -``` - -This test is as simple as asserting that the given value is true. - -Laravel uses `PHPUnit` for tests by default. -If we run `vendor/bin/phpunit` we should see the green output: - -```bash -vendor/bin/phpunit -OK (1 test, 1 assertions) -``` - -This test will be used later for continuously testing our app with GitLab CI/CD. - -### Push to GitLab - -Since we have our app up and running locally, it's time to push the codebase to our remote repository. -Let's create [a new project](../../gitlab-basics/create-project.md) in GitLab named `laravel-sample`. -After that, follow the command line instructions displayed on the project's homepage to initiate the repository on our machine and push the first commit. - - -```bash -cd laravel-sample -git init -git remote add origin git@gitlab.example.com:<USERNAME>/laravel-sample.git -git add . -git commit -m 'Initial Commit' -git push -u origin master -``` - -## Configure the production server - -Before we begin setting up Envoy and GitLab CI/CD, let's quickly make sure the production server is ready for deployment. -We have installed LEMP stack which stands for Linux, Nginx, MySQL and PHP on our Ubuntu 16.04. - -### Create a new user - -Let's now create a new user that will be used to deploy our website and give it -the needed permissions using [Linux ACL](https://serversforhackers.com/video/linux-acls): - -```bash -# Create user deployer -sudo adduser deployer -# Give the read-write-execute permissions to deployer user for directory /var/www -sudo setfacl -R -m u:deployer:rwx /var/www -``` - -If you don't have ACL installed on your Ubuntu server, use this command to install it: - -```bash -sudo apt install acl -``` - -### Add SSH key - -Let's suppose we want to deploy our app to the production server from a private repository on GitLab. First, we need to [generate a new SSH key pair **with no passphrase**](../../ssh/README.md) for the deployer user. - -After that, we need to copy the private key, which will be used to connect to our server as the deployer user with SSH, to be able to automate our deployment process: - -```bash -# As the deployer user on server -# -# Copy the content of public key to authorized_keys -cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys -# Copy the private key text block -cat ~/.ssh/id_rsa -``` - -Now, let's add it to your GitLab project as a [secret variable](../../ci/variables/README.md#secret-variables). -Secret variables are user-defined variables and are stored out of `.gitlab-ci.yml`, for security purposes. -They can be added per project by navigating to the project's **Settings** > **CI/CD**. - - - -To the field **KEY**, add the name `SSH_PRIVATE_KEY`, and to the **VALUE** field, paste the private key you've copied earlier. -We'll use this variable in the `.gitlab-ci.yml` later, to easily connect to our remote server as the deployer user without entering its password. - -We also need to add the public key to **Project** > **Settings** > **Repository** as [Deploy Keys](../../ssh/README.md/#deploy-keys), which gives us the ability to access our repository from the server through [SSH protocol](../../gitlab-basics/command-line-commands.md/#start-working-on-your-project). - - -```bash -# As the deployer user on the server -# -# Copy the public key -cat ~/.ssh/id_rsa.pub -``` - - - -To the field **Title**, add any name you want, and paste the public key into the **Key** field. - -Now, let's clone our repository on the server just to make sure the `deployer` user has access to the repository. - -```bash -# As the deployer user on server -# -git clone git@gitlab.example.com:<USERNAME>/laravel-sample.git -``` - ->**Note:** -Answer **yes** if asked `Are you sure you want to continue connecting (yes/no)?`. -It adds GitLab.com to the known hosts. - -### Configuring Nginx - -Now, let's make sure our web server configuration points to the `current/public` rather than `public`. - -Open the default Nginx server block configuration file by typing: - -```bash -sudo nano /etc/nginx/sites-available/default -``` - -The configuration should be like this. - -``` -server { - root /var/www/app/current/public; - server_name example.com; - # Rest of the configuration -} -``` - ->**Note:** -You may replace the app's name in `/var/www/app/current/public` with the folder name of your application. - -## Setting up Envoy - -So we have our Laravel app ready for production. -The next thing is to use Envoy to perform the deploy. - -To use Envoy, we should first install it on our local machine [using the given instructions by Laravel](https://laravel.com/docs/envoy/#introduction). - -### How Envoy works - -The pros of Envoy is that it doesn't require Blade engine, it just uses Blade syntax to define tasks. -To start, we create an `Envoy.blade.php` in the root of our app with a simple task to test Envoy. - - -```php -@servers(['web' => 'remote_username@remote_host']) - -@task('list', [on => 'web']) - ls -l -@endtask -``` - -As you may expect, we have an array within `@servers` directive at the top of the file, which contains a key named `web` with a value of the server's address (e.g. `deployer@192.168.1.1`). -Then within our `@task` directive we define the bash commands that should be run on the server when the task is executed. - -On the local machine use the `run` command to run Envoy tasks. - -```bash -envoy run list -``` - -It should execute the `list` task we defined earlier, which connects to the server and lists directory contents. - -Envoy is not a dependency of Laravel, therefore you can use it for any PHP application. - -### Zero downtime deployment - -Every time we deploy to the production server, Envoy downloads the latest release of our app from GitLab repository and replace it with preview's release. -Envoy does this without any [downtime](https://en.wikipedia.org/wiki/Downtime), -so we don't have to worry during the deployment while someone might be reviewing the site. -Our deployment plan is to clone the latest release from GitLab repository, install the Composer dependencies and finally, activate the new release. - -#### @setup directive - -The first step of our deployment process is to define a set of variables within [@setup](https://laravel.com/docs/envoy/#setup) directive. -You may change the `app` to your application's name: - - -```php -... - -@setup - $repository = 'git@gitlab.example.com:<USERNAME>/laravel-sample.git'; - $releases_dir = '/var/www/app/releases'; - $app_dir = '/var/www/app'; - $release = date('YmdHis'); - $new_release_dir = $releases_dir .'/'. $release; -@endsetup - -... -``` - -- `$repository` is the address of our repository -- `$releases_dir` directory is where we deploy the app -- `$app_dir` is the actual location of the app that is live on the server -- `$release` contains a date, so every time that we deploy a new release of our app, we get a new folder with the current date as name -- `$new_release_dir` is the full path of the new release which is used just to make the tasks cleaner - -#### @story directive - -The [@story](https://laravel.com/docs/envoy/#stories) directive allows us define a list of tasks that can be run as a single task. -Here we have three tasks called `clone_repository`, `run_composer`, `update_symlinks`. These variables are usable to making our task's codes more cleaner: - - -```php -... - -@story('deploy') - clone_repository - run_composer - update_symlinks -@endstory - -... -``` - -Let's create these three tasks one by one. - -#### Clone the repository - -The first task will create the `releases` directory (if it doesn't exist), and then clone the `master` branch of the repository (by default) into the new release directory, given by the `$new_release_dir` variable. -The `releases` directory will hold all our deployments: - -```php -... - -@task('clone_repository') - echo 'Cloning repository' - [ -d {{ $releases_dir }} ] || mkdir {{ $releases_dir }} - git clone --depth 1 {{ $repository }} {{ $new_release_dir }} -@endtask - -... -``` - -While our project grows, its Git history will be very very long over time. -Since we are creating a directory per release, it might not be necessary to have the history of the project downloaded for each release. -The `--depth 1` option is a great solution which saves systems time and disk space as well. - -#### Installing dependencies with Composer - -As you may know, this task just navigates to the new release directory and runs Composer to install the application dependencies: - -```php -... - -@task('run_composer') - echo "Starting deployment ({{ $release }})" - cd {{ $new_release_dir }} - composer install --prefer-dist --no-scripts -q -o -@endtask - -... -``` - -#### Activate new release - -Next thing to do after preparing the requirements of our new release, is to remove the storage directory from it and to create two symbolic links to point the application's `storage` directory and `.env` file to the new release. -Then, we need to create another symbolic link to the new release with the name of `current` placed in the app directory. -The `current` symbolic link always points to the latest release of our app: - -```php -... - -@task('update_symlinks') - echo "Linking storage directory" - rm -rf {{ $new_release_dir }}/storage - ln -nfs {{ $app_dir }}/storage {{ $new_release_dir }}/storage - - echo 'Linking .env file' - ln -nfs {{ $app_dir }}/.env {{ $new_release_dir }}/.env - - echo 'Linking current release' - ln -nfs {{ $new_release_dir }} {{ $app_dir }}/current -@endtask -``` - -As you see, we use `-nfs` as an option for `ln` command, which says that the `storage`, `.env` and `current` no longer points to the preview's release and will point them to the new release by force (`f` from `-nfs` means force), which is the case when we are doing multiple deployments. - -### Full script - -The script is ready, but make sure to change the `deployer@192.168.1.1` to your server and also change `/var/www/app` with the directory you want to deploy your app. - -At the end, our `Envoy.blade.php` file will look like this: - -```php -@servers(['web' => 'deployer@192.168.1.1']) - -@setup - $repository = 'git@gitlab.example.com:<USERNAME>/laravel-sample.git'; - $releases_dir = '/var/www/app/releases'; - $app_dir = '/var/www/app'; - $release = date('YmdHis'); - $new_release_dir = $releases_dir .'/'. $release; -@endsetup - -@story('deploy') - clone_repository - run_composer - update_symlinks -@endstory - -@task('clone_repository') - echo 'Cloning repository' - [ -d {{ $releases_dir }} ] || mkdir {{ $releases_dir }} - git clone --depth 1 {{ $repository }} {{ $new_release_dir }} -@endtask - -@task('run_composer') - echo "Starting deployment ({{ $release }})" - cd {{ $new_release_dir }} - composer install --prefer-dist --no-scripts -q -o -@endtask - -@task('update_symlinks') - echo "Linking storage directory" - rm -rf {{ $new_release_dir }}/storage - ln -nfs {{ $app_dir }}/storage {{ $new_release_dir }}/storage - - echo 'Linking .env file' - ln -nfs {{ $app_dir }}/.env {{ $new_release_dir }}/.env - - echo 'Linking current release' - ln -nfs {{ $new_release_dir }} {{ $app_dir }}/current -@endtask -``` - -One more thing we should do before any deployment is to manually copy our application `storage` folder to the `/var/www/app` directory on the server for the first time. -You might want to create another Envoy task to do that for you. -We also create the `.env` file in the same path to setup our production environment variables for Laravel. -These are persistent data and will be shared to every new release. - -Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../ci/environments.md), which will be described [later](#setting-up-gitlab-ci-cd) in this tutorial. - -Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `master` branch. -To keep things simple, we commit directly to `master`, without using [feature-branches](../../workflow/gitlab_flow.md/#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. -In a real world project, teams may use [Issue Tracker](../../user/project/issues/index.md) and [Merge Requests](../../user/project/merge_requests/index.md) to move their code across branches: - -```bash -git add Envoy.blade.php -git commit -m 'Add Envoy' -git push origin master -``` - -## Continuous Integration with GitLab - -We have our app ready on GitLab, and we also can deploy it manually. -But let's take a step forward to do it automatically with [Continuous Delivery](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#continuous-delivery) method. -We need to check every commit with a set of automated tests to become aware of issues at the earliest, and then, we can deploy to the target environment if we are happy with the result of the tests. - -[GitLab CI/CD](../../ci/README.md) allows us to use [Docker](https://docker.com/) engine to handle the process of testing and deploying our app. -In the case you're not familiar with Docker, refer to [How to Automate Docker Deployments](http://paislee.io/how-to-automate-docker-deployments/). - -To be able to build, test, and deploy our app with GitLab CI/CD, we need to prepare our work environment. -To do that, we'll use a Docker image which has the minimum requirements that a Laravel app needs to run. -[There are other ways](../../ci/examples/php.md/#test-php-projects-using-the-docker-executor) to do that as well, but they may lead our builds run slowly, which is not what we want when there are faster options to use. - -With Docker images our builds run incredibly faster! - -### Create a Container Image - -Let's create a [Dockerfile](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Dockerfile) in the root directory of our app with the following content: - -```bash -# Set the base image for subsequent instructions -FROM php:7.1 - -# Update packages -RUN apt-get update - -# Install PHP and composer dependencies -RUN apt-get install -qq git curl libmcrypt-dev libjpeg-dev libpng-dev libfreetype6-dev libbz2-dev - -# Clear out the local repository of retrieved package files -RUN apt-get clean - -# Install needed extensions -# Here you can install any other extension that you need during the test and deployment process -RUN docker-php-ext-install mcrypt pdo_mysql zip - -# Install Composer -RUN curl --silent --show-error https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer - -# Install Laravel Envoy -RUN composer global require "laravel/envoy=~1.0" -``` - -We added the [official PHP 7.1 Docker image](https://hub.docker.com/r/_/php/), which consist of a minimum installation of Debian Jessie with PHP pre-installed, and works perfectly for our use case. - -We used `docker-php-ext-install` (provided by the official PHP Docker image) to install the PHP extensions we need. - -#### Setting Up GitLab Container Registry - -Now that we have our `Dockerfile` let's build and push it to our [GitLab Container Registry](../../user/project/container_registry.md). - -> The registry is the place to store and tag images for later use. Developers may want to maintain their own registry for private, company images, or for throw-away images used only in testing. Using GitLab Container Registry means you don't need to set up and administer yet another service or use a public registry. - -On your GitLab project repository navigate to the **Registry** tab. - - - -You may need to [enable Container Registry](../../user/project/container_registry.md#enable-the-container-registry-for-your-project) to your project to see this tab. You'll find it under your project's **Settings > General > Sharing and permissions**. - - - -To start using Container Registry on our machine, we first need to login to the GitLab registry using our GitLab username and password: - -```bash -docker login registry.gitlab.com -``` -Then we can build and push our image to GitLab: - -```bash -docker build -t registry.gitlab.com/<USERNAME>/laravel-sample . - -docker push registry.gitlab.com/<USERNAME>/laravel-sample -``` - ->**Note:** -To run the above commands, we first need to have [Docker](https://docs.docker.com/engine/installation/) installed on our machine. - -Congratulations! You just pushed the first Docker image to the GitLab Registry, and if you refresh the page you should be able to see it: - - - ->**Note:** -You can also [use GitLab CI/CD](https://about.gitlab.com/2016/05/23/gitlab-container-registry/#use-with-gitlab-ci) to build and push your Docker images, rather than doing that on your machine. - -We'll use this image further down in the `.gitlab-ci.yml` configuration file to handle the process of testing and deploying our app. - -Let's commit the `Dockerfile` file. - -```bash -git add Dockerfile -git commit -m 'Add Dockerfile' -git push origin master -``` - -### Setting up GitLab CI/CD - -In order to build and test our app with GitLab CI/CD, we need a file called `.gitlab-ci.yml` in our repository's root. It is similar to Circle CI and Travis CI, but built-in GitLab. - -Our `.gitlab-ci.yml` file will look like this: - -```yaml -image: registry.gitlab.com/<USERNAME>/laravel-sample:latest - -services: - - mysql:5.7 - -variables: - MYSQL_DATABASE: homestead - MYSQL_ROOT_PASSWORD: secret - DB_HOST: mysql - DB_USERNAME: root - -stages: - - test - - deploy - -unit_test: - stage: test - script: - - cp .env.example .env - - composer install - - php artisan key:generate - - php artisan migrate - - vendor/bin/phpunit - -deploy_production: - stage: deploy - script: - - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' - - eval $(ssh-agent -s) - - ssh-add <(echo "$SSH_PRIVATE_KEY") - - mkdir -p ~/.ssh - - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' - - - ~/.composer/vendor/bin/envoy run deploy - environment: - name: production - url: http://192.168.1.1 - when: manual - only: - - master -``` - -That's a lot to take in, isn't it? Let's run through it step by step. - -#### Image and Services - -[GitLab Runners](../../ci/runners/README.md) run the script defined by `.gitlab-ci.yml`. -The `image` keyword tells the Runners which image to use. -The `services` keyword defines additional images [that are linked to the main image](../../ci/docker/using_docker_images.md/#what-is-a-service). -Here we use the container image we created before as our main image and also use MySQL 5.7 as a service. - -```yaml -image: registry.gitlab.com/<USERNAME>/laravel-sample:latest - -services: - - mysql:5.7 - -... -``` - ->**Note:** -If you wish to test your app with different PHP versions and [database management systems](../../ci/services/README.md), you can define different `image` and `services` keywords for each test job. - -#### Variables - -GitLab CI/CD allows us to use [environment variables](../../ci/yaml/README.md#variables) in our jobs. -We defined MySQL as our database management system, which comes with a superuser root created by default. - -So we should adjust the configuration of MySQL instance by defining `MYSQL_DATABASE` variable as our database name and `MYSQL_ROOT_PASSWORD` variable as the password of `root`. -Find out more about MySQL variables at the [official MySQL Docker Image](https://hub.docker.com/r/_/mysql/). - -Also set the variables `DB_HOST` to `mysql` and `DB_USERNAME` to `root`, which are Laravel specific variables. -We define `DB_HOST` as `mysql` instead of `127.0.0.1`, as we use MySQL Docker image as a service which [is linked to the main Docker image](../../ci/docker/using_docker_images.md/#how-services-are-linked-to-the-build). - -```yaml -... - -variables: - MYSQL_DATABASE: homestead - MYSQL_ROOT_PASSWORD: secret - DB_HOST: mysql - DB_USERNAME: root - -... -``` - -#### Unit Test as the first job - -We defined the required shell scripts as an array of the [script](../../ci/yaml/README.md#script) variable to be executed when running `unit_test` job. - -These scripts are some Artisan commands to prepare the Laravel, and, at the end of the script, we'll run the tests by `PHPUnit`. - -```yaml -... - -unit_test: - script: - # Install app dependencies - - composer install - # Setup .env - - cp .env.example .env - # Generate an environment key - - php artisan key:generate - # Run migrations - - php artisan migrate - # Run tests - - vendor/bin/phpunit - -... -``` - -#### Deploy to production - -The job `deploy_production` will deploy the app to the production server. -To deploy our app with Envoy, we had to set up the `$SSH_PRIVATE_KEY` variable as an [SSH private key](../../ci/ssh_keys/README.md/#ssh-keys-when-using-the-docker-executor). -If the SSH keys have added successfully, we can run Envoy. - -As mentioned before, GitLab supports [Continuous Delivery](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#continuous-delivery) methods as well. -The [environment](../../ci/yaml/README.md#environment) keyword tells GitLab that this job deploys to the `production` environment. -The `url` keyword is used to generate a link to our application on the GitLab Environments page. -The `only` keyword tells GitLab CI that the job should be executed only when the pipeline is building the `master` branch. -Lastly, `when: manual` is used to turn the job from running automatically to a manual action. - -```yaml -... - -deploy_production: - script: - # Add the private SSH key to the build environment - - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' - - eval $(ssh-agent -s) - - ssh-add <(echo "$SSH_PRIVATE_KEY") - - mkdir -p ~/.ssh - - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' - - # Run Envoy - - ~/.composer/vendor/bin/envoy run deploy - - environment: - name: production - url: http://192.168.1.1 - when: manual - only: - - master -``` - -You may also want to add another job for [staging environment](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments), to final test your application before deploying to production. - -### Turn on GitLab CI/CD - -We have prepared everything we need to test and deploy our app with GitLab CI/CD. -To do that, commit and push `.gitlab-ci.yml` to the `master` branch. It will trigger a pipeline, which you can watch live under your project's **Pipelines**. - - - -Here we see our **Test** and **Deploy** stages. -The **Test** stage has the `unit_test` build running. -click on it to see the Runner's output. - - - -After our code passed through the pipeline successfully, we can deploy to our production server by clicking the **play** button on the right side. - - - -Once the deploy pipeline passed successfully, navigate to **Pipelines > Environments**. - - - -If something doesn't work as expected, you can roll back to the latest working version of your app. - - - -By clicking on the external link icon specified on the right side, GitLab opens the production website. -Our deployment successfully was done and we can see the application is live. - - - -In the case that you're interested to know how is the application directory structure on the production server after deployment, here are three directories named `current`, `releases` and `storage`. -As you know, the `current` directory is a symbolic link that points to the latest release. -The `.env` file consists of our Laravel environment variables. - - - -If you navigate to the `current` directory, you should see the application's content. -As you see, the `.env` is pointing to the `/var/www/app/.env` file and also `storage` is pointing to the `/var/www/app/storage/` directory. - - - -## Conclusion - -We configured GitLab CI to perform automated tests and used the method of [Continuous Delivery](https://continuousdelivery.com/) to deploy to production a Laravel application with Envoy, directly from the codebase. - -Envoy also was a great match to help us deploy the application without writing our custom bash script and doing Linux magics. +This document was moved to [another location](../../ci/examples/laravel_with_gitlab_and_envoy/index.md). diff --git a/doc/articles/numerous_undo_possibilities_in_git/img/branching.png b/doc/articles/numerous_undo_possibilities_in_git/img/branching.png Binary files differdeleted file mode 100644 index 9a80c211c99..00000000000 --- a/doc/articles/numerous_undo_possibilities_in_git/img/branching.png +++ /dev/null diff --git a/doc/articles/numerous_undo_possibilities_in_git/img/rebase_reset.png b/doc/articles/numerous_undo_possibilities_in_git/img/rebase_reset.png Binary files differdeleted file mode 100644 index ac7ea9ecddc..00000000000 --- a/doc/articles/numerous_undo_possibilities_in_git/img/rebase_reset.png +++ /dev/null diff --git a/doc/articles/numerous_undo_possibilities_in_git/img/revert.png b/doc/articles/numerous_undo_possibilities_in_git/img/revert.png Binary files differdeleted file mode 100644 index 13b3a35ca45..00000000000 --- a/doc/articles/numerous_undo_possibilities_in_git/img/revert.png +++ /dev/null diff --git a/doc/articles/numerous_undo_possibilities_in_git/index.md b/doc/articles/numerous_undo_possibilities_in_git/index.md index 895bbccec08..3f46ee9a5e6 100644 --- a/doc/articles/numerous_undo_possibilities_in_git/index.md +++ b/doc/articles/numerous_undo_possibilities_in_git/index.md @@ -1,497 +1 @@ -# Numerous undo possibilities in Git - -> **Article [Type](../../development/writing_documentation.md#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Crt Mori](https://gitlab.com/Letme) || -> **Publication date:** 2017-08-17 - -## Introduction - -In this tutorial, we will show you different ways of undoing your work in Git, for which -we will assume you have a basic working knowledge of. Check GitLab's -[Git documentation](../../topics/git/index.md#git-documentation) for reference. -Also, we will only provide some general info of the commands, which is enough -to get you started for the easy cases/examples, but for anything more advanced please refer to the [Git book](https://git-scm.com/book/en/v2). - -We will explain a few different techniques to undo your changes based on the stage -of the change in your current development. Also, keep in mind that [nothing in -Git is really deleted.][git-autoclean-ref] -This means that until Git automatically cleans detached commits (which cannot be -accessed by branch or tag) it will be possible to view them with `git reflog` command -and access them with direct commit-id. Read more about _[redoing the undo](#redoing-the-undo)_ on the section below. - -This guide is organized depending on the [stage of development][git-basics] -where you want to undo your changes from and if they were shared with other developers -or not. Because Git is tracking changes a created or edited file is in the unstaged state -(if created it is untracked by Git). After you add it to a repository (`git add`) you put -a file into the **staged** state, which is then committed (`git commit`) to your -local repository. After that, file can be shared with other developers (`git push`). -Here's what we'll cover in this tutorial: - - - [Undo local changes](#undo-local-changes) which were not pushed to remote repository - - - Before you commit, in both unstaged and staged state - - After you committed - - - Undo changes after they are pushed to remote repository - - - [Without history modification](#undo-remote-changes-without-changing-history) (preferred way) - - [With history modification](#undo-remote-changes-with-modifying-history) (requires - coordination with team and force pushes). - - - [Usecases when modifying history is generally acceptable](#where-modifying-history-is-generally-acceptable) - - [How to modify history](#how-modifying-history-is-done) - - [How to remove sensitive information from repository](#deleting-sensitive-information-from-commits) - - -### Branching strategy - -[Git][git-official] is a de-centralized version control system, which means that beside regular -versioning of the whole repository, it has possibilities to exchange changes -with other repositories. To avoid chaos with -[multiple sources of truth][git-distributed], various -development workflows have to be followed, and it depends on your internal -workflow how certain changes or commits can be undone or changed. -[GitLab Flow][gitlab-flow] provides a good -balance between developers clashing with each other while -developing the same feature and cooperating seamlessly, but it does not enable -joined development of the same feature by multiple developers by default. -When multiple developers develop the same feature on the same branch, clashing -with every synchronization is unavoidable, but a proper or chosen Git Workflow will -prevent that anything is lost or out of sync when feature is complete. You can also -read through this blog post on [Git Tips & Tricks][gitlab-git-tips-n-tricks] -to learn how to easily **do** things in Git. - - -## Undo local changes - -Until you push your changes to any remote repository, they will only affect you. -That broadens your options on how to handle undoing them. Still, local changes -can be on various stages and each stage has a different approach on how to tackle them. - - -### Unstaged local changes (before you commit) - -When a change is made, but it is not added to the staged tree, Git itself -proposes a solution to discard changes to certain file. - -Suppose you edited a file to change the content using your favorite editor: - -```shell -vim <file> -``` - -Since you did not `git add <file>` to staging, it should be under unstaged files (or -untracked if file was created). You can confirm that with: - -```shell -$ git status -On branch master -Your branch is up-to-date with 'origin/master'. -Changes not staged for commit: - (use "git add <file>..." to update what will be committed) - (use "git checkout -- <file>..." to discard changes in working directory) - - modified: <file> -no changes added to commit (use "git add" and/or "git commit -a") -``` - -At this point there are 3 options to undo the local changes you have: - - - Discard all local changes, but save them for possible re-use [later](#quickly-save-local-changes) - - ```shell - git stash - ``` - - - Discarding local changes (permanently) to a file - - ```shell - git checkout -- <file> - ``` - - - Discard all local changes to all files permanently - - ```shell - git reset --hard - ``` - - -Before executing `git reset --hard`, keep in mind that there is also a way to -just temporary store the changes without committing them using `git stash`. -This command resets the changes to all files, but it also saves them in case -you would like to apply them at some later time. You can read more about it in -[section below](#quickly-save-local-changes). - -### Quickly save local changes - -You are working on a feature when a boss drops by with an urgent task. Since your -feature is not complete, but you need to swap to another branch, you can use -`git stash` to save what you had done, swap to another branch, commit, push, -test, then get back to previous feature branch, do `git stash pop` and continue -where you left. - -The example above shows that discarding all changes is not always a preferred option, -but Git provides a way to save them for later, while resetting the repository to state without -them. This is achieved by Git stashing command `git stash`, which in fact saves your -current work and runs `git reset --hard`, but it also has various -additional options like: - - - `git stash save`, which enables including temporary commit message, which will help you identify changes, among with other options - - `git stash list`, which lists all previously stashed commits (yes, there can be more) that were not `pop`ed - - `git stash pop`, which redoes previously stashed changes and removes them from stashed list - - `git stash apply`, which redoes previously stashed changes, but keeps them on stashed list - -### Staged local changes (before you commit) - -Let's say you have added some files to staging, but you want to remove them from the -current commit, yet you want to retain those changes - just move them outside -of the staging tree. You also have an option to discard all changes with -`git reset --hard` or think about `git stash` [as described earlier.](#quickly-save-local-changes) - -Lets start the example by editing a file, with your favorite editor, to change the -content and add it to staging - -``` -vim <file> -git add <file> -``` - -The file is now added to staging as confirmed by `git status` command: - -```shell -$ git status -On branch master -Your branch is up-to-date with 'origin/master'. -Changes to be committed: - (use "git reset HEAD <file>..." to unstage) - - new file: <file> -``` - -Now you have 4 options to undo your changes: - - - Unstage the file to current commit (HEAD) - - ```shell - git reset HEAD <file> - ``` - - - Unstage everything - retain changes - - ```shell - git reset - ``` - - - Discard all local changes, but save them for [later](#quickly-save-local-changes) - - ```shell - git stash - ``` - - - Discard everything permanently - - ```shell - git reset --hard - ``` - -## Committed local changes - -Once you commit, your changes are recorded by the version control system. -Because you haven't pushed to your remote repository yet, your changes are -still not public (or shared with other developers). At this point, undoing -things is a lot easier, we have quite some workaround options. Once you push -your code, you'll have less options to troubleshoot your work. - -### Without modifying history - -Through the development process some of the previously committed changes do not -fit anymore in the end solution, or are source of the bugs. Once you find the -commit which triggered bug, or once you have a faulty commit, you can simply -revert it with `git revert commit-id`. This command inverts (swaps) the additions and -deletions in that commit, so that it does not modify history. Retaining history -can be helpful in future to notice that some changes have been tried -unsuccessfully in the past. - -In our example we will assume there are commits `A`,`B`,`C`,`D`,`E` committed in this order: `A-B-C-D-E`, -and `B` is the commit you want to undo. There are many different ways to identify commit -`B` as bad, one of them is to pass a range to `git bisect` command. The provided range includes -last known good commit (we assume `A`) and first known bad commit (where bug was detected - we will assume `E`). - -```shell -git bisect A..E -``` - -Bisect will provide us with commit-id of the middle commit to test, and then guide us -through simple bisection process. You can read more about it [in official Git Tools][git-debug] -In our example we will end up with commit `B`, that introduced bug/error. We have -4 options on how to remove it (or part of it) from our repository. - -- Undo (swap additions and deletions) changes introduced by commit `B`. - - ```shell - git revert commit-B-id - ``` - -- Undo changes on a single file or directory from commit `B`, but retain them in the staged state - - ```shell - git checkout commit-B-id <file> - ``` - -- Undo changes on a single file or directory from commit `B`, but retain them in the unstaged state - - ```shell - git reset commit-B-id <file> - ``` - - - There is one command we also must not forget: **creating a new branch** - from the point where changes are not applicable or where the development has hit a - dead end. For example you have done commits `A-B-C-D` on your feature-branch - and then you figure `C` and `D` are wrong. At this point you either reset to `B` - and do commit `F` (which will cause problems with pushing and if forced pushed also with other developers) - since branch now looks `A-B-F`, which clashes with what other developers have locally (you will - [change history](#with-history-modification)), or you simply checkout commit `B` create - a new branch and do commit `F`. In the last case, everyone else can still do their work while you - have your new way to get it right and merge it back in later. Alternatively, with GitLab, - you can [cherry-pick](../../user/project/merge_requests/cherry_pick_changes.md#cherry-picking-a-commit) - that commit into a new merge request. - -  - - ```shell - git checkout commit-B-id - git checkout -b new-path-of-feature - # Create <commit F> - git commit -a - ``` - -### With history modification - -There is one command for history modification and that is `git rebase`. Command -provides interactive mode (`-i` flag) which enables you to: - - - **reword** commit messages (there is also `git commit --amend` for editing - last commit message) - - **edit** the commit content (changes introduced by commit) and message - - **squash** multiple commits into a single one, and have a custom or aggregated - commit message - - **drop** commits - simply delete them - - and few more options - -Let us check few examples. Again there are commits `A-B-C-D` where you want to -delete commit `B`. - -- Rebase the range from current commit D to A: - - ```shell - git rebase -i A - ``` - -- Command opens your favorite editor where you write `drop` in front of commit - `B`, but you leave default `pick` with all other commits. Save and exit the - editor to perform a rebase. Remember: if you want to cancel delete whole - file content before saving and exiting the editor - -In case you want to modify something introduced in commit `B`. - -- Rebase the range from current commit D to A: - - ```shell - git rebase -i A - ``` - -- Command opens your favorite text editor where you write `edit` in front of commit - `B`, but leave default `pick` with all other commits. Save and exit the editor to - perform a rebase - -- Now do your edits and commit changes: - - ```shell - git commit -a - ``` - -You can find some more examples in [below section where we explain how to modify -history](#how-modifying-history-is-done) - - -### Redoing the Undo - -Sometimes you realize that the changes you undid were useful and you want them -back. Well because of first paragraph you are in luck. Command `git reflog` -enables you to *recall* detached local commits by referencing or applying them -via commit-id. Although, do not expect to see really old commits in reflog, because -Git regularly [cleans the commits which are *unreachable* by branches or tags][git-autoclean-ref]. - -To view repository history and to track older commits you can use below command: - -```shell -$ git reflog show - -# Example output: -b673187 HEAD@{4}: merge 6e43d5987921bde189640cc1e37661f7f75c9c0b: Merge made by the 'recursive' strategy. -eb37e74 HEAD@{5}: rebase -i (finish): returning to refs/heads/master -eb37e74 HEAD@{6}: rebase -i (pick): Commit C -97436c6 HEAD@{7}: rebase -i (start): checkout 97436c6eec6396c63856c19b6a96372705b08b1b -... -88f1867 HEAD@{12}: commit: Commit D -97436c6 HEAD@{13}: checkout: moving from 97436c6eec6396c63856c19b6a96372705b08b1b to test -97436c6 HEAD@{14}: checkout: moving from master to 97436c6 -05cc326 HEAD@{15}: commit: Commit C -6e43d59 HEAD@{16}: commit: Commit B -``` - -Output of command shows repository history. In first column there is commit-id, -in following column, number next to `HEAD` indicates how many commits ago something -was made, after that indicator of action that was made (commit, rebase, merge, ...) -and then on end description of that action. - -## Undo remote changes without changing history - -This topic is roughly same as modifying committed local changes without modifying -history. **It should be the preferred way of undoing changes on any remote repository -or public branch.** Keep in mind that branching is the best solution when you want -to retain the history of faulty development, yet start anew from certain point. Branching -enables you to include the existing changes in new development (by merging) and -it also provides a clear timeline and development structure. - - - -If you want to revert changes introduced in certain `commit-id` you can simply -revert that `commit-id` (swap additions and deletions) in newly created commit: -You can do this with - -```shell -git revert commit-id -``` - -or creating a new branch: - -```shell -git checkout commit-id -git checkout -b new-path-of-feature -``` - -## Undo remote changes with modifying history - -This is useful when you want to *hide* certain things - like secret keys, -passwords, SSH keys, etc. It is and should not be used to hide mistakes, as -it will make it harder to debug in case there are some other bugs. The main -reason for this is that you loose the real development progress. **Also keep in -mind that, even with modified history, commits are just detached and can still be -accessed through commit-id** - at least until all repositories perform -the cleanup of detached commits (happens automatically). - - - -### Where modifying history is generally acceptable - -Modified history breaks the development chain of other developers, as changed -history does not have matching commits'ids. For that reason it should not -be used on any public branch or on branch that *might* be used by other -developers. When contributing to big open source repositories (e.g. [GitLab CE][gitlab-ce]), -it is acceptable to *squash* commits into a single one, to present -a nicer history of your contribution. -Keep in mind that this also removes the comments attached to certain commits -in merge requests, so if you need to retain traceability in GitLab, then -modifying history is not acceptable. -A feature-branch of a merge request is a public branch and might be used by -other developers, but project process and rules might allow or require -you to use `git rebase` (command that changes history) to reduce number of -displayed commits on target branch after reviews are done (for example -GitLab). There is a `git merge --squash` command which does exactly that -(squashes commits on feature-branch to a single commit on target branch -at merge). - ->**Note:** -Never modify the commit history of `master` or shared branch - -### How modifying history is done - -After you know what you want to modify (how far in history or how which range of -old commits), use `git rebase -i commit-id`. This command will then display all the commits from -current version to chosen commit-id and allow modification, squashing, deletion -of that commits. - -```shell -$ git rebase -i commit1-id..commit3-id -pick <commit1-id> <commit1-commit-message> -pick <commit2-id> <commit2-commit-message> -pick <commit3-id> <commit3-commit-message> - -# Rebase commit1-id..commit3-id onto <commit4-id> (3 command(s)) -# -# Commands: -# p, pick = use commit -# r, reword = use commit, but edit the commit message -# e, edit = use commit, but stop for amending -# s, squash = use commit, but meld into previous commit -# f, fixup = like "squash", but discard this commit's log message -# x, exec = run command (the rest of the line) using shell -# d, drop = remove commit -# -# These lines can be re-ordered; they are executed from top to bottom. -# -# If you remove a line here THAT COMMIT WILL BE LOST. -# -# However, if you remove everything, the rebase will be aborted. -# -# Note that empty commits are commented out -``` - ->**Note:** -It is important to notice that comment from the output clearly states that, if -you decide to abort, then do not just close your editor (as that will in-fact -modify history), but remove all uncommented lines and save. - -That is one of the reasons why `git rebase` should be used carefully on -shared and remote branches. But don't worry, there will be nothing broken until -you push back to the remote repository (so you can freely explore the -different outcomes locally). - -```shell -# Modify history from commit-id to HEAD (current commit) -git rebase -i commit-id -``` - -### Deleting sensitive information from commits - -Git also enables you to delete sensitive information from your past commits and -it does modify history in the progress. That is why we have included it in this -section and not as a standalone topic. To do so, you should run the -`git filter-branch`, which enables you to rewrite history with -[certain filters][git-filters-manual]. -This command uses rebase to modify history and if you want to remove certain -file from history altogether use: - -```shell -git filter-branch --tree-filter 'rm filename' HEAD -``` - -Since `git filter-branch` command might be slow on big repositories, there are -tools that can use some of Git specifics to enable faster execution of common -tasks (which is exactly what removing sensitive information file is about). -An alternative is [BFG Repo-cleaner][bfg-repo-cleaner]. Keep in mind that these -tools are faster because they do not provide a same fully feature set as `git filter-branch` -does, but focus on specific usecases. - -## Conclusion - -There are various options of undoing your work with any version control system, but -because of de-centralized nature of Git, these options are multiplied (or limited) -depending on the stage of your process. Git also enables rewriting history, but that -should be avoided as it might cause problems when multiple developers are -contributing to the same codebase. - -<!-- Identifiers, in alphabetical order --> - -[bfg-repo-cleaner]: https://rtyley.github.io/bfg-repo-cleaner/ -[git-autoclean-ref]: https://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery -[git-basics]: https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository -[git-debug]: https://git-scm.com/book/en/v2/Git-Tools-Debugging-with-Git -[git-distributed]: https://git-scm.com/about/distributed -[git-filters-manual]: https://git-scm.com/docs/git-filter-branch#_options -[git-official]: https://git-scm.com/ -[gitlab-ce]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#contribution-acceptance-criteria -[gitlab-flow]: https://about.gitlab.com/2014/09/29/gitlab-flow/ -[gitlab-git-tips-n-tricks]: https://about.gitlab.com/2016/12/08/git-tips-and-tricks/ +This document was moved to [another location](../../topics/git/numerous_undo_possibilities_in_git/index.md). diff --git a/doc/articles/openshift_and_gitlab/img/add-gitlab-to-project.png b/doc/articles/openshift_and_gitlab/img/add-gitlab-to-project.png Binary files differdeleted file mode 100644 index fcad4e59ae3..00000000000 --- a/doc/articles/openshift_and_gitlab/img/add-gitlab-to-project.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/add-to-project.png b/doc/articles/openshift_and_gitlab/img/add-to-project.png Binary files differdeleted file mode 100644 index bd915a229f6..00000000000 --- a/doc/articles/openshift_and_gitlab/img/add-to-project.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/create-project-ui.png b/doc/articles/openshift_and_gitlab/img/create-project-ui.png Binary files differdeleted file mode 100644 index e72866f252a..00000000000 --- a/doc/articles/openshift_and_gitlab/img/create-project-ui.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/gitlab-logs.png b/doc/articles/openshift_and_gitlab/img/gitlab-logs.png Binary files differdeleted file mode 100644 index 1e24080c7df..00000000000 --- a/doc/articles/openshift_and_gitlab/img/gitlab-logs.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/gitlab-overview.png b/doc/articles/openshift_and_gitlab/img/gitlab-overview.png Binary files differdeleted file mode 100644 index 3c5df0ea101..00000000000 --- a/doc/articles/openshift_and_gitlab/img/gitlab-overview.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/gitlab-running.png b/doc/articles/openshift_and_gitlab/img/gitlab-running.png Binary files differdeleted file mode 100644 index c7db691cb30..00000000000 --- a/doc/articles/openshift_and_gitlab/img/gitlab-running.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/gitlab-scale.png b/doc/articles/openshift_and_gitlab/img/gitlab-scale.png Binary files differdeleted file mode 100644 index 4903c7d7498..00000000000 --- a/doc/articles/openshift_and_gitlab/img/gitlab-scale.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/gitlab-settings.png b/doc/articles/openshift_and_gitlab/img/gitlab-settings.png Binary files differdeleted file mode 100644 index db4360ffef0..00000000000 --- a/doc/articles/openshift_and_gitlab/img/gitlab-settings.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/no-resources.png b/doc/articles/openshift_and_gitlab/img/no-resources.png Binary files differdeleted file mode 100644 index 480fb766468..00000000000 --- a/doc/articles/openshift_and_gitlab/img/no-resources.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/openshift-infra-project.png b/doc/articles/openshift_and_gitlab/img/openshift-infra-project.png Binary files differdeleted file mode 100644 index 8b9f85aa341..00000000000 --- a/doc/articles/openshift_and_gitlab/img/openshift-infra-project.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/pods-overview.png b/doc/articles/openshift_and_gitlab/img/pods-overview.png Binary files differdeleted file mode 100644 index e1cf08bd217..00000000000 --- a/doc/articles/openshift_and_gitlab/img/pods-overview.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/rc-name.png b/doc/articles/openshift_and_gitlab/img/rc-name.png Binary files differdeleted file mode 100644 index 889e34adbec..00000000000 --- a/doc/articles/openshift_and_gitlab/img/rc-name.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/running-pods.png b/doc/articles/openshift_and_gitlab/img/running-pods.png Binary files differdeleted file mode 100644 index 3fd4e56662f..00000000000 --- a/doc/articles/openshift_and_gitlab/img/running-pods.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/storage-volumes.png b/doc/articles/openshift_and_gitlab/img/storage-volumes.png Binary files differdeleted file mode 100644 index ae1e5381faa..00000000000 --- a/doc/articles/openshift_and_gitlab/img/storage-volumes.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/img/web-console.png b/doc/articles/openshift_and_gitlab/img/web-console.png Binary files differdeleted file mode 100644 index aa1425d4f94..00000000000 --- a/doc/articles/openshift_and_gitlab/img/web-console.png +++ /dev/null diff --git a/doc/articles/openshift_and_gitlab/index.md b/doc/articles/openshift_and_gitlab/index.md index c0bbcfe2a8a..b7594cfef7f 100644 --- a/doc/articles/openshift_and_gitlab/index.md +++ b/doc/articles/openshift_and_gitlab/index.md @@ -1,510 +1 @@ -# Getting started with OpenShift Origin 3 and GitLab - -> **Article [Type](../../development/writing_documentation.html#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Achilleas Pipinellis](https://gitlab.com/axil) || -> **Publication date:** 2016-06-28 - -## Introduction - -[OpenShift Origin][openshift] is an open source container application -platform created by [RedHat], based on [kubernetes] and [Docker]. That means -you can host your own PaaS for free and almost with no hassle. - -In this tutorial, we will see how to deploy GitLab in OpenShift using GitLab's -official Docker image while getting familiar with the web interface and CLI -tools that will help us achieve our goal. - ---- - -## Prerequisites - -OpenShift 3 is not yet deployed on RedHat's offered Online platform ([openshift.com]), -so in order to test it, we will use an [all-in-one Virtualbox image][vm] that is -offered by the OpenShift developers and managed by Vagrant. If you haven't done -already, go ahead and install the following components as they are essential to -test OpenShift easily: - -- [VirtualBox] -- [Vagrant] -- [OpenShift Client][oc] (`oc` for short) - -It is also important to mention that for the purposes of this tutorial, the -latest Origin release is used: - -- **oc** `v1.3.0` (must be [installed][oc-gh] locally on your computer) -- **openshift** `v1.3.0` (is pre-installed in the [VM image][vm-new]) -- **kubernetes** `v1.3.0` (is pre-installed in the [VM image][vm-new]) - ->**Note:** -If you intend to deploy GitLab on a production OpenShift cluster, there are some -limitations to bare in mind. Read on the [limitations](#current-limitations) -section for more information and follow the linked links for the relevant -discussions. - -Now that you have all batteries, let's see how easy it is to test OpenShift -on your computer. - -## Getting familiar with OpenShift Origin - -The environment we are about to use is based on CentOS 7 which comes with all -the tools needed pre-installed: Docker, kubernetes, OpenShift, etcd. - -### Test OpenShift using Vagrant - -As of this writing, the all-in-one VM is at version 1.3, and that's -what we will use in this tutorial. - -In short: - -1. Open a terminal and in a new directory run: - ```sh - vagrant init openshift/origin-all-in-one - ``` -1. This will generate a Vagrantfile based on the all-in-one VM image -1. In the same directory where you generated the Vagrantfile - enter: - - ```sh - vagrant up - ``` - -This will download the VirtualBox image and fire up the VM with some preconfigured -values as you can see in the Vagrantfile. As you may have noticed, you need -plenty of RAM (5GB in our example), so make sure you have enough. - -Now that OpenShift is setup, let's see how the web console looks like. - -### Explore the OpenShift web console - -Once Vagrant finishes its thing with the VM, you will be presented with a -message which has some important information. One of them is the IP address -of the deployed OpenShift platform and in particular <https://10.2.2.2:8443/console/>. -Open this link with your browser and accept the self-signed certificate in -order to proceed. - -Let's login as admin with username/password `admin/admin`. This is what the -landing page looks like: - - - -You can see that a number of [projects] are already created for testing purposes. - -If you head over the `openshift-infra` project, a number of services with their -respective pods are there to explore. - - - -We are not going to explore the whole interface, but if you want to learn about -the key concepts of OpenShift, read the [core concepts reference][core] in the -official documentation. - -### Explore the OpenShift CLI - -OpenShift Client (`oc`), is a powerful CLI tool that talks to the OpenShift API -and performs pretty much everything you can do from the web UI and much more. - -Assuming you have [installed][oc] it, let's explore some of its main -functionalities. - -Let's first see the version of `oc`: - -```sh -$ oc version - -oc v1.3.0 -kubernetes v1.3.0+52492b4 -``` - -With `oc help` you can see the top level arguments you can run with `oc` and -interact with your cluster, kubernetes, run applications, create projects and -much more. - -Let's login to the all-in-one VM and see how to achieve the same results like -when we visited the web console earlier. The username/password for the -administrator user is `admin/admin`. There is also a test user with username/ -password `user/user`, with limited access. Let's login as admin for the moment: - -```sh -$ oc login https://10.2.2.2:8443 - -Authentication required for https://10.2.2.2:8443 (openshift) -Username: admin -Password: -Login successful. - -You have access to the following projects and can switch between them with 'oc project <projectname>': - - * cockpit - * default (current) - * delete - * openshift - * openshift-infra - * sample - -Using project "default". -``` - -Switch to the `openshift-infra` project with: - -```sh -oc project openshift-infra -``` - -And finally, see its status: - -```sh -oc status -``` - -The last command should spit a bunch of information about the statuses of the -pods and the services, which if you look closely is what we encountered in the -second image when we explored the web console. - -You can always read more about `oc` in the [OpenShift CLI documentation][oc]. - -### Troubleshooting the all-in-one VM - -Using the all-in-one VM gives you the ability to test OpenShift whenever you -want. That means you get to play with it, shutdown the VM, and pick up where -you left off. - -Sometimes though, you may encounter some issues, like OpenShift not running -when booting up the VM. The web UI may not responding or you may see issues -when trying to login with `oc`, like: - -``` -The connection to the server 10.2.2.2:8443 was refused - did you specify the right host or port? -``` - -In that case, the OpenShift service might not be running, so in order to fix it: - -1. SSH into the VM by going to the directory where the Vagrantfile is and then - run: - - ```sh - vagrant ssh - ``` - -1. Run `systemctl` and verify by the output that the `openshift` service is not - running (it will be in red color). If that's the case start the service with: - - ```sh - sudo systemctl start openshift - ``` - -1. Verify the service is up with: - - ```sh - systemctl status openshift -l - ``` - -Now you will be able to login using `oc` (like we did before) and visit the web -console. - -## Deploy GitLab - -Now that you got a taste of what OpenShift looks like, let's deploy GitLab! - -### Create a new project - -First, we will create a new project to host our application. You can do this -either by running the CLI client: - -```bash -$ oc new-project gitlab -``` - -or by using the web interface: - - - -If you used the command line, `oc` automatically uses the new project and you -can see its status with: - -```sh -$ oc status - -In project gitlab on server https://10.2.2.2:8443 - -You have no services, deployment configs, or build configs. -Run 'oc new-app' to create an application. -``` - -If you visit the web console, you can now see `gitlab` listed in the projects list. - -The next step is to import the OpenShift template for GitLab. - -### Import the template - -The [template][templates] is basically a JSON file which describes a set of -related object definitions to be created together, as well as a set of -parameters for those objects. - -The template for GitLab resides in the Omnibus GitLab repository under the -docker directory. Let's download it locally with `wget`: - -```bash -wget https://gitlab.com/gitlab-org/omnibus-gitlab/raw/master/docker/openshift-template.json -``` - -And then let's import it in OpenShift: - -```bash -oc create -f openshift-template.json -n openshift -``` - ->**Note:** -The `-n openshift` namespace flag is a trick to make the template available to all -projects. If you recall from when we created the `gitlab` project, `oc` switched -to it automatically, and that can be verified by the `oc status` command. If -you omit the namespace flag, the application will be available only to the -current project, in our case `gitlab`. The `openshift` namespace is a global -one that the administrators should use if they want the application to be -available to all users. - -We are now ready to finally deploy GitLab! - -### Create a new application - -The next step is to use the template we previously imported. Head over to the -`gitlab` project and hit the **Add to Project** button. - - - -This will bring you to the catalog where you can find all the pre-defined -applications ready to deploy with the click of a button. Search for `gitlab` -and you will see the previously imported template: - - - -Select it, and in the following screen you will be presented with the predefined -values used with the GitLab template: - - - -Notice at the top that there are three resources to be created with this -template: - -- `gitlab-ce` -- `gitlab-ce-redis` -- `gitlab-ce-postgresql` - -While PostgreSQL and Redis are bundled in Omnibus GitLab, the template is using -separate images as you can see from [this line][line] in the template. - -The predefined values have been calculated for the purposes of testing out -GitLab in the all-in-one VM. You don't need to change anything here, hit -**Create** to start the deployment. - -If you are deploying to production you will want to change the **GitLab instance -hostname** and use greater values for the volume sizes. If you don't provide a -password for PostgreSQL, it will be created automatically. - ->**Note:** -The `gitlab.apps.10.2.2.2.xip.io` hostname that is used by default will -resolve to the host with IP `10.2.2.2` which is the IP our VM uses. It is a -trick to have distinct FQDNs pointing to services that are on our local network. -Read more on how this works in <http://xip.io>. - -Now that we configured this, let's see how to manage and scale GitLab. - -## Manage and scale GitLab - -Setting up GitLab for the first time might take a while depending on your -internet connection and the resources you have attached to the all-in-one VM. -GitLab's docker image is quite big (~500MB), so you'll have to wait until -it's downloaded and configured before you use it. - -### Watch while GitLab gets deployed - -Navigate to the `gitlab` project at **Overview**. You can notice that the -deployment is in progress by the orange color. The Docker images are being -downloaded and soon they will be up and running. - - - -Switch to the **Browse > Pods** and you will eventually see all 3 pods in a -running status. Remember the 3 resources that were to be created when we first -created the GitLab app? This is where you can see them in action. - - - -You can see GitLab being reconfigured by taking look at the logs in realtime. -Click on `gitlab-ce-2-j7ioe` (your ID will be different) and go to the **Logs** -tab. - - - -At a point you should see a _**gitlab Reconfigured!**_ message in the logs. -Navigate back to the **Overview** and hopefully all pods will be up and running. - - - -Congratulations! You can now navigate to your new shinny GitLab instance by -visiting <http://gitlab.apps.10.2.2.2.xip.io> where you will be asked to -change the root user password. Login using `root` as username and providing the -password you just set, and start using GitLab! - -### Scale GitLab with the push of a button - -If you reach to a point where your GitLab instance could benefit from a boost -of resources, you'd be happy to know that you can scale up with the push of a -button. - -In the **Overview** page just click the up arrow button in the pod where -GitLab is. The change is instant and you can see the number of [replicas] now -running scaled to 2. - - - -Upping the GitLab pods is actually like adding new application servers to your -cluster. You can see how that would work if you didn't use GitLab with -OpenShift by following the [HA documentation][ha] for the application servers. - -Bare in mind that you may need more resources (CPU, RAM, disk space) when you -scale up. If a pod is in pending state for too long, you can navigate to -**Browse > Events** and see the reason and message of the state. - - - -### Scale GitLab using the `oc` CLI - -Using `oc` is super easy to scale up the replicas of a pod. You may want to -skim through the [basic CLI operations][basic-cli] to get a taste how the CLI -commands are used. Pay extra attention to the object types as we will use some -of them and their abbreviated versions below. - -In order to scale up, we need to find out the name of the replication controller. -Let's see how to do that using the following steps. - -1. Make sure you are in the `gitlab` project: - - ```sh - oc project gitlab - ``` - -1. See what services are used for this project: - - ```sh - oc get svc - ``` - - The output will be similar to: - - ``` - NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE - gitlab-ce 172.30.243.177 <none> 22/TCP,80/TCP 5d - gitlab-ce-postgresql 172.30.116.75 <none> 5432/TCP 5d - gitlab-ce-redis 172.30.105.88 <none> 6379/TCP 5d - ``` - -1. We need to see the replication controllers of the `gitlab-ce` service. - Get a detailed view of the current ones: - - ```sh - oc describe rc gitlab-ce - ``` - - This will return a large detailed list of the current replication controllers. - Search for the name of the GitLab controller, usually `gitlab-ce-1` or if - that failed at some point and you spawned another one, it will be named - `gitlab-ce-2`. - -1. Scale GitLab using the previous information: - - ```sh - oc scale --replicas=2 replicationcontrollers gitlab-ce-2 - ``` - -1. Get the new replicas number to make sure scaling worked: - - ```sh - oc get rc gitlab-ce-2 - ``` - - which will return something like: - - ``` - NAME DESIRED CURRENT AGE - gitlab-ce-2 2 2 5d - ``` - -And that's it! We successfully scaled the replicas to 2 using the CLI. - -As always, you can find the name of the controller using the web console. Just -click on the service you are interested in and you will see the details in the -right sidebar. - - - -### Autoscaling GitLab - -In case you were wondering whether there is an option to autoscale a pod based -on the resources of your server, the answer is yes, of course there is. - -We will not expand on this matter, but feel free to read the documentation on -OpenShift's website about [autoscaling]. - -## Current limitations - -As stated in the [all-in-one VM][vm] page: - -> By default, OpenShift will not allow a container to run as root or even a -non-random container assigned userid. Most Docker images in the Dockerhub do not -follow this best practice and instead run as root. - -The all-in-one VM we are using has this security turned off so it will not -bother us. In any case, it is something to keep in mind when deploying GitLab -on a production cluster. - -In order to deploy GitLab on a production cluster, you will need to assign the -GitLab service account to the `anyuid` Security Context. - -1. Edit the Security Context: - ```sh - oc edit scc anyuid - ``` - -1. Add `system:serviceaccount:<project>:gitlab-ce-user` to the `users` section. - If you changed the Application Name from the default the user will - will be `<app-name>-user` instead of `gitlab-ce-user` - -1. Save and exit the editor - -## Conclusion - -By now, you should have an understanding of the basic OpenShift Origin concepts -and a sense of how things work using the web console or the CLI. - -GitLab was hard to install in previous versions of OpenShift, -but now that belongs to the past. Upload a template, create a project, add an -application and you are done. You are ready to login to your new GitLab instance. - -And remember that in this tutorial we just scratched the surface of what Origin -is capable of. As always, you can refer to the detailed -[documentation][openshift-docs] to learn more about deploying your own OpenShift -PaaS and managing your applications with the ease of containers. - -[RedHat]: https://www.redhat.com/en "RedHat website" -[openshift]: https://www.openshift.org "OpenShift Origin website" -[vm]: https://www.openshift.org/vm/ "OpenShift All-in-one VM" -[vm-new]: https://atlas.hashicorp.com/openshift/boxes/origin-all-in-one "Official OpenShift Vagrant box on Atlas" -[template]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/docker/openshift-template.json "OpenShift template for GitLab" -[openshift.com]: https://openshift.com "OpenShift Online" -[kubernetes]: http://kubernetes.io/ "Kubernetes website" -[Docker]: https://www.docker.com "Docker website" -[oc]: https://docs.openshift.org/latest/cli_reference/get_started_cli.html "Documentation - oc CLI documentation" -[VirtualBox]: https://www.virtualbox.org/wiki/Downloads "VirtualBox downloads" -[Vagrant]: https://www.vagrantup.com/downloads.html "Vagrant downloads" -[projects]: https://docs.openshift.org/latest/dev_guide/projects.html "Documentation - Projects overview" -[core]: https://docs.openshift.org/latest/architecture/core_concepts/index.html "Documentation - Core concepts of OpenShift Origin" -[templates]: https://docs.openshift.org/latest/architecture/core_concepts/templates.html "Documentation - OpenShift templates" -[old-post]: https://blog.openshift.com/deploy-gitlab-openshift/ "Old post - Deploy GitLab on OpenShift" -[line]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/658c065c8d022ce858dd63eaeeadb0b2ddc8deea/docker/openshift-template.json#L239 "GitLab - OpenShift template" -[oc-gh]: https://github.com/openshift/origin/releases/tag/v1.3.0 "Openshift 1.3.0 release on GitHub" -[ha]: http://docs.gitlab.com/ce/administration/high_availability/gitlab.html "Documentation - GitLab High Availability" -[replicas]: https://docs.openshift.org/latest/architecture/core_concepts/deployments.html#replication-controllers "Documentation - Replication controller" -[autoscaling]: https://docs.openshift.org/latest/dev_guide/pod_autoscaling.html "Documentation - Autoscale" -[basic-cli]: https://docs.openshift.org/latest/cli_reference/basic_cli_operations.html "Documentation - Basic CLI operations" -[openshift-docs]: https://docs.openshift.org "OpenShift documentation" +This document was moved to [another location](../../install/openshift_and_gitlab/index.html). diff --git a/doc/articles/runner_autoscale_aws/index.md b/doc/articles/runner_autoscale_aws/index.md index 9d4c4a57ce5..e2667aebc5f 100644 --- a/doc/articles/runner_autoscale_aws/index.md +++ b/doc/articles/runner_autoscale_aws/index.md @@ -1,410 +1 @@ ---- -last_updated: 2017-11-24 ---- - -> **[Article Type](../../development/writing_documentation.html#types-of-technical-articles):** Admin guide || -> **Level:** intermediary || -> **Author:** [Achilleas Pipinellis](https://gitlab.com/axil) || -> **Publication date:** 2017/11/24 - -# Autoscaling GitLab Runner on AWS - -One of the biggest advantages of GitLab Runner is its ability to automatically -spin up and down VMs to make sure your builds get processed immediately. It's a -great feature, and if used correctly, it can be extremely useful in situations -where you don't use your Runners 24/7 and want to have a cost-effective and -scalable solution. - -## Introduction - -In this tutorial, we'll explore how to properly configure a GitLab Runner in -AWS that will serve as the bastion where it will spawn new Docker machines on -demand. - -In addition, we'll make use of [Amazon's EC2 Spot instances][spot] which will -greatly reduce the costs of the Runner instances while still using quite -powerful autoscaling machines. - -## Prerequisites - -NOTE: **Note:** -A familiarity with Amazon Web Services (AWS) is required as this is where most -of the configuration will take place. - -Your GitLab instance is going to need to talk to the Runners over the network, -and that is something you need think about when configuring any AWS security -groups or when setting up your DNS configuration. - -For example, you can keep the EC2 resources segmented away from public traffic -in a different VPC to better strengthen your network security. Your environment -is likely different, so consider what works best for your situation. - -### AWS security groups - -Docker Machine will attempt to use a -[default security group](https://docs.docker.com/machine/drivers/aws/#security-group) -with rules for port `2376`, which is required for communication with the Docker -daemon. Instead of relying on Docker, you can create a security group with the -rules you need and provide that in the Runner options as we will -[see below](#the-runners-machine-section). This way, you can customize it to your -liking ahead of time based on your networking environment. - -### AWS credentials - -You'll need an [AWS Access Key](https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html) -tied to a user with permission to scale (EC2) and update the cache (via S3). -Create a new user with [policies](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-policies-for-amazon-ec2.html) -for EC2 (AmazonEC2FullAccess) and S3 (AmazonS3FullAccess). To be more secure, -you can disable console login for that user. Keep the tab open or copy paste the -security credentials in an editor as we'll use them later during the -[Runner configuration](#the-runners-machine-section). - -## Prepare the bastion instance - -The first step is to install GitLab Runner in an EC2 instance that will serve -as the bastion that spawns new machines. This doesn't have to be a powerful -machine since it will not run any jobs itself, a `t2.micro` instance will do. -This machine will be a dedicated host since we need it always up and running, -thus it will be the only standard cost. - -NOTE: **Note:** -For the bastion instance, choose a distribution that both Docker and GitLab -Runner support, for example either Ubuntu, Debian, CentOS or RHEL will work fine. - -Install the prerequisites: - -1. Log in to your server -1. [Install GitLab Runner from the official GitLab repository](https://docs.gitlab.com/runner/install/linux-repository.html) -1. [Install Docker](https://docs.docker.com/engine/installation/#server) -1. [Install Docker Machine](https://docs.docker.com/machine/install-machine/) - -Now that the Runner is installed, it's time to register it. - -## Registering the GitLab Runner - -Before configuring the GitLab Runner, you need to first register it, so that -it connects with your GitLab instance: - -1. [Obtain a Runner token](../../ci/runners/README.md) -1. [Register the Runner](https://docs.gitlab.com/runner/register/index.html#gnu-linux) -1. When asked the executor type, enter `docker+machine` - -You can now move on to the most important part, configuring the GitLab Runner. - -TIP: **Tip:** -If you want every user in your instance to be able to use the autoscaled Runners, -register the Runner as a shared one. - -## Configuring the GitLab Runner - -Now that the Runner is registered, you need to edit its configuration file and -add the required options for the AWS machine driver. - -Let's first break it down to pieces. - -### The global section - -In the global section, you can define the limit of the jobs that can be run -concurrently across all Runners (`concurrent`). This heavily depends on your -needs, like how many users your Runners will accommodate, how much time your -builds take, etc. You can start with something low like `10`, and increase or -decrease its value going forward. - -The `check_interval` option defines how often the Runner should check GitLab -for new jobs, in seconds. - -Example: - -```toml -concurrent = 10 -check_interval = 0 -``` - -[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) -about all the options you can use. - -### The `runners` section - -From the `[[runners]]` section, the most important part is the `executor` which -must be set to `docker+machine`. Most of those settings are taken care of when -you register the Runner for the first time. - -`limit` sets the maximum number of machines (running and idle) that this Runner -will spawn. For more info check the [relationship between `limit`, `concurrent` -and `IdleCount`](https://docs.gitlab.com/runner/configuration/autoscale.html#how-concurrent-limit-and-idlecount-generate-the-upper-limit-of-running-machines). - -Example: - -```toml -[[runners]] - name = "gitlab-aws-autoscaler" - url = "<URL of your GitLab instance>" - token = "<Runner's token>" - executor = "docker+machine" - limit = 20 -``` - -[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -about all the options you can use under `[[runners]]`. - -### The `runners.docker` section - -In the `[runners.docker]` section you can define the default Docker image to -be used by the child Runners if it's not defined in [`.gitlab-ci.yml`](../../ci/yaml/README.md). -By using `privileged = true`, all Runners will be able to run -[Docker in Docker](../../ci/docker/using_docker_build.md#use-docker-in-docker-executor) -which is useful if you plan to build your own Docker images via GitLab CI/CD. - -Next, we use `disable_cache = true` to disable the Docker executor's inner -cache mechanism since we will use the distributed cache mode as described -in the following section. - -Example: - -```toml - [runners.docker] - image = "alpine" - privileged = true - disable_cache = true -``` - -[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-docker-section) -about all the options you can use under `[runners.docker]`. - -### The `runners.cache` section - -To speed up your jobs, GitLab Runner provides a cache mechanism where selected -directories and/or files are saved and shared between subsequent jobs. -While not required for this setup, it is recommended to use the distributed cache -mechanism that GitLab Runner provides. Since new instances will be created on -demand, it is essential to have a common place where the cache is stored. - -In the following example, we use Amazon S3: - -```toml - [runners.cache] - Type = "s3" - ServerAddress = "s3.amazonaws.com" - AccessKey = "<your AWS Access Key ID>" - SecretKey = "<your AWS Secret Access Key>" - BucketName = "<the bucket where your cache should be kept>" - BucketLocation = "us-east-1" - Shared = true -``` - -Here's some more info to further explore the cache mechanism: - -- [Reference for `runners.cache`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-cache-section) -- [Deploying and using a cache server for GitLab Runner](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) -- [How cache works](../../ci/yaml/README.md#cache) - -### The `runners.machine` section - -This is the most important part of the configuration and it's the one that -tells GitLab Runner how and when to spawn new or remove old Docker Machine -instances. - -We will focus on the AWS machine options, for the rest of the settings read -about the: - -- [Autoscaling algorithm and the parameters it's based on](https://docs.gitlab.com/runner/configuration/autoscale.html#autoscaling-algorithm-and-parameters) - depends on the needs of your organization -- [Off peak time configuration](https://docs.gitlab.com/runner/configuration/autoscale.html#off-peak-time-mode-configuration) - useful when there are regular time periods in your organization when no work is done, for example weekends - -Here's an example of the `runners.machine` section: - -```toml - [runners.machine] - IdleCount = 1 - IdleTime = 1800 - MaxBuilds = 10 - OffPeakPeriods = [ - "* * 0-9,18-23 * * mon-fri *", - "* * * * * sat,sun *" - ] - OffPeakIdleCount = 0 - OffPeakIdleTime = 1200 - MachineDriver = "amazonec2" - MachineName = "gitlab-docker-machine-%s" - MachineOptions = [ - "amazonec2-access-key=XXXX", - "amazonec2-secret-key=XXXX", - "amazonec2-region=us-central-1", - "amazonec2-vpc-id=vpc-xxxxx", - "amazonec2-subnet-id=subnet-xxxxx", - "amazonec2-use-private-address=true", - "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", - "amazonec2-security-group=docker-machine-scaler", - "amazonec2-instance-type=m4.2xlarge", - ] -``` - -The Docker Machine driver is set to `amazonec2` and the machine name has a -standard prefix followed by `%s` (required) that is replaced by the ID of the -child Runner: `gitlab-docker-machine-%s`. - -Now, depending on your AWS infrastructure, there are many options you can set up -under `MachineOptions`. Below you can see the most common ones. - -| Machine option | Description | -| -------------- | ----------- | -| `amazonec2-access-key=XXXX` | The AWS access key of the user that has permissions to create EC2 instances, see [AWS credentials](#aws-credentials). | -| `amazonec2-secret-key=XXXX` | The AWS secret key of the user that has permissions to create EC2 instances, see [AWS credentials](#aws-credentials). | -| `amazonec2-region=eu-central-1` | The region to use when launching the instance. You can omit this entirely and the default `us-east-1` will be used. | -| `amazonec2-vpc-id=vpc-xxxxx` | Your [VPC ID](https://docs.docker.com/machine/drivers/aws/#vpc-id) to launch the instance in. | -| `amazonec2-subnet-id=subnet-xxxx` | The AWS VPC subnet ID. | -| `amazonec2-use-private-address=true` | Use the private IP address of Docker Machines, but still create a public IP address. Useful to keep the traffic internal and avoid extra costs.| -| `amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true` | AWS extra tag key-value pairs, useful to identify the instances on the AWS console. The "Name" tag is set to the machine name by default. We set the "runner-manager-name" to match the Runner name set in `[[runners]]`, so that we can filter all the EC2 instances created by a specific manager setup. Read more about [using tags in AWS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). | -| `amazonec2-security-group=docker-machine-scaler` | AWS VPC security group name, see [AWS security groups](#aws-security-groups). | -| `amazonec2-instance-type=m4.2xlarge` | The instance type that the child Runners will run on. | - -TIP: **Tip:** -Under `MachineOptions` you can add anything that the [AWS Docker Machine driver -supports](https://docs.docker.com/machine/drivers/aws/#options). You are highly -encouraged to read Docker's docs as your infrastructure setup may warrant -different options to be applied. - -NOTE: **Note:** -The child instances will use by default Ubuntu 16.04 unless you choose a -different AMI ID by setting `amazonec2-ami`. - -NOTE: **Note:** -If you specify `amazonec2-private-address-only=true` as one of the machine -options, your EC2 instance won't get assigned a public IP. This is ok if your -VPC is configured correctly with an Internet Gateway (IGW) and routing is fine, -but it’s something to consider if you've got a more complex configuration. Read -more in [Docker docs about VPC connectivity](https://docs.docker.com/machine/drivers/aws/#vpc-connectivity). - -[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-machine-section) -about all the options you can use under `[runners.machine]`. - -### Getting it all together - -Here's the full example of `/etc/gitlab-runner/config.toml`: - -```toml -concurrent = 10 -check_interval = 0 - -[[runners]] - name = "gitlab-aws-autoscaler" - url = "<URL of your GitLab instance>" - token = "<Runner's token>" - executor = "docker+machine" - limit = 20 - [runners.docker] - image = "alpine" - privileged = true - disable_cache = true - [runners.cache] - Type = "s3" - ServerAddress = "s3.amazonaws.com" - AccessKey = "<your AWS Access Key ID>" - SecretKey = "<your AWS Secret Access Key>" - BucketName = "<the bucket where your cache should be kept>" - BucketLocation = "us-east-1" - Shared = true - [runners.machine] - IdleCount = 1 - IdleTime = 1800 - MaxBuilds = 100 - OffPeakPeriods = [ - "* * 0-9,18-23 * * mon-fri *", - "* * * * * sat,sun *" - ] - OffPeakIdleCount = 0 - OffPeakIdleTime = 1200 - MachineDriver = "amazonec2" - MachineName = "gitlab-docker-machine-%s" - MachineOptions = [ - "amazonec2-access-key=XXXX", - "amazonec2-secret-key=XXXX", - "amazonec2-region=us-central-1", - "amazonec2-vpc-id=vpc-xxxxx", - "amazonec2-subnet-id=subnet-xxxxx", - "amazonec2-use-private-address=true", - "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", - "amazonec2-security-group=docker-machine-scaler", - "amazonec2-instance-type=m4.2xlarge", - ] -``` - -## Cutting down costs with Amazon EC2 Spot instances - -As [described by][spot] Amazon: - -> -Amazon EC2 Spot instances allow you to bid on spare Amazon EC2 computing capacity. -Since Spot instances are often available at a discount compared to On-Demand -pricing, you can significantly reduce the cost of running your applications, -grow your application’s compute capacity and throughput for the same budget, -and enable new types of cloud computing applications. - -In addition to the [`runners.machine`](#the-runners-machine-section) options -you picked above, in `/etc/gitlab-runner/config.toml` under the `MachineOptions` -section, add the following: - -```toml - MachineOptions = [ - "amazonec2-request-spot-instance=true", - "amazonec2-spot-price=0.03", - "amazonec2-block-duration-minutes=60" - ] -``` - -With this configuration, Docker Machines are created on Spot instances with a -maximum bid price of $0.03 per hour and the duration of the Spot instance is -capped at 60 minutes. The `0.03` number mentioned above is just an example, so -be sure to check on the current pricing based on the region you picked. - -To learn more about Amazon EC2 Spot instances, visit the following links: - -- https://aws.amazon.com/ec2/spot/ -- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html -- https://aws.amazon.com/blogs/aws/focusing-on-spot-instances-lets-talk-about-best-practices/ - -### Caveats of Spot instances - -While Spot instances is a great way to use unused resources and minimize the -costs of your infrastructure, you must be aware of the implications. - -Running CI jobs on Spot instances may increase the failure rates because of the -Spot instances pricing model. If the price exceeds your bid, the existing Spot -instances will be immediately terminated and all your jobs on that host will fail. - -As a consequence, the auto-scale Runner would fail to create new machines while -it will continue to request new instances. This eventually will make 60 requests -and then AWS won't accept any more. Then once the Spot price is acceptable, you -are locked out for a bit because the call amount limit is exceeded. - -If you encounter that case, you can use the following command in the bastion -machine to see the Docker Machines state: - -```sh -docker-machine ls -q --filter state=Error --format "{{.NAME}}" -``` - -NOTE: **Note:** -There are some issues regarding making GitLab Runner gracefully handle Spot -price changes, and there are reports of `docker-machine` attempting to -continually remove a Docker Machine. GitLab has provided patches for both cases -in the upstream project. For more information, see issues -[#2771](https://gitlab.com/gitlab-org/gitlab-runner/issues/2771) and -[#2772](https://gitlab.com/gitlab-org/gitlab-runner/issues/2772). - -## Conclusion - -In this guide we learned how to install and configure a GitLab Runner in -autoscale mode on AWS. - -Using the autoscale feature of GitLab Runner can save you both time and money. -Using the Spot instances that AWS provides can save you even more, but you must -be aware of the implications. As long as your bid is high enough, there shouldn't -be an issue. - -You can read the following use cases from which this tutorial was (heavily) -influenced: - -- [HumanGeo - Scaling GitLab CI](http://blog.thehumangeo.com/gitlab-autoscale-runners.html) -- [subtrakt Health - Autoscale GitLab CI Runners and save 90% on EC2 costs](https://substrakthealth.com/news/gitlab-ci-cost-savings/) - -[spot]: https://aws.amazon.com/ec2/spot/ +This document was moved to [another location](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html). |
