diff options
Diffstat (limited to 'doc')
526 files changed, 18648 insertions, 720 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md new file mode 100644 index 00000000000..d7a2e13b53e --- /dev/null +++ b/doc/administration/audit_events.md @@ -0,0 +1,116 @@ +--- +last_updated: 2019-02-04 +--- + +# Audit Events **[STARTER]** + +GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan][ee]. + +GitLab system administrators can also take advantage of the logs located on the +filesystem, see [the logs system documentation](logs.md) for more details. + +## Overview + +**Audit Events** is a tool for GitLab owners and administrators to be +able to track important events such as who performed certain actions and the +time they happened. These actions could be, for example, change a user +permission level, who added a new user, or who removed a user. + +## Use-cases + +- Check who was the person who changed the permission level of a particular + user for a project in GitLab. +- Use it to track which users have access to a certain group of projects + in GitLab, and who gave them that permission level. + +## List of events + +There are two kinds of events logged: + +- Events scoped to the group or project, used by group / project managers + to look up who made what change. +- Instance events scoped to the whole GitLab instance, used by your Compliance team to + perform formal audits. + +### Group events **[STARTER]** + +NOTE: **Note:** +You need Owner [permissions] to view the group Audit Events page. + +To view a group's audit events, navigate to **Group > Settings > Audit Events**. +From there, you can see the following actions: + +- Group name/path changed +- Group repository size limit changed +- Group created/deleted +- Group changed visibility +- User was added to group and with which [permissions] +- Permissions changes of a user assigned to a group +- Removed user from group +- Project added to group and with which visibility level +- Project removed from group +- [Project shared with group](../user/project/members/share_project_with_groups.md) + and with which [permissions] +- Removal of a previously shared group with a project +- LFS enabled/disabled +- Shared runners minutes limit changed +- Membership lock enabled/disabled +- Request access enabled/disabled +- 2FA enforcement/grace period changed +- Roles allowed to create project changed + +### Project events **[STARTER]** + +NOTE: **Note:** +You need Maintainer [permissions] or higher to view the project Audit Events page. + +To view a project's audit events, navigate to **Project > Settings > Audit Events**. +From there, you can see the following actions: + +- Added/removed deploy keys +- Project created/deleted/renamed/moved(transferred)/changed path +- Project changed visibility level +- User was added to project and with which [permissions] +- Permission changes of a user assigned to a project +- User was removed from project + +### Instance events **[PREMIUM ONLY]** + +> [Introduced][ee-2336] in [GitLab Premium][ee] 9.3. + +Server-wide audit logging introduces the ability to observe user actions across +the entire instance of your GitLab server, making it easy to understand who +changed what and when for audit purposes. + +To view the server-wide admin log, visit **Admin Area > Monitoring > Audit Log**. + +In addition to the group and project events, the following user actions are also +recorded: + +- Failed Logins +- Sign-in events and the authentication type (standard, LDAP, OmniAuth, etc.) +- Added SSH key +- Added/removed email +- Changed password +- Ask for password reset +- Grant OAuth access + +It is possible to filter particular actions by choosing an audit data type from +the filter drop-down. You can further filter by specific group, project or user +(for authentication events). + + + +### Missing events + +Some events are not being tracked in Audit Events. Please see the following +epics for more detail on which events are not being tracked and our progress +on adding these events into GitLab: + +- [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474) +- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475) +- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476) + +[ee-2336]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2336 +[ee]: https://about.gitlab.com/pricing/ +[permissions]: ../user/permissions.md diff --git a/doc/administration/audit_log.png b/doc/administration/audit_log.png Binary files differnew file mode 100644 index 00000000000..d4f4c2abf38 --- /dev/null +++ b/doc/administration/audit_log.png diff --git a/doc/administration/auditor_access_form.png b/doc/administration/auditor_access_form.png Binary files differnew file mode 100644 index 00000000000..c179a7d3b0a --- /dev/null +++ b/doc/administration/auditor_access_form.png diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md new file mode 100644 index 00000000000..ef8c8197d6d --- /dev/null +++ b/doc/administration/auditor_users.md @@ -0,0 +1,88 @@ +# Auditor users **[PREMIUM ONLY]** + +>[Introduced][ee-998] in [GitLab Premium][eep] 8.17. + +Auditor users are given read-only access to all projects, groups, and other +resources on the GitLab instance. + +## Overview + +Auditor users can have full access to their own resources (projects, groups, +snippets, etc.), and read-only access to **all** other resources, except the +Admin area. To put another way, they are just regular users (who can be added +to projects, create personal snippets, create milestones on their groups, etc.) +who also happen to have read-only access to all projects on the system that +they haven't been explicitly [given access][permissions] to. + +The Auditor role is _not_ a read-only version of the Admin role. Auditor users +will not be able to access the project/group settings pages, or the Admin Area. + +To sum up, assuming you have logged-in as an Auditor user: + +- For a project the Auditor is not member of, the Auditor should have + read-only access. If the project is public or internal, they would have the + same access as the users that are not members of that project/group. +- For a project the Auditor owns, the Auditor should have full access to + everything. +- For a project the Auditor has been added to as a member, the Auditor should + have the same access as the [permissions] they were given to. For example, if + they were added as a Developer, they could then push commits or comment on + issues. +- The Auditor cannot view the Admin area, or perform any admin actions. + +For more information about what an Auditor can or can't do, see the +[Permissions and restrictions of an Auditor user](#permissions-and-restrictions-of-an-auditor-user) +section. + +## Use cases + +1. Your compliance department wants to run tests against the entire GitLab base + to ensure users are complying with password, credit card, and other sensitive + data policies. With Auditor users, this can be achieved very easily without + resulting to tactics like giving a user admin rights or having to use the API + to add them to all projects. +1. If particular users need visibility or access to most of all projects in + your GitLab instance, instead of manually adding the user to all projects, + you can simply create an Auditor user and share the credentials with those + that you want to grant access to. + +## Adding an Auditor user + +1. Create a new user or edit an existing one by navigating to + **Admin Area > Users**. You will find the option of the access level under + the 'Access' section. + +  + +1. Click **Save changes** or **Create user** for the changes to take effect. + +To revoke the Auditor permissions from a user, simply make them a Regular user +following the same steps as above. + +## Permissions and restrictions of an Auditor user + +An Auditor user should be able to access all projects and groups of a GitLab +instance, with the following permissions/restrictions: + +- Has read-only access to the API +- Can access projects that are: + - Private + - Public + - Internal +- Can read all files in a repository +- Can read issues / MRs +- Can read project snippets +- Cannot be Admin and Auditor at the same time +- Cannot access the Admin area +- In a group / project they're not a member of: + - Cannot access project settings + - Cannot access group settings + - Cannot commit to repository + - Cannot create / comment on issues / MRs + - Cannot create/modify files from the Web UI + - Cannot merge a merge request + - Cannot create project snippets + +[ee-998]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/998 +[eep]: https://about.gitlab.com/pricing/ +[permissions]: ../user/permissions.md diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 54be7b616cc..e215a0df6ec 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -9,9 +9,11 @@ providers. - [LDAP](ldap.md) Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server + - [LDAP for GitLab EE](ldap-ee.md): LDAP additions to GitLab Enterprise Editions **[STARTER ONLY]** - [OmniAuth](../../integration/omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, Crowd, Azure, Authentiq ID, and JWT - [CAS](../../integration/cas.md) Configure GitLab to sign in using CAS - [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider - [Okta](okta.md) Configure GitLab to sign in using Okta - [Authentiq](authentiq.md): Enable the Authentiq OmniAuth provider for passwordless authentication +- [Smartcard](smartcard.md) Smartcard authentication **[PREMIUM ONLY]** diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 8d81b32e721..1f67e8f5744 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -14,7 +14,7 @@ Managing a large number of users in GitLab can become a burden for system admini In this guide we will focus on configuring GitLab with Active Directory. [Active Directory](https://en.wikipedia.org/wiki/Active_Directory) is a popular LDAP compatible directory service provided by Microsoft, included in all modern Windows Server operating systems. -GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html#group-sync) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. +GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](../how_to_configure_ldap_gitlab_ee/index.html#group-sync) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. ## Getting started @@ -111,7 +111,7 @@ The initial configuration of LDAP in GitLab requires changes to the `gitlab.rb` The two Active Directory specific values are `active_directory: true` and `uid: 'sAMAccountName'`. `sAMAccountName` is an attribute returned by Active Directory used for GitLab usernames. See the example output from `ldapsearch` for a full list of attributes a "person" object (user) has in **AD** - [`ldapsearch` example](#using-ldapsearch-unix) -> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html#gitlab-enterprise-edition---ldap-features) +> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](../how_to_configure_ldap_gitlab_ee/index.html#gitlab-enterprise-edition---ldap-features) **[STARTER ONLY]** ### Example `gitlab.rb` LDAP @@ -267,4 +267,4 @@ have extended functionalities with LDAP, such as: - Updating user permissions - Multiple LDAP servers -Read through the article on [LDAP for GitLab EE](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/) for an overview. +Read through the article on [LDAP for GitLab EE](../how_to_configure_ldap_gitlab_ee/index.md) **[STARTER ONLY]** for an overview. diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png Binary files differnew file mode 100644 index 00000000000..9896379d669 --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png Binary files differnew file mode 100644 index 00000000000..21fb5a7d0ce --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif Binary files differnew file mode 100644 index 00000000000..d35cf55804f --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif Binary files differnew file mode 100644 index 00000000000..29b28df1cbd --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif Binary files differnew file mode 100644 index 00000000000..d317add9837 --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md new file mode 100644 index 00000000000..4d82a7370bb --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -0,0 +1,119 @@ +--- +author: Chris Wilson +author_gitlab: MrChrisW +level: intermediary +article_type: admin guide +date: 2017-05-03 +--- + +# How to configure LDAP with GitLab EE **[STARTER ONLY]** + +## Introduction + +The present article follows [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. + +## GitLab Enterprise Edition - LDAP features + +[GitLab Enterprise Edition (EE)](https://about.gitlab.com/pricing/) has a number of advantages when it comes to integrating with Active Directory (LDAP): + +- [Administrator Sync](../ldap-ee.md#administrator-sync): As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the LDAP group will be given administrator privileges. +- [Group Sync](#group-sync): This allows GitLab group membership to be automatically updated based on LDAP group members. +- [Multiple LDAP servers](#multiple-ldap-servers): The ability to configure multiple LDAP servers. This is useful if an organization has different LDAP servers within departments. This is not designed for failover. We're working on [supporting LDAP failover](https://gitlab.com/gitlab-org/gitlab-ee/issues/139) in GitLab. + +- Daily user synchronization: Once a day, GitLab will run a synchronization to check and update GitLab users against LDAP. This process updates all user details automatically. + +On the following section, you'll find a description for each of these features. Read through [LDAP GitLab EE docs](../ldap-ee.md) for complementary information. + + + +All members of the group `Global Admins` will be given **administrator** access to GitLab, allowing them to view the `/admin` dashboard. + +### Group Sync + +Group syncing allows AD (LDAP) groups to be mapped to GitLab groups. This provides more control over per-group user management. To configure group syncing edit the `group_base` **DN** (`'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org'`). This **OU** contains all groups that will be associated with [GitLab groups](../../../user/group/index.md). + +#### Creating group links - example + +As an example, let's suppose we have a "UKGov" GitLab group, which deals with confidential government information. Therefore, it is important that users of this group are given the correct permissions to projects contained within the group. Granular group permissions can be applied based on the AD group. + +**UK Developers** of our "UKGov" group are given **"developer"** permissions. + +_The developer permission allows the development staff to effectively manage all project code, issues, and merge requests._ + +**UK Support** staff of our "UKGov" group are given **"reporter"** permissions. + +_The reporter permission allows support staff to manage issues, labels, and review project code._ + +**US People Ops** of our "UKGov" group are given **"guest"** permissions. + + + +> Guest permissions allows people ops staff to review and lodge new issues while allowing no read or write access to project code or [confidential issues](../../../user/project/issues/confidential_issues.md#permissions-and-access-to-confidential-issues) created by other users. + +See the [permission list](../../../user/permissions.md) for complementary info. + +#### Group permissions - example + +Considering the previous example, our staff will have +access to our GitLab instance with the following structure: + + + +Using this permission structure in our example allows only UK staff access to sensitive information stored in the projects code, while still allowing other teams to work effectively. As all permissions are controlled via AD groups new users can be quickly added to existing groups. New group members will then automatically inherit the required permissions. + +> [More information](../ldap-ee.md#group-sync) on group syncing. + +### Updating user permissions - new feature + +Since GitLab [v8.15](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/822) LDAP user permissions can now be manually overridden by an admin user. To override a user's permissions visit the groups **Members** page and select **Edit permissions**. + + + +### Multiple LDAP servers + +GitLab EE can support multiple LDAP servers. Simply configure another server in the `gitlab.rb` file within the `ldap_servers` block. In the example below we configure a new secondary server with the label **GitLab Secondary AD**. This is shown on the GitLab login screen. Large enterprises often utilize multiple LDAP servers for segregating organizational departments. + + + +Considering the example illustrated on the image above, +our `gitlab.rb` configuration would look like: + +```ruby +gitlab_rails['ldap_enabled'] = true +gitlab_rails['ldap_servers'] = { +'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.example.org', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'method' => 'ssl', + 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=org', + 'password' => 'Password1', + 'active_directory' => true, + 'base' => 'OU=GitLab INT,DC=GitLab,DC=org', + 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org', + 'admin_group' => 'Global Admins' + }, + +'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.example.net', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'method' => 'ssl', + 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=com', + 'password' => 'Password1', + 'active_directory' => true, + 'base' => 'OU=GitLab Secondary,DC=GitLab,DC=com', + 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=com', + 'admin_group' => 'Global Admins' + } +} +``` + +## Conclusion + +Integration of GitLab with Active Directory (LDAP) reduces the complexity of user management. +It has the advantage of improving user permission controls, whilst easing the deployment of GitLab into an existing [IT environment](https://www.techopedia.com/definition/29199/it-infrastructure). GitLab EE offers advanced group management and multiple LDAP servers. + +With the assistance of the [GitLab Support](https://about.gitlab.com/support) team, setting up GitLab with an existing AD/LDAP solution will be a smooth and painless process. diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md new file mode 100644 index 00000000000..30095d35705 --- /dev/null +++ b/doc/administration/auth/ldap-ee.md @@ -0,0 +1,557 @@ +# LDAP Additions in GitLab EE **[STARTER ONLY]** + +This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP +features specific to GitLab Enterprise Edition Starter, Premium and Ultimate. + +## Overview + +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which +is a standard application protocol for +accessing and maintaining distributed directory information services +over an Internet Protocol (IP) network. + +GitLab integrates with LDAP to support **user authentication**. This integration +works with most LDAP-compliant directory servers, including Microsoft Active +Directory, Apple Open Directory, Open LDAP, and 389 Server. +**GitLab Enterprise Edition** includes enhanced integration, including group +membership syncing. + +## Use cases + +- User sync: Once a day, GitLab will update users against LDAP +- Group sync: Once an hour, GitLab will update group membership + based on LDAP group members + +## Multiple LDAP servers + +With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers +that your GitLab instance will connect to. + +To add another LDAP server, you can start by duplicating the settings under +[the main configuration](ldap.md#configuration) and edit them to match the +additional LDAP server. + +Be sure to choose a different provider ID made of letters a-z and numbers 0-9. +This ID will be stored in the database so that GitLab can remember which LDAP +server a user belongs to. + +## User sync + +Once per day, GitLab will run a worker to check and update GitLab +users against LDAP. + +The process will execute the following access checks: + +1. Ensure the user is still present in LDAP +1. If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration [^1] + +The user will be set to `ldap_blocked` state in GitLab if the above conditions +fail. This means the user will not be able to login or push/pull code. + +The process will also update the following user information: + +1. Email address +1. If `sync_ssh_keys` is set, SSH public keys +1. If Kerberos is enabled, Kerberos identity + +NOTE: **Note:** +The LDAP sync process updates existing users while new users will +be created on first sign in. + +## Group Sync + +If your LDAP supports the `memberof` property, when the user signs in for the +first time GitLab will trigger a sync for groups the user should be a member of. +That way they don't need to wait for the hourly sync to be granted +access to their groups and projects. + +In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can +also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). + +A group sync process will run every hour on the hour, and `group_base` must be set +in LDAP configuration for LDAP synchronizations based on group CN to work. This allows +GitLab group membership to be automatically updated based on LDAP group members. + +The `group_base` configuration should be a base LDAP 'container', such as an +'organization' or 'organizational unit', that contains LDAP groups that should +be available to GitLab. For example, `group_base` could be +`ou=groups,dc=example,dc=com`. In the config file it will look like the +following. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +--- + +To take advantage of group sync, group owners or maintainers will need to create an +LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP +groups and/or filters can be linked with a single GitLab group. When the link is +created, an access level/role is specified (Guest, Reporter, Developer, Maintainer, +or Owner). + +## Administrator sync + +As an extension of group sync, you can automatically manage your global GitLab +administrators. Specify a group CN for `admin_group` and all members of the +LDAP group will be given administrator privileges. The configuration will look +like the following. + +NOTE: **Note:** +Administrators will not be synced unless `group_base` is also +specified alongside `admin_group`. Also, only specify the CN of the admin +group, as opposed to the full DN. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + + ## + ## The CN of a group containing GitLab administrators + ## + ## Ex. administrators + ## + ## Note: Not `cn=administrators` or the full DN + ## + ## + admin_group: my_admin_group + + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + admin_group: my_admin_group + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Adjusting LDAP user sync schedule + +> Introduced in GitLab Enterprise Edition Starter. + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example http://www.crontabgenerator.com/. + +By default, GitLab will run a worker once per day at 01:30 a.m. server time to +check and update GitLab users against LDAP. + +You can manually configure LDAP user sync times by setting the +following configuration values. The example below shows how to set LDAP user +sync to run once every 12 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_sync_worker_cron: + "0 */12 * * *" + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Adjusting LDAP group sync schedule + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example http://www.crontabgenerator.com/. + +By default, GitLab will run a group sync process every hour, on the hour. + +CAUTION: **Important:** +It's recommended that you do not run too short intervals as this +could lead to multiple syncs running concurrently. This is primarily a concern +for installations with a large number of LDAP users. Please review the +[LDAP group sync benchmark metrics](#benchmarks) to see how +your installation compares before proceeding. + +You can manually configure LDAP group sync times by setting the +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_group_sync_worker_cron: + "*/30 * * * *" + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## External groups + +> Introduced in GitLab Enterprise Edition Starter 8.9. + +Using the `external_groups` setting will allow you to mark all users belonging +to these groups as [external users](../../user/permissions.md#external-users-permissions). +Group membership is checked periodically through the `LdapGroupSync` background +task. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## An array of CNs of groups containing users that should be considered external + ## + ## Ex. ['interns', 'contractors'] + ## + ## Note: Not `cn=interns` or the full DN + ## + external_groups: ['interns', 'contractors'] + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + external_groups: ['interns', 'contractors'] + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Group sync technical details + +There is a lot going on with group sync 'under the hood'. This section +outlines what LDAP queries are executed and what behavior you can expect +from group sync. + +Group member access will be downgraded from a higher level if their LDAP group +membership changes. For example, if a user has 'Owner' rights in a group and the +next group sync reveals they should only have 'Developer' privileges, their +access will be adjusted accordingly. The only exception is if the user is the +*last* owner in a group. Groups need at least one owner to fulfill +administrative duties. + +### Supported LDAP group types/attributes + +GitLab supports LDAP groups that use member attributes `member`, `submember`, +`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at +least, LDAP groups with object class `groupOfNames`, `posixGroup`, and +`groupOfUniqueName`. Other object classes should work fine as long as members +are defined as one of the mentioned attributes. This also means GitLab supports +Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. +Other LDAP servers should work, too. + +Active Directory also supports nested groups. Group sync will recursively +resolve membership if `active_directory: true` is set in the configuration file. + +> **Note:** Nested group membership will only be resolved if the nested group + also falls within the configured `group_base`. For example, if GitLab sees a + nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but + the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` + will be ignored. + +### Queries + +- Each LDAP group is queried a maximum of one time with base `group_base` and + filter `(cn=<cn_from_group_link>)`. +- If the LDAP group has the `memberuid` attribute, GitLab will execute another + LDAP query per member to obtain each user's full DN. These queries are + executed with base `base`, scope 'base object', and a filter depending on + whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a + joining of `user_filter`. + +### Benchmarks + +Group sync was written to be as performant as possible. Data is cached, database +queries are optimized, and LDAP queries are minimized. The last benchmark run +revealed the following metrics: + +For 20,000 LDAP users, 11,000 LDAP groups and 1,000 GitLab groups with 10 +LDAP group links each: + +- Initial sync (no existing members assigned in GitLab) took 1.8 hours +- Subsequent syncs (checking membership, no writes) took 15 minutes + +These metrics are meant to provide a baseline and performance may vary based on +any number of factors. This was a pretty extreme benchmark and most instances will +not have near this many users or groups. Disk speed, database performance, +network and LDAP server response time will affect these metrics. + +## Troubleshooting + +### Referral error + +If you see `LDAP search error: Referral` in the logs, or when troubleshooting +LDAP Group Sync, this error may indicate a configuration problem. The LDAP +configuration `/etc/gitlab/gitlab.rb` (Omnibus) or `config/gitlab.yml` (source) +is in YAML format and is sensitive to indentation. Check that `group_base` and +`admin_group` configuration keys are indented 2 spaces past the server +identifier. The default identifier is `main` and an example snippet looks like +the following: + +```yaml +main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'LDAP' + host: 'ldap.example.com' + ... + group_base: 'cn=my_group,ou=groups,dc=example,dc=com' + admin_group: 'my_admin_group' +``` + +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source + +[^1]: In Active Directory, a user is marked as disabled/blocked if the user + account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) + has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/ + for more information. + +### User DN has changed + +When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. + +If GitLab cannot find a user by their DN, it will attempt to fallback +to finding the user by their email. If the lookup is successful, GitLab will +update the stored DN to the new value. + +### User is not being added to a group + +Sometimes you may think a particular user should be added to a GitLab group via +LDAP group sync, but for some reason it's not happening. There are several +things to check to debug the situation. + +- Ensure LDAP configuration has a `group_base` specified. This configuration is + required for group sync to work properly. +- Ensure the correct LDAP group link is added to the GitLab group. Check group + links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**. +- Check that the user has an LDAP identity + 1. Sign in to GitLab as an administrator user. + 1. Navigate to **Admin area -> Users**. + 1. Search for the user + 1. Open the user, by clicking on their name. Do not click 'Edit'. + 1. Navigate to the **Identities** tab. There should be an LDAP identity with + an LDAP DN as the 'Identifier'. + +If all of the above looks good, jump in to a little more advanced debugging. +Often, the best way to learn more about why group sync is behaving a certain +way is to enable debug logging. There is verbose output that details every +step of the sync. + +1. Start a Rails console + + ```bash + # For Omnibus installations + sudo gitlab-rails console + + # For installations from source + sudo -u git -H bundle exec rails console production + ``` +1. Set the log level to debug (only for this session): + + ```ruby + Rails.logger.level = Logger::DEBUG + ``` +1. Choose a GitLab group to test with. This group should have an LDAP group link + already configured. If the output is `nil`, the group could not be found. + If a bunch of group attributes are output, your group was found successfully. + + ```ruby + group = Group.find_by(name: 'my_group') + + # Output + => #<Group:0x007fe825196558 id: 1234, name: "my_group"...> + ``` +1. Run a group sync for this particular group. + + ```ruby + EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group) + ``` +1. Look through the output of the sync. See [example log output](#example-log-output) + below for more information about the output. +1. If you still aren't able to see why the user isn't being added, query the + LDAP group directly to see what members are listed. Still in the Rails console, + run the following query: + + ```ruby + adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider + ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter) + + # Output + => #<EE::Gitlab::Auth::LDAP::Group:0x007fcbdd0bb6d8 + ``` +1. Query the LDAP group's member DNs and see if the user's DN is in the list. + One of the DNs here should match the 'Identifier' from the LDAP identity + checked earlier. If it doesn't, the user does not appear to be in the LDAP + group. + + ```ruby + ldap_group.member_dns + + # Output + => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"] + ``` +1. Some LDAP servers don't store members by DN. Rather, they use UIDs instead. + If you didn't see results from the last query, try querying by UIDs instead. + + ```ruby + ldap_group.member_uids + + # Output + => ['john','mary'] + ``` + +#### Example log output + +The output of the last command will be very verbose, but contains lots of +helpful information. For the most part you can ignore log entries that are SQL +statements. + +Indicates the point where syncing actually begins: + +```bash +Started syncing all providers for 'my_group' group +``` + +The follow entry shows an array of all user DNs GitLab sees in the LDAP server. +Note that these are the users for a single LDAP group, not a GitLab group. If +you have multiple LDAP groups linked to this GitLab group, you will see multiple +log entries like this - one for each LDAP group. If you don't see an LDAP user +DN in this log entry, LDAP is not returning the user when we do the lookup. +Verify the user is actually in the LDAP group. + +```bash +Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com", +"uid=mary0,ou=people,dc=example,dc=com", "uid=john1,ou=people,dc=example,dc=com", +"uid=mary1,ou=people,dc=example,dc=com", "uid=john2,ou=people,dc=example,dc=com", +"uid=mary2,ou=people,dc=example,dc=com", "uid=john3,ou=people,dc=example,dc=com", +"uid=mary3,ou=people,dc=example,dc=com", "uid=john4,ou=people,dc=example,dc=com", +"uid=mary4,ou=people,dc=example,dc=com"] +``` + +Shortly after each of the above entries, you will see a hash of resolved member +access levels. This hash represents all user DNs GitLab thinks should have +access to this group, and at which access level (role). This hash is additive, +and more DNs may be added, or existing entries modified, based on additional +LDAP group lookups. The very last occurrence of this entry should indicate +exactly which users GitLab believes should be added to the group. + +> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' + and 50 is 'Owner' + +```bash +Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, +"uid=mary0,ou=people,dc=example,dc=com"=>30, "uid=john1,ou=people,dc=example,dc=com"=>30, +"uid=mary1,ou=people,dc=example,dc=com"=>30, "uid=john2,ou=people,dc=example,dc=com"=>30, +"uid=mary2,ou=people,dc=example,dc=com"=>30, "uid=john3,ou=people,dc=example,dc=com"=>30, +"uid=mary3,ou=people,dc=example,dc=com"=>30, "uid=john4,ou=people,dc=example,dc=com"=>30, +"uid=mary4,ou=people,dc=example,dc=com"=>30} +``` + +It's not uncommon to see warnings like the following. These indicate that GitLab +would have added the user to a group, but the user could not be found in GitLab. +Usually this is not a cause for concern. + +If you think a particular user should already exist in GitLab, but you're seeing +this entry, it could be due to a mismatched DN stored in GitLab. See +[User DN has changed](#User-DN-has-changed) to update the user's LDAP identity. + +```bash +User with DN `uid=john0,ou=people,dc=example,dc=com` should have access +to 'my_group' group but there is no user in GitLab with that +identity. Membership will be updated once the user signs in for +the first time. +``` + +Finally, the following entry says syncing has finished for this group: + +```bash +Finished syncing all providers for 'my_group' group +``` diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 2d057dc7509..423a79c56d8 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -14,7 +14,7 @@ including group membership syncing as well as multiple LDAP servers support. The information on this page is relevant for both GitLab CE and EE. For more details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html). +[LDAP Enterprise Edition documentation](ldap-ee.md). **[STARTER ONLY]** ## Security @@ -39,7 +39,7 @@ immediately block all access. NOTE: **Note**: GitLab Enterprise Edition Starter supports a -[configurable sync time](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#adjusting-ldap-user-and-group-sync-schedules), +[configurable sync time](ldap-ee.md#adjusting-ldap-user-sync-schedule), with a default of one hour. ## Git password authentication @@ -64,6 +64,7 @@ to connect to one GitLab server. For a complete guide on configuring LDAP with GitLab Community Edition, please check the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). +For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **[STARTER ONLY]** To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -388,7 +389,7 @@ group, you can use the following syntax: Find more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter at <https://docs.microsoft.com/en-us/windows/desktop/ADSI/search-filter-syntax>. Support for nested members in the user filter should not be confused with -[group sync nested groups support (EE only)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#supported-ldap-group-types-attributes). +[group sync nested groups support](ldap-ee.md#supported-ldap-group-typesattributes). **[STARTER ONLY]** Please note that GitLab does not support the custom filter syntax used by omniauth-ldap. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md new file mode 100644 index 00000000000..e55f7dbb4df --- /dev/null +++ b/doc/administration/auth/oidc.md @@ -0,0 +1,105 @@ +# OpenID Connect OmniAuth provider + +GitLab can use [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) as an OmniAuth provider. + +To enable the OpenID Connect OmniAuth provider, you must register your application with an OpenID Connect provider. +The OpenID Connect will provide you with a client details and secret for you to use. + +1. On your GitLab server, open the configuration file. + + For Omnibus GitLab: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + + See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration. + + For Omnibus GitLab: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { 'name' => 'openid_connect', + 'label' => '<your_oidc_label>', + 'args' => { + 'scope' => ['openid','profile'], + 'response_type' => 'code', + 'issuer' => '<your_oidc_url>', + 'discovery' => true, + 'client_auth_method' => 'query', + 'uid_field' => '<uid_field>', + 'client_options' => { + 'identifier' => '<your_oidc_client_id>', + 'secret' => '<your_oidc_client_secret>', + 'redirect_uri' => '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ] + ``` + + For installation from source: + + ```yaml + - { name: 'openid_connect', + label: '<your_oidc_label>', + args: { + scope: ['openid','profile'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ``` + + > **Note:** + > + > - For more information on each configuration option refer to + the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) and + the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). + +1. For the configuration above, change the values for the provider to match your OpenID Connect client setup. Use the following as a guide: + - `<your_oidc_label>` is the label that will be displayed on the login page. + - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. + If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. + - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. + - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. + If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. + - `client_options` are the OpenID Connect client-specific options. Specifically: + + - `identifier` is the client identifier as configured in the OpenID Connect service provider. + - `secret` is the client secret as configured in the OpenID Connect service provider. + - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. + - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. + + The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: + + - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. + - `token_endpoint` is the URL to the endpoint that provides Access Token. + - `userinfo_endpoint` is the URL to the endpoint that provides the user information. + - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. + +1. Save the configuration file. +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect if you installed GitLab via Omnibus or from source respectively. + +On the sign in page, there should now be an OpenID Connect icon below the regular sign in form. +Click the icon to begin the authentication process. The OpenID Connect provider will ask the user to +sign in and authorize the GitLab application (if confirmation required by the client). If everything goes well, the user +will be redirected to GitLab and will be signed in. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md new file mode 100644 index 00000000000..b33c5359b44 --- /dev/null +++ b/doc/administration/auth/smartcard.md @@ -0,0 +1,186 @@ +# Smartcard authentication **[PREMIUM ONLY]** + +GitLab supports authentication using smartcards. + +## Authentication methods + +GitLab supports two authentication methods: + +- X.509 certificates with local databases. +- LDAP servers. + +### Authentication against a local database with X.509 certificates + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/726) in +[GitLab Premium](https://about.gitlab.com/pricing/) 11.6 as an experimental +feature. Smartcard authentication against local databases may change or be +removed completely in future releases. + +Smartcards with X.509 certificates can be used to authenticate with GitLab. + +To use a smartcard with an X.509 certificate to authenticate against a local +database with GitLab, `CN` and `emailAddress` must be defined in the +certificate. For example: + +``` +Certificate: + Data: + Version: 1 (0x0) + Serial Number: 12856475246677808609 (0xb26b601ecdd555e1) + Signature Algorithm: sha256WithRSAEncryption + Issuer: O=Random Corp Ltd, CN=Random Corp + Validity + Not Before: Oct 30 12:00:00 2018 GMT + Not After : Oct 30 12:00:00 2019 GMT + Subject: CN=Gitlab User, emailAddress=gitlab-user@example.com +``` + +### Authentication against an LDAP server + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7693) in +[GitLab Premium](https://about.gitlab.com/pricing/) 11.8 as an experimental +feature. Smartcard authentication against an LDAP server may change or be +removed completely in future releases. + +GitLab implements a standard way of certificate matching following +[RFC4523](https://tools.ietf.org/html/rfc4523). It uses the +`certificateExactMatch` certificate matching rule against the `userCertificate` +attribute. As a prerequisite, you must use an LDAP server that: + +- Supports the `certificateExactMatch` matching rule. +- Has the certificate stored in the `userCertificate` attribute. + +## Configure GitLab for smartcard authentication + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['smartcard_enabled'] = true + gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" + gitlab_rails['smartcard_client_certificate_required_port'] = 3444 + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Configure NGINX to request a client side certificate + + In NGINX configuration, an **additional** server context must be defined with + the same configuration except: + + - The additional NGINX server context must be configured to run on a different + port: + + ``` + listen *:3444 ssl; + ``` + + - The additional NGINX server context must be configured to require the client + side certificate: + + ``` + ssl_verify_depth 2; + ssl_client_certificate /etc/ssl/certs/CA.pem; + ssl_verify_client on; + ``` + + - The additional NGINX server context must be configured to forward the client + side certificate: + + ``` + proxy_set_header X-SSL-Client-Certificate $ssl_client_escaped_cert; + ``` + + For example, the following is an example server context in an NGINX + configuration file (eg. in `/etc/nginx/sites-available/gitlab-ssl`): + + ``` + server { + listen *:3444 ssl; + + # certificate for configuring SSL + ssl_certificate /path/to/example.com.crt; + ssl_certificate_key /path/to/example.com.key; + + ssl_verify_depth 2; + # CA certificate for client side certificate verification + ssl_client_certificate /etc/ssl/certs/CA.pem; + ssl_verify_client on; + + location / { + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + + proxy_set_header X-SSL-Client-Certificate $ssl_client_escaped_cert; + + proxy_read_timeout 300; + + proxy_pass http://gitlab-workhorse; + } + } + ``` + +1. Edit `config/gitlab.yml`: + + ```yaml + ## Smartcard authentication settings + smartcard: + # Allow smartcard authentication + enabled: true + + # Path to a file containing a CA certificate + ca_file: '/etc/ssl/certs/CA.pem' + + # Port where the client side certificate is requested by NGINX + client_certificate_required_port: 3444 + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. + +### Additional steps when authenticating against an LDAP server + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + EOS + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 72cb57fb36c..9c13ff772b3 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -11,8 +11,8 @@ GitLab’s [security features](../security/README.md) may also help you meet rel |**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+|| |**[Email all users of a project, group, or entire server](../user/admin_area/settings/terms.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+|| -|**[Lock project membership to group](https://docs.gitlab.com/ee/user/group/index.html#member-lock-starter)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| +|**[Lock project membership to group](../user/group/index.md#member-lock-starter)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| |**[LDAP group sync](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| |**[LDAP group sync filters](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| -|**[Audit logs](https://docs.gitlab.com/ee/administration/audit_events.html)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze and track every change.|Premium+|| -|**[Auditor users](https://docs.gitlab.com/ee/administration/auditor_users.html)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+||
\ No newline at end of file +|**[Audit logs](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze and track every change.|Premium+|| +|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+|| diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index 48f599fa7e6..4d55f2357c1 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -384,14 +384,12 @@ CAUTION: **Warning:** GitLab will not backup Docker images that are not stored o filesystem. Remember to enable backups with your object storage provider if desired. ---- +NOTE: **Note:** +`regionendpoint` is only required when configuring an S3 compatible service such as Minio. It takes a URL such as `http://127.0.0.1:9000`. **Omnibus GitLab installations** -> **Note:** -`regionendpoint` is only required when configuring an S3 compatible service such as Minio, by entering a URL such as http://127.0.0.1:9000 - -To configure the storage driver in Omnibus: +To configure the `s3` storage driver in Omnibus: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -409,16 +407,14 @@ To configure the storage driver in Omnibus: 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** Configuring the storage driver is done in your registry config YML file created when you [deployed your docker registry][registry-deploy]. -Example: +`s3` storage driver example: -``` +```yml storage: s3: accesskey: 'AKIAKIAKI' @@ -663,6 +659,37 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). 1. Save the file and [restart GitLab][] for the changes to take effect. +### Supporting older Docker clients + +As of GitLab 11.9, we began shipping version 2.7.1 of the Docker container registry, which disables the schema1 manifest by default. If you are still using older Docker clients (1.9 or older), you may experience an error pushing images. See [omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4145) for more details. + +You can add a configuration option for backwards compatibility. + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['compatibility_schema1_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**For installations from source** + +1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: + + ```yaml + compatibility: + schema1: + enabled: true + ``` + +1. Restart the registry for the changes to take affect. + + [ce-18239]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18239 [docker-insecure-self-signed]: https://docs.docker.com/registry/insecure/#use-self-signed-certificates [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index f318cb38ef0..288cb1bf0bb 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -8,6 +8,9 @@ have filesystem access. For a user configurable Git hook interface, see [Push Rules](https://docs.gitlab.com/ee/push_rules/push_rules.html), available in GitLab Enterprise Edition. +NOTE: **Note:** +Custom Git hooks won't be replicated to secondary nodes if you use [GitLab Geo][gitlab-geo] + Git natively supports hooks that are executed on different actions. Examples of server-side git hooks include pre-receive, post-receive, and update. See [Git SCM Server-Side Hooks][hooks] for more information about each hook type. @@ -48,7 +51,7 @@ To create a Git hook that applies to all of your repositories in your instance, set a global Git hook. Since all the repositories' `hooks` directories are symlinked to gitlab-shell's `hooks` directory, adding any hook to the gitlab-shell `hooks` directory will also apply it to all repositories. Follow -the steps below to properly set up a custom hook all for repositories: +the steps below to properly set up a custom hook for all repositories: 1. On the GitLab server, navigate to the configured custom hook directory. The default is in the gitlab-shell directory. The gitlab-shell `hook` directory @@ -120,5 +123,6 @@ exit 1 [CI]: ../ci/README.md [hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks [webhooks]: ../user/project/integrations/webhooks.md +[gitlab-geo]: https://docs.gitlab.com/ee/administration/geo/replication/index.html [5073]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5073 [93]: https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/93 diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md new file mode 100644 index 00000000000..7f3be402b84 --- /dev/null +++ b/doc/administration/database_load_balancing.md @@ -0,0 +1,277 @@ +# Database Load Balancing **[PREMIUM ONLY]** + +> [Introduced][ee-1283] in [GitLab Premium][eep] 9.0. + +Distribute read-only queries among multiple database servers. + +## Overview + +Database load balancing improves the distribution of database workloads across +multiple computing resources. Load balancing aims to optimize resource use, +maximize throughput, minimize response time, and avoid overload of any single +resource. Using multiple components with load balancing instead of a single +component may increase reliability and availability through redundancy. +[_Wikipedia article_][wikipedia] + +When database load balancing is enabled in GitLab, the load is balanced using +a simple round-robin algorithm, without any external dependencies such as Redis. +Load balancing is not enabled for Sidekiq as this would lead to consistency +problems, and Sidekiq mostly performs writes anyway. + +In the following image, you can see the load is balanced rather evenly among +all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not +sent to the primary (unless necessary), the primary (`db3`) hardly has any load. + + + +## Requirements + +For load balancing to work you will need at least PostgreSQL 9.2 or newer, +[**MySQL is not supported**][db-req]. You also need to make sure that you have +at least 1 secondary in [hot standby][hot-standby] mode. + +Load balancing also requires that the configured hosts **always** point to the +primary, even after a database failover. Furthermore, the additional hosts to +balance load among must **always** point to secondary databases. This means that +you should put a load balance in front of every database, and have GitLab connect +to those load balancers. + +For example, say you have a primary (`db1.gitlab.com`) and two secondaries, +`db2.gitlab.com` and `db3.gitlab.com`. For this setup you will need to have 3 +load balancers, one for every host. For example: + +* `primary.gitlab.com` forwards to `db1.gitlab.com` +* `secondary1.gitlab.com` forwards to `db2.gitlab.com` +* `secondary2.gitlab.com` forwards to `db3.gitlab.com` + +Now let's say that a failover happens and db2 becomes the new primary. This +means forwarding should now happen as follows: + +* `primary.gitlab.com` forwards to `db2.gitlab.com` +* `secondary1.gitlab.com` forwards to `db1.gitlab.com` +* `secondary2.gitlab.com` forwards to `db3.gitlab.com` + +GitLab does not take care of this for you, so you will need to do so yourself. + +Finally, load balancing requires that GitLab can connect to all hosts using the +same credentials and port as configured in the +[Enabling load balancing](#enabling-load-balancing) section. Using +different ports or credentials for different hosts is not supported. + +## Use cases + +- For GitLab instances with thousands of users and high traffic, you can use + database load balancing to reduce the load on the primary database and + increase responsiveness, thus resulting in faster page load inside GitLab. + +## Enabling load balancing + +For the environment in which you want to use load balancing, you'll need to add +the following. This will balance the load between `host1.example.com` and +`host2.example.com`. + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/database.yml` and add or amend the following lines: + + ```yaml + production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + hosts: + - host1.example.com + - host2.example.com + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. + +## Service Discovery + +> [Introduced][ee-5883] in [GitLab Premium][eep] 11.0. + +Service discovery allows GitLab to automatically retrieve a list of secondary +databases to use, instead of having to manually specify these in the +`database.yml` configuration file. Service discovery works by periodically +checking a DNS A record, using the IPs returned by this record as the addresses +for the secondaries. For service discovery to work, all you need is a DNS server +and an A record containing the IP addresses of your secondaries. + +To use service discovery you need to change your `database.yml` configuration +file so it looks like the following: + +```yaml +production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + discover: + nameserver: localhost + record: secondary.postgresql.service.consul + port: 8600 + interval: 60 + disconnect_timeout: 120 +``` + +Here the `discover:` section specifies the configuration details to use for +service discovery. + +### Configuration + +The following options can be set: + +| Option | Description | Default | +|----------------------|---------------------------------------------------------------------------------------------------|-----------| +| `nameserver` | The nameserver to use for looking up the DNS record. | localhost | +| `record` | The A record to look up. This option is required for service discovery to work. | | +| `port` | The port of the nameserver. | 8600 | +| `interval` | The minimum time in seconds between checking the DNS record. | 60 | +| `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | +| `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | + +The `interval` value specifies the _minimum_ time between checks. If the A +record has a TTL greater than this value, then service discovery will honor said +TTL. For example, if the TTL of the A record is 90 seconds, then service +discovery will wait at least 90 seconds before checking the A record again. + +When the list of hosts is updated, it might take a while for the old connections +to be terminated. The `disconnect_timeout` setting can be used to enforce an +upper limit on the time it will take to terminate all old database connections. + +Some nameservers (like [Consul][consul-udp]) can return a truncated list of hosts when +queried over UDP. To overcome this issue, you can use TCP for querying by setting +`use_tcp` to `true`. + +### Forking + +If you use an application server that forks, such as Unicorn, you _have to_ +update your Unicorn configuration to start service discovery _after_ a fork. +Failure to do so will lead to service discovery only running in the parent +process. If you are using Unicorn, then you can add the following to your +Unicorn configuration file: + +```ruby +after_fork do |server, worker| + defined?(Gitlab::Database::LoadBalancing) && + Gitlab::Database::LoadBalancing.start_service_discovery +end +``` + +This will ensure that service discovery is started in both the parent and all +child processes. + +## Balancing queries + +Read-only `SELECT` queries will be balanced among all the secondary hosts. +Everything else (including transactions) will be executed on the primary. +Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. + +## Prepared statements + +Prepared statements don't work well with load balancing and are disabled +automatically when load balancing is enabled. This should have no impact on +response timings. + +## Primary sticking + +After a write has been performed, GitLab will stick to using the primary for a +certain period of time, scoped to the user that performed the write. GitLab will +revert back to using secondaries when they have either caught up, or after 30 +seconds. + +## Failover handling + +In the event of a failover or an unresponsive database, the load balancer will +try to use the next available host. If no secondaries are available the +operation is performed on the primary instead. + +In the event of a connection error being produced when writing data, the +operation will be retried up to 3 times using an exponential back-off. + +When using load balancing, you should be able to safely restart a database server +without it immediately leading to errors being presented to the users. + +## Logging + +The load balancer logs various messages, such as: + +* When a host is marked as offline +* When a host comes back online +* When all secondaries are offline + +Each log message contains the tag `[DB-LB]` to make searching/filtering of such +log entries easier. For example: + +``` +[DB-LB] Host 10.123.2.5 came back online +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Host 10.123.2.6 came back online +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Host 10.123.2.7 came back online +[DB-LB] Host 10.123.2.7 came back online +``` + +## Handling Stale Reads + +> [Introduced][ee-3526] in [GitLab Premium][eep] 10.3. + +To prevent reading from an outdated secondary the load balancer will check if it +is in sync with the primary. If the data is determined to be recent enough the +secondary can be used, otherwise it will be ignored. To reduce the overhead of +these checks we only perform these checks at certain intervals. + +There are three configuration options that influence this behaviour: + +| Option | Description | Default | +|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| +| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | +| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | +| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | + +The defaults should be sufficient for most users. Should you want to change them +you can specify them in `config/database.yml` like so: + +```yaml +production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + hosts: + - host1.example.com + - host2.example.com + max_replication_difference: 16777216 # 16 MB + max_replication_lag_time: 30 + replica_check_interval: 30 +``` + +[hot-standby]: https://www.postgresql.org/docs/9.6/static/hot-standby.html +[ee-1283]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1283 +[eep]: https://about.gitlab.com/pricing/ +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" +[wikipedia]: https://en.wikipedia.org/wiki/Load_balancing_(computing) +[db-req]: ../install/requirements.md#database +[ee-3526]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3526 +[ee-5883]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5883 +[consul-udp]: https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md new file mode 100644 index 00000000000..7d2fd51f834 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -0,0 +1,172 @@ +# Automatic background verification **[PREMIUM ONLY]** + +NOTE: **Note:** +Automatic background verification of repositories and wikis was added in +GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can +disable or enable this feature manually by following +[these instructions](#disabling-or-enabling-the-automatic-background-verification). + +Automatic background verification ensures that the transferred data matches a +calculated checksum. If the checksum of the data on the **primary** node matches checksum of the +data on the **secondary** node, the data transferred successfully. Following a planned failover, +any corrupted data may be **lost**, depending on the extent of the corruption. + +If verification fails on the **primary** node, this indicates that Geo is +successfully replicating a corrupted object; restore it from backup or remove it +it from the **primary** node to resolve the issue. + +If verification succeeds on the **primary** node but fails on the **secondary** node, +this indicates that the object was corrupted during the replication process. +Geo actively try to correct verification failures marking the repository to +be resynced with a backoff period. If you want to reset the verification for +these failures, so you should follow [these instructions][reset-verification]. + +If verification is lagging significantly behind replication, consider giving +the node more time before scheduling a planned failover. + +## Disabling or enabling the automatic background verification + +Run the following commands in a Rails console on the **primary** node: + +```sh +# Omnibus GitLab +gitlab-rails console + +# Installation from source +cd /home/git/gitlab +sudo -u git -H bin/rails console RAILS_ENV=production +``` + +To check if automatic background verification is enabled: + +```ruby +Gitlab::Geo.repository_verification_enabled? +``` + +To disable automatic background verification: + +```ruby +Feature.disable('geo_repository_verification') +``` + +To enable automatic background verification: + +```ruby +Feature.enable('geo_repository_verification') +``` + +## Repository verification + +Navigate to the **Admin Area > Geo** dashboard on the **primary** node and expand +the **Verification information** tab for that node to view automatic checksumming +status for repositories and wikis. Successes are shown in green, pending work +in grey, and failures in red. + + + +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node and expand +the **Verification information** tab for that node to view automatic verification +status for repositories and wikis. As with checksumming, successes are shown in +green, pending work in grey, and failures in red. + + + +## Using checksums to compare Geo nodes + +To check the health of Geo **secondary** nodes, we use a checksum over the list of +Git references and their values. The checksum includes `HEAD`, `heads`, `tags`, +`notes`, and GitLab-specific references to ensure true consistency. If two nodes +have the same checksum, then they definitely hold the same references. We compute +the checksum for every node after every update to make sure that they are all +in sync. + +## Repository re-verification + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8550) in GitLab Enterprise Edition 11.6. Available in [GitLab Premium](https://about.gitlab.com/pricing/). + +Due to bugs or transient infrastructure failures, it is possible for Git +repositories to change unexpectedly without being marked for verification. +Geo constantly reverifies the repositories to ensure the integrity of the +data. The default and recommended re-verification interval is 7 days, though +an interval as short as 1 day can be set. Shorter intervals reduce risk but +increase load and vice versa. + +Navigate to the **Admin Area > Geo** dashboard on the **primary** node, and +click the **Edit** button for the **primary** node to customize the minimum +re-verification interval: + + + +The automatic background re-verification is enabled by default, but you can +disable if you need. Run the following commands in a Rails console on the +**primary** node: + +```sh +# Omnibus GitLab +gitlab-rails console + +# Installation from source +cd /home/git/gitlab +sudo -u git -H bin/rails console RAILS_ENV=production +``` + +To disable automatic background re-verification: + +```ruby +Feature.disable('geo_repository_reverification') +``` + +To enable automatic background re-verification: + +```ruby +Feature.enable('geo_repository_reverification') +``` + +## Reset verification for projects where verification has failed + +Geo actively try to correct verification failures marking the repository to +be resynced with a backoff period. If you want to reset them manually, this +rake task marks projects where verification has failed or the checksum mismatch +to be resynced without the backoff period: + +For repositories: + +- Omnibus Installation + + ```sh + sudo gitlab-rake geo:verification:repository:reset + ``` + +- Source Installation + + ```sh + sudo -u git -H bundle exec rake geo:verification:repository:reset RAILS_ENV=production + ``` + +For wikis: + +- Omnibus Installation + + ```sh + sudo gitlab-rake geo:verification:wiki:reset + ``` + +- Source Installation + + ```sh + sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production + ``` + +## Current limitations + +Until [issue #5064][ee-5064] is completed, background verification doesn't cover +CI job artifacts and traces, LFS objects, or user uploads in file storage. +Verify their integrity manually by following [these instructions][foreground-verification] +on both nodes, and comparing the output between them. + +Data in object storage is **not verified**, as the object store is responsible +for ensuring the integrity of the data. + +[reset-verification]: background_verification.md#reset-verification-for-projects-where-verification-has-failed +[foreground-verification]: ../../raketasks/check.md +[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md new file mode 100644 index 00000000000..f4d31a98080 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -0,0 +1,61 @@ +# Bring a demoted primary node back online **[PREMIUM ONLY]** + +After a failover, it is possible to fail back to the demoted **primary** node to +restore your original configuration. This process consists of two steps: + +1. Making the old **primary** node a **secondary** node. +1. Promoting a **secondary** node to a **primary** node. + +CAUTION: **Caution:** +If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch. + +## Configure the former **primary** node to be a **secondary** node + +Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like +repositories and uploads will not be replayed when bringing the former **primary** node back +into sync, which may result in increased disk usage. +Alternatively, you can [set up a new **secondary** GitLab instance][setup-geo] to avoid this. + +To bring the former **primary** node up to date: + +1. SSH into the former **primary** node that has fallen behind. +1. Make sure all the services are up: + + ```sh + sudo gitlab-ctl start + ``` + + > **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], + > you need to undo those steps now. For Debian/Ubuntu you just need to run + > `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install + > the GitLab instance from scratch and set it up as a **secondary** node by + > following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. + > + > **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) + > for this node during disaster recovery procedure you may need to [block + > all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic) + > during this procedure. + +1. [Setup database replication][database-replication]. Note that in this + case, **primary** node refers to the current **primary** node, and **secondary** node refers to the + former **primary** node. + +If you have lost your original **primary** node, follow the +[setup instructions][setup-geo] to set up a new **secondary** node. + +## Promote the **secondary** node to **primary** node + +When the initial replication is complete and the **primary** node and **secondary** node are +closely in sync, you can do a [planned failover]. + +## Restore the **secondary** node + +If your objective is to have two nodes again, you need to bring your **secondary** +node back online as well by repeating the first step +([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node)) +for the **secondary** node. + +[setup-geo]: ../replication/index.md#setup-instructions +[database-replication]: ../replication/database.md +[disaster-recovery-disable-primary]: index.md#step-2-permanently-disable-the-primary-node +[planned failover]: planned_failover.md diff --git a/doc/administration/geo/disaster_recovery/img/replication-status.png b/doc/administration/geo/disaster_recovery/img/replication-status.png Binary files differnew file mode 100644 index 00000000000..d7085927c75 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/replication-status.png diff --git a/doc/administration/geo/disaster_recovery/img/reverification-interval.png b/doc/administration/geo/disaster_recovery/img/reverification-interval.png Binary files differnew file mode 100644 index 00000000000..ad4597a4f49 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/reverification-interval.png diff --git a/doc/administration/geo/disaster_recovery/img/verification-status-primary.png b/doc/administration/geo/disaster_recovery/img/verification-status-primary.png Binary files differnew file mode 100644 index 00000000000..2503408ec5d --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/verification-status-primary.png diff --git a/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png b/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png Binary files differnew file mode 100644 index 00000000000..462274d8b14 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md new file mode 100644 index 00000000000..71dc797f281 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/index.md @@ -0,0 +1,322 @@ +# Disaster Recovery **[PREMIUM ONLY]** + +Geo replicates your database, your Git repositories, and few other assets. +We will support and replicate more data in the future, that will enable you to +failover with minimal effort, in a disaster situation. + +See [Geo current limitations][geo-limitations] for more information. + +CAUTION: **Warning:** +Disaster recovery for multi-secondary configurations is in **Alpha**. +For the latest updates, check the multi-secondary [Disaster Recovery epic][gitlab-org&65]. + +## Promoting a **secondary** Geo node in single-secondary configurations + +We don't currently provide an automated way to promote a Geo replica and do a +failover, but you can do it manually if you have `root` access to the machine. + +This process promotes a **secondary** Geo node to a **primary** node. To regain +geographic redundancy as quickly as possible, you should add a new **secondary** node +immediately after following these instructions. + +### Step 1. Allow replication to finish if possible + +If the **secondary** node is still replicating data from the **primary** node, follow +[the planned failover docs][planned-failover] as closely as possible in +order to avoid unnecessary data loss. + +### Step 2. Permanently disable the **primary** node + +CAUTION: **Warning:** +If the **primary** node goes offline, there may be data saved on the **primary** node +that has not been replicated to the **secondary** node. This data should be treated +as lost if you proceed. + +If an outage on the **primary** node happens, you should do everything possible to +avoid a split-brain situation where writes can occur in two different GitLab +instances, complicating recovery efforts. So to prepare for the failover, we +must disable the **primary** node. + +1. SSH into the **primary** node to stop and disable GitLab, if possible: + + ```sh + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```sh + sudo systemctl disable gitlab-runsvdir + ``` + + > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being + > started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). + > It may be safest to uninstall the GitLab package completely: + + ```sh + yum remove gitlab-ee + ``` + + > **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu + > or any other distro based on the Upstart init system, you can prevent GitLab + > from starting if the machine reboots by doing the following: + + ```sh + initctl stop gitlab-runsvvdir + echo 'manual' > /etc/init/gitlab-runsvdir.override + initctl reload-configuration + ``` + +1. If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + - Reconfigure the load balancers. + - Change DNS records (e.g., point the primary DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +1. If you plan to + [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + +### Step 3. Promoting a **secondary** node + +NOTE: **Note:** +A new **secondary** should not be added at this time. If you want to add a new +**secondary**, do this after you have completed the entire process of promoting +the **secondary** to the **primary**. + +#### Promoting a **secondary** node running on a single machine + +1. SSH in to your **secondary** node and login as root: + + ```sh + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by + removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Promote the **secondary** node to the **primary** node. Execute: + + ```sh + gitlab-ctl promote-to-primary-node + ``` + +1. Verify you can connect to the newly promoted **primary** node using the URL used + previously for the **secondary** node. +1. If successful, the **secondary** node has now been promoted to the **primary** node. + +#### Promoting a **secondary** node with HA + +The `gitlab-ctl promote-to-primary-node` command cannot be used yet in +conjunction with High Availability or with multiple machines, as it can only +perform changes on a **secondary** with only a single machine. Instead, you must +do this manually. + +1. SSH in to the database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```bash + sudo gitlab-pg-ctl promote + ``` + +1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. + +1. Promote the **secondary** to **primary**. SSH into a single application + server and execute: + + ```bash + sudo gitlab-rake geo:set_secondary_as_primary + ``` + +1. Verify you can connect to the newly promoted **primary** using the URL used + previously for the **secondary**. +1. Success! The **secondary** has now been promoted to **primary**. + +### Step 4. (Optional) Updating the primary domain DNS record + +Updating the DNS records for the primary domain to point to the **secondary** node +will prevent the need to update all references to the primary domain to the +secondary domain, like changing Git remotes and API URLs. + +1. SSH into the **secondary** node and login as root: + + ```sh + sudo -i + ``` + +1. Update the primary domain's DNS record. After updating the primary domain's + DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the + **secondary** node to reflect the new URL: + + ```ruby + # Change the existing external_url configuration + external_url 'https://<new_external_url>' + ``` + + NOTE: **Note** + Changing `external_url` won't prevent access via the old secondary URL, as + long as the secondary DNS records are still intact. + +1. Reconfigure the **secondary** node for the change to take effect: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Execute the command below to update the newly promoted **primary** node URL: + + ```sh + gitlab-rake geo:update_primary_node_url + ``` + + This command will use the changed `external_url` configuration defined + in `/etc/gitlab/gitlab.rb`. + +1. Verify you can connect to the newly promoted **primary** using its URL. + If you updated the DNS records for the primary domain, these changes may + not have yet propagated depending on the previous DNS records TTL. + +### Step 5. (Optional) Add **secondary** Geo node to a promoted **primary** node + +Promoting a **secondary** node to **primary** node using the process above does not enable +Geo on the new **primary** node. + +To bring a new **secondary** node online, follow the [Geo setup instructions][setup-geo]. + +### Step 6. (Optional) Removing the secondary's tracking database + +Every **secondary** has a special tracking database that is used to save the status of the synchronization of all the items from the **primary**. +Because the **secondary** is already promoted, that data in the tracking database is no longer required. + +The data can be removed with the following command: + +```sh +sudo rm -rf /var/opt/gitlab/geo-postgresql +``` + +## Promoting secondary Geo replica in multi-secondary configurations + +If you have more than one **secondary** node and you need to promote one of them, we suggest you follow +[Promoting a **secondary** Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) +and after that you also need two extra steps. + +### Step 1. Prepare the new **primary** node to serve one or more **secondary** nodes + +1. SSH into the new **primary** node and login as root: + + ```sh + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` + + ```ruby + ## Enable a Geo Primary role (if you haven't yet) + roles ['geo_primary_role'] + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have + postgresql['max_replication_slots'] = 1 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + + ``` + + (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) + +1. Save the file and reconfigure GitLab for the database listen changes and + the replication slot changes to be applied. + + ```sh + gitlab-ctl reconfigure + ``` + + Restart PostgreSQL for its changes to take effect: + + ```sh + gitlab-ctl restart postgresql + ``` + +1. Re-enable migrations now that PostgreSQL is restarted and listening on the + private address. + + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + + Save the file and reconfigure GitLab: + + ```sh + gitlab-ctl reconfigure + ``` + +### Step 2. Initiate the replication process + +Now we need to make each **secondary** node listen to changes on the new **primary** node. To do that you need +to [initiate the replication process][initiate-the-replication-process] again but this time +for another **primary** node. All the old replication settings will be overwritten. + +## Troubleshooting + +### I followed the disaster recovery instructions and now two-factor auth is broken! + +The setup instructions for Geo prior to 10.5 failed to replicate the +`otp_key_base` secret, which is used to encrypt the two-factor authentication +secrets stored in the database. If it differs between **primary** and **secondary** +nodes, users with two-factor authentication enabled won't be able to log in +after a failover. + +If you still have access to the old **primary** node, you can follow the +instructions in the +[Upgrading to GitLab 10.5][updating-geo] +section to resolve the error. Otherwise, the secret is lost and you'll need to +[reset two-factor authentication for all users][sec-tfa]. + +[gitlab-org&65]: https://gitlab.com/groups/gitlab-org/-/epics/65 +[geo-limitations]: ../replication/index.md#current-limitations +[planned-failover]: planned_failover.md +[setup-geo]: ../replication/index.md#setup-instructions +[updating-geo]: ../replication/updating_the_geo_nodes.md#upgrading-to-gitlab-105 +[sec-tfa]: ../../../security/two_factor_authentication.md#disabling-2fa-for-everyone +[gitlab-org/omnibus-gitlab#3058]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058 +[gitlab-org/gitlab-ee#4284]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4284 +[initiate-the-replication-process]: ../replication/database.html#step-3-initiate-the-replication-process +[configure-the-primary-server]: ../replication/database.html#step-1-configure-the-primary-server diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md new file mode 100644 index 00000000000..88ab12d910a --- /dev/null +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -0,0 +1,227 @@ +# Disaster recovery for planned failover **[PREMIUM ONLY]** + +The primary use-case of Disaster Recovery is to ensure business continuity in +the event of unplanned outage, but it can be used in conjunction with a planned +failover to migrate your GitLab instance between regions without extended +downtime. + +As replication between Geo nodes is asynchronous, a planned failover requires +a maintenance window in which updates to the **primary** node are blocked. The +length of this window is determined by your replication capacity - once the +**secondary** node is completely synchronized with the **primary** node, the failover can occur without +data loss. + +This document assumes you already have a fully configured, working Geo setup. +Please read it and the [Disaster Recovery][disaster-recovery] failover +documentation in full before proceeding. Planned failover is a major operation, +and if performed incorrectly, there is a high risk of data loss. Consider +rehearsing the procedure until you are comfortable with the necessary steps and +have a high degree of confidence in being able to perform them accurately. + +## Not all data is automatically replicated + +If you are using any GitLab features that Geo [doesn't support][limitations], +you must make separate provisions to ensure that the **secondary** node has an +up-to-date copy of any data associated with that feature. This may extend the +required scheduled maintenance period significantly. + +A common strategy for keeping this period as short as possible for data stored +in files is to use `rsync` to transfer the data. An initial `rsync` can be +performed ahead of the maintenance window; subsequent `rsync`s (including a +final transfer inside the maintenance window) will then transfer only the +*changes* between the **primary** node and the **secondary** nodes. + +Repository-centric strategies for using `rsync` effectively can be found in the +[moving repositories][moving-repositories] documentation; these strategies can +be adapted for use with any other file-based data, such as GitLab Pages (to +be found in `/var/opt/gitlab/gitlab-rails/shared/pages` if using Omnibus). + +## Pre-flight checks + +Follow these steps before scheduling a planned failover to ensure the process +will go smoothly. + +### Object storage + +Some classes of non-repository data can use object storage in preference to +file storage. Geo [does not replicate data in object storage](../replication/object_storage.md), +leaving that task up to the object store itself. For a planned failover, this +means you can decouple the replication of this data from the failover of the +GitLab service. + +If you're already using object storage, simply verify that your **secondary** +node has access to the same data as the **primary** node - they must either they share the +same object storage configuration, or the **secondary** node should be configured to +access a [geographically-replicated][os-repl] copy provided by the object store +itself. + +If you have a large GitLab installation or cannot tolerate downtime, consider +[migrating to Object Storage][os-conf] **before** scheduling a planned failover. +Doing so reduces both the length of the maintenance window, and the risk of data +loss as a result of a poorly executed planned failover. + +### Review the configuration of each **secondary** node + +Database settings are automatically replicated to the **secondary** node, but the +`/etc/gitlab/gitlab.rb` file must be set up manually, and differs between +nodes. If features such as Mattermost, OAuth or LDAP integration are enabled +on the **primary** node but not the **secondary** node, they will be lost during failover. + +Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node +supports everything the **primary** node does **before** scheduling a planned failover. + +### Run system checks + +Run the following on both **primary** and **secondary** nodes: + +```sh +gitlab-rake gitlab:check +gitlab-rake gitlab:geo:check +``` + +If any failures are reported on either node, they should be resolved **before** +scheduling a planned failover. + +### Check that secrets match between nodes + +The SSH host keys and `/etc/gitlab/gitlab-secrets.json` files should be +identical on all nodes. Check this by running the following on all nodes and +comparing the output: + +```sh +sudo sha256sum /etc/ssh/ssh_host* /etc/gitlab/gitlab-secrets.json +``` + +If any files differ, replace the content on the **secondary** node with the +content from the **primary** node. + +### Ensure Geo replication is up-to-date + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node to +review status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in grey), consider giving the node more +time to complete + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. Following a planned failover, anything that +failed to replicate will be **lost**. + +You can use the [Geo status API](https://docs.gitlab.com/ee/api/geo_nodes.html#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and +the reasons for failure. + +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +### Verify the integrity of replicated data + +This [content was moved to another location][background-verification]. + +### Notify users of scheduled maintenance + +On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast +message. You can check under **Admin Area > Geo** to estimate how long it +will take to finish syncing. An example message would be: + +> A scheduled maintenance will take place at XX:XX UTC. We expect it to take +> less than 1 hour. + +## Prevent updates to the **primary** node + +Until a [read-only mode][ce-19739] is implemented, updates must be prevented +from happening manually. Note that your **secondary** node still needs read-only +access to the **primary** node during the maintenance window. + +1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you might run the following commands on the server(s) making up your **primary** node: + + ```sh + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + +1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + +1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + +1. Disable non-Geo periodic background jobs on the primary node by navigating + to **Admin Area > Monitoring > Background Jobs > Cron** , pressing `Disable All`, + and then pressing `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +## Finish replicating and verifying all data + +1. If you are manually replicating any data not managed by Geo, trigger the + final replication process now. +1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. +1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + +1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. +1. On the **secondary** node, use [these instructions][foreground-verification] + to verify the integrity of CI artifacts, LFS objects and uploads in file + storage. + +At this point, your **secondary** node will contain an up-to-date copy of everything the +**primary** node has, meaning nothing will be lost when you fail over. + +## Promote the **secondary** node + +Finally, follow the [Disaster Recovery docs][disaster-recovery] to promote the +**secondary** node to a **primary** node. This process will cause a brief outage on the **secondary** node, and users may need to log in again. + +Once it is completed, the maintenance window is over! Your new **primary** node will now +begin to diverge from the old one. If problems do arise at this point, failing +back to the old **primary** node [is possible][bring-primary-back], but likely to result +in the loss of any data uploaded to the new primary in the meantime. + +Don't forget to remove the broadcast message after failover is complete. + +[bring-primary-back]: bring_primary_back.md +[ce-19739]: https://gitlab.com/gitlab-org/gitlab-ce/issues/19739 +[container-registry]: ../replication/container_registry.md +[disaster-recovery]: index.md +[ee-4930]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4930 +[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 +[foreground-verification]: ../../raketasks/check.md +[background-verification]: background_verification.md +[limitations]: ../replication/index.md#current-limitations +[moving-repositories]: ../../operations/moving_repositories.md +[os-conf]: ../replication/object_storage.md#configuration +[os-repl]: ../replication/object_storage.md#replication diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md new file mode 100644 index 00000000000..57735b21cda --- /dev/null +++ b/doc/administration/geo/replication/configuration.md @@ -0,0 +1,314 @@ +# Geo configuration (GitLab Omnibus) **[PREMIUM ONLY]** + +NOTE: **Note:** +This is the documentation for the Omnibus GitLab packages. For installations +from source, follow the [**Geo nodes configuration for installations +from source**][configuration-source] guide. + +## Configuring a new **secondary** node + +NOTE: **Note:** +This is the final step in setting up a **secondary** Geo node. Stages of the +setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages][setup-geo-omnibus]. + +The basic steps of configuring a **secondary** node are to: + +- Replicate required configurations between the **primary** node and the **secondary** nodes. +- Configure a tracking database on each **secondary** node. +- Start GitLab on each **secondary** node. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +> **Notes:** +> - **Do not** setup any custom authentication for the **secondary** nodes. This will be + handled by the **primary** node. +> - Any change that requires access to the **Admin Area** needs to be done in the + **primary** node because the **secondary** node is a read-only replica. + +### Step 1. Manually replicate secret GitLab values + +GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` +file which *must* be the same on all nodes. Until there is +a means of automatically replicating these between nodes (see issue [gitlab-org/gitlab-ee#3789]), +they must be manually replicated to the **secondary** node. + +1. SSH into the **primary** node, and execute the command below: + + ```sh + sudo cat /etc/gitlab/gitlab-secrets.json + ``` + + This will display the secrets that need to be replicated, in JSON format. + +1. SSH into the **secondary** node and login as the `root` user: + + ```sh + sudo -i + ``` + +1. Make a backup of any existing secrets: + + ```sh + mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` + ``` + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or + copy-and-paste the file contents between nodes: + + ```sh + sudo editor /etc/gitlab/gitlab-secrets.json + + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` + +1. Ensure the file permissions are correct: + + ```sh + chown root:root /etc/gitlab/gitlab-secrets.json + chmod 0600 /etc/gitlab/gitlab-secrets.json + ``` + +1. Reconfigure the **secondary** node for the change to take effect: + + ```sh + gitlab-ctl reconfigure + gitlab-ctl restart + ``` + +### Step 2. Manually replicate the **primary** node's SSH host keys + +GitLab integrates with the system-installed SSH daemon, designating a user +(typically named git) through which all access requests are handled. + +In a [Disaster Recovery] situation, GitLab system +administrators will promote a **secondary** node to the **primary** node. DNS records for the +**primary** domain should also be updated to point to the new **primary** node +(previously a **secondary** node). Doing so will avoid the need to update Git remotes and API URLs. + +This will cause all SSH requests to the newly promoted **primary** node to +fail due to SSH host key mismatch. To prevent this, the primary SSH host +keys must be manually replicated to the **secondary** node. + +1. SSH into the **secondary** node and login as the `root` user: + + ```sh + sudo -i + ``` + +1. Make a backup of any existing SSH host keys: + + ```sh + find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; + ``` + +1. Copy OpenSSH host keys from the **primary** node: + + If you can access your **primary** node using the **root** user: + + ```sh + # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server + scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh + ``` + + If you only have access through a user with **sudo** privileges: + + ```sh + # Run this from your primary node: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* + + # Run this from your secondary node: + scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . + tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh + ``` + +1. On your **secondary** node, ensure the file permissions are correct: + + ```sh + chown root:root /etc/ssh/ssh_host_*_key* + chmod 0600 /etc/ssh/ssh_host_*_key* + ``` + +1. To verify key fingerprint matches, execute the following command on both nodes: + + ```sh + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ``` + + You should get an output similar to this one and they should be identical on both nodes: + + ```sh + 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) + 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) + 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) + 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) + ``` + +1. Verify that you have the correct public keys for the existing private keys: + + ```sh + # This will print the fingerprint for private keys: + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + + # This will print the fingerprint for public keys: + for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done + ``` + + NOTE: **Note**: + The output for private keys and public keys command should generate the same fingerprint. + +1. Restart sshd on your **secondary** node: + + ```sh + # Debian or Ubuntu installations + sudo service ssh reload + + # CentOS installations + sudo service sshd reload + ``` + +### Step 3. Add the **secondary** node + +1. Visit the **primary** node's **Admin Area > Geo** + (`/admin/geo/nodes`) in your browser. +1. Add the **secondary** node by providing its full URL. **Do NOT** check the + **This is a primary node** checkbox. +1. Optionally, choose which groups or storage shards should be replicated by the + **secondary** node. Leave blank to replicate all. Read more in + [selective synchronization](#selective-synchronization). +1. Click the **Add node** button. +1. SSH into your GitLab **secondary** server and restart the services: + + ```sh + gitlab-ctl restart + ``` + + Check if there are any common issue with your Geo setup by running: + + ```sh + gitlab-rake gitlab:geo:check + ``` + +1. SSH into your **primary** server and login as root to verify the + **secondary** node is reachable or there are any common issue with your Geo setup: + + ```sh + gitlab-rake gitlab:geo:check + ``` + +Once added to the admin panel and restarted, the **secondary** node will automatically start +replicating missing data from the **primary** node in a process known as **backfill**. +Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so +that the **secondary** node can act on those notifications immediately. + +Make sure the **secondary** node is running and accessible. +You can login to the **secondary** node with the same credentials as used for the **primary** node. + +### Step 4. Enabling Hashed Storage + +Using Hashed Storage significantly improves Geo replication. Project and group +renames no longer require synchronization between nodes. + +1. Visit the **primary** node's **Admin Area > Settings > Repository** + (`/admin/application_settings/repository`) in your browser. +1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**. + +### Step 5. (Optional) Configuring the **secondary** node to trust the **primary** node + +You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. + +If your **primary** node is using a self-signed certificate for *HTTPS* support, you will +need to add that certificate to the **secondary** node's trust store. Retrieve the +certificate from the **primary** node and follow +[these instructions][omnibus-ssl] +on the **secondary** node. + +### Step 6. Enable Git access over HTTP/HTTPS + +Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone +method to be enabled. Navigate to **Admin Area > Settings** +(`/admin/application_settings`) on the **primary** node, and set +`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. + +### Step 7. Verify proper functioning of the **secondary** node + +Your **secondary** node is now configured! + +You can login to the **secondary** node with the same credentials you used for the +**primary** node. Visit the **secondary** node's **Admin Area > Geo** +(`/admin/geo/nodes`) in your browser to check if it's correctly identified as a +**secondary** Geo node and if Geo is enabled. + +The initial replication, or 'backfill', will probably still be in progress. You +can monitor the synchronization process on each geo node from the **primary** +node's Geo Nodes dashboard in your browser. + + + +If your installation isn't working properly, check the +[troubleshooting document]. + +The two most obvious issues that can become apparent in the dashboard are: + +1. Database replication not working well. +1. Instance to instance notification not working. In that case, it can be + something of the following: + - You are using a custom certificate or custom CA (see the + [troubleshooting document]). + - The instance is firewalled (check your firewall rules). + +Please note that disabling a **secondary** node will stop the synchronization process. + +Please note that if `git_data_dirs` is customized on the **primary** node for multiple +repository shards you must duplicate the same configuration on each **secondary** node. + +Point your users to the ["Using a Geo Server" guide][using-geo]. + +Currently, this is what is synced: + +- Git repositories. +- Wikis. +- LFS objects. +- Issues, merge requests, snippets, and comment attachments. +- Users, groups, and project avatars. + +## Selective synchronization + +Geo supports selective synchronization, which allows admins to choose +which projects should be synchronized by **secondary** nodes. +A subset of projects can be chosen, either by group or by storage shard. The +former is ideal for replicating data belonging to a subset of users, while the +latter is more suited to progressively rolling out Geo to a large GitLab +instance. + +It is important to note that selective synchronization: + +1. Does not restrict permissions from **secondary** nodes. +1. Does not hide project metadata from **secondary** nodes. + - Since Geo currently relies on PostgreSQL replication, all project metadata + gets replicated to **secondary** nodes, but repositories that have not been + selected will be empty. +1. Does not reduce the number of events generated for the Geo event log. + - The **primary** node generates events as long as any **secondary** nodes are present. + Selective synchronization restrictions are implemented on the **secondary** nodes, + not the **primary** node. + +## Upgrading Geo + +See the [updating the Geo nodes document](updating_the_geo_nodes.md). + +## Troubleshooting + +See the [troubleshooting document](troubleshooting.md). + +[configuration-source]: configuration_source.md +[setup-geo-omnibus]: index.md#using-omnibus-gitlab +[Hashed Storage]: ../../repository_storage_types.md +[Disaster Recovery]: ../disaster_recovery/index.md +[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 +[gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 +[omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html +[troubleshooting document]: troubleshooting.md +[using-geo]: using_a_geo_server.md diff --git a/doc/administration/geo/replication/configuration_source.md b/doc/administration/geo/replication/configuration_source.md new file mode 100644 index 00000000000..10dd9405b4b --- /dev/null +++ b/doc/administration/geo/replication/configuration_source.md @@ -0,0 +1,172 @@ +# Geo configuration (source) **[PREMIUM ONLY]** + +NOTE: **Note:** +This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release. +Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations +using the Omnibus GitLab packages, follow the +[**Omnibus Geo nodes configuration**][configuration] guide. + +## Configuring a new **secondary** node + +NOTE: **Note:** +This is the final step in setting up a **secondary** node. Stages of the setup +process must be completed in the documented order. Before attempting the steps +in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated). + +The basic steps of configuring a **secondary** node are to: + +- Replicate required configurations between the **primary** and **secondary** nodes. +- Configure a tracking database on each **secondary** node. +- Start GitLab on the **secondary** node. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +NOTE: **Note:** +**Do not** set up any custom authentication on **secondary** nodes, this will be handled by the **primary** node. + +NOTE: **Note:** +**Do not** add anything in the **secondary** node's admin area (**Admin Area > Geo**). This is handled solely by the **primary** node. + +### Step 1. Manually replicate secret GitLab values + +GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml` +file which *must* match between the **primary** and **secondary** nodes. Until there is +a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must +be manually replicated to **secondary** nodes. + +1. SSH into the **primary** node, and execute the command below: + + ```sh + sudo cat /home/git/gitlab/config/secrets.yml + ``` + + This will display the secrets that need to be replicated, in YAML format. + +1. SSH into the **secondary** node and login as the `git` user: + + ```sh + sudo -i -u git + ``` + +1. Make a backup of any existing secrets: + + ```sh + mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F` + ``` + +1. Copy `/home/git/gitlab/config/secrets.yml` from the **primary** node to the **secondary** node, or + copy-and-paste the file contents between nodes: + + ```sh + sudo editor /home/git/gitlab/config/secrets.yml + + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` + +1. Ensure the file permissions are correct: + + ```sh + chown git:git /home/git/gitlab/config/secrets.yml + chmod 0600 /home/git/gitlab/config/secrets.yml + ``` + +1. Restart GitLab + + ```sh + service gitlab restart + ``` + +Once restarted, the **secondary** node will automatically start replicating missing data +from the **primary** node in a process known as backfill. Meanwhile, the **primary** node +will start to notify the **secondary** node of any changes, so that the **secondary** node can +act on those notifications immediately. + +Make sure the **secondary** node is running and accessible. You can login to +the **secondary** node with the same credentials as used for the **primary** node. + +### Step 2. Manually replicate the **primary** node's SSH host keys + +Read [Manually replicate the **primary** node's SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) + +### Step 3. Add the **secondary** GitLab node + +1. Navigate to the **primary** node's **Admin Area > Geo** + (`/admin/geo/nodes`) in your browser. +1. Add the **secondary** node by providing its full URL. **Do NOT** check the + **This is a primary node** checkbox. +1. Optionally, choose which namespaces should be replicated by the + **secondary** node. Leave blank to replicate all. Read more in + [selective synchronization](#selective-synchronization). +1. Click the **Add node** button. +1. SSH into your GitLab **secondary** server and restart the services: + + ```sh + service gitlab restart + ``` + + Check if there are any common issue with your Geo setup by running: + + ```sh + bundle exec rake gitlab:geo:check + ``` + +1. SSH into your GitLab **primary** server and login as root to verify the + **secondary** node is reachable or there are any common issue with your Geo setup: + + ```sh + bundle exec rake gitlab:geo:check + ``` + +Once reconfigured, the **secondary** node will automatically start +replicating missing data from the **primary** node in a process known as backfill. +Meanwhile, the **primary** node will start to notify the **secondary** node of any changes, so +that the **secondary** node can act on those notifications immediately. + +Make sure the **secondary** node is running and accessible. +You can log in to the **secondary** node with the same credentials as used for the **primary** node. + +### Step 4. Enabling Hashed Storage + +Read [Enabling Hashed Storage](configuration.md#step-4-enabling-hashed-storage). + +### Step 5. (Optional) Configuring the secondary to trust the primary + +You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. + +If your **primary** node is using a self-signed certificate for *HTTPS* support, you will +need to add that certificate to the **secondary** node's trust store. Retrieve the +certificate from the **primary** node and follow your distribution's instructions for +adding it to the **secondary** node's trust store. In Debian/Ubuntu, you would follow these steps: + +```sh +sudo -i +cp <primary_node_certification_file> /usr/local/share/ca-certificates +update-ca-certificates +``` + +### Step 6. Enable Git access over HTTP/HTTPS + +Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone +method to be enabled. Navigate to **Admin Area > Settings** +(`/admin/application_settings`) on the **primary** node, and set +`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. + +### Step 7. Verify proper functioning of the secondary node + +Read [Verify proper functioning of the secondary node][configuration-verify-node]. + +## Selective synchronization + +Read [Selective synchronization][configuration-selective-replication]. + +## Troubleshooting + +Read the [troubleshooting document][troubleshooting]. + +[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 +[configuration]: configuration.md +[configuration-selective-replication]: configuration.md#selective-synchronization +[configuration-verify-node]: configuration.md#step-7-verify-proper-functioning-of-the-secondary-node +[troubleshooting]: troubleshooting.md diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md new file mode 100644 index 00000000000..e57583a3bf9 --- /dev/null +++ b/doc/administration/geo/replication/database.md @@ -0,0 +1,507 @@ +# Geo database replication (GitLab Omnibus) **[PREMIUM ONLY]** + +NOTE: **Note:** +This is the documentation for the Omnibus GitLab packages. For installations +from source, follow the +[Geo database replication (source)](database_source.md) guide. + +NOTE: **Note:** +If your GitLab installation uses external (not managed by Omnibus) PostgreSQL +instances, the Omnibus roles will not be able to perform all necessary +configuration steps. In this case, +[follow the Geo with external PostgreSQL instances document instead](external_database.md). + +NOTE: **Note:** +The stages of the setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages][toc]. + +This document describes the minimal steps you have to take in order to +replicate your **primary** GitLab database to a **secondary** node's database. You may +have to change some values according to your database setup, how big it is, etc. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +## PostgreSQL replication + +The GitLab **primary** node where the write operations happen will connect to +the **primary** database server, and **secondary** nodes will +connect to their own database servers (which are also read-only). + +NOTE: **Note:** +In database documentation, you may see "**primary**" being referenced as "master" +and "**secondary**" as either "slave" or "standby" server (read-only). + +We recommend using [PostgreSQL replication slots][replication-slots-article] +to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to +recover. See below for more details. + +The following guide assumes that: + +- You are using Omnibus and therefore you are using PostgreSQL 9.6 or later + which includes the [`pg_basebackup` tool][pgback] and improved + [Foreign Data Wrapper][FDW] support. +- You have a **primary** node already set up (the GitLab server you are + replicating from), running Omnibus' PostgreSQL (or equivalent version), and + you have a new **secondary** server set up with the same versions of the OS, + PostgreSQL, and GitLab on all nodes. + +CAUTION: **Warning:** +Geo works with streaming replication. Logical replication is not supported at this time. +There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420). + +### Step 1. Configure the **primary** server + +1. SSH into your GitLab **primary** server and login as root: + + ```sh + sudo -i + ``` + +1. Execute the command below to define the node as **primary** node: + + ```sh + gitlab-ctl set-geo-primary-node + ``` + + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + +1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: + + Generate a MD5 hash of the desired password: + + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` + +1. Omnibus GitLab already has a [replication user] + called `gitlab_replicator`. You must set the password for this user manually. + You will be prompted to enter a password: + + ```sh + gitlab-ctl set-replication-password + ``` + + This command will also read the `postgresql['sql_replication_user']` Omnibus + setting in case you have changed `gitlab_replicator` username to something + else. + + If you are using an external database not managed by Omnibus GitLab, you need + to create the replicator user and define a password to it manually. + For information on how to create a replication user, refer to the + [appropriate step](database_source.md#step-1-configure-the-primary-server) + in [Geo database replication (source)](database_source.md). + +1. Configure PostgreSQL to listen on network interfaces: + + For security reasons, PostgreSQL does not listen on any network interfaces + by default. However, Geo requires the **secondary** node to be able to + connect to the **primary** node's database. For this reason, we need the address of + each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md). + + If you are using a cloud provider, you can lookup the addresses for each + Geo node through your cloud provider's management console. + + To lookup the address of a Geo node, SSH in to the Geo node and execute: + + ```sh + ## + ## Private address + ## + ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' + + ## + ## Public address + ## + echo "External address: $(curl --silent ipinfo.io/ip)" + ``` + + In most cases, the following addresses will be used to configure GitLab + Geo: + + | Configuration | Address | + |:----------------------------------------|:------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + + If you are using Google Cloud Platform, SoftLayer, or any other vendor that + provides a virtual private cloud (VPC) you can use the **secondary** node's private + address (corresponds to "internal address" for Google Cloud Platform) for + `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. + + The `listen_address` option opens PostgreSQL up to network connections + with the interface corresponding to the given address. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + Depending on your network configuration, the suggested addresses may not + be correct. If your **primary** node and **secondary** nodes connect over a local + area network, or a virtual network connecting availability zones like + [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) + you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## Primary address + ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node + ## + postgresql['listen_address'] = '<primary_node_ip>' + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + ## + ## Replication settings + ## - set this to be the number of Geo secondary nodes you have + ## + postgresql['max_replication_slots'] = 1 + # postgresql['max_wal_senders'] = 10 + # postgresql['wal_keep_segments'] = 10 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` + +1. Optional: If you want to add another **secondary** node, the relevant setting would look like: + + ```ruby + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + ``` + + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the [PostgreSQL - + Replication documentation][pg-docs-runtime-replication] + for more information. + +1. Save the file and reconfigure GitLab for the database listen changes and + the replication slot changes to be applied: + + ```sh + gitlab-ctl reconfigure + ``` + + Restart PostgreSQL for its changes to take effect: + + ```sh + gitlab-ctl restart postgresql + ``` + +1. Re-enable migrations now that PostgreSQL is restarted and listening on the + private address. + + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + + Save the file and reconfigure GitLab: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Now that the PostgreSQL server is set up to accept remote connections, run + `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port + `5432` to the **primary** server's private address. + +1. A certificate was automatically generated when GitLab was reconfigured. This + will be used automatically to protect your PostgreSQL traffic from + eavesdroppers, but to protect against active ("man-in-the-middle") attackers, + the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL + `server.crt` file on the **primary** node by running this command: + + ```sh + cat ~gitlab-psql/data/server.crt + ``` + + Copy the output into a clipboard or into a local file. You + will need it when setting up the **secondary** node! The certificate is not sensitive + data. + +### Step 2. Configure the **secondary** server + +1. SSH into your GitLab **secondary** server and login as root: + + ``` + sudo -i + ``` + +1. Stop application server and Sidekiq + + ``` + gitlab-ctl stop unicorn + gitlab-ctl stop sidekiq + ``` + + NOTE: **Note**: + This step is important so we don't try to execute anything before the node is fully configured. + +1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server: + + ```sh + gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + ``` + + NOTE: **Note**: + If this step fails, you may be using the wrong IP address, or a firewall may + be preventing access to the server. Check the IP address, paying close + attention to the difference between public and private addresses and ensure + that, if a firewall is present, the **secondary** node is permitted to connect to the + **primary** node on port 5432. + +1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: + + ``` + editor server.crt + ``` + +1. Set up PostgreSQL TLS verification on the **secondary** node: + + Install the `server.crt` file: + + ```sh + install \ + -D \ + -o gitlab-psql \ + -g gitlab-psql \ + -m 0400 \ + -T server.crt ~gitlab-psql/.postgresql/root.crt + ``` + + PostgreSQL will now only recognize that exact certificate when verifying TLS + connections. The certificate can only be replicated by someone with access + to the private key, which is **only** present on the **primary** node. + +1. Test that the `gitlab-psql` user can connect to the **primary** node's database + (the default Omnibus database name is gitlabhq_production): + + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + --list \ + -U gitlab_replicator \ + -d "dbname=gitlabhq_production sslmode=verify-ca" \ + -W \ + -h <primary_node_ip> + ``` + + When prompted enter the password you set in the first step for the + `gitlab_replicator` user. If all worked correctly, you should see + the list of **primary** node's databases. + + A failure to connect here indicates that the TLS configuration is incorrect. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + +1. Configure PostgreSQL to enable FDW support: + + This step is similar to how we configured the **primary** instance. + We need to enable this, to enable FDW support, even if using a single node. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## Enable FDW support for the Geo Tracking Database (improves performance) + ## + geo_secondary['db_fdw'] = true + ``` + + For external PostgreSQL instances, see [additional instructions](external_database.md). + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + +1. Reconfigure GitLab for the changes to take effect: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Restart PostgreSQL for the IP change to take effect and reconfigure again: + + ```sh + gitlab-ctl restart postgresql + gitlab-ctl reconfigure + ``` + + This last reconfigure will provision the FDW configuration and enable it. + +### Step 3. Initiate the replication process + +Below we provide a script that connects the database on the **secondary** node to +the database on the **primary** node, replicates the database, and creates the +needed files for streaming replication. + +The directories used are the defaults that are set up in Omnibus. If you have +changed any defaults or are using a source installation, configure it as you +see fit replacing the directories and paths. + +CAUTION: **Warning:** +Make sure to run this on the **secondary** server as it removes all PostgreSQL's +data before running `pg_basebackup`. + +1. SSH into your GitLab **secondary** server and login as root: + + ```sh + sudo -i + ``` + +1. Choose a database-friendly name to use for your **secondary** node to + use as the replication slot name. For example, if your domain is + `secondary.geo.example.com`, you may use `secondary_example` as the slot + name as shown in the commands below. + +1. Execute the command below to start a backup/restore and begin the replication + CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. + Using the same slot name between two secondaries will break PostgreSQL replication. + + ```sh + gitlab-ctl replicate-geo-database \ + --slot-name=<secondary_node_name> \ + --host=<primary_node_ip> + ``` + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + + This command also takes a number of additional options. You can use `--help` + to list them all, but here are a couple of tips: + - If PostgreSQL is listening on a non-standard port, add `--port=` as well. + - If your database is too large to be transferred in 30 minutes, you will need + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` + +The replication process is now complete. + +## PGBouncer support (optional) + +[PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool +PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a +high-availability configuration with a cluster of nodes supporting a Geo +**primary** node and another cluster of nodes supporting a Geo **secondary** node. For more +information, see the [Omnibus HA](https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability) +documentation. + +For a Geo **secondary** node to work properly with PGBouncer in front of the database, +it will need a separate read-only user to make [PostgreSQL FDW queries][FDW] +work: + +1. On the **primary** Geo database, enter the PostgreSQL on the console as an + admin user. If you are using an Omnibus-managed database, log onto the **primary** + node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): + + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` + +1. Then create the read-only user: + + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` + +1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`: + + ``` + geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' + ``` + +1. Save the file and reconfigure GitLab for the changes to be applied: + + ```sh + gitlab-ctl reconfigure + ``` + +## MySQL replication + +MySQL replication is not supported for Geo. + +## Troubleshooting + +Read the [troubleshooting document](troubleshooting.md). + +[replication-slots-article]: https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75 +[pgback]: http://www.postgresql.org/docs/9.2/static/app-pgbasebackup.html +[replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication +[FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html +[toc]: index.md#using-omnibus-gitlab +[rake-maintenance]: ../../raketasks/maintenance.md +[pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION +[pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html +[pg-docs-runtime-replication]: https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html diff --git a/doc/administration/geo/replication/database_source.md b/doc/administration/geo/replication/database_source.md new file mode 100644 index 00000000000..67cf8b6535f --- /dev/null +++ b/doc/administration/geo/replication/database_source.md @@ -0,0 +1,439 @@ +# Geo database replication (source) **[PREMIUM ONLY]** + +NOTE: **Note:** +This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release. +Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations +using the Omnibus GitLab packages, follow the +[**database replication for Omnibus GitLab**][database] guide. + +NOTE: **Note:** +The stages of the setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated). + +This document describes the minimal steps you have to take in order to +replicate your **primary** GitLab database to a **secondary** node's database. You may +have to change some values according to your database setup, how big it is, etc. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +## PostgreSQL replication + +The GitLab **primary** node where the write operations happen will connect to +**primary** database server, and the **secondary** ones which are read-only will +connect to **secondary** database servers (which are read-only too). + +NOTE: **Note:** +In many databases' documentation, you will see "**primary**" being referenced as "master" +and "**secondary**" as either "slave" or "standby" server (read-only). + +We recommend using [PostgreSQL replication slots][replication-slots-article] +to ensure the **primary** node retains all the data necessary for the secondaries to +recover. See below for more details. + +The following guide assumes that: + +- You are using PostgreSQL 9.6 or later which includes the + [`pg_basebackup` tool][pgback] and improved [Foreign Data Wrapper][FDW] support. +- You have a **primary** node already set up (the GitLab server you are + replicating from), running PostgreSQL 9.6 or later, and + you have a new **secondary** server set up with the same versions of the OS, + PostgreSQL, and GitLab on all nodes. +- The IP of the **primary** server for our examples is `198.51.100.1`, whereas the + **secondary** node's IP is `198.51.100.2`. Note that the **primary** and **secondary** servers + **must** be able to communicate over these addresses. These IP addresses can either + be public or private. + +CAUTION: **Warning:** +Geo works with streaming replication. Logical replication is not supported at this time. +There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420). + +### Step 1. Configure the **primary** server + +1. SSH into your GitLab **primary** server and login as root: + + ```sh + sudo -i + ``` + +1. Add this node as the Geo **primary** by running: + + ```sh + bundle exec rake geo:set_primary_node + ``` + +1. Create a [replication user] named `gitlab_replicator`: + + ```sql + --- Create a new user 'replicator' + CREATE USER gitlab_replicator; + + --- Set/change a password and grants replication privilege + ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; + ``` + +1. Make sure your the `gitlab` database user has a password defined: + + ```sh + sudo \ + -u postgres psql \ + -d template1 \ + -c "ALTER USER gitlab WITH ENCRYPTED PASSWORD '<database_password>';" + ``` + +1. Edit the content of `database.yml` in `production:` and add the password like the example below: + + ```yaml + # + # PRODUCTION + # + production: + adapter: postgresql + encoding: unicode + database: gitlabhq_production + pool: 10 + username: gitlab + password: <database_password> + host: /var/opt/gitlab/geo-postgresql + ``` + +1. Set up TLS support for the PostgreSQL **primary** server: + + CAUTION: **Warning**: + Only skip this step if you **know** that PostgreSQL traffic + between the **primary** and **secondary** nodes will be secured through some other + means, e.g., a known-safe physical network path or a site-to-site VPN that + you have configured. + + If you are replicating your database across the open Internet, it is + **essential** that the connection is TLS-secured. Correctly configured, this + provides protection against both passive eavesdroppers and active + "man-in-the-middle" attackers. + + To generate a self-signed certificate and key, run this command: + + ```sh + openssl req \ + -nodes \ + -batch \ + -x509 \ + -newkey rsa:4096 \ + -keyout server.key \ + -out server.crt \ + -days 3650 + ``` + + This will create two files - `server.key` and `server.crt` - that you can + use for authentication. + + Copy them to the correct location for your PostgreSQL installation: + + ```sh + # Copying a self-signed certificate and key + install -o postgres -g postgres -m 0400 -T server.crt ~postgres/9.x/main/data/server.crt + install -o postgres -g postgres -m 0400 -T server.key ~postgres/9.x/main/data/server.key + ``` + + Add this configuration to `postgresql.conf`, removing any existing + configuration for `ssl_cert_file` or `ssl_key_file`: + + ``` + ssl = on + ssl_cert_file='server.crt' + ssl_key_file='server.key' + ``` + +1. Edit `postgresql.conf` to configure the **primary** server for streaming replication + (for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`): + + ``` + listen_address = '<primary_node_ip>' + wal_level = hot_standby + max_wal_senders = 5 + min_wal_size = 80MB + max_wal_size = 1GB + max_replicaton_slots = 1 # Number of Geo secondary nodes + wal_keep_segments = 10 + hot_standby = on + ``` + + NOTE: **Note**: + Be sure to set `max_replication_slots` to the number of Geo **secondary** + nodes that you may potentially have (at least 1). + + For security reasons, PostgreSQL by default only listens on the local + interface (e.g. 127.0.0.1). However, Geo needs to communicate + between the **primary** and **secondary** nodes over a common network, such as a + corporate LAN or the public Internet. For this reason, we need to + configure PostgreSQL to listen on more interfaces. + + The `listen_address` option opens PostgreSQL up to external connections + with the interface corresponding to the given IP. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the + [PostgreSQL - Replication documentation][pg-docs-runtime-replication] for more information. + +1. Set the access control on the **primary** node to allow TCP connections using the + server's public IP and set the connection from the **secondary** node to require a + password. Edit `pg_hba.conf` (for Debian/Ubuntu that would be + `/etc/postgresql/9.x/main/pg_hba.conf`): + + ```sh + host all all <primary_node_ip>/32 md5 + host replication gitlab_replicator <secondary_node_ip>/32 md5 + ``` + + If you want to add another secondary, add one more row like the replication + one and change the IP address: + + ```sh + host all all <primary_node_ip>/32 md5 + host replication gitlab_replicator <secondary_node_ip>/32 md5 + host replication gitlab_replicator <another_secondary_node_ip>/32 md5 + ``` + +1. Restart PostgreSQL for the changes to take effect. + +1. Choose a database-friendly name to use for your secondary to use as the + replication slot name. For example, if your domain is + `secondary.geo.example.com`, you may use `secondary_example` as the slot + name. + +1. Create the replication slot on the **primary** node: + + ```sh + $ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');" + slot_name | xlog_position + ------------------+--------------- + secondary_example | + (1 row) + ``` + +1. Now that the PostgreSQL server is set up to accept remote connections, run + `netstat -plnt` to make sure that PostgreSQL is listening to the server's + public IP. + +### Step 2. Configure the secondary server + +Follow the first steps in ["configure the secondary server"][database-replication] and note that since you are installing from source, the username and +group listed as `gitlab-psql` in those steps should be replaced by `postgres` +instead. After completing the "Test that the `gitlab-psql` user can connect to +the **primary** node's database" step, continue here: + +1. Edit `postgresql.conf` to configure the secondary for streaming replication + (for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`): + + ```sh + wal_level = hot_standby + max_wal_senders = 5 + checkpoint_segments = 10 + wal_keep_segments = 10 + hot_standby = on + ``` + +1. Restart PostgreSQL for the changes to take effect. + +#### Enable tracking database on the secondary server + +Geo secondary nodes use a tracking database to keep track of replication status +and recover automatically from some replication issues. Follow the steps below to create +the tracking database. + +1. On the secondary node, run the following command to create `database_geo.yml` with the + information of your secondary PostgreSQL instance: + + ```sh + sudo cp /home/git/gitlab/config/database_geo.yml.postgresql /home/git/gitlab/config/database_geo.yml + ``` + +1. Edit the content of `database_geo.yml` in `production:` as in the example below: + + ```yaml + # + # PRODUCTION + # + production: + adapter: postgresql + encoding: unicode + database: gitlabhq_geo_production + pool: 10 + username: gitlab_geo + # password: + host: /var/opt/gitlab/geo-postgresql + ``` + +1. Create the database `gitlabhq_geo_production` on the PostgreSQL instance of the **secondary** node. + +1. Set up the Geo tracking database: + + ```sh + bundle exec rake geo:db:migrate + ``` + +1. Configure the [PostgreSQL FDW][FDW] connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```sh + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="/var/opt/gitlab/postgresql" # change to the public IP or VPC private IP if its an external server + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="/var/opt/gitlab/geo-postgresql" # change to the public IP or VPC private IP if its an external server + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + And edit the content of `database_geo.yml` and to add `fdw: true` to + the `production:` block. + +### Step 3. Initiate the replication process + +Below we provide a script that connects the database on the **secondary** node to +the database on the **primary** node, replicates the database, and creates the +needed files for streaming replication. + +The directories used are the defaults for Debian/Ubuntu. If you have changed +any defaults, configure it as you see fit replacing the directories and paths. + +CAUTION: **Warning:** +Make sure to run this on the **secondary** server as it removes all PostgreSQL's +data before running `pg_basebackup`. + +1. SSH into your GitLab **secondary** server and login as root: + + ```sh + sudo -i + ``` + +1. Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the + embedded paths if necessary: + + ``` + #!/bin/bash + + PORT="5432" + USER="gitlab_replicator" + echo --------------------------------------------------------------- + echo WARNING: Make sure this script is run from the secondary server + echo --------------------------------------------------------------- + echo + echo Enter the IP or FQDN of the primary PostgreSQL server + read HOST + echo Enter the password for $USER@$HOST + read -s PASSWORD + echo Enter the required sslmode + read SSLMODE + + echo Stopping PostgreSQL and all GitLab services + sudo service gitlab stop + sudo service postgresql stop + + echo Backing up postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ + + echo Cleaning up old cluster directory + sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data + + echo Starting base backup as the replicator user + echo Enter the password for $USER@$HOST + sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P + + echo Writing recovery.conf file + sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ + standby_mode = 'on' + primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' + _EOF1_ + " + + echo Restoring postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ + + echo Starting PostgreSQL + sudo service postgresql start + ``` + +1. Run it with: + + ```sh + bash /tmp/replica.sh + ``` + + When prompted, enter the IP/FQDN of the **primary** node, and the password you set up + for the `gitlab_replicator` user in the first step. + + You should use `verify-ca` for the `sslmode`. You can use `disable` if you + are happy to skip PostgreSQL TLS authentication altogether (e.g., you know + the network path is secure, or you are using a site-to-site VPN). This is + **not** safe over the public Internet! + + You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + +The replication process is now over. + +## PGBouncer support (optional) + +1. First, enter the PostgreSQL console as an admin user. + +1. Then create the read-only user: + + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password '<your_password_here>'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` + +1. Enter the PostgreSQL console on the **secondary** tracking database and change the user mapping to this new user: + + ``` + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET user 'gitlab_geo_fdw') + ``` + +## MySQL replication + +MySQL replication is not supported for Geo. + +## Troubleshooting + +Read the [troubleshooting document](troubleshooting.md). + +[replication-slots-article]: https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75 +[pgback]: http://www.postgresql.org/docs/9.6/static/app-pgbasebackup.html +[replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication +[FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html +[database]: database.md +[add-geo-node]: configuration.md#step-3-add-the-secondary-gitlab-node +[database-replication]: database.md#step-2-configure-the-secondary-server +[pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION +[pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html +[pg-docs-runtime-replication]: https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md new file mode 100644 index 00000000000..5b02b861c61 --- /dev/null +++ b/doc/administration/geo/replication/docker_registry.md @@ -0,0 +1,23 @@ +# Docker Registry for a secondary node **[PREMIUM ONLY]** + +You can set up a [Docker Registry] on your +**secondary** Geo node that mirrors the one on the **primary** Geo node. + +## Storage support + +CAUTION: **Warning:** +If you use [local storage][registry-storage] +for the Container Registry you **cannot** replicate it to a **secondary** node. + +Docker Registry currently supports a few types of storages. If you choose a +distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker +Registry on the **primary** node, you can use the same storage for a **secondary** +Docker Registry as well. For more information, read the +[Load balancing considerations][registry-load-balancing] +when deploying the Registry, and how to set up the storage driver for GitLab's +integrated [Container Registry][registry-storage]. + +[ee]: https://about.gitlab.com/pricing/ +[Docker Registry]: https://docs.docker.com/registry/ +[registry-storage]: ../../container_registry.md#container-registry-storage-driver +[registry-load-balancing]: https://docs.docker.com/registry/deploying/#load-balancing-considerations diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md new file mode 100644 index 00000000000..dae5ed911b0 --- /dev/null +++ b/doc/administration/geo/replication/external_database.md @@ -0,0 +1,219 @@ +# Geo with external PostgreSQL instances **[PREMIUM ONLY]** + +This document is relevant if you are using a PostgreSQL instance that is *not +managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or +manually installed and configured PostgreSQL instances. + +NOTE: **Note**: +We strongly recommend running Omnibus-managed instances as they are actively +developed and tested. We aim to be compatible with most external +(not managed by Omnibus) databases but we do not guarantee compatibility. + +## **Primary** node + +1. SSH into a GitLab **primary** application server and login as root: + + ```sh + sudo -i + ``` + +1. Execute the command below to define the node as **primary** node: + + ```sh + gitlab-ctl set-geo-primary-node + ``` + + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + + +### Configure the external database to be replicated + +To set up an external database, you can either: + +- Set up streaming replication yourself (for example, in AWS RDS). +- Perform the Omnibus configuration manually as follows. + +#### Leverage your cloud provider's tools to replicate the primary database + +Given you have a primary node set up on AWS EC2 that uses RDS. +You can now just create a read-only replica in a different region and the +replication process will be managed by AWS. Make sure you've set Network ACL, Subnet, and +Security Group according to your needs, so the secondary application node can access the database. +Skip to the [Configure secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica) section below. + +#### Manually configure the primary database for replication + +The [geo_primary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +configures the **primary** node's database to be replicated by making changes to +`pg_hba.conf` and `postgresql.conf`. Make the following configuration changes +manually to your external database configuration: + +``` +## +## Geo Primary Role +## - pg_hba.conf +## +host replication gitlab_replicator <trusted secondary IP>/32 md5 +``` + +``` +## +## Geo Primary Role +## - postgresql.conf +## +sql_replication_user = gitlab_replicator +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 50 +max_replication_slots = 1 # number of secondary instances +hot_standby = on +``` + +## **Secondary** nodes + +### Manually configure the replica database + +Make the following configuration changes manually to your `postgresql.conf` +of external replica database: + +``` +## +## Geo Secondary Role +## - postgresql.conf +## +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 10 +hot_standby = on +``` + +### Configure **secondary** application nodes to use the external read-replica + +With Omnibus, the +[geo_secondary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +has three main functions: + +1. Configure the replica database. +1. Configure the tracking database. +1. Enable the [Geo Log Cursor](index.md#geo-log-cursor) (not covered in this section). + +To configure the connection to the external read-replica database and enable Log Cursor: + +1. SSH into a GitLab **secondary** application server and login as root: + + ```bash + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the following + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + # note this is shared between both databases, + # make sure you define the same password in both + gitlab_rails['db_password'] = '<your_password_here>' + + gitlab_rails['db_username'] = 'gitlab' + gitlab_rails['db_host'] = '<database_read_replica_host>' + ``` +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + +### Configure the tracking database + +**Secondary** nodes use a separate PostgreSQL installation as a tracking +database to keep track of replication status and automatically recover from +potential replication issues. Omnibus automatically configures a tracking database +when `roles ['geo_secondary_role']` is set. For high availability, +refer to [Geo High Availability](https://docs.gitlab.com/ee/administration/high_availability). +If you want to run this database external to Omnibus, please follow the instructions below. + +The tracking database requires an [FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) +connection with the **secondary** replica database for improved performance. + +If you have an external database ready to be used as the tracking database, +follow the instructions below to use it: + +NOTE: **Note:** +If you want to use AWS RDS as a tracking database, make sure it has access to +the secondary database. Unfortunately, just assigning the same security group is not enough as +outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound +rule to the read-replica's security group allowing any TCP traffic from +the tracking database on port 5432. + +1. SSH into a GitLab **secondary** server and login as root: + + ```bash + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for + the machine with the PostgreSQL instance: + + ```ruby + geo_secondary['db_username'] = 'gitlab_geo' + geo_secondary['db_password'] = '<your_password_here>' + + geo_secondary['db_host'] = '<tracking_database_host>' + geo_secondary['db_port'] = <tracking_database_port> # change to the correct port + geo_secondary['db_fdw'] = true # enable FDW + geo_postgresql['enable'] = false # don't use internal managed instance + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + +1. Run the tracking database migrations: + + ```bash + gitlab-rake geo:db:create + gitlab-rake geo:db:migrate + ``` + +1. Configure the + [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```bash + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="<public_ip_or_vpc_private_ip>" + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PASS="<your_password_here>" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="<public_ip_or_vpc_private_ip>" + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, + but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using + `psql` for AWS RDS. + +1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) +1. Populate the FDW tables: + + ```bash + gitlab-rake geo:db:refresh_foreign_tables + ``` diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md new file mode 100644 index 00000000000..dd1af0dbf9c --- /dev/null +++ b/doc/administration/geo/replication/faq.md @@ -0,0 +1,62 @@ +# Geo Frequently Asked Questions **[PREMIUM ONLY]** + +## What are the minimum requirements to run Geo? + +The requirements are listed [on the index page](index.md#requirements-for-running-geo) + +## How does Geo know which projects to sync? + +On each **secondary** node, there is a read-only replicated copy of the GitLab database. +A **secondary** node also has a tracking database where it stores which projects have been synced. +Geo compares the two databases to find projects that are not yet tracked. + +At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. + +For each project to sync: + +1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node. +If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. +1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc. +1. Repeat until all projects are synced. + +When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. +The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced. + +To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when +the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. + +Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. +If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. +So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. + +## Can I use Geo in a disaster recovery situation? + +Yes, but there are limitations to what we replicate (see +[What data is replicated to a **secondary** node?](#what-data-is-replicated-to-a-secondary-node)). + +Read the documentation for [Disaster Recovery](../disaster_recovery/index.md). + +## What data is replicated to a **secondary** node? + +We currently replicate project repositories, LFS objects, generated +attachments / avatars and the whole database. This means user accounts, +issues, merge requests, groups, project data, etc., will be available for +query. + +## Can I git push to a **secondary** node? + +Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. + +## How long does it take to have a commit replicated to a **secondary** node? + +All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of +factors including the amount of traffic, how big your commit is, the +connectivity between your nodes, your hardware, etc. + +## What if the SSH server runs at a different port? + +That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to all **secondary** nodes. + +## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node? + +Yes. See [Docker Registry for a **secondary** node](docker_registry.md). diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md new file mode 100644 index 00000000000..715a83a9ff3 --- /dev/null +++ b/doc/administration/geo/replication/high_availability.md @@ -0,0 +1,295 @@ +# Geo High Availability **[PREMIUM ONLY]** + +This document describes a minimal reference architecture for running Geo +in a high availability configuration. If your HA setup differs from the one +described, it is possible to adapt these instructions to your needs. + +## Architecture overview + + + +_[diagram source - gitlab employees only][diagram-source]_ + +The topology above assumes that the **primary** and **secondary** Geo clusters +are located in two separate locations, on their own virtual network +with private IP addresses. The network is configured such that all machines within +one geographic location can communicate with each other using their private IP addresses. +The IP addresses given are examples and may be different depending on the +network topology of your deployment. + +The only external way to access the two Geo deployments is by HTTPS at +`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. + +NOTE: **Note:** +The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. + +## Redis and PostgreSQL High Availability + +The **primary** and **secondary** Redis and PostgreSQL should be configured +for high availability. Because of the additional complexity involved +in setting up this configuration for PostgreSQL and Redis, +it is not covered by this Geo HA documentation. + +For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for +[PostgreSQL](../../high_availability/database.md) and +[Redis](../../high_availability/redis.md), respectively. + +NOTE: **Note:** +It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. + +## Prerequisites: A working GitLab HA cluster + +This cluster will serve as the **primary** node. Use the +[GitLab HA documentation](../../high_availability/README.md) to set this up. + +## Configure the GitLab cluster to be the **primary** node + +The following steps enable a GitLab cluster to serve as the **primary** node. + +### Step 1: Configure the **primary** frontend servers + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` + +After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the +application servers, and connections from the application servers to those +services on the backend servers configured, during normal GitLab HA set up. See +high availability configuration documentation for +[PostgreSQL](https://docs.gitlab.com/ee/administration/high_availability/database.html#configuring-the-application-nodes) +and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). + +The **primary** database will require modification later, as part of +[step 2](#step-2-configure-the-main-read-only-replica-postgresql-database-on-the-secondary-node). + +## Configure a **secondary** node + +A **secondary** cluster is similar to any other GitLab HA cluster, with two +major differences: + +* The main PostgreSQL database is a read-only replica of the **primary** node's + PostgreSQL database. +* There is also a single PostgreSQL database for the **secondary** cluster, + called the "tracking database", which tracks the synchronization state of + various resources. + +Therefore, we will set up the HA components one-by-one, and include deviations +from the normal HA setup. + +### Step 1: Configure the Redis and NFS services on the **secondary** node + +Configure the following services, again using the non-Geo high availability +documentation: + +* [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high + availability. +* [NFS](../../high_availability/nfs.md) which will store data that is + synchronized from the **primary** node. + +### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node + +NOTE: **Note:** The following documentation assumes the database will be run on +a single node only, rather than as a PostgreSQL cluster. + +Configure the [**secondary** database](database.md) as a read-only replica of +the **primary** database. Use the following as a guide. + +1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the + following: + + ```ruby + ## + ## Configure the PostgreSQL role + ## + roles ['postgres_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## When running the Geo tracking database on a separate machine, disable it + ## here and allow connections from the tracking database host. And ensure + ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. + ## + geo_postgresql['enable'] = false + ``` + +After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 3: Configure the tracking database on the **secondary** node + +NOTE: **Note:** This documentation assumes the tracking database will be run on +only a single machine, rather than as a PostgreSQL cluster. + +Configure the tracking database. + +1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['listen_address'] = '<ip_address_of_this_host>' + geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' + + ## + ## Configure FDW connection to the replica database + ## + geo_secondary['db_fdw'] = true + geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' + geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] + gitlab_rails['db_host'] = '<replica_database_ip>' + + # Prevent reconfigure from attempting to run migrations on the replica DB + gitlab_rails['auto_migrate'] = false + + ## + ## Disable all other services that aren't needed, since we don't have a role + ## that does this. + ## + alertmanager['enable'] = false + consul['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + ``` + +After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 4: Configure the frontend application servers on the **secondary** node + +In the architecture overview, there are two machines running the GitLab +application services. These services are enabled selectively in the +configuration. + +Configure the application servers following +[Configuring GitLab for HA](../../high_availability/gitlab.md), then make the +following modifications: + +1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** + cluster, and add the following: + + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '<geo_tracking_db_host>' + geo_secondary['db_password'] = '<geo_tracking_db_password>' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '<replica_database_host>' + gitlab_rails['db_password'] = '<replica_database_password>' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '<redis_host>' + gitlab_rails['redis_password'] = '<redis_password>' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +NOTE: **Note:** +If you had set up PostgreSQL cluster using the omnibus package and you had set +up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` +mentioned above contains the plaintext passwords. This is used to let the Rails +servers connect to the databases. + +NOTE: **Note:** +Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. + +After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +On the secondary the following GitLab frontend services will be enabled: + +* geo-logcursor +* gitlab-pages +* gitlab-workhorse +* logrotate +* nginx +* registry +* remote-syslog +* sidekiq +* unicorn + +Verify these services by running `sudo gitlab-ctl status` on the frontend +application servers. + +### Step 5: Set up the LoadBalancer for the **secondary** node + +In this topology, a load balancer is required at each geographic location to +route traffic to the application servers. + +See [Load Balancer for GitLab HA](../../high_availability/load_balancer.md) for +more information. + +[diagram-source]: https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit +[gitlab-reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/geo/replication/img/geo_architecture.png b/doc/administration/geo/replication/img/geo_architecture.png Binary files differnew file mode 100644 index 00000000000..d318cd5d0f4 --- /dev/null +++ b/doc/administration/geo/replication/img/geo_architecture.png diff --git a/doc/administration/geo/replication/img/geo_node_dashboard.png b/doc/administration/geo/replication/img/geo_node_dashboard.png Binary files differnew file mode 100644 index 00000000000..99792d0770d --- /dev/null +++ b/doc/administration/geo/replication/img/geo_node_dashboard.png diff --git a/doc/administration/geo/replication/img/geo_node_healthcheck.png b/doc/administration/geo/replication/img/geo_node_healthcheck.png Binary files differnew file mode 100644 index 00000000000..33a31f7ab49 --- /dev/null +++ b/doc/administration/geo/replication/img/geo_node_healthcheck.png diff --git a/doc/administration/geo/replication/img/geo_overview.png b/doc/administration/geo/replication/img/geo_overview.png Binary files differnew file mode 100644 index 00000000000..01c1615212c --- /dev/null +++ b/doc/administration/geo/replication/img/geo_overview.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md new file mode 100644 index 00000000000..6abea2cf271 --- /dev/null +++ b/doc/administration/geo/replication/index.md @@ -0,0 +1,309 @@ +# Geo Replication **[PREMIUM ONLY]** + +Geo is the solution for widely distributed development teams. + +## Overview + +Fetching large repositories can take a long time for teams located far from a single GitLab instance. + +Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development. + +> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed). +> - Introduced in GitLab Enterprise Edition 8.9. +> - We recommend you use: +> - At least GitLab Enterprise Edition 10.0 for basic Geo features. +> - The latest version for a better experience. +> - Make sure that all nodes run the same GitLab version. +> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md). +> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + +For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). + +CAUTION: **Caution:** +Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. + +To make sure you're using the right version of the documentation, navigate to [the source version of this page on GitLab.com](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/administration/geo/replication/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v11.2.3-ee`](https://gitlab.com/gitlab-org/gitlab-ee/blob/v11.2.3-ee/doc/administration/geo/replication/index.md). + +## Use cases + +Implementing Geo provides the following benefits: + +- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects. +- Enable all of your developers to contribute ideas and work in parallel, no matter where they are. +- Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to a **secondary** node. + +In addition, it: + +- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)). +- Overcomes slow connections between distant offices, saving time by improving speed for distributed teams. +- Helps reducing the loading time for automated tasks, custom integrations, and internal workflows. +- Can quickly fail over to a **secondary** node in a [disaster recovery](../disaster_recovery/index.md) scenario. +- Allows [planned failover](../disaster_recovery/planned_failover.md) to a **secondary** node. + +Geo provides: + +- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams. +- Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. +- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. + +## How it works + +Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. + + + +When Geo is enabled, the: + +- Original instance is known as the **primary** node. +- Replicated read-only nodes are known as **secondary** nodes. + +Keep in mind that: + +- **Secondary** nodes talk to the **primary** node to: + - Get user data for logins (API). + - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). +- Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). +- Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. +- There are [limitations](#current-limitations) in the current implementation. + +### Architecture + +The following diagram illustrates the underlying architecture of Geo. + + + +In this diagram: + +- There is the **primary** node and the details of one **secondary** node. +- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database + updates via PostgreSQL streaming replication. +- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. +- A **secondary** node performs different type of synchronizations against the **primary** node, using a special + authorization protected by JWT: + - Repositories are cloned/updated via Git over HTTPS. + - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. + +From the perspective of a user performing Git operations: + +- The **primary** node behaves as a full read-write GitLab instance. +- **Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves. + +To simplify the diagram, some necessary components are omitted. Note that: + +- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. +- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). + +Note that a **secondary** node needs two different PostgreSQL databases: + +- A read-only database instance that streams data from the main GitLab database. +- [Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated. + +In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor). + +## Requirements for running Geo + +The following are required to run Geo: + +- An operating system that supports OpenSSH 6.9+ (needed for + [fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md)) + The following operating systems are known to ship with a current version of OpenSSH: + - [CentOS](https://www.centos.org) 7.4+ + - [Ubuntu](https://www.ubuntu.com) 16.04+ +- PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- Git 2.9+ + +### Firewall rules + +The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo. + +| **Primary** node | **Secondary** node | Protocol | +|:-----------------|:-------------------|:-------------| +| 80 | 80 | HTTP | +| 443 | 443 | TCP or HTTPS | +| 22 | 22 | TCP | +| 5432 | | PostgreSQL | + +See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) + +NOTE: **Note:** +[Web terminal](../../../ci/environments.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. +When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../../integration/terminal.md) integration guide for more details. + +NOTE: **Note:** +When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. +If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. + +### LDAP + +We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. + +NOTE: **Note:** +It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node. + +Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). + +### Geo Tracking Database + +The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example: + +- Download new assets. +- Fetch new LFS Objects. +- Fetch changes from a repository that has recently been updated. + +Because the replicated database instance is read-only, we need this additional database instance for each **secondary** node. +The tracking database requires the `postgres_fdw` extension. + +### Geo Log Cursor + +This daemon: + +- Reads a log of events replicated by the **primary** node to the **secondary** database instance. +- Updates the Geo Tracking Database instance with changes that need to be executed. + +When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state. + +This new architecture allows GitLab to be resilient to connectivity issues between the nodes. It doesn't matter how long the **secondary** node is disconnected from the **primary** node as it will be able to replay all the events in the correct order and become synchronized with the **primary** node again. + +## Setup instructions + +These instructions assume you have a working instance of GitLab. They guide you through: + +1. Making your existing instance the **primary** node. +1. Adding **secondary** nodes. + +CAUTION: **Caution:** +The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.** + +### Using Omnibus GitLab + +If you installed GitLab using the Omnibus packages (highly recommended): + +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/installation/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. +1. [Upload the GitLab License](https://docs.gitlab.com/ee/user/admin_area/license.html) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). +1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes. +1. [Configure GitLab](configuration.md) to set the **primary** and **secondary** nodes. +1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap). +1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). + +### Using GitLab installed from source (Deprecated) + +NOTE: **Note:** +In GitLab 11.5, support for using Geo in GitLab source installations was deprecated and will be removed in a future release. Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). + +If you installed GitLab from source: + +1. [Install GitLab Enterprise Edition](../../../install/installation.md) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. +1. [Upload the GitLab License](https://docs.gitlab.com/ee/user/admin_area/license.html) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Set up the database replication](database_source.md) (`primary (read-write) <-> secondary (read-only)` topology). +1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). Do this step for **both** **primary** and **secondary** nodes. +1. [Configure GitLab](configuration_source.md) to set the **primary** and **secondary** nodes. +1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). + +## Post-installation documentation + +After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information. + +### Configuring Geo + +For information on configuring Geo, see: + +- [Geo configuration (GitLab Omnibus)](configuration.md). +- [Geo configuration (source)](configuration_source.md). Configuring Geo in GitLab source installations was **deprecated** in GitLab 11.5. + +### Updating Geo + +For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md). + +### Configuring Geo high availability + +For information on configuring Geo for high availability, see [Geo High Availability](high_availability.md). + +### Configuring Geo with Object Storage + +For information on configuring Geo with object storage, see [Geo with Object storage](object_storage.md). + +### Disaster Recovery + +For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](../disaster_recovery/index.md). + +### Replicating the Container Registry + +For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** node](docker_registry.md). + +### Security Review + +For more information on Geo security, see [Geo security review](security_review.md). + +### Tuning Geo + +For more information on tuning Geo, see [Tuning Geo](tuning.md). + +## Remove Geo node + +For more information on removing a Geo node, see [Removing **secondary** Geo nodes](remove_geo_node.md). + +## Current limitations + +CAUTION: **Caution:** +This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. + +- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab-ee/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. +- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. +- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [gitlab-org/omnibus-gitlab#2978](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978) for details. +- Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. +- [Selective synchronization](configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. +- Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. +- [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. + +### Limitations on replication + +Only the following items are replicated to the **secondary** node: + +- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata. +- Project repositories. +- Project wiki repositories. +- User uploads. For example, attachments to issues, merge requests, epics, and avatars. +- CI job artifacts and traces. + +DANGER: **DANGER** +Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**. + +### Examples of data not replicated + +Take special note that these examples of GitLab features are both: + +- Commonly used. +- **Not** replicated by Geo at present. + +Examples include: + +- [Elasticsearch integration](https://docs.gitlab.com/ee/integration/elasticsearch.html). +- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this. +- [GitLab Pages](../../pages/index.md). +- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/). + +CAUTION: **Caution:** +If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means. + +## Frequently Asked Questions + +For answers to common questions, see the [Geo FAQ](faq.md). + +## Log files + +Since GitLab 9.5, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. + +This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into Elasticsearch, Splunk, etc. + +For example: + +```json +{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} +``` + +This message shows that Geo detected that a repository update was needed for project `1`. + +## Troubleshooting + +For troubleshooting steps, see [Geo Troubleshooting](troubleshooting.md). diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md new file mode 100644 index 00000000000..c3c11dbaf1e --- /dev/null +++ b/doc/administration/geo/replication/object_storage.md @@ -0,0 +1,43 @@ +# Geo with Object storage **[PREMIUM ONLY]** + +Geo can be used in combination with Object Storage (AWS S3, or +other compatible object storage). + +## Configuration + +At this time it is required that if object storage is enabled on the +**primary** node, it must also be enabled on each **secondary** node. + +**Secondary** nodes can use the same storage bucket as the **primary** node, or +they can use a replicated storage bucket. At this time GitLab does not +take care of content replication in object storage. + +For LFS, follow the documentation to +[set up LFS object storage](../../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). + +For CI job artifacts, there is similar documentation to configure +[jobs artifact object storage](../../job_artifacts.md#using-object-storage) + +For user uploads, there is similar documentation to configure [upload object storage](../../uploads.md#using-object-storage-core-only) + +You should enable and configure object storage on both **primary** and **secondary** +nodes. Migrating existing data to object storage should be performed on the +**primary** node only. **Secondary** nodes will automatically notice that the migrated +files are now in object storage. + +## Replication + +When using Amazon S3, you can use +[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to +have automatic replication between the bucket used by the **primary** node and +the bucket used by **secondary** nodes. + +If you are using Google Cloud Storage, consider using +[Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional). +Or you can use the [Storage Transfer Service](https://cloud.google.com/storage/transfer/), +although this only supports daily synchronization. + +For manual synchronization, or scheduled by `cron`, please have a look at: + +- [`s3cmd sync`](http://s3tools.org/s3cmd-sync) +- [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync) diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md new file mode 100644 index 00000000000..b190fe7d42d --- /dev/null +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -0,0 +1,50 @@ +# Removing secondary Geo nodes **[PREMIUM ONLY]** + +**Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: + +1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`). +1. Click the **Remove** button for the **secondary** node you want to remove. +1. Confirm by clicking **Remove** when the prompt appears. + +Once removed from the Geo admin page, you must stop and uninstall the **secondary** node: + +1. On the **secondary** node, stop GitLab: + + ```bash + sudo gitlab-ctl stop + ``` +1. On the **secondary** node, uninstall GitLab: + + ```bash + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall + + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee + + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` + +Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: + +1. On the **primary** node, start a PostgreSQL console session: + + ```bash + sudo gitlab-psql + ``` + + NOTE: **Note:** + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +1. Remove the replication slot for the **secondary** node: + + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md new file mode 100644 index 00000000000..46d3e68ab63 --- /dev/null +++ b/doc/administration/geo/replication/security_review.md @@ -0,0 +1,286 @@ +# Geo security review (Q&A) **[PREMIUM ONLY]** + +The following security review of the Geo feature set focuses on security +aspects of the feature as they apply to customers running their own GitLab +instances. The review questions are based in part on the [application security architecture](https://www.owasp.org/index.php/Application_Security_Architecture_Cheat_Sheet) +questions from [owasp.org](https://www.owasp.org). + +## Business Model + +### What geographic areas does the application service? + +- This varies by customer. Geo allows customers to deploy to multiple areas, + and they get to choose where they are. +- Region and node selection is entirely manual. + +## Data Essentials + +### What data does the application receive, produce, and process? + +- Geo streams almost all data held by a GitLab instance between sites. This + includes full database replication, most files (user-uploaded attachments, + etc) and repository + wiki data. In a typical configuration, this will + happen across the public Internet, and be TLS-encrypted. +- PostgreSQL replication is TLS-encrypted. +- See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2948) + +### How can the data be classified into categories according to its sensitivity? + +- GitLab’s model of sensitivity is centered around public vs. internal vs. + private projects. Geo replicates them all indiscriminately. “Selective sync” + exists for files and repositories (but not database content), which would permit + only less-sensitive projects to be replicated to a **secondary** node if desired. +- See also: [developing a data classification policy](https://gitlab.com/gitlab-com/security/issues/4). + +### What data backup and retention requirements have been defined for the application? + +- Geo is designed to provide replication of a certain subset of the application + data. It is part of the solution, rather than part of the problem. + +## End-Users + +### Who are the application's end‐users? + +- **Secondary** nodes are created in regions that are distant (in terms of + Internet latency) from the main GitLab installation (the **primary** node). They are + intended to be used by anyone who would ordinarily use the **primary** node, who finds + that the **secondary** node is closer to them (in terms of Internet latency). + +### How do the end‐users interact with the application? + +- **Secondary** nodes provide all the interfaces a **primary** node does + (notably a HTTP/HTTPS web application, and HTTP/HTTPS or SSH git repository + access), but is constrained to read-only activities. The principal use case is + envisioned to be cloning git repositories from the **secondary** node in favor of the + **primary** node, but end-users may use the GitLab web interface to view projects, + issues, merge requests, snippets, etc. + +### What security expectations do the end‐users have? + +- The replication process must be secure. It would typically be unacceptable to + transmit the entire database contents or all files and repositories across the + public Internet in plaintext, for instance. +- **Secondary** nodes must have the same access controls over its content as the + **primary** node - unauthenticated users must not be able to gain access to privileged + information on the **primary** node by querying the **secondary** node. +- Attackers must not be able to impersonate the **secondary** node to the **primary** node, and + thus gain access to privileged information. + +## Administrators + +### Who has administrative capabilities in the application? + +- Nothing Geo-specific. Any user where `admin: true` is set in the database is + considered an admin with super-user privileges. +- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-ce/issues/32730) + (not geo-specific) +- Much of Geo’s integration (database replication, for instance) must be + configured with the application, typically by system administrators. + +### What administrative capabilities does the application offer? + +- **Secondary** nodes may be added, modified, or removed by users with + administrative access. +- The replication process may be controlled (start/stop) via the Sidekiq + administrative controls. + +## Network + +### What details regarding routing, switching, firewalling, and load‐balancing have been defined? + +- Geo requires the **primary** node and **secondary** node to be able to communicate with each + other across a TCP/IP network. In particular, the **secondary** nodes must be able to + access HTTP/HTTPS and PostgreSQL services on the **primary** node. + +### What core network devices support the application? + +- Varies from customer to customer. + +### What network performance requirements exist? + +- Maximum replication speeds between **primary** node and **secondary** node is limited by the + available bandwidth between sites. No hard requirements exist - time to complete + replication (and ability to keep up with changes on the **primary** node) is a function + of the size of the data set, tolerance for latency, and available network + capacity. + +### What private and public network links support the application? + +- Customers choose their own networks. As sites are intended to be + geographically separated, it is envisioned that replication traffic will pass + over the public Internet in a typical deployment, but this is not a requirement. + +## Systems + +### What operating systems support the application? + +- Geo imposes no additional restrictions on operating system (see the + [GitLab installation](https://about.gitlab.com/installation/) page for more + details), however we recommend using the operating systems listed in the [Geo documentation](index.md#requirements-for-running-geo). + +### What details regarding required OS components and lock‐down needs have been defined? + +- The recommended installation method (Omnibus) packages most components itself. + A from-source installation method exists. Both are documented at + <https://docs.gitlab.com/ee/administration/geo/replication/index.html> +- There are significant dependencies on the system-installed OpenSSH daemon (Geo + requires users to set up custom authentication methods) and the omnibus or + system-provided PostgreSQL daemon (it must be configured to listen on TCP, + additional users and replication slots must be added, etc). +- The process for dealing with security updates (for example, if there is a + significant vulnerability in OpenSSH or other services, and the customer + wants to patch those services on the OS) is identical to the non-Geo + situation: security updates to OpenSSH would be provided to the user via the + usual distribution channels. Geo introduces no delay there. + +## Infrastructure Monitoring + +### What network and system performance monitoring requirements have been defined? + +- None specific to Geo. + +### What mechanisms exist to detect malicious code or compromised application components? + +- None specific to Geo. + +### What network and system security monitoring requirements have been defined? + +- None specific to Geo. + +## Virtualization and Externalization + +### What aspects of the application lend themselves to virtualization? + +- All. + +## What virtualization requirements have been defined for the application? + +- Nothing Geo-specific, but everything in GitLab needs to have full + functionality in such an environment. + +### What aspects of the product may or may not be hosted via the cloud computing model? + +- GitLab is “cloud native” and this applies to Geo as much as to the rest of the + product. Deployment in clouds is a common and supported scenario. + +## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)? + +- To be decided by our customers, according to their operational needs. + +## Environment + +### What frameworks and programming languages have been used to create the application? + +- Ruby on Rails, Ruby. + +### What process, code, or infrastructure dependencies have been defined for the application? + +- Nothing specific to Geo. + +### What databases and application servers support the application? + +- PostgreSQL >= 9.6, Redis, Sidekiq, Unicorn. + +### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? + +- There are some Geo-specific values. Some are shared secrets which must be + securely transmitted from the **primary** node to the **secondary** node at setup time. Our + documentation recommends transmitting them from the **primary** node to the system + administrator via SSH, and then back out to the **secondary** node in the same manner. + In particular, this includes the PostgreSQL replication credentials and a secret + key (`db_key_base`) which is used to decrypt certain columns in the database. + The `db_key_base` secret is stored unencrypted on the filesystem, in + `/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is + no at-rest protection for them. + +## Data Processing + +### What data entry paths does the application support? + +- Data is entered via the web application exposed by GitLab itself. Some data is + also entered using system administration commands on the GitLab servers (e.g., + `gitlab-ctl set-primary-node`). +- **Secondary** nodes also receive inputs via PostgreSQL streaming replication from the **primary** node. + +### What data output paths does the application support? + +- **Primary** nodes output via PostgreSQL streaming replication to the **secondary** node. + Otherwise, principally via the web application exposed by GitLab itself, and via + SSH `git clone` operations initiated by the end-user. + +### How does data flow across the application's internal components? + +- **Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web + tokens) and via PostgreSQL streaming replication. +- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database + (including Geo tracking database on **secondary** node). The various internal components + are orchestrated to make alterations to these stores. + +### What data input validation requirements have been defined? + +- **Secondary** nodes must have a faithful replication of the **primary** node’s data. + +### What data does the application store and how? + +- Git repositories and files, tracking information related to the them, and the GitLab database contents. + +### What data is or may need to be encrypted and what key management requirements have been defined? + +- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at + rest. A subset of database columns are encrypted at rest using the `db_otp_key`. +- A static secret shared across all hosts in a GitLab deployment. +- In transit, data should be encrypted, although the application does permit + communication to proceed unencrypted. The two main transits are the **secondary** node’s + replication process for PostgreSQL, and for git repositories/files. Both should + be protected using TLS, with the keys for that managed via Omnibus per existing + configuration for end-user access to GitLab. + +### What capabilities exist to detect the leakage of sensitive data? + +- Comprehensive system logs exist, tracking every connection to GitLab and PostgreSQL. + +### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:? + +- Data must have the option to be encrypted in transit, and be secure against + both passive and active attack (e.g., MITM attacks should not be possible). + +## Access + +### What user privilege levels does the application support? + +- Geo adds one type of privilege: **secondary** nodes can access a special Geo API to + download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS. + +### What user identification and authentication requirements have been defined? + +- **Secondary** nodes identify to Geo **primary** nodes via OAuth or JWT authentication + based on the shared database (HTTP access) or a PostgreSQL replication user (for + database replication). The database replication also requires IP-based access + controls to be defined. + +### What user authorization requirements have been defined? + +- **Secondary** nodes must only be able to *read* data. They are not currently able to mutate data on the **primary** node. + +### What session management requirements have been defined? + +- Geo JWTs are defined to last for only two minutes before needing to be regenerated. +- Geo JWTs are generated for one of the following specific scopes: + - Geo API access. + - Git access. + - LFS and File ID. + - Upload and File ID. + - Job Artifact and File ID. + +### What access requirements have been defined for URI and Service calls? + +- **Secondary** nodes make many calls to the **primary** node's API. This is how file + replication proceeds, for instance. This endpoint is only accessible with a JWT token. +- The **primary** node also makes calls to the **secondary** node to get status information. + +## Application Monitoring + +### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured? + +- Structured JSON log is written to the filesystem, and can also be ingested + into a Kibana installation for further analysis. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md new file mode 100644 index 00000000000..9c95720487d --- /dev/null +++ b/doc/administration/geo/replication/troubleshooting.md @@ -0,0 +1,457 @@ +# Geo Troubleshooting **[PREMIUM ONLY]** + +NOTE: **Note:** +This list is an attempt to document all the moving parts that can go wrong. +We are working into getting all this steps verified automatically in a +rake task in the future. + +Setting up Geo requires careful attention to details and sometimes it's easy to +miss a step. Here is a list of questions you should ask to try to detect +what you need to fix (all commands and path locations are for Omnibus installs): + +## First check the health of the **secondary** node + +Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in +your browser. We perform the following health checks on each **secondary** node +to help identify if something is wrong: + +- Is the node running? +- Is the node's secondary database configured for streaming replication? +- Is the node's secondary tracking database configured? +- Is the node's secondary tracking database connected? +- Is the node's secondary tracking database up-to-date? + + + +If the UI is not working, or you are unable to log in, you can run the Geo +health check manually to get this information as well as a few more details. +This rake task can be run on an app node in the **primary** or **secondary** +Geo nodes: + +```sh +sudo gitlab-rake gitlab:geo:check +``` + +Example output: + +``` +Checking Geo ... + +GitLab Geo is available ... yes +GitLab Geo is enabled ... yes +GitLab Geo secondary database is correctly configured ... yes +Using database streaming replication? ... yes +GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes +GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... yes +GitLab Geo HTTP(S) connectivity ... +* Can connect to the primary node ... yes +HTTP/HTTPS repository cloning is enabled ... yes +Machine clock is synchronized ... yes +Git user has default SSH configuration? ... yes +OpenSSH configured to use AuthorizedKeysCommand ... yes +GitLab configured to disable writing to authorized_keys file ... yes +GitLab configured to store new projects in hashed storage? ... yes +All projects are in hashed storage? ... yes + +Checking Geo ... Finished +``` + +Current sync information can be found manually by running this rake task on any +**secondary** app node: + +```sh +sudo gitlab-rake geo:status +``` + +Example output: + +``` +http://secondary.example.com/ +----------------------------------------------------- + GitLab Version: 11.8.1-ee + Geo Role: Secondary + Health Status: Healthy + Repositories: 190/190 (100%) + Verified Repositories: 190/190 (100%) + Wikis: 190/190 (100%) + Verified Wikis: 190/190 (100%) + LFS Objects: 35/35 (100%) + Attachments: 528/528 (100%) + CI job artifacts: 477/477 (100%) + Repositories Checked: 0/190 (0%) + Sync Settings: Full + Database replication lag: 0 seconds + Last event ID seen from primary: 2158 (about 2 minute ago) + Last event ID processed by cursor: 2158 (about 2 minute ago) + Last status report was: 4 minutes ago +``` + +## Is Postgres replication working? + +### Are my nodes pointing to the correct database instance? + +You should make sure your **primary** Geo node points to the instance with +writing permissions. + +Any **secondary** nodes should point only to read-only instances. + +### Can Geo detect my current node correctly? + +Geo uses the defined node from the **Admin Area > Geo** screen, and tries to match +it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file. +The relevant line looks like: `external_url "http://gitlab.example.com"`. + +To check if the node on the current machine is correctly detected type: + +```sh +sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" +``` + +and expect something like: + +``` +#<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...> +``` + +By running the command above, `primary` should be `true` when executed in +the **primary** node, and `false` on any **secondary** node. + +## How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"? + +This means that the `max_replication_slots` PostgreSQL variable needs to +be set on the **primary** database. In GitLab 9.4, we have made this setting +default to 1. You may need to increase this value if you have more +**secondary** nodes. Be sure to restart PostgreSQL for this to take +effect. See the [PostgreSQL replication +setup][database-pg-replication] guide for more details. + +## How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? + +This occurs when PostgreSQL does not have a replication slot for the +**secondary** node by that name. You may want to rerun the [replication +process](database.md) on the **secondary** node . + +## How do I fix the message, "Command exceeded allowed execution time" when setting up replication? + +This may happen while [initiating the replication process][database-start-replication] on the **secondary** node, +and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). + +Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for +`--backup-timeout`: + +```sh +sudo gitlab-ctl \ + replicate-geo-database \ + --host=<primary_node_hostname> \ + --slot-name=<secondary_slot_name> \ + --backup-timeout=21600 +``` + +This will give the initial replication up to six hours to complete, rather than +the default thirty minutes. Adjust as required for your installation. + +## How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" + +Determine if you have any unused replication slots in the **primary** database. This can cause large amounts of +log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`. + +1. Start a PostgreSQL console session: + + ```sh + sudo gitlab-psql gitlabhq_production + ``` + + > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. View your replication slots with: + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +Slots where `active` is `f` are not active. + +- When this slot should be active, because you have a **secondary** node configured using that slot, + log in to that **secondary** node and check the PostgreSQL logs why the replication is not running. + +- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the + PostgreSQL console session: + + ```sql + SELECT pg_drop_replication_slot('<name_of_extra_slot>'); + ``` + +## Very large repositories never successfully synchronize on the **secondary** node + +GitLab places a timeout on all repository clones, including project imports +and Geo synchronization operations. If a fresh `git clone` of a repository +on the primary takes more than a few minutes, you may be affected by this. +To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` +on the **secondary** node: + +```ruby +gitlab_rails['gitlab_shell_git_timeout'] = 10800 +``` + +Then reconfigure GitLab: + +```sh +sudo gitlab-ctl reconfigure +``` + +This will increase the timeout to three hours (10800 seconds). Choose a time +long enough to accommodate a full clone of your largest repositories. + +## How to reset Geo **secondary** node replication + +If you get a **secondary** node in a broken state and want to reset the replication state, +to start again from scratch, there are a few steps that can help you: + +1. Stop Sidekiq and the Geo LogCursor + + It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and + wait until the current jobs to finish processing. + + You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** + when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. + + ```sh + gitlab-ctl status sidekiq + # run: sidekiq: (pid 10180) <- this is the PID you will use + kill -TSTP 10180 # change to the correct PID + + gitlab-ctl stop sidekiq + gitlab-ctl stop geo-logcursor + ``` + + You can watch sidekiq logs to know when sidekiq jobs processing have finished: + + ```sh + gitlab-ctl tail sidekiq + ``` + +1. Rename repository storage folders and create new ones + + ```sh + mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old + mkdir -p /var/opt/gitlab/git-data/repositories + chown git:git /var/opt/gitlab/git-data/repositories + ``` + + TIP: **Tip** + You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future + as soon as you confirmed that you don't need it anymore, to save disk space. + +1. _(Optional)_ Rename other data folders and create new ones + + CAUTION: **Caution**: + You may still have files on the **secondary** node that have been removed from **primary** node but + removal have not been reflected. If you skip this step, they will never be removed + from this Geo node. + + Any uploaded content like file attachments, avatars or LFS objects are stored in a + subfolder in one of the two paths below: + + 1. /var/opt/gitlab/gitlab-rails/shared + 1. /var/opt/gitlab/gitlab-rails/uploads + + To rename all of them: + + ```sh + gitlab-ctl stop + + mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old + mkdir -p /var/opt/gitlab/gitlab-rails/shared + + mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old + mkdir -p /var/opt/gitlab/gitlab-rails/uploads + ``` + + Reconfigure in order to recreate the folders and make sure permissions and ownership + are correctly + + ```sh + gitlab-ctl reconfigure + ``` + +1. Reset the Tracking Database + + ```sh + gitlab-rake geo:db:reset + ``` + +1. Restart previously stopped services + + ```sh + gitlab-ctl start + ``` + +## How do I fix a "Foreign Data Wrapper (FDW) is not configured" error? + +When setting up Geo, you might see this warning in the `gitlab-rake +gitlab:geo:check` output: + +``` +GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured +``` + +There are a few key points to remember: + +1. The FDW settings are configured on the Geo **tracking** database. +1. The configured foreign server enables a login to the Geo +**secondary**, read-only database. + +By default, the Geo secondary and tracking database are running on the +same host on different ports. That is, 5432 and 5431 respectively. + +### Checking configuration + +NOTE: **Note:** +The following steps are for Omnibus installs only. Using Geo with source-based installs [is deprecated](index.md#using-gitlab-installed-from-source-deprecated). + +To check the configuration: + +1. Enter the database console: + + ```sh + gitlab-geo-psql + ``` + +1. Check whether any tables are present. If everything is working, you +should see something like this: + + ```sql + gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; + foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n + ame + -------------------------+----------------------+-------------------------------------------------+-------------------------+----------------- + ---- + gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary + <snip> + ``` + + However, if the query returns with `0 rows`, then continue onto the next steps. + +1. Check that the foreign server mapping is correct via `\des+`. The + results should look something like this: + + ```sql + gitlabhq_geo_production=# \des+ + List of foreign servers + -[ RECORD 1 ]--------+------------------------------------------------------------ + Name | gitlab_secondary + Owner | gitlab-psql + Foreign-data wrapper | postgres_fdw + Access privileges | "gitlab-psql"=U/"gitlab-psql" + + | gitlab_geo=U/"gitlab-psql" + Type | + Version | + FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') + Description | + ``` + + NOTE: **Note:** Pay particular attention to the host and port under + FDW options. That configuration should point to the Geo secondary + database. + + If you need to experiment with changing the host or password, the + following queries demonstrate how: + + ```sql + ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); + ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); + ``` + + If you change the host and/or port, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_host']` + - `gitlab_rails['db_port']` + +1. Check that the user mapping is configured properly via `\deu+`: + + ```sql + gitlabhq_geo_production=# \deu+ + List of user mappings + Server | User name | FDW Options + ------------------+------------+-------------------------------------------------------------------------------- + gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') + (1 row) + ``` + + Make sure the password is correct. You can test that logins work by running `psql`: + + ```sh + # Connect to the tracking database as the `gitlab_geo` user + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` + + If you need to correct the password, the following query shows how: + + ```sql + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); + ``` + + If you change the user or password, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_username']` + - `gitlab_rails['db_password']` + + If you are using [PgBouncer in front of the secondary + database](database.md#pgbouncer-support-optional), be sure to update + the following settings: + + - `geo_postgresql['fdw_external_user']` + - `geo_postgresql['fdw_external_password']` + +### Manual reload of FDW schema + +If you're still unable to get FDW working, you may want to try a manual +reload of the FDW schema. To manually reload the FDW schema: + +1. On the node running the Geo tracking database, enter the PostgreSQL console via + the `gitlab_geo` user: + + ```sh + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` + + Be sure to adjust the port and hostname for your configuration. You + may be asked to enter a password. + +1. Reload the schema via: + + ```sql + DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; + CREATE SCHEMA gitlab_secondary; + GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; + IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; + ``` + +1. Test that queries work: + + ```sql + SELECT * from information_schema.foreign_tables; + SELECT * FROM gitlab_secondary.projects limit 1; + ``` + +[database-start-replication]: database.md#step-3-initiate-the-replication-process +[database-pg-replication]: database.md#postgresql-replication diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md new file mode 100644 index 00000000000..b9921b2e69f --- /dev/null +++ b/doc/administration/geo/replication/tuning.md @@ -0,0 +1,17 @@ +# Tuning Geo **[PREMIUM ONLY]** + +## Changing the sync capacity values + +In the Geo admin page (`/admin/geo/nodes`), there are several variables that +can be tuned to improve performance of Geo: + +- Repository sync capacity. +- File sync capacity. + +Increasing these values will increase the number of jobs that are scheduled. +However, this may not lead to more downloads in parallel unless the number of +available Sidekiq threads is also increased. For example, if repository sync +capacity is increased from 25 to 50, you may also want to increase the number +of Sidekiq threads from 25 to 50. See the +[Sidekiq concurrency documentation](https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#number-of-threads) +for more details. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md new file mode 100644 index 00000000000..66728366e24 --- /dev/null +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -0,0 +1,403 @@ +# Updating the Geo nodes **[PREMIUM ONLY]** + +Depending on which version of Geo you are updating to/from, there may be +different steps. + +## General update steps + +In order to update the Geo nodes when a new GitLab version is released, +all you need to do is update GitLab itself: + +1. Log into each node (**primary** and **secondary** nodes). +1. [Update GitLab][update]. +1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when + the tracking database is enabled. +1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. + +## Upgrading to GitLab 10.8 + +Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node: + +```sh +sudo gitlab-rake cache:clear +``` + +## Upgrading to GitLab 10.6 + +In 10.4, we started to recommend that you define a password for database user (`gitlab`). + +We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize +the Geo Tracking Database. We are also improving security by disabling the use of **trust** +authentication method. + +1. **[primary]** Login to your **primary** node and run: + + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` + + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this + ``` + +1. **[primary]** Reconfigure and restart: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` + +1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + + # Enable Foreign Data Wrapper + geo_secondary['db_fdw'] = true + + # Secondary address in CIDR format, for example '5.6.7.8/32' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + ``` + + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this + ``` + +1. **[secondary]** Reconfigure and restart: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` + +## Upgrading to GitLab 10.5 + +For Geo Disaster Recovery to work with minimum downtime, your **secondary** node +should use the same set of secrets as the **primary** node. However, setup instructions +prior to the 10.5 release only synchronized the `db_key_base` secret. + +To rectify this error on existing installations, you should **overwrite** the +contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the +contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the +following command on each **secondary** node: + +```sh +sudo gitlab-ctl reconfigure +``` + +If you do not perform this step, you may find that two-factor authentication +[is broken following DR](../disaster_recovery/index.html#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken). + +To prevent SSH requests to the newly promoted **primary** node from failing +due to SSH host key mismatch when updating the **primary** node domain's DNS record +you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each +**secondary** node. + +## Upgrading to GitLab 10.4 + +There are no Geo-specific steps to take! + +## Upgrading to GitLab 10.3 + +### Support for SSH repository synchronization removed + +In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, +support is removed entirely. All installations will switch to the HTTP/HTTPS +cloning method instead. Before upgrading, ensure that all your Geo nodes are +configured to use this method and that it works for your installation. In +particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps). + +Synchronizing repositories over the public Internet using HTTP is insecure, so +you should ensure that you have HTTPS configured before upgrading. Note that +file synchronization is **also** insecure in these cases! + +## Upgrading to GitLab 10.2 + +### Secure PostgreSQL replication + +Support for TLS-secured PostgreSQL replication has been added. If you are +currently using PostgreSQL replication across the open internet without an +external means of securing the connection (e.g., a site-to-site VPN), then you +should immediately reconfigure your **primary** and **secondary** PostgreSQL instances +according to the [updated instructions][database]. + +If you *are* securing the connections externally and wish to continue doing so, +ensure you include the new option `--sslmode=prefer` in future invocations of +`gitlab-ctl replicate-geo-database`. + +### HTTPS repository sync + +Support for replicating repositories and wikis over HTTP/HTTPS has been added. +Replicating over SSH has been deprecated, and support for this option will be +removed in a future release. + +To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit +**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed, +press the "Edit" button, change the "Repository cloning" setting from +"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take +effect immediately. + +Any new secondaries should be created using HTTP/HTTPS replication - this is the +default setting. + +After you've verified that HTTP/HTTPS replication is working, you should remove +the now-unused SSH keys from your secondaries, as they may cause problems if the +**secondary** node if ever promoted to a **primary** node: + +1. **[secondary]** Login to **all** your **secondary** nodes and run: + + ```ruby + sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub + ``` + +### Hashed Storage + +CAUTION: **Warning:** +Hashed storage is in **Alpha**. It is considered experimental and not +production-ready. See [Hashed Storage] for more detail. + +If you previously enabled Hashed Storage and migrated all your existing +projects to Hashed Storage, disabling hashed storage will not migrate projects +to their previous project based storage path. As such, once enabled and +migrated we recommend leaving Hashed Storage enabled. + +## Upgrading to GitLab 10.1 + +CAUTION: **Warning:** +Hashed storage is in **Alpha**. It is considered experimental and not +production-ready. See [Hashed Storage] for more detail. + +[Hashed storage] was introduced in GitLab 10.0, and a [migration path][hashed-migration] +for existing repositories was added in GitLab 10.1. + +## Upgrading to GitLab 10.0 + +Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via +the database][ssh-fast-lookup] to avoid having to maintain consistency of the +`authorized_keys` file for SSH access. Failing to do this will prevent users +from being able to clone via SSH. + +Note that in older versions of Geo, attachments downloaded on the **secondary** +nodes would be saved to the wrong directory. We recommend that you do the +following to clean this up. + +On the **secondary** Geo nodes, run as root: + +```sh +mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old +mkdir /var/opt/gitlab/gitlab-rails/working +chmod 700 /var/opt/gitlab/gitlab-rails/working +chown git:git /var/opt/gitlab/gitlab-rails/working +``` + +You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time. + +Once this is done, we advise restarting GitLab on the **secondary** nodes for the +new working directory to be used: + +```sh +sudo gitlab-ctl restart +``` + +## Upgrading from GitLab 9.3 or older + +If you started running Geo on GitLab 9.3 or older, we recommend that you +resync your **secondary** PostgreSQL databases to use replication slots. If you +started using Geo with GitLab 9.4 or 10.x, no further action should be +required because replication slots are used by default. However, if you +started with GitLab 9.3 and upgraded later, you should still follow the +instructions below. + +When in doubt, it does not hurt to do a resync. The easiest way to do this in +Omnibus is the following: + + 1. Make sure you have Omnibus GitLab on the **primary** server. + 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. + 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. + 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. + 1. Install GitLab on the **secondary** server. + 1. Re-run the [database replication process][database-replication]. + +## Special update notes for 9.0.x + +> **IMPORTANT**: +With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are +required in order to update the **secondary** nodes and keep the Streaming +Replication working. Downtime is required, so plan ahead. + +The following steps apply only if you upgrade from a 8.17 GitLab version to +9.0+. For previous versions, update to GitLab 8.17 first before attempting to +upgrade to 9.0+. + +--- + +Make sure to follow the steps in the exact order as they appear below and pay +extra attention in what node (either **primary** or **secondary**) you execute them! Each step +is prepended with the relevant node for better clarity: + +1. **[secondary]** Login to **all** your **secondary** nodes and stop all services: + + ```ruby + sudo gitlab-ctl stop + ``` + +1. **[secondary]** Make a backup of the `recovery.conf` file on **all** + **secondary** nodes to preserve PostgreSQL's credentials: + + ```sh + sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ + ``` + +1. **[primary]** Update the **primary** node to GitLab 9.0 following the + [regular update docs][update]. At the end of the update, the **primary** node + will be running with PostgreSQL 9.6. + +1. **[primary]** To prevent a de-synchronization of the repository replication, + stop all services except `postgresql` as we will use it to re-initialize the + **secondary** node's database: + + ```sh + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` + +1. **[secondary]** Run the following steps on each of the **secondary** nodes: + + 1. **[secondary]** Stop all services: + + ```sh + sudo gitlab-ctl stop + ``` + + 1. **[secondary]** Prevent running database migrations: + + ```sh + sudo touch /etc/gitlab/skip-auto-migrations + ``` + + 1. **[secondary]** Move the old database to another directory: + + ```sh + sudo mv /var/opt/gitlab/postgresql{,.bak} + ``` + + 1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update]. + At the end of the update, the node will be running with PostgreSQL 9.6. + + 1. **[secondary]** Make sure all services are up: + + ```sh + sudo gitlab-ctl start + ``` + + 1. **[secondary]** Reconfigure GitLab: + + ```sh + sudo gitlab-ctl reconfigure + ``` + + 1. **[secondary]** Run the PostgreSQL upgrade command: + + ```sh + sudo gitlab-ctl pg-upgrade + ``` + + 1. **[secondary]** See the stored credentials for the database that you will + need to re-initialize the replication: + + ```sh + sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf + ``` + + 1. **[secondary]** Create the `replica.sh` script as described in the + [database configuration document][database-source-replication]. + + 1. **[secondary]** Run the recovery script using the credentials from the + previous step: + + ```sh + sudo bash /tmp/replica.sh + ``` + + 1. **[secondary]** Reconfigure GitLab: + + ```sh + sudo gitlab-ctl reconfigure + ``` + + 1. **[secondary]** Start all services: + + ```sh + sudo gitlab-ctl start + ``` + + 1. **[secondary]** Repeat the steps for the remaining **secondary** nodes. + +1. **[primary]** After all **secondary** nodes are updated, start all services in + **primary** node: + + ```sh + sudo gitlab-ctl start + ``` + +## Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + +## Update tracking database on **secondary** node + +After updating a **secondary** node, you might need to run migrations on +the tracking database. The tracking database was added in GitLab 9.1, +and it is required since 10.0. + +1. Run database migrations on tracking database: + + ```sh + sudo gitlab-rake geo:db:migrate + ``` + +1. Repeat this step for each **secondary** node. + +[update]: ../../../update/README.md +[database]: database.md +[database-replication]: database.md#step-3-initiate-the-replication-process +[database-source-replication]: database_source.md#step-3-initiate-the-replication-process +[Hashed Storage]: ../../repository_storage_types.md +[hashed-migration]: ../../raketasks/storage.md +[ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md new file mode 100644 index 00000000000..f1f1fe48748 --- /dev/null +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -0,0 +1,18 @@ +[//]: # (Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) + +# Using a Geo Server **[PREMIUM ONLY]** + +After you set up the [database replication and configure the Geo nodes][req], use your closest GitLab node as you would a normal standalone GitLab instance. + +Pushing directly to a **secondary** node (for both HTTP, SSH including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. + +Example of the output you will see when pushing to a **secondary** node: + +```bash +$ git push +> GitLab: You're pushing to a Geo secondary. +> GitLab: We'll help you by proxying this request to the primary: ssh://git@primary.geo/user/repo.git +Everything up-to-date +``` + +[req]: index.md#setup-instructions diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 49fe80fb2a6..de61c01991b 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -1,4 +1,4 @@ -# High Availability +# Scaling and High Availability GitLab supports several different types of clustering and high-availability. The solution you choose will be based on the level of scalability and @@ -13,51 +13,173 @@ of Git, developers can still commit code locally even when GitLab is not available. However, some GitLab features such as the issue tracker and Continuous Integration are not available when GitLab is down. -**Keep in mind that all Highly Available solutions come with a trade-off between +**Keep in mind that all highly-available solutions come with a trade-off between cost/complexity and uptime**. The more uptime you want, the more complex the solution. And the more complex the solution, the more work is involved in setting up and maintaining it. High availability is not free and every HA solution should balance the costs against the benefits. -## Architecture - -There are two kinds of setups: - -- active/active -- active/passive - -### Active/Active - -This architecture scales easily because all application servers handle -user requests simultaneously. The database, Redis, and GitLab application are -all deployed on separate servers. The configuration is **only** highly-available -if the database, Redis and storage are also configured as such. - -Follow the steps below to configure an active/active setup: +There are many options when choosing a highly-available GitLab architecture. We +recommend engaging with GitLab Support to choose the best architecture for your +use-case. This page contains some various options and guidelines based on +experience with GitLab.com and Enterprise Edition on-premises customers. + +For a detailed insight into how GitLab scales and configures GitLab.com, you can +watch [this 1 hour Q&A](https://www.youtube.com/watch?v=uCU8jdYzpac) +with [John Northrup](https://gitlab.com/northrup), one of our infrastructure +engineers, and live questions coming in from some of our customers. + +## GitLab Components + +The following components need to be considered for a scaled or highly-available +environment. In many cases components can be combined on the same nodes to reduce +complexity. + +- Unicorn/Workhorse - Web-requests (UI, API, Git over HTTP) +- Sidekiq - Asynchronous/Background jobs +- PostgreSQL - Database + - Consul - Database service discovery and health checks/failover + - PGBouncer - Database pool manager +- Redis - Key/Value store (User sessions, cache, queue for Sidekiq) + - Sentinel - Redis health check/failover manager +- Gitaly - Provides high-level RPC access to Git repositories + +## Scalable Architecture Examples + +When an organization reaches a certain threshold it will be necessary to scale +the GitLab instance. Still, true high availability may not be necessary. There +are options for scaling GitLab instances relatively easily without incurring the +infrastructure and maintenance costs of full high availability. + +### Basic Scaling + +This is the simplest form of scaling and will work for the majority of +cases. Backend components such as PostgreSQL, Redis and storage are offloaded +to their own nodes while the remaining GitLab components all run on 2 or more +application nodes. + +This form of scaling also works well in a cloud environment when it is more +cost-effective to deploy several small nodes rather than a single +larger one. + +- 1 PostgreSQL node +- 1 Redis node +- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq) +- 1 NFS/Gitaly storage server + +#### Installation Instructions + +Complete the following installation steps in order. A link at the end of each +section will bring you back to the Scalable Architecture Examples section so +you can continue with the next step. + +1. [PostgreSQL](./database.md#postgresql-in-a-scaled-environment) +1. [Redis](./redis.md#redis-in-a-scaled-environment) +1. [Gitaly](./gitaly.md) (recommended) or [NFS](./nfs.md) +1. [GitLab application nodes](./gitlab.md) + +### Full Scaling + +For very large installations it may be necessary to further split components +for maximum scalability. In a fully-scaled architecture the application node +is split into separate Sidekiq and Unicorn/Workhorse nodes. One indication that +this architecture is required is if Sidekiq queues begin to periodically increase +in size, indicating that there is contention or not enough resources. + +- 1 PostgreSQL node +- 1 Redis node +- 2 or more GitLab application nodes (Unicorn, Workhorse) +- 2 or more Sidekiq nodes +- 2 or more NFS/Gitaly storage servers + +## High Availability Architecture Examples + +When organizations require scaling *and* high availability the following +architectures can be utilized. As the introduction section at the top of this +page mentions, there is a tradeoff between cost/complexity and uptime. Be sure +this complexity is absolutely required before taking the step into full +high availability. + +For all examples below, we recommend running Consul and Redis Sentinel on +dedicated nodes. If Consul is running on PostgreSQL nodes or Sentinel on +Redis nodes there is a potential that high resource usage by PostgreSQL or +Redis could prevent communication between the other Consul and Sentinel nodes. +This may lead to the other nodes believing a failure has occurred and automated +failover is necessary. Isolating them from the services they monitor reduces +the chances of split-brain. + +The examples below do not really address high availability of NFS. Some enterprises +have access to NFS appliances that manage availability. This is the best case +scenario. In the future, GitLab may offer a more user-friendly solution to +[GitLab HA Storage](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2472). + +There are many options in between each of these examples. Work with GitLab Support +to understand the best starting point for your workload and adapt from there. + +### Horizontal + +This is the simplest form of high availability and scaling. It requires the +fewest number of individual servers (virtual or physical) but does have some +trade-offs and limits. + +This architecture will work well for many GitLab customers. Larger customers +may begin to notice certain events cause contention/high load - for example, +cloning many large repositories with binary files, high API usage, a large +number of enqueued Sidekiq jobs, etc. If this happens you should consider +moving to a hybrid or fully distributed architecture depending on what is causing +the contention. + +- 3 PostgreSQL nodes +- 2 Redis nodes +- 3 Consul/Sentinel nodes +- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer) +- 1 NFS/Gitaly server + + + +### Hybrid + +In this architecture, certain components are split on dedicated nodes so high +resource usage of one component does not interfere with others. In larger +environments this is a good architecture to consider if you foresee or do have +contention due to certain workloads. + +- 3 PostgreSQL nodes +- 2 Redis nodes +- 3 Consul/Sentinel nodes +- 2 or more Sidekiq nodes +- 2 or more Web nodes (Unicorn, Workhorse, PGBouncer) +- 1 or more NFS/Gitaly servers + + + +### Fully Distributed + +This architecture scales to hundreds of thousands of users and projects and is +the basis of the GitLab.com architecture. While this scales well it also comes +with the added complexity of many more nodes to configure, manage and monitor. + +- 3 PostgreSQL nodes +- 4 or more Redis nodes (2 separate clusters for persistent and cache data) +- 3 Consul nodes +- 3 Sentinel nodes +- Multiple dedicated Sidekiq nodes (Split into real-time, best effort, ASAP, + CI Pipeline and Pull Mirror sets) +- 2 or more Git nodes (Git over SSH/Git over HTTP) +- 2 or more API nodes (All requests to `/api`) +- 2 or more Web nodes (All other web requests) +- 2 or more NFS/Gitaly servers + + + +The following pages outline the steps necessary to configure each component +separately: 1. [Configure the database](database.md) 1. [Configure Redis](redis.md) 1. [Configure Redis for GitLab source installations](redis_source.md) 1. [Configure NFS](nfs.md) + 1. [NFS Client and Host setup](nfs_host_client_setup.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) - - -### Active/Passive - -For pure high-availability/failover with no scaling you can use an -active/passive configuration. This utilizes DRBD (Distributed Replicated -Block Device) to keep all data in sync. DRBD requires a low latency link to -remain in sync. It is not advisable to attempt to run DRBD between data centers -or in different cloud availability zones. - -> **Note:** GitLab recommends against choosing this HA method because of the - complexity of managing DRBD and crafting automatic failover. This is - *compatible* with GitLab, but not officially *supported*. If you are - an EE customer, support will help you with GitLab related problems, but if the - root cause is identified as DRBD, we will not troubleshoot further. - -Components/Servers Required: 2 servers/virtual machines (one active/one passive) - - diff --git a/doc/administration/high_availability/alpha_database.md b/doc/administration/high_availability/alpha_database.md new file mode 100644 index 00000000000..7bf20be60e6 --- /dev/null +++ b/doc/administration/high_availability/alpha_database.md @@ -0,0 +1,6 @@ +--- +redirect_to: 'database.md' +--- + +This documentation has been moved to the main +[database documentation](database.md#configure_using_omnibus_for_high_availability). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md new file mode 100644 index 00000000000..056b7fc15d9 --- /dev/null +++ b/doc/administration/high_availability/consul.md @@ -0,0 +1,105 @@ +# Working with the bundled Consul service **[PREMIUM ONLY]** + +## Overview + +As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](http://consul.io) that can be managed through `/etc/gitlab/gitlab.rb`. + +A Consul cluster consists of multiple server agents, as well as client agents that run on other nodes which need to talk to the consul cluster. + +## Operations + +### Checking cluster membership + +To see which nodes are part of the cluster, run the following on any member in the cluster +``` +# /opt/gitlab/embedded/bin/consul members +Node Address Status Type Build Protocol DC +consul-b XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +consul-c XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +consul-c XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +db-a XX.XX.X.Y:8301 alive client 0.9.0 2 gitlab_consul +db-b XX.XX.X.Y:8301 alive client 0.9.0 2 gitlab_consul +``` + +Ideally all nodes will have a `Status` of `alive`. + +### Restarting the server cluster + +**Note**: This section only applies to server agents. It is safe to restart client agents whenever needed. + +If it is necessary to restart the server cluster, it is important to do this in a controlled fashion in order to maintain quorum. If quorum is lost, you will need to follow the consul [outage recovery](#outage-recovery) process to recover the cluster. + +To be safe, we recommend you only restart one server agent at a time to ensure the cluster remains intact. + +For larger clusters, it is possible to restart multiple agents at a time. See the [Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) for how many failures it can tolerate. This will be the number of simulateneous restarts it can sustain. + +## Troubleshooting + +### Consul server agents unable to communicate + +By default, the server agents will attempt to [bind](https://www.consul.io/docs/agent/options.html#_bind) to '0.0.0.0', but they will advertise the first private IP address on the node for other agents to communicate with them. If the other nodes cannot communicate with a node on this address, then the cluster will have a failed status. + +You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: + +``` +2017-09-25_19:53:39.90821 2017/09/25 19:53:39 [WARN] raft: no known peers, aborting election +2017-09-25_19:53:41.74356 2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader +``` + + +To fix this: + +1. Pick an address on each node that all of the other nodes can reach this node through. +1. Update your `/etc/gitlab/gitlab.rb` + + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` +1. Run `gitlab-ctl reconfigure` + +If you still see the errors, you may have to [erase the consul database and reinitialize](#recreate-from-scratch) on the affected node. + +### Consul agents do not start - Multiple private IPs + +In the case that a node has multiple private IPs the agent be confused as to which of the private addresses to advertise, and then immediately exit on start. + +You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: + +``` +2017-11-09_17:41:45.52876 ==> Starting Consul agent... +2017-11-09_17:41:45.53057 ==> Error creating agent: Failed to get advertise address: Multiple private IPs found. Please configure one. +``` + +To fix this: + +1. Pick an address on the node that all of the other nodes can reach this node through. +1. Update your `/etc/gitlab/gitlab.rb` + + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` +1. Run `gitlab-ctl reconfigure` + +### Outage recovery + +If you lost enough server agents in the cluster to break quorum, then the cluster is considered failed, and it will not function without manual intervenetion. + +#### Recreate from scratch +By default, GitLab does not store anything in the consul cluster that cannot be recreated. To erase the consul database and reinitialize + +``` +# gitlab-ctl stop consul +# rm -rf /var/opt/gitlab/consul/data +# gitlab-ctl start consul +``` + +After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well. + +#### Recover a failed cluster +If you have taken advantage of consul to store other data, and want to restore the failed cluster, please follow the [Consul guide](https://www.consul.io/docs/guides/outage.html) to recover a failed cluster. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index c1eeb40b98f..1648b6b848a 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -1,11 +1,6 @@ -# Configuring a Database for GitLab HA +# Configuring PostgreSQL for Scaling and High Availability -You can choose to install and manage a database server (PostgreSQL/MySQL) -yourself, or you can use GitLab Omnibus packages to help. GitLab recommends -PostgreSQL. This is the database that will be installed if you use the -Omnibus package to manage your database. - -## Configure your own database server +## Provide your own PostgreSQL instance **[CORE ONLY]** If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. For example, AWS offers a managed Relational @@ -20,91 +15,1147 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring GitLab for HA](gitlab.md). -## Configure using Omnibus +## PostgreSQL in a Scaled Environment -1. Download/install GitLab Omnibus using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. - Be sure to change the `external_url` to match your eventual GitLab front-end - URL. If there is a directive listed below that you do not see in the configuration, be sure to add it. +This section is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples) +environments including [Basic Scaling](./README.md#basic-scaling) and +[Full Scaling](./README.md#full-scaling). - ```ruby - external_url 'https://gitlab.example.com' +### Provide your own PostgreSQL instance **[CORE ONLY]** + +If you want to use your own deployed PostgreSQL instance(s), +see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled PostgreSQL. + +### Standalone PostgreSQL using GitLab Omnibus **[CORE ONLY]** + +1. SSH into the PostgreSQL server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. +1. Generate a password hash for PostgreSQL. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of `POSTGRESQL_PASSWORD_HASH`. + ```sh + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder + values appropriately. + + - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step + - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP + addresses of the GitLab application servers that will connect to the + database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` + + ```ruby # Disable all components except PostgreSQL roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_monitor['enable'] = false + + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + # ???? + postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` + + NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. +1. Note the PostgreSQL node's IP address or hostname, port, and + plain text password. These will be necessary when configuring the GitLab + application servers later. + +Advanced configuration options are supported and can be added if +needed. + +Continue configuration of other components by going +[back to Scaled Architectures](./README.md#scalable-architecture-examples) + +## PostgreSQL with High Availability + +This section is relevant for [High Availability Architecture](./README.md#high-availability-architecture-examples) +environments including [Horizontal](./README.md#horizontal), +[Hybrid](./README.md#hybrid), and +[Fully Distributed](./README.md#fully-distributed). + +### Provide your own PostgreSQL instance **[CORE ONLY]** + +If you want to use your own deployed PostgreSQL instance(s), +see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled PostgreSQL. + +### High Availability with GitLab Omnibus **[PREMIUM ONLY]** + +> Important notes: +> - This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +> - If you are a Community Edition or Starter user, consider using a cloud hosted solution. +> - This document will not cover installations from source. +> +> - If HA setup is not what you were looking for, see the [database configuration document](http://docs.gitlab.com/omnibus/settings/database.html) +> for the Omnibus GitLab packages. + +> Please read this document fully before attempting to configure PostgreSQL HA +> for GitLab. +> +> This configuration is GA in EE 10.2. + +The recommended configuration for a PostgreSQL HA requires: + +- A minimum of three database nodes + - Each node will run the following services: + - `PostgreSQL` - The database itself + - `repmgrd` - A service to monitor, and handle failover in case of a failure + - `Consul` agent - Used for service discovery, to alert other nodes when failover occurs +- A minimum of three `Consul` server nodes +- A minimum of one `pgbouncer` service node + +You also need to take into consideration the underlying network topology, +making sure you have redundant connectivity between all Database and GitLab instances, +otherwise the networks will become a single point of failure. + +#### Architecture + + + +Database nodes run two services with PostgreSQL: + +- Repmgrd. Monitors the cluster and handles failover when issues with the master occur. The failover consists of: + - Selecting a new master for the cluster. + - Promoting the new node to master. + - Instructing remaining servers to follow the new master node. + + On failure, the old master node is automatically evicted from the cluster, and should be rejoined manually once recovered. +- Consul. Monitors the status of each node in the database cluster and tracks its health in a service definition on the consul cluster. + +Alongside pgbouncer, there is a consul agent that watches the status of the PostgreSQL service. If that status changes, consul runs a script which updates the configuration and reloads pgbouncer + +##### Connection flow + +Each service in the package comes with a set of [default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports). You may need to make specific firewall rules for the connections listed below: + +- Application servers connect to [PgBouncer default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) +- PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Postgres secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul) + +#### Required information + +Before proceeding with configuration, you will need to collect all the necessary +information. + +##### Network information + +PostgreSQL does not listen on any network interface by default. It needs to know +which IP address to listen on in order to be accessible to other services. +Similarly, PostgreSQL access is controlled based on the network source. + +This is why you will need: + +> IP address of each nodes network interface +> - This can be set to `0.0.0.0` to listen on all interfaces. It cannot +> be set to the loopack address `127.0.0.1` +> +> Network Address +> - This can be in subnet (i.e. `192.168.0.0/255.255.255.0`) or CIDR (i.e. +> `192.168.0.0/24`) form. + +##### User information + +Various services require different configuration to secure +the communication as well as information required for running the service. +Bellow you will find details on each service and the minimum required +information you need to provide. + +##### Consul information + +When using default setup, minimum configuration requires: + +- `CONSUL_USERNAME`. Defaults to `gitlab-consul` +- `CONSUL_DATABASE_PASSWORD`. Password for the database user. +- `CONSUL_PASSWORD_HASH`. This is a hash generated out of consul username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME + ``` + +- `CONSUL_SERVER_NODES`. The IP addresses or DNS records of the Consul server nodes. + +Few notes on the service itself: + +- The service runs under a system account, by default `gitlab-consul`. + - If you are using a different username, you will have to specify it. We +will refer to it with `CONSUL_USERNAME`, +- There will be a database user created with read only access to the repmgr +database +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + - `/var/opt/gitlab/gitlab-consul/.pgpass`: plaintext + +##### PostgreSQL information + +When configuring PostgreSQL, we will set `max_wal_senders` to one more than +the number of database nodes in the cluster. +This is used to prevent replication from using up all of the +available database connections. + +> Note: +> - In this document we are assuming 3 database nodes, which makes this configuration: + +``` +postgresql['max_wal_senders'] = 4 +``` + +As previously mentioned, you'll have to prepare the network subnets that will +be allowed to authenticate with the database. +You'll also need to supply the IP addresses or DNS records of Consul +server nodes. + +We will need the following password information for the application's database user: + +- `POSTGRESQL_USERNAME`. Defaults to `gitlab` +- `POSTGRESQL_USER_PASSWORD`. The password for the database user +- `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME + ``` + +##### Pgbouncer information + +When using default setup, minimum configuration requires: + +- `PGBOUNCER_USERNAME`. Defaults to `pgbouncer` +- `PGBOUNCER_PASSWORD`. This is a password for pgbouncer service. +- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of pgbouncer username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME + ``` + +- `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running Pgbouncer. + +Few notes on the service itself: + +- The service runs as the same system account as the database + - In the package, this is by default `gitlab-psql` +- If you use a non-default user account for Pgbouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. +- The service will have a regular database user account generated for it + - This defaults to `repmgr` +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed, and in plain text + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + +##### Repmgr information + +When using default setup, you will only have to prepare the network subnets that will +be allowed to authenticate with the service. + +Few notes on the service itself: + +- The service runs under the same system account as the database + - In the package, this is by default `gitlab-psql` +- The service will have a superuser database user account generated for it + - This defaults to `gitlab_repmgr` + +#### Installing Omnibus GitLab + +First, make sure to [download/install](https://about.gitlab.com/installation) +GitLab Omnibus **on each node**. + +Make sure you install the necessary dependencies from step 1, +add GitLab package repository from step 2. +When installing the GitLab package, do not supply `EXTERNAL_URL` value. + +#### Configuring the Consul nodes + +On each Consul node perform the following: + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Consul + roles ['consul_role'] + + # START user configuration + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + server: true, + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + # + # END user configuration + ``` + + > `consul_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. + +##### Consul Checkpoint + +Before moving on, make sure Consul is configured correctly. Run the following +command to verify all server nodes are communicating: + +``` +/opt/gitlab/embedded/bin/consul members +``` + +The output should be similar to: + +``` +Node Address Status Type Build Protocol DC +CONSUL_NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_THREE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +``` + +If any of the nodes isn't `alive` or if any of the three nodes are missing, +check the [Troubleshooting section](#troubleshooting) before proceeding. + +#### Configuring the Database nodes + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. + +1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] # PostgreSQL configuration - gitlab_rails['db_password'] = 'DB password' - postgresql['md5_auth_cidr_addresses'] = ['0.0.0.0/0'] postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' # Disable automatic database migrations gitlab_rails['auto_migrate'] = false + + # Configure the consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + # Replace X with value of number of db nodes + 1 + postgresql['max_wal_senders'] = X + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) + + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration ``` -1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. + > `postgres_role` was introduced with GitLab 10.3 + +1. On secondary nodes, add all the configuration specified above for primary node + to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration + to inform gitlab-ctl that they are standby nodes initially and it need not + attempt to register them as primary node + ``` + # HA setting to specify if a node should attempt to be master on initialization + repmgr['master_on_initialization'] = false + ``` - > **Note**: This `reconfigure` step will result in some errors. - That's OK - don't be alarmed. +1. [Reconfigure GitLab] for te changes to take effect. + +> Please note: +> - If you want your database to listen on a specific interface, change the config: +> `postgresql['listen_address'] = '0.0.0.0'` +> - If your Pgbouncer service runs under a different user account, +> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in +> your configuration + +##### Database nodes post-configuration + +###### Primary node + +Select one node as a primary node. 1. Open a database prompt: + ```sh + gitlab-psql -d gitlabhq_production ``` - su - gitlab-psql - /bin/bash - psql -h /var/opt/gitlab/postgresql -d template1 - # Output: +1. Enable the `pg_trgm` extension: - psql (9.2.15) - Type "help" for help. + ```sh + CREATE EXTENSION pg_trgm; + ``` + +1. Exit the database prompt by typing `\q` and Enter. + +1. Verify the cluster is initialized with one node: - template1=# + ```sh + gitlab-ctl repmgr cluster show ``` -1. Run the following command at the database prompt and you will be asked to - enter the new password for the PostgreSQL superuser. + The output should be similar to the following: ``` - \password + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` - # Output: +1. Note down the hostname/ip in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) - Enter new password: - Enter it again: + +###### Secondary nodes + +1. Set up the repmgr standby: + + ```sh + gitlab-ctl repmgr standby setup MASTER_NODE_NAME ``` -1. Similarly, set the password for the `gitlab` database user. Use the same - password that you specified in the `/etc/gitlab/gitlab.rb` file for - `gitlab_rails['db_password']`. + Do note that this will remove the existing data on the node. The command + has a wait time. + The output should be similar to the following: + + ```console + # gitlab-ctl repmgr standby setup MASTER_NODE_NAME + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s ``` - \password gitlab - # Output: +1. Verify the node now appears in the cluster: - Enter new password: - Enter it again: + ```sh + gitlab-ctl repmgr cluster show ``` -1. Exit from editing `template1` prompt by typing `\q` and Enter. -1. Enable the `pg_trgm` extension within the `gitlabhq_production` database: - + + The output should be similar to the following: + + ``` + Role | Name | Upstream | Connection String + ----------+---------|-----------|------------------------------------------------ + * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +Repeat the above steps on all secondary nodes. + +##### Database checkpoint + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly: + +``` +gitlab-ctl repmgr cluster show +``` + +The output should be similar to: + +``` +Role | Name | Upstream | Connection String +----------+--------------|--------------|-------------------------------------------------------------------- +* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](#troubleshooting) before proceeding. + +Also, check that the check master command works successfully on each node: + +``` +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](#troubleshooting) before proceeding. + +#### Configuring the Pgbouncer node + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure Pgbouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # Replace CONSUL_PASSWORD_HASH with with a generated md5 value + # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value + pgbouncer['users'] = { + 'gitlab-consul': { + password: 'CONSUL_PASSWORD_HASH' + }, + 'pgbouncer': { + password: 'PGBOUNCER_PASSWORD_HASH' + } + } + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration ``` + + > `pgbouncer_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. + +1. Create a `.pgpass` file so Consule is able to + reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +##### PGBouncer Checkpoint + +1. Ensure the node is talking to the current master: + + ```sh + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + + If there is an error `psql: ERROR: Auth failed` after typing in the + password, ensure you previously generated the MD5 password hashes with the correct + format. The correct format is to concatenate the password and the username: + `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text + needed to generate an MD5 password hash for the `pgbouncer` user. + +1. Once the console prompt is available, run the following queries: + + ```sh + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ``` + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` + +#### Configuring the Application nodes + +These will be the nodes running the `gitlab-rails` service. You may have other +attributes set, but the following need to be set. + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Disable PostgreSQL on the application node + postgresql['enable'] = false + + gitlab_rails['db_host'] = 'PGBOUNCER_NODE' + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +##### Application node post-configuration + +Ensure that all migrations ran: + +```sh +gitlab-rake gitlab:db:configure +``` + +> **Note**: If you encounter a `rake aborted!` error stating that PGBouncer is failing to connect to +PostgreSQL it may be that your PGBouncer node's IP address is missing from +PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See +[PGBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +in the Troubleshooting section before proceeding. + +##### Ensure GitLab is running + +At this point, your GitLab instance should be up and running. Verify you are +able to login, and create issues and merge requests. If you have troubles check +the [Troubleshooting section](#troubleshooting). + +#### Example configuration + +Here we'll show you some fully expanded example configurations. + +##### Example recommended setup + +This example uses 3 consul servers, 3 postgresql servers, and 1 application node. + +We start with all servers on the same 10.6.0.0/16 private network range, they +can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +* `10.6.0.11`: Consul 1 +* `10.6.0.12`: Consul 2 +* `10.6.0.13`: Consul 3 +* `10.6.0.21`: PostgreSQL master +* `10.6.0.22`: PostgreSQL secondary +* `10.6.0.23`: PostgreSQL secondary +* `10.6.0.31`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. + +The external_url for GitLab is `http://gitlab.example.com` + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +##### Example recommended setup for Consul servers + +On each server edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except Consul +roles ['consul_role'] + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup for PostgreSQL servers + +###### Primary node + +On primary node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL and Repmgr and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the consul agent +consul['services'] = %w(postgresql) + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +###### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration +added to primary node, noted above. In addition, append the following +configuration + +``` +# HA setting to specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = true +consul['enable'] = true + +# Configure Pgbouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup manual steps + +After deploying the configuration follow these steps: + +1. On `10.6.0.21`, our primary database + + Enable the `pg_trgm` extension + + ```sh gitlab-psql -d gitlabhq_production - + ``` + + ``` CREATE EXTENSION pg_trgm; + ``` + +1. On `10.6.0.22`, our first standby database - # Output: + Make this node a standby of the primary - CREATE EXTENSION + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 ``` -1. Exit the database prompt by typing `\q` and Enter. -1. Exit the `gitlab-psql` user by running `exit` twice. -1. Run `sudo gitlab-ctl reconfigure` a final time. -1. Configure the GitLab application servers with the appropriate details. - This step is covered in [Configuring GitLab for HA](gitlab.md). + +1. On `10.6.0.23`, our second standby database + + Make this node a standby of the primary + + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` + +1. On `10.6.0.31`, our application server + + Set gitlab-consul's pgbouncer password to `toomanysecrets` + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + + Run database migrations + + ```sh + gitlab-rake gitlab:db:configure + ``` + +#### Example minimal setup + +This example uses 3 postgresql servers, and 1 application node. + +It differs from the [recommended setup](#example-recommended-setup) by moving the consul servers into the same servers we use for PostgreSQL. +The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with postgres [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [consul outage recovery](consul.md#outage-recovery) on the same set of machines. + +In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +* `10.6.0.21`: PostgreSQL master +* `10.6.0.22`: PostgreSQL secondary +* `10.6.0.23`: PostgreSQL secondary +* `10.6.0.31`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. + +The external_url for GitLab is `http://gitlab.example.com` + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +##### Example minimal configuration for database servers + +##### Primary node +On primary database node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL, Repmgr, and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the consul agent +consul['services'] = %w(postgresql) + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +###### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added +to primary node, noted above. In addition, append the following configuration + +``` +# HA setting to specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +##### Example minimal configuration for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = true +consul['enable'] = true + +# Configure Pgbouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example minimal setup manual steps + +The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). + +#### Failover procedure + +By default, if the master database fails, `repmgrd` should promote one of the +standby nodes to master automatically, and consul will update pgbouncer with +the new master. + +If you need to failover manually, you have two options: + +**Shutdown the current master database** + +Run: + +```sh +gitlab-ctl stop postgresql +``` + +The automated failover process will see this and failover to one of the +standby nodes. + +**Or perform a manual failover** + +1. Ensure the old master node is not still active. +1. Login to the server that should become the new master and run: + + ```sh + gitlab-ctl repmgr standby promote + ``` + +1. If there are any other standby servers in the cluster, have them follow + the new master server: + + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + ``` + +#### Restore procedure + +If a node fails, it can be removed from the cluster, or added back as a standby +after it has been restored to service. + +- If you want to remove the node from the cluster, on any other node in the + cluster, run: + + ```sh + gitlab-ctl repmgr standby unregister --node=X + ``` + + where X is the value of node in `repmgr.conf` on the old server. + + To find this, you can use: + + ```sh + awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf + ``` + + It will output something like: + + ``` + 959789412 + ``` + + Then you will use this id to unregister the node: + + ```sh + gitlab-ctl repmgr standby unregister --node=959789412 + ``` + +- To add the node as a standby server: + + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + gitlab-ctl restart repmgrd + ``` + + CAUTION: **Warning:** When the server is brought back online, and before + you switch it to a standby node, repmgr will report that there are two masters. + If there are any clients that are still attempting to write to the old master, + this will cause a split, and the old master will need to be resynced from + scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. + +#### Alternate configurations + +##### Database authorization + +By default, we give any host on the database network the permission to perform +repmgr operations using PostgreSQL's `trust` method. If you do not want this +level of trust, there are alternatives. + +You can trust only the specific nodes that will be database clusters, or you +can require md5 authentication. + +##### Trust specific addresses + +If you know the IP address, or FQDN of all database and pgbouncer nodes in the +cluster, you can trust only those nodes. + +In `/etc/gitlab/gitlab.rb` on all of the database nodes, set +`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of +the addresses. + +If setting to a node's FQDN, they must have a corresponding PTR record in DNS. +If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. + +For example: + +```ruby +repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) +``` + + +##### MD5 Authentication + +If you are running on an untrusted network, repmgr can use md5 authentication +with a [.pgpass file](https://www.postgresql.org/docs/9.6/static/libpq-pgpass.html) +to authenticate. + +You can specify by IP address, FQDN, or by subnet, using the same format as in +the previous section: + +1. On the current master node, create a password for the `gitlab` and + `gitlab_repmgr` user: + + ```sh + gitlab-psql -d template1 + template1=# \password gitlab_repmgr + Enter password: **** + Confirm password: **** + template1=# \password gitlab + ``` + +1. On each database node: + + 1. Edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set + 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value + 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` + 1. Reconfigure with `gitlab-ctl reconfigure` + 1. Restart postgresql with `gitlab-ctl restart postgresql` + + 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to + when asked: + + ```sh + gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' + ``` + +1. On each pgbouncer node, edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for + the `gitlab` database user + 1. [Reconfigure GitLab] for the changes to take effect + +## Troubleshooting + +#### Consul and PostgreSQL changes not taking effect. + +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. + +To restart either service, run `gitlab-ctl restart SERVICE` + +For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. + +On the consul server nodes, it is important to restart the consul service in a controlled fashion. Read our [consul documentation](consul.md#restarting-the-server-cluster) for instructions on how to restart the service. + +#### `gitlab-ctl repmgr-check-master` command produces errors + +If this command displays errors about database permissions it is likely that something failed during +install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these +steps to fix the problem: + +1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` +1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` +1. Exit the database prompt - `\q` +1. [Reconfigure GitLab] and the user will be re-added with the proper permissions. +1. Change to the `gitlab-consul` user - `su - gitlab-consul` +1. Try the check command again - `gitlab-ctl repmgr-check-master`. + +Now there should not be errors. If errors still occur then there is another problem. + +#### PGBouncer error `ERROR: pgbouncer cannot connect to server` + +You may get this error when running `gitlab-rake gitlab:db:configure` or you +may see the error in the PGBouncer log file. + +``` +PG::ConnectionBad: ERROR: pgbouncer cannot connect to server +``` + +The problem may be that your PGBouncer node's IP address is not included in the +`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. + +You can confirm that this is the issue by checking the PostgreSQL log on the master +database node. If you see the following error then `trust_auth_cidr_addresses` +is the problem. + +``` +2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off +``` + +To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. + +``` +postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) +``` + +[Reconfigure GitLab] for the changes to take effect. + +#### Issues with other components + +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. + +- [Consul](consul.md#troubleshooting) +- [PostgreSQL](http://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) +- [GitLab application](gitlab.md#troubleshooting) + +## Configure using Omnibus + +**Note**: We recommend that you follow the instructions here for a full [PostgreSQL cluster](#high-availability-with-gitlab-omnibus-premium-only). +If you are reading this section due to an old bookmark, you can find that old documentation [in the repository](https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.1.4/doc/administration/high_availability/database.md#configure-using-omnibus). --- @@ -114,3 +1165,6 @@ Read more on high-availability configuration: 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) +1. [Manage the bundled Consul cluster](consul.md) + +[reconfigure GitLab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md new file mode 100644 index 00000000000..d44744f2af8 --- /dev/null +++ b/doc/administration/high_availability/gitaly.md @@ -0,0 +1,90 @@ +# Configuring Gitaly for Scaled and High Availability + +Gitaly does not yet support full high availability. However, Gitaly is quite +stable and is in use on GitLab.com. Scaled and highly available GitLab environments +should consider using Gitaly on a separate node. + +See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to +track plans and progress toward high availability support. + +This document is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples) +environments and [High Availability Architecture](./README.md#high-availability-architecture-examples). + +## Running Gitaly on its own server + +Starting with GitLab 11.4, Gitaly is a replacement for NFS except +when the [Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer) +is used. + +NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend using EFS as it may impact GitLab's performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. + +NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to +restrict access to your Gitaly server. + +The steps below are the minimum necessary to configure a Gitaly server with +Omnibus: + +1. SSH into the Gitaly server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result, + the GitLab Shell secret must be the same between the other GitLab servers and + the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json` + from an existing GitLab server to the Gitaly server. Without this shared secret, + Git operations in GitLab will result in an API error. + + > **NOTE:** In most or all cases the storage paths below end in `repositories` which is + different than `path` in `git_data_dirs` of Omnibus installations. Check the + directory layout on your Gitaly server to be sure. + + ```ruby + # Enable Gitaly + gitaly['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_monitor['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + gitaly['listen_addr'] = "0.0.0.0:8075" + gitaly['auth_token'] = 'abc123secret' + + gitaly['storage'] = [ + { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' }, + { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' }, + ] + + # To use tls for gitaly you need to add + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` + +Again, reconfigure (Omnibus) or restart (source). + +Continue configuration of other components by going back to: + +- [Scaled Architectures](./README.md#scalable-architecture-examples) +- [High Availability Architectures](./README.md#high-availability-architecture-examples) diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index d95c3acec54..888426ece5c 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -1,8 +1,4 @@ -# Configuring GitLab for HA - -Assuming you have already configured a [database](database.md), [Redis](redis.md), and [NFS](nfs.md), you can -configure the GitLab application server(s) now. Complete the steps below -for each GitLab application server in your environment. +# Configuring GitLab Scaling and High Availability > **Note:** There is some additional configuration near the bottom for additional GitLab application servers. It's important to read and understand diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md new file mode 100644 index 00000000000..a8bc101dee6 --- /dev/null +++ b/doc/administration/high_availability/nfs_host_client_setup.md @@ -0,0 +1,135 @@ +# Configuring NFS for GitLab HA + +Setting up NFS for a GitLab HA setup allows all applications nodes in a cluster +to share the same files and maintain data consistency. Application nodes in an HA +setup act as clients while the NFS server plays host. + +> Note: The instructions provided in this documentation allow for setting a quick +proof of concept but will leave NFS as potential single point of failure and +therefore not recommended for use in production. Explore options such as [Pacemaker +and Corosync](http://clusterlabs.org/) for highly available NFS in production. + +Below are instructions for setting up an application node(client) in an HA cluster +to read from and write to a central NFS server(host). + +NOTE: **Note:** +Using EFS may negatively impact performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for additional details. + +## NFS Server Setup + +> Follow the instructions below to set up and configure your NFS server. + +### Step 1 - Install NFS Server on Host + +Installing the nfs-kernel-server package allows you to share directories with the clients running the GitLab application. + +```sh +apt-get update +apt-get install nfs-kernel-server +``` + +### Step 2 - Export Host's Home Directory to Client + +In this setup we will share the home directory on the host with the client. Edit the exports file as below to share the host's home directory with the client. If you have multiple clients running GitLab you must enter the client IP addresses in line in the `/etc/exports` file. + +```text +#/etc/exports for one client +/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) + +#/etc/exports for three clients +/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-2-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-3-ip-address>(rw,sync,no_root_squash,no_subtree_check) +``` + +Restart the NFS server after making changes to the `exports` file for the changes +to take effect. + +```sh +systemctl restart nfs-kernel-server +``` + +NOTE: **Note:** +You may need to update your server's firewall. See the [firewall section](#nfs-in-a-firewalled-environment) at the end of this guide. + +## Client/ GitLab application node Setup + +> Follow the instructions below to connect any GitLab rails application node running +inside your HA environment to the NFS server configured above. + +### Step 1 - Install NFS Common on Client + +The nfs-common provides NFS functionality without installing server components which +we don't need running on the application nodes. + +```sh +apt-get update +apt-get install nfs-common +``` + +### Step 2 - Create Mount Points on Client + +Create a directroy on the client that we can mount the shared directory from the host. +Please note that if your mount point directory contains any files they will be hidden +once the remote shares are mounted. An empty/new directory on the client is recommended +for this purpose. + +```sh +mkdir -p /nfs/home +``` + +Confirm that the mount point works by mounting it on the client and checking that +it is mounted with the command below: + +```sh +mount <host_ip_address>:/home +df -h +``` + +### Step 3 - Set up Automatic Mounts on Boot + +Edit `/etc/fstab` on client as below to mount the remote shares automatically at boot. +Note that GitLab requires advisory file locking, which is only supported natively in +NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used. +We recommend using version 4 and do not specifically test NFSv3. + +```text +#/etc/fstab +165.227.159.85:/home /nfs/home nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +``` + +Reboot the client and confirm that the mount point is mounted automatically. + +### Step 4 - Set up GitLab to Use NFS mounts + +When using the default Omnibus configuration you will need to share 5 data locations +between all GitLab cluster nodes. No other locations should be shared. Changing the +default file locations in `gitlab.rb` on the client allows you to have one main mount +point and have all the required locations as subdirectories to use the NFS mount for +git-data. + +```text +git_data_dirs({"default" => {"path" => "/nfs/home/var/opt/gitlab-data/git-data"}}) +gitlab_rails['uploads_directory'] = '/nfs/home/var/opt/gitlab-data/uploads' +gitlab_rails['shared_path'] = '/nfs/home/var/opt/gitlab-data/shared' +gitlab_ci['builds_directory'] = '/nfs/home/var/opt/gitlab-data/builds' +``` + +Save the changes in `gitlab.rb` and run `gitlab-ctl reconfigure`. + +## NFS in a Firewalled Environment + +If the traffic between your NFS server and NFS client(s) is subject to port filtering +by a firewall, then you will need to reconfigure that firewall to allow NFS communication. + +[This guide from TDLP](http://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) +covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to +search for and review the specific documentation for your OS/distro and your firewall software. + +Example for Ubuntu: + +Check that NFS traffic from the client is allowed by the firewall on the host by running +the command: `sudo ufw status`. If it's being blocked, then you can allow traffic from a specific +client with the command below. + +```sh +sudo ufw allow from <client-ip-address> to any port nfs +``` diff --git a/doc/administration/high_availability/pg_ha_architecture.png b/doc/administration/high_availability/pg_ha_architecture.png Binary files differnew file mode 100644 index 00000000000..ef870f652ae --- /dev/null +++ b/doc/administration/high_availability/pg_ha_architecture.png diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md new file mode 100644 index 00000000000..762179cf756 --- /dev/null +++ b/doc/administration/high_availability/pgbouncer.md @@ -0,0 +1,132 @@ +# Working with the bundle Pgbouncer service + +## Overview + +As part of its High Availability stack, GitLab Premium includes a bundled version of [Pgbouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`. + +In a High Availability setup, Pgbouncer is used to seamlessly migrate database connections between servers in a failover scenario. + +Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage. + +It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on its own dedicated node in a cluster. + +## Operations + +### Running Pgbouncer as part of an HA GitLab installation + +See our [HA documentation for PostgreSQL](database.md) for information on running pgbouncer as part of a HA setup + +### Running Pgbouncer as part of a non-HA GitLab installation + +1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer` + +1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab`. We'll also need to enter the plaintext SQL_USER_PASSWORD later + +1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` + + ```ruby + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' + postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' + postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on + postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node + ``` + +1. Run `gitlab-ctl reconfigure` + + **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. + +1. On the node you are running pgbouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + pgbouncer['enable'] = true + pgbouncer['databases'] = { + gitlabhq_production: { + host: 'DATABASE_HOST', + user: 'pgbouncer', + password: 'PGBOUNCER_USER_PASSWORD_HASH' + } + } + ``` + +1. Run `gitlab-ctl reconfigure` + +1. On the node running unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + gitlab_rails['db_host'] = 'PGBOUNCER_HOST' + gitlab_rails['db_port'] = '6432' + gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' + ``` + +1. Run `gitlab-ctl reconfigure` + +1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section + +### Interacting with pgbouncer + +#### Administrative console + +As part of omnibus-gitlab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the pgbouncer administrative console. Please see the [pgbouncer documentation](https://pgbouncer.github.io/usage.html#admin-console) for detailed instructions on how to interact with the console. + +To start a session, run + +```shell +# gitlab-ctl pgb-console +Password for user pgbouncer: +psql (9.6.8, server 1.7.2/bouncer) +Type "help" for help. + +pgbouncer=# +``` + +The password you will be prompted for is the PGBOUNCER_USER_PASSWORD + +To get some basic information about the instance, run +```shell +pgbouncer=# show databases; show clients; show servers; + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections +---------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production | | 100 | 5 | | 0 | 1 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 +(2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link +| remote_pid | tls +------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------ ++------------+----- + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 | +| 0 | + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 44660 | 127.0.0.1 | 6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 | +| 0 | +(8 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | rem +ote_pid | tls +------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+---- +--------+----- + S | gitlab | gitlabhq_production | idle | 127.0.0.1 | 5432 | 127.0.0.1 | 35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 | | + 19980 | +(1 row) +``` + +## Troubleshooting + +In case you are experiencing any issues connecting through pgbouncer, the first place to check is always the logs: + +```shell +# gitlab-ctl tail pgbouncer +``` + +Additionally, you can check the output from `show databases` in the [Administrative console](#administrative-console). In the output, you would expect to see values in the `host` field for the `gitlabhq_production` database. Additionally, `current_connections` should be greater than 1. diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 3daebc4d84b..46ad3ecd9bb 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,6 +1,103 @@ -# Configuring Redis for GitLab HA +# Configuring Redis for Scaling and High Availability -> Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. +## Provide your own Redis instance **[CORE ONLY]** + +The following are the requirements for providing your own Redis instance: + +- Redis version 2.8 or higher. Version 3.2 or higher is recommend as this is + what ships with the GitLab Omnibus package. +- Standalone Redis or Redis high availability with Sentinel are supported. Redis + Cluster is not supported. +- Managed Redis from cloud providers such as AWS Elasticache will work. If these + services support high availability, be sure it is not the Redis Cluster type. + +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the GitLab application servers later. + +## Redis in a Scaled Environment + +This section is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples) +environments including [Basic Scaling](./README.md#basic-scaling) and +[Full Scaling](./README.md#full-scaling). + +### Provide your own Redis instance **[CORE ONLY]** + +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. + +### Standalone Redis using GitLab Omnibus **[CORE ONLY]** + +The GitLab Omnibus package can be used to configure a standalone Redis server. +In this configuration Redis is not highly available, and represents a single +point of failure. However, in a scaled environment the objective is to allow +the environment to handle more users or to increase throughput. Redis itself +is generally stable and can handle many requests so it is an acceptable +trade off to have only a single instance. See [Scaling and High Availability](./README.md) +for an overview of GitLab scaling and high availability options. + +The steps below are the minimum necessary to configure a Redis server with +Omnibus: + +1. SSH into the Redis server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + ## Enable Redis + redis['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + gitlab_monitor['enable'] = false + gitaly['enable'] = false + + redis['bind'] = '0.0.0.0' + redis['port'] = '6379' + redis['password'] = 'SECRET_PASSWORD_HERE' + + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +1. Note the Redis node's IP address or hostname, port, and + Redis password. These will be necessary when configuring the GitLab + application servers later. + +Advanced configuration options are supported and can be added if +needed. + +Continue configuration of other components by going +[back to Scaled Architectures](./README.md#scalable-architecture-examples) + +## Redis with High Availability + +This section is relevant for [High Availability Architecture](./README.md#high-availability-architecture-examples) +environments including [Horizontal](./README.md#horizontal), +[Hybrid](./README.md#hybrid), and +[Fully Distributed](./README.md#fully-distributed). + +### Provide your own Redis instance **[CORE ONLY]** + +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. + +### High Availability with GitLab Omnibus **[PREMIUM ONLY]** + +> Experimental Redis Sentinel support was [introduced in GitLab 8.11][ce-1877]. Starting with 8.14, Redis Sentinel is no longer experimental. If you've used it with versions `< 8.14` before, please check the updated documentation here. @@ -53,8 +150,6 @@ failure. Make sure that you read this document once as a whole before configuring the components below. -### High Availability with Sentinel - > **Notes:** > > - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel @@ -270,10 +365,9 @@ The prerequisites for a HA Redis setup are the following: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Enable the master role and disable all other services in the machine - # (you can still enable Sentinel). - redis_master_role['enable'] = true - + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -287,6 +381,7 @@ The prerequisites for a HA Redis setup are the following: # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'redis-password-goes-here' ``` + 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following @@ -298,6 +393,10 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high +> availability roles at https://docs.gitlab.com/omnibus/roles/ + ### Step 2. Configuring the slave Redis instances 1. SSH into the **slave** Redis server. @@ -310,11 +409,9 @@ The prerequisites for a HA Redis setup are the following: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Enable the slave role and disable all other services in the machine - # (you can still enable Sentinel). This will also set automatically - # `redis['master'] = false`. - redis_slave_role['enable'] = true - + # Specify server role as 'redis_slave_role' + roles ['redis_slave_role'] + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -336,17 +433,19 @@ The prerequisites for a HA Redis setup are the following: #redis['master_port'] = 6379 ``` -1. To prevent database migrations from running on upgrade, run: +1. To prevent reconfigure from running automatically on upgrade, run: ``` sudo touch /etc/gitlab/skip-auto-reconfigure ``` - Only the primary GitLab application server should handle migrations. - 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high +> availability roles at https://docs.gitlab.com/omnibus/roles/ + --- These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -400,13 +499,13 @@ multiple machines with the Sentinel daemon. be duplicate below): ```ruby - redis_sentinel_role['enable'] = true + roles ['redis_sentinel_role'] # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' # The same password for Redis authentication you set up for the master node. - redis['password'] = 'redis-password-goes-here' + redis['master_password'] = 'redis-password-goes-here' # The IP of the master Redis node. redis['master_ip'] = '10.0.0.1' @@ -573,8 +672,7 @@ or a failover promotes a different **Master** node. In `/etc/gitlab/gitlab.rb`: ```ruby -redis_master_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_master_role'] redis['bind'] = '10.0.0.1' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -596,8 +694,7 @@ sentinel['quorum'] = 2 In `/etc/gitlab/gitlab.rb`: ```ruby -redis_slave_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_slave_role'] redis['bind'] = '10.0.0.2' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -619,8 +716,7 @@ sentinel['quorum'] = 2 In `/etc/gitlab/gitlab.rb`: ```ruby -redis_slave_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_slave_role'] redis['bind'] = '10.0.0.3' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -643,7 +739,7 @@ In `/etc/gitlab/gitlab.rb`: ```ruby redis['master_name'] = 'gitlab-redis' -redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' gitlab_rails['redis_sentinels'] = [ {'host' => '10.0.0.1', 'port' => 26379}, {'host' => '10.0.0.2', 'port' => 26379}, @@ -764,15 +860,11 @@ Before proceeding with the troubleshooting below, check your firewall rules: ### Troubleshooting Redis replication You can check if everything is correct by connecting to each server using -`redis-cli` application, and sending the `INFO` command. +`redis-cli` application, and sending the `info replication` command as below. -If authentication was correctly defined, it should fail with: -`NOAUTH Authentication required` error. Try to authenticate with the -previous defined password with `AUTH redis-password-goes-here` and -try the `INFO` command again. - -Look for the `# Replication` section where you should see some important -information like the `role` of the server. +``` +/opt/gitlab/embedded/bin/redis-cli -a <redis-password> info replication +``` When connected to a `master` redis, you will see the number of connected `slaves`, and a list of each with connection details: @@ -842,7 +934,7 @@ To make sure your configuration is correct: 1. Run in the console: ```ruby - redis = Redis.new(Gitlab::Redis.params) + redis = Redis.new(Gitlab::Redis::SharedState.params) redis.info ``` diff --git a/doc/administration/img/db_load_balancing_postgres_stats.png b/doc/administration/img/db_load_balancing_postgres_stats.png Binary files differnew file mode 100644 index 00000000000..8b311616e7b --- /dev/null +++ b/doc/administration/img/db_load_balancing_postgres_stats.png diff --git a/doc/administration/img/high_availability/active-active-diagram.png b/doc/administration/img/high_availability/active-active-diagram.png Binary files differdeleted file mode 100644 index 4f5984b88fe..00000000000 --- a/doc/administration/img/high_availability/active-active-diagram.png +++ /dev/null diff --git a/doc/administration/img/high_availability/active-passive-diagram.png b/doc/administration/img/high_availability/active-passive-diagram.png Binary files differdeleted file mode 100644 index 3b42ce5911c..00000000000 --- a/doc/administration/img/high_availability/active-passive-diagram.png +++ /dev/null diff --git a/doc/administration/img/high_availability/fully-distributed.png b/doc/administration/img/high_availability/fully-distributed.png Binary files differnew file mode 100644 index 00000000000..ad23207134e --- /dev/null +++ b/doc/administration/img/high_availability/fully-distributed.png diff --git a/doc/administration/img/high_availability/geo-ha-diagram.png b/doc/administration/img/high_availability/geo-ha-diagram.png Binary files differnew file mode 100644 index 00000000000..da5d612827c --- /dev/null +++ b/doc/administration/img/high_availability/geo-ha-diagram.png diff --git a/doc/administration/img/high_availability/horizontal.png b/doc/administration/img/high_availability/horizontal.png Binary files differnew file mode 100644 index 00000000000..c3bd489d96f --- /dev/null +++ b/doc/administration/img/high_availability/horizontal.png diff --git a/doc/administration/img/high_availability/hybrid.png b/doc/administration/img/high_availability/hybrid.png Binary files differnew file mode 100644 index 00000000000..7d4a56bf0ea --- /dev/null +++ b/doc/administration/img/high_availability/hybrid.png diff --git a/doc/administration/img/instance_review_button.png b/doc/administration/img/instance_review_button.png Binary files differnew file mode 100644 index 00000000000..b7604d7c7e5 --- /dev/null +++ b/doc/administration/img/instance_review_button.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 658b2f55d30..9ac310af248 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -10,6 +10,8 @@ GitLab has several features based on receiving incoming emails: - [New merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email): allow GitLab users to create a new merge request by sending an email to a user-specific email address. +- [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html): provide e-mail support to + your customers through GitLab. **[PREMIUM]** ## Requirements diff --git a/doc/administration/index.md b/doc/administration/index.md index 5f368ea8d49..02e88dbd2a6 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -32,8 +32,14 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Installing GitLab - [Install](../install/README.md): Requirements, directory structures, and installation methods. + - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **[STARTER ONLY]** + - [Omnibus support for external MySQL DB](https://docs.gitlab.com/omnibus/settings/database.html#using-a-mysql-database-management-server-enterprise-edition-only): Omnibus package supports configuring an external MySQL database. **[STARTER ONLY]** + - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **[STARTER ONLY]** - [High Availability](high_availability/README.md): Configure multiple servers for scaling or high availability. - [High Availability on AWS](../university/high-availability/aws/README.md): Set up GitLab HA on Amazon AWS. +- [Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **[PREMIUM ONLY]** +- [Disaster Recovery](https://docs.gitlab.com/ee/administration/geo/disaster_recovery/index.html): Quickly fail-over to a different site with minimal effort in a disaster situation. **[PREMIUM ONLY]** +- [Pivotal Tile](https://docs.gitlab.com/ee/install/pivotal/index.html): Deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **[PREMIUM ONLY]** ### Configuring GitLab @@ -44,8 +50,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Global user settings](user_settings.md): Configure instance-wide user permissions. - [Polling](polling.md): Configure how often the GitLab UI polls for updates. - [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. -- [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on - [source installations](../install/installation.md#installation-from-source). +- [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). +- [Uploads configuration](uploads.md): Configure GitLab uploads storage. - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. - [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) @@ -54,6 +60,9 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages. - [Merge request diffs storage](merge_request_diffs.md): Configure merge requests diffs external storage. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. +- [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html): Enable Elasticsearch to empower GitLab's Advanced Global Search. Useful when you deal with a huge amount of data. **[STARTER ONLY]** +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **[PREMIUM ONLY]** +- [Upload a license](https://docs.gitlab.com/ee/user/admin_area/license.html): Upload a license to unlock features that are in paid tiers of GitLab. **[STARTER ONLY]** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. #### Customizing GitLab's appearance @@ -63,6 +72,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Branded login page](../customization/branded_login_page.md): Customize the login page with your own logo, title, and description. - [Welcome message](../customization/welcome_message.md): Add a custom welcome message to the sign-in page. - ["New Project" page](../customization/new_project_page.md): Customize the text to be displayed on the page that opens whenever your users create a new project. +- [Additional custom email text](https://docs.gitlab.com/ee/user/admin_area/settings/email.html#custom-additional-text-premium-only): Add additional custom text to emails sent from GitLab. **[PREMIUM ONLY]** ### Maintaining GitLab @@ -96,7 +106,16 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Libravatar](../customization/libravatar.md): Use Libravatar instead of Gravatar for user avatars. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). -- [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). +- [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. + - [Sync LDAP](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html) **[STARTER ONLY]** + - [Kerberos authentication](https://docs.gitlab.com/ee/integration/kerberos.html) **[STARTER ONLY]** + - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). +- [Email users](https://docs.gitlab.com/ee/tools/email.html): Email GitLab users from within GitLab. **[STARTER ONLY]** +- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. +- [Audit logs and events](audit_events.md): View the changes made within the GitLab server for: + - Groups and projects. **[STARTER]** + - Instances. **[PREMIUM ONLY]** +- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **[PREMIUM ONLY]** - [Incoming email](incoming_email.md): Configure incoming emails to allow users to [reply by email], create [issues by email] and [merge requests by email], and to enable [Service Desk]. @@ -108,6 +127,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. [reply by email]: reply_by_email.md [issues by email]: ../user/project/issues/create_new_issue.md#new-issue-via-email [merge requests by email]: ../user/project/merge_requests/index.md#create-new-merge-requests-by-email +[Service Desk]: https://docs.gitlab.com/ee/user/project/service_desk.html ## Project settings @@ -116,13 +136,15 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Gitaly](gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service. - [Default labels](../user/admin_area/labels.html): Create labels that will be automatically added to every new project. - [Restrict the use of public or internal projects](../public_access/public_access.md#restricting-the-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. -- [Custom project templates](https://docs.gitlab.com/ee/user/admin_area/custom_project_templates.html): Configure a set of projects to be used as custom templates when creating a new project. **[PREMIUM ONLY]** +- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. **[PREMIUM ONLY]** +- [Packages](packages.md): Enable GitLab to act as a Maven repository or NPM registry. **[PREMIUM ONLY]** ### Repository settings - [Repository checks](repository_checks.md): Periodic Git repository checks. - [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. - [Repository storage rake tasks](raketasks/storage.md): A collection of rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. +- [Limit repository size](https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html): Set a hard limit for your repositories' size. **[STARTER ONLY]** ## Continuous Integration settings @@ -159,6 +181,10 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Request Profiling](monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. - [Performance Bar](monitoring/performance/performance_bar.md): Get performance information for the current page. +## Analytics + +- [Pseudonymizer](pseudonymizer.md): Export data from GitLab's database to CSV files in a secure way. **[ULTIMATE]** + ## Troubleshooting - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md new file mode 100644 index 00000000000..b1244f44e95 --- /dev/null +++ b/doc/administration/instance_review.md @@ -0,0 +1,17 @@ +# Instance Review **[CORE ONLY]** + +> [Introduced][6995] in [GitLab Core][ee] 11.3. + +If you are running a medium size instance of GitLab Core edition you are qualified for a free Instance Review. You can find the button in the User menu. + + + +When you click the button you will be redirected to a form with prefilled data obtained from your instance. + +Once you submit the data to GitLab Inc. you can see the initial report. + +Additionally you will be contacted by our team for further review which should help you to improve your usage of GitLab. + +[6995]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6995 +[ee]: https://about.gitlab.com/pricing/ + diff --git a/doc/administration/maven_packages.md b/doc/administration/maven_packages.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/maven_repository.md b/doc/administration/maven_repository.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/maven_repository.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 3bfcc9a289e..c243dd9edbb 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -48,6 +48,47 @@ The following metrics are available: | unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | | unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | +## Sidekiq Metrics available for Geo **[PREMIUM]** + +Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled (e.g. via +the `monitoring.sidekiq_exporter` configuration option in `gitlab.yml`. + +| Metric | Type | Since | Description | Labels | +|:-------------------------------------------- |:------- |:----- |:----------- |:------ | +| geo_db_replication_lag_seconds | Gauge | 10.2 | Database replication lag (seconds) | url +| geo_repositories | Gauge | 10.2 | Total number of repositories available on primary | url +| geo_repositories_synced | Gauge | 10.2 | Number of repositories synced on secondary | url +| geo_repositories_failed | Gauge | 10.2 | Number of repositories failed to sync on secondary | url +| geo_lfs_objects | Gauge | 10.2 | Total number of LFS objects available on primary | url +| geo_lfs_objects_synced | Gauge | 10.2 | Number of LFS objects synced on secondary | url +| geo_lfs_objects_failed | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url +| geo_attachments | Gauge | 10.2 | Total number of file attachments available on primary | url +| geo_attachments_synced | Gauge | 10.2 | Number of attachments synced on secondary | url +| geo_attachments_failed | Gauge | 10.2 | Number of attachments failed to sync on secondary | url +| geo_last_event_id | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url +| geo_last_event_timestamp | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url +| geo_cursor_last_event_id | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url +| geo_cursor_last_event_timestamp | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url +| geo_status_failed_total | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url +| geo_last_successful_status_check_timestamp | Gauge | 10.2 | Last timestamp when the status was successfully updated | url +| geo_lfs_objects_synced_missing_on_primary | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url +| geo_job_artifacts_synced_missing_on_primary | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url +| geo_attachments_synced_missing_on_primary | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url +| geo_repositories_checksummed_count | Gauge | 10.7 | Number of repositories checksummed on primary | url +| geo_repositories_checksum_failed_count | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url +| geo_wikis_checksummed_count | Gauge | 10.7 | Number of wikis checksummed on primary | url +| geo_wikis_checksum_failed_count | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url +| geo_repositories_verified_count | Gauge | 10.7 | Number of repositories verified on secondary | url +| geo_repositories_verification_failed_count | Gauge | 10.7 | Number of repositories failed to verify on secondary | url +| geo_repositories_checksum_mismatch_count | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url +| geo_wikis_verified_count | Gauge | 10.7 | Number of wikis verified on secondary | url +| geo_wikis_verification_failed_count | Gauge | 10.7 | Number of wikis failed to verify on secondary | url +| geo_wikis_checksum_mismatch_count | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url +| geo_repositories_checked_count | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url +| geo_repositories_checked_failed_count | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url +| geo_repositories_retrying_verification_count | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url +| geo_wikis_retrying_verification_count | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url + ### Ruby metrics Some basic Ruby runtime metrics are available: diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 222ac0b3ae1..ce65d77274b 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -217,6 +217,12 @@ The Postgres exporter allows you to measure various PostgreSQL metrics. [➔ Read more about the Postgres exporter.](postgres_exporter.md) +### PgBouncer exporter + +The PgBouncer exporter allows you to measure various PgBouncer metrics. + +[➔ Read more about the PgBouncer exporter.](pgbouncer_exporter.md) + ### GitLab monitor exporter The GitLab monitor exporter allows you to measure various GitLab metrics, pulled from Redis and the database. diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md new file mode 100644 index 00000000000..d76834fdbea --- /dev/null +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -0,0 +1,34 @@ +# PgBouncer exporter + +>**Note:** +Available since [Omnibus GitLab 11.0][2493]. For installations from source +you'll have to install and configure it yourself. + +The [PgBouncer exporter] allows you to measure various PgBouncer metrics. + +To enable the PgBouncer exporter: + +1. [Enable Prometheus](index.md#configuring-prometheus) +1. Edit `/etc/gitlab/gitlab.rb` +1. Add or find and uncomment the following line, making sure it's set to `true`: + + ```ruby + pgbouncer_exporter['enable'] = true + ``` + +1. Save the file and [reconfigure GitLab][reconfigure] for the changes to + take effect. + +Prometheus will now automatically begin collecting performance data from +the PgBouncer exporter exposed under `localhost:9188`. + +The PgBouncer exporter will also be enabled by default if the [pgbouncer_role][postgres roles] +is enabled. + +[← Back to the main Prometheus page](index.md) + +[2493]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/2493 +[PgBouncer exporter]: https://github.com/stanhu/pgbouncer_exporter +[postgres roles]: https://docs.gitlab.com/omnibus/roles/#postgres-roles +[prometheus]: https://prometheus.io +[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/npm_registry.md b/doc/administration/npm_registry.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/npm_registry.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index b45ca99fd80..20c19445404 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -27,7 +27,7 @@ rcli() { # This example works for Omnibus installations of GitLab 7.3 or newer. For an # installation from source you will have to change the socket path and the # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.shared_state.socket "$@" } # test the new shell function; the response should be PONG diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md new file mode 100644 index 00000000000..286b99aceb5 --- /dev/null +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -0,0 +1,218 @@ +# Extra Sidekiq processes **[STARTER ONLY]** + +GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes +besides the default one. These processes can be used to consume a dedicated set +of queues. This can be used to ensure certain queues always have dedicated +workers, no matter the number of jobs that need to be processed. + +## Starting extra processes via Omnibus GitLab + +To enable `sidekiq-cluster`, you must apply the `sidekiq_cluster['enable'] = true` +setting `/etc/gitlab/gitlab.rb`: + +```ruby +sidekiq_cluster['enable'] = true +``` + +You will then specify how many additional processes to create via `sidekiq-cluster` +as well as which queues for them to handle. This is done via the +`sidekiq_cluster['queue_groups']` setting. This is an array whose items contain +which queues to process. Each item in the array will equate to one additional +sidekiq process. + +As an example, to make additional sidekiq processes that process the +`elastic_indexer` and `mailers` queues, you would apply the following: + +```ruby +sidekiq_cluster['queue_groups'] = [ + "elastic_indexer", + "mailers" +] +``` + +To have an additional sidekiq process handle multiple queues, you simply put a +comma after the first queue name and then put the next queue name: + +```ruby +sidekiq_cluster['queue_groups'] = [ + "elastic_indexer,elastic_commit_indexer", + "mailers" +] +``` + +Keep in mind, all changes must be followed by reconfiguring your GitLab +application via `sudo gitlab-ctl reconfigure`. + +### Monitoring + +Once the Sidekiq processes are added, you can visit the "Background Jobs" +section under the admin area in GitLab (`/admin/background_jobs`). + + + +### All queues with exceptions + +To have the additional sidekiq processes work on every queue EXCEPT the ones +you list: + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['negate'] = true + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + +### Limiting concurrency + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['concurrency'] = 25 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +Keep in mind, this normally would not exceed the number of CPU cores available. + +### Modifying the check interval + +To modify the check interval for the additional Sidekiq processes: + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['interval'] = 5 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +This tells the additional processes how often to check for enqueued jobs. + +## Starting extra processes via command line + +Starting extra Sidekiq processes can be done using the command +`/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster`. This command +takes arguments using the following syntax: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...] +``` + +Each separate argument denotes a group of queues that have to be processed by a +Sidekiq process. Multiple queues can be processed by the same process by +separating them with a comma instead of a space. + +Instead of a queue, a queue namespace can also be provided, to have the process +automatically listen on all queues in that namespace without needing to +explicitly list all the queue names. For more information about queue namespaces, +see the relevant section in the +[Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). + +For example, say you want to start 2 extra processes: one to process the +"process_commit" queue, and one to process the "post_receive" queue. This can be +done as follows: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit post_receive +``` + +If you instead want to start one process processing both queues you'd use the +following syntax: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive +``` + +If you want to have one Sidekiq process process the "process_commit" and +"post_receive" queues, and one process to process the "gitlab_shell" queue, +you'd use the following: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive gitlab_shell +``` + +### Monitoring + +The `sidekiq-cluster` command will not terminate once it has started the desired +amount of Sidekiq processes. Instead, the process will continue running and +forward any signals to the child processes. This makes it easy to stop all +Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process, +instead of having to send it to the individual processes. + +If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child +processes will terminate themselves after a few seconds. This ensures you don't +end up with zombie Sidekiq processes. + +All of this makes monitoring the processes fairly easy. Simply hook up +`sidekiq-cluster` to your supervisor of choice (e.g. runit) and you're good to +go. + +If a child process died the `sidekiq-cluster` command will signal all remaining +process to terminate, then terminate itself. This removes the need for +`sidekiq-cluster` to re-implement complex process monitoring/restarting code. +Instead you should make sure your supervisor restarts the `sidekiq-cluster` +process whenever necessary. + +### PID files + +The `sidekiq-cluster` command can store its PID in a file. By default no PID +file is written, but this can be changed by passing the `--pidfile` option to +`sidekiq-cluster`. For example: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit +``` + +Keep in mind that the PID file will contain the PID of the `sidekiq-cluster` +command and not the PID(s) of the started Sidekiq processes. + +### Environment + +The Rails environment can be set by passing the `--environment` flag to the +`sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The +default value is "development". + +### All queues with exceptions + +You're able to run all queues in `sidekiq_queues.yml` file on a single or +multiple processes with exceptions using the `--negate` flag. + +For example, say you want to run a single process for all queues, +except "process_commit" and "post_receive". You can do so by executing: + +```bash +sidekiq-cluster process_commit,post_receive --negate +``` + +For multiple processes of all queues (except "process_commit" and "post_receive"): + +```bash +sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate +``` + +### Limiting concurrency + +By default, `sidekiq-cluster` will spin up extra Sidekiq processes that use +one thread per queue up to a maximum of 50. If you wish to change the cap, use +the `-m N` option. For example, this would cap the maximum number of threads to 1: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive -m 1 +``` + +For each queue group, the concurrency factor will be set to min(number of +queues, N). Setting the value to 0 will disable the limit. + +Note that each thread requires a Redis connection, so adding threads may +increase Redis latency and potentially cause client timeouts. See the [Sidekiq +documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) +for more details. + +## Number of threads + +Each process started using `sidekiq-cluster` (whether it be via command line or +via the gitlab.rb file) starts with a number of threads that equals the number +of queues, plus one spare thread. For example, a process that handles the +"process_commit" and "post_receive" queues will use 3 threads in total. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index f8d58c6ce28..69f110805b7 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -30,6 +30,19 @@ instructions will break installations using older versions of OpenSSH, such as those included with CentOS 6 as of September 2017. If you want to use this feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing. +## Fast lookup is required for Geo **[PREMIUM]** + +By default, GitLab manages an `authorized_keys` file, which contains all the +public SSH keys for users allowed to access GitLab. However, to maintain a +single source of truth, [Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) needs to be configured to perform SSH fingerprint +lookups via database lookup. + +As part of [setting up Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html#setup-instructions), +you will be required to follow the steps outlined below for both the primary and +secondary nodes, but note that the `Write to "authorized keys" file` checkbox +only needs to be unchecked on the primary node since it will be reflected +automatically on the secondary if database replication is working. + ## Setting up fast lookup via GitLab Shell GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup @@ -69,7 +82,7 @@ file will still be scanned. So git SSH performance will still be slow for many users as long as a large file exists. You can disable any more writes to the `authorized_keys` file by unchecking -`Write to "authorized_keys" file` in the Application Settings of your GitLab +`Write to "authorized_keys" file` in the **Admin Area > Settings > Network > Performance optimization** of your GitLab installation.  diff --git a/doc/administration/operations/img/sidekiq-cluster.png b/doc/administration/operations/img/sidekiq-cluster.png Binary files differnew file mode 100644 index 00000000000..4eb1849010e --- /dev/null +++ b/doc/administration/operations/img/sidekiq-cluster.png diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 32f36d68c50..df795a48169 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -11,6 +11,7 @@ Keep your GitLab instance up and running smoothly. by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. +- [Extra Sidekiq operations](extra_sidekiq_processes.md): Configure an extra set of Sidekiq processes to ensure certain queues always have dedicated workers, no matter the amount of jobs that need to be processed. **[STARTER ONLY]** - [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or diff --git a/doc/administration/packages.md b/doc/administration/packages.md new file mode 100644 index 00000000000..5b9a13e3859 --- /dev/null +++ b/doc/administration/packages.md @@ -0,0 +1,174 @@ +# GitLab Packages administration **[PREMIUM ONLY]** + +GitLab Packages allows organizations to utilize GitLab as a private repository +for a variety of common package managers. Users are able to build and publish +packages, which can be easily consumed as a dependency in downstream projects. + +The Packages feature allows GitLab to act as a repository for the following: + +| Software repository | Description | Available in GitLab version | +| ------------------- | ----------- | --------------------------- | +| [Maven Repository](https://docs.gitlab.com/ee/user/project/packages/maven_repository.html) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | +| [NPM Registry](https://docs.gitlab.com/ee/user/project/packages/npm_registry.html) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | + +Don't you see your package management system supported yet? +Please consider contributing +to GitLab. This [development documentation](https://docs.gitlab.com/ee/development/packages.html) will guide you through the process. + +## Enabling the Packages feature + +NOTE: **Note:** +After the Packages feature is enabled, the repositories are available +for all new projects by default. To enable it for existing projects, users will +have to explicitly do so in the project's settings. + +To enable the Packages feature: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['packages_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. After the installation is complete, you will have to configure the `packages` + section in `config/gitlab.yml`. Set to `true` to enable it: + + ```yaml + packages: + enabled: true + ``` +1. [Restart GitLab] for the changes to take effect. + +## Changing the storage path + +By default, the packages are stored locally, but you can change the default +local location or even use object storage. + +### Changing the local storage path + +The packages for Omnibus GitLab installations are stored under +`/var/opt/gitlab/gitlab-rails/shared/packages/` and for source +installations under `shared/packages/` (relative to the git homedir). +To change the local storage path: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['packages_storage_path'] = "/mnt/packages" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `packages` section in `config/gitlab.yml`: + + ```yaml + packages: + enabled: true + storage_path: shared/packages + ``` +1. [Restart GitLab] for the changes to take effect. + +### Using object storage + +Instead of relying on the local storage, you can use an object storage to +upload packages: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where + necessary): + + ```ruby + gitlab_rails['packages_enabled'] = true + gitlab_rails['packages_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/packages" + gitlab_rails['packages_object_store_enabled'] = true + gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name. + gitlab_rails['packages_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + gitlab_rails['packages_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). + gitlab_rails['packages_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + gitlab_rails['packages_object_store_connection'] = { + ## + ## If the provider is AWS S3, uncomment the following + ## + #'provider' => 'AWS', + #'region' => 'eu-west-1', + #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY', + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #'host' => 's3.amazonaws.com', + #'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `packages` section in `config/gitlab.yml` (uncomment where necessary): + + ```yaml + packages: + enabled: true + ## + ## The location where build packages are stored (default: shared/packages). + ## + #storage_path: shared/packages + object_store: + enabled: false + remote_directory: packages # The bucket name. + #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + #background_upload: true # Temporary option to limit automatic upload (Default: true). + #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + connection: + ## + ## If the provider is AWS S3, uncomment the following + ## + #provider: AWS + #region: us-east-1 + #aws_access_key_id: AWS_ACCESS_KEY_ID + #aws_secret_access_key: AWS_SECRET_ACCESS_KEY + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #host: 's3.amazonaws.com' # default: s3.amazonaws.com. + #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + ``` + +1. [Restart GitLab] for the changes to take effect. + +### Migrating local packages to object storage + +After [configuring the object storage](#using-object-storage), you may use the +following task to migrate existing packages from the local storage to the remote one. +The processing will be done in a background worker and requires **no downtime**. + +For Omnibus GitLab: + +```sh +sudo gitlab-rake "gitlab:packages:migrate" +``` + +For installations from source: + +```bash +RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:packages:migrate +``` + +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md new file mode 100644 index 00000000000..036e1d3fe61 --- /dev/null +++ b/doc/administration/pseudonymizer.md @@ -0,0 +1,103 @@ +# Pseudonymizer **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5532) in [GitLab Ultimate][ee] 11.1. + +As GitLab's database hosts sensitive information, using it unfiltered for analytics +implies high security requirements. To help alleviate this constraint, the Pseudonymizer +service is used to export GitLab's data in a pseudonymized way. + +CAUTION: **Warning:** +This process is not impervious. If the source data is available, it's possible for +a user to correlate data to the pseudonymized version. + +The Pseudonymizer currently uses `HMAC(SHA256)` to mutate fields that shouldn't +be textually exported. This ensures that: + +- the end-user of the data source cannot infer/revert the pseudonymized fields +- the referential integrity is maintained + +## Configuration + +To configure the pseudonymizer, you need to: + +- Provide a manifest file that describes which fields should be included or + pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/config/pseudonymizer.yml)). + A default manifest is provided with the GitLab installation. Using a relative file path will be resolved from the Rails root. + Alternatively, you can use an absolute file path. +- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. + +**For Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['pseudonymizer_manifest'] = 'config/pseudonymizer.yml' + gitlab_rails['pseudonymizer_upload_remote_directory'] = 'gitlab-elt' # bucket name + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: **Note:** + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. + + ```ruby + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +--- + +**For installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + pseudonymizer: + manifest: config/pseudonymizer.yml + upload: + remote_directory: 'gitlab-elt' # bucket name + connection: + provider: AWS + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) + for the changes to take effect. + +## Usage + +You can optionally run the pseudonymizer using the following environment variables: + +- `PSEUDONYMIZER_OUTPUT_DIR` - where to store the output CSV files (defaults to `/tmp`) +- `PSEUDONYMIZER_BATCH` - the batch size when querying the DB (defaults to `100000`) + +```bash +## Omnibus +sudo gitlab-rake gitlab:db:pseudonymizer + +## Source +sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production +``` + +This will produce some CSV files that might be very large, so make sure the +`PSEUDONYMIZER_OUTPUT_DIR` has sufficient space. As a rule of thumb, at least +10% of the database size is recommended. + +After the pseudonymizer has run, the output CSV files should be uploaded to the +configured object storage and deleted from the local disk. + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md new file mode 100644 index 00000000000..9f3b31442f3 --- /dev/null +++ b/doc/administration/raketasks/geo.md @@ -0,0 +1,57 @@ +# Geo Rake Tasks **[PREMIUM ONLY]** + +## Git housekeeping + +There are few tasks you can run to schedule a git housekeeping to start at the +next repository sync in a **Secondary node**: + +### Incremental Repack + +This is equivalent of running `git repack -d` on a _bare_ repository. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:incremental_repack +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_ENV=production +``` + +### Full Repack + +This is equivalent of running `git repack -d -A --pack-kept-objects` on a +_bare_ repository which will optionally, write a reachability bitmap index +when this is enabled in GitLab. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:full_repack +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:full_repack RAILS_ENV=production +``` + +### GC + +This is equivalent of running `git gc` on a _bare_ repository, optionally writing +a reachability bitmap index when this is enabled in GitLab. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:gc +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:gc RAILS_ENV=production +``` diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index c39fef907db..42a1a1c2e60 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -43,6 +43,9 @@ If you find it necessary, you can run this migration script again to schedule mi Any error or warning will be logged in Sidekiq's log file. +NOTE: **Note:** +If Geo is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes. + You only need the `gitlab:storage:migrate_to_hashed` rake task to migrate your repositories, but we have additional commands below that helps you inspect projects and attachments in both legacy and hashed storage. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 88bf55296b2..4db3cbb9958 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -47,15 +47,15 @@ Any change in the URL will need to be reflected on disk (when groups / users or projects are renamed). This can add a lot of load in big installations, especially if using any type of network based filesystem. -For GitLab Geo in particular: Geo does work with legacy storage, but in some +CAUTION: **Caution:** +For Geo in particular: Geo does work with legacy storage, but in some edge cases due to race conditions it can lead to errors when a project is renamed multiple times in short succession, or a project is deleted and recreated under the same name very quickly. We expect these race events to be rare, and we have not observed a race condition side-effect happening yet. - This pattern also exists in other objects stored in GitLab, like issue Attachments, GitLab Pages artifacts, Docker Containers for the integrated -Registry, etc. +Registry, etc. Hashed storage is a requirement for Geo. ## Hashed Storage diff --git a/doc/api/project_templates.md b/doc/api/project_templates.md index 3b5b12c8da3..0a94a8d47ae 100644 --- a/doc/api/project_templates.md +++ b/doc/api/project_templates.md @@ -15,7 +15,7 @@ templates are also available from this API endpoint. Support will be added for [Issue and Merge Request templates](../user/project/description_templates.md) in a future release. -Support for [Group-level file templates](../user/group/index.md#group-level-file-templates-premium) +Support for [Group-level file templates](../user/group/index.md#group-file-templates-premium) **[PREMIUM]** was [added](https://gitlab.com/gitlab-org/gitlab-ee/issues/5987) in GitLab 11.5 diff --git a/doc/articles/how_to_configure_ldap_gitlab_ee/index.md b/doc/articles/how_to_configure_ldap_gitlab_ee/index.md new file mode 100644 index 00000000000..4ce96fcb230 --- /dev/null +++ b/doc/articles/how_to_configure_ldap_gitlab_ee/index.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html). diff --git a/doc/ci/README.md b/doc/ci/README.md index 440a79c7782..27bde2ac0f1 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -3,7 +3,7 @@ comments: false description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." --- -# GitLab Continuous Integration (GitLab CI/CD) +# GitLab CI/CD GitLab CI/CD is a tool built into GitLab for software development through the [continuous methodologies](introduction/index.md#introduction-to-cicd-methodologies): @@ -61,12 +61,12 @@ for all the attributes you can set and use. NOTE: **Note:** GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#extra-shared-runners-pipeline-minutes-quota). -## GitLab CI/CD configuration +## Configuration GitLab CI/CD supports numerous configuration options: -| Configuration | Description | -|:--- |:--- | +| Configuration | Description | +|:--------------|:-------------| | [Pipelines](pipelines.md) | Structure your CI/CD process through pipelines. | | [Environment variables](variables/README.md) | Reuse values based on a variable/value key pair. | | [Environments](environments.md) | Deploy your application to different environments (e.g., staging, production). | @@ -74,7 +74,7 @@ GitLab CI/CD supports numerous configuration options: | [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | | [Schedule pipelines](../user/project/pipelines/schedules.md) | Schedule pipelines to run as often as you need. | | [Custom path for `.gitlab-ci.yml`](../user/project/pipelines/settings.md#custom-ci-config-path) | Define a custom path for the CI/CD configuration file. | -| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | +| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules.| | [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | | [Pipelines triggers](triggers/README.md) | Trigger pipelines through the API. | | [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | @@ -85,33 +85,43 @@ GitLab CI/CD supports numerous configuration options: Note that certain operations can only be performed according to the [user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. -## GitLab CI/CD feature set +## Feature set -You can also use the vast GitLab CI/CD feature set to easily configure -it for specific purposes: +Use the vast GitLab CI/CD to easily configure it for specific purposes. +Its feature set is listed on the table below according to DevOps stages. | Feature | Description | -|:--- |:--- | -| [Auto Deploy](../topics/autodevops/index.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | +|:--------|:------------| +| **Configure** || | [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | +| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | +|---+---| +| **Verify** || +| [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html) | Quickly determine the performance impact of pending code changes. | +| [CI services](services/README.md) | Link Docker containers with your base image.| +| [Code Quality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** | Analyze your source code quality. | +| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | +| [Interactive Web Terminals](interactive_web_terminal/index.md) **[CORE ONLY]** | Open an interactive web terminal to debug the running jobs. | +| [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | +| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | +|---+---| +| **Release** || +| [Auto Deploy](../topics/autodevops/index.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | | [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | | [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) **[PREMIUM]** | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | -| [CI services](services/README.md)| Link Docker containers with your base image. | -| [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]**| Check your Docker containers for known vulnerabilities. | -| [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]**| Analyze your dependencies for known vulnerabilities. | | [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | | [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html) **[PREMIUM]** | Deploy your features behind Feature Flags. | -| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | | [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | | [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | -| [Interactive Web Terminals](interactive_web_terminal/index.md) **[CORE ONLY]** | Open an interactive web terminal to debug the running jobs. | -| [JUnit tests](junit_test_reports.md)| Identify script failures directly on merge requests. | | [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | +|---+---| +| **Secure** || +| [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]** | Check your Docker containers for known vulnerabilities.| +| [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [License Management](https://docs.gitlab.com/ee/user/application_security/license_management/) **[ULTIMATE]** | Search your project dependencies for their licenses. | | [Security Test reports](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate) **[ULTIMATE]** | Check for app vulnerabilities. | -| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | -## GitLab CI/CD examples +## Examples GitLab provides examples of configuring GitLab CI/CD in the form of: @@ -120,7 +130,7 @@ GitLab provides examples of configuring GitLab CI/CD in the form of: - [`multi-project-pipelines`](https://gitlab.com/gitlab-examples/multi-project-pipelines) for examples of implementing multi-project pipelines. - [`review-apps-nginx`](https://gitlab.com/gitlab-examples/review-apps-nginx/) provides an example of using Review Apps. -## GitLab CI/CD administration **[CORE ONLY]** +## Administration **[CORE ONLY]** As a GitLab administrator, you can change the default behavior of GitLab CI/CD for: diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md new file mode 100644 index 00000000000..1c8e12d229c --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -0,0 +1,147 @@ +# Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]** + +GitLab CI/CD can be used with Bitbucket Cloud by creating a +[CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) and connecting +your Git repository via URL. + +1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and + create the project. + +  + + GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. + +1. In GitLab create a + [Personal Access Token](../../user/profile/personal_access_tokens.md) + with `api` scope. This will be used to authenticate requests from the web + hook that will be created in Bitbucket to notify GitLab of new commits. + +1. In Bitbucket from **Settings > Webhooks** create a new web hook to notify + GitLab of new commits. + + The web hook URL should be set to the GitLab API to trigger pull mirroring, + using the Personal Access Token we just generated for authentication. + + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + + The web hook Trigger should be set to 'Repository Push'. + +  + + After saving, test the web hook by pushing a change to your Bitbucket + repository. + +1. In Bitbucket create an **App Password** from **Bitbucket Settings > App + Passwords** to authenticate the build status script setting commit build + statuses in Bitbucket. Repository write permissions are required. + +  + +1. In GitLab from **Settings > CI/CD > Environment variables** add variables to allow + communication with Bitbucket via the Bitbucket API. + + `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above + + `BITBUCKET_USERNAME`: the username of the Bitbucket account + + `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ + + `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ + +1. In Bitbucket add a script to push the pipeline status to Bitbucket. + + > Note: changes made in GitLab will be overwritten by any changes made + upstream in Bitbucket. + + Create a file `build_status` and insert the script below and run + `chmod +x build_status` in your terminal to make the script executable. + + ```bash + #!/usr/bin/env bash + + # Push GitLab CI/CD build status to Bitbucket Cloud + + if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then + echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" + exit 1 + fi + if [ -z "$BITBUCKET_USERNAME" ]; then + echo "ERROR: BITBUCKET_USERNAME is not set" + exit 1 + fi + if [ -z "$BITBUCKET_NAMESPACE" ]; then + echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" + BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE + fi + if [ -z "$BITBUCKET_REPOSITORY" ]; then + echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" + BITBUCKET_REPOSITORY=$CI_PROJECT_NAME + fi + + BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" + BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" + BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" + + case "$BUILD_STATUS" in + running) + BITBUCKET_STATE="INPROGRESS" + BITBUCKET_DESCRIPTION="The build is running!" + ;; + passed) + BITBUCKET_STATE="SUCCESSFUL" + BITBUCKET_DESCRIPTION="The build passed!" + ;; + failed) + BITBUCKET_STATE="FAILED" + BITBUCKET_DESCRIPTION="The build failed." + ;; + esac + + echo "Pushing status to $BITBUCKET_STATUS_API..." + curl --request POST $BITBUCKET_STATUS_API \ + --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ + --header "Content-Type:application/json" \ + --silent \ + --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": + \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" + ``` + +1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push + pipeline success and failures to Bitbucket. + + ``` + stages: + - test + - ci_status + + unit-tests: + script: + - echo "Success. Add your tests!" + + success: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=passed BUILD_KEY=push ./build_status + when: on_success + + failure: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=failed BUILD_KEY=push ./build_status + when: on_failure + ``` + +GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines +configured in `.gitlab-ci.yml` and push the status to Bitbucket. + +[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md new file mode 100644 index 00000000000..0e2acf957e0 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -0,0 +1,111 @@ +# Using GitLab CI/CD with a GitHub repository **[PREMIUM]** + +GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by +creating a [CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) to connect your GitHub repository to +GitLab. + +NOTE: **Note:** +To use **GitHub Enterprise** with **GitLab.com** you should use the +manual method. + +## Connect with GitHub integration + +If the [GitHub integration](../../integration/github.md) has been enabled by your GitLab +administrator: + +NOTE: **Note:** +Due to a 10-token limitation on the [GitHub OAuth Implementation](https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps/#creating-multiple-tokens-for-oauth-apps), +if you import more than 10 times, your oldest imported project's token will be +revoked. See issue [#9147](https://gitlab.com/gitlab-org/gitlab-ee/issues/9147) +for more information. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Once authenticated, you will be redirected to a list of your repositories to + connect. Click **Connect** to select the repository. + +  + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable +[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook +on GitHub to notify GitLab of new commits. + +## Connect with Personal Access Token + +NOTE: **Note:** Personal access tokens can only be used to connect GitHub.com +repositories to GitLab. + +If you are not using the [GitHub integration](../../integration/github.md), you can +still perform a one-off authorization with GitHub to grant GitLab access your +repositories: + +1. Open https://github.com/settings/tokens/new to create a **Personal Access + Token**. This token with be used to access your repository and push commit + statuses to GitHub. + + The `repo` and `admin:repo_hook` should be enable to allow GitLab access to + your project, update commit statuses, and create a web hook to notify + GitLab of new commits. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Paste the token into the **Personal access token** field and click **List + Repositories**. Click **Connect** to select the repository. + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable +[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook +on GitHub to notify GitLab of new commits. + +## Connect manually + +If the [GitHub integration](../../integration/github.md) is not enabled, or is enabled +for a different GitHub instance, you GitLab CI/CD can be manually enabled for +your repository. + +1. In GitHub open https://github.com/settings/tokens/new create a **Personal + Access Token.** GitLab will use this token to access your repository and + push commit statuses. + + Enter a **Token description** and update the scope to allow: + + `repo` so that GitLab can access your project and update commit statuses + +1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS + URL for your GitHub repository. If your project is private, use the personal + access token you just created for authentication. + + GitLab will automatically configure polling-based pull mirroring. + +1. Still in GitLab, enable the [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html) + from **Settings > Integrations.** + + Check the **Active** checkbox to enable the integration, paste your + personal access token and HTTPS repository URL into the form, and **Save.** + +1. Still in GitLab create a **Personal Access Token** with `API` scope to + authenticate the GitHub web hook notifying GitLab of new commits. + +1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of + new commits. + + The web hook URL should be set to the GitLab API to + [trigger pull mirroring](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project-starter), + using the GitLab personal access token we just created. + + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + +  + +1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png Binary files differnew file mode 100644 index 00000000000..ac5be3c3058 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png Binary files differnew file mode 100644 index 00000000000..0a3476d9035 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png Binary files differnew file mode 100644 index 00000000000..f068688146b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png diff --git a/doc/ci/ci_cd_for_external_repos/img/external_repository.png b/doc/ci/ci_cd_for_external_repos/img/external_repository.png Binary files differnew file mode 100644 index 00000000000..b850d91f56b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/external_repository.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png Binary files differnew file mode 100644 index 00000000000..71a3a057a41 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png Binary files differnew file mode 100644 index 00000000000..3f2059504f5 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png Binary files differnew file mode 100644 index 00000000000..e8c17d664e1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png Binary files differnew file mode 100644 index 00000000000..73579dd3cf1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md new file mode 100644 index 00000000000..450ffe512dc --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -0,0 +1,31 @@ +# GitLab CI/CD for external repositories **[PREMIUM]** + +NOTE: **Note:** +This feature [is available for free](https://about.gitlab.com/2019/03/21/six-more-months-ci-cd-github/) to +GitLab.com users until September 22nd, 2019. + +>[Introduced][ee-4642] in [GitLab Premium][eep] 10.6. + +GitLab CI/CD can be used with GitHub or any other Git server. +Instead of moving your entire project to GitLab, you can connect your +external repository to get the benefits of GitLab CI/CD. + +- [GitHub](github_integration.md) +- [Bitbucket Cloud](bitbucket_integration.md) + +Connecting an external repository will set up [repository mirroring][mirroring] +and create a lightweight project where issues, merge requests, wiki, and +snippets disabled. These features +[can be re-enabled later][settings]. + +1. From your GitLab dashboard click **New project** +1. Switch to the **CI/CD for external repo** tab +1. Choose **GitHub** or **Repo by URL** +1. The next steps are similar to the [import flow](../../user/project/import/index.md) + + + +[ee-4642]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4642 +[eep]: https://about.gitlab.com/pricing/ +[mirroring]: ../../workflow/repository_mirroring.md +[settings]: ../../user/project/settings/index.md#sharing-and-permissions diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 6b4d4f1b9d4..d5e6fbe8113 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -669,7 +669,7 @@ fetch = +refs/environments/*:refs/remotes/origin/environments/* Some GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) features can behave differently for each environment. For example, you can -[create a secret variable to be injected only into a production environment](https://docs.gitlab.com/ee/ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium). +[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables-premium). In most cases, these features use the _environment specs_ mechanism, which offers an efficient way to implement scoping within each environment group. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 7f686781e3c..6b9f8181d1d 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -18,30 +18,30 @@ Examples are available in several forms. As a collection of: The following table lists examples for different use cases: -| Use case | Resource | -|:-----------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------| -| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | -| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | -| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | -| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). | -| Dependency scanning | [Dependency Scanning with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html). **[ULTIMATE]** | -| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | -| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** | -| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | -| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | -| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | -| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | -| JUnit | [JUnit test reports](../junit_test_reports.md). | -| License management | [Dependencies license management with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/license_management.html) **[ULTIMATE]** | -| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | -| PHP | [Testing PHP projects](php.md). | -| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | -| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | -| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | -| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | -| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/sast.html) **[ULTIMATE]** | -| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | +| Use case | Resource | +|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------| +| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | +| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | +| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | +| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). | +| Dependency scanning | [Dependency Scanning with GitLab CI/CD](dependency_scanning.md). **[ULTIMATE]** | +| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | +| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** | +| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | +| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | +| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | +| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | +| JUnit | [JUnit test reports](../junit_test_reports.md). | +| License management | [Dependencies license management with GitLab CI/CD](license_management.md) **[ULTIMATE]** | +| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | +| PHP | [Testing PHP projects](php.md). | +| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | +| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | +| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | +| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | +| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](sast.md) **[ULTIMATE]** | +| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | ### Contributing examples diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index b47038011de..442d0788d37 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -56,7 +56,7 @@ provide a list of URLs to test, please consult TIP: **Tip:** For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically extracted and shown right in the merge request widget. -[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html). +[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). ## Performance testing on Review Apps diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md index 36fdc29fe65..4d41e424f4a 100644 --- a/doc/ci/examples/container_scanning.md +++ b/doc/ci/examples/container_scanning.md @@ -1,119 +1,5 @@ -# Container Scanning with GitLab CI/CD +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +--- -CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. -It also requires the GitLab Runner 11.5 or later. -For earlier versions, use the [previous job definitions](#previous-job-definitions). - -You can check your Docker images (or more precisely the containers) for known -vulnerabilities by using [Clair](https://github.com/coreos/clair) and -[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools -for Vulnerability Static Analysis for containers. - -First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml` that -generates the expected report: - -```yaml -container_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:stable-dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - reports: - container_scanning: gl-container-scanning-report.json -``` - -The above example will create a `container_scanning` job in your CI/CD pipeline, pull -the image from the [Container Registry](../../user/project/container_registry.md) -(whose name is defined from the two `CI_APPLICATION_` variables) and scan it -for possible vulnerabilities. The report will be saved as a -[Container Scanning report artifact](../yaml/README.md#artifactsreportscontainer_scanning-ultimate) -that you can later download and analyze. -Due to implementation limitations we always take the latest Container Scanning artifact available. - -If you want to whitelist some specific vulnerabilities, you can do so by defining -them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file), -in our case its named `clair-whitelist.yml`. - -TIP: **Tip:** -For [GitLab Ultimate][ee] users, this information will -be automatically extracted and shown right in the merge request widget. -[Learn more on Container Scanning in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). - -CAUTION: **Caution:** -Starting with GitLab 11.5, Container Scanning feature is licensed under the name `container_scanning`. -While the old name `sast_container` is still maintained, it has been deprecated with GitLab 11.5 and -may be removed in next major release, GitLab 12.0. You are advised to update your current `.gitlab-ci.yml` -configuration to reflect that change if you are using the `$GITLAB_FEATURES` environment variable. - -## Previous job definitions - -CAUTION: **Caution:** -Before GitLab 11.5, Container Scanning job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained they have been deprecated -and may be removed in next major release, GitLab 12.0. -You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -container_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:stable-dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - paths: [gl-container-scanning-report.json] -``` - -Alternatively the job name could be `sast:container` -and the artifact name could be `gl-sast-container-report.json`. -These names have been deprecated with GitLab 11.0 -and may be removed in next major release, GitLab 12.0. - -[ee]: https://about.gitlab.com/pricing/ +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html). diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index ab0ca13d2cf..b676c661267 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -1,102 +1,5 @@ -# Dynamic Application Security Testing with GitLab CI/CD +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dast/index.html' +--- -CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. -It also requires the GitLab Runner 11.5 or later. -For earlier versions, use the [previous job definitions](#previous-job-definitions). - -[Dynamic Application Security Testing (DAST)](https://en.wikipedia.org/wiki/Dynamic_program_analysis) -is using the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy) -to perform an analysis on your running web application. -Since it is based on [ZAP Baseline](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan) -DAST will perform passive scanning only; -it will not actively attack your application. - -It can be very useful combined with [Review Apps](../review_apps/index.md). - -## Example - -First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml` that -generates the expected report: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` - -The above example will create a `dast` job in your CI/CD pipeline which will run -the tests on the URL defined in the `website` variable (change it to use your -own) and scan it for possible vulnerabilities. The report will be saved as a -[DAST report artifact](../yaml/README.md#artifactsreportsdast-ultimate) -that you can later download and analyze. -Due to implementation limitations we always take the latest DAST artifact available. - -It's also possible to authenticate the user before performing DAST checks: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - login_url: "https://example.com/sign-in" - username: "john.doe@example.com" - password: "john-doe-password" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website - --auth-url $login_url - --auth-username $username - --auth-password $password || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` -See [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) -to learn more about authentication settings. - -TIP: **Tip:** -For [GitLab Ultimate][ee] users, this information will -be automatically extracted and shown right in the merge request widget. -[Learn more on DAST in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html). - -## Previous job definitions - -CAUTION: **Caution:** -Before GitLab 11.5, DAST job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained they have been deprecated -and may be removed in next major release, GitLab 12.0. -You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - paths: [gl-dast-report.json] -``` - -[ee]: https://about.gitlab.com/pricing/ +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dast/index.html). diff --git a/doc/ci/examples/dependency_scanning.md b/doc/ci/examples/dependency_scanning.md new file mode 100644 index 00000000000..3a8b53b425c --- /dev/null +++ b/doc/ci/examples/dependency_scanning.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). diff --git a/doc/ci/examples/license_management.md b/doc/ci/examples/license_management.md new file mode 100644 index 00000000000..08704425a75 --- /dev/null +++ b/doc/ci/examples/license_management.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/license_management/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/license_management/index.html). diff --git a/doc/ci/examples/sast.md b/doc/ci/examples/sast.md new file mode 100644 index 00000000000..688cc79d0f6 --- /dev/null +++ b/doc/ci/examples/sast.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/sast/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/sast/index.html). diff --git a/doc/ci/examples/sast_docker.md b/doc/ci/examples/sast_docker.md index 70b269046e5..570898b2d23 100644 --- a/doc/ci/examples/sast_docker.md +++ b/doc/ci/examples/sast_docker.md @@ -1,5 +1,5 @@ --- -redirect_to: 'container_scanning.md' +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' --- -This document was moved to [another location](container_scanning.md). +This document was moved to [another location](../../user/application_security/container_scanning/index.html). diff --git a/doc/ci/img/metrics_reports.png b/doc/ci/img/metrics_reports.png Binary files differnew file mode 100644 index 00000000000..ffd9f6830a2 --- /dev/null +++ b/doc/ci/img/metrics_reports.png diff --git a/doc/ci/img/multi_pipeline_mini_graph.gif b/doc/ci/img/multi_pipeline_mini_graph.gif Binary files differnew file mode 100644 index 00000000000..de49ba5aa12 --- /dev/null +++ b/doc/ci/img/multi_pipeline_mini_graph.gif diff --git a/doc/ci/img/multi_project_pipeline_graph.png b/doc/ci/img/multi_project_pipeline_graph.png Binary files differnew file mode 100644 index 00000000000..723a455cb4a --- /dev/null +++ b/doc/ci/img/multi_project_pipeline_graph.png diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 2a4160f62b0..7109b2ec583 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -54,3 +54,8 @@ terminal will block the job from finishing for the duration configured in close the terminal window.  + +## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** + +Read the Web IDE docs to learn how to run [Interactive Terminals through the Web IDE](../../user/project/web_ide/index.md). + diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md new file mode 100644 index 00000000000..83a7094faaa --- /dev/null +++ b/doc/ci/metrics_reports.md @@ -0,0 +1,40 @@ +# Metrics Reports **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing) 11.10. +Requires GitLab Runner 11.10 and above. + +## Overview + +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](./junit_test_reports.md), [codequality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. + +You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. + + + +## Use cases + +Consider the following examples of data that can utilize Metrics Reports: + +1. Memory usage +1. Load testing results +1. Code complexity +1. Code coverage stats + +## How it works + +Metrics are read from the metrics report (default: `metrics.txt`). They are parsed and displayed in the MR widget. + +## How to set it up + +Add a job that creates a [metrics report](yaml/README.md#artifactsreportsmetrics-premium) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. + +For example: + +```yaml +metrics: + script: + - echo 'metric_name metric_value' > metrics.txt + artifacts: + reports: + metrics: metrics.txt +``` diff --git a/doc/ci/multi_project_pipeline_graphs.md b/doc/ci/multi_project_pipeline_graphs.md new file mode 100644 index 00000000000..af10e5b6126 --- /dev/null +++ b/doc/ci/multi_project_pipeline_graphs.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'multi_project_pipelines.md' +--- + +This document was moved to [another location](multi_project_pipelines.md). diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md new file mode 100644 index 00000000000..556059c01b6 --- /dev/null +++ b/doc/ci/multi_project_pipelines.md @@ -0,0 +1,152 @@ +# Multi-project pipelines **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/2121) in +[GitLab Premium 9.3](https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). + +When you set up [GitLab CI/CD](README.md) across multiple projects, you can visualize +the entire pipeline, including all cross-project inter-dependencies. + +## Overview + +GitLab CI/CD is a powerful continuous integration tool that works not only per project, but also across projects. When you +configure GitLab CI for your project, you can visualize the stages +of your [jobs](pipelines.md#configuring-pipelines) on a [pipeline graph](pipelines.md#visualizing-pipelines). + + + +In the Merge Request Widget, multi-project pipeline mini-graphs are displayed, +and when hovering or tapping (on touchscreen devices) they will expand and be shown adjacent to each other. + + + +Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those +adopting a [microservices architecture](https://about.gitlab.com/2016/08/16/trends-in-version-control-land-microservices/). + +## Use cases + +Let's assume you deploy your web app from different projects in GitLab: + +- One for the free version, which has its own pipeline that builds and tests your app +- One for the paid version add-ons, which also pass through builds and tests +- One for the documentation, which also builds, tests, and deploys with an SSG + +With Multi-Project Pipelines, you can visualize the entire pipeline, including all stages of builds and tests for the three projects. + +## Triggering multi-project pipelines through API + +When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab +recognizes the source of the job token, and thus internally ties these pipelines +together, allowing you to visualize their relationships on pipeline graphs. + +These relationships are displayed in the pipeline graph by showing inbound and +outbound connections for upstream and downstream pipeline dependencies. + +## Creating multi-project pipelines from `.gitlab-ci.yml` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +### Triggering a downstream pipeline using a bridge job + +Before GitLab 11.8, it was necessary to implement a pipeline job that was +responsible for making the API request [to trigger a pipeline](#triggering-multi-project-pipelines-through-api) +in a different project. + +In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this +task easier, and avoid needing GitLab Runner for triggering cross-project +pipelines. The following illustrates configuring a bridge job: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +In the example above, as soon as `rspec` job succeeds in the `test` stage, +the `staging` bridge job is going to be started. The initial status of this +job will be `pending`. GitLab will create a downstream pipeline in the +`my/deployment` project and, as soon as the pipeline gets created, the +`staging` job will succeed. `my/deployment` is a full path to that project. + +The user that created the upstream pipeline needs to have access rights to the +downstream project (`my/deployment` in this case). If a downstream project can +not be found, or a user does not have access rights to create pipeline there, +the `staging` job is going to be marked as _failed_. + +CAUTION: **Caution:** +`staging` will succeed as soon as a downstream pipeline gets created. +GitLab does not support status attribution yet, however adding first-class +`trigger` configuration syntax is ground work for implementing +[status attribution](https://gitlab.com/gitlab-org/gitlab-ce/issues/39640). + +NOTE: **Note:** +Bridge jobs do not support every configuration entry that a user can use +in the case of regular jobs. Bridge jobs will not to be picked by a Runner, +thus there is no point in adding support for `script`, for example. If a user +tries to use unsupported configuration syntax, YAML validation will fail upon +pipeline creation. + +### Specifying a downstream pipeline branch + +It is possible to specify a branch name that a downstream pipeline will use: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable-11-2 +``` + +Use a `project` keyword to specify full path to a downstream project. Use +a `branch` keyword to specify a branch name. + +GitLab will use a commit that is currently on the HEAD of the branch when +creating a downstream pipeline. + +### Passing variables to a downstream pipeline + +Sometimes you might want to pass variables to a downstream pipeline. +You can do that using the `variables` keyword, just like you would when +defining a regular job. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +The `ENVIRONMENT` variable will be passed to every job defined in a downstream +pipeline. It will be available as an environment variable when GitLab Runner picks a job. + +### Limitations + +Because bridge jobs are a little different to regular jobs, it is not +possible to use exactly the same configuration syntax here, as one would +normally do when defining a regular job that will be picked by a runner. + +Some features are not implemented yet. For example, support for environments. + +[Configuration keywords](yaml/README.md) available for bridge jobs are: + +- `trigger` (to define a downstream pipeline trigger) +- `stage` +- `allow_failure` +- `only` and `except` +- `when` +- `extends` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 07129eb4186..342c2ab972a 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -19,7 +19,7 @@ If all the jobs in a stage: - Fail, the next stage is not (usually) executed and the pipeline ends early. NOTE: **Note:** -If you have a [mirrored repository that GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository that GitLab pulls from](../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -220,7 +220,7 @@ For information on adding pipeline badges to projects, see [Pipeline badges](../ Pipelines for different projects can be combined and visualized together. -For more information, see [Multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html). +For more information, see [Multi-project pipelines](multi_project_pipelines.md). ## Working with pipelines diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 65886400c64..015f1c0dc0f 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -127,7 +127,7 @@ Now if you go to the **Pipelines** page you will see that the pipeline is pending. NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 0e72bdddee7..ad80b5d8818 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -23,6 +23,63 @@ attackers can impersonate the user that exposed their trigger token publicly in their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#gitlab-cicd-environment-variables) to protect trigger tokens. +### CI job token + +You can use the `CI_JOB_TOKEN` [variable][predef] (used to authenticate +with the [GitLab Container Registry][registry]) in the following cases. + +#### When used with multi-project pipelines **[PREMIUM]** + +> **Note**: +The use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced][ee-2017] +in [GitLab Premium][ee] 9.3. + +This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, +and it creates a dependent pipeline relation visible on the +[pipeline graph](../multi_project_pipelines.md#overview). For example: + +```yaml +build_docs: + stage: deploy + script: + - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline + only: + - tags +``` + +Pipelines triggered that way also expose a special variable: +`CI_PIPELINE_SOURCE=pipeline`. + +Read more about the [pipelines trigger API][trigapi]. + +#### When a pipeline depends on the artifacts of another pipeline **[PREMIUM]** + +> The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] + in [GitLab Premium][ee] 9.5. + +With the introduction of dependencies between different projects, one of +them may need to access artifacts created by a previous one. This process +must be granted for authorized accesses, and it can be done using the +`CI_JOB_TOKEN` variable that identifies a specific job. For example: + +```yaml +build_submodule: + image: debian + stage: test + script: + - apt update && apt install -y unzip + - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN" + - unzip artifacts.zip + only: + - tags +``` + +This allows you to use that for multi-project pipelines and download artifacts +from any project to which you have access as this follows the same principles +with the [permission model][permissions]. + +Read more about the [jobs API](../../api/jobs.md#download-the-artifacts-archive). + ## Adding a new trigger You can add a new trigger by going to your project's @@ -225,7 +282,10 @@ removed with one of the future versions of GitLab. You are advised to [take ownership](#taking-ownership-of-a-trigger) of any legacy triggers. [ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 +[ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 [ee]: https://about.gitlab.com/pricing/ [variables]: ../variables/README.md [predef]: ../variables/README.md#predefined-environment-variables [registry]: ../../user/project/container_registry.md +[permissions]: ../../user/permissions.md#job-permissions +[trigapi]: ../../api/pipeline_triggers.md diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 6313ffc584d..1ba22070abe 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -52,6 +52,33 @@ or directly in the `.gitlab-ci.yml` file and reuse them as you wish. That can be very powerful as it can be used for scripting without the need to specify the value itself. +#### Variable types + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46806) in GitLab 11.11. + +There are two types of variables supported by GitLab: + +- `env_var`: the runner will create environment variable named same as the variable key and set its value to the variable value. +- `file`: the runner will write the variable value to a temporary file and set the path to this file as the value of an environment variable named same as the variable key. + +#### Masked variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/13784) in GitLab 11.10 + +By default, variables will be created as masked variables. +This means that the value of the variable will be hidden in job logs, +though it must match certain requirements to do so: + +- The value must be in a single line. +- The value must not have escape characters. +- The value must not use variables. +- The value must not have any whitespace. +- The value must be at least 8 characters long. + +If the value does not meet the requirements above, then the CI variable will fail to save. +In order to save, either alter the value to meet the masking requirements +or disable **Masked** for the variable. + ## Getting started To get started with environment variables in the scope of GitLab @@ -104,7 +131,10 @@ let's say you want to output `HELLO WORLD` for a `TEST` variable. You can either set the variable directly in the `.gitlab-ci.yml` file or through the UI. -#### Via [`.gitlab-ci.yml`](../yaml/README.md#variables) +#### Via `.gitlab-ci.yml` + +To create a new custom `env_var` variable via [`.gitlab-ci.yml`](../yaml/README.md#variables), define their variable/value pair under +`variables`: ```yaml variables: @@ -116,11 +146,13 @@ For a deeper look into them, see [`.gitlab-ci.yml` defined variables](#gitlab-ci #### Via the UI From the UI, navigate to your project's **Settings > CI/CD** and -expand **Environment variables**. Create a new variable by naming -it in the field **Input variable key**, and define its value in the +expand **Variables**. Create a new variable by choosing its **type**, naming +it in the field **Input variable key**, and defining its value in the **Input variable value** field: - + + +You'll also see the option to mask and/or protect your variables. Once you've set the variables, call them from the `.gitlab-ci.yml` file: @@ -129,30 +161,14 @@ test_variable: stage: test script: - echo $CI_JOB_STAGE # calls a predefined variable - - echo $TEST # calls a custom variable + - echo $TEST # calls a custom variable of type `env_var` + - echo $GREETING # calls a custom variable of type `file` that contains the path to the temp file + - cat $GREETING # the temp file itself contains the variable value ``` The output will be: - - -### Masked Variables - -By default, variables will be created as masked variables. -This means that the value of the variable will be hidden in job logs, -though it must match certain requirements to do so: - -- The value must be in a single line. -- The value must contain only letters, numbers, or underscores. -- The value must not have escape characters, such as `\"` -- The value must not use variables. -- The value must not have any whitespace. -- The value must be at least 8 characters long. - -The above rules are validated using the regex `/\A\w{8,}\z/`. If the value -does not meet the requirements above, then the CI variable will fail to save. -In order to save, either alter the value to meet the masking requirements or -disable `Masked` for the variable. + ### Syntax of environment variables in job scripts @@ -299,7 +315,7 @@ use for storing things like passwords, SSH keys, and credentials. Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. -1. Inputing variable keys and values in the **Environment variables** section. +1. Inputing variable types, keys, and values in the **Variables** section. Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. Once you set them, they will be available for all subsequent pipelines. @@ -350,6 +366,25 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. +### Limiting environment scopes of environment variables **[PREMIUM]** + +> [Introduced][ee-2112] in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. + +You can limit the environment scope of a variable by +[defining which environments][envs] it can be available for. + +Wildcards can be used, and the default environment scope is `*` which means +any jobs will have this variable, not matter if an environment is defined or +not. + +For example, if the environment scope is `production`, then only the jobs +having the environment `production` defined would have this specific variable. +Wildcards (`*`) can be used along with the environment name, therefore if the +environment scope is `review/*` then any jobs with environment names starting +with `review/` would have that particular variable. + +To learn more about about scoping environments, see [Scoping environments with specs](../environments.md#scoping-environments-with-specs-premium). + ### Deployment environment variables > Introduced in GitLab 8.15. @@ -390,8 +425,7 @@ For instance, suppose you added a [custom variable `$TEST`](#creating-a-custom-environment-variable) as exemplified above and you want to override it in a manual pipeline. Navigate to your project's **CI/CD > Pipelines** and click **Run pipeline**. -Choose the branch you want to run the pipeline for, then add a new variable -pair through the UI: +Choose the branch you want to run the pipeline for, then add a new variable through the UI:  @@ -662,13 +696,15 @@ MIIFQzCCBCugAwIBAgIRAL/ElDjuf15xwja1ZnCocWAwDQYJKoZIhvcNAQELBQAw' ... ``` +[ee-2112]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2112 [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI variables" -[eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium" [envs]: ../environments.md [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md [shellexecutors]: https://docs.gitlab.com/runner/executors/ [triggered]: ../triggers/README.md +[trigger-job-token]: ../triggers/README.md#ci-job-token [gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token [registry]: ../../user/project/container_registry.md [dependent-repositories]: ../../user/project/new_ci_build_permissions_model.md#dependent-repositories +[get-job-artifacts]: ../../api/jobs.html#get-job-artifacts diff --git a/doc/ci/variables/img/custom_variable_output.png b/doc/ci/variables/img/custom_variable_output.png Binary files differdeleted file mode 100644 index 50f3bceff9a..00000000000 --- a/doc/ci/variables/img/custom_variable_output.png +++ /dev/null diff --git a/doc/ci/variables/img/custom_variables_output.png b/doc/ci/variables/img/custom_variables_output.png Binary files differnew file mode 100644 index 00000000000..29f5c63b3d9 --- /dev/null +++ b/doc/ci/variables/img/custom_variables_output.png diff --git a/doc/ci/variables/img/new_custom_variable_example.png b/doc/ci/variables/img/new_custom_variable_example.png Binary files differdeleted file mode 100644 index d169c5f1806..00000000000 --- a/doc/ci/variables/img/new_custom_variable_example.png +++ /dev/null diff --git a/doc/ci/variables/img/new_custom_variables_example.png b/doc/ci/variables/img/new_custom_variables_example.png Binary files differnew file mode 100644 index 00000000000..4b78e0ff587 --- /dev/null +++ b/doc/ci/variables/img/new_custom_variables_example.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differindex 3bcd354e096..3c8c59720cf 100644 --- a/doc/ci/variables/img/override_variable_manual_pipeline.png +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 091ddcb0bae..0470cf52654 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -112,3 +112,22 @@ They are: - Not supported: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - In the `only` and `except` [variables expressions](README.md#environment-variables-expressions). + +## Variables with an environment scope + +Variables defined with an environment scope are supported. Given that +there is a variable `$STAGING_SECRET` defined in a scope of +`review/staging/*`, the following job that is using dynamic environments +is going to be created, based on the matching variable expression: + +```yaml +my-job: + stage: staging + environment: + name: review/$CI_JOB_STAGE/deploy + script: + - 'deploy staging' + only: + variables: + - $STAGING_SECRET == 'something' +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 03383d11c14..99e4c64ff86 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -19,7 +19,7 @@ We have complete examples of configuring pipelines: - To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab-ce`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml). NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -56,7 +56,7 @@ independently from each other. Each instance of GitLab CI has an embedded debug tool called Lint, which validates the content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your -project namespace. For example, `http://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. +project namespace. For example, `https://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. ### Unavailable names for jobs @@ -101,7 +101,7 @@ The following table lists available parameters for jobs: | [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | | [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, and `environment:action`. | | [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. | -| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, and `artifacts:reports:junit`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, and `artifacts:reports:performance`. | +| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, and `artifacts:reports:junit`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, `artifacts:reports:performance` and `artifacts:reports:metrics`. | | [`dependencies`](#dependencies) | Other jobs that a job depends on so that you can pass artifacts between them. | | [`coverage`](#coverage) | Code coverage settings for a given job. | | [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | @@ -1457,7 +1457,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html) as artifacts. The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will @@ -1468,7 +1468,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html) +The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html) as artifacts. The collected Container Scanning report will be uploaded to GitLab as an artifact and will @@ -1479,7 +1479,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html) +The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dast/index.html) as artifacts. The collected DAST report will be uploaded to GitLab as an artifact and will @@ -1507,6 +1507,14 @@ as artifacts. The collected Performance report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests. +##### `artifacts:reports:metrics` **[PREMIUM]** + +The `metrics` report collects [Metrics](../../ci/metrics_reports.md) +as artifacts. + +The collected Metrics report will be uploaded to GitLab as an artifact and will +be automatically shown in merge requests. + ### `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1702,7 +1710,7 @@ test: from `trigger` definition is started by GitLab, a downstream pipeline gets created. -Learn more about [multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html#creating-cross-project-pipelines-from-gitlab-ci-yml). +Learn more about [multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml). #### Simple `trigger` syntax diff --git a/doc/customization/issue_and_merge_request_template.md b/doc/customization/issue_and_merge_request_template.md new file mode 100644 index 00000000000..01c31728c21 --- /dev/null +++ b/doc/customization/issue_and_merge_request_template.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/project/description_templates.html#setting-a-default-template-for-issues-and-merge-requests--starter' +--- + +This document was moved to [description_templates](https://docs.gitlab.com/ee/user/project/description_templates.html#setting-a-default-template-for-issues-and-merge-requests--starter). diff --git a/doc/development/README.md b/doc/development/README.md index 5a33c46c620..83a1145c020 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -38,6 +38,7 @@ description: 'Learn how to contribute to GitLab.' - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Manage feature flags](feature_flags.md) +- [Licensed feature availability](licensed_feature_availability.md) - [View sent emails or preview mailers](emails.md) - [Shell commands](shell_commands.md) in the GitLab codebase - [`Gemfile` guidelines](gemfile.md) @@ -48,6 +49,7 @@ description: 'Learn how to contribute to GitLab.' - [How to dump production data to staging](db_dump.md) - [Working with the GitHub importer](github_importer.md) - [Import/Export development documentation](import_export.md) +- [Elasticsearch integration docs](elasticsearch.md) - [Working with Merge Request diffs](diffs.md) - [Kubernetes integration guidelines](kubernetes.md) - [Permissions](permissions.md) @@ -55,6 +57,7 @@ description: 'Learn how to contribute to GitLab.' - [Guidelines for reusing abstractions](reusing_abstractions.md) - [DeclarativePolicy framework](policies.md) - [How Git object deduplication works in GitLab](git_object_deduplication.md) +- [Geo development](geo.md) ## Performance guides @@ -101,6 +104,10 @@ description: 'Learn how to contribute to GitLab.' - [Query Count Limits](query_count_limits.md) - [Database helper modules](database_helpers.md) +## Integration guides + +- [Jira Connect app](integrations/jira_connect.md) + ## Testing guides - [Testing standards and style guidelines](testing_guide/index.md) diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 5e310092a6e..8a4aa5dfa7f 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -155,7 +155,7 @@ the contribution acceptance criteria below: restarting the failing CI job, rebasing from master to bring in updates that may resolve the failure, or if it has not been fixed yet, ask a developer to help you fix the test. -1. The MR initially contains a a few logically organized commits. +1. The MR initially contains a few logically organized commits. 1. The changes can merge without problems. If not, you should rebase if you're the only one working on your feature branch, otherwise merge `master`. 1. Only one specific issue is fixed or one specific feature is implemented. Do not diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md new file mode 100644 index 00000000000..0c9e7908713 --- /dev/null +++ b/doc/development/elasticsearch.md @@ -0,0 +1,166 @@ +# Elasticsearch knowledge **[STARTER ONLY]** + +This area is to maintain a compendium of useful information when working with elasticsearch. + +Information on how to enable ElasticSearch and perform the initial indexing is kept in https://docs.gitlab.com/ee/integration/elasticsearch.html#enabling-elasticsearch + +## Initial installation on OS X + +It is recommended to use the Docker image. After installing docker you can immediately spin up an instance with + +``` +docker run --name elastic56 -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:5.6.12 +``` + +and use `docker stop elastic56` and `docker start elastic56` to stop/start it. + +### Installing on the host + +We currently only support Elasticsearch [5.6 to 6.x](https://docs.gitlab.com/ee/integration/elasticsearch.html#requirements) + +Version 5.6 is available on homebrew and is the recommended version to use in order to test compatibility. + +``` +brew install elasticsearch@5.6 +``` + +There is no need to install any plugins + +## New repo indexer (beta) + +If you're interested on working with the new beta repo indexer, all you need to do is: + +- git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git +- make +- make install + +this adds `gitlab-elasticsearch-indexer` to `$GOPATH/bin`, please make sure that is in your `$PATH`. After that GitLab will find it and you'll be able to enable it in the admin settings area. + +**note:** `make` will not recompile the executable unless you do `make clean` beforehand + +## Helpful rake tasks + +- `gitlab:elastic:test:index_size`: Tells you how much space the current index is using, as well as how many documents are in the index. +- `gitlab:elastic:test:index_size_change`: Outputs index size, reindexes, and outputs index size again. Useful when testing improvements to indexing size. + +Additionally, if you need large repos or multiple forks for testing, please consider [following these instructions](https://docs.gitlab.com/ee/development/rake_tasks.html#extra-project-seed-options) + +## How does it work? + +The ElasticSearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb). + +All indexing after the initial one is done via `ElasticIndexerWorker` (sidekiq jobs). + +Search queries are generated by the concerns found in [ee/app/models/concerns/elastic](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! + +## Existing Analyzers/Tokenizers/Filters +These are all defined in https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/model.rb + +### Analyzers +#### `path_analyzer` +Used when indexing blobs' paths. Uses the `path_tokenizer` and the `lowercase` and `asciifolding` filters. + +Please see the `path_tokenizer` explanation below for an example. + +#### `sha_analyzer` +Used in blobs and commits. Uses the `sha_tokenizer` and the `lowercase` and `asciifolding` filters. + +Please see the `sha_tokenizer` explanation later below for an example. + +#### `code_analyzer` +Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the filters: `code`, `edgeNGram_filter`, `lowercase`, and `asciifolding` + +The `whitespace` tokenizer was selected in order to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` in order to be properly searched. + +Please see the `code` filter for an explanation on how tokens are split. + +#### `code_search_analyzer` +Not directly used for indexing, but rather used to transform a search input. Uses the `whitespace` tokenizer and the `lowercase` and `asciifolding` filters. + +### Tokenizers +#### `sha_tokenizer` +This is a custom tokenizer that uses the [`edgeNGram` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenizer.html) to allow SHAs to be searcheable by any sub-set of it (minimum of 5 chars). + +example: + +`240c29dc7e` becomes: +- `240c2` +- `240c29` +- `240c29d` +- `240c29dc` +- `240c29dc7` +- `240c29dc7e` + +#### `path_tokenizer` +This is a custom tokenizer that uses the [`path_hierarchy` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pathhierarchy-tokenizer.html) with `reverse: true` in order to allow searches to find paths no matter how much or how little of the path is given as input. + +example: + +`'/some/path/application.js'` becomes: +- `'/some/path/application.js'` +- `'some/path/application.js'` +- `'path/application.js'` +- `'application.js'` + +### Filters +#### `code` +Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. + +Patterns: +- `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCased and lowedCameCased strings as separate tokens +- `"(\\d+)"`: extracts digits +- `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCased strings recursively. Ex: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` +- `'"((?:\\"|[^"]|\\")*)"'`: captures terms inside quotes, removing the quotes +- `"'((?:\\'|[^']|\\')*)'"`: same as above, for single-quotes +- `'\.([^.]+)(?=\.|\s|\Z)'`: separate terms with periods in-between +- `'\/?([^\/]+)(?=\/|\b)'`: separate path terms `like/this/one` + +#### `edgeNGram_filter` +Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenfilter.html) to allow inputs with only parts of a token to find the token. For example it would turn `glasses` into permutations starting with `gl` and ending with `glasses`, which would allow a search for "`glass`" to find the original token `glasses` + +## Gotchas + +- Searches can have their own analyzers. Remember to check when editing analyzers +- `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches + +## Troubleshooting + +### Getting "flood stage disk watermark [95%] exceeded" + +You might get an error such as + +``` +[2018-10-31T15:54:19,762][WARN ][o.e.c.r.a.DiskThresholdMonitor] [pval5Ct] + flood stage disk watermark [95%] exceeded on + [pval5Ct7SieH90t5MykM5w][pval5Ct][/usr/local/var/lib/elasticsearch/nodes/0] free: 56.2gb[3%], + all indices on this node will be marked read-only +``` + +This is because you've exceeded the disk space threshold - it thinks you don't have enough disk space left, based on the default 95% threshold. + +In addition, the `read_only_allow_delete` setting will be set to `true`. It will block indexing, `forcemerge`, etc + +``` +curl "http://localhost:9200/gitlab-development/_settings?pretty" +``` + +Add this to your `elasticsearch.yml` file: + +``` +# turn off the disk allocator +cluster.routing.allocation.disk.threshold_enabled: false +``` + +_or_ + +``` +# set your own limits +cluster.routing.allocation.disk.threshold_enabled: true +cluster.routing.allocation.disk.watermark.flood_stage: 5gb # ES 6.x only +cluster.routing.allocation.disk.watermark.low: 15gb +cluster.routing.allocation.disk.watermark.high: 10gb +``` + +Restart ElasticSearch, and the `read_only_allow_delete` will clear on it's own. + +_from "Disk-based Shard Allocation | Elasticsearch Reference" [5.6](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/disk-allocator.html#disk-allocator) and [6.x](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/disk-allocator.html)_ diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index 548d72bea93..36880dd746d 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -16,10 +16,12 @@ New utility classes should be added to [`utilities.scss`](https://gitlab.com/git **Background color**: `.bg-variant-shade` e.g. `.bg-warning-400` **Text color**: `.text-variant-shade` e.g. `.text-success-500` + - variant is one of 'primary', 'secondary', 'success', 'warning', 'error' - shade is on of the shades listed on [colors](https://design.gitlab.com/foundations/colors/) **Font size**: `.text-size` e.g. `.text-2` + - **size** is number from 1-6 from our [Type scale](https://design.gitlab.com/foundations/typography) ### Naming diff --git a/doc/development/geo.md b/doc/development/geo.md new file mode 100644 index 00000000000..c8e6a86eb52 --- /dev/null +++ b/doc/development/geo.md @@ -0,0 +1,463 @@ +# Geo (development) **[PREMIUM ONLY]** + +Geo connects GitLab instances together. One GitLab instance is +designated as a **primary** node and can be run with multiple +**secondary** nodes. Geo orchestrates quite a few components that can be seen on +the diagram below and are described in more detail within this document. + + + +## Replication layer + +Geo handles replication for different components: +- [Database](#database-replication): includes the entire application, except cache and jobs. +- [Git repositories](#repository-replication): includes both projects and wikis. +- [Uploaded blobs](#uploads-replication): includes anything from images attached on issues +to raw logs and assets from CI. + +With the exception of the Database replication, on a *secondary* node, everything is coordinated +by the [Geo Log Cursor](#geo-log-cursor). + +### Geo Log Cursor daemon + +The [Geo Log Cursor daemon](#geo-log-cursor-daemon) is a separate process running on +each **secondary** node. It monitors the [Geo Event Log](#geo-event-log) +for new events and creates background jobs for each specific event type. + +For example when a repository is updated, the Geo **primary** node creates +a Geo event with an associated repository updated event. The Geo Log Cursor daemon +picks the event up and schedules a `Geo::ProjectSyncWorker` job which will +use the `Geo::RepositorySyncService` and `Geo::WikiSyncService` classes +to update the repository and the wiki respectively. + +The Geo Log Cursor daemon can operate in High Availability mode automatically. +The daemon will try to acquire a lock from time to time and once acquired, it +will behave as the *active* daemon. + +Any additional running daemons on the same node, will be in standby +mode, ready to resume work if the *active* daemon releases its lock. + +We use the [`ExclusiveLease`](https://www.rubydoc.info/github/gitlabhq/gitlabhq/Gitlab/ExclusiveLease) lock type with a small TTL, that is renewed at every +pooling cycle. That allows us to implement this global lock with a timeout. + +At the end of the pooling cycle, if the daemon can't renew and/or reacquire +the lock, it switches to standby mode. + +### Database replication + +Geo uses [streaming replication](#streaming-replication) to replicate +the database from the **primary** to the **secondary** nodes. This +replication gives the **secondary** nodes access to all the data saved +in the database. So users can log in on the **secondary** and read all +the issues, merge requests, etc. on the **secondary** node. + +### Repository replication + +Geo also replicates repositories. Each **secondary** node keeps track of +the state of every repository in the [tracking database](#tracking-database). + +There are a few ways a repository gets replicated by the: + +- [Repository Sync worker](#repository-sync-worker). +- [Geo Log Cursor](#geo-log-cursor). + +#### Project Registry + +The `Geo::ProjectRegistry` class defines the model used to track the +state of repository replication. For each project in the main +database, one record in the tracking database is kept. + +It records the following about repositories: + +- The last time they were synced. +- The last time they were successfully synced. +- If they need to be resynced. +- When a retry should be attempted. +- The number of retries. +- If and when they were verified. + +It also stores these attributes for project wikis in dedicated columns. + +#### Repository Sync worker + +The `Geo::RepositorySyncWorker` class runs periodically in the +background and it searches the `Geo::ProjectRegistry` model for +projects that need updating. Those projects can be: + +- Unsynced: Projects that have never been synced on the **secondary** + node and so do not exist yet. +- Updated recently: Projects that have a `last_repository_updated_at` + timestamp that is more recent than the `last_repository_successful_sync_at` + timestamp in the `Geo::ProjectRegistry` model. +- Manual: The admin can manually flag a repository to resync in the + [Geo admin panel](https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html). + +When we fail to fetch a repository on the secondary `RETRIES_BEFORE_REDOWNLOAD` +times, Geo does a so-called _redownload_. It will do a clean clone +into the `@geo-temporary` directory in the root of the storage. When +it's successful, we replace the main repo with the newly cloned one. + +### Uploads replication + +File uploads are also being replicated to the **secondary** node. To +track the state of syncing, the `Geo::FileRegistry` model is used. + +#### File Registry + +Similar to the [Project Registry](#project-registry), there is a +`Geo::FileRegistry` model that tracks the synced uploads. + +CI Job Artifacts are synced in a similar way as uploads or LFS +objects, but they are tracked by `Geo::JobArtifactRegistry` model. + +#### File Download Dispatch worker + +Also similar to the [Repository Sync worker](#repository-sync-worker), +there is a `Geo::FileDownloadDispatchWorker` class that is run +periodically to sync all uploads that aren't synced to the Geo +**secondary** node yet. + +Files are copied via HTTP(s) and initiated via the +`/api/v4/geo/transfers/:type/:id` endpoint, +e.g. `/api/v4/geo/transfers/lfs/123`. + +## Authentication + +To authenticate file transfers, each `GeoNode` record has two fields: + +- A public access key (`access_key` field). +- A secret access key (`secret_access_key` field). + +The **secondary** node authenticates itself via a [JWT request](https://jwt.io/). +When the **secondary** node wishes to download a file, it sends an +HTTP request with the `Authorization` header: + +``` +Authorization: GL-Geo <access_key>:<JWT payload> +``` + +The **primary** node uses the `access_key` field to look up the +corresponding **secondary** node and decrypts the JWT payload, +which contains additional information to identify the file +request. This ensures that the **secondary** node downloads the right +file for the right database ID. For example, for an LFS object, the +request must also include the SHA256 sum of the file. An example JWT +payload looks like: + +``` +{ "data": { sha256: "31806bb23580caab78040f8c45d329f5016b0115" }, iat: "1234567890" } +``` + +If the requested file matches the requested SHA256 sum, then the Geo +**primary** node sends data via the [X-Sendfile](https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/) +feature, which allows NGINX to handle the file transfer without tying +up Rails or Workhorse. + +NOTE: **Note:** +JWT requires synchronized clocks between the machines +involved, otherwise it may fail with an encryption error. + +## Git Push to Geo secondary + +The Git Push Proxy exists as a functionality built inside the `gitlab-shell` component. +It is active on a **secondary** node only. It allows the user that has cloned a repository +from the secondary node to push to the same URL. + +Git `push` requests directed to a **secondary** node will be sent over to the **primary** node, +while `pull` requests will continue to be served by the **secondary** node for maximum efficiency. + +HTTPS and SSH requests are handled differently: + +- With HTTPS, we will give the user a `HTTP 302 Redirect` pointing to the project on the **primary** node. +The git client is wise enough to understand that status code and process the redirection. +- With SSH, because there is no equivalent way to perform a redirect, we have to proxy the request. +This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request +to the HTTP protocol, and then proxying it to the **primary** node. + +The [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) daemon knows when to proxy based on the response +from `/api/v4/allowed`. A special `HTTP 300` status code is returned and we execute a "custom action", +specified in the response body. The response contains additional data that allows the proxied `push` operation +to happen on the **primary** node. + +## Using the Tracking Database + +Along with the main database that is replicated, a Geo **secondary** +node has its own separate [Tracking database](#tracking-database). + +The tracking database contains the state of the **secondary** node. + +Any database migration that needs to be run as part of an upgrade +needs to be applied to the tracking database on each **secondary** node. + +### Configuration + +The database configuration is set in [`config/database_geo.yml`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/config/database_geo.yml.postgresql). +The directory [`ee/db/geo`](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/db/geo) +contains the schema and migrations for this database. + +To write a migration for the database, use the `GeoMigrationGenerator`: + +``` +rails g geo_migration [args] [options] +``` + +To migrate the tracking database, run: + +``` +bundle exec rake geo:db:migrate +``` + +### Foreign Data Wrapper + +The use of [FDW](#fdw) was introduced in GitLab 10.1. + +This is useful for the [Geo Log Cursor](#geo-log-cursor) and improves +the performance of some synchronization operations. + +While FDW is available in older versions of PostgreSQL, we needed to +raise the minimum required version to 9.6 as this includes many +performance improvements to the FDW implementation. + +#### Refeshing the Foreign Tables + +Whenever the database schema changes on the **primary** node, the +**secondary** node will need to refresh its foreign tables by running +the following: + +```sh +bundle exec rake geo:db:refresh_foreign_tables +``` + +Failure to do this will prevent the **secondary** node from +functioning properly. The **secondary** node will generate error +messages, as the following PostgreSQL error: + +``` +ERROR: relation "gitlab_secondary.ci_job_artifacts" does not exist at character 323 +STATEMENT: SELECT a.attname, format_type(a.atttypid, a.atttypmod), + pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod + FROM pg_attribute a LEFT JOIN pg_attrdef d + ON a.attrelid = d.adrelid AND a.attnum = d.adnum + WHERE a.attrelid = '"gitlab_secondary"."ci_job_artifacts"'::regclass + AND a.attnum > 0 AND NOT a.attisdropped + ORDER BY a.attnum +``` + +## Finders + +Geo uses [Finders](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/app/finders), +which are classes take care of the heavy lifting of looking up +projects/attachments/etc. in the tracking database and main database. + +### Finders Performance + +The Finders need to compare data from the main database with data in +the tracking database. For example, counting the number of synced +projects normally involves retrieving the project IDs from one +database and checking their state in the other database. This is slow +and requires a lot of memory. + +To overcome this, the Finders use [FDW](#fdw), or Foreign Data +Wrappers. This allows a regular `JOIN` between the main database and +the tracking database. + +## Redis + +Redis on the **secondary** node works the same as on the **primary** +node. It is used for caching, storing sessions, and other persistent +data. + +Redis data replication between **primary** and **secondary** node is +not used, so sessions etc. aren't shared between nodes. + +## Object Storage + +GitLab can optionally use Object Storage to store data it would +otherwise store on disk. These things can be: + + - LFS Objects + - CI Job Artifacts + - Uploads + +Objects that are stored in object storage, are not handled by Geo. Geo +ignores items in object storage. Either: + +- The object storage layer should take care of its own geographical + replication. +- All secondary nodes should use the same storage node. + +## Verification + +### Repository verification + +Repositories are verified with a checksum. + +The **primary** node calculates a checksum on the repository. It +basically hashes all Git refs together and stores that hash in the +`project_repository_states` table of the database. + +The **secondary** node does the same to calculate the hash of its +clone, and compares the hash with the value the **primary** node +calculated. If there is a mismatch, Geo will mark this as a mismatch +and the administrator can see this in the [Geo admin panel](https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html). + +## Glossary + +### Primary node + +A **primary** node is the single node in a Geo setup that read-write +capabilities. It's the single source of truth and the Geo +**secondary** nodes replicate their data from there. + +In a Geo setup, there can only be one **primary** node. All +**secondary** nodes connect to that **primary**. + +### Secondary node + +A **secondary** node is a read-only replica of the **primary** node +running in a different geographical location. + +### Streaming replication + +Geo depends on the streaming replication feature of PostgreSQL. It +completely replicates the database data and the database schema. The +database replica is a read-only copy. + +Streaming replication depends on the Write Ahead Logs, or WAL. Those +logs are copied over to the replica and replayed there. + +Since streaming replication also replicates the schema, the database +migration do not need to run on the secondary nodes. + +### Tracking database + +A database on each Geo **secondary** node that keeps state for the node +on which it resides. Read more in [Using the Tracking database](#using-the-tracking-database). + +### FDW + +Foreign Data Wrapper, or FDW, is a feature built-in in PostgreSQL. It +allows data to be queried from different data sources. In Geo, it's +used to query data from different PostgreSQL instances. + +## Geo Event Log + +The Geo **primary** stores events in the `geo_event_log` table. Each +entry in the log contains a specific type of event. These type of +events include: + + - Repository Deleted event + - Repository Renamed event + - Repositories Changed event + - Repository Created event + - Hashed Storage Migrated event + - Lfs Object Deleted event + - Hashed Storage Attachments event + - Job Artifact Deleted event + - Upload Deleted event + +### Geo Log Cursor + +The process running on the **secondary** node that looks for new +`Geo::EventLog` rows. + +## Code features + +### `Gitlab::Geo` utilities + +Small utility methods related to Geo go into the +[`ee/lib/gitlab/geo.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/geo.rb) +file. + +Many of these methods are cached using the `RequestStore` class, to +reduce the performance impact of using the methods throughout the +codebase. + +#### Current node + +The class method `.current_node` returns the `GeoNode` record for the +current node. + +We use the `host`, `port`, and `relative_url_root` values from +`gitlab.yml` and search in the database to identify which node we are +in (see `GeoNode.current_node`). + +#### Primary or secondary + +To determine whether the current node is a **primary** node or a +**secondary** node use the `.primary?` and `.secondary?` class +methods. + +It is possible for these methods to both return `false` on a node when +the node is not enabled. See [Enablement](#enablement). + +#### Geo Database configured? + +There is also an additional gotcha when dealing with things that +happen during initialization time. In a few places, we use the +`Gitlab::Geo.geo_database_configured?` method to check if the node has +the tracking database, which only exists on the **secondary** +node. This overcomes race conditions that could happen during +bootstrapping of a new node. + +#### Enablement + +We consider Geo feature enabled when the user has a valid license with the +feature included, and they have at least one node defined at the Geo Nodes +screen. + +See `Gitlab::Geo.enabled?` and `Gitlab::Geo.license_allows?` methods. + +#### Read-only + +All Geo **secondary** nodes are read-only. + +The general principle of a [read-only database](verifying_database_capabilities.md#read-only-database) +applies to all Geo **secondary** nodes. So the +`Gitlab::Database.read_only?` method will always return `true` on a +**secondary** node. + +When some write actions are not allowed because the node is a +**secondary**, consider adding the `Gitlab::Database.read_only?` or +`Gitlab::Database.read_write?` guard, instead of `Gitlab::Geo.secondary?`. + +The database itself will already be read-only in a replicated setup, +so we don't need to take any extra step for that. + +## History of communication channel + +The communication channel has changed since first iteration, you can +check here historic decisions and why we moved to new implementations. + +### Custom code (GitLab 8.6 and earlier) + +In GitLab versions before 8.6, custom code is used to handle +notification from **primary** node to **secondary** nodes by HTTP +requests. + +### System hooks (GitLab 8.7 to 9.5) + +Later, it was decided to move away from custom code and begin using +system hooks. More people were using them, so +many would benefit from improvements made to this communication layer. + +There is a specific **internal** endpoint in our API code (Grape), +that receives all requests from this System Hooks: +`/api/v4/geo/receive_events`. + +We switch and filter from each event by the `event_name` field. + +### Geo Log Cursor (GitLab 10.0 and up) + +Since GitLab 10.0, [System Webhooks](#system-hooks-gitlab-87-to-95) are no longer +used and Geo Log Cursor is used instead. The Log Cursor traverses the +`Geo::EventLog` rows to see if there are changes since the last time +the log was checked and will handle repository updates, deletes, +changes, and renames. + +The table is within the replicated database. This has two advantages over the +old method: + +- Replication is synchronous and we preserve the order of events. +- Replication of the events happen at the same time as the changes in the + database. diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 6dcade3bb51..b9dc3797e5b 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -93,7 +93,7 @@ become available, you will be able to share job templates like this Dependencies should be kept to the minimum. The introduction of a new dependency should be argued in the merge request, as per our [Approval -Guidelines](../code_review.html#approval-guidelines). Both [License +Guidelines](../code_review.md#approval-guidelines). Both [License Management](https://docs.gitlab.com/ee/user/project/merge_requests/license_management.html) **[ULTIMATE]** and [Dependency Scanning](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index aa463e467d4..ac04a21b37a 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -43,6 +43,7 @@ are very appreciative of the work done by translators and proofreaders! - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) - German - Michael Hahnle - [GitLab](https://gitlab.com/mhah), [Crowdin](https://crowdin.com/profile/mhah) + - Katrin Leinweber - [GitLab](https://gitlab.com/katrinleinweber/), [Crowdin](https://crowdin.com/profile/katrinleinweber) - Greek - Proofreaders needed. - Hebrew diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md new file mode 100644 index 00000000000..5bf43d320c6 --- /dev/null +++ b/doc/development/integrations/jira_connect.md @@ -0,0 +1,41 @@ +# Setting up a development environment + +The following are required to install and test the app: + +1. A Jira Cloud instance + + Atlassian provides free instances for development and testing. [Click here to sign up](http://go.atlassian.com/cloud-dev). + +1. A GitLab instance available over the internet + + For the app to work, Jira Cloud should be able to connect to the GitLab instance through the internet. + + To easily expose your local development environment, you can use tools like [serveo](https://serveo.net) or [ngrok](https://ngrok.com). + These also take care of SSL for you because Jira requires all connections to the app host to be over SSL. + +> This feature is currently behind the `:jira_connect_app` feature flag + +# Installing the app in Jira + +1. Enable Jira development mode to install apps that are not from the Atlassian Marketplace + + 1. Navigate to **Jira settings** (cog icon) > **Apps** > **Manage apps**. + 1. Scroll to the bottom of the **Manage apps** page and click **Settings**. + 1. Select **Enable development mode** and click **Apply**. + +1. Install the app + + 1. Navigate to Jira, then choose **Jira settings** (cog icon) > **Apps** > **Manage apps**. + 1. Click **Upload app**. + 1. In the **From this URL** field, provide a link to the app descriptor. The host and port must point to your GitLab instance. + + For example: + ``` + https://xxxx.serveo.net/-/jira_connect/app_descriptor.json + ``` + 1. Click **Upload**. + + If the install was successful, you should see the **GitLab for Jira** app under **Manage apps**. + You can also click **Getting Started** to open the configuration page rendered from your GitLab instance. + + _Note that any changes to the app descriptor requires you to uninstall then reinstall the app._ diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md new file mode 100644 index 00000000000..1657d73e0c9 --- /dev/null +++ b/doc/development/licensed_feature_availability.md @@ -0,0 +1,37 @@ +# Licensed feature availability **[STARTER]** + +As of GitLab 9.4, we've been supporting a simplified version of licensed +feature availability checks via `ee/app/models/license.rb`, both for +on-premise or GitLab.com plans and features. + +## Restricting features scoped by namespaces or projects + +GitLab.com plans are persisted on user groups and namespaces, therefore, if you're adding a +feature such as [Related issues](https://docs.gitlab.com/ee/user/project/issues/related_issues.html) or +[Service desk](https://docs.gitlab.com/ee/user/project/service_desk.html), +it should be restricted on namespace scope. + +1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in + `ee/app/models/license.rb`. Note on `ee/app/models/ee/namespace.rb` that _Bronze_ GitLab.com + features maps to on-premise _EES_, _Silver_ to _EEP_ and _Gold_ to _EEU_. +2. Check using: + +```ruby +project.feature_available?(:feature_symbol) +``` + +## Restricting global features (instance) + +However, for features such as [Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) and +[Load balancing](https://docs.gitlab.com/ee/administration/database_load_balancing.html), which cannot be restricted +to only a subset of projects or namespaces, the check will be made directly in +the instance license. + +1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in + `ee/app/models/license.rb`. +2. Add the same feature symbol to `GLOBAL_FEATURES` +3. Check using: + +```ruby +License.feature_available?(:feature_symbol) +``` diff --git a/doc/development/packages.md b/doc/development/packages.md new file mode 100644 index 00000000000..a3b891d7834 --- /dev/null +++ b/doc/development/packages.md @@ -0,0 +1,68 @@ +# Packages **[PREMIUM]** + +This document will guide you through adding another [package management system](https://docs.gitlab.com/ee/administration/packages.html) support to GitLab. + +See already supported package types in [Packages documentation](https://docs.gitlab.com/ee/administration/packages.html) + +Since GitLab packages' UI is pretty generic, it is possible to add new +package system support by solely backend changes. This guide is superficial and does +not cover the way the code should be written. However, you can find a good example +by looking at existing merge requests with Maven and NPM support: + +- [NPM registry support](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8673). +- [Maven repository](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6607). +- [Instance level endpoint for Maven repository](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8757) + +## General information + +The existing database model requires the following: + +- Every package belongs to a project. +- Every package file belongs to a package. +- A package can have one or more package files. +- The package model is based on storing information about the package and its version. + +## API endpoints + +Package systems work with GitLab via API. For example `ee/lib/api/npm_packages.rb` +implements API endpoints to work with NPM clients. So, the first thing to do is to +add a new `ee/lib/api/your_name_packages.rb` file with API endpoints that are +necessary to make the package system client to work. Usually that means having +endpoints like: + +- GET package information. +- GET package file content. +- PUT upload package. + +Since the packages belong to a project, it's expected to have project-level endpoint +for uploading and downloading them. For example: + +``` +GET https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ +PUT https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ +``` + +Group-level and instance-level endpoints are good to have but are optional. + +NOTE: **Note:** +To avoid name conflict for instance-level endpoints we use +[the package naming convention](https://docs.gitlab.com/ee/user/project/packages/npm_registry.html#package-naming-convention) + +## Configuration + +GitLab has a `packages` section in its configuration file (`gitlab.rb`). +It applies to all package systems supported by GitLab. Usually you don't need +to add anything there. + +Packages can be configured to use object storage, therefore your code must support it. + +## Database + +The current database model allows you to store a name and a version for each package. +Every time you upload a new package, you can either create a new record of `Package` +or add files to existing record. `PackageFile` should be able to store all file-related +information like the file `name`, `side`, `sha1`, etc. + +If there is specific data necessary to be stored for only one package system support, +consider creating a separate metadata model. See `packages_maven_metadata` table +and `Packages::MavenMetadatum` model as example for package specific data. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 1ae69127295..27fc3231218 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -28,6 +28,24 @@ bin/rake "gitlab:seed:issues[group-path/project-path]" By default, this seeds an average of 2 issues per week for the last 5 weeks per project. +#### Seeding issues for Insights charts **[ULTIMATE]** + +You can seed issues specifically for working with the +[Insights charts](https://docs.gitlab.com/ee/user/group/insights/index.html) with the +`gitlab:seed:insights:issues` task: + +```shell +# All projects +bin/rake gitlab:seed:insights:issues + +# A specific project +bin/rake "gitlab:seed:insights:issues[group-path/project-path]" +``` + +By default, this seeds an average of 10 issues per week for the last 52 weeks +per project. All issues will also be randomly labeled with team, type, severity, +and priority. + ### Automation If you're very sure that you want to **wipe the current database** and refill diff --git a/doc/getting-started/subscription.md b/doc/getting-started/subscription.md new file mode 100644 index 00000000000..65999183d4a --- /dev/null +++ b/doc/getting-started/subscription.md @@ -0,0 +1,3 @@ +--- +redirect_to: '../subscriptions/index.md' +---
\ No newline at end of file diff --git a/doc/git_hooks/git_hooks.md b/doc/git_hooks/git_hooks.md new file mode 100644 index 00000000000..9b8ad1578a0 --- /dev/null +++ b/doc/git_hooks/git_hooks.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/push_rules/push_rules.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/push_rules/push_rules.html) diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index 3e99496d531..785e2ffb650 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -16,6 +16,7 @@ To create a project in GitLab: - [Import a project](../user/project/import/index.md) from a different repository, if enabled on your GitLab instance. Contact your GitLab admin if this is unavailable. + - Run [CI/CD pipelines for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html). **[PREMIUM]** ## Blank projects @@ -57,7 +58,7 @@ Built-in templates are project templates that are: To use a built-in template on the **New project** page: -1. On the **Create from template** tab. +1. On the **Create from template** tab, select the **Built-in** tab. 1. From the list of available built-in templates, click the: - **Preview** button to look at the template source itself. - **Use template** button to start creating the project. diff --git a/doc/gitlab-geo/README.md b/doc/gitlab-geo/README.md new file mode 100644 index 00000000000..30d21db7de5 --- /dev/null +++ b/doc/gitlab-geo/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/index.md' +--- + +This document was moved to [another location](../administration/geo/replication/index.md). diff --git a/doc/gitlab-geo/after_setup.md b/doc/gitlab-geo/after_setup.md new file mode 100644 index 00000000000..c8a7b9d1096 --- /dev/null +++ b/doc/gitlab-geo/after_setup.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/using_a_geo_server.md' +--- + +This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/gitlab-geo/bring-primary-back.md b/doc/gitlab-geo/bring-primary-back.md new file mode 100644 index 00000000000..8c43f4d805f --- /dev/null +++ b/doc/gitlab-geo/bring-primary-back.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/bring_primary_back.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/bring_primary_back.md). diff --git a/doc/gitlab-geo/configuration.md b/doc/gitlab-geo/configuration.md new file mode 100644 index 00000000000..b46a2caea4a --- /dev/null +++ b/doc/gitlab-geo/configuration.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/configuration.md' +--- + +This document was moved to [another location](../administration/geo/replication/configuration.md). diff --git a/doc/gitlab-geo/configuration_source.md b/doc/gitlab-geo/configuration_source.md new file mode 100644 index 00000000000..f1aab86fadc --- /dev/null +++ b/doc/gitlab-geo/configuration_source.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/configuration_source.md' +--- + +This document was moved to [another location](../administration/geo/replication/configuration_source.md). diff --git a/doc/gitlab-geo/database.md b/doc/gitlab-geo/database.md new file mode 100644 index 00000000000..b4156dc4ec6 --- /dev/null +++ b/doc/gitlab-geo/database.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/database.md' +--- + +This document was moved to [another location](../administration/geo/replication/database.md). diff --git a/doc/gitlab-geo/database_source.md b/doc/gitlab-geo/database_source.md new file mode 100644 index 00000000000..3392d0f02c0 --- /dev/null +++ b/doc/gitlab-geo/database_source.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/database_source.md' +--- + +This document was moved to [another location](../administration/geo/replication/database_source.md). diff --git a/doc/gitlab-geo/disaster-recovery.md b/doc/gitlab-geo/disaster-recovery.md new file mode 100644 index 00000000000..d42e815a879 --- /dev/null +++ b/doc/gitlab-geo/disaster-recovery.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/index.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/index.md). diff --git a/doc/gitlab-geo/docker_registry.md b/doc/gitlab-geo/docker_registry.md new file mode 100644 index 00000000000..26a708f6845 --- /dev/null +++ b/doc/gitlab-geo/docker_registry.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/docker_registry.md' +--- + +This document was moved to [another location](../administration/geo/replication/docker_registry.md). diff --git a/doc/gitlab-geo/faq.md b/doc/gitlab-geo/faq.md new file mode 100644 index 00000000000..f1952bc7e4c --- /dev/null +++ b/doc/gitlab-geo/faq.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/faq.md' +--- + +This document was moved to [another location](../administration/geo/replication/faq.md). diff --git a/doc/gitlab-geo/ha.md b/doc/gitlab-geo/ha.md new file mode 100644 index 00000000000..23ed11eaf09 --- /dev/null +++ b/doc/gitlab-geo/ha.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/high_availability.md' +--- + +This document was moved to [another location](../administration/geo/replication/high_availability.md). diff --git a/doc/gitlab-geo/object_storage.md b/doc/gitlab-geo/object_storage.md new file mode 100644 index 00000000000..1f29b7b7e8c --- /dev/null +++ b/doc/gitlab-geo/object_storage.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/object_storage.md' +--- + +This document was moved to [another location](../administration/geo/replication/object_storage.md). diff --git a/doc/gitlab-geo/planned-failover.md b/doc/gitlab-geo/planned-failover.md new file mode 100644 index 00000000000..720b6bc9424 --- /dev/null +++ b/doc/gitlab-geo/planned-failover.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/planned_failover.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/planned_failover.md). diff --git a/doc/gitlab-geo/security-review.md b/doc/gitlab-geo/security-review.md new file mode 100644 index 00000000000..a0a5b0e536c --- /dev/null +++ b/doc/gitlab-geo/security-review.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/security_review.md' +--- + +This document was moved to [another location](../administration/geo/replication/security_review.md). diff --git a/doc/gitlab-geo/ssh.md b/doc/gitlab-geo/ssh.md new file mode 100644 index 00000000000..4f8db687850 --- /dev/null +++ b/doc/gitlab-geo/ssh.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/operations/fast_ssh_key_lookup.md' +--- + +This document was moved to [another location](../administration/operations/fast_ssh_key_lookup.md). diff --git a/doc/gitlab-geo/troubleshooting.md b/doc/gitlab-geo/troubleshooting.md new file mode 100644 index 00000000000..25fe1372c69 --- /dev/null +++ b/doc/gitlab-geo/troubleshooting.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/troubleshooting.md' +--- + +This document was moved to [another location](../administration/geo/replication/troubleshooting.md). diff --git a/doc/gitlab-geo/tuning.md b/doc/gitlab-geo/tuning.md new file mode 100644 index 00000000000..84ac40f99db --- /dev/null +++ b/doc/gitlab-geo/tuning.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/tuning.md' +--- + +This document was moved to [another location](../administration/geo/replication/tuning.md). diff --git a/doc/gitlab-geo/updating_the_geo_nodes.md b/doc/gitlab-geo/updating_the_geo_nodes.md new file mode 100644 index 00000000000..28234ec02ed --- /dev/null +++ b/doc/gitlab-geo/updating_the_geo_nodes.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/updating_the_geo_nodes.md' +--- + +This document was moved to [another location](../administration/geo/replication/updating_the_geo_nodes.md). diff --git a/doc/gitlab-geo/using_a_geo_server.md b/doc/gitlab-geo/using_a_geo_server.md new file mode 100644 index 00000000000..c8a7b9d1096 --- /dev/null +++ b/doc/gitlab-geo/using_a_geo_server.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/using_a_geo_server.md' +--- + +This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/install/ldap.md b/doc/install/ldap.md new file mode 100644 index 00000000000..a19f0342b65 --- /dev/null +++ b/doc/install/ldap.md @@ -0,0 +1,7 @@ +--- +redirect_to: '../administration/auth/ldap.md' +--- + +# GitLab LDAP integration + +This document was moved under [`administration/auth/ldap`](../administration/auth/ldap.md). diff --git a/doc/install/pivotal/index.md b/doc/install/pivotal/index.md new file mode 100644 index 00000000000..896e01ad975 --- /dev/null +++ b/doc/install/pivotal/index.md @@ -0,0 +1,45 @@ +# GitLab Pivotal Tile **[PREMIUM ONLY]** + +> Introduced in [GitLab Premium][eep] 8.2. + +Easily deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for +[Pivotal Cloud Foundry][pcf]. + +## Overview + +Enterprise admins want their development toolkit to be more customizable, more +integrated, and more secure. With Pivotal Cloud Foundry, GitLab is installed and +scales easily in a highly available environment. + +The upgrades are pain-free and well tested. All it takes is upload the new tile +and click a button to begin the upgrade process. + +## Use cases + +- You want a highly available deployment with minimal effort. Scale horizontally + as your user base grows. + +## Features + +The GitLab Pivotal Tile is based on [GitLab Premium][eep] and includes nearly all of its features. The features in Premium but _not_ supported on the Tile are: + +* PostgreSQL +* Pages +* Geo +* Registry +* Mattermost +* Subgroups +* Elasticsearch +* Service Desk +* OAuth & Kerberos Authentication + +## Installing GitLab with Pivotal + +The product information and installation documentation is hosted on Pivotal's +website: + +- [Product page](https://network.pivotal.io/products/p-gitlab/) +- [Documentation](https://docs.pivotal.io/partners/gitlab/index.html) + +[eep]: https://about.gitlab.com/pricing/ +[pcf]: https://pivotal.io/platform diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 9a6c2ce1976..17099c1d051 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -105,8 +105,10 @@ features of GitLab work with MySQL/MariaDB: 1. MySQL support for subgroups was [dropped with GitLab 9.3][post]. See [issue #30472][30472] for more information. -1. Geo does [not support MySQL](https://docs.gitlab.com/ee/administration/geo/replication/database.html#mysql-replication). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** +1. Geo does [not support MySQL](https://docs.gitlab.com/ee/administration/geo/replication/database.html). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** 1. [Zero downtime migrations](../update/README.md#upgrading-without-downtime) do not work with MySQL. +1. [Database load balancing](https://docs.gitlab.com/ee/administration/database_load_balancing.html) is + supported only for PostgreSQL. **[PREMIUM ONLY]** 1. GitLab [optimizes the loading of dashboard events](https://gitlab.com/gitlab-org/gitlab-ce/issues/31806) using [PostgreSQL LATERAL JOINs](https://blog.heapanalytics.com/postgresqls-powerful-new-join-type-lateral/). 1. In general, SQL optimized for PostgreSQL may run much slower in MySQL due to differences in query planners. For example, subqueries that work well in PostgreSQL @@ -141,7 +143,17 @@ On some systems you may need to install an additional package (e.g. #### Additional requirements for GitLab Geo -If you are using [GitLab Geo](https://docs.gitlab.com/ee/development/geo.html), the [tracking database](https://docs.gitlab.com/ee/development/geo.html#geo-tracking-database) also requires the `postgres_fdw` extension. +If you are using [GitLab Geo](https://docs.gitlab.com/ee/development/geo.html): + +- We strongly recommend running Omnibus-managed instances as they are actively + developed and tested. We aim to be compatible with most external (not managed + by Omnibus) databases (for example, AWS RDS) but we do not guarantee + compatibility. +- The + [tracking database](https://docs.gitlab.com/ee/development/geo.html#geo-tracking-database) + requires the + [postgres_fdw](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + extension. ``` CREATE EXTENSION postgres_fdw; diff --git a/doc/integration/README.md b/doc/integration/README.md index a539933f223..1fea6a32c28 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -15,7 +15,9 @@ See the documentation below for details on how to configure these services. - [CAS](cas.md) Configure GitLab to sign in using CAS - [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages +- [Jenkins](jenkins.md) Integrate with the Jenkins CI - [JIRA](../user/project/integrations/jira.md) Integrate with the JIRA issue tracker +- [Kerberos](kerberos.md) Integrate with Kerberos - [LDAP](ldap.md) Set up sign in via LDAP - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, SAML, Crowd, Azure and Authentiq ID diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md new file mode 100644 index 00000000000..7cef664bc98 --- /dev/null +++ b/doc/integration/elasticsearch.md @@ -0,0 +1,584 @@ +# Elasticsearch integration **[STARTER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request") in GitLab [Starter](https://about.gitlab.com/pricing/) 8.4. Support +> for [Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) was [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305) in GitLab +> [Starter](https://about.gitlab.com/pricing/) 9.0. + +This document describes how to set up Elasticsearch with GitLab. Once enabled, +you'll have the benefit of fast search response times and the advantage of two +special searches: + +- [Advanced Global Search](https://docs.gitlab.com/ee/user/search/advanced_global_search.html) +- [Advanced Syntax Search](https://docs.gitlab.com/ee/user/search/advanced_search_syntax.html) + +## Version Requirements +<!-- Please remember to update ee/lib/system_check/app/elasticsearch_check.rb if this changes --> + +| GitLab version | Elasticsearch version | +| -------------- | --------------------- | +| GitLab Enterprise Edition 8.4 - 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed | +| GitLab Enterprise Edition 9.0 - 11.4 | Elasticsearch 5.1 - 5.5 | +| GitLab Enterprise Edition 11.5+ | Elasticsearch 5.6 - 6.x | + +## Installing Elasticsearch + +Elasticsearch is _not_ included in the Omnibus packages. You will have to +install it yourself whether you are using the Omnibus package or installed +GitLab from source. Providing detailed information on installing Elasticsearch +is out of the scope of this document. + +Once the data is added to the database or repository and [Elasticsearch is +enabled in the admin area](#enabling-elasticsearch) the search index will be +updated automatically. Elasticsearch can be installed on the same machine as +GitLab or on a separate server, or you can use the [Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) +service. + +You can follow the steps as described in the [official web site](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html "Elasticsearch installation documentation") or +use the packages that are available for your OS. + +## Elasticsearch repository indexer (beta) + +In order to improve elasticsearch indexing performance, GitLab has made available a [new indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +This will replace the included Ruby indexer in the future but should be considered beta software for now, so there may be some bugs. + +If you would like to use it, please follow the instructions below. + +### Installation + +First, we need to install some dependencies, then we'll build and install +the indexer itself. + +#### Dependencies + +This project relies on [ICU](http://site.icu-project.org/) for text encoding, +therefore we need to ensure the development packages for your platform are +installed before running `make`. + +##### Debian / Ubuntu + +To install on Debian or Ubuntu, run: + +```sh +sudo apt install libicu-dev +``` + +##### CentOS / RHEL + +To install on CentOS or RHEL, run: + +```sh +sudo yum install libicu-devel +``` + +##### Mac OSX + +To install on macOS, run: + +```sh +brew install icu4c +export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" +``` + +#### Building and installing + +To build and install the indexer, run: + +```sh +git clone https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer.git +cd /gitlab-elasticsearch-indexer +make +sudo make install +``` + +The `gitlab-elasticsearch-indexer` will be installed to `/usr/local/bin`. + +You can change the installation path with the `PREFIX` env variable. +Please remember to pass the `-E` flag to `sudo` if you do so. + +Example: + +```sh +PREFIX=/usr sudo -E make install +``` + +Once installed, enable it under your instance's elasticsearch settings explained [below](#enabling-elasticsearch). + +## System Requirements + +Elasticsearch requires additional resources in excess of those documented in the +[GitLab system requirements](../install/requirements.md). These will vary by +installation size, but you should ensure **at least** an additional **8 GiB of RAM** +for each Elasticsearch node, per the [official guidelines](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). + +Keep in mind, this is the **minimum requirements** as per Elasticsearch. For +production instances, they recommend considerably more resources. + +Storage requirements also vary based on the installation side, but as a rule of +thumb, you should allocate the total size of your production database, **plus** +two-thirds of the total size of your git repositories. Efforts to reduce this +total are being tracked in this epic: [gitlab-org&153](https://gitlab.com/groups/gitlab-org/-/epics/153). + +## Enabling Elasticsearch + +In order to enable Elasticsearch, you need to have admin access. Go to +**Admin > Settings > Integrations** and find the "Elasticsearch" section. + +The following Elasticsearch settings are available: + +| Parameter | Description | +| --------- | ----------- | +| `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). | +| `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). | +| `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. | +| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., "http://host1, https://host2:9200"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects). +| `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or [AWS EC2 Instance Profile Credentials](http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli). The policies must be configured to allow `es:*` actions. | +| `AWS Region` | The AWS region your Elasticsearch service is located in. | +| `AWS Access Key` | The AWS access key. | +| `AWS Secret Access Key` | The AWS secret access key. | + +### Limiting namespaces and projects + +If you select `Limit namespaces and projects that can be indexed`, more options will become available + + +You can select namespaces and projects to index exclusively. Please note that if the namespace is a group it will include +any sub-groups and projects belonging to those sub-groups to be indexed as well. + +You can filter the selection dropdown by writing part of the namespace or project name you're interested in. + + +NOTE: **Note**: +If no namespaces or projects are selected, no Elasticsearch indexing will take place. + +CAUTION: **Warning**: +If you have already indexed your instance, you will have to regenerate the index in order to delete all existing data +for filtering to work correctly. To do this run the rake tasks `gitlab:elastic:create_empty_index` and +`gitlab:elastic:clear_index_status` Afterwards, removing a namespace or a projeect from the list will delete the data +from the Elasticsearch index as expected. + +## Disabling Elasticsearch + +To disable the Elasticsearch integration: + +1. Navigate to the **Admin > Settings > Integrations** +1. Find the 'Elasticsearch' section and uncheck 'Search with Elasticsearch enabled' + and 'Elasticsearch indexing' +1. Click **Save** for the changes to take effect +1. [Optional] Delete the existing index by running the command `sudo gitlab-rake gitlab:elastic:delete_index` + +## Adding GitLab's data to the Elasticsearch index + +### Indexing small instances (database size less than 500 MiB, size of repos less than 5 GiB) + +Configure Elasticsearch's host and port in **Admin > Settings**. Then index the data using one of the following commands: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index + +# Installations from source +bundle exec rake gitlab:elastic:index RAILS_ENV=production +``` + +After it completes the indexing process, [enable Elasticsearch searching](elasticsearch.md#enabling-elasticsearch). + +### Indexing large instances + +WARNING: **Warning**: +Performing asynchronous indexing, as this will describe, will generate a lot of sidekiq jobs. +Make sure to prepare for this task by either [Horizontally Scaling](../administration/high_availability/README.md#basic-scaling) +or creating [extra sidekiq processes](../administration/operations/extra_sidekiq_processes.md) + +NOTE: **Note**: +After indexing the repositories asynchronously, you **MUST** index the database to be able to search. + +Configure Elasticsearch's host and port in **Admin > Settings > Integrations**. Then create empty indexes using one of the following commands: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:create_empty_index + +# Installations from source +bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production +``` + +Indexing large Git repositories can take a while. To speed up the process, you +can temporarily disable auto-refreshing and replicating. In our experience, you can expect a 20% +decrease in indexing time. We'll enable them when indexing is done. This step is optional! + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "refresh_interval" : "-1", + "number_of_replicas" : 0 + } }' +``` + +Then enable Elasticsearch indexing and run repository indexing tasks: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories_async + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories_async RAILS_ENV=production +``` + +This enqueues a number of Sidekiq jobs to index your existing repositories. +You can view the jobs in the admin panel (they are placed in the `elastic_batch_project_indexer`) +queue), or you can query indexing status using a rake task: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories_status + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories_status RAILS_ENV=production + +Indexing is 65.55% complete (6555/10000 projects) +``` + +By default, one job is created for every 300 projects. For large numbers of +projects, you may wish to increase the batch size, by setting the `BATCH` +environment variable. + +You can also run the initial indexing synchronously - this is most useful if +you have a small number of projects or need finer-grained control over indexing +than Sidekiq permits: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production +``` + +It might take a while depending on how big your Git repositories are. + +If you want to run several tasks in parallel (probably in separate terminal +windows) you can provide the `ID_FROM` and `ID_TO` parameters: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories ID_FROM=1001 ID_TO=2000 + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories ID_FROM=1001 ID_TO=2000 RAILS_ENV=production +``` + +Where `ID_FROM` and `ID_TO` are project IDs. Both parameters are optional. +As an example, if you have 3,000 repositories and you want to run three separate indexing tasks, you might run: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories ID_TO=1000 +sudo gitlab-rake gitlab:elastic:index_repositories ID_FROM=1001 ID_TO=2000 +sudo gitlab-rake gitlab:elastic:index_repositories ID_FROM=2001 + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production ID_TO=1000 +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production ID_FROM=1001 ID_TO=2000 +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production ID_FROM=2001 +``` + +Sometimes your repository index process `gitlab:elastic:index_repositories` or +`gitlab:elastic:index_repositories_async` can get interrupted. This may happen +for many reasons, but it's always safe to run the indexing job again - it will +skip those repositories that have already been indexed. + +As the indexer stores the last commit SHA of every indexed repository in the +database, you can run the indexer with the special parameter `UPDATE_INDEX` and +it will check every project repository again to make sure that every commit in +that repository is indexed, it can be useful in case if your index is outdated: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories UPDATE_INDEX=true ID_TO=1000 + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories UPDATE_INDEX=true ID_TO=1000 RAILS_ENV=production +``` + +You can also use the `gitlab:elastic:clear_index_status` Rake task to force the +indexer to "forget" all progress, so retrying the indexing process from the +start. + +To index all wikis: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_wikis + +# Installations from source +bundle exec rake gitlab:elastic:index_wikis RAILS_ENV=production +``` + +The wiki indexer also supports the `ID_FROM` and `ID_TO` parameters if you want +to limit a project set. + +Index all database entities (Keep in mind it can take a while, so consider using `screen` or `tmux`): + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_database + +# Installations from source +bundle exec rake gitlab:elastic:index_database RAILS_ENV=production +``` + +Enable replication and refreshing again after indexing (only if you previously disabled it): + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "number_of_replicas" : 1, + "refresh_interval" : "1s" + } }' +``` + +A force merge should be called after enabling the refreshing above: + +```bash +curl --request POST 'http://localhost:9200/_forcemerge?max_num_segments=5' +``` + +Enable Elasticsearch search in **Admin > Settings > Integrations**. That's it. Enjoy it! + +## GitLab Elasticsearch Rake Tasks + +There are several rake tasks available to you via the command line: + +- [sudo gitlab-rake gitlab:elastic:index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This is a wrapper task. It does the following: + - `sudo gitlab-rake gitlab:elastic:create_empty_index` + - `sudo gitlab-rake gitlab:elastic:clear_index_status` + - `sudo gitlab-rake gitlab:elastic:index_wikis` + - `sudo gitlab-rake gitlab:elastic:index_database` + - `sudo gitlab-rake gitlab:elastic:index_repositories` +- [sudo gitlab-rake gitlab:elastic:index_repositories_async](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This iterates over all projects and places them in batches. It then sends these batches to the background via sidekiq jobs to be indexed. +- [sudo gitlab-rake gitlab:elastic:index_repositories_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. +- [sudo gitlab-rake gitlab:elastic:index_repositories](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This iterates over all projects and places them in batches. It then performs indexing on said batches synchronously. +- [sudo gitlab-rake gitlab:elastic:index_wikis](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Iterates over every project, determines if said project contains wiki data, and then indexes the blobs (content) of said wiki data. +- [sudo gitlab-rake gitlab:elastic:index_database](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This is a [rake multitask](https://www.rubydoc.info/github/ruby/rake/Rake/MultiTask). It does the following: + - `sudo gitlab-rake gitlab:elastic:index_projects` + - `sudo gitlab-rake gitlab:elastic:index_issues` + - `sudo gitlab-rake gitlab:elastic:index_merge_requests` + - `sudo gitlab-rake gitlab:elastic:index_snippets` + - `sudo gitlab-rake gitlab:elastic:index_notes` + - `sudo gitlab-rake gitlab:elastic:index_milestones` +- [sudo gitlab-rake gitlab:elastic:create_empty_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This generates an empty index on the Elasticsearch side. +- [sudo gitlab-rake gitlab:elastic:clear_index_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This deletes all instances of IndexStatus for all projects. +- [sudo gitlab-rake gitlab:elastic:delete_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This removes the GitLab index on the Elasticsearch instance. +- [sudo gitlab-rake gitlab:elastic:recreate_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Does the same thing as `sudo gitlab-rake gitlab:elastic:create_empty_index` +- [sudo gitlab-rake gitlab:elastic:add_feature_visibility_levels_to_project](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Adds visibility information to the indices for projects. +- [sudo gitlab-rake gitlab:elastic:index_projects](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Performs an Elasticsearch import that indexes projects data. +- [sudo gitlab-rake gitlab:elastic:index_issues](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Performs an Elasticsearch import that indexes issues data. +- [sudo gitlab-rake gitlab:elastic:index_merge_requests](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Performs an Elasticsearch import that indexes merge requests data. +- [sudo gitlab-rake gitlab:elastic:index_snippets](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Performs an Elasticsearch import that indexes the snippets data. +- [sudo gitlab-rake gitlab:elastic:index_notes](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Performs an Elasticsearch import that indexes the notes data. +- [sudo gitlab-rake gitlab:elastic:index_milestones](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Performs an Elasticsearch import that indexes the milestones data. + +### Environment Variables + +In addition to the rake tasks, there are some environment variables that can be used to modify the process: + +| Environment Variable | Data Type | What it does | +| -------------------- |:---------:| ---------------------------------------------------------------------------- | +| `BATCH` | Integer | Modifies the size of the indexing batch (default 300 projects). | +| `UPDATE_INDEX` | Boolean | Tells the indexer to overwrite any existing index data (true/false). | +| `ID_TO` | Integer | Tells the indexer to only index projects less than or equal to the value. | +| `ID_FROM` | Integer | Tells the indexer to only index projects greater than or equal to the value. | + +### Batching + +The ability to apply batching makes the indexer run more efficiently. The default +size of a batch is 300 projects, which may or may not be ideal for your setup. +Depending on the resources available to your GitLab instance (sidekiq) and your +Elasticsearch instance (RAM, CPU), you may be able to increase or decrease the +batch size for more efficiency. + +- The larger the batch size is, the less sidekiq jobs and indexing requests get created. +- The larger the batch size is, the more time and RAM it takes to process. +- The smaller the batch size, the more sidekiq jobs, and indexing requests get created. +- The smaller the batch size, the more CPU gets utilized. + +Finding the ideal size can be tricky, and will vary from GitLab instance to GitLab instance. +Generally speaking, if the default is not ideal for you, try reducing it to somewhere in +the 50-150 range (for bigger sized repos) or 450-600 range (for many small-sized repos). + +Example use: + +```sh +sudo gitlab-rake gitlab:elastic:index_repositories_async BATCH=50 +``` + +### Indexing a specific project + +Because the `ID_TO` and `ID_FROM` environment variables use the `or equal to` comparison, you can index only one project by using both these variables with the same project ID number: + +```sh +root@git:~# sudo gitlab-rake gitlab:elastic:index_repositories ID_TO=5 ID_FROM=5 +Indexing project repositories...I, [2019-03-04T21:27:03.083410 #3384] INFO -- : Indexing GitLab User / test (ID=33)... +I, [2019-03-04T21:27:05.215266 #3384] INFO -- : Indexing GitLab User / test (ID=33) is done! +``` + +## Elasticsearch Index Scopes + +When performing a search, the GitLab index will use the following scopes: + +| Scope Name | What it searches | +| ---------------- | ---------------------- | +| `commits` | Commit data | +| `projects` | Project data (default) | +| `blobs` | Code | +| `issues` | Issue data | +| `merge_requests` | Merge Request data | +| `milestones` | Milestone data | +| `notes` | Note data | +| `snippets` | Snippet data | +| `wiki_blobs` | Wiki contents | + +## Tuning + +### Deleted documents + +Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index. + +In general, we recommend simply letting Elasticseach merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ + +However, some larger installations may wish to tune the merge policy settings: + +- Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently. + + ```bash + curl --request PUT http://localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "merge.policy.max_merged_segment": "2gb" + } + }' + ``` + +- You can also adjust `index.merge.policy.reclaim_deletes_weight`, which controls how aggressively deletions are targeted. But this can lead to costly merge decisions, so we recommend not changing this unless you understand the tradeoffs. + + ```bash + curl --request PUT http://localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "merge.policy.reclaim_deletes_weight": "3.0" + } + }' + ``` + +- Do not do a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") to remove deleted documents. A warning in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") states that this can lead to very large segments that may never get reclaimed, and can also cause significant performance or availability issues. + +## Troubleshooting + +Here are some common pitfalls and how to overcome them: + +- **How can I verify my GitLab instance is using Elasticsearch?** + + The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following: + + ```ruby + u = User.find_by_username('your-username') + s = SearchService.new(u, {:search => 'search_term'}) + pp s.search_objects.class.name + ``` + + If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. + +- **I updated GitLab and now I can't find anything** + + We continuously make updates to our indexing strategies and aim to support + newer versions of Elasticsearch. When indexing changes are made, it may + be necessary for you to [reindex](#adding-gitlabs-data-to-the-elasticsearch-index) after updating GitLab. + +- **I indexed all the repositories but I can't find anything** + + Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index). + + Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side. + + If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`): + + ```ruby + u = User.find_by_username('your-username') + s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’}) + pp s.search_objects.to_a + ``` + + See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data. + +- **I indexed all the repositories but then switched elastic search servers and now I can't find anything** + + You will need to re-run all the rake tasks to re-index the database, repositories, and wikis. + +- **The indexing process is taking a very long time** + + The more data present in your GitLab instance, the longer the indexing process takes. You might want to try adjusting the BATCH sizes for asynchronous indexing to help speed up the process. + +- **No new data is added to the Elasticsearch index when I push code** + + When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could + happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, + run the `gitlab:elastic:clear_locked_projects` rake task. + +- **"Can't specify parent if no parent field has been configured"** + + If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get + exception in lots of different cases: + + ```text + Elasticsearch::Transport::Transport::Errors::BadRequest([400] { + "error": { + "root_cause": [{ + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }], + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }, + "status": 400 + }): + ``` + + This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again, + see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch). + +- Exception `Elasticsearch::Transport::Transport::Errors::BadRequest` + + If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#system-requirements). + There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command. + +- Exception `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge` + + ```text + [413] {"Message":"Request size exceeded 10485760 bytes"} + ``` + + This exception is seen when your Elasticsearch cluster is configured to reject + requests above a certain size (10MiB in this case). This corresponds to the + `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a + larger size and restart your Elasticsearch cluster. + + AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html) + for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of + the underlying instance. + diff --git a/doc/integration/img/jenkins_gitlab_plugin_config.png b/doc/integration/img/jenkins_gitlab_plugin_config.png Binary files differnew file mode 100644 index 00000000000..3e1f83e96dc --- /dev/null +++ b/doc/integration/img/jenkins_gitlab_plugin_config.png diff --git a/doc/integration/img/jenkins_gitlab_service.png b/doc/integration/img/jenkins_gitlab_service.png Binary files differnew file mode 100644 index 00000000000..682a5ae8ee2 --- /dev/null +++ b/doc/integration/img/jenkins_gitlab_service.png diff --git a/doc/integration/img/jenkins_gitlab_service_settings.png b/doc/integration/img/jenkins_gitlab_service_settings.png Binary files differnew file mode 100644 index 00000000000..5a12e9cb39a --- /dev/null +++ b/doc/integration/img/jenkins_gitlab_service_settings.png diff --git a/doc/integration/img/jenkins_project.png b/doc/integration/img/jenkins_project.png Binary files differnew file mode 100644 index 00000000000..126b05c8879 --- /dev/null +++ b/doc/integration/img/jenkins_project.png diff --git a/doc/integration/img/jira_dev_panel_gl_setup_1.png b/doc/integration/img/jira_dev_panel_gl_setup_1.png Binary files differnew file mode 100644 index 00000000000..75279877c93 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_gl_setup_1.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_1.png b/doc/integration/img/jira_dev_panel_jira_setup_1.png Binary files differnew file mode 100644 index 00000000000..5c0f594cc1d --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_1.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_2.png b/doc/integration/img/jira_dev_panel_jira_setup_2.png Binary files differnew file mode 100644 index 00000000000..a4778a00dd5 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_2.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_3.png b/doc/integration/img/jira_dev_panel_jira_setup_3.png Binary files differnew file mode 100644 index 00000000000..4049a65f56b --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_3.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_4.png b/doc/integration/img/jira_dev_panel_jira_setup_4.png Binary files differnew file mode 100644 index 00000000000..81d84cb173d --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_4.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_5.png b/doc/integration/img/jira_dev_panel_jira_setup_5.png Binary files differnew file mode 100644 index 00000000000..73dc867d301 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_5.png diff --git a/doc/integration/img/jira_dev_panel_manual_refresh.png b/doc/integration/img/jira_dev_panel_manual_refresh.png Binary files differnew file mode 100644 index 00000000000..dc92d533bde --- /dev/null +++ b/doc/integration/img/jira_dev_panel_manual_refresh.png diff --git a/doc/integration/img/limit_namespace_filter.png b/doc/integration/img/limit_namespace_filter.png Binary files differnew file mode 100644 index 00000000000..88f5caa41db --- /dev/null +++ b/doc/integration/img/limit_namespace_filter.png diff --git a/doc/integration/img/limit_namespaces_projects_options.png b/doc/integration/img/limit_namespaces_projects_options.png Binary files differnew file mode 100644 index 00000000000..488341f7e92 --- /dev/null +++ b/doc/integration/img/limit_namespaces_projects_options.png diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md new file mode 100644 index 00000000000..f60333aa07c --- /dev/null +++ b/doc/integration/jenkins.md @@ -0,0 +1,129 @@ +# Jenkins CI service **[STARTER]** + +>**Note:** +In GitLab 8.3, Jenkins integration using the +[GitLab Hook Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +was deprecated in favor of the +[GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). +The deprecated integration has been renamed to [Jenkins CI (Deprecated)](jenkins_deprecated.md) in the +project service settings. We may remove this in a future release and recommend +using the new 'Jenkins CI' project service instead which is described in this +document. + +## Overview + +[Jenkins](https://jenkins.io/) is a great Continuous Integration tool, similar to our built-in +[GitLab CI](../ci/README.md). + +GitLab's Jenkins integration allows you to trigger a Jenkins build when you +push code to a repository, or when a merge request is created. Additionally, +it shows the pipeline status on merge requests widgets and on the project's home page. + +## Use cases + +- Suppose you are new to GitLab, and want to keep using Jenkins until you prepare +your projects to build with [GitLab CI/CD](../ci/README.md). You set up the +integration between GitLab and Jenkins, then you migrate to GitLab CI later. While +you organize yourself and your team to onboard GitLab, you keep your pipelines +running with Jenkins, but view the results in your project's repository in GitLab. +- Your team uses [Jenkins Plugins](https://plugins.jenkins.io/) for other proceedings, +therefore, you opt for keep using Jenkins to build your apps. Show the results of your +pipelines directly in GitLab. + +## Requirements + +* [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +* [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin) +* Git clone access for Jenkins from the GitLab repository +* GitLab API access to report build status + +## Configure GitLab users + +Create a user or choose an existing user that Jenkins will use to interact +through the GitLab API. This user will need to be a global Admin or added +as a member to each Group/Project. Developer permission is required for reporting +build status. This is because a successful build status can trigger a merge +when 'Merge when pipeline succeeds' feature is used. Some features of the GitLab +Plugin may require additional privileges. For example, there is an option to +accept a merge request if the build is successful. Using this feature would +require developer, maintainer or owner-level permission. + +Copy the private API token from **Profile Settings -> Account**. You will need this +when configuring the Jenkins server later. + +## Configure the Jenkins server + +Install [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +and [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin). + +Go to Manage Jenkins -> Configure System and scroll down to the 'GitLab' section. +Enter the GitLab server URL in the 'GitLab host URL' field and paste the API token +copied earlier in the 'API Token' field. + +For more information, see GitLab Plugin documentation about +[Jenkins-to-GitLab authentication](https://github.com/jenkinsci/gitlab-plugin#jenkins-to-gitlab-authentication) + + + +## Configure a Jenkins project + +Follow the GitLab Plugin documentation about [Jenkins Job Configuration](https://github.com/jenkinsci/gitlab-plugin#jenkins-job-configuration). + +NOTE: **Note:** +Be sure to include the steps about [Build status configuration](https://github.com/jenkinsci/gitlab-plugin#build-status-configuration). +The 'Publish build status to GitLab' post-build step is required to view +Jenkins build status in GitLab Merge Requests. + +## Configure a GitLab project + +Create a new GitLab project or choose an existing one. Then, go to **Integrations -> +Jenkins CI**. + +Check the 'Active' box. Select whether you want GitLab to trigger a build +on push, Merge Request creation, tag push, or any combination of these. We +recommend unchecking 'Merge Request events' unless you have a specific use-case +that requires re-building a commit when a merge request is created. With 'Push +events' selected, GitLab will build the latest commit on each push and the build +status will be displayed in the merge request. + +Enter the Jenkins URL and Project name. The project name should be URL-friendly +where spaces are replaced with underscores. To be safe, copy the project name +from the URL bar of your browser while viewing the Jenkins project. + +Optionally, enter a username and password if your Jenkins server requires +authentication. + + + +## Plugin functional overview + +GitLab does not contain a database table listing commits. Commits are always +read from the repository directly. Therefore, it is not possible to retain the +build status of a commit in GitLab. This is overcome by requesting build +information from the integrated CI tool. The CI tool is responsible for creating +and storing build status for Commits and Merge Requests. + +### Steps required to implement a similar integration + +>**Note:** +All steps are implemented using AJAX requests on the merge request page. + +1. In order to display the build status in a merge request you must create a project service in GitLab. +2. Your project service will do a (JSON) query to a URL of the CI tool with the SHA1 of the commit. +3. The project service builds this URL and payload based on project service settings and knowledge of the CI tool. +4. The response is parsed to give a response in GitLab (success/failed/pending). + +## Troubleshooting + +### Error in merge requests - "Could not connect to the CI server" + +This integration relies on Jenkins reporting the build status back to GitLab via +the [Commit Status API](../api/commits.md#commit-status). + +The error 'Could not connect to the CI server' usually means that GitLab did not +receive a build status update via the API. Either Jenkins was not properly +configured or there was an error reporting the status via the API. + +1. [Configure the Jenkins server](#configure-the-jenkins-server) for GitLab API access +2. [Configure a Jenkins project](#configure-a-jenkins-project), including the + 'Publish build status to GitLab' post-build action. diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md new file mode 100644 index 00000000000..8001c5dbd83 --- /dev/null +++ b/doc/integration/jenkins_deprecated.md @@ -0,0 +1,53 @@ +# Jenkins CI (deprecated) service + +>**Note:** In GitLab 8.3, Jenkins integration using the +[GitLab Hook Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +was deprecated in favor of the +[GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). +Please use documentation for the new [Jenkins CI service](jenkins.md). + +Integration includes: + +* Trigger Jenkins build after push to repo +* Show build status on Merge Request page + +Requirements: + +* [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +* git clone access for Jenkins from GitLab repo (via ssh key) + +## Jenkins + +1. Install [GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +2. Set up jenkins project + + + +## GitLab + +In GitLab, perform the following steps. + +### Read access to repository + +Jenkins needs read access to the GitLab repository. We already specified a +private key to use in Jenkins, now we need to add a public one to the GitLab +project. For that case we will need a Deploy key. Read the documentation on +[how to set up a Deploy key](../ssh/README.md#deploy-keys). + +### Jenkins service + +Now navigate to GitLab services page and activate Jenkins + + + +Done! Now when you push to GitLab - it will create a build for Jenkins. +And also you will be able to see merge request build status with a link to the Jenkins build. + +### Multi-project Configuration + +The GitLab Hook plugin in Jenkins supports the automatic creation of a project +for each feature branch. After configuration GitLab will trigger feature branch +builds and a corresponding project will be created in Jenkins. + +Configure the GitLab Hook plugin in Jenkins. Go to 'Manage Jenkins' and then +'Configure System'. Find the 'GitLab Web Hook' section and configure as shown below. diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md new file mode 100644 index 00000000000..703736eeb3c --- /dev/null +++ b/doc/integration/jira_development_panel.md @@ -0,0 +1,141 @@ +# GitLab Jira development panel integration **[PREMIUM]** + +> [Introduced][ee-2381] in [GitLab Premium][eep] 10.0. + +Complementary to our [existing Jira][existing-jira] project integration, you're now able to integrate +GitLab projects with [Jira Development Panel][jira-development-panel]. Both can be used +simultaneously. This works with self-hosted GitLab or GitLab.com integrated with self-hosted Jira +or cloud Jira. + +By doing this you can easily access related GitLab merge requests, branches, and commits directly from a Jira issue. + +This integration connects all GitLab projects within a top-level group or a personal namespace to projects in the Jira instance. +A top-level GitLab group is one that does not have any parent group itself. All the projects of that top-level group, +as well as projects of the top-level group's subgroups nesting down, are connected. Alternatively, you can specify +a GitLab personal namespace in the Jira configuration, which will then connect the projects in that personal namespace to Jira. + +NOTE: **Note**: +Note this is different from the [existing Jira][existing-jira] project integration, where the mapping +is one GitLab project to the entire Jira instance. + +We recommend that a GitLab group admin +or instance admin (in the case of self-hosted GitLab) set up the integration, +in order to simplify administration. + +TIP: **Tip:** +Create and use a single-purpose "jira" user in GitLab, so that removing +regular users won't impact your integration. + +## Requirements + +### Self-hosted GitLab + +If you are using self-hosted GitLab, make sure your GitLab instance is accessible by Jira. + +- If you are connecting to Jira Cloud, make sure your instance is accessible via the internet. +- If you are using Jira Server, make sure your instance is accessible however your network is set up. + +### GitLab.com + +There are no special requirements if you are using GitLab.com. + +## GitLab Configuration + +1. In GitLab, create a new application in order to allow Jira to connect with your GitLab account + + While logged-in, go to `Settings -> Applications`. (Click your profile avatar at + the top right, choose `Settings`, and then navigate to `Applications` from the left + navigation menu.) Use the form to create a new application. + + Enter a useful name for the `Name` field. + + For the `Redirect URI` field, enter `https://<your-gitlab-instance-domain>/login/oauth/callback`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/login/oauth/callback`. + + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Redirect URI` value should be `https://<your-gitlab-instance-domain>/-/jira/login/oauth/callback`. + +  + - Check `api` in the Scopes section. + +2. Click `Save application`. You will see the generated 'Application Id' and 'Secret' values. + Copy these values that you will use on the Jira configuration side. + +## Jira Configuration + +1. In Jira, from the gear menu at the top right, go to `Applications`. Navigate to `DVCS accounts` + from the left navigation menu. Click `Link GitHub account` to start creating a new integration. + (We are pretending to be GitHub in this integration until there is further platform support from Jira.) + +  + +2. Complete the form + + Select GitHub Enterprise for the `Host` field. + + For the `Team or User Account` field, enter the relative path of a top-level GitLab group that you have access to, + or the relative path of your personal namespace. + +  + + For the `Host URL` field, enter `https://<your-gitlab-instance-domain>/`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/`. + + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Host URL` value should be `https://<your-gitlab-instance-domain>/-/jira` + + For the `Client ID` field, use the `Application ID` value from the previous section. + + For the `Client Secret` field, use the `Secret` value from the previous section. + + Ensure that the rest of the checkboxes are checked. + +3. Click `Add` to complete and create the integration. + + Jira takes up to a few minutes to know about (import behind the scenes) all the commits and branches + for all the projects in the GitLab group you specified in the previous step. These are refreshed + every 60 minutes. + + > **Note:** + > In the future, we plan on implementating real-time integration. If you need + > to refresh the data manually, you can do this from the `Applications -> DVCS + > accounts` screen where you initially set up the integration: + > + >  + +To connect additional GitLab projects from other GitLab top-level groups (or personal namespaces), repeat the above +steps with additional Jira DVCS accounts. + +You may now refer any Jira issue by its ID in branch names, commit messages and merge request names on GitLab's side, +and you will be able to see the linked `branches`, `commits`, and `merge requests` when entering a Jira issue +(inside the Jira issue, merge requests will be called "pull requests"). + + + +Click the links to see your GitLab repository data. + + + + + +## Limitations + +- This integration is currently not supported on GitLab instances under a [relative url][relative-url] (e.g. `http://example.com/gitlab`). + +## Changelog + +### 11.10 + +- [Instance admins can now setup integration for all namespaces](https://gitlab.com/gitlab-org/gitlab-ee/issues/8902) + +### 11.1 + +- [Support GitLab subgroups in Jira development panel](https://gitlab.com/gitlab-org/gitlab-ee/issues/3561) + +[existing-jira]: ../user/project/integrations/jira.md +[jira-development-panel]: https://confluence.atlassian.com/adminjiraserver070/integrating-with-development-tools-776637096.html#Integratingwithdevelopmenttools-Developmentpanelonissues +[eep]: https://about.gitlab.com/pricing/ +[ee-2381]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2381 +[relative-url]: https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md new file mode 100644 index 00000000000..44117755b83 --- /dev/null +++ b/doc/integration/kerberos.md @@ -0,0 +1,315 @@ +# Kerberos integration **[STARTER ONLY]** + +GitLab can integrate with [Kerberos][kerb] as an authentication mechanism. + +## Overview + +[Kerberos][kerb] is a secure method for authenticating a request for a service in a +computer network. Kerberos was developed in the Athena Project at the +[Massachusetts Institute of Technology (MIT)][mit]. The name is taken from Greek +mythology; Kerberos was a three-headed dog who guarded the gates of Hades. + +## Use-cases + +- GitLab can be configured to allow your users to sign with their Kerberos credentials. +- You can use Kerberos to [prevent][why-kerb] anyone from intercepting or eavesdropping on the transmitted password. + +## Configuration + +For GitLab to offer Kerberos token-based authentication, perform the +following prerequisites. You still need to configure your system for +Kerberos usage, such as specifying realms. GitLab will make use of the +system's Kerberos settings. + +### GitLab keytab + +1. Create a Kerberos Service Principal for the HTTP service on your GitLab server. + If your GitLab server is `gitlab.example.com` and your Kerberos realm + `EXAMPLE.COM`, create a Service Principal `HTTP/gitlab.example.com@EXAMPLE.COM` + in your Kerberos database. +1. Create a keytab on the GitLab server for the above Service Principal, e.g. + `/etc/http.keytab`. + +The keytab is a sensitive file and must be readable by the GitLab user. Set +ownership and protect the file appropriately: + +``` +sudo chown git /etc/http.keytab +sudo chmod 0600 /etc/http.keytab +``` + +### Configure GitLab + +**Installations from source** + +>**Note:** +For source installations, make sure the `kerberos` gem group +[has been installed](../install/installation.md#install-gems). + +1. Edit the kerberos section of [gitlab.yml] to enable Kerberos ticket-based + authentication. In most cases, you only need to enable Kerberos and specify + the location of the keytab: + + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ['kerberos'] + + kerberos: + # Allow the HTTP Negotiate authentication method for Git clients + enabled: true + + # Kerberos 5 keytab file. The keytab file must be readable by the GitLab user, + # and should be different from other keytabs in the system. + # (default: use default keytab from Krb5 config) + keytab: /etc/http.keytab + ``` + +1. [Restart GitLab] for the changes to take effect. + +--- + +**Omnibus package installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = ['kerberos'] + + gitlab_rails['kerberos_enabled'] = true + gitlab_rails['kerberos_keytab'] = "/etc/http.keytab" + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +--- + +GitLab will now offer the `negotiate` authentication method for signing in and +HTTP Git access, enabling Git clients that support this authentication protocol +to authenticate with Kerberos tokens. + +## Creating and linking Kerberos accounts + +The Administrative user can navigate to **Admin > Users > Example User > Identities** +and attach a Kerberos account. Existing GitLab users can go to **Profile > Account** +and attach a Kerberos account. If you want to allow users without a GitLab +account to login, you should enable the option `allow_single_sign_on` as +described in the [Configure GitLab](#configure-gitlab) section. Then, the first +time a user signs in with Kerberos credentials, GitLab will create a new GitLab +user associated with the email, which is built from the Kerberos username and +realm. User accounts will be created automatically when authentication was +successful. + +## Linking Kerberos and LDAP accounts together + +If your users log in with Kerberos, but you also have [LDAP integration](../administration/auth/ldap.md) +enabled, then your users will be automatically linked to their LDAP accounts on +first login. For this to work, some prerequisites must be met: + +The Kerberos username must match the LDAP user's UID. You can choose which LDAP +attribute is used as the UID in GitLab's [LDAP configuration](../administration/auth/ldap.md#configuration) +but for Active Directory, this should be `sAMAccountName`. + +The Kerberos realm must match the domain part of the LDAP user's Distinguished +Name. For instance, if the Kerberos realm is `AD.EXAMPLE.COM`, then the LDAP +user's Distinguished Name should end in `dc=ad,dc=example,dc=com`. + +Taken together, these rules mean that linking will only work if your users' +Kerberos usernames are of the form `foo@AD.EXAMPLE.COM` and their +LDAP Distinguished Names look like `sAMAccountName=foo,dc=ad,dc=example,dc=com`. + +## HTTP Git access + +A linked Kerberos account enables you to `git pull` and `git push` using your +Kerberos account, as well as your standard GitLab credentials. + +GitLab users with a linked Kerberos account can also `git pull` and `git push` +using Kerberos tokens, i.e., without having to send their password with each +operation. + +### HTTP Git access with Kerberos token (passwordless authentication) + +#### Support for Git before 2.4 + +Until Git version 2.4, the `git` command uses only the `negotiate` authentication +method if the HTTP server offers it, even if this method fails (such as when +the client does not have a Kerberos token). It is thus not possible to fall back +to username/password (also known as `basic`) authentication if Kerberos +authentication fails. + +For GitLab users to be able to use either `basic` or `negotiate` authentication +with older Git versions, it is possible to offer Kerberos ticket-based +authentication on a different port (e.g. 8443) while the standard port will +keep offering only `basic` authentication. + +**For source installations with HTTPS** + +1. Edit the NGINX configuration file for GitLab + (e.g., `/etc/nginx/sites-available/gitlab-ssl`) and configure NGINX to + listen to port `8443` in addition to the standard HTTPS port: + + ```conf + server { + listen 0.0.0.0:443 ssl; + listen [::]:443 ipv6only=on ssl default_server; + listen 0.0.0.0:8443 ssl; + listen [::]:8443 ipv6only=on ssl; + ``` + +1. Update the Kerberos section of [gitlab.yml]: + + ```yaml + kerberos: + # Dedicated port: Git before 2.4 does not fall back to Basic authentication if Negotiate fails. + # To support both Basic and Negotiate methods with older versions of Git, configure + # nginx to proxy GitLab on an extra port (e.g. 8443) and uncomment the following lines + # to dedicate this port to Kerberos authentication. (default: false) + use_dedicated_port: true + port: 8443 + https: true + ``` + +1. [Restart GitLab] and NGINX for the changes to take effect. + +--- + +**For Omnibus package installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['kerberos_use_dedicated_port'] = true + gitlab_rails['kerberos_port'] = 8443 + gitlab_rails['kerberos_https'] = true + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +--- + +After this change, all Git remote URLs will have to be updated to +`https://gitlab.example.com:8443/mygroup/myproject.git` in order to use +Kerberos ticket-based authentication. + +## Upgrading from password-based to ticket-based Kerberos sign-ins + +Prior to GitLab 8.10 Enterprise Edition, users had to submit their +Kerberos username and password to GitLab when signing in. We will +remove support for password-based Kerberos sign-ins in a future +release, so we recommend that you upgrade to ticket-based sign-ins. + +Depending on your existing GitLab configuration, the 'Sign in with: +Kerberos Spnego' button may already be visible on your GitLab sign-in +page. If not, then add the settings [described above](#configuration). + +Once you have verified that the 'Kerberos Spnego' button works +without entering any passwords, you can proceed to disable +password-based Kerberos sign-ins. To do this you need only need to +remove the OmniAuth provider named `kerberos` from your `gitlab.yml` / +`gitlab.rb` file. + +**For installations from source** + +1. Edit [gitlab.yml] and remove the `- { name: 'kerberos' }` line under omniauth + providers: + + ```yaml + omniauth: + # ... + providers: + - { name: 'kerberos' } # <-- remove this line + ``` + +1. [Restart GitLab] for the changes to take effect. + +--- + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb` and remove the `{ "name" => "kerberos" }` line + under `gitlab_rails['omniauth_providers']`: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { "name" => "kerberos" } # <-- remove this entry + ] + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +## Support for Active Directory Kerberos environments + +When using Kerberos ticket-based authentication in an Active Directory domain, +it may be necessary to increase the maximum header size allowed by NGINX, +as extensions to the Kerberos protocol may result in HTTP authentication headers +larger than the default size of 8kB. Configure `large_client_header_buffers` +to a larger value in [the NGINX configuration][nginx]. + +## Troubleshooting + +### Unsupported GSSAPI mechanism + +With Kerberos SPNEGO authentication, the browser is expected to send a list of +mechanisms it supports to GitLab. If it doesn't support any of the mechanisms +GitLab supports, authentication will fail with a message like this in the log: + +``` +OmniauthKerberosSpnegoController: failed to process Negotiate/Kerberos authentication: gss_accept_sec_context did not return GSS_S_COMPLETE: An unsupported mechanism was requested Unknown error +``` + +This is usually seen when the browser is unable to contact the Kerberos server +directly. It will fall back to an unsupported mechanism known as +[`IAKERB`](https://k5wiki.kerberos.org/wiki/Projects/IAKERB), which tries to use +the GitLab server as an intermediary to the Kerberos server. + +If you're experiencing this error, ensure there is connectivity between the +client machine and the Kerberos server - this is a prerequisite! Traffic may be +blocked by a firewall, or the DNS records may be incorrect. + +Another failure mode occurs when the forward and reverse DNS records for the +GitLab server do not match. Often, Windows clients will work in this case, while +Linux clients will fail. They use reverse DNS while detecting the Kerberos +realm. If they get the wrong realm, then ordinary Kerberos mechanisms will fail, +so the client will fall back to attempting to negotiate `IAKERB`, leading to the +above error message. + +To fix this, ensure that the forward and reverse DNS for your GitLab server +match. So for instance, if you acces GitLab as `gitlab.example.com`, resolving +to IP address `1.2.3.4`, then `4.3.2.1.in-addr.arpa` must be a PTR record for +`gitlab.example.com`. + +Finally, it's possible that the browser or client machine lack Kerberos support +completely. Ensure that the Kerberos libraries are installed and that you can +authenticate to other Kerberos services. + +### HTTP Basic: Access denied when cloning + +```sh +remote: HTTP Basic: Access denied +fatal: Authentication failed for '<KRB5 path>' +``` + +If you are using Git v2.11 or newer and see the above error when cloning, you can +set the `http.emptyAuth` Git option to `true` to fix this: + +``` +git config --global http.emptyAuth true +``` + +See also: [Git v2.11 release notes](https://github.com/git/git/blob/master/Documentation/RelNotes/2.11.0.txt#L482-L486) + +## Helpful links + +- <https://help.ubuntu.com/community/Kerberos> +- <http://blog.manula.org/2012/04/setting-up-kerberos-server-with-debian.html> +- <http://www.roguelynn.com/words/explain-like-im-5-kerberos/> + +[gitlab.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example +[restart gitlab]: ../administration/restart_gitlab.md#installations-from-source +[reconfigure gitlab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure +[nginx]: http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers +[kerb]: https://web.mit.edu/kerberos/ +[mit]: http://web.mit.edu/ +[why-kerb]: http://web.mit.edu/sipb/doc/working/guide/guide/node20.html +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 7fd39b02fbe..ef1f2df77f8 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -33,6 +33,7 @@ contains some settings that are common for all providers. - [Authentiq](../administration/auth/authentiq.md) - [OAuth2Generic](oauth2_generic.md) - [JWT](../administration/auth/jwt.md) +- [OpenID Connect](../administration/auth/oidc.md) - [UltraAuth](ultra_auth.md) ## Initial OmniAuth Configuration diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 8ee07a7fcdc..15d9d8c9c74 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -1,5 +1,7 @@ # SAML OmniAuth Provider +> This topic is for SAML on self-managed GitLab instances. For SAML on GitLab.com, see [SAML SSO for GitLab.com Groups](https://docs.gitlab.com/ee/user/group/saml_sso/index.html). + NOTE: **Note:** You need to [enable OmniAuth](omniauth.md) in order to use this. @@ -10,115 +12,115 @@ Microsoft ADFS to authenticate users. First configure SAML 2.0 support in GitLab, then register the GitLab application in your SAML IdP: -1. Make sure GitLab is configured with HTTPS. - See [Using HTTPS](../install/installation.md#using-https) for instructions. - -1. On your GitLab server, open the configuration file. +1. Make sure GitLab is configured with HTTPS. + See [Using HTTPS](../install/installation.md#using-https) for instructions. + +1. On your GitLab server, open the configuration file. - For omnibus package: + For omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` -1. To allow your users to use SAML to sign up without having to manually create - an account first, don't forget to add the following values to your configuration: +1. To allow your users to use SAML to sign up without having to manually create + an account first, don't forget to add the following values to your configuration: - For omnibus package: + For omnibus package: - ```ruby - gitlab_rails['omniauth_enabled'] = true - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] - gitlab_rails['omniauth_block_auto_created_users'] = false - ``` + ```ruby + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] + gitlab_rails['omniauth_block_auto_created_users'] = false + ``` - For installations from source: + For installations from source: - ```yaml - omniauth: - enabled: true - allow_single_sign_on: ["saml"] - block_auto_created_users: false - ``` + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ["saml"] + block_auto_created_users: false + ``` -1. You can also automatically link SAML users with existing GitLab users if their - email addresses match by adding the following setting: +1. You can also automatically link SAML users with existing GitLab users if their + email addresses match by adding the following setting: - For omnibus package: + For omnibus package: - ```ruby - gitlab_rails['omniauth_auto_link_saml_user'] = true - ``` + ```ruby + gitlab_rails['omniauth_auto_link_saml_user'] = true + ``` - For installations from source: + For installations from source: - ```yaml - auto_link_saml_user: true - ``` + ```yaml + auto_link_saml_user: true + ``` -1. Add the provider configuration: +1. Add the provider configuration: - For omnibus package: + For omnibus package: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` - For installations from source: - - ```yaml - omniauth: - providers: - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ``` + For installations from source: -1. Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint - of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab - installation to generate the correct value). + ```yaml + omniauth: + providers: + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ``` -1. Change the values of `idp_cert_fingerprint`, `idp_sso_target_url`, - `name_identifier_format` to match your IdP. If a fingerprint is used it must - be a SHA1 fingerprint; check - [the omniauth-saml documentation](https://github.com/omniauth/omniauth-saml) - for more details on these options. +1. Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint + of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab + installation to generate the correct value). -1. Change the value of `issuer` to a unique name, which will identify the application - to the IdP. +1. Change the values of `idp_cert_fingerprint`, `idp_sso_target_url`, + `name_identifier_format` to match your IdP. If a fingerprint is used it must + be a SHA1 fingerprint; check + [the omniauth-saml documentation](https://github.com/omniauth/omniauth-saml) + for more details on these options. -1. For the changes to take effect, you must [reconfigure][] GitLab if you installed via Omnibus or [restart GitLab][] if you installed from source. +1. Change the value of `issuer` to a unique name, which will identify the application + to the IdP. -1. Register the GitLab SP in your SAML 2.0 IdP, using the application name specified - in `issuer`. +1. For the changes to take effect, you must [reconfigure][] GitLab if you installed via Omnibus or [restart GitLab][] if you installed from source. + +1. Register the GitLab SP in your SAML 2.0 IdP, using the application name specified + in `issuer`. To ease configuration, most IdP accept a metadata URL for the application to provide configuration information to the IdP. To build the metadata URL for GitLab, append @@ -185,6 +187,78 @@ tell GitLab which groups are external via the `external_groups:` element: } } ``` +## Required groups + +>**Note:** +This setting is only available on GitLab 10.2 EE and above. + +This setting works like `External Groups` setting. Just like there, your IdP has to +pass Group Information to GitLab, you have to tell GitLab where to look or the +groups SAML response, and which group membership should be requisite for logging in. +When `required_groups` is not set or it is empty, anyone with proper authentication +will be able to use the service. + +Example: + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + required_groups: ['Developers', 'Managers', 'Admins'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + +## Admin Groups + +>**Note:** +This setting is only available on GitLab 8.8 EE and above. + +This setting works very similarly to the `External Groups` setting. The requirements +are the same, your IdP needs to pass Group information to GitLab, you need to tell +GitLab where to look for the groups in the SAML response, and which group should be +considered `admin groups`. + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + admin_groups: ['Managers', 'Admins'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + +## Auditor Groups + +>**Note:** +This setting is only available on GitLab 11.4 EE and above. + +This setting also follows the requirements documented for the `External Groups` setting. GitLab uses the Group information provided by your IdP to determine if a user should be assigned the `auditor` role. + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + auditor_groups: ['Auditors', 'Security'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + ## Bypass two factor authentication If you want some SAML authentication methods to count as 2FA on a per session basis, you can register them in the @@ -194,28 +268,28 @@ If you want some SAML authentication methods to count as 2FA on a per session ba 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', - upstream_two_factor_authn_contexts: - %w( - urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport - urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS - urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN - ) - - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', + upstream_two_factor_authn_contexts: + %w( + urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport + urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS + urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN + ) + + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` 1. Save the file and [reconfigure][] GitLab for the changes to take effect. @@ -225,27 +299,27 @@ If you want some SAML authentication methods to count as 2FA on a per session ba 1. Edit `config/gitlab.yml`: - ```yaml - omniauth: - providers: - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', - upstream_two_factor_authn_contexts: - [ - 'urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport', - 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS', - 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN' - ] - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ``` + ```yaml + omniauth: + providers: + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', + upstream_two_factor_authn_contexts: + [ + 'urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport', + 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS', + 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN' + ] + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ``` 1. Save the file and [restart GitLab][] for the changes ot take effect diff --git a/doc/legal/corporate_contributor_license_agreement.md b/doc/legal/corporate_contributor_license_agreement.md index e5fc7a3c85f..7f08188bd65 100644 --- a/doc/legal/corporate_contributor_license_agreement.md +++ b/doc/legal/corporate_contributor_license_agreement.md @@ -1,3 +1,29 @@ ---- -redirect_to: 'README.md' ---- +# Corporate contributor license agreement + +You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. + +1. Definitions. + + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + + "Contribution" shall mean the code, documentation or other original works of authorship, including any modifications or additions to an existing work, that is submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + +2. Grant of Copyright License. + +Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + +3. Grant of Patent License. + +Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + +4. You represent that You are legally entitled to grant the above license. You represent further that each of Your employees is authorized to submit Contributions on Your behalf, but excluding employees that are designated in writing by You as "Not authorized to submit Contributions on behalf of [name of Your corporation here]." Such designations of exclusion for unauthorized employees are to be submitted via email to legal@gitlab.com. + +5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). + +6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]". + +8. It is Your responsibility to notify GitLab.com when any change is required to the list of designated employees excluded from submitting Contributions on Your behalf per Section 4. Such notification should be sent via email to legal@gitlab.com. + +This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/legal/individual_contributor_license_agreement.md b/doc/legal/individual_contributor_license_agreement.md index e5fc7a3c85f..59803aea080 100644 --- a/doc/legal/individual_contributor_license_agreement.md +++ b/doc/legal/individual_contributor_license_agreement.md @@ -1,3 +1,25 @@ ---- -redirect_to: 'README.md' ---- +# Individual contributor license agreement + +You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. + +1. Definitions. + + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + + "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + +2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + +3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + +4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to GitLab B.V., or that your employer has executed a separate Corporate CLA with GitLab B.V.. + +5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions. + +6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [insert_name_here]". + +8. You agree to notify GitLab B.V. of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect. + +This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/license/README.md b/doc/license/README.md new file mode 100644 index 00000000000..4cc387ba95f --- /dev/null +++ b/doc/license/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/admin_area/license.html' +--- + +This document was moved to [user/admin_area/license](https://docs.gitlab.com/ee/user/admin_area/license.html). diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md new file mode 100644 index 00000000000..7654023f266 --- /dev/null +++ b/doc/push_rules/push_rules.md @@ -0,0 +1,156 @@ +# Push Rules **[STARTER]** + +Gain additional control over pushes to your repository. + +## Overview + +GitLab already offers [protected branches][protected-branches], but there are +cases when you need some specific rules like preventing git tag removal or +enforcing a special format for commit messages. + +Push rules are essentially [pre-receive Git hooks][hooks] that are easy to +enable in a user-friendly interface. They are defined globally if you are an +admin or per project so you can have different rules applied to different +projects depending on your needs. + +## Use cases + +Every push rule could have its own use case, but let's consider some examples. + +### Commit messages with a specific reference + +Let's assume you have the following requirements for your workflow: + +- every commit should reference a JIRA issue, for example: `Refactored css. Fixes JIRA-123.` +- users should not be able to remove git tags with `git push` + +All you need to do is write a simple regular expression that requires the mention +of a JIRA issue in the commit message, like `JIRA\-\d+`. + +Now when a user tries to push a commit with a message `Bugfix`, their push will +be declined. Only pushing commits with messages like `Bugfix according to JIRA-123` +will be accepted. + +### Restrict branch names + +Let's assume there's a strictly policy for branch names in your company, and +you want the branches to start with a certain name because you have different +GitLab CI jobs (`feature`, `hotfix`, `docker`, `android`, etc.) that rely on the +branch name. + +Your developers however, don't always remember that policy, so they push +various branches and CI pipelines do not work as expected. By restricting the +branch names globally in Push Rules, you can now sleep without the anxiety +of your developers' mistakes. Every branch that doesn't match your push rule +will get rejected. + +## Enabling push rules + +>**Note:** +GitLab administrators can set push rules globally under +**Admin area > Push Rules** that all new projects will inherit. You can later +override them in a project's settings. + +1. Navigate to your project's **Settings > Repository** and expand **Push Rules** +1. Set the rule you want +1. Click **Save Push Rules** for the changes to take effect + +The following options are available. + +| Push rule | GitLab version | Description | +| --------- | :------------: | ----------- | +| Removal of tags with `git push` | **Starter** 7.10 | Forbid users to remove git tags with `git push`. Tags will still be able to be deleted through the web UI. | +| Check whether author is a GitLab user | **Starter** 7.10 | Restrict commits by author (email) to existing GitLab users. | +| Check whether committer is the current authenticated user | **Premium** 10.2 | GitLab will reject any commit that was not committed by the current authenticated user | +| Check whether commit is signed through GPG | **Premium** 10.1 | Reject commit when it is not signed through GPG. Read [signing commits with GPG][signing-commits]. | +| Prevent committing secrets to Git | **Starter** 8.12 | GitLab will reject any files that are likely to contain secrets. Read [what files are forbidden](#prevent-pushing-secrets-to-the-repository). | +| Restrict by commit message | **Starter** 7.10 | Only commit messages that match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by commit message (negative match)| **Starter** 11.1 | Only commit messages that do not match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by branch name | **Starter** 9.3 | Only branch names that match this regular expression are allowed to be pushed. Leave empty to allow any branch name. | +| Restrict by commit author's email | **Starter** 7.10 | Only commit author's email that match this regular expression are allowed to be pushed. Leave empty to allow any email. | +| Prohibited file names | **Starter** 7.10 | Any committed filenames that match this regular expression are not allowed to be pushed. Leave empty to allow any filenames. | +| Maximum file size | **Starter** 7.12 | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. | + +>**Tip:** +GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules. You can check your regular expressions at <https://regex-golang.appspot.com>. + +## Prevent pushing secrets to the repository + +> [Introduced][ee-385] in [GitLab Starter][ee] 8.12. + +You can turn on a predefined blacklist of files which won't be allowed to be +pushed to a repository. + +By selecting the checkbox *Prevent committing secrets to Git*, GitLab prevents +pushes to the repository when a file matches a regular expression as read from +[`files_blacklist.yml`][list] (make sure you are at the right branch +as your GitLab version when viewing this file). + +NOTE: **Note**: +Files already committed won't get restricted by this push rule. + +Below is an example list of what will be rejected by these regular expressions: + +```shell +##################### +# AWS CLI credential blobs +##################### +.aws/credentials +aws/credentials +homefolder/aws/credentials + +##################### +# Private RSA SSH keys +##################### +/ssh/id_rsa +/.ssh/personal_rsa +/config/server_rsa +id_rsa +.id_rsa + +##################### +# Private DSA SSH keys +##################### +/ssh/id_dsa +/.ssh/personal_dsa +/config/server_dsa +id_dsa +.id_dsa + +##################### +# Private ed25519 SSH keys +##################### +/ssh/id_ed25519 +/.ssh/personal_ed25519 +/config/server_ed25519 +id_ed25519 +.id_ed25519 + +##################### +# Private ECDSA SSH keys +##################### +/ssh/id_ecdsa +/.ssh/personal_ecdsa +/config/server_ecdsa +id_ecdsa +.id_ecdsa + +##################### +# Any file with .pem or .key extensions +##################### +*.pem +*.key + +##################### +# Any file ending with _history or .history extension +##################### +pry.history +bash_history +``` + +[protected-branches]: ../user/project/protected_branches.md +[signing-commits]: ../user/project/repository/gpg_signed_commits/index.md +[ee-385]: https://gitlab.com/gitlab-org/gitlab-ee/issues/385 +[list]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/checks/files_blacklist.yml +[hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index e70a009323e..f5c788af578 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -23,6 +23,16 @@ sudo gitlab-rake gitlab:cleanup:repos bundle exec rake gitlab:cleanup:repos RAILS_ENV=production ``` +Remove old repository copies from repositories moved to another storage. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:cleanup:moved + +# installation from source +bundle exec rake gitlab:cleanup:moved RAILS_ENV=production +``` + Clean up local project upload files if they don't exist in GitLab database. The task attempts to fix the file if it can find its project, otherwise it moves the file to a lost and found directory. diff --git a/doc/subscriptions/billing_table.png b/doc/subscriptions/billing_table.png Binary files differnew file mode 100644 index 00000000000..acd1b6193ec --- /dev/null +++ b/doc/subscriptions/billing_table.png diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md new file mode 100644 index 00000000000..b6d16536fee --- /dev/null +++ b/doc/subscriptions/index.md @@ -0,0 +1,103 @@ +# Subscription setup and management + +This page will help get you started with your new subscription or manage an existing one, whether you have subscribed to GitLab.com or self-managed GitLab. + +To subscribe, upgrade, or read more about the types of subscriptions, please see [Subscribe to GitLab](../README.md#subscribe-to-gitlab) on the GitLab Documentation landing page. + +## Set up GitLab + +Learn how GitLab helps you in the stages of the DevOps lifecycle by learning more [about the GitLab product](https://about.gitlab.com/product/), [GitLab features](https://about.gitlab.com/features/), and [GitLab Documentation](../README.md). + +### Self-managed: Install GitLab + +Take a look at [installing GitLab](https://about.gitlab.com/install/) and our [administrator documentation](../administration/index.md). Then, follow the instructions below under [Your subscription](#your-subscription) to apply your license file. + +### GitLab.com: Create a user and group + +Start with creating a user account for yourself using our [sign up page](https://gitlab.com/users/sign_in#register-pane). + +[GitLab groups](../user/group/index.md) help assemble related projects together allowing you to grant members access to several projects at once. A group is not required if you plan on having [projects](../user/project/) inside a personal namespace. + +## Your subscription + +You can view and manage subscriptions through our [Customers portal](https://customers.gitlab.com/). Information on applying your subscription is below. + +Please also see our [subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/) + +### View subscription and seats + +To view and manage the subscriptions you have purchased and the number of seats associated with the subscription, please visit and log into the [Customers’ Portal](https://customers.gitlab.com/subscriptions). For more information, please see our [subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/) and [pricing page](https://about.gitlab.com/pricing/), which includes information on our [true-up pricing policy](https://about.gitlab.com/handbook/product/pricing/#true-up-pricing) when adding more users than at the time of purchase. + +Please note that this account may have the same email, but is a _separate_ login from your GitLab.com account. If the two accounts are linked together, then you can use the "sign in with GitLab.com account" link underneath the `Sign In` button. + +### Change billing information + +In the customers portal, go to the `My Account` page, then revise the `Account Details` information and click on the `Update Account` button. + +Future purchases will use the information in this section. The email listed in this section is used for the Customers Portal login and for license related email communication. + +### Self-managed: Apply your license file + +After purchase, the license file is sent to the email address tied to the Customers portal account, which needs to be [uploaded to the GitLab instance](https://docs.gitlab.com/ee/user/admin_area/license.html#uploading-your-license). + +### Link your GitLab.com account with your Customers Portal account + +NOTE: **Note:** This is *required* for GitLab.com subscriptions. + +Once signed into the customers portal, if your account is not already linked, you should be prompted to link your account with a "Link my GitLab Account" button. + +You can also go to the [My Account](https://customers.gitlab.com/customers/edit) page to add or change the GitLab.com account link. + +### Change the linked GitLab.com account for your Customers Portal account + +To change which GitLab.com account is associated with a Customers Portal account, please follow these steps: + +1. Log into the [Customers Portal](https://customers.gitlab.com/customers/sign_in). +1. In a separate browser tab, visit [GitLab.com](https://gitlab.com) to ensure you are not logged in, or if you are, log out. +1. Back on the Customers Portal page, click [My Account](https://customers.gitlab.com/customers/edit) in the top menu. +1. Under `Your GitLab.com account`, click the `Change linked account` button. +1. Have the user you want associated log in to their [GitLab.com](https://gitlab.com) account. + +### GitLab.com: Associate your namespace with your subscription + +Once your GitLab.com account is linked, you can go to your [Subscriptions](https://customers.gitlab.com/subscriptions) page to choose or change the namespace your subscription applies to. + +Please note that you need to be a group owner to associate a group to your subscription. + +### Confirm or upgrade your GitLab.com subscription details within GitLab + +To see the status of your GitLab.com subscription, you can click on the Billings +section of the relevant namespace: + +* For individuals, this is located at https://gitlab.com/profile/billings under +in your Settings, +* For groups, this is located under the group's Settings dropdown, under Billing. + +For groups, you can see details of your subscription - including your current +plan - in the included table: + + + +| Field | Description | +| ------ | ------ | +| Seats in subscription | If this is a paid plan, this represents the number of seats you've paid to support in your group. | +| Seats currently in use | The number of active seats currently in use. | +| Max seats used | The highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | +| Seats owed | If your max seats used exceeds the seats in your subscription, you'll owe an additional fee for the users you've added. | +| Subscription start date | The date your subscription started. If this is for a Free plan, this is the date you transitioned off your group's paid plan. | +| Subscription end date | The date your current subscription will end. This does not apply to Free plans. | + +## Need help? + +[GitLab's Documentation](https://docs.gitlab.com/) offers a wide range of topics covering the use and administration of GitLab. + +We also encourage all users to search our project trackers for known issues and existing feature requests in: + +- [GitLab CE](https://gitlab.com/gitlab-org/gitlab-ce/issues/) for features included in all tiers, and +- [GitLab EE](https://gitlab.com/gitlab-org/gitlab-ee/issues/) for paid-tier features. + +These issues are the best avenue for getting updates on specific product plans and for communicating directly with the relevant GitLab team members. + +### Contacting Support + +Learn more about the tiers of [GitLab Support](https://about.gitlab.com/support/) or [submit a request via the Support Portal](https://support.gitlab.com/hc/en-us/requests/new). diff --git a/doc/tools/email.md b/doc/tools/email.md new file mode 100644 index 00000000000..ab39206ffa4 --- /dev/null +++ b/doc/tools/email.md @@ -0,0 +1,38 @@ +# Email from GitLab **[STARTER ONLY]** + +As a GitLab administrator you can email GitLab users from within GitLab. + +## Overview + +GitLab provides a simple tool to email all users or users of a chosen group or +project right from the admin area. Users will receive the email to their primary +email address. + +## Use-cases + +- Notify your users about a new project, a new feature, or a new product launch. +- Notify your users about a new deployment, or that will be downtime expected + for a particular reason. + +## Sending emails to users from within GitLab + +1. Go to the admin area using the wrench icon in the top right corner and + navigate to **Overview > Users > Send email to users**. + +  + +1. Compose an email and choose where it will be sent (all users or users of a + chosen group or project): + +  + +## Unsubscribing from emails + +User can choose to unsubscribe from receiving emails from GitLab by following +the unsubscribe link from the email. Unsubscribing is unauthenticated in order +to keep the simplicity of this feature. + +On unsubscribe, user will receive an email notifying that unsubscribe happened. +The endpoint that provides the unsubscribe option is rate-limited. + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/tools/email1.png b/doc/tools/email1.png Binary files differnew file mode 100644 index 00000000000..e79ccc3e9a9 --- /dev/null +++ b/doc/tools/email1.png diff --git a/doc/tools/email2.png b/doc/tools/email2.png Binary files differnew file mode 100644 index 00000000000..d073c0e42da --- /dev/null +++ b/doc/tools/email2.png diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 0fc7f57feab..0531627b7b3 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -30,6 +30,8 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Atlassian Crowd OmniAuth Provider](../../administration/auth/crowd.md) - [CAS OmniAuth Provider](../../integration/cas.md) - [SAML OmniAuth Provider](../../integration/saml.md) + - [SAML for GitLab.com Groups](https://docs.gitlab.com/ee/user/group/saml_sso/index.html) + - [SCIM user provisioning for GitLab.com Groups](https://docs.gitlab.com/ee/user/group/saml_sso/scim_setup.html) - [Okta SSO provider](../../administration/auth/okta.md) - [Kerberos integration (GitLab EE)](https://docs.gitlab.com/ee/integration/kerberos.html) diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 69834f7ae47..2884458a44c 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -179,7 +179,7 @@ Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so except for the environment scope, they would also need to have a different domain they would be deployed to. This is why you need to define a separate `KUBE_INGRESS_BASE_DOMAIN` variable for all the above -[based on the environment](https://docs.gitlab.com/ee/ci/variables/index.html#limiting-environment-scopes-of-variables-premium). +[based on the environment](https://docs.gitlab.com/ee/ci/variables/#limiting-environment-scopes-of-environment-variables-premium). The following table is an example of how the three different clusters would be configured. @@ -377,8 +377,8 @@ analysis on the current code and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/sast.html). +Any security warnings are also shown in the merge request widget. Read more how +[SAST works](https://docs.gitlab.com/ee/user/application_security/sast/index.html). NOTE: **Note:** The Auto SAST stage will be skipped on licenses other than Ultimate. @@ -396,8 +396,8 @@ to run analysis on the project dependencies and checks for potential security is report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dependency_scanning.html). +Any security warnings are also shown in the merge request widget. Read more about +[Dependency Scanning](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). NOTE: **Note:** The Auto Dependency Scanning stage will be skipped on licenses other than Ultimate. @@ -415,8 +415,8 @@ to search the project dependencies for their license. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any licenses are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/license_management.html). +Any licenses are also shown in the merge request widget. Read more how +[License Management works](https://docs.gitlab.com/ee/user/application_security/license_management/index.html). NOTE: **Note:** The Auto License Management stage will be skipped on licenses other than Ultimate. @@ -431,8 +431,8 @@ Docker image and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/container_scanning.html). +Any security warnings are also shown in the merge request widget. Read more how +[Container Scanning works](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html). NOTE: **Note:** The Auto Container Scanning stage will be skipped on licenses other than Ultimate. @@ -487,8 +487,8 @@ to perform an analysis on the current code and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dast.html). +Any security warnings are also shown in the merge request widget. Read how +[DAST works](https://docs.gitlab.com/ee/user/application_security/dast/index.html). NOTE: **Note:** The Auto DAST stage will be skipped on licenses other than Ultimate. diff --git a/doc/update/10.0-ce-to-ee.md b/doc/update/10.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.1-ce-to-ee.md b/doc/update/10.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.2-ce-to-ee.md b/doc/update/10.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.3-ce-to-ee.md b/doc/update/10.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.4-ce-to-ee.md b/doc/update/10.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.5-ce-to-ee.md b/doc/update/10.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.6-ce-to-ee.md b/doc/update/10.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.7-ce-to-ee.md b/doc/update/10.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.8-ce-to-ee.md b/doc/update/10.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.0-ce-to-ee.md b/doc/update/11.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.1-ce-to-ee.md b/doc/update/11.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.2-ce-to-ee.md b/doc/update/11.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.3-ce-to-ee.md b/doc/update/11.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.4-ce-to-ee.md b/doc/update/11.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.5-ce-to-ee.md b/doc/update/11.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.6-ce-to-ee.md b/doc/update/11.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.7-ce-to-ee.md b/doc/update/11.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.8-ce-to-ee.md b/doc/update/11.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.0-ce-to-ee.md b/doc/update/6.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.1-ce-to-ee.md b/doc/update/6.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.2-ce-to-ee.md b/doc/update/6.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.3-ce-to-ee.md b/doc/update/6.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.4-ce-to-ee.md b/doc/update/6.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.5-ce-to-ee.md b/doc/update/6.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.6-ce-to-ee.md b/doc/update/6.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.7-ce-to-ee.md b/doc/update/6.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.8-ce-to-ee.md b/doc/update/6.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.9-ce-to-ee.md b/doc/update/6.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.0-ce-to-ee.md b/doc/update/7.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.1-ce-to-ee.md b/doc/update/7.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.10-ce-to-ee.md b/doc/update/7.10-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.10-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.11-ce-to-ee.md b/doc/update/7.11-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.11-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.12-ce-to-ee.md b/doc/update/7.12-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.12-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.13-ce-to-ee.md b/doc/update/7.13-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.13-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.14-ce-to-ee.md b/doc/update/7.14-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.14-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.3-ce-to-ee.md b/doc/update/7.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.4-ce-to-ee.md b/doc/update/7.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.5-ce-to-ee.md b/doc/update/7.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.6-ce-to-ee.md b/doc/update/7.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.7-ce-to-ee.md b/doc/update/7.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.8-ce-to-ee.md b/doc/update/7.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.9-ce-to-ee.md b/doc/update/7.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.0-ce-to-ee.md b/doc/update/8.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.1-ce-to-ee.md b/doc/update/8.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.10-ce-to-ee.md b/doc/update/8.10-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.10-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.11-ce-to-ee.md b/doc/update/8.11-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.11-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.12-ce-to-ee.md b/doc/update/8.12-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.12-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.13-ce-to-ee.md b/doc/update/8.13-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.13-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.14-ce-to-ee.md b/doc/update/8.14-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.14-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.15-ce-to-ee.md b/doc/update/8.15-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.15-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.16-ce-to-ee.md b/doc/update/8.16-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.16-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.17-ce-to-ee.md b/doc/update/8.17-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.17-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.2-ce-to-ee.md b/doc/update/8.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.3-ce-to-ee.md b/doc/update/8.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.4-ce-to-ee.md b/doc/update/8.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.5-ce-to-ee.md b/doc/update/8.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.6-ce-to-ee.md b/doc/update/8.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.7-ce-to-ee.md b/doc/update/8.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.8-ce-to-ee.md b/doc/update/8.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.9-ce-to-ee.md b/doc/update/8.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.0-ce-to-ee.md b/doc/update/9.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.1-ce-to-ee.md b/doc/update/9.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.2-ce-to-ee.md b/doc/update/9.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.3-ce-to-ee.md b/doc/update/9.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.4-ce-to-ee.md b/doc/update/9.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.5-ce-to-ee.md b/doc/update/9.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 2e8380aa5d8..f90bff662e2 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -106,14 +106,20 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) sudo -u git -H make ``` -### 8. Start application +### 8. Install/Update `gitlab-elasticsearch-indexer` (optional) **[STARTER ONLY]** + +If you're interested in using GitLab's new [elasticsearch repository indexer](https://docs.gitlab.com/ee/integration/elasticsearch.html#elasticsearch-repository-indexer-beta) (currently in beta) +please follow the instructions on the document linked above and enable the +indexer usage in the GitLab admin settings. + +### 9. Start application ```bash sudo service gitlab start sudo service nginx restart ``` -### 9. Check application status +### 10. Check application status Check if GitLab and its environment are configured correctly: diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index 0d1ecab5f8e..e74a5c00f7e 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -9,6 +9,10 @@ Community Edition to Enterprise Edition. These documents can be found in the [`doc/update` directory of Enterprise Edition's source code][old-ee-upgrade-docs]. +If you want to upgrade the version only, for example 11.8 to 11.9, *without* changing the +GitLab edition you are using (Community or Enterprise), see the +[Upgrading from source](upgrading_from_source.md) documentation. + ## General upgrading steps This guide assumes you have a correctly configured and tested installation of diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index b0c9aaa1eed..fea89669831 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -16,6 +16,9 @@ If the highest number stable branch is unclear please check the [GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation guide links by version. +If you are changing from GitLab Community Edition to GitLab Enterprise Edition, see +the [Upgrading from CE to EE](upgrading_from_ce_to_ee.md) documentation. + ## Guidelines for all versions This section contains all the steps necessary to upgrade Community Edition or diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md new file mode 100644 index 00000000000..776ab139c64 --- /dev/null +++ b/doc/user/admin_area/geo_nodes.md @@ -0,0 +1,70 @@ +# Geo nodes admin area **[PREMIUM ONLY]** + +For more information about setting up GitLab Geo, read the +[Geo documentation](https://docs.gitlab.com/ee/administration/geo/replication/index.html). + +When you're done, you can navigate to **Admin area > Geo** (`/admin/geo/nodes`). + +## Common settings + +All Geo nodes have the following settings: + +| Setting | Description | +| --------| ----------- | +| Primary | This marks a Geo Node as **primary** node. There can be only one **primary** node; make sure that you first add the **primary** node and then all the others. | +| Name | The unique identifier for the Geo node. Must match the setting `gitlab_rails[geo_node_name]` in `/etc/gitlab/gitlab.rb`. The setting defaults to `external_url` with a trailing slash. | +| URL | The instance's user-facing URL. | + +The node you're reading from is indicated with a green `Current node` label, and +the **primary** node is given a blue `Primary` label. Remember that you can only make +changes on the **primary** node! + +## **Secondary** node settings + +**Secondary** nodes have a number of additional settings available: + +| Setting | Description | +|---------------------------|-------------| +| Selective synchronization | Enable Geo [selective sync](https://docs.gitlab.com/ee/administration/geo/replication/configuration.html#selective-synchronization) for this **secondary** node. | +| Repository sync capacity | Number of concurrent requests this **secondary** node will make to the **primary** node when backfilling repositories. | +| File sync capacity | Number of concurrent requests this **secondary** node will make to the **primary** node when backfilling files. | + +## Geo backfill + +**Secondary** nodes are notified of changes to repositories and files by the **primary** node, +and will always attempt to synchronize those changes as quickly as possible. + +Backfill is the act of populating the **secondary** node with repositories and files that +existed *before* the **secondary** node was added to the database. Since there may be +extremely large numbers of repositories and files, it's infeasible to attempt to +download them all at once, so GitLab places an upper limit on the concurrency of +these operations. + +How long the backfill takes is a function of the maximum concurrency, but higher +values place more strain on the **primary** node. From [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3107), +the limits are configurable. If your **primary** node has lots of surplus capacity, +you can increase the values to complete backfill in a shorter time. If it's +under heavy load and backfill is reducing its availability for normal requests, +you can decrease them. + +## Using a different URL for synchronization + +The **primary** node's Internal URL is used by **secondary** nodes to contact it +(to sync repositories, for example). The name Internal URL distinguishes it from +[External URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab) +which is used by users. Internal URL does not need to be a private address. + +Internal URL defaults to External URL, but you can customize it under +**Admin area > Geo Nodes**. + +## Multiple secondary nodes behind a load balancer + +In GitLab 11.11, **secondary** nodes can use identical external URLs as long as +a unique `name` is set for each Geo node. The `gitlab.rb` setting +`gitlab_rails[geo_node_name]` must: + +- Be set for each GitLab instance that runs `unicorn`, `sidekiq`, or `geo_logcursor`. +- Match a Geo node name. + +The load balancer must use sticky sessions in order to avoid authentication +failures and cross site request errors. diff --git a/doc/user/admin_area/img/admin_wrench.png b/doc/user/admin_area/img/admin_wrench.png Binary files differnew file mode 100644 index 00000000000..17eee143e87 --- /dev/null +++ b/doc/user/admin_area/img/admin_wrench.png diff --git a/doc/user/admin_area/img/license_admin_area.png b/doc/user/admin_area/img/license_admin_area.png Binary files differnew file mode 100644 index 00000000000..b5662b81c5e --- /dev/null +++ b/doc/user/admin_area/img/license_admin_area.png diff --git a/doc/user/admin_area/img/license_details.png b/doc/user/admin_area/img/license_details.png Binary files differnew file mode 100644 index 00000000000..2085bb437ad --- /dev/null +++ b/doc/user/admin_area/img/license_details.png diff --git a/doc/user/admin_area/img/license_no_license_message.png b/doc/user/admin_area/img/license_no_license_message.png Binary files differnew file mode 100644 index 00000000000..87b397f7905 --- /dev/null +++ b/doc/user/admin_area/img/license_no_license_message.png diff --git a/doc/user/admin_area/img/license_upload.png b/doc/user/admin_area/img/license_upload.png Binary files differnew file mode 100644 index 00000000000..29d55175a2d --- /dev/null +++ b/doc/user/admin_area/img/license_upload.png diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index f924ff8dfde..dd4e96c1f4a 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -14,19 +14,22 @@ Only admin users can access the Admin Area. The Admin Area is made up of the following sections: -| Section | Description | -|:------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| -| Overview | View your GitLab [Dashboard](#admin-dashboard), and administer [projects](#administer-projects), users, groups, jobs, runners, and Gitaly servers. | -| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | -| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | -| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | -| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | -| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | -| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | -| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | -| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | -| Appearance | Customize [GitLab's appearance](../../customization/index.md). | -| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | +| Section | Description | +|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| +| Overview | View your GitLab [Dashboard](#admin-dashboard), and administer [projects](#administer-projects), users, groups, jobs, runners, and Gitaly servers. | +| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | +| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | +| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | +| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | +| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | +| License **[STARTER ONLY]** | Upload, display, and remove [licenses](license.md). | +| Push Rules **[STARTER]** | Configure pre-defined git [push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html) for projects. | +| Geo **[PREMIUM ONLY]** | Configure and maintain [Geo nodes](geo_nodes.md). | +| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | +| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | +| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | +| Appearance | Customize [GitLab's appearance](../../customization/index.md). | +| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | ## Admin Dashboard @@ -70,4 +73,4 @@ them as you type. Select from the **Namespace** dropdown to filter only projects in that namespace. You can combine the filter options. For example, click the **Public** tab, and enter `score` in -the **Filter by name...** input box to list only public projects with `score` in their name.
\ No newline at end of file +the **Filter by name...** input box to list only public projects with `score` in their name. diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md new file mode 100644 index 00000000000..45f986d480f --- /dev/null +++ b/doc/user/admin_area/license.md @@ -0,0 +1,79 @@ +# Activate all GitLab Enterprise Edition functionality with a license **[STARTER ONLY]** + +To activate all GitLab Enterprise Edition (EE) functionality, you need to upload +a license. Once you've received your license from GitLab Inc., you can upload it +by **signing into your GitLab instance as an admin**. + +The license has the form of a base64 encoded ASCII text with a `.gitlab-license` +extension and can be obtained when you [purchase one][pricing] or when you sign +up for a [free trial]. + +NOTE: **Note:** +As of GitLab Enterprise Edition 9.4.0, a newly-installed instance without an +uploaded license will only have the Core features active. A trial license will +activate all Ultimate features, but after +[the trial expires](#what-happens-when-your-license-expires), some functionality +will be locked. + +## Uploading your license + +The very first time you visit your GitLab EE installation signed in as an admin, +you should see a note urging you to upload a license with a link that takes you +straight to the License admin area. + +Otherwise, you can: + +1. Navigate manually to the **Admin Area** by clicking the wrench icon in the menu bar. + +  + +1. And then going to the **License** tab and click on **Upload New License**. + +  + +1. If you've received a `.gitlab-license` file, you should have already downloaded + it in your local machine. You can then upload it directly by choosing the + license file and clicking the **Upload license** button. In the image below, + you can see that the selected license file is named `GitLab.gitlab-license`. + +  + + If you've received your license as plain text, you need to select the + "Enter license key" option, copy the license, paste it into the "License key" + field and click **Upload license**. + +--- + +Once the license is uploaded, all GitLab Enterprise Edition functionality +will be active until the end of the license period. When that period ends, the +instance will [fall back](#what-happens-when-your-license-expires) to Core-only +functionality. + +You can review the license details at any time in the License section of the +Admin Area. + + + +## Notification before the license expires + +One month before the license expires, a message informing when the expiration +is due to, will start appearing to GitLab admins. Make sure that you update your +license, otherwise you will miss all the paid features if it expires. + +## What happens when your license expires + +In case your license expires, GitLab will lock down some features like Git pushes, +issue creation, etc., and a message to inform of the expired license will be +presented to all admins. + +In order to get back all the previous functionality, a new license must be uploaded. +To fall back to having only the Core features active, you'll need to delete the +expired license(s). + +## License history + +It's possible to upload and view more than one license, +but only the latest license will be used as the active license. + +[free trial]: https://about.gitlab.com/free-trial/ +[pricing]: https://about.gitlab.com/pricing/ diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md new file mode 100644 index 00000000000..4a5bfb6b677 --- /dev/null +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -0,0 +1,53 @@ +# Account and limit settings + +## Repository size limit **[STARTER]** + +> [Introduced][ee-740] in [GitLab Enterprise Edition 8.12][ee-8.12]. + +Repositories within your GitLab instance can grow quickly, especially if you are +using LFS. Their size can grow exponentially and eat up your storage device quite +fast. + +In order to avoid this from happening, you can set a hard limit for your +repositories' size. This limit can be set globally, per group, or per project, +with per project limits taking the highest priority. + +There are numerous cases where you'll need to set up a limit for repository size. +For instance, consider the following workflow: + +1. Your team develops apps which demand large files to be stored in + the application repository. +1. Although you have enabled [Git LFS](../../../workflow/lfs/manage_large_binaries_with_git_lfs.html#git-lfs) + to your project, your storage has grown significantly. +1. Before you blow your storage limit up, you set up a limit of 10 GB + per repository. + +### How it works + +Only a GitLab administrator can set those limits. Setting the limit to `0` means +there are no restrictions. + +These settings can be found within: + +- Each project's settings. +- A group's settings. +- The **Size limit per repository (MB)** field in the **Account and limit** section of a GitLab instance's + settings by navigating to either: + - **Admin Area > Settings > General**. + - The path `/admin/application_settings`. + +The very first push of a new project cannot be checked for size as of now, so +the first push will allow you to upload more than the limit dictates, but every +subsequent push will be denied. LFS objects, however, can be checked on first +push and **will** be rejected if the sum of their sizes exceeds the maximum +allowed repository size. + +For more manually purging the files, read the docs on +[reducing the repository size using Git][repo-size]. + +> **Note:** +> For GitLab.com, the repository size limit is 10 GB. + +[ee-740]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/740 +[repo-size]: ../../project/repository/reducing_the_repo_size_using_git.md +[ee-8.12]: https://about.gitlab.com/2016/09/22/gitlab-8-12-released/#limit-project-size-ee diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 22011cc6967..834a1e2423a 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -10,8 +10,8 @@ You can find it in the admin area, under **Settings > Continuous Integration and To enable (or disable) [Auto DevOps](../../../topics/autodevops/index.md) for all projects: -1. Go to **Admin area > Settings > Continuous Integration and Deployment**. -1. Check (or uncheck to disable) the box that says "Default to Auto DevOps pipeline for all projects". +1. Go to **Admin area > Settings > Continuous Integration and Deployment** +1. Check (or uncheck to disable) the box that says "Default to Auto DevOps pipeline for all projects" 1. Optionally, set up the [Auto DevOps base domain](../../../topics/autodevops/index.md#auto-devops-base-domain) which is going to be used for Auto Deploy and Auto Review Apps. 1. Hit **Save changes** for the changes to take effect. @@ -24,9 +24,9 @@ If you want to disable it for a specific project, you can do so in ## Maximum artifacts size **[CORE ONLY]** -The maximum size of the [job artifacts][art-yml] can be set in the Admin area -of your GitLab instance. The value is in *MB* and the default is 100MB per job; -on GitLab.com it's [set to 1G](../../gitlab_com/index.md#gitlab-cicd). +The maximum size of the [job artifacts](../../../administration/job_artifacts.md) +can be set in the Admin area of your GitLab instance. The value is in *MB* and +the default is 100MB per job; on GitLab.com it's [set to 1G](../../gitlab_com/index.md#gitlab-cicd). To change it: @@ -50,6 +50,79 @@ This setting is set per job and can be overridden in [`.gitlab-ci.yml`](../../../ci/yaml/README.md#artifactsexpire_in). To disable the expiration, set it to `0`. The default unit is in seconds. +## Shared Runners pipeline minutes quota **[STARTER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1078) +in GitLab Starter 8.16. + +If you have enabled shared Runners for your GitLab instance, you can limit their +usage by setting a maximum number of pipeline minutes that a group can use on +shared Runners per month. Setting this to `0` (default value) will grant +unlimited pipeline minutes. While build limits are stored as minutes, the +counting is done in seconds. Usage resets on the first day of each month. +On GitLab.com, the quota is calculated based on your +[subscription plan](https://about.gitlab.com/pricing/#gitlab-com). + +To change the pipelines minutes quota: + +1. Go to **Admin area > Settings > Continuous Integration and Deployment** +1. Set the pipeline minutes quota limit. +1. Hit **Save changes** for the changes to take effect + +--- + +While the setting in the Admin area has a global effect, as an admin you can +also change each group's pipeline minutes quota to override the global value. + +1. Navigate to the **Groups** admin area and hit the **Edit** button for the + group you wish to change the pipeline minutes quota. +1. Set the pipeline minutes quota to the desired value +1. Hit **Save changes** for the changes to take effect. + +Once saved, you can see the build quota in the group admin view. +The quota can also be viewed in the project admin view if shared Runners +are enabled. + + + +When the pipeline minutes quota for a group is set to a value different than 0, +the **Pipelines quota** page is available to the group page settings list. +You can see there an overview of the pipeline minutes quota of all projects of +the group. + + + + +## Extra Shared Runners pipeline minutes quota + +NOTE: **Note:** +Only available on GitLab.com. + +If you have a Group with a [paid plan](https://about.gitlab.com/pricing/#gitlab-com) on GitLab.com, +then you can purchase additional CI minutes so your pipelines will not be blocked after you have +used all your CI minutes from your main quota. + +In order to purchase additional minutes, you should follow these steps: + +1. Go to **Group > Settings > Pipelines quota**. Once you are on that page, click on **Buy additional minutes**. + +  + +1. Locate the subscription card that is linked to your group on GitLab.com, +click on **Buy more CI minutes**, and complete the details about the transaction. + +  + +1. Once we have processed your payment, the extra CI minutes +will be synced to your Group and you can visualize it from the +**Group > Settings > Pipelines quota** page: + +  + +NOTE: **Important note**: If you have some minutes used over your default quota, these minutes will +be deducted from your Additional Minutes quota immediately after your purchase of additional +minutes. + ## Archive jobs **[CORE ONLY]** Archiving jobs is useful for reducing the CI/CD footprint on the system by diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 50c318a4969..01a98cf15dc 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -4,6 +4,22 @@ The logo in the header of some emails can be customized, see the [logo customization section](../../../customization/branded_page_and_email_header.md). +## Custom additional text **[PREMIUM ONLY]** + +>[Introduced][ee-5031] in [GitLab Premium][eep] 10.7. + +The additional text will appear at the bottom of any email and can be used for +legal/auditing/compliance reasons. + +1. Go to **Admin area > Settings** (`/admin/application_settings`). +1. Under the **Email** section, change the **Additional text** field. +1. Hit **Save** for the changes to take effect. + + + +[ee-5031]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5031 +[eep]: https://about.gitlab.com/pricing/ + ## Custom hostname for private commit emails > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22560) in GitLab 11.5. diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index 72e76ac2a84..06e00e02f3d 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -1,4 +1,4 @@ -# External authorization control +# External authorization control **[CORE ONLY]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4216) in > [GitLab Premium](https://about.gitlab.com/pricing) 10.6. diff --git a/doc/user/admin_area/settings/img/additional_minutes.png b/doc/user/admin_area/settings/img/additional_minutes.png Binary files differnew file mode 100644 index 00000000000..d148ed79b92 --- /dev/null +++ b/doc/user/admin_area/settings/img/additional_minutes.png diff --git a/doc/user/admin_area/settings/img/admin_area_group_edit.png b/doc/user/admin_area/settings/img/admin_area_group_edit.png Binary files differnew file mode 100644 index 00000000000..c9bd2f10b36 --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_area_group_edit.png diff --git a/doc/user/admin_area/settings/img/admin_area_groups.png b/doc/user/admin_area/settings/img/admin_area_groups.png Binary files differnew file mode 100644 index 00000000000..ebdee0eafdc --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_area_groups.png diff --git a/doc/user/admin_area/settings/img/admin_project_quota_view.png b/doc/user/admin_area/settings/img/admin_project_quota_view.png Binary files differnew file mode 100644 index 00000000000..8320be860da --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_project_quota_view.png diff --git a/doc/user/admin_area/settings/img/buy_btn.png b/doc/user/admin_area/settings/img/buy_btn.png Binary files differnew file mode 100644 index 00000000000..0cc88b8a48f --- /dev/null +++ b/doc/user/admin_area/settings/img/buy_btn.png diff --git a/doc/user/admin_area/settings/img/buy_minutes_card.png b/doc/user/admin_area/settings/img/buy_minutes_card.png Binary files differnew file mode 100644 index 00000000000..cf4ad34ead7 --- /dev/null +++ b/doc/user/admin_area/settings/img/buy_minutes_card.png diff --git a/doc/user/admin_area/settings/img/ci_shared_runners_build_minutes_quota.png b/doc/user/admin_area/settings/img/ci_shared_runners_build_minutes_quota.png Binary files differnew file mode 100644 index 00000000000..269a3cf1fbc --- /dev/null +++ b/doc/user/admin_area/settings/img/ci_shared_runners_build_minutes_quota.png diff --git a/doc/user/admin_area/settings/img/email_settings.png b/doc/user/admin_area/settings/img/email_settings.png Binary files differnew file mode 100644 index 00000000000..ed0a80d10ce --- /dev/null +++ b/doc/user/admin_area/settings/img/email_settings.png diff --git a/doc/user/admin_area/settings/img/file_template_admin_area.png b/doc/user/admin_area/settings/img/file_template_admin_area.png Binary files differnew file mode 100644 index 00000000000..269d997e1d9 --- /dev/null +++ b/doc/user/admin_area/settings/img/file_template_admin_area.png diff --git a/doc/user/admin_area/settings/img/file_template_user_dropdown.png b/doc/user/admin_area/settings/img/file_template_user_dropdown.png Binary files differnew file mode 100644 index 00000000000..8c9eb49f6c9 --- /dev/null +++ b/doc/user/admin_area/settings/img/file_template_user_dropdown.png diff --git a/doc/user/admin_area/settings/img/group_pipelines_quota.png b/doc/user/admin_area/settings/img/group_pipelines_quota.png Binary files differnew file mode 100644 index 00000000000..d94b609ad6f --- /dev/null +++ b/doc/user/admin_area/settings/img/group_pipelines_quota.png diff --git a/doc/user/admin_area/settings/img/group_quota_view.png b/doc/user/admin_area/settings/img/group_quota_view.png Binary files differnew file mode 100644 index 00000000000..791bfd868e0 --- /dev/null +++ b/doc/user/admin_area/settings/img/group_quota_view.png diff --git a/doc/user/admin_area/settings/img/group_settings.png b/doc/user/admin_area/settings/img/group_settings.png Binary files differnew file mode 100644 index 00000000000..a849d9cfdc1 --- /dev/null +++ b/doc/user/admin_area/settings/img/group_settings.png diff --git a/doc/user/admin_area/settings/img/mirror_settings.png b/doc/user/admin_area/settings/img/mirror_settings.png Binary files differnew file mode 100644 index 00000000000..090db6808a7 --- /dev/null +++ b/doc/user/admin_area/settings/img/mirror_settings.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 8358fe64f18..0e697b9e445 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -6,6 +6,7 @@ instance like sign-up restrictions, account limits and quota, metrics, etc. Navigate to it by going to **Admin area > Settings**. Some of the settings include: +- [Account and limit settings](account_and_limit_settings.md) **[STARTER]** - [Continuous Integration and Deployment](continuous_integration.md) - [Email](email.md) - [Sign up restrictions](sign_up_restrictions.md) @@ -13,6 +14,7 @@ include: - [Third party offers](third_party_offers.md) - [Usage statistics](usage_statistics.md) - [Visibility and access controls](visibility_and_access_controls.md) +- [Custom templates repository](instance_template_repository.md) **[PREMIUM]** NOTE: **Note:** You can change the [first day of the week](../../profile/preferences.md) for the entire GitLab instance diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md new file mode 100644 index 00000000000..4010008f694 --- /dev/null +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -0,0 +1,63 @@ +# Instance template repository **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5986) in +> [GitLab Premium](https://about.gitlab.com/pricing) 11.3. + +## Overview + +In hosted systems, enterprises often have a need to share their own templates +across teams. This feature allows an administrator to pick a project to be the +instance-wide collection of file templates. These templates are then exposed to +all users [via the web editor](../../project/repository/web_editor.md#template-dropdowns) +while the project remains secure. + +## Configuration + +As an administrator, navigate to **Admin area > Settings > Templates** and +select the project to serve as the custom template repository. + + + +Once a project has been selected, you can add custom templates to the repository, +and they will appear in the appropriate places in the +[frontend](../../project/repository/web_editor.md#template-dropdowns) and +[API](../../../api/settings.md). + +Templates must be added to a specific subdirectory in the repository, +corresponding to the kind of template. The following types of custom templates +are supported: + +| Type | Directory | Extension | +| :---------------: | :-----------: | :-----------: | +| `Dockerfile` | `Dockerfile` | `.dockerfile` | +| `.gitignore` | `gitignore` | `.gitignore` | +| `.gitlab-ci.yml` | `gitlab-ci` | `.yml` | +| `LICENSE` | `LICENSE` | `.txt` | + +Each template must go in its respective subdirectory, have the correct +extension and not be empty. So, the hierarchy should look like this: + +```text +|-- README.md +|-- Dockerfile + |-- custom_dockerfile.dockerfile + |-- another_dockerfile.dockerfile +|-- gitignore + |-- custom_gitignore.gitignore + |-- another_gitignore.gitignore +|-- gitlab-ci + |-- custom_gitlab-ci.yml + |-- another_gitlab-ci.yml +|-- LICENSE + |-- custom_license.txt + |-- another_license.txt +``` + +Once this is established, the list of custom templates will be included when +creating a new file and the template type is selected. These will appear at the +top of the list. + + + +If this feature is disabled or no templates are present, there will be +no "Custom" section in the selection dropdown. diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 6a1e8004f87..4d1b7d0f252 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -49,5 +49,15 @@ block access to the server itself. The ports used for the protocol, be it SSH or HTTP, will still be accessible. What GitLab does is restrict access on the application level. +## Allow mirrors to be set up for projects + +> [Introduced][ee-3586] in GitLab 10.3. + +This option is enabled by default. By disabling it, both pull and push mirroring will no longer +work in every repository and can only be re-enabled on a per-project basis by an admin. + + + [ce-4696]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4696 [ce-18021]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18021 +[ee-3586]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3586 diff --git a/doc/user/application_security/container_scanning/img/container_scanning.png b/doc/user/application_security/container_scanning/img/container_scanning.png Binary files differnew file mode 100644 index 00000000000..e47f62acd9d --- /dev/null +++ b/doc/user/application_security/container_scanning/img/container_scanning.png diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md new file mode 100644 index 00000000000..ce86ade3c1a --- /dev/null +++ b/doc/user/application_security/container_scanning/index.md @@ -0,0 +1,202 @@ +# Container Scanning **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3672) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can check your Docker +images (or more precisely the containers) for known vulnerabilities by using +[Clair](https://github.com/coreos/clair) and [clair-scanner](https://github.com/arminc/clair-scanner), +two open source tools for Vulnerability Static Analysis for containers. + +You can take advantage of Container Scanning by either [including the CI job](#including-the-provided-template) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto Container Scanning](../../../topics/autodevops/index.md#auto-container-scanning) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the Container Scanning report, compares the found vulnerabilities +between the source and target branches, and shows the information right on the +merge request. + + + +## Use cases + +If you distribute your application with Docker, then there's a great chance +that your image is based on other Docker images that may in turn contain some +known vulnerabilities that could be exploited. + +Having an extra job in your pipeline that checks for those vulnerabilities, +and the fact that they are displayed inside a merge request, makes it very easy +to perform audits for your Docker-based apps. + +## Requirements + +To enable Container Scanning in your pipeline, you need: + +- A GitLab Runner with the + [`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or + [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) + executor running in privileged mode. If you're using the shared Runners on GitLab.com, + this is enabled by default. +- To [build and push](../../../ci/docker/using_docker_build.md#container-registry-examples) + your Docker image to your project's [Container Registry](../../project/container_registry.md). + The name of the Docker image should match the following scheme: + + ``` + $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA + ``` + + The variables above can be found in the + [predefined environment variables](../../../ci/variables/predefined_variables.md) + document. + +## Configuring Container Scanning + +To enable Container Scanning in your project, define a job in your +`.gitlab-ci.yml` file that generates the +[Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided + `Container-Scanning.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD Container Scanning template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [Container Scanning template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) +with the default Container Scanning job definition is provided as a part of your GitLab +installation that you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable Container Scanning using the provided template, add the following to +your `.gitlab-ci.yml` file: + +```yaml +include: + template: Container-Scanning.gitlab-ci.yml +``` + +The included template will: + +- Create a `container_scanning` job in your CI/CD pipeline. +- Pull the already built Docker image from your project's + [Container Registry](../../project/container_registry.md) (see [requirements](#requirements)) + and scan it for possible vulnerabilities. + +The report will be saved as a +[Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning-ultimate) +that you can later download and analyze. +Due to implementation limitations, we always take the latest Container Scanning +artifact available. Behind the scenes, the +[GitLab Container Scanning analyzer](https://gitlab.com/gitlab-org/security-products/container-scanning) +is used and runs the scans. + +If you want to whitelist some specific vulnerabilities, you can do so by defining +them in a YAML file named `clair-whitelist.yml`. Read more in the +[Clair documentation](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file). + +### Manual job definition for GitLab 11.5 and later + +CAUTION: **Caution:** +The job definition shown below is supported on GitLab 11.5 and later versions. +However, if you're using GitLab 11.9+, it's recommended to use +[the provided Container Scanning template](#including-the-provided-template). + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `container_scanning` +job can be added: + +```yaml +container_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + ## Define two new variables based on GitLab's CI/CD predefined variables + ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables + CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG + CI_APPLICATION_TAG: $CI_COMMIT_SHA + allow_failure: true + services: + - docker:stable-dind + script: + - docker run -d --name db arminc/clair-db:latest + - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 + - apk add -U wget ca-certificates + - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} + - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 + - mv clair-scanner_linux_amd64 clair-scanner + - chmod +x clair-scanner + - touch clair-whitelist.yml + - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done + - retries=0 + - echo "Waiting for clair daemon to start" + - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done + - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true + artifacts: + reports: + container_scanning: gl-container-scanning-report.json +``` + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Deprecated:** +Before GitLab 11.5, the Container Scanning job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly +advised to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the Container Scanning job should look like: + +```yaml +container_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + ## Define two new variables based on GitLab's CI/CD predefined variables + ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables + CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG + CI_APPLICATION_TAG: $CI_COMMIT_SHA + allow_failure: true + services: + - docker:stable-dind + script: + - docker run -d --name db arminc/clair-db:latest + - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 + - apk add -U wget ca-certificates + - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} + - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 + - mv clair-scanner_linux_amd64 clair-scanner + - chmod +x clair-scanner + - touch clair-whitelist.yml + - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done + - retries=0 + - echo "Waiting for clair daemon to start" + - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done + - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true + artifacts: + paths: [gl-container-scanning-report.json] +``` + +Alternatively, the job name could be `sast:container` +and the artifact name could be `gl-sast-container-report.json`. +These names have been deprecated with GitLab 11.0 +and may be removed in the next major release, GitLab 12.0. + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). diff --git a/doc/user/application_security/dast/img/dast_all.png b/doc/user/application_security/dast/img/dast_all.png Binary files differnew file mode 100644 index 00000000000..b6edc928dc3 --- /dev/null +++ b/doc/user/application_security/dast/img/dast_all.png diff --git a/doc/user/application_security/dast/img/dast_single.png b/doc/user/application_security/dast/img/dast_single.png Binary files differnew file mode 100644 index 00000000000..26ca4bde786 --- /dev/null +++ b/doc/user/application_security/dast/img/dast_single.png diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md new file mode 100644 index 00000000000..904c9e8fefe --- /dev/null +++ b/doc/user/application_security/dast/index.md @@ -0,0 +1,262 @@ +# Dynamic Application Security Testing (DAST) **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4348) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. + +Running [static checks](../sast/index.md) on your code is the first step to detect +vulnerabilities that can put the security of your code at risk. Yet, once +deployed, your application is exposed to a new category of possible attacks, +such as cross-site scripting or broken authentication flaws. This is where +Dynamic Application Security Testing (DAST) comes into place. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your running web application(s) +for known vulnerabilities using Dynamic Application Security Testing (DAST). + +You can take advantage of DAST by either [including the CI job](#configuring-dast) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto DAST](../../../topics/autodevops/index.md#auto-dast-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the DAST report, compares the found vulnerabilities between the source and target +branches, and shows the information right on the merge request. + + + +By clicking on one of the detected linked vulnerabilities, you will be able to +see the details and the URL(s) affected. + + + +[Dynamic Application Security Testing (DAST)](https://en.wikipedia.org/wiki/Dynamic_Application_Security_Testing) +is using the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy) +to perform an analysis on your running web application. + +By default, DAST executes [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan) and will perform passive scanning only. It will not actively attack your application. + +However, DAST can be [configured](#full-scan) +to also perform a so-called "active scan". That is, attack your application and produce a more extensive security report. +It can be very useful combined with [Review Apps](../../../ci/review_apps/index.md). + +## Use cases + +It helps you automatically find security vulnerabilities in your running web +applications while you are developing and testing your applications. + +## Requirements + +To run a DAST job, you need GitLab Runner with the +[`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or +[`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) +executor running in privileged mode. If you're using the shared Runners on GitLab.com, +this is enabled by default. + +## Configuring DAST + +To enable DAST in your project, define a job in your `.gitlab-ci.yml` file that generates the +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided DAST `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD DAST template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [DAST template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml) +with the default DAST job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable DAST using the provided template, add the following to your `.gitlab-ci.yml` +file: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com +``` + +The included template will create a `dast` job in your CI/CD pipeline and scan +your project's source code for possible vulnerabilities. + +The report will be saved as a +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate) +that you can later download and analyze. Due to implementation limitations we +always take the latest DAST artifact available. Behind the scenes, the +[GitLab DAST Docker image](https://gitlab.com/gitlab-org/security-products/dast) +is used to run the tests on the specified URL and scan it for possible vulnerabilities. + +There are two ways to define the URL to be scanned by DAST: + +- Set the `DAST_WEBSITE` [variable](../../../ci/yaml/README.md#variables). +- Add it in an `environment_url.txt` file at the root of your project. + +#### Authenticated scan + +It's also possible to authenticate the user before performing the DAST checks: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com + DAST_AUTH_URL: https://example.com/sign-in + DAST_USERNAME: john.doe@example.com + DAST_PASSWORD: john-doe-password + DAST_USERNAME_FIELD: session[user] # the name of username field at the sign-in HTML form + DAST_PASSWORD_FIELD: session[password] # the name of password field at the sign-in HTML form + DAST_AUTH_EXCLUDE_URLS: http://example.com/sign-out,http://example.com/sign-out-2 # optional, URLs to skip during the authenticated scan; comma-separated, no spaces in between +``` + +The report will be saved as a +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate) +that you can later download and analyze. +Due to implementation limitations, we always take the latest DAST artifact available. + +#### Full scan + +DAST can be configured to perform [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan), which +includes both passive and active scanning against the same target website: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_FULL_SCAN_ENABLED: "true" +``` + +#### Customizing the DAST settings + +The SAST settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. +These variables are documented in the [DAST README](https://gitlab.com/gitlab-org/security-products/dast#settings). + +For example: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com + DAST_TARGET_AVAILABILITY_TIMEOUT: 120 +``` + +Because the template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline +configuration, the last mention of the variable will take precedence. + +#### Overriding the DAST template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `dast` job after the +template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + stage: dast # IMPORTANT: don't forget to add this + variables: + DAST_WEBSITE: https://example.com + CI_DEBUG_TRACE: "true" +``` + +As the DAST job belongs to a separate `dast` stage that runs after all +[default stages](../../../ci/yaml/README.md#stages), +don't forget to add `stage: dast` when you override the template job definition. + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dast` +job can be added: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + reports: + dast: gl-dast-report.json +``` + +Where the `website` variable holds the URL to run the tests against. + +For an authenticated scan, use the following definition: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + login_url: "https://example.com/sign-in" + username: "john.doe@example.com" + password: "john-doe-password" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website + --auth-url $login_url + --auth-username $username + --auth-password $password || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + reports: + dast: gl-dast-report.json +``` + +See the [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) +to learn more about the authentication settings. + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Caution:** +Before GitLab 11.5, DAST job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained they have been deprecated +and may be removed in next major release, GitLab 12.0. You are strongly advised +to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + paths: [gl-dast-report.json] +``` + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). diff --git a/doc/user/application_security/dependency_scanning/img/dependency_scanning.png b/doc/user/application_security/dependency_scanning/img/dependency_scanning.png Binary files differnew file mode 100644 index 00000000000..18df356f846 --- /dev/null +++ b/doc/user/application_security/dependency_scanning/img/dependency_scanning.png diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md new file mode 100644 index 00000000000..2d0c2be4233 --- /dev/null +++ b/doc/user/application_security/dependency_scanning/index.md @@ -0,0 +1,225 @@ +# Dependency Scanning **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5105) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.7. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your dependencies for known +vulnerabilities using Dependency Scanning. + +You can take advantage of Dependency Scanning by either [including the CI job](#including-the-provided-template) +in your existing `.gitlab-ci.yml` file or by implicitly using +[Auto Dependency Scanning](../../../topics/autodevops/index.md#auto-dependency-scanning-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the Dependency Scanning report, compares the found vulnerabilities +between the source and target branches, and shows the information right on the +merge request. + + + +The results are sorted by the priority of the vulnerability: + +1. High +1. Medium +1. Low +1. Unknown +1. Everything else + +## Use cases + +It helps to automatically find security vulnerabilities in your dependencies +while you are developing and testing your applications. For example when your +application is using an external (open source) library which is known to be vulnerable. + +## Requirements + +To run a Dependency Scanning job, you need GitLab Runner with the +[`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or +[`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) +executor running in privileged mode. If you're using the shared Runners on GitLab.com, +this is enabled by default. + +## Supported languages and package managers + +The following languages and dependency managers are supported. + +| Language (package managers) | Scan tool | +|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| +| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [Retire.js](https://retirejs.github.io/retire.js) | +| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | +| Ruby ([gem](https://rubygems.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [bundler-audit](https://github.com/rubysec/bundler-audit) | +| Java ([Maven](https://maven.apache.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | +| PHP ([Composer](https://getcomposer.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | + +Some scanners require to send a list of project dependencies to GitLab's central +servers to check for vulnerabilities. To learn more about this or to disable it, +refer to the [GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). + +## Configuring Dependency Scanning + +To enable Dependency Scanning in your project, define a job in your `.gitlab-ci.yml` +file that generates the +[Dependency Scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided Dependency Scanning + `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD Dependency Scanning template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [Dependency Scanning template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) +with the default Dependency Scanning job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable Dependency Scanning using the provided template, add the following to +your `.gitlab-ci.yml` file: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml +``` + +The included template will create a `dependency_scanning` job in your CI/CD +pipeline and scan your project's source code for possible vulnerabilities. + +The report will be saved as a +[Dependency Scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate) +that you can later download and analyze. Due to implementation limitations, we +always take the latest Dependency Scanning artifact available. + +Some security scanners require to send a list of project dependencies to GitLab +central servers to check for vulnerabilities. To learn more about this or to +disable it, check the +[GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). + +#### Customizing the Dependency Scanning settings + +The Dependency Scanning settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. +These variables are documented in the +[Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings). + +For example: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DEP_SCAN_DISABLE_REMOTE_CHECKS: true +``` + +Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline +configuration, the last mention of the variable will take precedence. + +#### Overriding the Dependency Scanning template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `dependency_scanning` job +after the template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +dependency_scanning: + variables: + CI_DEBUG_TRACE: "true" +``` + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dependency_scanning` +job can be added: + +```yaml +dependency_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export DS_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - | + docker run \ + --env DS_ANALYZER_IMAGES \ + --env DS_ANALYZER_IMAGE_PREFIX \ + --env DS_ANALYZER_IMAGE_TAG \ + --env DS_DEFAULT_ANALYZERS \ + --env DEP_SCAN_DISABLE_REMOTE_CHECKS \ + --env DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ + --env DS_PULL_ANALYZER_IMAGE_TIMEOUT \ + --env DS_RUN_ANALYZER_TIMEOUT \ + --volume "$PWD:/code" \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$DS_VERSION" /code + dependencies: [] + artifacts: + reports: + dependency_scanning: gl-dependency-scanning-report.json +``` + +You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings) +via `docker run --env` to customize your job execution. + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Caution:** +Before GitLab 11.5, the Dependency Scanning job and artifact had to be named specifically +to automatically extract the report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly advised +to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +dependency_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export DS_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - | + docker run \ + --env DS_ANALYZER_IMAGES \ + --env DS_ANALYZER_IMAGE_PREFIX \ + --env DS_ANALYZER_IMAGE_TAG \ + --env DS_DEFAULT_ANALYZERS \ + --env DS_EXCLUDED_PATHS \ + --env DEP_SCAN_DISABLE_REMOTE_CHECKS \ + --env DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ + --env DS_PULL_ANALYZER_IMAGE_TIMEOUT \ + --env DS_RUN_ANALYZER_TIMEOUT \ + --volume "$PWD:/code" \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$DS_VERSION" /code + artifacts: + paths: [gl-dependency-scanning-report.json] +``` + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). diff --git a/doc/user/application_security/img/create_issue_with_list_hover.png b/doc/user/application_security/img/create_issue_with_list_hover.png Binary files differnew file mode 100644 index 00000000000..7d70e8299f5 --- /dev/null +++ b/doc/user/application_security/img/create_issue_with_list_hover.png diff --git a/doc/user/application_security/img/interactive_reports.png b/doc/user/application_security/img/interactive_reports.png Binary files differnew file mode 100644 index 00000000000..9f9812dc69d --- /dev/null +++ b/doc/user/application_security/img/interactive_reports.png diff --git a/doc/user/application_security/img/issue.png b/doc/user/application_security/img/issue.png Binary files differnew file mode 100644 index 00000000000..6467201df3f --- /dev/null +++ b/doc/user/application_security/img/issue.png diff --git a/doc/user/application_security/img/vulnerability_solution.png b/doc/user/application_security/img/vulnerability_solution.png Binary files differnew file mode 100644 index 00000000000..7443b9b6eea --- /dev/null +++ b/doc/user/application_security/img/vulnerability_solution.png diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md new file mode 100644 index 00000000000..64e72fab198 --- /dev/null +++ b/doc/user/application_security/index.md @@ -0,0 +1,104 @@ +# GitLab Secure **[ULTIMATE]** + +Check your application for security vulnerabilities that may lead to unauthorized access, +data leaks, and denial of services. GitLab will perform static and dynamic tests on the +code of your application, looking for known flaws and report them in the merge request +so you can fix them before merging. Security teams can use dashboards to get a +high-level view on projects and groups, and start remediation processes when needed. + +## Security scanning tools + +GitLab can scan and report any vulnerabilities found in your project. + +| Secure scanning tools | Description | +|:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------| +| [Container Scanning](container_scanning/index.md) **[ULTIMATE]** | Scan Docker containers for known vulnerabilities. | +| [Dependency Scanning](dependency_scanning/index.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [Dynamic Application Security Testing (DAST)](dast/index.md) **[ULTIMATE]** | Analyze running web applications for known vulnerabilities. | +| [License Management](license_management/index.md) **[ULTIMATE]** | Search your project's dependencies for their licenses. | +| [Security Dashboard](security_dashboard/index.md) **[ULTIMATE]** | View vulnerabilities in all your projects and groups. | +| [Static Application Security Testing (SAST)](sast/index.md) **[ULTIMATE]** | Analyze source code for known vulnerabilities. | + +## Interacting with the vulnerabilities + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.8. + +CAUTION: **Warning:** +This feature is currently [Alpha](https://about.gitlab.com/handbook/product/#alpha-beta-ga) and while you can start using it, it may receive important changes in the future. + +Each security vulnerability in the merge request report or the +[Security Dashboard](security_dashboard/index.md) is actionable. Clicking on an +entry, a detailed information will pop up with different possible options: + +- [Dismiss vulnerability](#dismissing-a-vulnerability): Dismissing a vulnerability + will place a <s>strikethrough</s> styling on it. +- [Create issue](#creating-an-issue-for-a-vulnerability): The new issue will + have the title and description pre-populated with the information from the + vulnerability report and will be created as [confidential](../project/issues/confidential_issues.md) by default. +- [Solution](#solutions-for-vulnerabilities): For some vulnerabilities + ([Dependency Scanning](dependency_scanning/index.md) and [Container Scanning](container_scanning/index.md)) + a solution is provided for how to fix the vulnerability. + + + +### Dismissing a vulnerability + +You can dismiss vulnerabilities by clicking the **Dismiss vulnerability** button. +This will dismiss the vulnerability and re-render it to reflect its dismissed state. +If you wish to undo this dismissal, you can click the **Undo dismiss** button. + +### Creating an issue for a vulnerability + +You can create an issue for a vulnerability by selecting the **Create issue** +button from within the vulnerability modal or using the action buttons to the right of +a vulnerability row when in the group security dashboard. + +This will create a [confidential issue](../project/issues/confidential_issues.md) +on the project this vulnerability came from and pre-fill it with some useful +information taken from the vulnerability report. Once the issue is created, you +will be redirected to it so you can edit, assign, or comment on it. + +Upon returning to the group security dashboard, you'll see that +the vulnerability will now have an associated issue next to the name. + + + +### Solutions for vulnerabilities + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.7. + +CAUTION: **Warning:** +Automatic Patch creation is only available for a subset of +[Dependency Scanning](dependency_scanning/index.md). At the moment only Node.JS +projects managed with yarn are supported. + +Some vulnerabilities can be fixed by applying the solution that GitLab +automatically generates. + +#### Manually applying the suggested patch + +Some vulnerabilities can be fixed by applying a patch that is automatically +generated by GitLab. To apply the fix: + +1. Click on the vulnerability. +1. Download and review the patch file `remediation.patch`. +2. Ensure your local project has the same commit checked out that was used to generate the patch. +3. Run `git apply remediation.patch`. +4. Verify and commit the changes to your branch. + + + +#### Creating a merge request from a vulnerability + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9224) in + [GitLab Ultimate](https://about.gitlab.com/pricing) 11.9. + +In certain cases, GitLab will allow you to create a merge request that will +automatically remediate the vulnerability. Any vulnerability that has a +[solution](#solutions-for-vulnerabilities) can have a merge request created to +automatically solve the issue. + +If this action is available there will be a **Create merge request** button in the vulnerability modal. +Clicking on this button will create a merge request to apply the solution onto the source branch. + + diff --git a/doc/user/application_security/license_management/img/license_management.png b/doc/user/application_security/license_management/img/license_management.png Binary files differnew file mode 100644 index 00000000000..cdce6b5fe38 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management.png diff --git a/doc/user/application_security/license_management/img/license_management_decision.png b/doc/user/application_security/license_management/img/license_management_decision.png Binary files differnew file mode 100644 index 00000000000..0763130c375 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_decision.png diff --git a/doc/user/application_security/license_management/img/license_management_pipeline_tab.png b/doc/user/application_security/license_management/img/license_management_pipeline_tab.png Binary files differnew file mode 100644 index 00000000000..80ffca815b9 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_pipeline_tab.png diff --git a/doc/user/application_security/license_management/img/license_management_settings.png b/doc/user/application_security/license_management/img/license_management_settings.png Binary files differnew file mode 100644 index 00000000000..b5490e59074 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_settings.png diff --git a/doc/user/application_security/license_management/index.md b/doc/user/application_security/license_management/index.md new file mode 100644 index 00000000000..8b75995c377 --- /dev/null +++ b/doc/user/application_security/license_management/index.md @@ -0,0 +1,264 @@ +# License Management **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5483) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can search your project dependencies for their licenses +using License Management. + +You can take advantage of License Management by either [including the job](#configuring-license-management) +in your existing `.gitlab-ci.yml` file or by implicitly using +[Auto License Management](../../../topics/autodevops/index.md#auto-license-management-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the License Management report, compares the licenses between the +source and target branches, and shows the information right on the merge request. +Blacklisted licenses will be clearly visible with an `x` red icon next to them +as well as new licenses which need a decision from you. In addition, you can +[manually approve or blacklist](#project-policies-for-license-management) +licenses in your project's settings. + +NOTE: **Note:** +If the license management report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +`license_management` job in your `.gitlab-ci.yml` for the first time. +Consecutive merge requests will have something to compare to and the license +management report will be shown properly. + + + +If you are a project or group Maintainer, you can click on a license to be given +the choice to approve it or blacklist it. + + + +## Use cases + +It helps you find what licenses your project uses in its dependencies, and decide for each of then +whether to allow it or forbid it. For example, your application is using an external (open source) +library whose license is incompatible with yours. + +## Supported languages and package managers + +The following languages and package managers are supported. + +| Language | Package managers | +|------------|-------------------------------------------------------------------| +| JavaScript | [Bower](https://bower.io/), [npm](https://www.npmjs.com/) | +| Go | [Godep](https://github.com/tools/godep), go get | +| Java | [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) | +| .NET | [Nuget](https://www.nuget.org/) | +| Python | [pip](https://pip.pypa.io/en/stable/) | +| Ruby | [gem](https://rubygems.org/) | + +## Requirements + +To run a License Management scanning job, you need GitLab Runner with the +[`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). + +## Configuring License Management + +To enable License Management in your project, define a job in your `.gitlab-ci.yml` +file that generates the [License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided License Management `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD License Management template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [License Management template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml) +with the default License Management job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable License Management using the provided template, add the following to +your `.gitlab-ci.yml` file: + +```yaml +include: + template: License-Management.gitlab-ci.yml +``` + +The included template will create a `license_management` job in your CI/CD pipeline +and scan your dependencies to find their licenses. + +The report will be saved as a +[License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate) +that you can later download and analyze. Due to implementation limitations, we +always take the latest License Management artifact available. Behind the scenes, the +[GitLab License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management) +is used to detect the languages/frameworks and in turn analyzes the licenses. + +#### Installing custom dependencies + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. + +The `license_management` image already embeds many auto-detection scripts, languages, +and packages. Nevertheless, it's almost impossible to cover all cases for all projects. +That's why sometimes it's necessary to install extra packages, or to have extra steps +in the project automated setup, like the download and installation of a certificate. +For that, a `LICENSE_MANAGEMENT_SETUP_CMD` environment variable can be passed to the container, +with the required commands to run before the license detection. + +If present, this variable will override the setup step necessary to install all the packages +of your application (e.g.: for a project with a `Gemfile`, the setup step could be +`bundle install`). + +For example: + +```yaml +include: + template: License-Management.gitlab-ci.yml + +variables: + LICENSE_MANAGEMENT_SETUP_CMD: sh my-custom-install-script.sh +``` + +In this example, `my-custom-install-script.sh` is a shell script at the root +directory of your project. + +#### Overriding the template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `license_management` job +after the template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: License-Management.gitlab-ci.yml + +license_management: + variables: + CI_DEBUG_TRACE: "true" +``` + +#### Configuring Maven projects + +The License Management tool provides a `MAVEN_CLI_OPTS` environment variable which can hold +the command line arguments to pass to the `mvn install` command which is executed under the hood. +Feel free to use it for the customization of Maven execution. For example: + +```yaml +include: + template: License-Management.gitlab-ci.yml + +license_management: + variables: + MAVEN_CLI_OPTS: --debug +``` + +`mvn install` runs through all of the [build life cycle](http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html) +stages prior to `install`, including `test`. Running unit tests is not directly +necessary for the license scanning purposes and consumes time, so it's skipped +by having the default value of `MAVEN_CLI_OPTS` as `-DskipTests`. If you want +to supply custom `MAVEN_CLI_OPTS` and skip tests at the same time, don't forget +to explicitly add `-DskipTests` to your options. +If you still need to run tests during `mvn install`, add `-DskipTests=false` to +`MAVEN_CLI_OPTS`. + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `license_management` +job can be added: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + allow_failure: true + script: + - /run.sh analyze . + artifacts: + reports: + license_management: gl-license-management-report.json +``` + +If you want to install custom project dependencies via the `SETUP_CMD` variable: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + variables: + SETUP_CMD: ./my-custom-install-script.sh + allow_failure: true + script: + - /run.sh analyze . + artifacts: + reports: + license_management: gl-license-management-report.json +``` + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Caution:** +Before GitLab 11.5, the License Management job and artifact had to be named specifically +to automatically extract the report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly advised +to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + allow_failure: true + script: + - /run.sh analyze . + artifacts: + paths: [gl-license-management-report.json] +``` + +## Project policies for License Management + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5940) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. + +From the project's settings: + +- The list of licenses and their status can be managed. +- Licenses can be manually approved or blacklisted. + +To approve or blacklist a license: + +1. Either use the **Manage licenses** button in the merge request widget, or + navigate to the project's **Settings > CI/CD** and expand the + **License Management** section. +1. Click the **Add a license** button. +1. In the **License name** dropdown, either: + - Select one of the available licenses. You can search for licenses in the field + at the top of the list. + - Enter arbitrary text in the field at the top of the list. This will cause the text to be + added as a license name to the list. +1. Select the **Approve** or **Blacklist** radio button to approve or blacklist respectively + the selected license. + +  + +## License Management report under pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5491) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2. + +From your project's left sidebar, navigate to **CI/CD > Pipelines** and click on the +pipeline ID that has a `license_management` job to see the Licenses tab with the listed +licenses (if any). + + diff --git a/doc/user/application_security/sast/img/sast.png b/doc/user/application_security/sast/img/sast.png Binary files differnew file mode 100644 index 00000000000..2c75592c32a --- /dev/null +++ b/doc/user/application_security/sast/img/sast.png diff --git a/doc/user/application_security/sast/img/security_report.png b/doc/user/application_security/sast/img/security_report.png Binary files differnew file mode 100644 index 00000000000..ba41b707238 --- /dev/null +++ b/doc/user/application_security/sast/img/security_report.png diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md new file mode 100644 index 00000000000..02c115b7f22 --- /dev/null +++ b/doc/user/application_security/sast/index.md @@ -0,0 +1,254 @@ +# Static Application Security Testing (SAST) **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3775) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.3. + +NOTE: **4 of the top 6 attacks were application based.** +Download our whitepaper, +["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) +to learn how to protect your organization. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your source code for known +vulnerabilities using Static Application Security Testing (SAST). + +You can take advantage of SAST by either [including the CI job](#configuring-sast) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto SAST](../../../topics/autodevops/index.md#auto-sast-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the SAST report, compares the found vulnerabilities between the +source and target branches, and shows the information right on the merge request. + + + +The results are sorted by the priority of the vulnerability: + +1. Critical +1. High +1. Medium +1. Low +1. Unknown +1. Everything else + +## Use cases + +- Your code has a potentially dangerous attribute in a class, or unsafe code + that can lead to unintended code execution. +- Your application is vulnerable to cross-site scripting (XSS) attacks that can + be leveraged to unauthorized access to session data. + +## Requirements + +To run a SAST job, you need GitLab Runner with the +[`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or +[`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) +executor running in privileged mode. If you're using the shared Runners on GitLab.com, +this is enabled by default. + +## Supported languages and frameworks + +The following table shows which languages, package managers and frameworks are supported and which tools are used. + +| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | +|-----------------------------------------------------------------------------|----------------------------------------------------------------------------------------|------------------------------| +| .NET | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | +| Any | [Gitleaks](https://github.com/zricethezav/gitleaks) and [TruffleHog](https://github.com/dxa4481/truffleHog) | 11.9 | +| C/C++ | [Flawfinder](https://www.dwheeler.com/flawfinder/) | 10.7 | +| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.10 | +| Go | [Gosec](https://github.com/securego/gosec) | 10.7 | +| Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | +| Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | +| Javascript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | +| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | +| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | +| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/openstack/bandit) | 10.3 | +| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | +| Scala ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven) | +| Typescript | [TSLint config security](https://github.com/webschik/tslint-config-security/) | 11.9 | + +NOTE: **Note:** +The Java analyzers can also be used for variants like the +[Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), +[Grails](https://grails.org/) and the [Maven wrapper](https://github.com/takari/maven-wrapper). + +## Configuring SAST + +To enable SAST in your project, define a job in your `.gitlab-ci.yml` file that generates the +[SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided SAST `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD SAST template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [SAST template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) +with the default SAST job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable SAST using the provided template, add the following to your `.gitlab-ci.yml` +file: + +```yaml +include: + template: SAST.gitlab-ci.yml +``` + +The included template will create a `sast` job in your CI/CD pipeline and scan +your project's source code for possible vulnerabilities. + +The report will be saved as a +[SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast-ultimate) +that you can later download and analyze. Due to implementation limitations, we +always take the latest SAST artifact available. Behind the scenes, the +[GitLab SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) +is used to detect the languages/frameworks and in turn runs the matching scan tools. + +#### Customizing the SAST settings + +The SAST settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. +These variables are documented in the +[SAST tool documentation](https://gitlab.com/gitlab-org/security-products/sast#settings). + +In the following example, we include the SAST template and at the same time we +set the `SAST_GOSEC_LEVEL` variable to `2`: + +```yaml +include: + template: SAST.gitlab-ci.yml + +variables: + SAST_GOSEC_LEVEL: 2 +``` + +Because the template is [evaluated before](../../../ci/yaml/README.md#include) +the pipeline configuration, the last mention of the variable will take precedence. + +#### Overriding the SAST template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `sast` job after the +template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: SAST.gitlab-ci.yml + +sast: + variables: + CI_DEBUG_TRACE: "true" +``` + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `sast` +job can be added: + +```yaml +sast: + stage: test + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SAST_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - | + docker run \ + --env SAST_ANALYZER_IMAGES \ + --env SAST_ANALYZER_IMAGE_PREFIX \ + --env SAST_ANALYZER_IMAGE_TAG \ + --env SAST_DEFAULT_ANALYZERS \ + --env SAST_EXCLUDED_PATHS \ + --env SAST_BANDIT_EXCLUDED_PATHS \ + --env SAST_BRAKEMAN_LEVEL \ + --env SAST_GOSEC_LEVEL \ + --env SAST_FLAWFINDER_LEVEL \ + --env SAST_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ + --env SAST_PULL_ANALYZER_IMAGE_TIMEOUT \ + --env SAST_RUN_ANALYZER_TIMEOUT \ + --volume "$PWD:/code" \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + "registry.gitlab.com/gitlab-org/security-products/sast:$SAST_VERSION" /app/bin/run /code + dependencies: [] + artifacts: + reports: + sast: gl-sast-report.json +``` + +You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/sast#settings) +via `docker run --env` to customize your job execution. + +## Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Deprecated:** +Before GitLab 11.5, the SAST job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly +advised to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the SAST job should look like: + +```yaml +sast: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SAST_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - docker run + --env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}" + --volume "$PWD:/code" + --volume /var/run/docker.sock:/var/run/docker.sock + "registry.gitlab.com/gitlab-org/security-products/sast:$SAST_VERSION" /app/bin/run /code + artifacts: + paths: [gl-sast-report.json] +``` + +## Secret detection + +GitLab is also able to detect secrets and credentials that have been unintentionally pushed to the repository. +For example, an API key that allows write access to third-party deployment environments. + +This check is performed by a specific analyzer during the `sast` job. It runs regardless of the programming +language of your app, and you don't need to change anything to your +CI/CD configuration file to turn it on. Results are available in the SAST report. + +GitLab currently includes [Gitleaks](https://github.com/zricethezav/gitleaks) and [TruffleHog](https://github.com/dxa4481/truffleHog) checks. + +## Security report under pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3776) +in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.6. + +Visit any pipeline page which has a `sast` job and you will be able to see +the security report tab with the listed vulnerabilities (if any). + + + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). diff --git a/doc/user/application_security/security_dashboard/img/dashboard.png b/doc/user/application_security/security_dashboard/img/dashboard.png Binary files differnew file mode 100644 index 00000000000..d52a6dacdbf --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/dashboard.png diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard.png Binary files differnew file mode 100644 index 00000000000..3294e59e943 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/project_security_dashboard.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md new file mode 100644 index 00000000000..ca31e15c65f --- /dev/null +++ b/doc/user/application_security/security_dashboard/index.md @@ -0,0 +1,103 @@ +# GitLab Security Dashboard **[ULTIMATE]** + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. + +You can also drill down into a vulnerability and get extra information, see which +project it comes from, the file it's in, and various metadata to help you analyze +the risk. You can also action these vulnerabilities by creating an issue for them, +or by dismissing them. + +To benefit from the Security Dashboard you must first configure one of the +[security reports](../index.md). + +## Supported reports + +The Security Dashboard supports the following reports: + +- [Container Scanning](../container_scanning/index.md) +- [DAST](../dast/index.md) +- [Dependency Scanning](../dependency_scanning/index.md) +- [SAST](../sast/index.md) + +## Requirements + +To use the project or group security dashboard: + +1. At least one project inside a group must be configured with at least one of + the [supported reports](#supported-reports). +2. The configured jobs must use the [new `reports` syntax](../../../ci/yaml/README.md#artifactsreports). +3. [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 or newer must be used. + If you're using the shared Runners on GitLab.com, this is already the case. + +## Project Security Dashboard + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6165) in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.1. + +At the project level, the Security Dashboard displays the latest security reports +for your project. Use it to find and fix vulnerabilities affecting the +[default branch](../../project/repository/branches/index.md#default-branch). + + + +## Group Security Dashboard + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6709) in + [GitLab Ultimate](https://about.gitlab.com/pricing) 11.5. + +The group Security Dashboard gives an overview of the vulnerabilities of all the +projects in a group and its subgroups. + +First, navigate to the Security Dashboard found under your group's +**Overview > Security Dashboard**. + +Once you're on the dashboard, at the top you should see a series of filters for: + +- Severity +- Report type +- Project + + + +Selecting one or more filters will filter the results in this page. +The first section is an overview of all the vulnerabilities, grouped by severity. +Underneath this overview is a timeline chart that shows how many open +vulnerabilities your projects had at various points in time. You can filter among 30, 60, and +90 days, with the default being 90. Hover over the chart to get more details about +the open vulnerabilities at a specific time. + +Finally, there is a list of all the vulnerabilities in the group, sorted by severity. +In that list, you can see the severity of the vulnerability, its name, its +confidence (likelihood of the vulnerability to be a positive one), and the project +it's from. + +If you hover over a row, there will appear some actions you can take: + +- "More info" +- "Create issue" +- "Dismiss vulnerability" + +Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Keeping the dashboards up to date + +The Security Dashboard displays information from the results of the most recent +security scan on the [default branch](../../project/repository/branches/index.md#default-branch), +which means that security scans are performed every time the branch is updated. + +If the default branch is updated infrequently, scans are run infrequently and the +information on the Security Dashboard can become outdated as new vulnerabilities +are discovered. + +To ensure the information on the Security Dashboard is regularly updated, +[configure a scheduled pipeline](../../project/pipelines/schedules.md) to run a +daily security scan. This will update the information displayed on the Security +Dashboard regardless of how often the default branch is updated. + +That way, reports are created even if no code change happens. + +## Security scans using Auto DevOps + +When using [Auto DevOps](../../../topics/autodevops/index.md), use +[special environment variables](../../../topics/autodevops/index.md#environment-variables) +to configure daily security scans. diff --git a/doc/user/discussions/img/mr_review_resolve.png b/doc/user/discussions/img/mr_review_resolve.png Binary files differnew file mode 100644 index 00000000000..34176f3fa8e --- /dev/null +++ b/doc/user/discussions/img/mr_review_resolve.png diff --git a/doc/user/discussions/img/mr_review_resolve2.png b/doc/user/discussions/img/mr_review_resolve2.png Binary files differnew file mode 100644 index 00000000000..e4adb5f2c2d --- /dev/null +++ b/doc/user/discussions/img/mr_review_resolve2.png diff --git a/doc/user/discussions/img/mr_review_second_comment.png b/doc/user/discussions/img/mr_review_second_comment.png Binary files differnew file mode 100644 index 00000000000..920ea05ad66 --- /dev/null +++ b/doc/user/discussions/img/mr_review_second_comment.png diff --git a/doc/user/discussions/img/mr_review_second_comment_added.png b/doc/user/discussions/img/mr_review_second_comment_added.png Binary files differnew file mode 100644 index 00000000000..1fb54348547 --- /dev/null +++ b/doc/user/discussions/img/mr_review_second_comment_added.png diff --git a/doc/user/discussions/img/mr_review_start.png b/doc/user/discussions/img/mr_review_start.png Binary files differnew file mode 100644 index 00000000000..38b44bda0d2 --- /dev/null +++ b/doc/user/discussions/img/mr_review_start.png diff --git a/doc/user/discussions/img/mr_review_unresolve.png b/doc/user/discussions/img/mr_review_unresolve.png Binary files differnew file mode 100644 index 00000000000..da895ceb89f --- /dev/null +++ b/doc/user/discussions/img/mr_review_unresolve.png diff --git a/doc/user/discussions/img/mr_review_unresolve2.png b/doc/user/discussions/img/mr_review_unresolve2.png Binary files differnew file mode 100644 index 00000000000..a824b806e4a --- /dev/null +++ b/doc/user/discussions/img/mr_review_unresolve2.png diff --git a/doc/user/discussions/img/pending_review_comment.png b/doc/user/discussions/img/pending_review_comment.png Binary files differnew file mode 100644 index 00000000000..916ef5b7452 --- /dev/null +++ b/doc/user/discussions/img/pending_review_comment.png diff --git a/doc/user/discussions/img/review_comment_quickactions.png b/doc/user/discussions/img/review_comment_quickactions.png Binary files differnew file mode 100644 index 00000000000..bd9880c329a --- /dev/null +++ b/doc/user/discussions/img/review_comment_quickactions.png diff --git a/doc/user/discussions/img/review_preview.png b/doc/user/discussions/img/review_preview.png Binary files differnew file mode 100644 index 00000000000..4bf53a81b9c --- /dev/null +++ b/doc/user/discussions/img/review_preview.png diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index c7e8bb5b33b..248f8395db1 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -5,6 +5,7 @@ The ability to contribute conversationally is offered throughout GitLab. You can leave a comment in the following places: - issues +- epics **[ULTIMATE]** - merge requests - snippets - commits @@ -279,6 +280,75 @@ edit existing comments. Non-team members are restricted from adding or editing c Additionally, locked issues and merge requests can not be reopened. +## Merge Request Reviews **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4213) in GitLab 11.4. + +When looking at a Merge Request diff, you are able to start a review. +This allows you to create comments inside a Merge Request that are **only visible to you** until published, +in order to allow you to submit them all as a single action. + +### Starting a review + +In order to start a review, simply write a comment on a diff as normal under the **Changes** tab +in an MR and click on the **Start a review** button. + + + +Once a review is started, you will see any comments that are part of this review marked `Pending`. +All comments that are part of a review show two buttons: + +- **Submit review**: Submits all comments that are part of the review, making them visible to other users. +- **Add comment now**: Submits the specific comment as a regular comment instead of as part of the review. + + + +You can use [quick actions] inside review comments. The comment will show the actions that will be performed once published. + + + +To add more comments to a review, start writing a comment as normal and click the **Add to review** button. + + + +This will add the comment to the review. + + + +### Resolving/Unresolving discussions + +Review comments can also resolve/unresolve [resolvable discussions](#resolvable-comments-and-discussions). +When replying to a comment, you will see a checkbox that you can click in order to resolve or unresolve +the discussion once published. + + + + +If a particular pending comment will resolve or unresolve the discussion, this will be shown on the pending +comment itself. + + + + +### Submitting a review + +If you have any comments that have not been submitted, you will see a bar at the +bottom of the screen with two buttons: + +- **Discard**: Discards all comments that have not been submitted. +- **Finish review**: Opens a list of comments ready to be submitted for review. + Clicking **Submit review** will publish all comments. Any quick actions + submitted are performed at this time. + +Alternatively, every pending comment has a button to finish the entire review. + + + +Submitting the review will send a single email to every notifiable user of the +merge request with all the comments associated to it. + +Replying to this email will, consequentially, create a new comment on the associated merge request. + ## Filtering notes > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/26723) in GitLab 11.5. diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 0f71587830f..ae09453ce52 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -111,7 +111,7 @@ The domain should have a wildcard DNS configured to the Ingress IP address. When adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the -[environment-specific variables](https://docs.gitlab.com/ee/ci/variables/README.html#limiting-environment-scopes-of-variables-premium) +[environment-specific variables](https://docs.gitlab.com/ee/ci/variables/#limiting-environment-scopes-of-environment-variables-premium) work. While evaluating which environment matches the environment scope of a diff --git a/doc/user/group/contribution_analytics/img/group_stats_cal.png b/doc/user/group/contribution_analytics/img/group_stats_cal.png Binary files differnew file mode 100644 index 00000000000..0238c7bf088 --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_cal.png diff --git a/doc/user/group/contribution_analytics/img/group_stats_graph.png b/doc/user/group/contribution_analytics/img/group_stats_graph.png Binary files differnew file mode 100644 index 00000000000..ccfd3782c6f --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_graph.png diff --git a/doc/user/group/contribution_analytics/img/group_stats_table.png b/doc/user/group/contribution_analytics/img/group_stats_table.png Binary files differnew file mode 100644 index 00000000000..f1d1031fa18 --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_table.png diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md new file mode 100644 index 00000000000..bc88eff9ed2 --- /dev/null +++ b/doc/user/group/contribution_analytics/index.md @@ -0,0 +1,55 @@ +# Contribution Analytics **[STARTER]** + +>**Note:** +Introduced in [GitLab Starter][ee] 8.3. + +Track your team members' activity across your organization. + +## Overview + +With Contribution Analytics you can get an overview of the activity of +issues, merge requests, and push events of your organization and its members. + +The analytics page is located at **Group > Contribution Analytics** +under the URL `/groups/<groupname>/analytics`. + +## Use cases + +- Analyze your team's contributions over a period of time and offer a bonus for the top contributors +- Identify opportunities for improvement with group members who may benefit from additional support + +## Using Contribution Analytics + +There are three main bar graphs that are deducted from the number of +contributions per group member. These contributions include push events, merge +requests and closed issues. Hovering on each bar you can see the number of +events for a specific member. + + + +## Changing the period time + +There are three periods you can choose from, 'Last week', 'Last month' and +'Last three months'. The default is 'Last week'. + +You can choose which period to display by using the dropdown calendar menu in +the upper right corner. + + + +## Sorting by different factors + +Apart from the bar graphs you can also see the contributions per group member +which are depicted in a table that can be sorted by: + +* Member name +* Number of pushed events +* Number of opened issues +* Number of closed issues +* Number of opened MRs +* Number of accepted MRs +* Number of total contributions + + + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/group/epics/img/button_close_epic.png b/doc/user/group/epics/img/button_close_epic.png Binary files differnew file mode 100644 index 00000000000..aa1a889ea23 --- /dev/null +++ b/doc/user/group/epics/img/button_close_epic.png diff --git a/doc/user/group/epics/img/button_reopen_epic.png b/doc/user/group/epics/img/button_reopen_epic.png Binary files differnew file mode 100644 index 00000000000..7a953189270 --- /dev/null +++ b/doc/user/group/epics/img/button_reopen_epic.png diff --git a/doc/user/group/epics/img/child_epics_roadmap.png b/doc/user/group/epics/img/child_epics_roadmap.png Binary files differnew file mode 100644 index 00000000000..819fed58989 --- /dev/null +++ b/doc/user/group/epics/img/child_epics_roadmap.png diff --git a/doc/user/group/epics/img/containing_epic.png b/doc/user/group/epics/img/containing_epic.png Binary files differnew file mode 100644 index 00000000000..5ed693e6d6a --- /dev/null +++ b/doc/user/group/epics/img/containing_epic.png diff --git a/doc/user/group/epics/img/epic_view.png b/doc/user/group/epics/img/epic_view.png Binary files differnew file mode 100644 index 00000000000..dbda98e4351 --- /dev/null +++ b/doc/user/group/epics/img/epic_view.png diff --git a/doc/user/group/epics/img/epics_list_view.png b/doc/user/group/epics/img/epics_list_view.png Binary files differnew file mode 100644 index 00000000000..b30608d9d31 --- /dev/null +++ b/doc/user/group/epics/img/epics_list_view.png diff --git a/doc/user/group/epics/img/epics_search.png b/doc/user/group/epics/img/epics_search.png Binary files differnew file mode 100644 index 00000000000..96335527468 --- /dev/null +++ b/doc/user/group/epics/img/epics_search.png diff --git a/doc/user/group/epics/img/epics_sort.png b/doc/user/group/epics/img/epics_sort.png Binary files differnew file mode 100644 index 00000000000..b23c65fd0ef --- /dev/null +++ b/doc/user/group/epics/img/epics_sort.png diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md new file mode 100644 index 00000000000..6035f0c7326 --- /dev/null +++ b/doc/user/group/epics/index.md @@ -0,0 +1,217 @@ +# Epics **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 10.2. + +Epics let you manage your portfolio of projects more efficiently and with less +effort by tracking groups of issues that share a theme, across projects and +milestones. + + + +## Use cases + +- Suppose your team is working on a large feature that involves multiple discussions throughout different issues created in distinct projects within a [Group](../index.md). With Epics, you can track all the related activities that together contribute to that single feature. +- Track when the work for the group of issues is targeted to begin, and when it is targeted to end. +- Discuss and collaborate on feature ideas and scope at a high-level. + +## Creating an epic + +A paginated list of epics is available in each group from where you can create +a new epic. The list of epics includes also epics from all subgroups of the +selected group. From your group page: + +1. Go to **Epics** +1. Click the **New epic** button at the top right +1. Enter a descriptive title and hit **Create epic** + +Once created, you will be taken to the view for that newly-created epic where +you can change its title, description, start date, and due date. + + + +## Adding an issue to an epic + +An epic contains a list of issues and an issue can be associated with at most +one epic. When on an epic, you can add its associated issues: + +1. Click the plus icon (<kbd>+</kbd>) under the epic description. +1. Paste the link of the issue (you can hit <kbd>Spacebar</kbd> to add more than + one issues at a time). +1. Click **Add**. + +Any issue belonging to a project in the epic's group or any of the epic's +subgroups are eligible to be added. To remove an issue from an epic, click +on the <kbd>x</kbd> button in the epic's issue list. + +NOTE: **Note:** +When you add an issue or an epic to an epic that's already associated with another epic, +the issue or the epic is automatically removed from the previous epic. + +## Multi-level child epics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8333) in GitLab Ultimate 11.7. + +Much like adding issues to an epic, an epic can have multiple child epics with +the maximum depth being 5. To add a child epic: + +1. Click the plus icon (<kbd>+</kbd>) under the epic description. +1. Paste the link of the epic. +1. Click **Add**. + +Any epic that belongs to a group or subgroup of the parent epic's group is +eligible to be added. To remove a child epic from a parent epic, +click on the <kbd>x</kbd> button in the parent epic's epic list. + +## Start date and due date + +For each of the dates in the sidebar of an epic, you can choose to either: + +- Enter a fixed value. +- Inherit a dynamic value called "From milestones". + +If you select "From milestones" for the start date, GitLab will automatically set the +date to be earliest start date across all milestones that are currently assigned +to the issues that are attached to the epic. Similarly, if you select "From milestones" +for the due date, GitLab will set it to be the latest due date across all +milestones that are currently assigned to those issues. + +These are dynamic dates in that if milestones are re-assigned to the issues, if the +milestone dates change, or if issues are added or removed from the epic, then +the re-calculation will happen immediately to set a new dynamic date. + +## Roadmap in epics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7327) in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.10. + +If your epic contains one or more [child epics](#multi-level-child-epics) which +have a [start or due date](#start-date-and-due-date), then you can see a +[roadmap](../roadmap/index.md) view of the child epics under the parent epic itself. + + + +## Reordering issues and child epics + +Drag and drop to reorder issues and child epics. New issues and child epics added to an epic appear at the top of the list. + +## Deleting an epic + +NOTE: **Note:** +To delete an epic, you need to be an [Owner][permissions] of a group/subgroup. + +When inside a single epic view, click the **Delete** button to delete the epic. +A modal will pop-up to confirm your action. + +Deleting an epic releases all existing issues from their associated epic in the +system. + +## Closing and reopening epics + +### Using buttons + +Whenever you decide that there is no longer need for that epic, +close the epic using the close button: + + + +You can always reopen it using the reopen button. + + + +### Using quick actions + +You can close or reopen an epic using [Quick actions](../../project/quick_actions.md) + +## Navigating to an epic from an issue + +If an issue belongs to an epic, you can navigate to the containing epic with the +link in the issue sidebar. + + + +## Promoting an issue to an epic + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3777) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.6. + +If you have [permissions](../../permissions.md) to close an issue and create an +epic in the parent group, you can promote an issue to an epic with the `/promote` +[quick action](../../project/quick_actions.md#quick-actions-for-epics-ultimate). +Only issues from projects that are in groups can be promoted. + +When the quick action is executed: + +- An epic is created in the same group as the project of the issue. +- Subscribers of the issue are notified that the epic was created. + +The following issue metadata will be copied to the epic: + +- Title, description, activity/comment thread. +- Upvotes/downvotes. +- Participants. +- Group labels that the issue already has. + +## Searching for an epic from epics list page + +> Introduced in [GitLab Ultimate][ee] 10.5. + +You can search for an epic from the list of epics using filtered search bar (similar to +that of Issues and Merge requests) based on following parameters: + +- Title or description +- Author name / username +- Labels + + + +To search, go to the list of epics and click on the field **Search or filter results...**. +It will display a dropdown menu, from which you can add an author. You can also enter plain +text to search by epic title or description. When done, press <kbd>Enter</kbd> on your +keyboard to filter the list. + +You can also sort epics list by: + +- **Created date** +- **Last updated** +- **Start date** +- **Due date** + +Each option contains a button that can toggle the order between **ascending** and **descending**. The sort option and order will be persisted to be used wherever epics are browsed including the [roadmap](../roadmap/index.md). + + + +## Permissions + +If you have access to view an epic and have access to view an issue already +added to that epic, then you can view the issue in the epic issue list. + +If you have access to edit an epic and have access to edit an issue, then you +can add the issue to or remove it from the epic. + +Note that for a given group, the visibility of all projects must be the same as +the group, or less restrictive. That means if you have access to a group's epic, +then you already have access to its projects' issues. + +You may also consult the [group permissions table][permissions]. + +[ee]: https://about.gitlab.com/pricing/ +[permissions]: ../../permissions.md#group-members-permissions + +## Thread + +- Comments: collaborate on that epic by posting comments in its thread. +These text fields also fully support +[GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). + +## Comment, or start a discussion + +Once you wrote your comment, you can either: + +- Click "Comment" and your comment will be published. +- Click "Start discussion": start a thread within that epic's thread to discuss specific points. + +## Award emoji + +- You can [award an emoji](../../award_emojis.md) to that epic or its comments. + +## Notifications + +- [Receive notifications](../../../workflow/notifications.md) for epic events. diff --git a/doc/user/group/img/group_file_template_dropdown.png b/doc/user/group/img/group_file_template_dropdown.png Binary files differnew file mode 100644 index 00000000000..d9caae1a223 --- /dev/null +++ b/doc/user/group/img/group_file_template_dropdown.png diff --git a/doc/user/group/img/group_file_template_settings.png b/doc/user/group/img/group_file_template_settings.png Binary files differnew file mode 100644 index 00000000000..ca42f7726db --- /dev/null +++ b/doc/user/group/img/group_file_template_settings.png diff --git a/doc/user/group/img/group_issue_board.png b/doc/user/group/img/group_issue_board.png Binary files differnew file mode 100644 index 00000000000..a0da74a320f --- /dev/null +++ b/doc/user/group/img/group_issue_board.png diff --git a/doc/user/group/img/member_lock.png b/doc/user/group/img/member_lock.png Binary files differnew file mode 100644 index 00000000000..3f1382e76c6 --- /dev/null +++ b/doc/user/group/img/member_lock.png diff --git a/doc/user/group/img/membership_lock.png b/doc/user/group/img/membership_lock.png Binary files differdeleted file mode 100644 index c9ad82c90f2..00000000000 --- a/doc/user/group/img/membership_lock.png +++ /dev/null diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 2d887673fd6..06564fd6cd1 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -5,10 +5,12 @@ and grant members access to several projects at once. Groups can also be nested in [subgroups](subgroups/index.md). -Find your groups by expanding the left menu and clicking **Groups**: +Find your groups by clicking **Groups** in the top navigation.  +> The groups dropdown in the top navigation was [introduced][ce-36234] in [GitLab 11.1](https://about.gitlab.com/2018/07/22/gitlab-11-1-released/#groups-dropdown-in-navigation). + The Groups page displays all groups you are a member of, how many projects it holds, how many members it has, the group visibility, and, if you have enough permissions, a link to the group settings. By clicking the last button you can leave that group. @@ -44,7 +46,7 @@ For example, consider a user named Alex: 1. Alex creates an account on GitLab.com with the username `alex`; their profile will be accessed under `https://gitlab.example.com/alex` -1. Alex creates a group for their team with the groupname `alex-team`; +1. Alex creates a group for their team with the group name `alex-team`; the group and its projects will be accessed under `https://gitlab.example.com/alex-team` 1. Alex creates a subgroup of `alex-team` with the subgroup name `marketing`; this subgroup and its projects will be accessed under `https://gitlab.example.com/alex-team/marketing` @@ -160,6 +162,10 @@ There are two different ways to add a new project to a group: ### Default project creation level +> [Introduced][ee-2534] in [GitLab Premium][ee] 10.5. +> Brought to [GitLab Starter][ee] in 10.7. +> [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25975) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.10. + Group owners or administrators can allow users with the Developer role to create projects under groups. @@ -185,6 +191,32 @@ Alternatively, you can [lock the sharing with group feature](#share-with-group-l In GitLab Enterprise Edition it is possible to manage GitLab group memberships using LDAP groups. See [the GitLab Enterprise Edition documentation](../../integration/ldap.md) for more information. +## Epics **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 10.2. + +Epics let you manage your portfolio of projects more efficiently and with less +effort by tracking groups of issues that share a theme, across projects and +milestones. + +[Learn more about Epics.](epics/index.md) + +## Group Security Dashboard **[ULTIMATE]** + +Get an overview of the vulnerabilities of all the projects in a group and its subgroups. + +[Learn more about the Group Security Dashboard.](security_dashboard/index.md) + +## Insights **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 11.9 behind the `insights` feature flag. + +Configure the Insights that matter for your groups or projects to explore data +such as triage hygiene, issues created/closed per a given period, average time +for merge requests to be merged and much more. + +[Learn more about Insights](insights/index.md). + ## Transferring groups From GitLab 10.5, groups can be transferred in the following ways: @@ -257,7 +289,7 @@ working together in a project. To inherit the group membership, you share the project between the two groups A and B. **Share with group lock** prevents any project within the group from being shared with another group. By doing so, you -guarantee only the right group members have access to that projects. +guarantee only the right group members have access to those projects. To enable this feature, navigate to the group settings page. Select **Share with group lock** and **Save the group**. @@ -266,17 +298,50 @@ To enable this feature, navigate to the group settings page. Select #### Member Lock **[STARTER]** -With **Member Lock** it is possible to lock membership in project to the -level of members in group. +With Member lock, it is possible to lock membership in a project to the +level of members in the group. + +Member lock lets a group owner lock down any new project membership to all the +projects within the group, allowing tighter control over project membership. -Learn more about [Member Lock](https://docs.gitlab.com/ee/user/group/index.html#member-lock). +For instance, if you want to lock the group for an [Audit Event](https://docs.gitlab.com/ee/administration/audit_events.html), +you enable Member lock to guarantee that membership of a project cannot be modified during that audit. -#### Group-level file templates **[PREMIUM]** +To enable this feature: -Group-level file templates allow you to share a set of templates for common file -types with every project in a group. +1. Navigate to the group's **Settings > General** page. +1. Expand the **Permissions, LFS, 2FA** section and select **Member lock**. +1. Click the **Save changes** button. -Learn more about [Group-level file templates](https://docs.gitlab.com/ee/user/group/index.html#group-level-file-templates-premium). + + +This will disable the option for all users who previously had permissions to +operate project memberships so no new users can be added. Furthermore, any +request to add a new user to a project through API will not be possible. + +#### Group file templates **[PREMIUM]** + +Group file templates allow you to share a set of templates for common file +types with every project in a group. It is analogous to the +[instance template repository](https://docs.gitlab.com/ee/user/admin_area/settings/instance_template_repository.html) +feature, and the selected project should follow the same naming conventions as +are documented on that page. + +Only projects that are in the group may be chosen as the source of templates. +This includes projects shared with the group, but **excludes** projects in +subgroups or parent groups of the group being configured. + +This feature may be configured for subgroups as well as parent groups. A project +in a subgroup will have access to the templates for that subgroup, as well as +any parent groups. + + + +To enable this feature, navigate to the group settings page, expand the +**Templates** section, choose a project to act as the template repository, and +**Save group**. + + #### Group-level project templates **[PREMIUM]** @@ -289,6 +354,19 @@ Define project templates at a group-level by setting a group as a template sourc access each project's settings, and remove any project from the same screen. - **Webhooks**: configure [webhooks](../project/integrations/webhooks.md) to your group. - **Kubernetes cluster integration**: connect your GitLab group with [Kubernetes clusters](clusters/index.md). -- **Audit Events**: view [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html#audit-events) +- **Audit Events**: view [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html) for the group. **[STARTER ONLY]** - **Pipelines quota**: keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group. + +## User contribution analysis **[STARTER]** + +With [GitLab Contribution Analytics](contribution_analytics/index.md) +you have an overview of the contributions (pushes, merge requests, +and issues) performed by your group members. + +## Issues analytics **[PREMIUM]** + +With [GitLab Issues Analytics](issues_analytics/index.md), in groups, you can see a bar chart of the number of issues created each month. + +[ee]: https://about.gitlab.com/pricing/ +[ee-2534]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2534 diff --git a/doc/user/group/insights/img/insights_example_stacked_bar_chart.png b/doc/user/group/insights/img/insights_example_stacked_bar_chart.png Binary files differnew file mode 100644 index 00000000000..791d0e4bcdf --- /dev/null +++ b/doc/user/group/insights/img/insights_example_stacked_bar_chart.png diff --git a/doc/user/group/insights/img/insights_group_configuration.png b/doc/user/group/insights/img/insights_group_configuration.png Binary files differnew file mode 100644 index 00000000000..0af0073e448 --- /dev/null +++ b/doc/user/group/insights/img/insights_group_configuration.png diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md new file mode 100644 index 00000000000..5283d8da77d --- /dev/null +++ b/doc/user/group/insights/index.md @@ -0,0 +1,39 @@ +# Insights **[ULTIMATE]** + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. + +CAUTION: **Beta:** +Insights is considered beta, and is not ready for production use. +Follow [gitlab-org&725](https://gitlab.com/groups/gitlab-org/-/epics/725) for +updates. + +Configure the Insights that matter for your groups to explore data such as +triage hygiene, issues created/closed per a given period, average time for merge +requests to be merged and much more. + + + +## Configure your Insights + +Navigate to your group's **Settings > General**, expand **Insights**, and choose +the project that holds your `.gitlab/insights.yml` configuration file: + + + +If no configuration was set, a [default configuration file]( +https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/fixtures/insights/ee/fixtures/insights/default.yml) +will be used. + +See the [Project's Insights documentation](https://docs.gitlab.com/ee/user/project/insights/index.html) for +more details about the `.gitlab/insights.yml` configuration file. + +## Permissions + +If you have access to view a group, then you have access to view their Insights. + +NOTE: **Note:** +Issues or merge requests that you don't have access to (because you don't have +access to the project they belong to, or because they are confidential) are +filtered out of the Insights charts. + +You may also consult the [group permissions table](../../permissions.md#group-members-permissions). diff --git a/doc/user/group/issues_analytics/img/issues_created_per_month.png b/doc/user/group/issues_analytics/img/issues_created_per_month.png Binary files differnew file mode 100644 index 00000000000..96f0d36c917 --- /dev/null +++ b/doc/user/group/issues_analytics/img/issues_created_per_month.png diff --git a/doc/user/group/issues_analytics/index.md b/doc/user/group/issues_analytics/index.md new file mode 100644 index 00000000000..cf53d7423a6 --- /dev/null +++ b/doc/user/group/issues_analytics/index.md @@ -0,0 +1,16 @@ +# Issues Analytics **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7478) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.5. + +GitLab by default displays a bar chart of the number of issues created each month, for the +current month, and 12 months prior, for a total of 13 months. + +You can change the total number of months displayed by setting a URL parameter. +For example, `https://gitlab.com/groups/gitlab-org/-/issues_analytics?months_back=15` +would show a total of 15 months for the chart in the GitLab.org group. + +The **Search or filter results...** field can be used for filtering the issues by any attribute. For example, labels, assignee, milestone, and author. + +To access the chart, navigate to a group's sidebar and select **Issues > Analytics**. + + diff --git a/doc/user/group/roadmap/img/epics_state_dropdown.png b/doc/user/group/roadmap/img/epics_state_dropdown.png Binary files differnew file mode 100644 index 00000000000..cbcc3658a54 --- /dev/null +++ b/doc/user/group/roadmap/img/epics_state_dropdown.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_months.png b/doc/user/group/roadmap/img/roadmap_timeline_months.png Binary files differnew file mode 100644 index 00000000000..5a046b21507 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_months.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_quarters.png b/doc/user/group/roadmap/img/roadmap_timeline_quarters.png Binary files differnew file mode 100644 index 00000000000..56f428cb471 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_quarters.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_weeks.png b/doc/user/group/roadmap/img/roadmap_timeline_weeks.png Binary files differnew file mode 100644 index 00000000000..bf4c1dc0284 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_weeks.png diff --git a/doc/user/group/roadmap/img/roadmap_view.png b/doc/user/group/roadmap/img/roadmap_view.png Binary files differnew file mode 100644 index 00000000000..ff41a2e0441 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_view.png diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md new file mode 100644 index 00000000000..310c4bb88d0 --- /dev/null +++ b/doc/user/group/roadmap/index.md @@ -0,0 +1,64 @@ +# Roadmap **[ULTIMATE]** + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.5. + +An Epic within a group containing **Start date** and/or **Due date** +can be visualized in a form of a timeline (e.g. a Gantt chart). The Epics Roadmap page +shows such a visualization for all the epics which are under a group and/or its subgroups. + + + +A dropdown allows you to show only open or closed epics. By default, all epics are shown. + + + +Epics in the view can be sorted by: + +- **Created date** +- **Last updated** +- **Start date** +- **Due date** + +Each option contains a button that can toggle the order between **ascending** and **descending**. The sort option and order will be persisted to be used wherever epics are browsed including the [epics list view](../epics/index.md). + +Roadmaps can also be [visualized inside an epic](../epics/index.md#roadmap-in-epics). + +## Timeline duration + +Starting with [GitLab Ultimate][ee] 11.0, Roadmap supports three different date ranges; Quarters, Months (Default) and Weeks. + +### Quarters + + + +In _Quarters_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past quarter**, **current quarter** and **next 4 quarters**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the quarter name on +the timeline header represent the month of the quarter. + +### Months + + + +In _Months_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past month**, **current month** and **next 5 months**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the month name on +the timeline header represent the date on starting day (Sunday) of the week. This preset is +selected by default. + +### Weeks + + + +In _Weeks_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past week**, **current week** and **next 4 weeks**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the week name on +the timeline header represent the days of the week. + +## Timeline bar for an epic + +The timeline bar indicates the approximate position of an epic based on its start +and due date. If an epic doesn't have a due date, the timeline bar fades +away towards the future. Similarly, if an epic doesn't have a start date, the +timeline bar becomes more visible as it approaches the epic's due date on the +timeline. diff --git a/doc/user/group/saml_sso/img/group_saml_configuration_information.png b/doc/user/group/saml_sso/img/group_saml_configuration_information.png Binary files differnew file mode 100644 index 00000000000..98b83d0cb0f --- /dev/null +++ b/doc/user/group/saml_sso/img/group_saml_configuration_information.png diff --git a/doc/user/group/saml_sso/img/group_saml_settings.png b/doc/user/group/saml_sso/img/group_saml_settings.png Binary files differnew file mode 100644 index 00000000000..d95acb5075f --- /dev/null +++ b/doc/user/group/saml_sso/img/group_saml_settings.png diff --git a/doc/user/group/saml_sso/img/scim_advanced.png b/doc/user/group/saml_sso/img/scim_advanced.png Binary files differnew file mode 100644 index 00000000000..3b70e3fbe83 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_advanced.png diff --git a/doc/user/group/saml_sso/img/scim_attribute_mapping.png b/doc/user/group/saml_sso/img/scim_attribute_mapping.png Binary files differnew file mode 100644 index 00000000000..c9f6b71f5b0 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_attribute_mapping.png diff --git a/doc/user/group/saml_sso/img/scim_token.png b/doc/user/group/saml_sso/img/scim_token.png Binary files differnew file mode 100644 index 00000000000..7eb52bf6ea2 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_token.png diff --git a/doc/user/group/saml_sso/img/unlink_group_saml.png b/doc/user/group/saml_sso/img/unlink_group_saml.png Binary files differnew file mode 100644 index 00000000000..0561443b5f4 --- /dev/null +++ b/doc/user/group/saml_sso/img/unlink_group_saml.png diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md new file mode 100644 index 00000000000..ee3137d032e --- /dev/null +++ b/doc/user/group/saml_sso/index.md @@ -0,0 +1,91 @@ +# SAML SSO for GitLab.com Groups **[SILVER ONLY]** + +> Introduced in [GitLab.com Silver](https://about.gitlab.com/pricing/) 11.0. + +NOTE: **Note:** +This topic is for SAML on GitLab.com Silver tier and above. For SAML on self-managed GitLab instances, see [SAML OmniAuth Provider](../../../integration/saml.md). + +Currently SAML on GitLab.com can be used to automatically add users to a group, and does not yet sign users into GitLab.com. Users should already have an account on the GitLab instance, or can create one when logging in for the first time. + +User synchronization for GitLab.com is partially supported using [SCIM](scim_setup.md). + +NOTE: **Note:** +SAML SSO for groups is used only as a convenient way to add users and does not sync users between providers without using SCIM. If a group is not using SCIM, group Owners will still need to manage user accounts, such as removing users when necessary. + +## Configuring your Identity Provider + +1. Navigate to the group and click **Settings > SAML SSO**. +1. Configure your SAML server using the **Assertion consumer service URL** and **Issuer**. See [your identity provider's documentation](#providers) for more details. +1. Configure the SAML response to include a NameID that uniquely identifies each user. +1. Configure required assertions using the [table below](#assertions). +1. Once the identity provider is set up, move on to [configuring GitLab](#configuring-gitlab). + + + +NOTE: **Note:** +Partial SSO enforcement was introduced in [11.8](https://gitlab.com/gitlab-org/gitlab-ee/issues/5291). With this option enabled, users must use your group's GitLab single sign on URL to be added to the group or be added via SCIM. Users can no longer be added manually. After a user has been added to the group, GitLab does not continue to enforce the use of SSO, but we'll [add a persistent check](https://gitlab.com/gitlab-org/gitlab-ee/issues/9255) in a later version. + +### NameID + +GitLab.com uses the SAML NameID to identify users. The NameID element: + +- Is a required field in the SAML response. +- Must be unique to each user. +- Must be a persistent value that will never change, such as a unique ID or username. Email could also be used as the NameID, but only if it can be guaranteed to never change. + +### Assertions + +| Field | Supported keys | Notes | +|-|----------------|-------------| +| Email | `email`, `mail` | (required) | +| Full Name | `name` | | +| First Name | `first_name`, `firstname`, `firstName` | | +| Last Name | `last_name`, `lastname`, `lastName` | | + +## Configuring GitLab + +Once you've set up your identity provider to work with GitLab, you'll need to configure GitLab to use it for authentication: + +1. Navigate to the group's **Settings > SAML SSO**. +1. Find the SSO URL from your Identity Provider and enter it the **Identity provider single sign on URL** field. +1. Find and enter the fingerprint for the SAML token signing certificate in the **Certificate** field. +1. Check the **Enable SAML authentication for this group** checkbox. +1. Click the **Save changes** button. + + + +## Providers + +| Provider | Documentation | +|----------|---------------| +| ADFS (Active Directory Federation Services) | [Create a Relying Party Trust](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) | +| Azure | [Configuring single sign-on to applications](https://docs.microsoft.com/en-us/azure/active-directory/active-directory-saas-custom-apps) | +| Auth0 | [Auth0 as Identity Provider](https://auth0.com/docs/protocols/saml/saml-idp-generic) | +| G Suite | [Set up your own custom SAML application](https://support.google.com/a/answer/6087519?hl=en) | +| JumpCloud | [Single Sign On (SSO) with GitLab](https://support.jumpcloud.com/customer/en/portal/articles/2810701-single-sign-on-sso-with-gitlab) | +| Okta | [Setting up a SAML application in Okta](https://developer.okta.com/standards/SAML/setting_up_a_saml_application_in_okta) | +| OneLogin | [Use the OneLogin SAML Test Connector](https://onelogin.service-now.com/support?id=kb_article&sys_id=93f95543db109700d5505eea4b96198f) | +| Ping Identity | [Add and configure a new SAML application](https://docs.pingidentity.com/bundle/p1_enterpriseConfigSsoSaml_cas/page/enableAppWithoutURL.html) | + +## Unlinking accounts + +Users can unlink SAML for a group from their profile page. This can be helpful if: + +- You no longer want a group to be able to sign you in to GitLab.com. +- Your SAML NameID has changed and so GitLab can no longer find your user. + +For example, to unlink the `MyOrg` account, the following **Disconnect** button will be available under **Profile > Accounts**: + + + +## Glossary + +| Term | Description | +|------|-------------| +| Identity Provider | The service which manages your user identities such as ADFS, Okta, Onelogin or Ping Identity. | +| Service Provider | SAML considers GitLab to be a service provider. | +| Assertion | A piece of information about a user's identity, such as their name or role. Also know as claims or attributes. | +| SSO | Single Sign On. | +| Assertion consumer service URL | The callback on GitLab where users will be redirected after successfully authenticating with the identity provider. | +| Issuer | How GitLab identifies itself to the identity provider. Also known as a "Relying party trust identifier". | +| Certificate fingerprint | Used to confirm that communications over SAML are secure by checking that the server is signing communications with the correct certificate. Also known as a certificate thumbprint. | diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md new file mode 100644 index 00000000000..ec27ec3e336 --- /dev/null +++ b/doc/user/group/saml_sso/scim_setup.md @@ -0,0 +1,102 @@ +# SCIM provisioning using SAML SSO for Groups **[SILVER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9388) in [GitLab.com Silver](https://about.gitlab.com/pricing/) 11.10. + +GitLab's [SCIM API](https://docs.gitlab.com/ee/api/scim.html) implements part of [the RFC7644 protocol](https://tools.ietf.org/html/rfc7644). + +Currently, the following actions are available: + +- CREATE +- UPDATE +- DELETE (deprovisioning) + +The following identity providers are supported: + +- Azure + +## Requirements + +- [Group SSO](index.md) needs to be configured. +- The `scim_group` feature flag must be enabled: + + Run the following commands in a Rails console: + + ```sh + # Omnibus GitLab + gitlab-rails console + + # Installation from source + cd /home/git/gitlab + sudo -u git -H bin/rails console RAILS_ENV=production + ``` + + To enable SCIM for a group named `group_name`: + + ```ruby + group = Group.find_by_full_path('group_name') + Feature.enable(:group_scim, group) + ``` + +### GitLab configuration + +Once [Single sign-on](index.md) has been configured, we can: + +1. Navigate to the group and click **Settings > SAML SSO**. +1. Click on the **Generate a SCIM token** button. +1. Save the token and URL so they can be used in the next step. + + + +## SCIM IdP configuration + +### Configuration on Azure + +In the [Single sign-on](index.md) configuration for the group, make sure +that the **Name identifier value** (NameID) points to a unique identifier, such +as the `user.objectid`. This will match the `extern_uid` used on GitLab. + +The GitLab app in Azure needs to be configured following +[Azure's SCIM setup](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#getting-started). + +Note the following: + +- The `Tenant URL` and `secret token` are the ones retrieved in the +[previous step](#gitlab-configuration). +- Should there be any problems with the availability of GitLab or similar +errors, the notification email set will get those. +- For mappings, we will only leave `Synchronize Azure Active Directory Users to AppName` enabled. + +You can then test the connection clicking on `Test Connection`. + +### Synchronize Azure Active Directory users + +1. Click on `Synchronize Azure Active Directory Users to AppName`, to configure +the attribute mapping. +1. Select the unique identifier (in the example `objectId`) as the `id` and `externalId`, +and enable the `Create`, `Update`, and `Delete` actions. +1. Map the `userPricipalName` to `emails[type eq "work"].value` and `mailNickname` to +`userName`. + + Example configuration: + +  + +1. Click on **Show advanced options > Edit attribute list for AppName**. +1. Leave the `id` as the primary and only required field. + + NOTE: **Note:** + `username` should neither be primary nor required as we don't support + that field on GitLab SCIM yet. + +  + +1. Save all the screens and, in the **Provisioning** step, set +the `Provisioning Status` to `ON`. + + NOTE: **Note:** + You can control what is actually synced by selecting the `Scope`. For example, + `Sync only assigned users and groups` will only sync the users assigned to + the application (`Users and groups`), otherwise it will sync the whole Active Directory. + +Once enabled, the synchronization details and any errors will appear on the +bottom of the **Provisioning** screen, together with a link to the audit logs. diff --git a/doc/user/group/security_dashboard/index.md b/doc/user/group/security_dashboard/index.md new file mode 100644 index 00000000000..43e910b29fe --- /dev/null +++ b/doc/user/group/security_dashboard/index.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/security_dashboard/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/security_dashboard/index.html). diff --git a/doc/user/index.md b/doc/user/index.md index 8164b31c37e..67d5cbf06ec 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -51,19 +51,20 @@ With GitLab Enterprise Edition, you can also: - Provide support with [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html). - Improve collaboration with - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/index.html#merge-request-approvals), + [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/index.html#merge-request-approvals-starter), [Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), and [Multiple Issue Boards](project/issue_board.md#multiple-issue-boards-starter). - Create formal relationships between issues with [Related Issues](https://docs.gitlab.com/ee/user/project/issues/related_issues.html). - Use [Burndown Charts](https://docs.gitlab.com/ee/user/project/milestones/burndown_charts.html) to track progress during a sprint or while working on a new version of their software. -- Leverage [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) with [Advanced Global Search](https://docs.gitlab.com/ee/user/search/advanced_global_search.html) and [Advanced Syntax Search](https://docs.gitlab.com/ee/user/search/advanced_search_syntax.html) for faster, more advanced code search across your entire GitLab instance. +- Leverage [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) with [Advanced Global Search](search/advanced_global_search.md) and [Advanced Syntax Search](search/advanced_search_syntax.md) for faster, more advanced code search across your entire GitLab instance. - [Authenticate users with Kerberos](https://docs.gitlab.com/ee/integration/kerberos.html). -- [Mirror a repository](https://docs.gitlab.com/ee/workflow/repository_mirroring.html) from elsewhere on your local server. +- [Mirror a repository](../workflow/repository_mirroring.md) from elsewhere on your local server. - [Export issues as CSV](https://docs.gitlab.com/ee/user/project/issues/csv_export.html). - View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html). - [Lock files](https://docs.gitlab.com/ee/user/project/file_lock.html) to prevent conflicts. - View the current health and status of each CI environment running on Kubernetes with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html). - Leverage continuous delivery method with [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html). +- Scan your code for vulnerabilities and [display them in merge requests](https://docs.gitlab.com/ee/user/application_security/sast/index.html). You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, JIRA, and a lot more. @@ -106,8 +107,8 @@ to enjoy the best of GitLab. - [Permissions](permissions.md): Learn the different set of permissions levels for each user type (guest, reporter, developer, maintainer, owner). - [Feature highlight](feature_highlight.md): Learn more about the little blue dots - around the app that explain certain features -- [Abuse reports](abuse_reports.md): Report abuse from users to GitLab administrators + around the app that explain certain features. +- [Abuse reports](abuse_reports.md): Report abuse from users to GitLab administrators. ## Groups @@ -171,3 +172,8 @@ Learn what is [Git](../topics/git/index.md) and its best practices. ## Instance statistics See [various statistics](instance_statistics/index.md) of your GitLab instance. + +## Operations Dashboard **[PREMIUM]** + +See [Operations Dashboard](operations_dashboard/index.md) for a summary of each +project's operational health. diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 9891a43aa61..c1c2c036bae 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -20,6 +20,7 @@ You can use GFM in the following areas: - Snippets (the snippet must be named with a `.md` extension) - Wiki pages - Markdown documents inside repositories +- Epics **[ULTIMATE]** You can also use other rich text files in GitLab. You might have to install a dependency to do so. Please see the [`github-markup` gem readme](https://github.com/gitlabhq/markup#markups) for more information. @@ -320,6 +321,7 @@ GFM will recognize the following: | `#12345` | issue | | `!123` | merge request | | `$123` | snippet | +| `&123` | epic **[ULTIMATE]** | | `~123` | label by ID | | `~bug` | one-word label by name | | `~"feature request"` | multi-word label by name | @@ -340,6 +342,7 @@ GFM also recognizes certain cross-project references: | `namespace/project%123` | project milestone | | `namespace/project$123` | snippet | | `namespace/project@9ba12248` | specific commit | +| `group1/subgroup&123` | epic **[ULTIMATE]** | | `namespace/project@9ba12248...b19a04f5` | commit range comparison | | `namespace/project~"Some label"` | issues with given label | diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png Binary files differnew file mode 100644 index 00000000000..9d6a509ea72 --- /dev/null +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png Binary files differnew file mode 100644 index 00000000000..ee33eee8134 --- /dev/null +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md new file mode 100644 index 00000000000..66362f27299 --- /dev/null +++ b/doc/user/operations_dashboard/index.md @@ -0,0 +1,37 @@ +# Operations Dashboard **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5781) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. +[Moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/9218) to +[GitLab Premium](https://about.gitlab.com/pricing/) in 11.10. + +The Operations Dashboard provides a summary of each project's operational health, +including pipeline and alert status. + +The dashboard can be accessed via the top bar, by clicking on the new +dashboard icon: + + + +## Adding a project to the dashboard + +NOTE: **Note:** +For GitLab.com, the Operations Dashboard is available for free for public projects. +If your project is private, the group it belongs to must have a +[Gold](https://about.gitlab.com/pricing/) plan. + +To add a project to the dashboard: + +1. Click the **Add projects** button in the homescreen of the dashboard. +1. Search and add one or more projects using the **Search your projects** field. +1. Click the **Add projects** button. + +Once added, the dashboard will display the project's number of active alerts, +last commit, pipeline status, and when it was last deployed. + + + +## Making it the default dashboard when you sign in + +The Operations Dashboard can also be made the default GitLab dashboard shown when +you sign in. To make it the default, visit your [profile preferences](../profile/preferences.md). diff --git a/doc/user/permissions.md b/doc/user/permissions.md index a340dd063e4..9b298e4eb30 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -42,8 +42,7 @@ The following table depicts the various user permission levels in a project. | Create confidential issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | View confidential issues | (✓) [^2] | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | -| Lock merge request discussions | | | ✓ | ✓ | ✓ | +| See related issues | ✓ | ✓ | ✓ | ✓ | ✓ | | See a list of jobs | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | See a job log | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | Download and browse job artifacts | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | @@ -87,11 +86,12 @@ The following table depicts the various user permission levels in a project. | Update a container registry | | | ✓ | ✓ | ✓ | | Remove a container registry image | | | ✓ | ✓ | ✓ | | Create/edit/delete project milestones | | | ✓ | ✓ | ✓ | -| View approved/blacklisted licenses **[ULTIMATE]** | | | ✓ | ✓ | ✓ | +| View approved/blacklisted licenses **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | | Use security dashboard **[ULTIMATE]** | | | ✓ | ✓ | ✓ | | Dismiss vulnerability **[ULTIMATE]** | | | ✓ | ✓ | ✓ | | Apply code change suggestions | | | ✓ | ✓ | ✓ | | Use environment terminals | | | | ✓ | ✓ | +| Run Web IDE's Interactive Web Terminals **[ULTIMATE ONLY]** | | | | ✓ | ✓ | | Add new team members | | | | ✓ | ✓ | | Push to protected branches | | | | ✓ | ✓ | | Enable/disable branch protection | | | | ✓ | ✓ | @@ -120,6 +120,7 @@ The following table depicts the various user permission levels in a project. | Remove protected branches [^4] | | | | | | | View project Audit Events | | | | ✓ | ✓ | | View project statistics | | ✓ | ✓ | ✓ | ✓ | +| View Insights charts **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | ## Project features permissions @@ -172,8 +173,12 @@ read through the documentation on [permissions and access to confidential issues ### Releases permissions -[Project Releases](project/releases/index.md) can be read by all project -members (Reporters, Developers, Maintainers, Owners) **except Guests**. +[Project Releases](project/releases/index.md) can be read by project +members with Reporter, Developer, Maintainer, and Owner permissions. +Guest users can access Release pages for downloading assets but +are not allowed to download the source code nor see repository +information such as tags and commits. + Releases can be created, updated, or deleted via [Releases APIs](../api/releases/index.md) by project Developers, Maintainers, and Owners. @@ -200,6 +205,7 @@ group. | Create/edit group epic **[ULTIMATE]** | | ✓ | ✓ | ✓ | ✓ | | Delete group epic **[ULTIMATE]** | | | | | ✓ | | View group Audit Events | | | | | ✓ | +| View Insights charts **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | ### Subgroup permissions @@ -344,7 +350,7 @@ for details about the pipelines security model. ## LDAP users permissions Since GitLab 8.15, LDAP user permissions can now be manually overridden by an admin user. -Read through the documentation on [LDAP users permissions](https://docs.gitlab.com/ee/articles/how_to_configure_ldap_gitlab_ee/index.html#updating-user-permissions-new-feature) to learn more. +Read through the documentation on [LDAP users permissions](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html) to learn more. [^1]: On public and internal projects, all users are able to perform this action [^2]: Guest users can only view the confidential issues they created themselves diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index f399dc40164..b61216b7b67 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -79,6 +79,7 @@ You have 8 options here that you can use for your default dashboard view: - Your [Todos](../../workflow/todos.md) - Assigned Issues - Assigned Merge Requests +- Operations Dashboard **[PREMIUM]** ### Project overview content diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md new file mode 100644 index 00000000000..cce862b0911 --- /dev/null +++ b/doc/user/project/canary_deployments.md @@ -0,0 +1,71 @@ +# Canary Deployments **[PREMIUM]** + +> [Introduced][ee-1659] in [GitLab Premium][eep] 9.1. + +A popular [Continuous Integration](https://en.wikipedia.org/wiki/Continuous_integration) +strategy, where a small portion of the fleet is updated to the new version of +your application. + +## Overview + +When embracing [Continuous Delivery][cd-blog], a company needs to decide what +type of deployment strategy to use. One of the most popular strategies is canary +deployments, where a small portion of the fleet is updated to the new version +first. This subset, the canaries, then serve as the proverbial +[canary in the coal mine](https://en.wiktionary.org/wiki/canary_in_a_coal_mine). + +If there is a problem with the new version of the application, only a small +percentage of users are affected and the change can either be fixed or quickly +reverted. + +Leveraging [Kubernetes' Canary deployments][kube-canary], visualize your canary +deployments right inside the [Deploy Board], without the need to leave GitLab. + +## Use cases + +Canary deployments can be used when you want to ship features to only a portion of +your pods fleet and watch their behavior as a percentage of your user base +visits the temporarily deployed feature. If all works well, you can deploy the +feature to production knowing that it won't cause any problems. + +Canary deployments are also especially useful for backend refactors, performance +improvements, or other changes where the user interface doesn't change, but you +want to make sure the performance stays the same, or improves. Developers need +to be careful when using canaries with user-facing changes, because by default, +requests from the same user will be randomly distributed between canary and +non-canary pods, which could result in confusion or even errors. If needed, you +may want to consider [setting `service.spec.sessionAffinity` to `ClientIP` in +your Kubernetes service definitions][kube-net], but that is beyond the scope of +this document. + +## Enabling Canary Deployments + +Canary deployments require that you properly configure Deploy Boards: + +1. Follow the steps to [enable Deploy Boards](deploy_boards.md#enabling-deploy-boards). +1. To track canary deployments you need to label your Kubernetes deployments and + pods with `track: canary`. To get started quickly, you can use the [Auto Deploy] + template for canary deployments that GitLab provides. + +Depending on the deploy, the label should be either `stable` or `canary`. +Usually, `stable` and blank or missing label means the same thing, and `canary` +or any other track means canary/temporary. +This allows GitLab to discover whether deployment is stable or canary (temporary). + +Once all of the above are set up and the pipeline has run at least once, +navigate to the environments page under **Pipelines > Environments**. +As the pipeline executes Deploy Boards will clearly mark canary pods, enabling +quick and easy insight into the status of each environment and deployment. + +Canary deployments are marked with a yellow dot in the Deploy Board so that you +can easily notice them. + + + +[autodeploy]: ../../ci/autodeploy/index.md "GitLab Autodeploy" +[eep]: https://about.gitlab.com/pricing/ +[ee-1659]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1659 +[kube-canary]: https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments +[deploy board]: deploy_boards.md +[cd-blog]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/ +[kube-net]: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies diff --git a/doc/user/project/ci_cd_for_external_repo.md b/doc/user/project/ci_cd_for_external_repo.md new file mode 100644 index 00000000000..51b86a68c7b --- /dev/null +++ b/doc/user/project/ci_cd_for_external_repo.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html). diff --git a/doc/user/project/clusters/img/k8s_cluster_monitoring.png b/doc/user/project/clusters/img/k8s_cluster_monitoring.png Binary files differnew file mode 100644 index 00000000000..e449893a606 --- /dev/null +++ b/doc/user/project/clusters/img/k8s_cluster_monitoring.png diff --git a/doc/user/project/clusters/img/kubernetes_pod_logs.png b/doc/user/project/clusters/img/kubernetes_pod_logs.png Binary files differnew file mode 100644 index 00000000000..e664a47386a --- /dev/null +++ b/doc/user/project/clusters/img/kubernetes_pod_logs.png diff --git a/doc/user/project/clusters/img/pod_logs_deploy_board.png b/doc/user/project/clusters/img/pod_logs_deploy_board.png Binary files differnew file mode 100644 index 00000000000..7f83382968b --- /dev/null +++ b/doc/user/project/clusters/img/pod_logs_deploy_board.png diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 52b1708fe2d..157afb3a78c 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -406,6 +406,27 @@ Upgrades will reset values back to the values built into the `runner` chart plus the values set by [`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/runner/values.yaml) +### Uninstalling applications + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/60665) in +> GitLab 11.11. + +The applications below can be uninstalled. + +| Application | GitLab version | Notes | +| ----------- | -------------- | ----- | +| Prometheus | 11.11+ | All data will be deleted and cannot be restored. | + +To uninstall an application: + +1. Navigate to your project's **Operations > Kubernetes**. +1. Select your cluster. +1. Click the **Uninstall** button for the application. + +Support for uninstalling all applications will be progressively +introduced (see [related +epic](https://gitlab.com/groups/gitlab-org/-/epics/1201)). + ### Troubleshooting applications Applications can fail with the following error: @@ -527,7 +548,7 @@ differentiate the new cluster with the rest. When adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the -[environment-specific variables](https://docs.gitlab.com/ee/ci/variables/README.html#limiting-environment-scopes-of-variables-premium) work. +[environment-specific variables](https://docs.gitlab.com/ee/ci/variables/index.html#limiting-environment-scopes-of-environment-variables-premium) work. The default environment scope is `*`, which means all jobs, regardless of their environment, will use that cluster. Each scope can only be used by a single @@ -625,7 +646,7 @@ Common reasons for failure include: When [Prometheus is deployed](#installing-applications), GitLab will automatically monitor the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. - + ## Enabling or disabling the Kubernetes cluster integration @@ -656,7 +677,7 @@ and add a Kubernetes cluster again. ## View Kubernetes pod logs from GitLab **[ULTIMATE]** Learn how to easily -[view the logs of running pods in connected Kubernetes clusters](https://docs.gitlab.com/ee/user/project/clusters/kubernetes_pod_logs.html). +[view the logs of running pods in connected Kubernetes clusters](kubernetes_pod_logs.md). ## What you can get with the Kubernetes integration diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md new file mode 100644 index 00000000000..d5b60250860 --- /dev/null +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -0,0 +1,21 @@ +# Kubernetes Pod Logs **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4752) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. + +GitLab makes it easy to view the logs of running pods in [connected Kubernetes clusters](index.md). +By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface. + +## Overview + +[Kubernetes](https://kubernetes.io) pod logs can be viewed directly within GitLab. Logs can be displayed by clicking on a specific pod from [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html): + +1. Go to **Operations > Environments** and find the environment which contains the desired pod, like `production`. +1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html). +1. When mousing over the list of pods, a tooltip will appear with the exact pod name and status. + +1. Click on the desired pod to bring up the logs view, which will contain the last 500 lines for that pod. Support for pods with multiple containers is coming [in a future release](https://gitlab.com/gitlab-org/gitlab-ee/issues/6502). + + +## Requirements + +[Enabling Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html#enabling-deploy-boards) is required in order to be able to use Pod Logs. diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md new file mode 100644 index 00000000000..a4937e6cf6b --- /dev/null +++ b/doc/user/project/code_owners.md @@ -0,0 +1,85 @@ +# Code Owners **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6916) +in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. + +You can use a `CODEOWNERS` file to specify users that are responsible +for certain files in a repository. + +You can choose and add the `CODEOWNERS` file in three places: + +- to the root directory of the repository +- inside the `.gitlab/` directory +- inside the `docs/` directory + +The `CODEOWNERS` file is scoped to a branch, which means that with the +introduction of new files, the person adding the new content can +specify themselves as a code owner, all before the new changes +get merged to the default branch. + +When a file matches multiple entries in the `CODEOWNERS` file, +the users from all entries are displayed on the blob page of +the given file. + +## The syntax of Code Owners files + +Files can be specified using the same kind of patterns you would use +in the `.gitignore` file followed by the `@username` or email of one +or more users that should be owners of the file. + +The order in which the paths are defined is significant: the last +pattern that matches a given path will be used to find the code +owners. + +Starting a line with a `#` indicates a comment. This needs to be +escaped using `\#` to address files for which the name starts with a +`#`. + +Example `CODEOWNERS` file: + +``` +# This is an example code owners file, lines starting with a `#` will +# be ignored. + +# app/ @commented-rule + +# We can specifiy a default match using wildcards: +* @default-codeowner + +# Rules defined later in the file take precedence over the rules +# defined before. +# This will match all files for which the file name ends in `.rb` +*.rb @ruby-owner + +# Files with a `#` can still be accesssed by escaping the pound sign +\#file_with_pound.rb @owner-file-with-pound + +# Multiple codeowners can be specified, separated by whitespace +CODEOWNERS @multiple @owners @tab-separated + +# Both usernames or email addresses can be used to match +# users. Everything else will be ignored. For example this will +# specify `@legal` and a user with email `janedoe@gitlab.com` as the +# owner for the LICENSE file +LICENSE @legal this_does_not_match janedoe@gitlab.com + +# Ending a path in a `/` will specify the code owners for every file +# nested in that directory, on any level +/docs/ @all-docs + +# Ending a path in `/*` will specify code owners for every file in +# that directory, but not nested deeper. This will match +# `docs/index.md` but not `docs/projects/index.md` +/docs/* @root-docs + +# This will make a `lib` directory nested anywhere in the repository +# match +lib/ @lib-owner + +# This will only match a `config` directory in the root of the +# repository +/config/ @config-owner + +# If the path contains spaces, these need to be escaped like this: +path\ with\ spaces/ @space-owner +``` diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md new file mode 100644 index 00000000000..2f2a9c5eec3 --- /dev/null +++ b/doc/user/project/deploy_boards.md @@ -0,0 +1,132 @@ +# Deploy Boards **[PREMIUM]** + +> [Introduced][ee-1589] in [GitLab Premium][ee] 9.0. + +GitLab's Deploy Boards offer a consolidated view of the current health and +status of each CI [environment] running on [Kubernetes], displaying the status +of the pods in the deployment. Developers and other teammates can view the +progress and status of a rollout, pod by pod, in the workflow they already use +without any need to access Kubernetes. + +## Overview + +With Deploy Boards you can gain more insight into deploys with benefits such as: + +- Following a deploy from the start, not just when it's done +- Watching the rollout of a build across multiple servers +- Finer state detail (Waiting, Deploying, Finished, Unknown) +- See [Canary Deployments](canary_deployments.md) + +Here's an example of a Deploy Board of the production environment. + + + +The squares represent pods in your Kubernetes cluster that are associated with +the given environment. Hovering above each square you can see the state of a +deploy rolling out. The percentage is the percent of the pods that are updated +to the latest release. + +Since Deploy Boards are tightly coupled with Kubernetes, there is some required +knowledge. In particular you should be familiar with: + +- [Kubernetes pods](https://kubernetes.io/docs/user-guide/pods) +- [Kubernetes labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) +- [Kubernetes namespaces](https://kubernetes.io/docs/user-guide/namespaces/) +- [Kubernetes canary deployments](https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments) + +## Use cases + +Since the Deploy Board is a visual representation of the Kubernetes pods for a +specific environment, there are lot of uses cases. To name a few: + +- You want to promote what's running in staging, to production. You go to the + environments list, verify that what's running in staging is what you think is + running, then click on the [manual action](../../ci/yaml/README.md#whenmanual) to deploy to production. +- You trigger a deploy, and you've got lots of containers to upgrade so you know + it'll take a while (you've also throttled your deploy to only take down X + containers at a time). But you need to tell someone when it's deployed, so you + go to the environments list, look at the production environment to see what + the progress is in real-time as each pod is rolled. +- You get a report that something is weird in production, so you look at the + production environment to see what is running, and if a deploy is ongoing or + stuck or failed. +- You've got an MR that looks good, but you want to run it on staging because + staging is set up in some way closer to production. You go to the environment + list, find the [Review App][review apps] you're interested in, and click the + manual action to deploy it to staging. + +## Enabling Deploy Boards + +To display the Deploy Boards for a specific [environment] you should: + +1. Have a Kubernetes cluster up and running. + + NOTE: **Running on OpenShift:** + If you are using OpenShift, ensure that you're using the `Deployment` resource + instead of `DeploymentConfiguration`, otherwise the Deploy Boards won't render + correctly. For more information, read the + [OpenShift docs](https://docs.openshift.com/container-platform/3.7/dev_guide/deployments/kubernetes_deployments.html#kubernetes-deployments-vs-deployment-configurations) + and [GitLab issue #4584](https://gitlab.com/gitlab-org/gitlab-ee/issues/4584). + +1. [Configure GitLab Runner][runners] with the [Docker][docker-exec] or + [Kubernetes][kube-exec] executor. +1. Configure the [Kubernetes service][kube-service] in your project for the + cluster. The Kubernetes namespace is of particular note as you will need it + for your deployment scripts (exposed by the `KUBE_NAMESPACE` env variable). +1. Ensure Kubernetes annotations of `app.gitlab.com/env: $CI_ENVIRONMENT_SLUG` + and `app.gitlab.com/app: $CI_PROJECT_PATH_SLUG` are applied to the + deployments, replica sets, and pods, where `$CI_ENVIRONMENT_SLUG` and + `$CI_PROJECT_PATH_SLUG` are the values of the CI variables. This is so we can + lookup the proper environment in a cluster/namespace which may have more + than one. These resources should be contained in the namespace defined in + the Kubernetes service setting. You can use an [Autodeploy] `.gitlab-ci.yml` + template which has predefined stages and commands to use, and automatically + applies the annotations. Each project will need to have a unique namespace in + Kubernetes as well. The image below demonstrates how this is shown inside + Kubernetes. + + NOTE: **Note:** + The Kubernetes label of `app` is deprecated and may be removed in next major + release, GitLab 12.0. + +  + +Once all of the above are set up and the pipeline has run at least once, +navigate to the environments page under **Operations > Environments**. + +Deploy Boards are visible by default. You can explicitly click +the triangle next to their respective environment name in order to hide them. +GitLab will then query Kubernetes for the state of each pod (e.g., waiting, +deploying, finished, unknown), and the Deploy Board status will finally appear. + +GitLab will only display a Deploy Board for top-level environments. Foldered +environments like `review/*` (usually used for [Review Apps]) won't have a +Deploy Board attached to them. + +## Canary Deployments + +A popular CI strategy, where a small portion of the fleet is updated to the new +version of your application. + +[Read more about Canary Deployments.](canary_deployments.md) + +## Further reading + +- [GitLab Autodeploy][autodeploy] +- [GitLab CI environment variables][variables] +- [Environments and deployments][environment] +- [Kubernetes deploy example][kube-deploy] + +[ee-1589]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1589 "Deploy Boards initial issue" +[ee]: https://about.gitlab.com/pricing/ "GitLab Enterprise Edition landing page" +[kube-deploy]: https://gitlab.com/gitlab-examples/kubernetes-deploy "Kubernetes deploy example project" +[kubernetes]: https://kubernetes.io "Kubernetes website" +[environment]: ../../ci/environments.md "Environments and deployments documentation" +[docker-exec]: https://docs.gitlab.com/runner/executors/docker.html "GitLab Runner Docker executor" +[kube-exec]: https://docs.gitlab.com/runner/executors/kubernetes.html "GitLab Runner Kubernetes executor" +[kube-service]: integrations/kubernetes.md "Kubernetes project service" +[review apps]: ../../ci/review_apps/index.md "Review Apps documentation" +[variables]: ../../ci/variables/README.md "GitLab CI variables" +[autodeploy]: ../../ci/autodeploy/index.md "GitLab Autodeploy" +[kube-image]: https://gitlab.com/gitlab-examples/kubernetes-deploy/container_registry "Kubernetes deploy Container Registry" +[runners]: ../../ci/runners/README.md diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 08647caaf07..e230444fa67 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -2,8 +2,12 @@ >[Introduced][ce-4981] in GitLab 8.11. +We all know that a properly submitted issue is more likely to be addressed in +a timely manner by the developers of a project. + Description templates allow you to define context-specific templates for issue -and merge request description fields for your project. +and merge request description fields for your project, as well as help filter +out a lot of unnecessary noise from issues. ## Overview @@ -18,6 +22,18 @@ Description templates must be written in [Markdown](../markdown.md) and stored in your project's repository under a directory named `.gitlab`. Only the templates of the default branch will be taken into account. +## Use-cases + +- Add a template to be used in every issue for a specific project, +giving instructions and guidelines, requiring for information specific to that subject. +For example, if you have a project for tracking new blog posts, you can require the +title, outlines, author name, author social media information, etc. +- Following the previous example, you can make a template for every MR submitted +with a new blog post, requiring information about the post date, frontmatter data, +images guidelines, link to the related issue, reviewer name, etc. +- You can also create issues and merge request templates for different +stages of your workflow, e.g., feature proposal, feature improvement, bug report, etc. + ## Creating issue templates Create a new Markdown (`.md`) file inside the `.gitlab/issue_templates/` @@ -39,6 +55,32 @@ changes you made after picking the template and return it to its initial status.  +## Setting a default template for issues and merge requests **[STARTER]** + +> **Notes:** +> - This feature was introduced before [description templates](#overview) and is +> available in [GitLab Starter][products]. It can be enabled +> in the project's settings. +> - Templates for issues were [introduced][ee-28] in GitLab EE 8.1. +> - Templates for merge requests were [introduced][ee-7478ece] in GitLab EE 6.9. + +The visibility of issues and/or merge requests should be set to either "Everyone +with access" or "Only team members" in your project's **Settings** otherwise the +template text areas won't show. This is the default behavior so in most cases +you should be fine. + +Go to your project's **Settings** and fill in the "Default description template +for issues" and "Default description template for merge requests" text areas +for issues and merge requests respectively. Since GitLab issues and merge +request support [Markdown](../markdown.md), you can use special markup like +headings, lists, etc. + + + +After you add the description, hit **Save changes** for the settings to take +effect. Now, every time a new issue or merge request is created, it will be +pre-filled with the text you entered in the template(s). + ## Description template example We make use of Description Templates for Issues and Merge Requests within the GitLab Community Edition project. Please refer to the [`.gitlab` folder][gitlab-ce-templates] for some examples. @@ -93,3 +135,6 @@ Possible fixes [ce-4981]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4981 [gitlab-ce-templates]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/.gitlab +[ee-28]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/28 "Merge Request for adding issues template" +[ee-7478ece]: https://gitlab.com/gitlab-org/gitlab-ee/commit/7478ece8b48e80782b5465b96c79f85cc91d391b "Commit that introduced merge requests templates" +[products]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md new file mode 100644 index 00000000000..541023692ea --- /dev/null +++ b/doc/user/project/file_lock.md @@ -0,0 +1,107 @@ +# File Locking **[PREMIUM]** + +> **Notes:** +> - [Introduced][ee-440] in [GitLab Premium][ee] 8.9. +> - This feature needs to have a license with the "File Lock" option enabled. If +> you are using Premium but you don't see the "Lock" button, +> ask your GitLab administrator. + +File Locking helps you avoid merge conflicts and better manage your binary files. +Lock any file or directory, make your changes, and then unlock it so another +member of the team can edit it. + +## Overview + +Working with multiple people on the same file can be a risk. Conflicts +when merging a non-text file are hard to overcome and will require a lot +of manual work to resolve. With GitLab Premium, File +Locking helps you avoid merge conflicts and better manage your binary +files by preventing everyone, except you, from modifying a specific file +or entire directory. + +## Use-cases + +The file locking feature is useful in situations when: + +- Multiple people are working on the same file and you want to avoid merge + conflicts. +- Your repository contains binary files in which situation there is no easy + way to tell the diff between yours and your colleagues' changes. +- Prevent design assets from being overwritten. + +Locked directories are locked recursively, which means that everything that +lies under them is also locked. + +## Permissions on file locking + +The user that locks a file or directory **is the only one** that can edit and +push their changes back to the repository where the locked objects are located. + +Locks can be created by any person who has [push access] to the repository; i.e., +Developer and higher level, and can be removed solely by their author and any +user with Maintainer permissions and above. + +If a file is locked and you are not the author of its locked state, a +pre-receive hook will reject your changes when you try to push. In the +following example, a user who has no permissions on the locked `.gitignore` +file will see the message below: + +```bash +Counting objects: 3, done. +Delta compression using up to 4 threads. +Compressing objects: 100% (3/3), done. +Writing objects: 100% (3/3), 320 bytes | 0 bytes/s, done. +Total 3 (delta 1), reused 0 (delta 0) +remote: GitLab: The path '.gitignore' is locked by Administrator +To https://example.com/gitlab-org/gitlab-ce.git + ! [remote rejected] master -> master (pre-receive hook declined) + error: failed to push some refs to 'https://example.com/gitlab-org/gitlab-ce.git' +``` + +Similarly, when a user that is not the author of the locked state of a file +accepts a merge request, an error message will appear stating that the file +is locked. + + + +## Locking a file or a directory + +>**Note:** +Locking only works for the default branch you have set in the project's settings +(usually `master`). + +To lock a file, navigate to the repository tree under the **Repository > Files** tab, +pick the file you want to lock and hit the "Lock" button. + + + +--- + +To lock an entire directory, look for the "Lock" link next to "History". + + + +--- + +After you lock a file or directory, it will appear as locked in the repository +view. + + + +## Unlocking a file or a directory + +To unlock a file or a directory, follow the same procedure as when you locked +them. For a detailed view of every existing lock, see the next section on +"Viewing and managing existing locks". + +## Viewing and managing existing locks + +To view or manage every existing lock, navigate to the +**Project > Repository > Locked Files** area. There, you can view all existing +locks and [remove the ones you have permission for](#permissions-on-file-locking). + + + +[ee-440]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/440 "File Lock" +[ee]: https://about.gitlab.com/pricing/ +[push access]: ../../user/permissions.md diff --git a/doc/user/project/img/assignee_lists.png b/doc/user/project/img/assignee_lists.png Binary files differnew file mode 100644 index 00000000000..f2660cd8f80 --- /dev/null +++ b/doc/user/project/img/assignee_lists.png diff --git a/doc/user/project/img/deploy_boards_canary_deployments.png b/doc/user/project/img/deploy_boards_canary_deployments.png Binary files differnew file mode 100644 index 00000000000..037b4a908ab --- /dev/null +++ b/doc/user/project/img/deploy_boards_canary_deployments.png diff --git a/doc/user/project/img/deploy_boards_kubernetes_label.png b/doc/user/project/img/deploy_boards_kubernetes_label.png Binary files differnew file mode 100644 index 00000000000..19785c74d07 --- /dev/null +++ b/doc/user/project/img/deploy_boards_kubernetes_label.png diff --git a/doc/user/project/img/deploy_boards_landing_page.png b/doc/user/project/img/deploy_boards_landing_page.png Binary files differnew file mode 100644 index 00000000000..c9621a06860 --- /dev/null +++ b/doc/user/project/img/deploy_boards_landing_page.png diff --git a/doc/user/project/img/description_templates_default_settings.png b/doc/user/project/img/description_templates_default_settings.png Binary files differnew file mode 100644 index 00000000000..ab314e83d06 --- /dev/null +++ b/doc/user/project/img/description_templates_default_settings.png diff --git a/doc/user/project/img/file_lock.png b/doc/user/project/img/file_lock.png Binary files differnew file mode 100644 index 00000000000..33f96f20e91 --- /dev/null +++ b/doc/user/project/img/file_lock.png diff --git a/doc/user/project/img/file_lock_folders.png b/doc/user/project/img/file_lock_folders.png Binary files differnew file mode 100644 index 00000000000..5ff3d4d9dbd --- /dev/null +++ b/doc/user/project/img/file_lock_folders.png diff --git a/doc/user/project/img/file_lock_list.png b/doc/user/project/img/file_lock_list.png Binary files differnew file mode 100644 index 00000000000..2c276335c83 --- /dev/null +++ b/doc/user/project/img/file_lock_list.png diff --git a/doc/user/project/img/file_lock_merge_request_error_message.png b/doc/user/project/img/file_lock_merge_request_error_message.png Binary files differnew file mode 100644 index 00000000000..65bc3692da0 --- /dev/null +++ b/doc/user/project/img/file_lock_merge_request_error_message.png diff --git a/doc/user/project/img/file_lock_repository_view.png b/doc/user/project/img/file_lock_repository_view.png Binary files differnew file mode 100644 index 00000000000..8f4ef52aacd --- /dev/null +++ b/doc/user/project/img/file_lock_repository_view.png diff --git a/doc/user/project/img/key_value_labels.png b/doc/user/project/img/key_value_labels.png Binary files differnew file mode 100644 index 00000000000..bec901f127f --- /dev/null +++ b/doc/user/project/img/key_value_labels.png diff --git a/doc/user/project/img/labels_epic_sidebar.png b/doc/user/project/img/labels_epic_sidebar.png Binary files differnew file mode 100644 index 00000000000..f8162d89e9d --- /dev/null +++ b/doc/user/project/img/labels_epic_sidebar.png diff --git a/doc/user/project/img/project_security_dashboard.png b/doc/user/project/img/project_security_dashboard.png Binary files differnew file mode 100644 index 00000000000..3294e59e943 --- /dev/null +++ b/doc/user/project/img/project_security_dashboard.png diff --git a/doc/user/project/img/protected_branches_select_roles_and_users.png b/doc/user/project/img/protected_branches_select_roles_and_users.png Binary files differnew file mode 100644 index 00000000000..4f62b057cc7 --- /dev/null +++ b/doc/user/project/img/protected_branches_select_roles_and_users.png diff --git a/doc/user/project/img/protected_branches_select_roles_and_users_list.png b/doc/user/project/img/protected_branches_select_roles_and_users_list.png Binary files differnew file mode 100644 index 00000000000..519f2267496 --- /dev/null +++ b/doc/user/project/img/protected_branches_select_roles_and_users_list.png diff --git a/doc/user/project/img/service_desk_confirmation_email.png b/doc/user/project/img/service_desk_confirmation_email.png Binary files differnew file mode 100644 index 00000000000..e08dced1e38 --- /dev/null +++ b/doc/user/project/img/service_desk_confirmation_email.png diff --git a/doc/user/project/img/service_desk_disabled.png b/doc/user/project/img/service_desk_disabled.png Binary files differnew file mode 100644 index 00000000000..3ae64dcbe96 --- /dev/null +++ b/doc/user/project/img/service_desk_disabled.png diff --git a/doc/user/project/img/service_desk_enabled.png b/doc/user/project/img/service_desk_enabled.png Binary files differnew file mode 100644 index 00000000000..329348e4b52 --- /dev/null +++ b/doc/user/project/img/service_desk_enabled.png diff --git a/doc/user/project/img/service_desk_issue_tracker.png b/doc/user/project/img/service_desk_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..485751fab0a --- /dev/null +++ b/doc/user/project/img/service_desk_issue_tracker.png diff --git a/doc/user/project/img/service_desk_nav_item.png b/doc/user/project/img/service_desk_nav_item.png Binary files differnew file mode 100644 index 00000000000..3420e355f67 --- /dev/null +++ b/doc/user/project/img/service_desk_nav_item.png diff --git a/doc/user/project/img/service_desk_reply.png b/doc/user/project/img/service_desk_reply.png Binary files differnew file mode 100644 index 00000000000..ae6ce31713b --- /dev/null +++ b/doc/user/project/img/service_desk_reply.png diff --git a/doc/user/project/img/service_desk_thread.png b/doc/user/project/img/service_desk_thread.png Binary files differnew file mode 100644 index 00000000000..8d8ac39cc5b --- /dev/null +++ b/doc/user/project/img/service_desk_thread.png diff --git a/doc/user/project/import/gemnasium.md b/doc/user/project/import/gemnasium.md new file mode 100644 index 00000000000..dc5b3fcd0bb --- /dev/null +++ b/doc/user/project/import/gemnasium.md @@ -0,0 +1,102 @@ +# Gemnasium **[ULTIMATE]** + +This guide describes how to migrate from Gemnasium.com to your own GitLab +instance or GitLab.com. + +## Why is Gemnasium.com closed? + +Gemnasium has been [acquired by GitLab](https://about.gitlab.com/press/releases/2018-01-30-gemnasium-acquisition.html) +in January 2018. Since May 15, 2018, the services provided by Gemnasium are no longer available. +The team behind Gemnasium has joined GitLab as the new Security Products team +and is working on a wider range of tools than just Dependency Scanning: +[SAST](https://docs.gitlab.com/ee/user/application_security/sast/index.html), +[DAST](https://docs.gitlab.com/ee/user/application_security/dast/index.html), +[Container Scanning](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html) and more. +If you want to continue monitoring your dependencies, see the +[Migrating to GitLab](#migrating-to-gitlab) section below. + +## What happened to my account? + +Your account has been automatically closed on May 15th, 2018. If you had a paid +subscription at that time, your card will be refunded on a pro rata temporis basis. +You may contact `gemnasium@gitlab.com` regarding your closed account. + +## Will my account/data be transferred to GitLab Inc.? + +All accounts and data have been deleted on May 15th, 2018. GitLab Inc. +doesn't know anything about your private data, nor your projects, and therefore +if they were vulnerable or not. GitLab Inc. takes personal information very seriously. + +## What happened to my badge? + +To avoid broken 404 images, all badges pointing to Gemnasium.com will be a +placeholder, inviting you to migrate to GitLab (and pointing to this page). + +## Migrating to GitLab + +Gemnasium has been ported and integrated directly into GitLab CI/CD. +You can still benefit from our dependency monitoring features, and it requires +some steps to migrate your projects. There is no automatic import since GitLab +doesn't know anything about any projects which existed on Gemnasium.com. +Security features are free for public (open-source) projects hosted on GitLab.com. + +### If your project is hosted on GitLab (https://gitlab.com / self-hosted) + +You're almost set! If you're already using +[Auto DevOps](../../../topics/autodevops/), you are already covered. +Otherwise, you must configure your `.gitlab-ci.yml` according to the +[dependency scanning page](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). + +### If your project is hosted on GitHub (https://github.com / GitHub Enterprise) + +Since [GitLab 10.6 comes with GitHub integration](https://about.gitlab.com/features/github/), +GitLab users can now create a CI/CD project in GitLab connected to an external +GitHub.com or GitHub Enterprise repository. This will automatically prompt +GitLab CI/CD to run whenever code is pushed to GitHub and post CI/CD results +back to both GitLab and GitHub when completed. + +1. Create a new project, and select the "CI/CD for external repo" tab: + +  + +1. Use the "GitHub" button to connect your repositories. + +  + +1. Select the project(s) to be set up with GitLab CI/CD and chose "Connect". + +  + + Once the configuration is done, you may click on your new + project on GitLab. + +  + + Your project is now mirrored on GitLab, where the Runners will be able to access + your source code and run your tests. + + Optional step: If you set this up on GitLab.com, make sure the project is + public (in the project settings) if your GitHub project is public, since + the security feature is available only for [GitLab Ultimate](https://about.gitlab.com/pricing). + +1. To set up the dependency scanning job, corresponding to what Gemnasium was + doing, you must create a `.gitlab-ci.yml` file, or update it according to + the [dependency scanning docs](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). + The mirroring is pull-only by default, so you may create or update the file on + GitHub: + +  + +1. Once your file has been committed, a new pipeline will be automatically + triggered if your file is valid: + +  + +1. The result of the job will be visible directly from the pipeline view: + +  + +NOTE: **Note:** +If you don't commit very often to your project, you may want to use +[scheduled pipelines](../pipelines/schedules.md) to run the job on a regular +basis. diff --git a/doc/user/project/import/img/gemnasium/connect_github.png b/doc/user/project/import/img/gemnasium/connect_github.png Binary files differnew file mode 100644 index 00000000000..5933f4d5d0a --- /dev/null +++ b/doc/user/project/import/img/gemnasium/connect_github.png diff --git a/doc/user/project/import/img/gemnasium/create_project.png b/doc/user/project/import/img/gemnasium/create_project.png Binary files differnew file mode 100644 index 00000000000..6e00bf63405 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/create_project.png diff --git a/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png b/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png Binary files differnew file mode 100644 index 00000000000..8336c803b83 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png diff --git a/doc/user/project/import/img/gemnasium/pipeline.png b/doc/user/project/import/img/gemnasium/pipeline.png Binary files differnew file mode 100644 index 00000000000..ae4d5295ffa --- /dev/null +++ b/doc/user/project/import/img/gemnasium/pipeline.png diff --git a/doc/user/project/import/img/gemnasium/project_connected.png b/doc/user/project/import/img/gemnasium/project_connected.png Binary files differnew file mode 100644 index 00000000000..8e7216c015b --- /dev/null +++ b/doc/user/project/import/img/gemnasium/project_connected.png diff --git a/doc/user/project/import/img/gemnasium/report.png b/doc/user/project/import/img/gemnasium/report.png Binary files differnew file mode 100644 index 00000000000..5c4d58662c0 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/report.png diff --git a/doc/user/project/import/img/gemnasium/select_project.png b/doc/user/project/import/img/gemnasium/select_project.png Binary files differnew file mode 100644 index 00000000000..588c5ca7ce1 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/select_project.png diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index cc73ea95f51..ebbc5ca133b 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -13,11 +13,14 @@ 1. [From TFS](tfs.md) 1. [From repo by URL](repo_by_url.md) 1. [By uploading a manifest file (AOSP)](manifest.md) +1. [From Gemnasium](gemnasium.md) In addition to the specific migration documentation above, you can import any Git repository via HTTP from the New Project page. Be aware that if the repository is too large the import can timeout. +There is also the option of [connecting your external repository to get CI/CD benefits](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html). **[PREMIUM]** + ## Migrating from self-hosted GitLab to GitLab.com If you only need to migrate git repos, you can [import each project by URL](repo_by_url.md), but issues and merge requests can't be imported. diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 64139f9dbe9..6b3b40bf9f8 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -13,7 +13,7 @@ the number of private projects you create. When you create a project in GitLab, you'll have access to a large number of [features](https://about.gitlab.com/features/): -**Issues and merge requests:** +**Repositories:** - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow @@ -28,6 +28,13 @@ When you create a project in GitLab, you'll have access to a large number of permission to create tags, and prevent accidental update or deletion - [Signing commits](gpg_signed_commits/index.md): use GPG to sign your commits - [Deploy tokens](deploy_tokens/index.md): Manage project-based deploy tokens that allow permanent access to the repository and Container Registry. +- [Web IDE](web_ide/index.md) + +**Issues and merge requests:** + +- [Issue tracker](issues/index.md): Discuss implementations with your team within issues + - [Issue Boards](issue_board.md): Organize and prioritize your workflow + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards-starter): Allow your teams to create their own workflows (Issue Boards) for the same project **[STARTER]** - [Merge Requests](merge_requests/index.md): Apply your branching strategy and get reviewed by your team - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html): Ask for approval before @@ -67,6 +74,8 @@ When you create a project in GitLab, you'll have access to a large number of timeout (defines the maximum amount of time in minutes that a job is able run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline's visibility, and much more - [Kubernetes cluster integration](clusters/index.md): Connecting your GitLab project with a Kubernetes cluster + - [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html): Feature flags allow you to ship a project in + different flavors by dynamically toggling certain functionality **[PREMIUM]** - [GitLab Pages](pages/index.md): Build, test, and deploy your static website with GitLab Pages @@ -75,12 +84,17 @@ When you create a project in GitLab, you'll have access to a large number of - [Wiki](wiki/index.md): document your GitLab project in an integrated Wiki. - [Snippets](../snippets.md): store, share and collaborate on code snippets. - [Cycle Analytics](cycle_analytics.md): review your development lifecycle. +- [Security Dashboard](security_dashboard.md): Security Dashboard. **[ULTIMATE]** - [Syntax highlighting](highlighting.md): an alternative to customize your code blocks, overriding GitLab's default choice of language. - [Badges](badges.md): badges for the project overview. - [Releases](releases/index.md): a way to track deliverables in your project as snapshot in time of the source, build output, and other metadata or artifacts associated with a released version of your code. +- [Maven packages](https://docs.gitlab.com/ee/user/project/packages/maven_repository.html): your private Maven repository in GitLab. **[PREMIUM]** +- [NPM packages](https://docs.gitlab.com/ee/user/project/packages/npm_registry.html): your private NPM package registry in GitLab. **[PREMIUM]** +- [Code owners](code_owners.md): specify code owners for certain files **[STARTER]** +- [License Management](https://docs.gitlab.com/ee/user/application_security/license_management/index.html): approve and blacklist licenses for projects. **[ULTIMATE]** ### Project integrations @@ -116,6 +130,13 @@ Read through the documentation on [project settings](settings/index.md). - [Export a project from GitLab](settings/import_export.md#exporting-a-project-and-its-data) - [Importing and exporting projects between GitLab instances](settings/import_export.md) +## CI/CD for external repositories **[PREMIUM]** + +Instead of importing a repository directly to GitLab, you can connect your repository +as a CI/CD project. + +Read through the documentation on [CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html). + ## Project members Learn how to [add members to your projects](members/index.md). diff --git a/doc/user/project/insights/img/insights_example_bar_chart.png b/doc/user/project/insights/img/insights_example_bar_chart.png Binary files differnew file mode 100644 index 00000000000..eb96eb4a90f --- /dev/null +++ b/doc/user/project/insights/img/insights_example_bar_chart.png diff --git a/doc/user/project/insights/img/insights_example_bar_time_series_chart.png b/doc/user/project/insights/img/insights_example_bar_time_series_chart.png Binary files differnew file mode 100644 index 00000000000..28aa81939d8 --- /dev/null +++ b/doc/user/project/insights/img/insights_example_bar_time_series_chart.png diff --git a/doc/user/project/insights/img/insights_example_line_chart.png b/doc/user/project/insights/img/insights_example_line_chart.png Binary files differnew file mode 100644 index 00000000000..131dbedd35e --- /dev/null +++ b/doc/user/project/insights/img/insights_example_line_chart.png diff --git a/doc/user/project/insights/img/insights_example_pie_chart.png b/doc/user/project/insights/img/insights_example_pie_chart.png Binary files differnew file mode 100644 index 00000000000..518ed7338f9 --- /dev/null +++ b/doc/user/project/insights/img/insights_example_pie_chart.png diff --git a/doc/user/project/insights/img/insights_example_stacked_bar_chart.png b/doc/user/project/insights/img/insights_example_stacked_bar_chart.png Binary files differnew file mode 100644 index 00000000000..aafec4b394e --- /dev/null +++ b/doc/user/project/insights/img/insights_example_stacked_bar_chart.png diff --git a/doc/user/project/insights/img/project_insights.png b/doc/user/project/insights/img/project_insights.png Binary files differnew file mode 100644 index 00000000000..2d0292dda54 --- /dev/null +++ b/doc/user/project/insights/img/project_insights.png diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md new file mode 100644 index 00000000000..720f4c6604e --- /dev/null +++ b/doc/user/project/insights/index.md @@ -0,0 +1,300 @@ +# Insights **[ULTIMATE]** + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. + +CAUTION: **Beta:** +Insights is considered beta, and is not ready for production use. +Follow [gitlab-org&725](https://gitlab.com/groups/gitlab-org/-/epics/725) for +updates. + +Configure the Insights that matter for your projects to explore data such as +triage hygiene, issues created/closed per a given period, average time for merge +requests to be merged and much more. + + + +NOTE: **Note:** +This feature is [also available at the group level](https://docs.gitlab.com/ee/user/group/insights/index.html). + +## Configure your Insights + +Insights are configured using a YAML file called `.gitlab/insights.yml` within +a project. That file will then be used in the project's Insights page. + +See [Writing your `.gitlab/insights.yml`](#writing-your-gitlabinsightsyml) below +for details about the content of this file. + +NOTE: **Note:** +Once the configuration file is created, you can also +[use it for your project's group](https://docs.gitlab.com/ee/user/group/insights/index.html#configure-your-insights). + +NOTE: **Note:** +If the project doesn't have any configuration file, it'll try to use +the group configuration if possible. If the group doesn't have any +configuration, the default configuration will be used. + +## Permissions + +If you have access to view a project, then you have access to view their +Insights. + +NOTE: **Note:** +Issues or merge requests that you don't have access to (because you don't have +access to the project they belong to, or because they are confidential) are +filtered out of the Insights charts. + +You may also consult the [group permissions table](../../permissions.md#group-members-permissions). + +## Writing your `.gitlab/insights.yml` + +The `.gitlab/insights.yml` file defines the structure and order of the Insights +charts that will be displayed in each Insights page of your project or group. + +Each page has a unique key and a collection of charts to fetch and display. + +For example, here's a single definition for Insights that will display one page with one chart: + +```yaml +bugsCharts: + title: 'Charts for Bugs' + charts: + - title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + group_by: month + period_limit: 24 +``` + +Each chart definition is made up of a hash composed of key-value pairs. + +For example, here's single chart definition: + +```yaml +monthlyBugsCreated: + title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + group_by: month + period_limit: 24 +``` + +## Configuration parameters + +A chart is defined as a list of parameters that define the chart's behavior. + +The following table lists available parameters for charts: + +| Keyword | Description | +|:---------------------------------------------------|:------------| +| [`title`](#title) | The title of the chart. This will displayed on the Insights page. | +| [`type`](#type) | The type of chart: `bar`, `line`, `stacked-bar`, `pie` etc. | +| [`query`](#query) | A hash that defines the conditions for issues / merge requests to be part of the chart. | + +## Parameter details + +The following are detailed explanations for parameters used to configure +Insights charts. + +### `title` + +`title` is the title of the chart as it will be displayed on the Insights page. +For example: + +```yaml +monthlyBugsCreated: + title: Monthly Bugs Created (bar) +``` + +### `type` + +`type` is the chart type. + +For example: + +```yaml +monthlyBugsCreated: + title: Monthly Bugs Created (bar) + type: bar +``` + +Supported values are: + +| Name | Example | +| ----- | ------- | +| `bar` |  | +| `bar` (time series, i.e. when `group_by` is used) |  | +| `pie` |  | +| `line` |  | +| `stacked-bar` |  | + +### `query` + +`query` allows to define the conditions for issues / merge requests to be part +of the chart. + +Example: + +```yaml +monthlyBugsCreated: + title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + collection_labels: + - S1 + - S2 + - S3 + - S4 + group_by: week + period_limit: 104 +``` + +#### `query.issuable_type` + +Defines the type of "issuable" you want to create a chart for. + +Supported values are: + +- `issue`: The chart will display issues' data. +- `merge_request`: The chart will display merge requests' data. + +#### `query.issuable_state` + +Filter by the state of the queried "issuable". + +If you omit it, no state filter will be applied. + +Supported values are: + +- `opened`: Open issues / merge requests. +- `closed`: Closed Open issues / merge requests. +- `locked`: Issues / merge requests that have their discussion locked. +- `merged`: Merged merge requests. + +#### `query.filter_labels` + +Filter by labels applied to the queried "issuable". + +If you omit it, no labels filter will be applied. All the defined labels must be +applied to the "issuable" in order for it to be selected. + +Example: + +```yaml +monthlyBugsCreated: + title: Monthly regressions Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + - regression +``` + +#### `query.collection_labels` + +Group "issuable" by the configured labels. + +If you omit it, no grouping will be done. When using this keyword, you need to +set `type` to either `line` or `stacked-bar`. + +Example: + +```yaml +weeklyBugsBySeverity: + title: Weekly Bugs By Severity (stacked bar) + type: stacked-bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + collection_labels: + - S1 + - S2 + - S3 + - S4 +``` + +#### `query.group_by` + +Define the X-axis of your chart. + +Supported values are: + +- `day`: Group data per day. +- `week`: Group data per week. +- `month`: Group data per month. + +#### `query.period_limit` + +Define how far "issuables" are queried in the past. + +The unit is related to the `query.group_by` you defined. For instance if you +defined `query.group_by: 'day'` then `query.period_limit: 365` would mean +"Gather and display data for the last 365 days". + +If you omit it, default values will be applied depending on the `query.group_by` +you defined. + +| `query.group_by` | Default value | +| ---------------- | ------------- | +| `day` | 30 | +| `week` | 4 | +| `month` | 12 | + +## Complete example + +```yaml +bugsCharts: + title: 'Charts for Bugs' + charts: + - title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + group_by: month + period_limit: 24 + - title: Weekly Bugs By Severity (stacked bar) + type: stacked-bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + collection_labels: + - S1 + - S2 + - S3 + - S4 + group_by: week + period_limit: 104 + - title: Monthly Bugs By Team (line) + type: line + query: + issuable_type: merge_request + issuable_state: opened + filter_labels: + - bug + collection_labels: + - Manage + - Plan + - Create + group_by: month + period_limit: 24 +``` diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md new file mode 100644 index 00000000000..cdb0e34fdf6 --- /dev/null +++ b/doc/user/project/integrations/github.md @@ -0,0 +1,48 @@ +# GitHub project integration **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3836) in GitLab Premium 10.6. + +GitLab provides an integration for updating the pipeline statuses on GitHub. +This is especially useful if using GitLab for CI/CD only. + +This project integration is separate from the [instance wide GitHub integration](../import/github.md#mirroring-and-pipeline-status-sharing) +and is automatically configured on [GitHub import](../../../integration/github.md). + + + +## Configuration + +### Complete these steps on GitHub + +This integration requires a [GitHub API token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) +with `repo:status` access granted: + +1. Go to your "Personal access tokens" page at https://github.com/settings/tokens +1. Click "Generate New Token" +1. Ensure that `repo:status` is checked and click "Generate token" +1. Copy the generated token to use on GitLab + +### Complete these steps on GitLab + +1. Navigate to the project you want to configure. +1. Navigate to the [Integrations page](project_services.md#accessing-the-project-services) +1. Click "GitHub". +1. Select the "Active" checkbox. +1. Paste the token you've generated on GitHub +1. Enter the path to your project on GitHub, such as `https://github.com/username/repository` +1. Optionally check "Static status check names" checkbox to enable static status check names. +1. Save or optionally click "Test Settings". + +#### Static / dynamic status check names + +Since GitLab 11.5 it is possible to opt-in to using static status check names. + +This makes it possible to mark these status checks as _Required_ on GitHub. +If you check "Static status check names" checkbox on the integration page, your +GitLab instance host name is going to be appended to a status check name, +whereas in case of dynamic status check names, a branch name is going to be +appended. + +Dynamic status check name is a default behavior. + + diff --git a/doc/user/project/integrations/img/github_configuration.png b/doc/user/project/integrations/img/github_configuration.png Binary files differnew file mode 100644 index 00000000000..9f2d47921c7 --- /dev/null +++ b/doc/user/project/integrations/img/github_configuration.png diff --git a/doc/user/project/integrations/img/github_status_check_pipeline_update.png b/doc/user/project/integrations/img/github_status_check_pipeline_update.png Binary files differnew file mode 100644 index 00000000000..9d02ca18497 --- /dev/null +++ b/doc/user/project/integrations/img/github_status_check_pipeline_update.png diff --git a/doc/user/project/integrations/img/prometheus_add_metric.png b/doc/user/project/integrations/img/prometheus_add_metric.png Binary files differnew file mode 100644 index 00000000000..e85670e1a13 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_add_metric.png diff --git a/doc/user/project/integrations/img/prometheus_alert.png b/doc/user/project/integrations/img/prometheus_alert.png Binary files differnew file mode 100644 index 00000000000..a37f0477fd9 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_alert.png diff --git a/doc/user/project/integrations/img/prometheus_service_alerts.png b/doc/user/project/integrations/img/prometheus_service_alerts.png Binary files differnew file mode 100644 index 00000000000..a81dfe7da14 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_service_alerts.png diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index 339e6873c41..f560de427c5 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -34,10 +34,12 @@ Click on the service links to see further configuration instructions and details | [Emails on push](emails_on_push.md) | Email the commits and diff of each push to a list of recipients | | External Wiki | Replaces the link to the internal wiki with a link to an external wiki | | Flowdock | Flowdock is a collaboration web app for technical teams | +| [GitHub](github.md) **[PREMIUM]** | Sends pipeline notifications to GitHub | | [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | | [HipChat](hipchat.md) | Private group chat and IM | | [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | | [JIRA](jira.md) | JIRA issue tracker | +| [Jenkins](https://docs.gitlab.com/ee/integration/jenkins.html) **[STARTER]** | An extendable open source continuous integration server | | JetBrains TeamCity CI | A continuous integration and build server | | [Mattermost slash commands](mattermost_slash_commands.md) | Mattermost chat and ChatOps slash commands | | [Mattermost Notifications](mattermost.md) | Receive event notifications in Mattermost | diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 43a9e24526d..40113624430 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -13,7 +13,7 @@ There are two ways to set up Prometheus integration, depending on where your app - For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes). - For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). -Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-cicd-environments). +Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-cicd-environments). You are also able to [add your own metrics](#adding-additional-metrics-premium) as well. ## Enabling Prometheus Integration @@ -93,6 +93,85 @@ GitLab will automatically scan the Prometheus server for metrics from known serv You can view the performance dashboard for an environment by [clicking on the monitoring button](../../../ci/environments.md#monitoring-environments). +### Adding additional metrics **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3799) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.6. + +Additional metrics can be monitored by adding them on the Prometheus integration page. Once saved, they will be displayed on the environment performance dashboard. + + + +A few fields are required: + +- **Name**: Chart title +- **Type**: Type of metric. Metrics of the same type will be shown together. +- **Query**: Valid [PromQL query](https://prometheus.io/docs/prometheus/latest/querying/basics/). +- **Y-axis label**: Y axis title to display on the dashboard. +- **Unit label**: Query units, for example `req / sec`. Shown next to the value. + +Multiple metrics can be displayed on the same chart if the fields **Name**, **Type**, and **Y-axis label** match between metrics. For example, a metric with **Name** `Requests Rate`, **Type** `Business`, and **Y-axis label** `rec / sec` would display on the same chart as a second metric with the same values. A **Legend label** is suggested if this feature used. + +#### Query Variables + +GitLab supports a limited set of [CI variables](../../../ci/variables/README.html) in the Prometheus query. This is particularly useful for identifying a specific environment, for example with `CI_ENVIRONMENT_SLUG`. The supported variables are: + +- CI_ENVIRONMENT_SLUG +- KUBE_NAMESPACE + +To specify a variable in a query, enclose it in curly braces with a leading percent. For example: `%{ci_environment_slug}`. + +### Setting up alerts for Prometheus metrics **[ULTIMATE]** + +#### Managed Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](#adding-additional-metrics-premium), and 11.3 for [library metrics](prometheus_library/metrics.md). + +For managed Prometheus instances using auto configuration, alerts for metrics [can be configured](#adding-additional-metrics-premium) directly in the performance dashboard. + +To set an alert, click on the alarm icon in the top right corner of the metric you want to create the alert for. A dropdown +will appear, with options to set the threshold and operator. Click **Add** to save and activate the alert. + + + +To remove the alert, click back on the alert icon for the desired metric, and click **Delete**. + +#### External Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8. + +For manually configured Prometheus servers, a notify endpoint is provided to use with Prometheus webhooks. If you have manual configuration enabled, an **Alerts** section is added to **Settings > Integrations > Prometheus**. This contains the *URL* and *Authorization Key*. The **Reset Key** button will invalidate the key and generate a new one. + + + +To send GitLab alert notifications, copy the *URL* and *Authorization Key* into the [`webhook_configs`](https://prometheus.io/docs/alerting/configuration/#webhook_config) section of your Prometheus Alertmanager configuration: + +```yaml +receivers: + name: gitlab + webhook_configs: + - http_config: + bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 + send_resolved: true + url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json + ... +``` + +### Taking action on incidents **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. + +Alerts can be used to trigger actions, like open an issue automatically. To configure the actions: + +1. Navigate to your project's **Settings > Operations > Incidents**. +1. Enable the option to create issues. +1. Choose the [issue template](../description_templates.md) to create the issue from. +1. Optionally, select whether to send an email notification to the developers of the project. +1. Click **Save changes**. + +Once enabled, an issue will be opened automatically when an alert is triggered. To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template. + +If the metric exceeds the threshold of the alert for over 5 minutes, an email will be sent to all [Maintainers and Owners](../../permissions.md#project-members-permissions) of the project. + ## Determining the performance impact of a merge > [Introduced][ce-10408] in GitLab 9.2. diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 7a45c87ada0..436129f1dbc 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -1,6 +1,6 @@ # Monitoring Kubernetes -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0. GitLab has support for automatically detecting and monitoring Kubernetes metrics. @@ -35,3 +35,25 @@ Prometheus needs to be deployed into the cluster and configured properly in orde In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available. Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with [CI_ENVIRONMENT_SLUG](../../../../ci/variables/README.md#predefined-environment-variables). It can be followed by a `-` and additional content if desired. For example, a deployment name of `review-homepage-5620p5` would match the `review/homepage` environment. + +## Displaying Canary metrics **[PREMIUM]** + +> Introduced in [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15201). + +GitLab also gathers Kubernetes metrics for [canary deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html), allowing easy comparison between the current deployed version and the canary. + +These metrics expect the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name to begin with `$CI_ENVIRONMENT_SLUG-canary`, to isolate the canary metrics. + +### Canary metrics supported + +- Average Memory Usage (MB) + + ``` + avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 + ``` + +- Average CPU Utilization (%) + + ``` + avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) + ``` diff --git a/doc/user/project/issues/create_new_issue.md b/doc/user/project/issues/create_new_issue.md index 9a147deecd4..5e05846b77f 100644 --- a/doc/user/project/issues/create_new_issue.md +++ b/doc/user/project/issues/create_new_issue.md @@ -65,6 +65,26 @@ In GitLab 11.7, we updated the format of the generated email address. However the older format is still supported, allowing existing aliases or contacts to continue working._ +## New issue via Service Desk **[PREMIUM]** + +Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) to your project and offer email support. +By doing so, when your customer sends a new email, a new issue can be created in +the appropriate project and followed up from there. + +## New issue from the group-level Issue Tracker + +Head to the Group dashboard and click "Issues" in the sidebar to visit the Issue Tracker +for all projects in your Group. Select the project you'd like to add an issue for +using the dropdown button at the top-right of the page. + + + +We'll keep track of the project you selected most recently, and use it as the default +for your next visit. This should save you a lot of time and clicks, if you mostly +create issues for the same project. + + + ## New issue via URL with prefilled fields You can link directly to the new issue page for a given project, with prefilled diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md new file mode 100644 index 00000000000..56b94585672 --- /dev/null +++ b/doc/user/project/issues/csv_export.md @@ -0,0 +1,77 @@ +# Export Issues to CSV **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1126) in [GitLab Starter 9.0](https://about.gitlab.com/2017/03/22/gitlab-9-0-released/#export-issues-ees-eep). + +Issues can be exported as CSV from GitLab and are sent to your default notification email as an attachment. + +## Overview + +**Export Issues to CSV** enables you and your team to export all the data collected from issues into +a **[comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)** (CSV) file, +which stores tabular data in plain text. + +> _CSVs are a handy way of getting data from one program to another where one program cannot read the other ones normal output._ [Ref](https://www.quora.com/What-is-a-CSV-file-and-its-uses) + +CSV files can be used with any plotter or spreadsheet-based program, such as Microsoft Excel, +Open Office Calc, or Google Spreadsheets. + +## Use cases + +Among numerous use cases for exporting issues for CSV, we can name a few: + +- Make a snapshot of issues for offline analysis or to communicate with other teams who may not be in GitLab +- Create diagrams, graphs, and charts from the CSV data +- Present the data in any other format for auditing or sharing reasons +- Import the issues elsewhere to a system outside of GitLab +- Long-term issues' data analysis with multiple snapshots created along the time +- Use the long-term data to gather relevant feedback given in the issues, and improve your product based on real metrics + +## Choosing which issues to include + +From the issues page you can narrow down which issues to export using the search bar, along with the All/Open/Closed tabs. All issues returned will be exported, including those not shown on the first page. + + + +You will be asked to confirm the number of issues and email address for the export, after which the email will begin being prepared. + + + +## Sorting + +Exported issues are always sorted by `Issue ID`. + +## Format + +> **Time Estimate** and **Time Spent** columns were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2627) in GitLab Starter 10.0. +> +> **Weight** and **Locked** columns were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5300) in GitLab Starter 10.8. + +Data will be encoded with a comma as the column delimiter, with `"` used to quote fields if needed, and newlines to separate rows. The first row will be the headers, which are listed in the following table along with a description of the values: + + +| Column | Description | +|---------|-------------| +| Issue ID | Issue `iid` | +| URL | A link to the issue on GitLab | +| Title | Issue `title` | +| State | `Open` or `Closed` | +| Description | Issue `description` | +| Author | Full name of the issue author | +| Author Username | Username of the author, with the `@` symbol omitted | +| Assignee | Full name of the issue assignee | +| Assignee Username | Username of the author, with the `@` symbol omitted | +| Confidential | `Yes` or `No` | +| Locked | `Yes` or `No` | +| Due Date | Formated as `YYYY-MM-DD` | +| Created At (UTC) | Formated as `YYYY-MM-DD HH:MM:SS` | +| Updated At (UTC) | Formated as `YYYY-MM-DD HH:MM:SS` | +| Milestone | Title of the issue milestone | +| Weight | Issue weight | +| Labels | Title of any labels joined with a `,` | +| Time Estimate | [Time estimate](../../../workflow/time_tracking.md#estimates) in seconds | +| Time Spent | [Time spent](../../../workflow/time_tracking.md#time-spent) in seconds | + + +## Limitations + +As the issues will be sent as an email attachment, there is a limit on how much data can be exported. Currently this limit is 20MB to ensure successful delivery across a range of email providers. If this limit is reached we suggest narrowing the search before export, perhaps by exporting open and closed issues separately. diff --git a/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png b/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..8996490ae63 --- /dev/null +++ b/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png diff --git a/doc/user/project/issues/img/csv_export_button.png b/doc/user/project/issues/img/csv_export_button.png Binary files differnew file mode 100644 index 00000000000..f9fcfd71c2f --- /dev/null +++ b/doc/user/project/issues/img/csv_export_button.png diff --git a/doc/user/project/issues/img/csv_export_modal.png b/doc/user/project/issues/img/csv_export_modal.png Binary files differnew file mode 100644 index 00000000000..f988deec966 --- /dev/null +++ b/doc/user/project/issues/img/csv_export_modal.png diff --git a/doc/user/project/issues/img/multiple_assignees.gif b/doc/user/project/issues/img/multiple_assignees.gif Binary files differnew file mode 100644 index 00000000000..055a0efd9ae --- /dev/null +++ b/doc/user/project/issues/img/multiple_assignees.gif diff --git a/doc/user/project/issues/img/multiple_assignees_for_issues.png b/doc/user/project/issues/img/multiple_assignees_for_issues.png Binary files differnew file mode 100644 index 00000000000..e8ae37d427d --- /dev/null +++ b/doc/user/project/issues/img/multiple_assignees_for_issues.png diff --git a/doc/user/project/issues/img/related_issues_add.png b/doc/user/project/issues/img/related_issues_add.png Binary files differnew file mode 100644 index 00000000000..f59d2335386 --- /dev/null +++ b/doc/user/project/issues/img/related_issues_add.png diff --git a/doc/user/project/issues/img/related_issues_remove.png b/doc/user/project/issues/img/related_issues_remove.png Binary files differnew file mode 100644 index 00000000000..be2ec59e61b --- /dev/null +++ b/doc/user/project/issues/img/related_issues_remove.png diff --git a/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png b/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..97d93620b2a --- /dev/null +++ b/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md index 653bd94e513..ef9fcaec3e6 100644 --- a/doc/user/project/issues/issue_data_and_actions.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -50,7 +50,7 @@ where there is shared ownership of an issue. In [GitLab Starter](https://about.gitlab.com/pricing/), you can assign multiple people to an issue. -Learn more in the [Multiple Assignees documentation](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html). +Learn more in the [Multiple Assignees documentation](multiple_assignees_for_issues.md). #### 4. Milestone @@ -103,7 +103,7 @@ Learn more in the [Issue Weight documentation](https://docs.gitlab.com/ee/workfl - Unsubscribe: if you are receiving notifications on that issue but no longer want to receive them, unsubscribe from it. -Read more in the [notifications documentation](../../../workflow/notifications.md#issue--merge-request-events). +Read more in the [notifications documentation](https://docs.gitlab.com/ee/workflow/notifications.html#issue--epics--merge-request-events). #### 11. Reference diff --git a/doc/user/project/issues/multiple_assignees_for_issues.md b/doc/user/project/issues/multiple_assignees_for_issues.md new file mode 100644 index 00000000000..8781ebdd5b0 --- /dev/null +++ b/doc/user/project/issues/multiple_assignees_for_issues.md @@ -0,0 +1,41 @@ +# Multiple Assignees for Issues **[STARTER]** + +> **Note:** +[Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1904) +in [GitLab Starter 9.2](https://about.gitlab.com/2017/05/22/gitlab-9-2-released/#multiple-assignees-for-issues). + +## Overview + +In large teams, where there is shared ownership of an issue, it can be difficult +to track who is working on it, who already completed their contributions, who +didn't even start yet. + +In [GitLab Enterprise Edition](https://about.gitlab.com/pricing/), +you can also select multiple assignees to an issue, making it easier to +track, and making clearer who is accountable for it. + + + +## Use cases + +Consider a team formed by frontend developers, backend developers, +UX designers, QA testers, and a product manager working together to bring an idea to +market. + +Multiple Assignees for Issues makes collaboration smother, +and allows shared responsibilities to be clearly displayed. +All assignees are shown across your team's workflows and receive notifications (as they +would as single assignees), simplifying communication and ownership. + +Once an assignee had their work completed, they would remove themselves as assignees, making +it clear that their role is complete. + +## How it works + +From an opened issue, expand the right sidebar, locate the assignees entry, +and click on **Edit**. From the dropdown menu, select as many users as you want +to assign the issue to. + + + +An assignee can be easily removed by deselecting them from the same dropdown menu. diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md new file mode 100644 index 00000000000..db0ab65b442 --- /dev/null +++ b/doc/user/project/issues/related_issues.md @@ -0,0 +1,45 @@ +# Related issues **[STARTER]** + +> [Introduced][ee-1797] in [GitLab Starter][ee] 9.4. + +Related issues are a bi-directional relationship between any two issues +and appear in a block below the issue description. Issues can be across groups +and projects. + +The relationship only shows up in the UI if the user can see both issues. + +## Adding a related issue + +You can relate one issue to another by clicking the related issues "+" button +in the header of the related issue block. Then, input the issue reference number +or paste in the full URL of the issue. + +Issues of the same project can be specified just by the reference number. +Issues from a different project require additional information like the +group and the project name. For example: + +- same project: `#44` +- same group: `project#44 ` +- different group: `group/project#44` + +Valid references will be added to a temporary list that you can review. +When ready, click the green "Add related issues" button to submit. + + + +## Removing a related issue + +In the related issues block, click the "x" icon on the right-side of each issue +token that you wish to remove. Due to the bi-directional relationship, it +will no longer appear in either issue. + + + +Please access our [permissions] page for more information. + +Additionally, you are also able to manage related issues through [our API]. + +[ee]: https://about.gitlab.com/pricing/ +[ee-1797]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1797 +[permissions]: ../../permissions.md +[Our API]: https://docs.gitlab.com/ee/api/issue_links.html diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 73d61bedde1..9003018a521 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -8,9 +8,48 @@ Labels allow you to categorize issues or merge requests using descriptive titles In GitLab, you can create project and group labels: -- **Project labels** can be assigned to issues or merge requests in that project only. +- **Project labels** can be assigned to issues or merge requests in that project only. - **Group labels** can be assigned to any issue or merge request of any project in that group or any subgroups of the group. +## Scoped labels **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9175) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. + +Scoped labels allow teams to use the simple and familiar feature of labels to +annotate their issues, merge requests, and epics to achieve custom fields and +custom workflow states by leveraging a special label title syntax. + +A scoped label is a kind of label defined only by a special double-colon syntax +in the label’s title, using the format `key::value`. For example: + + + +Two scoped labels with the same key but a different value cannot simultaneously +apply to an issue, epic, or merge request. For example, if an issue already has `priority::3` +and you apply `priority::2` to it, `priority::3` is automatically removed from the issue. + +An issue, epic, or merge request cannot have two scoped labels with the same key. +For example, if an issue is already labeled `priority::3` and you apply the label `priority::2` to it, +`priority::3` is automatically removed. + +### Workflows with scoped labels **[PREMIUM]** + +Suppose you wanted a custom field in issues to track the platform operating system +that your features target, where each issue should only target one platform. You +would then create labels `platform::iOS`, `platform::Android`, `platform::Linux`, +etc., as necessary. Applying any one of these labels on a given issue would +automatically remove any other existing label that starts with `platform::`. + +The same pattern could be applied to represent the workflow states of your teams. +Suppose you have the labels `workflow::development`, `workflow::review`, and +`workflow::deployed`. If an issue already has the label `workflow::development` +applied, and a developer wanted to advance the issue to `workflow::review`, they +would simply apply that label, and the `workflow::development` label would +automatically be removed. This behavior already exists when you move issues +across label lists in an [issue board](issue_board.md#creating-workflows), but +now, team members who may not be working in an issue board directly would still +be able to advance workflow states consistently in issues themselves. + ## Creating labels >**Note:** @@ -35,6 +74,9 @@ GitLab will add the following default labels to the project: To create a **group label**, follow similar steps from above to project labels. Navigate to **Issues > Labels** in the group and create it from there. This page only shows group labels in this group. +Alternatively, you can create group labels also from Epic sidebar. Please note that the created label will belong to the immediate group to which epic belongs. + + Group labels appear in every label list page of the group's child projects. @@ -81,7 +123,7 @@ top-right: GitLab will consider the label title and description for the search. -## Filtering issues and merge requests by label +## Filtering issues, merge requests and epics by label ### Filtering in list pages @@ -89,11 +131,16 @@ From the project issue list page and the project merge request list page, you ca From the group issue list page and the group merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels (including subgroup ancestors and subgroup descendants) and project labels. +From the group epic list page, you can [filter](../search/index.md#issues-and-merge-requests) by both current group labels as well as decendent group labels. +  ### Filtering in issue boards - From [project boards](issue_board.md), you can filter by both group labels and project labels in the [search and filter bar](../search/index.md#issue-boards). +- From [group issue boards](issue_board.md#group-issue-boards-premium), you can filter by only group labels in the [search and filter bar](../search/index.md#issue-boards). **[PREMIUM]** +- From [project boards](issue_board.md), you can filter by both group labels and project labels in the [issue board configuration](issue_board.md#configurable-issue-boards-starter). **[STARTER]** +- From [group issue boards](issue_board.md#group-issue-boards-premium), you can filter by only group labels in the [issue board configuration](issue_board.md#configurable-issue-boards-starter). **[STARTER]** ## Subscribing to labels diff --git a/doc/user/project/maven_packages.md b/doc/user/project/maven_packages.md new file mode 100644 index 00000000000..d32d6084b38 --- /dev/null +++ b/doc/user/project/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/project/packages/maven_repository.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/project/packages/maven_repository.html). diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md new file mode 100644 index 00000000000..65ee2e128ae --- /dev/null +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -0,0 +1,54 @@ +# Browser Performance Testing **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3507) +in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. + +## Overview + +If your application offers a web interface and you are using +[GitLab CI/CD](../../../ci/README.md), you can quickly determine the performance +impact of pending code changes. + +GitLab uses [Sitespeed.io](https://www.sitespeed.io), a free and open source +tool for measuring the performance of web sites, and has built a simple +[Sitespeed plugin](https://gitlab.com/gitlab-org/gl-performance) +which outputs the results in a file called `performance.json`. This plugin +outputs the performance score for each page that is analyzed. + +The [Sitespeed.io performance score](http://examples.sitespeed.io/6.0/2017-11-23-23-43-35/help.html) +is a composite value based on best practices, and we will be expanding support +for [additional metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/4370) +in a future release. + +Going a step further, GitLab can show the Performance report right +in the merge request widget area: + +## Use cases + +For instance, consider the following workflow: + +1. A member of the marketing team is attempting to track engagement by adding a new tool +1. With browser performance metrics, they see how their changes are impacting the usability of the page for end users +1. The metrics show that after their changes the performance score of the page has gone down +1. When looking at the detailed report, they see that the new Javascript library was included in `<head>` which affects loading page speed +1. They ask a front end developer to help them, who sets the library to load asynchronously +1. The frontend developer approves the merge request and authorizes its deployment to production + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Performance report artifact](../../../ci/yaml/README.md#artifactsreportsperformance-premium). +For more information on how the Performance job should look like, check the +example on [Testing Browser Performance](../../../ci/examples/browser_performance.md). + +GitLab then checks this report, compares key performance metrics for each page +between the source and target branches, and shows the information right on the merge request. + +>**Note:** +If the Performance report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +Performance job in your `.gitlab-ci.yml` for the very first time. +Consecutive merge requests will have something to compare to and the Performance +report will be shown properly. + + diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md new file mode 100644 index 00000000000..e6811b5df5e --- /dev/null +++ b/doc/user/project/merge_requests/code_quality.md @@ -0,0 +1,82 @@ +# Code Quality **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1984) +in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your +source code quality using GitLab Code Quality. +Code Quality uses [Code Climate Engines](https://codeclimate.com), which are +free and open source. Code Quality doesn’t require a Code Climate subscription. + +Going a step further, GitLab can show the Code Quality report right +in the merge request widget area: + + + +## Use cases + +For instance, consider the following workflow: + +1. Your backend team member starts a new implementation for making certain feature in your app faster +1. With Code Quality reports, they analyze how their implementation is impacting the code quality +1. The metrics show that their code degrade the quality in 10 points +1. You ask a co-worker to help them with this modification +1. They both work on the changes until Code Quality report displays no degradations, only improvements +1. You approve the merge request and authorize its deployment to staging +1. Once verified, their changes are deployed to production + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Code Quality report artifact](../../../ci/yaml/README.md#artifactsreportscodequality-starter). + +The Code Quality report artifact is a subset of the +[Code Climate spec](https://github.com/codeclimate/spec/blob/master/SPEC.md#data-types). +It must be a JSON file containing an array of objects with the following properties: + +| Name | Description | +| ---------------------- | -------------------------------------------------------------------------------------- | +| `description` | A description of the code quality violation. | +| `fingerprint` | A unique fingerprint to identify the code quality violation. For example, an MD5 hash. | +| `location.path` | The relative path to the file containing the code quality violation. | +| `location.lines.begin` | The line on which the code quality violation occurred. | + +Example: + +```json +[ + { + "description": "'unused' is assigned a value but never used.", + "fingerprint": "7815696ecbf1c96e6894b779456d330e", + "location": { + "path": "lib/index.js", + "lines": { + "begin": 42 + } + } + } +] +``` + +NOTE: **Note:** +Although the Code Climate spec supports more properties, those are ignored by GitLab. + +For more information on how the Code Quality job should look like, check the +example on [analyzing a project's code quality](../../../ci/examples/code_quality.md). + +GitLab then checks this report, compares the metrics between the source and target +branches, and shows the information right on the merge request. + +CAUTION: **Caution:** +If multiple jobs in a pipeline generate a code quality artifact, only the artifact from +the last created job (the job with the largest job ID) is used. To avoid confusion, +configure only one job to generate a code quality artifact. + +NOTE: **Note:** +If the Code Quality report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +Code Quality job in your `.gitlab-ci.yml` for the very first time. +Consecutive merge requests will have something to compare to and the Code Quality +report will be shown properly. diff --git a/doc/user/project/merge_requests/code_quality_diff.md b/doc/user/project/merge_requests/code_quality_diff.md new file mode 100644 index 00000000000..890058eec6f --- /dev/null +++ b/doc/user/project/merge_requests/code_quality_diff.md @@ -0,0 +1,6 @@ +--- +redirect_from: 'https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html' +redirect_to: 'code_quality.md' +--- + +This document was moved to [another location](code_quality.md). diff --git a/doc/user/project/merge_requests/container_scanning.md b/doc/user/project/merge_requests/container_scanning.md new file mode 100644 index 00000000000..4d41e424f4a --- /dev/null +++ b/doc/user/project/merge_requests/container_scanning.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html). diff --git a/doc/user/project/merge_requests/dast.md b/doc/user/project/merge_requests/dast.md new file mode 100644 index 00000000000..b676c661267 --- /dev/null +++ b/doc/user/project/merge_requests/dast.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dast/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dast/index.html). diff --git a/doc/user/project/merge_requests/dependency_scanning.md b/doc/user/project/merge_requests/dependency_scanning.md new file mode 100644 index 00000000000..3a8b53b425c --- /dev/null +++ b/doc/user/project/merge_requests/dependency_scanning.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). diff --git a/doc/user/project/merge_requests/img/approvals_can_override.png b/doc/user/project/merge_requests/img/approvals_can_override.png Binary files differnew file mode 100644 index 00000000000..8d207d018e0 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_can_override.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png Binary files differnew file mode 100644 index 00000000000..b6dc86f312e --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png Binary files differnew file mode 100644 index 00000000000..b6f6188b9cd --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png diff --git a/doc/user/project/merge_requests/img/approvals_remove_on_push.png b/doc/user/project/merge_requests/img/approvals_remove_on_push.png Binary files differnew file mode 100644 index 00000000000..73964827587 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_remove_on_push.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_edit.png b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png Binary files differnew file mode 100644 index 00000000000..868b9d58740 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_empty.png b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png Binary files differnew file mode 100644 index 00000000000..7375820224c --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png diff --git a/doc/user/project/merge_requests/img/approve.png b/doc/user/project/merge_requests/img/approve.png Binary files differnew file mode 100644 index 00000000000..e68259ac5c2 --- /dev/null +++ b/doc/user/project/merge_requests/img/approve.png diff --git a/doc/user/project/merge_requests/img/approve_additionally.png b/doc/user/project/merge_requests/img/approve_additionally.png Binary files differnew file mode 100644 index 00000000000..3db5a9159e5 --- /dev/null +++ b/doc/user/project/merge_requests/img/approve_additionally.png diff --git a/doc/user/project/merge_requests/img/browser_performance_testing.png b/doc/user/project/merge_requests/img/browser_performance_testing.png Binary files differnew file mode 100644 index 00000000000..eea77fb8b93 --- /dev/null +++ b/doc/user/project/merge_requests/img/browser_performance_testing.png diff --git a/doc/user/project/merge_requests/img/code_quality.gif b/doc/user/project/merge_requests/img/code_quality.gif Binary files differnew file mode 100644 index 00000000000..bab921cf38b --- /dev/null +++ b/doc/user/project/merge_requests/img/code_quality.gif diff --git a/doc/user/project/merge_requests/img/container_scanning.png b/doc/user/project/merge_requests/img/container_scanning.png Binary files differnew file mode 100644 index 00000000000..e47f62acd9d --- /dev/null +++ b/doc/user/project/merge_requests/img/container_scanning.png diff --git a/doc/user/project/merge_requests/img/create-issue-with-list-hover.png b/doc/user/project/merge_requests/img/create-issue-with-list-hover.png Binary files differnew file mode 100644 index 00000000000..7d70e8299f5 --- /dev/null +++ b/doc/user/project/merge_requests/img/create-issue-with-list-hover.png diff --git a/doc/user/project/merge_requests/img/dast_all.png b/doc/user/project/merge_requests/img/dast_all.png Binary files differnew file mode 100644 index 00000000000..b6edc928dc3 --- /dev/null +++ b/doc/user/project/merge_requests/img/dast_all.png diff --git a/doc/user/project/merge_requests/img/dast_single.png b/doc/user/project/merge_requests/img/dast_single.png Binary files differnew file mode 100644 index 00000000000..26ca4bde786 --- /dev/null +++ b/doc/user/project/merge_requests/img/dast_single.png diff --git a/doc/user/project/merge_requests/img/dependency_scanning.png b/doc/user/project/merge_requests/img/dependency_scanning.png Binary files differnew file mode 100644 index 00000000000..18df356f846 --- /dev/null +++ b/doc/user/project/merge_requests/img/dependency_scanning.png diff --git a/doc/user/project/merge_requests/img/filter_approver_merge_requests.png b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png Binary files differnew file mode 100644 index 00000000000..9c386391a4f --- /dev/null +++ b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png diff --git a/doc/user/project/merge_requests/img/interactive_reports.png b/doc/user/project/merge_requests/img/interactive_reports.png Binary files differnew file mode 100644 index 00000000000..9f9812dc69d --- /dev/null +++ b/doc/user/project/merge_requests/img/interactive_reports.png diff --git a/doc/user/project/merge_requests/img/license_management.png b/doc/user/project/merge_requests/img/license_management.png Binary files differnew file mode 100644 index 00000000000..cdce6b5fe38 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management.png diff --git a/doc/user/project/merge_requests/img/license_management_decision.png b/doc/user/project/merge_requests/img/license_management_decision.png Binary files differnew file mode 100644 index 00000000000..0763130c375 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management_decision.png diff --git a/doc/user/project/merge_requests/img/license_management_pipeline_tab.png b/doc/user/project/merge_requests/img/license_management_pipeline_tab.png Binary files differnew file mode 100644 index 00000000000..80ffca815b9 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management_pipeline_tab.png diff --git a/doc/user/project/merge_requests/img/license_management_settings.png b/doc/user/project/merge_requests/img/license_management_settings.png Binary files differnew file mode 100644 index 00000000000..b5490e59074 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management_settings.png diff --git a/doc/user/project/merge_requests/img/remove_approval.png b/doc/user/project/merge_requests/img/remove_approval.png Binary files differnew file mode 100644 index 00000000000..6083e1745ef --- /dev/null +++ b/doc/user/project/merge_requests/img/remove_approval.png diff --git a/doc/user/project/merge_requests/img/sast.png b/doc/user/project/merge_requests/img/sast.png Binary files differnew file mode 100644 index 00000000000..2c75592c32a --- /dev/null +++ b/doc/user/project/merge_requests/img/sast.png diff --git a/doc/user/project/merge_requests/img/security_report.png b/doc/user/project/merge_requests/img/security_report.png Binary files differnew file mode 100644 index 00000000000..ba41b707238 --- /dev/null +++ b/doc/user/project/merge_requests/img/security_report.png diff --git a/doc/user/project/merge_requests/img/vulnerability_solution.png b/doc/user/project/merge_requests/img/vulnerability_solution.png Binary files differnew file mode 100644 index 00000000000..7443b9b6eea --- /dev/null +++ b/doc/user/project/merge_requests/img/vulnerability_solution.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 2765a32c845..2bb2d906453 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -23,7 +23,7 @@ With GitLab merge requests, you can: - Assign it to any registered user, and change the assignee how many times you need - Assign a [milestone](../../project/milestones/index.md) and track the development of a broader implementation - Organize your issues and merge requests consistently throughout the project with [labels](../../project/labels.md) -- Add a time estimation and the time spent with that merge request with [Time Tracking](../../../workflow/time_tracking.html#time-tracking) +- Add a time estimation and the time spent with that merge request with [Time Tracking](../../../workflow/time_tracking.md#time-tracking) - [Resolve merge conflicts from the UI](#resolve-conflicts) - Enable [fast-forward merge requests](#fast-forward-merge-requests) - Enable [semi-linear history merge requests](#semi-linear-history-merge-requests) as another security layer to guarantee the pipeline is passing in the target branch @@ -33,9 +33,16 @@ With GitLab merge requests, you can: With **[GitLab Enterprise Edition][ee]**, you can also: -- View the deployment process across projects with [Multi-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html#multi-project-pipeline-graphs) **[PREMIUM]** -- Request [approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers **[STARTER]** -- Analyze the impact of your changes with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** +- Prepare a full review and submit it once it's ready with [Merge Request Reviews](https://docs.gitlab.com/ee/user/discussions/index.md#merge-request-reviews-premium) **[PREMIUM]** +- View the deployment process across projects with [Multi-Project Pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.md) **[PREMIUM]** +- Request [approvals](merge_request_approvals.md) from your managers **[STARTER]** +- Analyze the impact of your changes with [Code Quality reports](code_quality.md) **[STARTER]** +- Manage the licenses of your dependencies with [License Management](https://docs.gitlab.com/ee/user/application_security/license_management/index.md) **[ULTIMATE]** +- Analyze your source code for vulnerabilities with [Static Application Security Testing](https://docs.gitlab.com/ee/user/application_security/sast/index.md) **[ULTIMATE]** +- Analyze your running web applications for vulnerabilities with [Dynamic Application Security Testing](https://docs.gitlab.com/ee/user/application_security/dast/index.md) **[ULTIMATE]** +- Analyze your dependencies for vulnerabilities with [Dependency Scanning](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.md) **[ULTIMATE]** +- Analyze your Docker images for vulnerabilities with [Container Scanning](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.md) **[ULTIMATE]** +- Determine the performance impact of changes with [Browser Performance Testing](#browser-performance-testing-premium) **[PREMIUM]** ## Use cases @@ -43,19 +50,21 @@ A. Consider you are a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team +1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) **[STARTER]** 1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD -1. You request the approval from your manager -1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) +1. You avoid using dependencies whose license is not compatible with your project with [License Management reports](license_management.md) **[ULTIMATE]** +1. You request the [approval](#merge-request-approvals-starter) from your manager +1. Your manager pushes a commit with their final review, [approves the merge request](merge_request_approvals.md), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#whenmanual) for GitLab CI/CD 1. Your implementations were successfully shipped to your customer -B. Consider you're a web developer writing a webpage for your company's: +B. Consider you're a web developer writing a webpage for your company's website: 1. You checkout a new branch, and submit a new page through a merge request 1. You gather feedback from your reviewers 1. Your changes are previewed with [Review Apps](../../../ci/review_apps/index.md) 1. You request your web designers for their implementation -1. You request the [approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your manager **[STARTER]** +1. You request the [approval](merge_request_approvals.md) from your manager **[STARTER]** 1. Once approved, your merge request is [squashed and merged](squash_and_merge.md), and [deployed to staging with GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) 1. Your production team [cherry picks](#cherry-pick-changes) the merge commit into production @@ -159,6 +168,21 @@ in a Merge Request. To do so, click the **...** button in the gutter of the Merg  +## Perform a Review **[PREMIUM]** + +Start a review in order to create multiple comments on a diff and publish them once you're ready. +Starting a review allows you to get all your thoughts in order and ensure you haven't missed anything +before submitting all your comments. + +[Learn more about Merge Request Reviews](https://docs.gitlab.com/ee/user/discussions/index.html#merge-request-reviews-premium) + +## Squash and merge + +GitLab allows you to squash all changes present in a merge request into a single +commit when merging, to allow for a neater commit history. + +[Learn more about squash and merge.](squash_and_merge.md) + ## Suggest changes > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/18008) in GitLab 11.6. @@ -182,7 +206,7 @@ To assign multiple assignees to a merge request: 1. From a merge request, expand the right sidebar and locate the **Assignees** section. 1. Click on **Edit** and from the dropdown menu, select as many users as you want -to assign the merge request to. + to assign the merge request to. Similarly, assignees are removed by deselecting them from the same dropdown menu. @@ -336,6 +360,52 @@ have been marked as a **Work In Progress**. [Learn more about setting a merge request as "Work In Progress".](work_in_progress_merge_requests.md) +## Merge request approvals **[STARTER]** + +> Included in [GitLab Starter][products]. + +If you want to make sure every merge request is approved by one or more people, +you can enforce this workflow by using merge request approvals. Merge request +approvals allow you to set the number of necessary approvals and predefine a +list of approvers that will need to approve every merge request in a project. + +[Read more about merge request approvals.](merge_request_approvals.md) + +## Code Quality **[STARTER]** + +> Introduced in [GitLab Starter][products] 9.3. + +If you are using [GitLab CI][ci], you can analyze your source code quality using +the [Code Climate][cc] analyzer [Docker image][cd]. Going a step further, GitLab +can show the Code Climate report right in the merge request widget area. + +[Read more about Code Quality reports.](code_quality.md) + +## Browser Performance Testing **[PREMIUM]** + +> Introduced in [GitLab Premium][products] 10.3. + +If your application offers a web interface and you are using [GitLab CI/CD][ci], you can quickly determine the performance impact of pending code changes. GitLab uses [Sitespeed.io][sitespeed], a free and open source tool for measuring the performance of web sites, to analyze the performance of specific pages. + +GitLab runs the [Sitespeed.io container][sitespeed-container] and displays the difference in overall performance scores between the source and target branches. + +[Read more about Browser Performance Testing.](browser_performance_testing.md) + +## Security reports **[ULTIMATE]** + +GitLab can scan and report any vulnerabilities found in your project. + +[Read more about security reports.](https://docs.gitlab.com/ee/user/application_security/index.html) + +## Live preview with Review Apps + +If you configured [Review Apps](https://about.gitlab.com/features/review-apps/) for your project, +you can preview the changes submitted to a feature-branch through a merge request +in a per-branch basis. No need to checkout the branch, install and preview locally; +all your changes will be available to preview by anyone with the Review Apps link. + +[Read more about Review Apps.](../../../ci/review_apps/index.md) + ## Merge request diff file navigation When reviewing changes in the **Changes** tab the diff can be navigated using @@ -528,5 +598,11 @@ And to check out a particular merge request: git checkout origin/merge-requests/1 ``` +[products]: https://about.gitlab.com/products/ "GitLab products page" [protected branches]: ../protected_branches.md +[ci]: ../../../ci/README.md +[cc]: https://codeclimate.com/ +[cd]: https://hub.docker.com/r/codeclimate/codeclimate/ +[sitespeed]: https://www.sitespeed.io +[sitespeed-container]: https://hub.docker.com/r/sitespeedio/sitespeed.io/ [ee]: https://about.gitlab.com/pricing/ "GitLab Enterprise Edition" diff --git a/doc/user/project/merge_requests/license_management.md b/doc/user/project/merge_requests/license_management.md new file mode 100644 index 00000000000..08704425a75 --- /dev/null +++ b/doc/user/project/merge_requests/license_management.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/license_management/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/license_management/index.html). diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md new file mode 100644 index 00000000000..265871a7b4b --- /dev/null +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -0,0 +1,319 @@ +# Merge request approvals **[STARTER]** + +> Introduced in [GitLab Enterprise Edition 7.12](https://about.gitlab.com/2015/06/22/gitlab-7-12-released/#merge-request-approvers-ee-only). + +NOTE: **Note:** +If you are running a self-managed instance, the new interface shown on +this page will not be available unless the feature flag +`approval_rules` is enabled, which can be done from the Rails console by +instance administrators. + +Use these commands to start the Rails console: + +```sh +# Omnibus GitLab +gitlab-rails console + +# Installation from source +cd /home/git/gitlab +sudo -u git -H bin/rails console RAILS_ENV=production +``` + +Then run `Feature.enable(:approval_rules)` to enable the feature flag. + +The documentation for the older interface can be accessed +[here](/11.7/ee/user/project/merge_requests/merge_request_approvals.html). + +## Overview + +Merge request approvals enable enforced code review by requiring specified people to approve a merge request before it can be unblocked for merging. + +## Use cases + +1. Enforcing review of all code that gets merged into a repository. +2. Specifying code maintainers for an entire repository. +3. Specifying reviewers for a given proposed code change. +4. Specifying categories of reviewers, such as BE, FE, QA, DB, etc., for all proposed code changes. + +## Editing approvals + +To edit the merge request approvals: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals**. + +  + +1. Click **Edit**. +1. Search for users or groups that will be [eligible to approve](#eligible-approvers) + merge requests and click the **Add** button to add them as approvers. Note: selecting + approvers is optional. +1. Set the minimum number of required approvals under the **No. approvals required** + box. Note: the minimum can be 0. +1. Click **Update approvers**. + +  + +The steps above are the minimum required to get approvals working in your +merge requests, but there are a couple more options available that might be +suitable to your workflow: + +- Choose whether the default settings can be + [overridden per merge request](#overriding-the-merge-request-approvals-default-settings) +- Choose whether [approvals will be reset with new pushed commits](#resetting-approvals-on-push) + +## Editing approvals **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1979) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +CAUTION: **Caution:** +There was a [regression affecting this feature in 11.8](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9648). We recommend upgrading _at least_ to version 11.8.2. to avoid any issues. + +NOTE: **Note:** +In 11.8 this feature does not work in [private groups](https://gitlab.com/gitlab-org/gitlab-ee/issues/10356). + +For GitLab Premium, [multiple approver rules](#multiple-approval-rules-premium) can be configured. To configure the merge +request approval rules: + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Click **Add approvers** to create a new approval rule. +1. Just like in [GitLab Starter](#editing-approvals), select the approval members and aprovals required. +1. Give the approval rule a name that describes the set of approvers selected. +1. Click **Add approvers** to submit the new rule. + +  + +## Multiple approval rules **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1979) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +For GitLab Premium, a merge request's overall approval status is determined by a set of rules. Each rule contains: + +- A set of [eligible approvers](#eligible-approvers). +- A minimum number of approvals required. + +When an [eligible approver](#eligible-approvers) approves a merge request, it will reduce the number of approvals left for +all rules that the approver belongs to. + + + +If no approval rules are set, then the overall minimum number of approvals required can be configured. With no approval rules, +any [eligible approver](#eligible-approvers) may approve. + +## Eligible approvers + +The following can approve merge requests: + +- Users being added as approvers at project or merge request level. +- [Code owners](https://docs.gitlab.com/ee/user/project/code_owners.html) related to the merge request ([introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7933) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.5). + +An individual user can be added as an approver for a project if they are a member of: + +- The project. +- The project's immediate parent group. +- A group that has access to the project via a [share](../members/share_project_with_groups.md). + +A group can also be added as an approver. [In the future](https://gitlab.com/gitlab-org/gitlab-ee/issues/2048), +group approvers will be restricted. + +If a user is added as an individual approver and is also part of a group approver, +then that user is just counted once. The merge request author, as well as users who have committed +to the merge request, do not count as eligible approvers, +if [**Prevent author approval**](#allowing-merge-request-authors-to-approve-their-own-merge-requests) (enabled by default) +and [**Prevent committers approval**](#prevent-approval-of-merge-requests-by-their-committers) (disabled by default) +are enabled on the project settings. + +### Implicit approvers + +If the number of required approvals is greater than the number of approvers, +other users will become implicit approvers to fill the gap. +Those implicit approvers include members of the given project with Developer role or higher. + +## Adding or removing an approval + +If approvals are activated for the given project, when a user visits an open +merge request, depending on their [eligibility](#eligible-approvers), one of +the following is possible: + +- **They are not an eligible approver**: They cannot do anything with respect + to approving this merge request. + +- **They have not approved this merge request**: + + - If the required number of approvals has _not_ been yet met, they can approve + it by clicking the displayed **Approve** button. +  + - If the required number of approvals has already been met, they can still + approve it by clicking the displayed **Approve additionally** button. +  + +- **They have already approved this merge request**: They can remove their approval. + +  + +NOTE: **Note:** +The merge request author is only allowed to approve their own merge request +if [**Prevent author approval**](#allowing-merge-request-authors-to-approve-their-own-merge-requests) is disabled on the project settings. + +For a given merge request, if the approval restrictions have been satisfied, +the merge request is unblocked and can be merged. +Note, that meeting the required number of approvals is a necessary, but not +sufficient condition for unblocking a merge request from being merged. There +are other conditions that may block it, such as merge conflicts, +[pending discussions](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved) +or a [failed CI/CD pipeline](merge_when_pipeline_succeeds.md). + +## Code Owners approvals **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4418) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.9. + +It is possible to require at least one approval for each entry in the +[`CODEOWNERS` file](https://docs.gitlab.com/ee/user/project/code_owners.html) that matches a file changed in +the merge request. To enable this feature: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals**. +1. Tick the **Require approval from code owners** checkbox + checkbox. +1. Click **Save changes**. + +When this feature is enabled, all merge requests will need approval +from one code owner per matched rule before it can be merged. + +## Overriding the merge request approvals default settings + +> Introduced in GitLab Enterprise Edition 9.4. + +NOTE: **Note:** +If you are using GitLab Premium, things are a little different with [multiple approval rules](#multiple-approval-rules-premium). +Read the differences [in GitLab Premium when overriding merge request approvals](#overriding-merge-request-approvals-default-settings-premium). + +If approvals are [set at the project level](#editing-approvals), the +default configuration (number of required approvals and approvers) can be +overridden for each merge request in that project. + +One possible scenario would be to to assign a group of approvers at the project +level and change them later when creating or editing the merge request. + +First, you have to enable this option in the project's settings: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals** +1. Tick the "Can override approvers and approvals required per merge request" + checkbox + +  + +1. Click **Save changes** + +NOTE: **Note:** +If approver overriding is enabled +and the project level approvers are changed after a merge request is created, +the merge request retains the previous approvers. +However, the approvers can be changed by [editing the merge request](#overriding-the-merge-request-approvals-default-settings). + +--- + +The default approval settings can now be overridden when creating a +[merge request](index.md) or by editing it after it's been created: + +1. Click **Edit** under the **Approvers** section. +1. Search for users or groups that will be [eligible to approve](#eligible-approvers) + merge requests and click the **Add** button to add them as approvers or + remove existing approvers that were set in the project's settings. +1. If you want to change the number of required approvals, set a new number + in the **No. approvals required** box. +1. Click **Update approvers**. + +There are however some restrictions: + +- The amount of required approvals, if changed, must be greater than the default + set at the project level. This ensures that you're not forced to adjust settings + when someone is unavailable for approval, yet the process is still enforced. + +NOTE: **Note:** +If you are contributing to a forked project, things are a little different. +Read what happens when the +[source and target branches are not the same](#merge-requests-with-different-source-branch-and-target-branch-projects). + +## Overriding merge request approvals default settings **[PREMIUM]** + +In GitLab Premium, when the approval rules are [set at the project level](#editing-approvals-premium), and +**Can override approvers and approvals required per merge request** is checked, there are a few more +restrictions (compared to [GitLab Starter](#overriding-the-merge-request-approvals-default-settings)): + +- Approval rules can be added to an MR with no restriction. +- For project sourced approval rules, editing and removing approvers is not allowed. +- The approvals required of all approval rules is configurable, but if a rule is backed by a project rule, then it is restricted +to the minimum approvals required set in the project's corresponding rule. + +## Resetting approvals on push + +If approvals are [set at the project level](#editing-approvals), +you can choose whether all approvals on a merge request are removed when +new commits are pushed to the source branch of the merge request: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals** +1. Tick the "Remove all approvals in a merge request when new commits are pushed to its source branch" + checkbox + +  + +1. Click **Save changes** + +NOTE: **Note:** +Approvals do not get reset when [rebasing a merge request](fast_forward_merge.md) +from the UI. +However, approvals will be reset if the target branch is changed. + +If you want approvals to persist, independent of changes to the merge request, +turn this setting to off by unchecking the box and saving the changes. + +## Allowing merge request authors to approve their own merge requests + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3349) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. + +You can allow merge request authors to self-approve merge requests by +enabling it [at the project level](#editing-approvals). Authors +also need to be included in the approvers list in order to be able to +approve their merge request. + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Uncheck the **Prevent approval of merge requests by merge request author** checkbox, which is enabled by default. +1. Click **Save changes**. + +## Prevent approval of merge requests by their committers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10441) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.10. + +You can prevent users that have committed to a merge request from approving it by +enabling [**Prevent approval of merge requests by their committers**](#prevent-approval-of-merge-requests-by-their-committers). + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Tick the checkbox **Prevent approval of merge requests by their committers**. +1. Click **Save changes**. + +## Merge requests with different source branch and target branch projects + +If the merge request source branch and target branch belong to different +projects (which happens in merge requests in forked projects), everything is +with respect to the target branch's project (typically the original project). +In particular, since the merge request in this case is part of the target +branch's project, the relevant settings are the target project's. The source +branch's project settings are not applicable. Even if you start the merge +request from the source branch's project UI, pay attention to the created merge +request itself. It belongs to the target branch's project. + +## Approver suggestions + +Approvers are suggested for merge requests based on the previous authors of the files affected by the merge request. + +## Filtering merge requests by approvers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9468) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.9. + +To filter merge requests by an individual approver, you can type (or select from +the dropdown) `approver` and select the user. + + diff --git a/doc/user/project/merge_requests/sast.md b/doc/user/project/merge_requests/sast.md new file mode 100644 index 00000000000..688cc79d0f6 --- /dev/null +++ b/doc/user/project/merge_requests/sast.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/sast/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/sast/index.html). diff --git a/doc/user/project/merge_requests/sast_docker.md b/doc/user/project/merge_requests/sast_docker.md new file mode 100644 index 00000000000..4d41e424f4a --- /dev/null +++ b/doc/user/project/merge_requests/sast_docker.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html). diff --git a/doc/user/project/milestones/burndown_charts.md b/doc/user/project/milestones/burndown_charts.md new file mode 100644 index 00000000000..0ad08da8ff5 --- /dev/null +++ b/doc/user/project/milestones/burndown_charts.md @@ -0,0 +1,70 @@ +# Burndown Charts **[STARTER]** + +> **Notes:** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1540) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.1 for project milestones. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5354) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.8 for group milestones. +> - [Added](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6495) to [GitLab Starter](https://about.gitlab.com/pricing/) 11.2 for group milestones. +> - Closed or reopened issues prior to GitLab 9.1 won't have a `closed_at` +> value, so the burndown chart considers them as closed on the milestone +> `start_date`. In that case, a warning will be displayed. + +## Overview + +Burndown Charts are visual representations of the progress of completing a milestone. + + + +At a glance, you see the current state for the completion a given milestone. +Without them, you would have to organize the data from the milestone and plot it +yourself to have the same sense of progress. + +GitLab Starter plots it for you and presents it in a clear and beautiful chart. + +For an overview, check the video demonstration on [Mapping Work Versus Time, With Burndown Charts](https://about.gitlab.com/2017/04/25/mapping-work-to-do-versus-time-with-burndown-charts/). + +## Use cases + +Burndown Charts, in general, are used for tracking and analyzing the completion of +a milestone. Therefore, their use cases are tied to the +[use you are assigning your milestone to](index.md). + +To exemplify, suppose you lead a team of developers in a large company, +and you follow this workflow: + +- Your company set the goal for the quarter to deliver 10 new features for your app + in the upcoming major release. +- You create a milestone, and remind your team to assign that milestone to every new issue + and merge request that's part of the launch of your app. +- Every week, you open the milestone, visualize the progress, identify the gaps, + and help your team to get their work done. +- Every month, you check in with your supervisor, and show the progress of that milestone + from the Burndown Chart. +- By the end of the quarter, your team successfully delivered 100% of that milestone, as + it was taken care of closely throughout the whole quarter. + +## How it works + +A Burndown Chart is available for every project or group milestone that has been attributed a **start +date** and a **due date**. + +Find your project's **Burndown Chart** under **Project > Issues > Milestones**, +and select a milestone from your current ones, while for group's, access the **Groups** dashboard, +select a group, and go through **Issues > Milestones** on the sidebar. + +NOTE: **Note:** +You're able to [promote project](https://docs.gitlab.com/ee/user/project/milestones/#promoting-project-milestones-to-group-milestones) to group milestones and still see the **Burndown Chart** for them, respecting license limitations. + +The chart indicates the project's progress throughout that milestone (for issues assigned to it). + +In particular, it shows how many issues were or are still open for a given day in the +milestone's corresponding period. + +The Burndown Chart tracks when issues were created and when they were last closed—not their full history. For each day, it takes the number of issues still open and issues created that day and subtracts the number of issues closed that day. +**Issues that were created and assigned a milestone before its start date—and remain open as of the start date—are considered as having been opened on the start date**. Therefore, when the milestone start date is changed the number of opened issues on each day may change. +Reopened issues are +considered as having been opened on the day after they were last closed. + +The Burndown Chart can also be toggled to display the cumulative open issue +weight for a given day. When using this feature, make sure issue weights have +been properly assigned, since an open issue with no weight adds zero to the +cumulative value. diff --git a/doc/user/project/milestones/img/burndown_chart.png b/doc/user/project/milestones/img/burndown_chart.png Binary files differnew file mode 100644 index 00000000000..e06b24f9907 --- /dev/null +++ b/doc/user/project/milestones/img/burndown_chart.png diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index a7d6144e3ec..0d8ee0a6cd2 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -83,7 +83,10 @@ From the project issue/merge request list pages and the group issue/merge reques ### Filtering in issue boards -From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [search and filter bar](../../search/index.md#issue-boards). +- From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [search and filter bar](../../search/index.md#issue-boards). +- From [group issue boards](../issue_board.md#group-issue-boards-premium), you can filter by only group milestones in the [search and filter bar](../../search/index.md#issue-boards). **[PREMIUM]** +- From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [issue board configuration](../issue_board.md#configurable-issue-boards-starter). **[STARTER]** +- From [group issue boards](../issue_board.md#group-issue-boards-premium) you can filter by only group milestones in the [issue board configuration](../issue_board.md#configurable-issue-boards-starter). **[STARTER]** ### Special milestone filters @@ -98,16 +101,17 @@ When filtering by milestone, in addition to choosing a specific project mileston Not all features in the project milestone view are available in the group milestone view. This table summarizes the differences: -| Feature | Project milestone view | Group milestone view | -|---|:---:|:---:| -| Title an description | ✓ | ✓ | -| Issues assigned to milestone | ✓ | | -| Merge requests assigned to milestone | ✓ | | -| Participants and labels used | ✓ | | -| Percentage complete | ✓ | ✓ | -| Start date and due date | ✓ | ✓ | -| Total issue time spent | ✓ | ✓ | -| Total issue weight | ✓ | | +| Feature | Project milestone view | Group milestone view | +|--------------------------------------|:----------------------:|:--------------------:| +| Title an description | ✓ | ✓ | +| Issues assigned to milestone | ✓ | | +| Merge requests assigned to milestone | ✓ | | +| Participants and labels used | ✓ | | +| Percentage complete | ✓ | ✓ | +| Start date and due date | ✓ | ✓ | +| Total issue time spent | ✓ | ✓ | +| Total issue weight | ✓ | | +| Burndown chart **[STARTER}** | ✓ | ✓ | The milestone view shows the title and description. @@ -118,6 +122,17 @@ These features are only available for project milestones and not group milestone - Issues assigned to the milestone are displayed in three columns: Unstarted issues, ongoing issues, and completed issues. - Merge requests assigned to the milestone are displayed in four columns: Work in progress merge requests, waiting for merge, rejected, and closed. - Participants and labels that are used in issues and merge requests that have the milestone assigned are displayed. +- [Burndown chart](#project-burndown-charts-starter). + +### Project Burndown Charts **[STARTER]** + +For project milestones in [GitLab Starter](https://about.gitlab.com/pricing), a [burndown chart](burndown_charts.md) is in the milestone view, showing the progress of completing a milestone. + + + +### Group Burndown Charts **[PREMIUM]** + +For group milestones in [GitLab Premium](https://about.gitlab.com/pricing), a [burndown chart](burndown_charts.md) is in the milestone view, showing the progress of completing a milestone. ### Milestone sidebar diff --git a/doc/user/project/operations/feature_flags.md b/doc/user/project/operations/feature_flags.md new file mode 100644 index 00000000000..a5db3d70bb9 --- /dev/null +++ b/doc/user/project/operations/feature_flags.md @@ -0,0 +1,177 @@ +# Feature Flags **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845) in GitLab 11.4. + +CAUTION: **Warning:** +This an _alpha_ feature and is subject to change at any time without +prior notice. + +Feature flags allow you to ship a project in different flavors by +dynamically toggling certain functionality. + +## Overview + +Feature Flags offer a feature toggle system for your application. They enable teams +to achieve Continuous Delivery by deploying new features to production at smaller +batches for controlled testing, separating feature delivery from customer launch. +This helps reducing risk and allows you to easily manage which features to enable. + +GitLab offers a Feature Flags interface that allows you to create, toggle and +remove feature flags. + +## How it works + +Underneath, GitLab uses [unleash](https://github.com/Unleash/unleash), a feature +toggle service. GitLab provides an API where your application can talk to and get the +list of feature flags you set in GitLab. + +The application must be configured to talk to GitLab, so that's up to the +developers to use a compatible [client library](#client-libraries) and +integrate it in their app. + +By setting a flag active or inactive via GitLab, your application will automatically +know which features to enable or disable respectively. + +## Adding a new feature flag + +To add a new feature flag: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **New Feature Flag** button. +1. Give it a name. + + NOTE: **Note:** + A name can contain only lowercase letters, digits, underscores (`_`) + and dashes (`-`), must start with a letter, and cannot end with a dash (`-`) + or an underscore (`_`). + +1. Give it a description (optional, 255 characters max). +1. Define environment [specs](#define-environment-specs). If you want the flag on by default, enable the catch-all [wildcard spec (`*`)](#define-environment-specs) +1. Click **Create feature flag**. + +Once a feature flag is created, the list of existing feature flags will be presented +with ability to edit or remove them. + +To make a feature flag active or inactive, click the pencil icon to edit it, +and toggle the status for each [spec](#define-environment-specs). + + + +## Define environment specs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8621) in GitLab 11.8. + +In general, an application is deployed to multiple environments, such as +production, staging and [review apps](../../../ci/review_apps/index.md). +For example, you may not want to enable a feature flag on production until your QA team has +first confirmed that the feature is working correctly on testing environments. + +To handle these situations, you can enable a feature flag on a particular environment +with [Environment specs](../../../ci/environments.md#scoping-environments-with-specs-premium). +You can define multiple specs per flag so that you can control your feature flag more granularly. + +To define specs for each environment: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **New Feature Flag** button or edit an existing flag. +1. Set the status of the default [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) (`*`). This status will be used for _all_ environments. +1. If you want to enable/disable the feature on a specific environment, create a new [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) and type the environment name. +1. Set the status of the additional spec. This status takes precedence over the default spec's status since we always use the most specific match available. +1. Click **Create feature flag** or **Update feature flag**. + + + +NOTE: **NOTE** +We'd highly recommend you to use the [Environment](../../../ci/environments.md) +feature in order to quickly assess which flag is enabled per environment. + +## Integrating with your application + +In order to use Feature Flags, you need to first +[get the access credentials](#configuring-feature-flags) from GitLab and then +prepare your application and hook it with a [client library](#client-libraries). + +### Configuring Feature Flags + +To get the access credentials that your application will need to talk to GitLab: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **Configure** button to see the values: + - **API URL**: URL where the client (application) connects to get a list of feature flags. + - **Instance ID**: Unique token that authorizes the retrieval of the feature flags. + - **Application name**: The name of the running environment. For instance, + if the application runs for production server, application name would be + `production` or similar. This value is used for + [the environment spec evaluation](#define-environment-specs). + +NOTE: **Note:** +The meaning of these fields might change over time. For example, we are not sure +if **Instance ID** will be single token or multiple tokens, assigned to the +**Environment**. Also, **Application name** could describe the version of +application instead of the running environment. + +### Client libraries + +GitLab currently implements a single backend that is compatible with +[Unleash](https://github.com/Unleash/unleash#client-implementations) clients. + +Unleash clients allow the developers to define in the app's code the default +values for flags. Each feature flag evaluation can express the desired +outcome in case the flag isn't present on the provided configuration file. + +Unleash currently offers a number of official SDKs for various frameworks and +a number of community contributed libraries. + +Official clients: + +- [unleash/unleash-client-java](https://github.com/unleash/unleash-client-java) +- [unleash/unleash-client-node](https://github.com/unleash/unleash-client-node) +- [unleash/unleash-client-go](https://github.com/unleash/unleash-client-go) +- [unleash/unleash-client-ruby](https://github.com/unleash/unleash-client-ruby) + +Community contributed clients: + +- [stiano/unleash-client-dotnet](https://github.com/stiano/unleash-client-dotnet) (.Net Core) +- [onybo/unleash-client-core](https://github.com/onybo/unleash-client-core) (.Net Core) +- [aes/unleash-client-python](https://github.com/aes/unleash-client-python) (Python 3) + +### Golang application example + +Here's an example of how to integrate the feature flags in a Golang application: + +```golang +package main + +import ( + "io" + "log" + "net/http" + + "github.com/Unleash/unleash-client-go" +) + +type metricsInterface struct { +} + +func init() { + unleash.Initialize( + unleash.WithUrl("https://gitlab.com/api/v4/feature_flags/unleash/42"), + unleash.WithInstanceId("29QmjsW6KngPR5JNPMWx"), + unleash.WithAppName("production"), + unleash.WithListener(&metricsInterface{}), + ) +} + +func helloServer(w http.ResponseWriter, req *http.Request) { + if unleash.IsEnabled("my_feature_name") { + io.WriteString(w, "Feature enabled\n") + } else { + io.WriteString(w, "hello, world!\n") + } +} + +func main() { + http.HandleFunc("/", helloServer) + log.Fatal(http.ListenAndServe(":8080", nil)) +} +``` diff --git a/doc/user/project/operations/img/feature_flags_list.png b/doc/user/project/operations/img/feature_flags_list.png Binary files differnew file mode 100644 index 00000000000..5313a163fec --- /dev/null +++ b/doc/user/project/operations/img/feature_flags_list.png diff --git a/doc/user/project/operations/img/specs_list.png b/doc/user/project/operations/img/specs_list.png Binary files differnew file mode 100644 index 00000000000..9630c907cfc --- /dev/null +++ b/doc/user/project/operations/img/specs_list.png diff --git a/doc/user/project/operations/index.md b/doc/user/project/operations/index.md index b0f9936be5c..0086c15c98a 100644 --- a/doc/user/project/operations/index.md +++ b/doc/user/project/operations/index.md @@ -7,5 +7,5 @@ your applications: - Deploy to different [environments](../../../ci/environments.md). - Connect your project to a [Kubernetes cluster](../clusters/index.md). - Discover and view errors generated by your applications with [Error Tracking](error_tracking.md). -- Create, toggle, and remove [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html). **[PREMIUM]** -- [Trace](https://docs.gitlab.com/ee/user/project/operations/tracing.html) the performance and health of a deployed application. **[ULTIMATE]** +- Create, toggle, and remove [Feature Flags](feature_flags.md). **[PREMIUM]** +- [Trace](tracing.md) the performance and health of a deployed application. **[ULTIMATE]** diff --git a/doc/user/project/operations/tracing.md b/doc/user/project/operations/tracing.md new file mode 100644 index 00000000000..0416e096cf2 --- /dev/null +++ b/doc/user/project/operations/tracing.md @@ -0,0 +1,34 @@ +# Tracing **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7903) in GitLab Ultimate 11.5. + +Tracing provides insight into the performance and health of a deployed application, +tracking each function or microservice which handles a given request. + +This makes it easy to +understand the end-to-end flow of a request, regardless of whether you are using a monolithic or distributed system. + +## Jaeger tracing + +[Jaeger](https://www.jaegertracing.io/) is an open source, end-to-end distributed +tracing system used for monitoring and troubleshooting microservices-based distributed +systems. + +### Deploying Jaeger + +To learn more about deploying Jaeger, read the official +[Getting Started documentation](https://www.jaegertracing.io/docs/latest/getting-started/). +There is an easy to use [all-in-one Docker image](https://www.jaegertracing.io/docs/latest/getting-started/#AllinoneDockerimage), +as well as deployment options for [Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes) +and [OpenShift](https://github.com/jaegertracing/jaeger-openshift). + +### Enabling Jaeger + +GitLab provides an easy way to open the Jaeger UI from within your project: + +1. [Set up Jaeger](#deploying-jaeger) and configure your application using one of the + [client libraries](https://www.jaegertracing.io/docs/latest/client-libraries/). +1. Navigate to your project's **Settings > Operations** and provide the Jaeger URL. +1. Click **Save changes** for the changes to take effect. +1. You can now visit **Operations > Tracing** in your project's sidebar and + GitLab will redirect you to the configured Jaeger URL.
\ No newline at end of file diff --git a/doc/user/project/packages/img/maven_package_view.png b/doc/user/project/packages/img/maven_package_view.png Binary files differnew file mode 100644 index 00000000000..2eb4b6f76b4 --- /dev/null +++ b/doc/user/project/packages/img/maven_package_view.png diff --git a/doc/user/project/packages/img/npm_package_view.png b/doc/user/project/packages/img/npm_package_view.png Binary files differnew file mode 100644 index 00000000000..8baf7d0ef9f --- /dev/null +++ b/doc/user/project/packages/img/npm_package_view.png diff --git a/doc/user/project/packages/maven.md b/doc/user/project/packages/maven.md new file mode 100644 index 00000000000..266225fdb8d --- /dev/null +++ b/doc/user/project/packages/maven.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'maven_repository.md' +--- + +This document was moved to [another location](maven_repository.md). diff --git a/doc/user/project/packages/maven_packages.md b/doc/user/project/packages/maven_packages.md new file mode 100644 index 00000000000..266225fdb8d --- /dev/null +++ b/doc/user/project/packages/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'maven_repository.md' +--- + +This document was moved to [another location](maven_repository.md). diff --git a/doc/user/project/packages/maven_repository.md b/doc/user/project/packages/maven_repository.md new file mode 100644 index 00000000000..94785eb6aec --- /dev/null +++ b/doc/user/project/packages/maven_repository.md @@ -0,0 +1,341 @@ +# GitLab Maven Repository **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5811) in + [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. + +With the GitLab [Maven](https://maven.apache.org) Repository, every +project can have its own space to store its Maven artifacts. + + + +## Enabling the Maven Repository + +NOTE: **Note:** +This option is available only if your GitLab administrator has +[enabled support for the Maven repository](https://docs.gitlab.com/ee/administration/packages.html).**[PREMIUM ONLY]** + +After the Packages feature is enabled, the Maven Repository will be available for +all new projects by default. To enable it for existing projects, or if you want +to disable it: + +1. Navigate to your project's **Settings > General > Permissions**. +1. Find the Packages feature and enable or disable it. +1. Click on **Save changes** for the changes to take effect. + +You should then be able to see the **Packages** section on the left sidebar. +Next, you must configure your project to authorize with the GitLab Maven +repository. + +## Authenticating to the GitLab Maven Repository + +If a project is private or you want to upload Maven artifacts to GitLab, +credentials will need to be provided for authorization. Support is available for +[personal access tokens](#authenticating-with-a-personal-access-token) and +[CI job tokens](#authenticating-with-a-ci-job-token) only. +[Deploy tokens](../deploy_tokens/index.md) and regular username/password +credentials do not work. + +### Authenticating with a personal access token + +To authenticate with a [personal access token](../../profile/personal_access_tokens.md), +add a corresponding section to your +[`settings.xml`](https://maven.apache.org/settings.html) file: + +```xml +<settings> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Private-Token</name> + <value>REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> +</settings> +``` + +You should now be able to upload Maven artifacts to your project. + +### Authenticating with a CI job token + +If you're using Maven with GitLab CI/CD, a CI job token can be used instead +of a personal access token. + +To authenticate with a CI job token, add a corresponding section to your +[`settings.xml`](https://maven.apache.org/settings.html) file: + +```xml +<settings> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Job-Token</name> + <value>${env.CI_JOB_TOKEN}</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> +</settings> +``` + +You can read more on +[how to create Maven packages using GitLab CI/CD](#creating-maven-packages-with-gitlab-cicd). + +## Configuring your project to use the GitLab Maven repository URL + +To download and upload packages from GitLab, you need a `repository` and +`distributionManagement` section in your `pom.xml` file. + +Depending on your workflow and the amount of Maven packages you have, there are +3 ways you can configure your project to use the GitLab endpoint for Maven packages: + +- **Project level**: Useful when you have few Maven packages which are not under + the same GitLab group. +- **Group level**: Useful when you have many Maven packages under the same GitLab + group. +- **Instance level**: Useful when you have many Maven packages under different + GitLab groups or on their own namespace. + +NOTE: **Note:** +In all cases, you need a project specific URL for uploading a package in +the `distributionManagement` section. + +### Project level Maven endpoint + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `PROJECT_ID` with your project ID which can be found on the home page +of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project +(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +project's ID can be used for uploading. + +### Group level Maven endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8798) in GitLab Premium 11.7. + +If you rely on many packages, it might be inefficient to include the `repository` section +with a unique URL for each package. Instead, you can use the group level endpoint for +all your Maven packages stored within one GitLab group. Only packages you have access to +will be available for download. + +The group level endpoint works with any package names, which means the you +have the flexibility of naming compared to [instance level endpoint](#instance-level-maven-endpoint). +However, GitLab will not guarantee the uniqueness of the package names within +the group. You can have two projects with the same package name and package +version. As a result, GitLab will serve whichever one is more recent. + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like. You still need a project specific URL for uploading a package in +the `distributionManagement` section: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/groups/my-group/-/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `my-group` with your group name and `PROJECT_ID` with your project ID +which can be found on the home page of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the group +(e.g., `group%2Fsubgroup`) or the group's ID (e.g., `12`). + +### Instance level Maven endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8274) in GitLab Premium 11.7. + +If you rely on many packages, it might be inefficient to include the `repository` section +with a unique URL for each package. Instead, you can use the instance level endpoint for +all maven packages stored in GitLab and the packages you have access to will be available +for download. + +Note that **only packages that have the same path as the project** are exposed via +the instance level endpoint. + +| Project | Package | Instance level endpoint available | +| ------- | ------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab-ce` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab-ce` | `gitlab-org/gitlab-ce/1.0-SNAPSHOT` | Yes | + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like. You still need a project specific URL for uploading a package in +the `distributionManagement` section: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `PROJECT_ID` with your project ID which can be found on the home page +of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project +(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +project's ID can be used for uploading. + +## Uploading packages + +Once you have set up the [authentication](#authenticating-to-the-gitlab-maven-repository) +and [configuration](#configuring-your-project-to-use-the-gitlab-maven-repository-url), +test to upload a Maven artifact from a project of yours: + +```sh +mvn deploy +``` + +You can then navigate to your project's **Packages** page and see the uploaded +artifacts or even delete them. + +## Creating Maven packages with GitLab CI/CD + +Once you have your repository configured to use the GitLab Maven Repository, +you can configure GitLab CI/CD to build new packages automatically. The example below +shows how to create a new package each time the `master` branch is updated: + +1. Create a `ci_settings.xml` file that will serve as Maven's `settings.xml` file. + Add the server section with the same id you defined in your `pom.xml` file. + For example, in our case it's `gitlab-maven`: + + ```xml + <settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd"> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Job-Token</name> + <value>${env.CI_JOB_TOKEN}</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> + </settings> + ``` + +1. Make sure your `pom.xml` file includes the following: + + ```xml + <repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + </repositories> + <distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </snapshotRepository> + </distributionManagement> + ``` + + TIP: **Tip:** + You can either let Maven utilize the CI environment variables or hardcode your project's ID. + +1. Add a `deploy` job to your `.gitlab-ci.yml` file: + + ```yaml + deploy: + image: maven:3.3.9-jdk-8 + script: + - 'mvn deploy -s ci_settings.xml' + only: + - master + ``` + +1. Push those files to your repository. + +The next time the `deploy` job runs, it will copy `ci_settings.xml` to the +user's home location (in this case the user is `root` since it runs in a +Docker container), and Maven will utilize the configured CI +[environment variables](../../../ci/variables/README.md#predefined-environment-variables). diff --git a/doc/user/project/packages/npm_registry.md b/doc/user/project/packages/npm_registry.md new file mode 100644 index 00000000000..9f4c01c9a0a --- /dev/null +++ b/doc/user/project/packages/npm_registry.md @@ -0,0 +1,120 @@ +# GitLab NPM Registry **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5934) + in [GitLab Premium](https://about.gitlab.com/pricing/) 11.7. + +With the GitLab NPM Registry, every +project can have its own space to store NPM packages. + + + +NOTE: **Note:** +Only [scoped](https://docs.npmjs.com/misc/scope) packages are supported. + + +NOTE: **Note:** +As `@group/subgroup/project` is not a valid NPM package name, publishing a package +within a subgroup is not supported yet. + +## Enabling the NPM Registry + +NOTE: **Note:** +This option is available only if your GitLab administrator has +[enabled support for the NPM registry](https://docs.gitlab.com/ee/administration/packages.html).**[PREMIUM ONLY]** + +After the NPM registry is enabled, it will be available for all new projects +by default. To enable it for existing projects, or if you want to disable it: + +1. Navigate to your project's **Settings > General > Permissions**. +1. Find the Packages feature and enable or disable it. +1. Click on **Save changes** for the changes to take effect. + +You should then be able to see the **Packages** section on the left sidebar. + +Before proceeding to authenticating with the GitLab NPM Registry, you should +get familiar with the package naming convention. + +## Package naming convention + +**Only packages that have the same path as the project** are supported. For + example: + +| Project | Package | Supported | +| ---------------------- | ----------------------- | --------- | +| `foo/bar` | `@foo/bar` | Yes | +| `gitlab-org/gitlab-ce` | `@gitlab-org/gitlab-ce` | Yes | +| `gitlab-org/gitlab-ce` | `@foo/bar` | No | + +Now, you can configure your project to authenticate with the GitLab NPM +Registry. + +## Authenticating to the GitLab NPM Registry + +If a project is private or you want to upload an NPM package to GitLab, +credentials will need to be provided for authentication. Support is available +only for [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow). + +CAUTION: **2FA not supported:** +Authentication for personal access tokens is not yet supported +([#9140](https://gitlab.com/gitlab-org/gitlab-ee/issues/9140)). If you have 2FA +enabled, you won't be able to authenticate to the GitLab NPM Registry. + +### Authenticating with an OAuth token + +To authenticate with an [OAuth token](../../../api/oauth2.md#resource-owner-password-credentials-flow), +add a corresponding section to your `.npmrc` file: + +```ini +; Set URL for your scoped packages. +; For example package with name `@foo/bar` will use this URL for download +@foo:registry=https://gitlab.com/api/v4/packages/npm/ + +; Add the OAuth token for the scoped packages URL. This will allow you to download +; `@foo/` packages from private projects. +//gitlab.com/api/v4/packages/npm/:_authToken=<your_oauth_token> + +; Add OAuth token for uploading to the registry. Replace <your_project_id> +; with the project you want your package to be uploaded to. +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=<your_oauth_token> +``` + +Replace `<your_project_id>` with your project ID which can be found on the home page +of your project and `<your_oauth_token>` with your OAuth token. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +You should now be able to download and upload NPM packages to your project. + +## Uploading packages + +Before you will be able to upload a package, you need to specify the registry +for NPM. To do this, add the following section to the bottom of `package.json`: + +```json + "publishConfig": { + "@foo:registry":"https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/" + } +``` + +Replace `<your_project_id>` with your project ID, which can be found on the home +page of your project, and replace `@foo` with your own scope. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +Once you have enabled it and set up [authentication](#authenticating-to-the-gitlab-npm-registry), +you can upload an NPM package to your project: + +```sh +npm publish +``` + +You can then navigate to your project's **Packages** page and see the uploaded +packages or even delete them. + +## Uploading a package with the same version twice + +If you upload a package with a same name and version twice, GitLab will show +both packages in the UI, but the GitLab NPM Registry will expose the most recent +one as it supports only one package per version for `npm install`. diff --git a/doc/user/project/pipelines/img/pipeline_schedule_variables.png b/doc/user/project/pipelines/img/pipeline_schedule_variables.png Binary files differindex 74692add93a..29846206491 100644 --- a/doc/user/project/pipelines/img/pipeline_schedule_variables.png +++ b/doc/user/project/pipelines/img/pipeline_schedule_variables.png diff --git a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png Binary files differindex 95203ec861b..e135dd51070 100644 --- a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png +++ b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 2060b5dd4a2..56e8f1731ae 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -15,12 +15,10 @@ By default, a protected branch does four simple things: - it prevents **anyone** from force pushing to the branch - it prevents **anyone** from deleting the branch -See the [Changelog](#changelog) section for changes over time. +>**Note**: +A GitLab admin is allowed to push to the protected branches. -> ->Additional functionality for GitLab Enterprise Edition: -> ->- Restrict push and merge access to [certain users][ee-restrict] +See the [Changelog](#changelog) section for changes over time. ## Configuring protected branches @@ -68,6 +66,21 @@ dropdown list in the "Already protected" area. If you don't choose any of those options while creating a protected branch, they are set to "Maintainers" by default. +## Restricting push and merge access to certain users **[STARTER]** + +> This feature was [introduced][ce-5081] in [GitLab Starter][ee] 8.11. + +With GitLab Enterprise Edition you can restrict access to protected branches +by choosing a role (Maintainers, Developers) as well as certain users. From the +dropdown menu select the role and/or the users you want to have merge or push +access. + + + +Click **Protect** and the branch will appear in the "Protected branch" list. + + + ## Wildcard protected branches > [Introduced][ce-4665] in GitLab 8.10. @@ -169,3 +182,4 @@ for details about the pipelines security model. [ce-21393]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21393 [ee-restrict]: http://docs.gitlab.com/ee/user/project/protected_branches.html#restricting-push-and-merge-access-to-certain-users [perm]: ../permissions.md +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 2040e2ee004..15eb862b431 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -46,13 +46,15 @@ discussions, and descriptions: | `/remove_due_date` | Remove due date | ✓ | | | `/weight 0,1,2, ...` | Set weight **[STARTER]** | ✓ | | | `/clear_weight` | Clears weight **[STARTER]** | ✓ | | -| `/epic <group&epic | Epic URL>` | Add to epic **[ULTIMATE]** | ✓ | | +| `/epic <&epic | group&epic | Epic URL>` | Add to epic **[ULTIMATE]** | ✓ | | | `/remove_epic` | Removes from epic **[ULTIMATE]** | ✓ | | +| `/promote` | Promote issue to epic **[ULTIMATE]** | ✓ | | | `/confidential` | Make confidential | ✓ | | | `/duplicate #issue` | Mark this issue as a duplicate of another issue | ✓ | | `/move path/to/project` | Move this issue to another project | ✓ | | | `/target_branch <Local branch Name>` | Set target branch | | ✓ | | `/wip` | Toggle the Work In Progress status | | ✓ | +| `/approve` | Approve the merge request | | ✓ | | `/merge` | Merge (when pipeline succeeds) | | ✓ | | `/create_merge_request <branch name>` | Create a new merge request starting from the current issue | ✓ | | diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 6495ede42e0..3260a355fdc 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -264,7 +264,7 @@ To remove a GPG key from your account: ## Rejecting commits that are not signed **[PREMIUM]** You can configure your project to reject commits that aren't GPG-signed -via [push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html). +via [push rules](../../../../push_rules/push_rules.md). ## GPG signing API diff --git a/doc/user/project/security_dashboard.md b/doc/user/project/security_dashboard.md new file mode 100644 index 00000000000..43e910b29fe --- /dev/null +++ b/doc/user/project/security_dashboard.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/security_dashboard/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/security_dashboard/index.html). diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md new file mode 100644 index 00000000000..1a582164d03 --- /dev/null +++ b/doc/user/project/service_desk.md @@ -0,0 +1,124 @@ +# Service Desk **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/149) in [GitLab Premium 9.1](https://about.gitlab.com/2017/04/22/gitlab-9-1-released/#service-desk-eep). + +## Overview + +Service Desk is a module that allows your team to connect directly +with any external party through email right inside of GitLab; no external tools required. +An ongoing conversation right where your software is built ensures that user feedback ends +up directly where it's needed, helping you build the right features to solve your users' +real problems. + +With Service Desk, you can provide efficient email support to your customers, who can now +email you bug reports, feature requests, or general feedback that will all end up in your +GitLab project as new issues. In turn, your team can respond straight from the project. + +As Service Desk is built right into GitLab itself, the complexity and inefficiencies +of multiple tools and external integrations are eliminated, significantly shortening +the cycle time from feedback to software update. + +For an overview, check the video demonstration on [GitLab Service Desk](https://about.gitlab.com/2017/05/09/demo-service-desk/). + +## Use cases + +For instance, let's assume you develop a game for iOS or Android. +The codebase is hosted in your GitLab instance, built and deployed +with GitLab CI. + +Here's how Service Desk will work for you: + +1. You'll provide a project-specific email address to your paying customers, who can email you directly from within the app +1. Each email they send creates an issue in the appropriate project +1. Your team members navigate to the Service Desk issue tracker, where they can see new support requests and respond inside associated issues +1. Your team communicates back and forth with the customer to understand the request +1. Your team starts working on implementing code to solve your customer's problem +1. When your team finishes the implementation, whereupon the merge request is merged and the issue is closed automatically +1. The customer will have been attended successfully via email, without having real access to your GitLab instance +1. Your team saved time by not having to leave GitLab (or setup any integrations) to follow up with your customer + +## How it works + +GitLab Service Desk is a simple way to allow people to create issues in your +GitLab instance without needing their own user account. + +It provides a unique email address for end users to create issues in a project, +and replies can be sent either through the GitLab interface or by email. End +users will only see the thread through email. + +## Configuring Service Desk + +> **Note:** +Service Desk is enabled on GitLab.com. If you're a +[Silver subscriber](https://about.gitlab.com/gitlab-com/), +you can skip the step 1 below; you only need to enable it per project. + +1. [Set up incoming email](../../administration/incoming_email.md#set-it-up) for the GitLab instance. This must + support [email sub-addressing](../../administration/incoming_email.md#email-sub-addressing). +2. Navigate to your project's **Settings** and scroll down to the **Service Desk** + section. +3. If you have the correct access and an Premium license, + you will see an option to set up Service Desk: + +  + +4. Checking that box will enable Service Desk for the project, and show a + unique email address to email issues to the project. These issues will be + [confidential](issues/confidential_issues.md), so they will only be visible to project members. + + **Warning**: this email address can be used by anyone to create an issue on + this project, whether or not they have access to your GitLab instance. + We recommend **putting this behind an alias** so that it can be changed if + needed, and **[enabling Akismet](../../integration/akismet.md)** on your GitLab instance to add spam + checking to this service. Unblocked email spam would result in many spam + issues being created, and may disrupt your GitLab service. + +  + + _In GitLab 11.7, we updated the format of the generated email address. + However the older format is still supported, allowing existing aliases + or contacts to continue working._ + + +5. Service Desk is now enabled for this project! You should be able to access it from your project's navigation **Issue submenu**: + +  + +## Using Service Desk + +### As an end user (issue creator) + +To create a Service Desk issue, an end user doesn't need to know anything about +the GitLab instance. They just send an email to the address they are given, and +receive an email back confirming receipt: + + + +This also gives the end user an option to unsubscribe. + +If they don't choose to unsubscribe, then any new comments added to the issue +will be sent as emails: + + + +And any responses they send will be displayed in the issue itself. + +### As a responder to the issue + +For responders to the issue, everything works as usual. They'll see a familiar looking +issue tracker, where they can see issues created via customer support requests and +filter and interact with them just like other GitLab issues. + + + +Messages from the end user will show as coming from the special Support Bot user, but apart from that, +you can read and write comments as you normally do: + + + +> Note that the project's visibility (private, internal, public) does not affect Service Desk. + +### Support Bot user + +Behind the scenes, Service Desk works by the special Support Bot user creating issues. This user +does not count toward the license limit count. diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index d5f4a7fd4ac..99dd018a3ba 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -42,9 +42,9 @@ Set up your project's merge request settings:  -### Service Desk +### Service Desk **[PREMIUM]** -Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. Service Desk is available in [GitLab Premium](https://about.gitlab.com/pricing/). +Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. ### Export project @@ -128,3 +128,7 @@ namespace if needed. ### Error Tracking Configure Error Tracking to discover and view [Sentry errors within GitLab](../operations/error_tracking.md). + +### Jaeger tracing **[ULTIMATE]** + +Add the URL of a Jaeger server to allow your users to [easily access the Jaeger UI from within GitLab](../operations/tracing.md). diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 46a1b2bc3aa..2a2507d98a3 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -128,5 +128,123 @@ IDE. An example `package.json` is below. } ``` +## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5426) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.6. + +CAUTION: **Warning:** +Interactive Web Terminals for the Web IDE is currently in **Beta**. + +[Interactive web terminals](../../../ci/interactive_web_terminal/index.md) +give the user access to a terminal to interact with the Runner directly from +GitLab, including through the Web IDE. + +Only project [**maintainers**](../../permissions.md#project-members-permissions) +can run Interactive Web Terminals through the Web IDE. + +CAUTION: **Warning:** +GitLab.com [does not support Interactive Web Terminals yet](https://gitlab.com/gitlab-org/gitlab-ce/issues/52611). +Shared Runners in private instances are not supported either. + +### Runner configuration + +Some things need to be configured in the runner for the interactive web terminal +to work: + +- The Runner needs to have + [`[session_server]` configured properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section). +- If you are using a reverse proxy with your GitLab instance, web terminals need to be + [enabled](../../../administration/integration/terminal.md#enabling-and-disabling-terminal-support). **[ULTIMATE ONLY]** + +If you have the terminal open and the job has finished with its tasks, the +terminal will block the job from finishing for the duration configured in +[`[session_server].terminal_max_retention_time`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) +until you close the terminal window. + +NOTE: **Note:** Not all executors are +[supported](https://docs.gitlab.com/runner/executors/#compatibility-chart) + +### Web IDE configuration file + +In order to enable the Web IDE terminals you need to create the file +`.gitlab/.gitlab-webide.yml` inside the repository's root. This +file is fairly similar to the [CI configuration file](../../../ci/yaml/README.md) +syntax but with some restrictions: + +- No global blocks can be defined (ie: `before_script` or `after_script`) +- Only one job named `terminal` can be added to this file. +- Only the keywords `image`, `services`, `tags`, `before_script`, `script`, and +`variables` are allowed to be used to configure the job. +- To connect to the interactive terminal, the `terminal` job must be still alive +and running, otherwise the terminal won't be able to connect to the job's session. +By default the `script` keyword has the value `sleep 60` to prevent +the job from ending and giving the Web IDE enough time to connect. This means +that, if you override the default `script` value, you'll have to add a command +which would keep the job running, like `sleep`. + +In the code below there is an example of this configuration file: + +```yaml +terminal: + before_script: + - apt-get update + script: sleep 60 + variables: + RAILS_ENV: "test" + NODE_ENV: "test" +``` + +Once the terminal has started, the console will be displayed and we could access +the project repository files. + +**Important**. The terminal job is branch dependant. This means that the +configuration file used to trigger and configure the terminal will be the one in +the selected branch of the Web IDE. + +If there is no configuration file in a branch, an error message will be shown. + +### Running Interactive Terminals in the Web IDE + +If Interactive Terminals are available for the current user, the **Terminal** button +will be visible in the right sidebar of the Web IDE. Click this button to open +or close the terminal tab. + +Once open, the tab will show the **Start Web Terminal** button. This button may +be disabled if the environment is not configured correctly. If so, a status +message will describe the issue. Here are some reasons why **Start Web Terminal** +may be disabled: + +- `.gitlab/.gitlab-webide.yml` does not exist or is set up incorrectly. +- No active private runners are available for the project. + +If active, clicking the **Start Web Terminal** button will load the terminal view +and start connecting to the runner's terminal. At any time, the **Terminal** tab +can be closed and reopened and the state of the terminal will not be affected. + +When the terminal is started and is successfully connected to the runner, then the +runner's shell prompt will appear in the terminal. From here, you can enter +commands that will be executed within the runner's environment. This is similar +to running commands in a local terminal or through SSH. + +While the terminal is running, it can be stopped by clicking **Stop Terminal**. +This will disconnect the terminal and stop the runner's terminal job. From here, +click **Restart Terminal** to start a new terminal session. + +### Limitations + +Interactive Terminals is in a beta phase and will continue to be improved upon in upcoming +releases. In the meantime, please note that the user is limited to having only one +active terminal at a time. + +### Troubleshooting + +- If the terminal's text is gray and unresponsive, then the terminal has stopped + and it can no longer be used. A stopped terminal can be restarted by clicking + **Restart Terminal**. +- If the terminal displays **Connection Failure**, then the terminal could not + connect to the runner. Please try to stop and restart the terminal. If the + problem persists, double check your runner configuration. + + [ce]: https://about.gitlab.com/pricing/ [ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/search/advanced_global_search.md b/doc/user/search/advanced_global_search.md new file mode 100644 index 00000000000..38b26f31417 --- /dev/null +++ b/doc/user/search/advanced_global_search.md @@ -0,0 +1,75 @@ +# Advanced Global Search **[STARTER ONLY]** + +> - [Introduced][ee-109] in GitLab [Starter][ee] 8.4. +> - This is the user documentation. To install and configure Elasticsearch, +> visit the [administrator documentation](https://docs.gitlab.com/ee/integration/elasticsearch.html). + +NOTE: **Note** +Advanced Global Search (powered by Elasticsearch) is not yet available on GitLab.com. We are working on adding it. [Follow this epic for the latest updates](https://gitlab.com/groups/gitlab-org/-/epics/153). + +Leverage Elasticsearch for faster, more advanced code search across your entire +GitLab instance. + +## Overview + +The Advanced Global Search in GitLab is a powerful search service that saves +you time. Instead of creating duplicate code and wasting time, you can +now search for code within other teams that can help your own project. + +GitLab leverages the search capabilities of [Elasticsearch] and enables it when +searching in: + +- GitLab application +- Projects +- Repositories +- Commits +- Issues +- Merge requests +- Milestones +- Notes (comments) +- Snippets +- Wiki + +## Use cases + +The Advanced Global Search can be useful in various scenarios. + +### Faster searches + +If you are dealing with huge amount of data and want to keep GitLab's search +fast, the Advanced Global Search will help you achieve that. + +### Promote innersourcing + +Your company may consist of many different developer teams each of which has +their own group where the various projects are hosted. Some of your applications +may be connected to each other, so your developers need to instantly search +throughout the GitLab instance and find the code they search for. + +## Searching globally + +Just use the search as before and GitLab will show you matching code from each +project you have access to. + + + +You can also use the [Advanced Syntax Search](advanced_search_syntax.md) which +provides some useful queries. + +>**Note:** +Elasticsearch has only data for the default branch. That means that if you go +to the repository tree and switch the branch from the default to something else, +then the "Code" tab in the search result page will be served by the regular +search even if Elasticsearch is enabled. + +[ee-1305]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305 +[aws-elastic]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html +[aws-iam]: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html +[aws-instance-profile]: http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli +[ee-109]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request" +[elasticsearch]: https://www.elastic.co/products/elasticsearch "Elasticsearch website" +[install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation" +[pkg]: https://about.gitlab.com/downloads/ "Download Omnibus GitLab" +[elastic-settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#settings "Elasticsearch configuration settings" +[ee]: https://about.gitlab.com/pricing/ +[es]: https://www.elastic.co/products/elasticsearch diff --git a/doc/user/search/advanced_search_syntax.md b/doc/user/search/advanced_search_syntax.md new file mode 100644 index 00000000000..8d4aac5f502 --- /dev/null +++ b/doc/user/search/advanced_search_syntax.md @@ -0,0 +1,69 @@ +# Advanced Syntax Search **[STARTER ONLY]** + +> **Notes:** +> - Introduced in [GitLab Enterprise Starter][ee] 9.2 +> - This is the user documentation. To install and configure Elasticsearch, +> visit the [administrator documentation](https://docs.gitlab.com/ee/integration/elasticsearch.html). + +NOTE: **Note** +Advanced Global Search (powered by Elasticsearch) is not yet available on GitLab.com. We are working on adding it. [Follow this epic for the latest updates](https://gitlab.com/groups/gitlab-org/-/epics/153). + +Use advanced queries for more targeted search results. + +## Overview + +The Advanced Syntax Search is a subset of the +[Advanced Global Search](advanced_global_search.md), which you can use if you +want to have more specific search results. + +## Use cases + +Let's say for example that the product you develop relies on the code of another +product that's hosted under some other group. + +Since under your GitLab instance there are hosted hundreds of different projects, +you need the search results to be as efficient as possible. You have a feeling +of what you want to find (e.g., a function name), but at the same you're also +not so sure. + +In that case, using the advanced search syntax in your query will yield much +better results. + +## Using the Advanced Syntax Search + +The Advanced Syntax Search supports fuzzy or exact search queries with prefixes, +boolean operators, and much more. + +Full details can be found in the [Elasticsearch documentation][elastic], but +here's a quick guide: + +- Searches look for all the words in a query, in any order - e.g.: searching + issues for `display bug` will return all issues matching both those words, in any order. +- To find the exact phrase (stemming still applies), use double quotes: `"display bug"` +- To find bugs not mentioning display, use `-`: `bug -display` +- To find a bug in display or sound, use `|`: `bug display | sound` +- To group terms together, use parentheses: `bug | (display +sound)` +- To match a partial word, use `*`: `bug find_by_*` +- To find a term containing one of these symbols, use `\`: `argument \-last` + +### Syntax search filters + +The Advanced Syntax Search also supports the use of filters. The available filters are: + + - filename: Filters by filename. You can use the glob (`*`) operator for fuzzy matching. + - path: Filters by path. You can use the glob (`*`) operator for fuzzy matching. + - extension: Filters by extension in the filename. Please write the extension without a leading dot. Exact match only. + +To use them, simply add them to your query in the format `<filter_name>:<value>` without + any spaces between the colon (`:`) and the value. + +Examples: + +- Finding a file with any content named `hello_world.rb`: `* filename:hello_world.rb` +- Finding a file named `hello_world` with the text `whatever` inside of it: `whatever filename:hello_world` +- Finding the text 'def create' inside files with the `.rb` extension: `def create extension:rb` +- Finding the text `sha` inside files in a folder called `encryption`: `sha path:encryption` +- Finding any file starting with `hello` containing `world` and with the `.js` extension: `world filename:hello* extension:js` + +[ee]: https://about.gitlab.com/pricing/ +[elastic]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/query-dsl-simple-query-string-query.html#_simple_query_string_syntax diff --git a/doc/user/search/img/advanced_global_search.png b/doc/user/search/img/advanced_global_search.png Binary files differnew file mode 100644 index 00000000000..4903bbb07e1 --- /dev/null +++ b/doc/user/search/img/advanced_global_search.png diff --git a/doc/user/search/img/multiple_assignees.png b/doc/user/search/img/multiple_assignees.png Binary files differnew file mode 100644 index 00000000000..5c46f3dda46 --- /dev/null +++ b/doc/user/search/img/multiple_assignees.png diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 705983cce2a..bb6c48471c7 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -84,6 +84,12 @@ You can view recent searches by clicking on the little arrow-clock icon, which i Individual filters can be removed by clicking on the filter's (x) button or backspacing. The entire search filter can be cleared by clicking on the search box's (x) button. +## Filtering with multiple filters of the same type + +Some filters can be added multiple times. These include but are not limited to assignees and labels. When you filter with these multiple filters of the same type, the AND logic is applied. For example, if you were filtering `assignee:@sam assignee:@sarah`, your results will only include entries whereby the assignees are assigned to both Sam and Sarah are returned. + + + ### Shortcut You'll also find a shortcut on the search field on the top-right of the project's dashboard to @@ -136,3 +142,18 @@ you'll be able to, besides filtering them by **Name**, **Author**, **Assignee**, and **Labels**, select multiple issues to add to a list of your choice:  + +## Advanced Global Search **[STARTER]** + +Leverage Elasticsearch for faster, more advanced code search across your entire +GitLab instance. + +[Learn how to use the Advanced Global Search.](advanced_global_search.md) + +## Advanced Syntax Search **[STARTER]** + +Use advanced queries for more targeted search results. + +[Learn how to use the Advanced Syntax Search.](advanced_search_syntax.md) + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/workflow/README.md b/doc/workflow/README.md index 272f7807ac0..84ab7840140 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -13,12 +13,15 @@ comments: false - [Groups](../user/group/index.md) - Issues - The GitLab Issue Tracker is an advanced and complete tool for tracking the evolution of a new idea or the process of solving a problem. + - [Exporting Issues](https://docs.gitlab.com/ee/user/project/issues/csv_export.html) **[STARTER]** Export issues as a CSV, emailed as an attachment. - [Confidential issues](../user/project/issues/confidential_issues.md) - [Due date for issues](../user/project/issues/due_dates.md) - [Issue Board](../user/project/issue_board.md) - [Keyboard shortcuts](shortcuts.md) - [File finder](file_finder.md) +- [File lock](https://docs.gitlab.com/ee/user/project/file_lock.html) **[PREMIUM]** - [Labels](../user/project/labels.md) +- [Issue weight](https://docs.gitlab.com/ee/workflow/issue_weight.html) **[STARTER]** - [Notification emails](notifications.md) - [Projects](../user/project/index.md) - [Project forking workflow](forking_workflow.md) @@ -41,6 +44,9 @@ comments: false - [Merge requests versions](../user/project/merge_requests/versions.md) - ["Work In Progress" merge requests](../user/project/merge_requests/work_in_progress_merge_requests.md) - [Fast-forward merge requests](../user/project/merge_requests/fast_forward_merge.md) + - [Merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) **[STARTER]** +- [Repository mirroring](repository_mirroring.md) **[STARTER]** +- [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) **[PREMIUM]** - [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md) - [Importing from SVN, GitHub, Bitbucket, etc](importing/README.md) - [Todos](todos.md) diff --git a/doc/workflow/ff_merge.md b/doc/workflow/ff_merge.md new file mode 100644 index 00000000000..11e9e1bbd6b --- /dev/null +++ b/doc/workflow/ff_merge.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/fast_forward_merge.md' +--- + +This document was moved to [user/project/merge_requests/fast_forward_merge](../user/project/merge_requests/fast_forward_merge.md). diff --git a/doc/workflow/git_annex.md b/doc/workflow/git_annex.md new file mode 100644 index 00000000000..84d25951908 --- /dev/null +++ b/doc/workflow/git_annex.md @@ -0,0 +1,238 @@ +# Git annex + +> **Warning:** GitLab has [completely +removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22). +Read through the [migration guide from git-annex to git-lfs][guide]. + +The biggest limitation of Git, compared to some older centralized version +control systems, has been the maximum size of the repositories. + +The general recommendation is to not have Git repositories larger than 1GB to +preserve performance. Although GitLab has no limit (some repositories in GitLab +are over 50GB!), we subscribe to the advice to keep repositories as small as +you can. + +Not being able to version control large binaries is a big problem for many +larger organizations. +Videos, photos, audio, compiled binaries and many other types of files are too +large. As a workaround, people keep artwork-in-progress in a Dropbox folder and +only check in the final result. This results in using outdated files, not +having a complete history and increases the risk of losing work. + +This problem is solved in GitLab Enterprise Edition by integrating the +[git-annex] application. + +`git-annex` allows managing large binaries with Git without checking the +contents into Git. +You check-in only a symlink that contains the SHA-1 of the large binary. If you +need the large binary, you can sync it from the GitLab server over `rsync`, a +very fast file copying tool. + +## GitLab git-annex Configuration + +`git-annex` is disabled by default in GitLab. Below you will find the +configuration options required to enable it. + +### Requirements + +`git-annex` needs to be installed both on the server and the client side. + +For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running: + +``` +sudo apt-get update && sudo apt-get install git-annex +``` + +For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running: + +``` +sudo yum install epel-release && sudo yum install git-annex +``` + +### Configuration for Omnibus packages + +For omnibus-gitlab packages, only one configuration setting is needed. +The Omnibus package will internally set the correct options in all locations. + +1. In `/etc/gitlab/gitlab.rb` add the following line: + + ```ruby + gitlab_shell['git_annex_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +### Configuration for installations from source + +There are 2 settings to enable git-annex on your GitLab server. + +One is located in `config/gitlab.yml` of the GitLab repository and the other +one is located in `config.yml` of gitlab-shell. + +1. In `config/gitlab.yml` add or edit the following lines: + + ```yaml + gitlab_shell: + git_annex_enabled: true + ``` + +1. In `config.yml` of gitlab-shell add or edit the following lines: + + ```yaml + git_annex_enabled: true + ``` + +1. Save the files and [restart GitLab][] for the changes to take effect. + +## Using GitLab git-annex + +> **Note:** +> Your Git remotes must be using the SSH protocol, not HTTP(S). + +Here is an example workflow of uploading a very large file and then checking it +into your Git repository: + +```bash +git clone git@example.com:group/project.git + +git annex init 'My Laptop' # initialize the annex project and give an optional description +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git annex add debian.iso # add the large file to git annex +git commit -am "Add Debian iso" # commit the file metadata +git annex sync --content # sync the Git repo and large file to the GitLab server +``` + +The output should look like this: + +``` +commit +On branch master +Your branch is ahead of 'origin/master' by 1 commit. + (use "git push" to publish your local commits) +nothing to commit, working tree clean +ok +pull origin +remote: Counting objects: 5, done. +remote: Compressing objects: 100% (4/4), done. +remote: Total 5 (delta 2), reused 0 (delta 0) +Unpacking objects: 100% (5/5), done. +From example.com:group/project + 497842b..5162f80 git-annex -> origin/git-annex +ok +(merging origin/git-annex into git-annex...) +(recording state in git...) +copy debian.iso (checking origin...) (to origin...) +SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso + 26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1) +ok +pull origin +ok +(recording state in git...) +push origin +Counting objects: 15, done. +Delta compression using up to 4 threads. +Compressing objects: 100% (13/13), done. +Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done. +Total 15 (delta 1), reused 0 (delta 0) +To example.com:group/project.git + * [new branch] git-annex -> synced/git-annex + * [new branch] master -> synced/master +ok +``` + +Your files can be found in the `master` branch, but you'll notice that there +are more branches created by the `annex sync` command. + +Git Annex will also create a new directory at `.git/annex/` and will record the +tracked files in the `.git/config` file. The files you assign to be tracked +with `git-annex` will not affect the existing `.git/config` records. The files +are turned into symbolic links that point to data in `.git/annex/objects/`. + +The `debian.iso` file in the example will contain the symbolic link: + +``` +.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso +``` + +Use `git annex info` to retrieve the information about the local copy of your +repository. + +--- + +Downloading a single large file is also very simple: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync # sync Git branches but not the large file +git annex get debian.iso # download the large file +``` + +To download all files: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync --content # sync Git branches and download all the large files +``` + +By using `git-annex` without GitLab, anyone that can access the server can also +access the files of all projects, but GitLab Annex ensures that you can only +access files of projects you have access to (developer, maintainer, or owner role). + +## How it works + +Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great +integration point for `git-annex`. +There is a setting in gitlab-shell so you can disable GitLab Annex support +if you want to. + +## Troubleshooting tips + +Differences in version of `git-annex` on the GitLab server and on local machines +can cause `git-annex` to raise unpredicted warnings and errors. + +Consult the [Annex upgrade page][annex-upgrade] for more information about +the differences between versions. You can find out which version is installed +on your server by navigating to <https://pkgs.org/download/git-annex> and +searching for your distribution. + +Although there is no general guide for `git-annex` errors, there are a few tips +on how to go around the warnings. + +### git-annex-shell: Not a git-annex or gcrypt repository. + +This warning can appear on the initial `git annex sync --content` and is caused +by differences in `git-annex-shell`. You can read more about it +[in this git-annex issue][issue]. + +One important thing to note is that despite the warning, the `sync` succeeds +and the files are pushed to the GitLab repository. + +If you get hit by this, you can run the following command inside the repository +that the warning was raised: + +``` +git config remote.origin.annex-ignore false +``` + +Consecutive runs of `git annex sync --content` **should not** produce this +warning and the output should look like this: + +``` +commit ok +pull origin +ok +pull origin +ok +push origin +``` + +[annex-upgrade]: https://git-annex.branchable.com/upgrades/ +[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648 +[git-annex]: https://git-annex.branchable.com/ "git-annex website" +[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository" +[guide]: lfs/migrate_from_git_annex_to_git_lfs.html +[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue" +[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure +[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source diff --git a/doc/workflow/git_lfs.md b/doc/workflow/git_lfs.md new file mode 100644 index 00000000000..da217b0a5da --- /dev/null +++ b/doc/workflow/git_lfs.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'lfs/manage_large_binaries_with_git_lfs.md' +--- + +This document was moved to [another location](lfs/manage_large_binaries_with_git_lfs.md). diff --git a/doc/workflow/issue_weight.md b/doc/workflow/issue_weight.md new file mode 100644 index 00000000000..267160dae2a --- /dev/null +++ b/doc/workflow/issue_weight.md @@ -0,0 +1,22 @@ +# Issue Weight **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/76) +> in [GitLab Starter](https://about.gitlab.com/pricing/) 8.3. + +When you have a lot of issues, it can be hard to get an overview. +By adding a weight to each issue, you can get a better idea of how much time, +value or complexity a given issue has or will cost. + +You can set the weight of an issue during its creation, by simply changing the +value in the dropdown menu. You can set it to a non-negative integer +value from 0, 1, 2, and so on. (The database stores a 4-byte value, so the +upper bound is essentially limitless). +You can remove weight from an issue +as well. + +This value will appear on the right sidebar of an individual issue, as well as +in the issues page next to a distinctive balance scale icon. + +As an added bonus, you can see the total sum of all issues on the milestone page. + + diff --git a/doc/workflow/issue_weight/issue.png b/doc/workflow/issue_weight/issue.png Binary files differnew file mode 100644 index 00000000000..3800b5940b8 --- /dev/null +++ b/doc/workflow/issue_weight/issue.png diff --git a/doc/workflow/lfs/images/git-annex-branches.png b/doc/workflow/lfs/images/git-annex-branches.png Binary files differnew file mode 100644 index 00000000000..3d614f68177 --- /dev/null +++ b/doc/workflow/lfs/images/git-annex-branches.png diff --git a/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md new file mode 100644 index 00000000000..1aeab5980a3 --- /dev/null +++ b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md @@ -0,0 +1,261 @@ +# Migration guide from Git Annex to Git LFS + +>**Note:** +Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise +Edition 9.0 (2017/03/22). + +Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git. + +## History + +Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time +where Git LFS didn't yet exist. A few months later, GitLab brought support for +Git LFS in [GitLab 8.2][post-2] and is available for both Community and +Enterprise editions. + +## Differences between Git Annex and Git LFS + +Some items below are general differences between the two protocols and some are +ones that GitLab developed. + +- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS + (SSH support was added in GitLab 8.12). +- Annex files are stored in a sub-directory of the normal repositories, whereas + LFS files are stored outside of the repositories in a place you can define. +- Git Annex requires a more complex setup, but has much more options than Git + LFS. You can compare the commands each one offers by running `man git-annex` + and `man git-lfs`. +- Annex files cannot be browsed directly in GitLab's interface, whereas LFS + files can. + +## Migration steps + +>**Note:** +Since Git Annex files are stored in a sub-directory of the normal repositories +(`.git/annex/objects`) and LFS files are stored outside of the repositories, +they are not compatible as they are using a different scheme. Therefore, the +migration has to be done manually per repository. + +There are basically two steps you need to take in order to migrate from Git +Annex to Git LFS. + +### TL; DR + +If you know what you are doing and want to skip the reading, this is what you +need to do (we assume you have [git-annex enabled][annex-gitlab-use] in your +repository and that you have made backups in case something goes wrong). +Fire up a terminal, navigate to your Git repository and: + + +1. Disable `git-annex`: + + ```bash + git annex sync --content + git annex direct + git annex uninit + git annex indirect + ``` + +1. Enable `git-lfs`: + + ``` + git lfs install + git lfs track <files> + git add . + git commit -m "commit message" + git push + ``` + +### Disabling Git Annex in your repo + +Before changing anything, make sure you have a backup of your repository first. +There are a couple of ways to do that, but you can simply clone it to another +local path and maybe push it to GitLab if you want a remote backup as well. +Here you'll find a guide on +[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive]. + +Since Annex files are stored as objects with symlinks and cannot be directly +modified, we need to first remove those symlinks. + +>**Note:** +Make sure the you read about the [`direct` mode][annex-direct] as it contains +useful information that may fit in your use case. Note that `annex direct` is +deprecated in Git Annex version 6, so you may need to upgrade your repository +if the server also has Git Annex 6 installed. Read more in the +[Git Annex troubleshooting tips][annex-tips] section. + +1. Backup your repository + + ```bash + cd repository + git annex sync --content + cd .. + git clone repository repository-backup + cd repository-backup + git annex get + cd .. + ``` + +1. Use `annex direct`: + + ```bash + cd repository + git annex direct + ``` + + The output should be similar to this: + + ```bash + commit + On branch master + Your branch is up-to-date with 'origin/master'. + nothing to commit, working tree clean + ok + direct debian.iso ok + direct ok + ``` + +1. Disable Git Annex with [`annex uninit`][uninit]: + + ```bash + git annex uninit + ``` + + The output should be similar to this: + + ```bash + unannex debian.iso ok + Deleted branch git-annex (was 2534d2c). + ``` + + This will `unannex` every file in the repository, leaving the original files. + +1. Switch back to `indirect` mode: + + ```bash + git annex indirect + ``` + + The output should be similar to this: + + ```bash + (merging origin/git-annex into git-annex...) + (recording state in git...) + commit (recording state in git...) + + ok + (recording state in git...) + [master fac3194] commit before switching to indirect mode + 1 file changed, 1 deletion(-) + delete mode 120000 alpine-virt-3.4.4-x86_64.iso + ok + indirect ok + ok + ``` + +--- + +At this point, you have two options. Either add, commit and push the files +directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in +the next section. + +### Enabling Git LFS in your repo + +Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE, +GitLab.com), therefore, you don't need to do anything server-side. + +1. First, make sure you have `git-lfs` installed locally: + + ```bash + git lfs help + ``` + + If the terminal doesn't prompt you with a full response on `git-lfs` commands, + [install the Git LFS client][install-lfs] first. + +1. Inside the repo, run the following command to initiate LFS: + + ```bash + git lfs install + ``` + +1. Enable `git-lfs` for the group of files you want to track. You + can track specific files, all files containing the same extension, or an + entire directory: + + ```bash + git lfs track images/01.png # per file + git lfs track **/*.png # per extension + git lfs track images/ # per directory + ``` + + Once you do that, run `git status` and you'll see `.gitattributes` added + to your repo. It collects all file patterns that you chose to track via + `git-lfs`. + +1. Add the files, commit and push them to GitLab: + + ```bash + git add . + git commit -m "commit message" + git push + ``` + + If your remote is set up with HTTP, you will be asked to enter your login + credentials. If you have [2FA enabled][2fa], make sure to use a + [personal access token][token] instead of your password. + +## Removing the Git Annex branches + +After the migration finishes successfully, you can remove all `git-annex` +related branches from your repository. + +On GitLab, navigate to your project's **Repository ➔ Branches** and delete all +branches created by Git Annex: `git-annex`, and all under `synced/`. + + + +You can also do this on the commandline with: + + ```bash + git branch -d synced/master + git branch -d synced/git-annex + git push origin :synced/master + git push origin :synced/git-annex + git push origin :git-annex + git remote prune origin + ``` + +If there are still some Annex objects inside your repository (`.git/annex/`) +or references inside `.git/config`, run `annex uninit` again: + +```bash +git annex uninit +``` + +## Further Reading + +- (Blog Post) [Getting Started with Git FLS][post-1] +- (Blog Post) [Announcing LFS Support in GitLab][post-2] +- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3] +- (GitLab Docs) [Git Annex][doc-1] +- (GitLab Docs) [Git LFS][doc-2] + +[2fa]: ../../user/profile/account/two_factor_authentication.md +[token]: ../../user/profile/account/two_factor_authentication.html#personal-access-tokens +[annex-tips]: ../git_annex.html#troubleshooting-tips +[annex-direct]: https://git-annex.branchable.com/direct_mode/ +[annex-gitlab-use]: ../git_annex.md#using-gitlab-git-annex +[annex-ee]: https://docs.gitlab.com/ee/workflow/git_annex.html +[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive +[doc-1]: https://docs.gitlab.com/ee/workflow/git_annex.html +[doc-2]: https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html +[Git Annex]: http://git-annex.branchable.com/ +[Git LFS]: https://git-lfs.github.com/ +[install-lfs]: https://git-lfs.github.com/ +[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648 +[lfs-track]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs +[post-1]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/ +[post-2]: https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/ +[post-3]: https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ +[uninit]: https://git-annex.branchable.com/git-annex-uninit/ diff --git a/doc/workflow/merge_request_approvals.md b/doc/workflow/merge_request_approvals.md new file mode 100644 index 00000000000..f8a99ec3d57 --- /dev/null +++ b/doc/workflow/merge_request_approvals.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html' +--- + +This document was moved to [user/project/merge_requests/merge_request_approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html). diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index 6ce789998a4..0eb4f85e0ec 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -73,40 +73,43 @@ Below is the table of events users can be notified of: | Group access level changed | User | Sent when user group access level is changed | | Project moved | Project members [1] | [1] not disabled | -### Issue / Merge request events +### Issue / Epics / Merge request events In most of the below cases, the notification will be sent to: - Participants: - the author and assignee of the issue/merge request - authors of comments on the issue/merge request - - anyone mentioned by `@username` in the issue/merge request title or description - - anyone mentioned by `@username` in any of the comments on the issue/merge request - ...with notification level "Participating" or higher + - anyone mentioned by `@username` in the title or description of the issue, merge request or epic **[ULTIMATE]** + - anyone with notification level "Participating" or higher that is mentioned by `@username` + in any of the comments on the issue, merge request, or epic **[ULTIMATE]** - Watchers: users with notification level "Watch" -- Subscribers: anyone who manually subscribed to the issue/merge request +- Subscribers: anyone who manually subscribed to the issue, merge request, or epic **[ULTIMATE]** - Custom: Users with notification level "custom" who turned on notifications for any of the events present in the table below | Event | Sent to | |------------------------|---------| -| New issue | | -| Close issue | | +| New issue | | +| Close issue | | | Reassign issue | The above, plus the old assignee | -| Reopen issue | | +| Reopen issue | | | Due issue | Participants and Custom notification level with this event selected | | Change milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected | | Remove milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected | -| New merge request | | +| New merge request | | | Push to merge request | Participants and Custom notification level with this event selected | | Reassign merge request | The above, plus the old assignee | -| Close merge request | | -| Reopen merge request | | -| Merge merge request | | +| Close merge request | | +| Reopen merge request | | +| Merge merge request | | | Change milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected | | Remove milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected | | New comment | The above, plus anyone mentioned by `@username` in the comment, with notification level "Mention" or higher | | Failed pipeline | The author of the pipeline | | Successful pipeline | The author of the pipeline, if they have the custom notification setting for successful pipelines set | +| New epic **[ULTIMATE]** | | +| Close epic **[ULTIMATE]** | | +| Reopen epic **[ULTIMATE]** | | In addition, if the title or description of an Issue or Merge Request is changed, notifications will be sent to any **new** mentions by `@username` as diff --git a/doc/workflow/rebase_before_merge.md b/doc/workflow/rebase_before_merge.md new file mode 100644 index 00000000000..10e768d3371 --- /dev/null +++ b/doc/workflow/rebase_before_merge.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/fast_forward_merge.md' +--- + +This document was moved to [another location](../user/project/merge_requests/fast_forward_merge.md). diff --git a/doc/workflow/shortcuts.md b/doc/workflow/shortcuts.md index 6ed6b0bda66..6714d08f2f7 100644 --- a/doc/workflow/shortcuts.md +++ b/doc/workflow/shortcuts.md @@ -85,6 +85,14 @@ You can see GitLab's keyboard shortcuts by using 'shift + ?' | <kbd>]</kbd> or <kbd>j</kbd> | Move to next file | | <kbd>[</kbd> or <kbd>k</kbd> | Move to previous file | +## Epics **[ULTIMATE]** + +| Keyboard Shortcut | Description | +| ----------------- | ----------- | +| <kbd>r</kbd> | Reply (quoting selected text) | +| <kbd>e</kbd> | Edit description | +| <kbd>l</kbd> | Change label | + ## Wiki pages | Keyboard Shortcut | Description | diff --git a/doc/workflow/todos.md b/doc/workflow/todos.md index 830f17aa7f2..32907db4f46 100644 --- a/doc/workflow/todos.md +++ b/doc/workflow/todos.md @@ -25,9 +25,8 @@ will still be shown in the body of the _To do_ tab. A Todo appears in your Todos dashboard when: -- an issue or merge request is assigned to you, -- you are `@mentioned` in an issue or merge request, be it the description of - the issue/merge request or in a comment, +- an issue or merge request is assigned to you +- you are `@mentioned` in the description or in a comment of an issue, merge request, or epic **[ULTIMATE]** - you are `@mentioned` in a comment on a commit, - a job in the CI pipeline running for your merge request failed, but this job is not allowed to fail. @@ -63,14 +62,14 @@ for filtering; otherwise, they appear as normal. ### Manually creating a Todo -You can also add an issue or merge request to your Todos dashboard by clicking -the "Add todo" button in the issue or merge request sidebar. +You can also add an issue, merge request or epic to your Todos dashboard by clicking +the "Add todo" button in the sidebar of the issue, merge request, or epic **[ULTIMATE]**.  ## Marking a Todo as done -Any action to the corresponding issue or merge request will mark your Todo as +Any action to the corresponding issue, merge request or epic **[ULTIMATE]** will mark your Todo as **Done**. Actions that dismiss Todos include: - changing the assignee @@ -84,10 +83,10 @@ Todos are personal, and they're only marked as done if the action is coming from you. If you close the issue or merge request, your Todo will automatically be marked as done. -If someone else closes, merges, or takes action on the issue or merge +If someone else closes, merges, or takes action on the issue, epic or merge request, your Todo will remain pending. This prevents other users from closing issues without you being notified. -There is just one Todo per issue or merge request, so mentioning a user a +There is just one Todo per issue, epic or merge request, so mentioning a user a hundred times in an issue will only trigger one Todo. --- @@ -97,7 +96,7 @@ corresponding **Done** button, and it will disappear from your Todo list.  -A Todo can also be marked as done from the issue or merge request sidebar using +A Todo can also be marked as done from the issue, merge request or epic sidebar using the "Mark todo as done" button.  @@ -114,7 +113,7 @@ There are four kinds of filters you can use on your Todos dashboard. | Project | Filter by project | | Group | Filter by group | | Author | Filter by the author that triggered the Todo | -| Type | Filter by issue or merge request | +| Type | Filter by issue, merge request, or epic **[ULTIMATE]** | | Action | Filter by the action that triggered the Todo | You can also filter by more than one of these at the same time. The possible Actions are `Any Action`, `Assigned`, `Mentioned`, `Added`, `Pipelines`, and `Directly Addressed`, [as described above](#what-triggers-a-todo). |
