diff options
Diffstat (limited to 'doc/administration')
33 files changed, 1500 insertions, 230 deletions
diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 07e548aaabe..2fc5d0355b5 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -7,5 +7,6 @@ providers. and 389 Server - [OmniAuth](../../integration/omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, Crowd and Azure -- [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider - [CAS](../../integration/cas.md) Configure GitLab to sign in using CAS +- [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider +- [Okta](okta.md) Configure GitLab to sign in using Okta diff --git a/doc/administration/auth/img/okta_admin_panel.png b/doc/administration/auth/img/okta_admin_panel.png Binary files differnew file mode 100644 index 00000000000..12e21956715 --- /dev/null +++ b/doc/administration/auth/img/okta_admin_panel.png diff --git a/doc/administration/auth/img/okta_saml_settings.png b/doc/administration/auth/img/okta_saml_settings.png Binary files differnew file mode 100644 index 00000000000..ee275ece369 --- /dev/null +++ b/doc/administration/auth/img/okta_saml_settings.png diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index bf7814875bf..b8b63df091e 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -35,6 +35,10 @@ of one hour. To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml`. +There is a Rake task to check LDAP configuration. After configuring LDAP +using the documentation below, see [LDAP check Rake task](../raketasks/check.md#ldap-check) +for information on the LDAP check Rake task. + >**Note**: In GitLab EE, you can configure multiple LDAP servers to connect to one GitLab server. @@ -217,7 +221,7 @@ Tip: If you want to limit access to the nested members of an Active Directory group you can use the following syntax: ``` -(memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) +(memberOf=CN=My Group,DC=Example,DC=com) ``` Please note that GitLab does not support the custom filter syntax used by @@ -253,6 +257,24 @@ the LDAP server's SSL certificate is performed. ## Troubleshooting +### Debug LDAP user filter with ldapsearch + +This example uses ldapsearch and assumes you are using ActiveDirectory. The +following query returns the login names of the users that will be allowed to +log in to GitLab if you configure your own user_filter. + +``` +ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName +``` + +- Variables beginning with a `$` refer to a variable from the LDAP section of + your configuration file. +- Replace ldaps:// with ldap:// if you are using the plain authentication method. + Port `389` is the default `ldap://` port and `636` is the default `ldaps://` + port. +- We are assuming the password for the bind_dn user is in bind_dn_password.txt. + + ### Invalid credentials when logging in - Make sure the user you are binding with has enough permissions to read the user's diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md new file mode 100644 index 00000000000..cb42b7743c5 --- /dev/null +++ b/doc/administration/auth/okta.md @@ -0,0 +1,160 @@ +# Okta SSO provider + +Okta is a [Single Sign-on provider][okta-sso] that can be used to authenticate +with GitLab. + +The following documentation enables Okta as a SAML provider. + +## Configure the Okta application + +1. On Okta go to the admin section and choose to **Add an App**. +1. When the app screen comes up you see another button to **Create an App** and + choose SAML 2.0 on the next screen. +1. Now, very important, add a logo + (you can choose it from https://about.gitlab.com/press/). You'll have to + crop and resize it. +1. Next, you'll need the to fill in the SAML general config. Here's an example + image. + +  + +1. The last part of the configuration is the feedback section where you can + just say you're a customer and creating an app for internal use. +1. When you have your app you'll have a few tabs on the top of the app's + profile. Click on the SAML 2.0 config instructions button which should + look like the following: + +  + +1. On the screen that comes up take note of the + **Identity Provider Single Sign-On URL** which you'll use for the + `idp_sso_target_url` on your GitLab config file. + +1. **Before you leave Okta make sure you add your user and groups if any.** + +--- + +Now that the Okta app is configured, it's time to enable it in GitLab. + +## Configure GitLab + +1. On your GitLab server, open the configuration file: + + **For Omnibus GitLab installations** + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + **For installations from source** + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) + for initial settings. + +1. To allow your users to use Okta to sign up without having to manually create + an account first, don't forget to add the following values to your + configuration: + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] + gitlab_rails['omniauth_block_auto_created_users'] = false + ``` + + **For installations from source** + + ```yaml + allow_single_sign_on: ["saml"] + block_auto_created_users: false + ``` + +1. You can also automatically link Okta users with existing GitLab users if + their email addresses match by adding the following setting: + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_auto_link_saml_user'] = true + ``` + + **For installations from source** + + ```yaml + auto_link_saml_user: true + ``` + +1. Add the provider configuration. + + >**Notes:** + >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint + of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab + installation to generate the correct value). + >- To get the `idp_cert_fingerprint` fingerprint, first download the + certificate from the Okta app you registered and then run: + `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` + with the location of your certificate. + >- Change the value of `idp_sso_target_url`, with the value of the + **Identity Provider Single Sign-On URL** from the step when you + configured the Okta app. + >- Change the value of `issuer` to a unique name, which will identify the application + to the IdP. + >- Leave `name_identifier_format` as-is. + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Okta' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` + + **For installations from source** + + ```yaml + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Okta' # optional label for SAML login button, defaults to "Saml" + } + ``` + + +1. [Reconfigure][reconf] or [restart] GitLab for Omnibus and installations + from source respectively for the changes to take effect. + +You might want to try this out on a incognito browser window. + +## Configuring groups + +>**Note:** +Make sure the groups exist and are assigned to the Okta app. + +You can take a look of the [SAML documentation][saml] on external groups since +it works the same. + +[okta-sso]: https://www.okta.com/products/single-sign-on/ +[saml]: ../../integration/saml.md#external-groups +[reconf]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source diff --git a/doc/administration/build_artifacts.md b/doc/administration/build_artifacts.md index 64353f7282b..3ba8387c7f0 100644 --- a/doc/administration/build_artifacts.md +++ b/doc/administration/build_artifacts.md @@ -84,7 +84,7 @@ _The artifacts are stored by default in ## Set the maximum file size of the artifacts Provided the artifacts are enabled, you can change the maximum file size of the -artifacts through the [Admin area settings](../user/admin_area/settings/continuous_integration#maximum-artifacts-size). +artifacts through the [Admin area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). [reconfigure gitlab]: restart_gitlab.md "How to restart GitLab" [restart gitlab]: restart_gitlab.md "How to restart GitLab" diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 0387d730489..80e5d80aa41 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -42,6 +42,33 @@ Follow the steps below to set up a custom hook: That's it! Assuming the hook code is properly implemented the hook will fire as appropriate. +## Chained hooks support + +> [Introduced][93] in GitLab Shell 4.1.0 and GitLab 8.15. + +Hooks can be also placed in `hooks/<hook_name>.d` (global) or +`custom_hooks/<hook_name>.d` (per project) directories supporting chained +execution of the hooks. + +To look in a different directory for the global custom hooks (those in +`hooks/<hook_name.d>`), set `custom_hooks_dir` in gitlab-shell config. For +Omnibus installations, this can be set in `gitlab.rb`; and in source +installations, this can be set in `gitlab-shell/config.yml`. + +The hooks are searched and executed in this order: + +1. `<project>.git/hooks/` - symlink to `gitlab-shell/hooks` global dir +1. `<project>.git/hooks/<hook_name>` - executed by `git` itself, this is `gitlab-shell/hooks/<hook_name>` +1. `<project>.git/custom_hooks/<hook_name>` - per project hook (this is already existing behavior) +1. `<project>.git/custom_hooks/<hook_name>.d/*` - per project hooks +1. `<project>.git/hooks/<hook_name>.d/*` OR `<custom_hooks_dir>/<hook_name.d>/*` - global hooks: all executable files (minus editor backup files) + +Files in `.d` directories need to be executable and not match the backup file +pattern (`*~`). + +The hooks of the same type are executed in order and execution stops on the +first script exiting with a non-zero value. + ## Custom error messages > [Introduced][5073] in GitLab 8.10. @@ -54,3 +81,4 @@ STDERR takes precedence over STDOUT. [hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks [5073]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5073 +[93]: https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/93 diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index b4a953d1ccc..76029b30dd8 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -13,17 +13,18 @@ override certain values. Variable | Type | Description -------- | ---- | ----------- -`GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation -`GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) -`RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging` or `test` -`DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development` -`GITLAB_EMAIL_FROM` | string | The e-mail address used in the "From" field in e-mails sent by GitLab -`GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the "From" field in e-mails sent by GitLab -`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab -`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab -`GITLAB_EMAIL_SUBJECT_SUFFIX` | string | The e-mail subject suffix used in e-mails sent by GitLab -`GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the Unicorn worker killer -`GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the Unicorn worker killer +`GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation +`GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) +`RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging` or `test` +`DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development` +`GITLAB_EMAIL_FROM` | string | The e-mail address used in the "From" field in e-mails sent by GitLab +`GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the "From" field in e-mails sent by GitLab +`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab +`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab +`GITLAB_EMAIL_SUBJECT_SUFFIX` | string | The e-mail subject suffix used in e-mails sent by GitLab +`GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the Unicorn worker killer +`GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the Unicorn worker killer +`GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for GitLab Runners ## Complete database variables diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index d74a786ac24..d5a5aef7ec0 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -7,19 +7,10 @@ highly available. ## Architecture -### Active/Passive - -For pure high-availability/failover with no scaling you can use an -active/passive configuration. This utilizes DRBD (Distributed Replicated -Block Device) to keep all data in sync. DRBD requires a low latency link to -remain in sync. It is not advisable to attempt to run DRBD between data centers -or in different cloud availability zones. +There are two kinds of setups: -Components/Servers Required: - -- 2 servers/virtual machines (one active/one passive) - - +- active/active +- active/passive ### Active/Active @@ -28,12 +19,24 @@ user requests simultaneously. The database, Redis, and GitLab application are all deployed on separate servers. The configuration is **only** highly-available if the database, Redis and storage are also configured as such. - - -**Steps to configure active/active:** +Follow the steps below to configure an active/active setup: 1. [Configure the database](database.md) 1. [Configure Redis](redis.md) 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) + + + +### Active/Passive + +For pure high-availability/failover with no scaling you can use an +active/passive configuration. This utilizes DRBD (Distributed Replicated +Block Device) to keep all data in sync. DRBD requires a low latency link to +remain in sync. It is not advisable to attempt to run DRBD between data centers +or in different cloud availability zones. + +Components/Servers Required: 2 servers/virtual machines (one active/one passive) + + diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 538dada1bae..e4f94eb7cb6 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -41,9 +41,12 @@ If you use a cloud-managed service, or provide your own PostgreSQL: mailroom['enable'] = false # PostgreSQL configuration - postgresql['sql_password'] = 'DB password' + gitlab_rails['db_password'] = 'DB password' postgresql['md5_auth_cidr_addresses'] = ['0.0.0.0/0'] postgresql['listen_address'] = '0.0.0.0' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false ``` 1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. @@ -80,7 +83,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Similarly, set the password for the `gitlab` database user. Use the same password that you specified in the `/etc/gitlab/gitlab.rb` file for - `postgresql['sql_password']`. + `gitlab_rails['db_password']`. ``` \password gitlab @@ -102,9 +105,6 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Exit the database prompt by typing `\q` and Enter. 1. Exit the `gitlab-psql` user by running `exit` twice. 1. Run `sudo gitlab-ctl reconfigure` a final time. -1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations - from running on upgrade. Only the primary GitLab application server should - handle migrations. --- diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 537f4f3501d..5602d70f1ef 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -76,7 +76,7 @@ configuration to move each data location to a subdirectory: user['home'] = '/gitlab-data/home' git_data_dir '/gitlab-data/git-data' gitlab_rails['shared_path'] = '/gitlab-data/shared' -gitlab_rails['uploads_directory'] = "/gitlab-data/uploads" +gitlab_rails['uploads_directory'] = '/gitlab-data/uploads' gitlab_ci['builds_directory'] = '/gitlab-data/builds' ``` diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index bc424330656..b4e7bf21e35 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,265 +1,780 @@ # Configuring Redis for GitLab HA -You can choose to install and manage Redis yourself, or you can use the one -that comes bundled with GitLab Omnibus packages. - -> **Note:** Redis does not require authentication by default. See +> +Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. +Starting with 8.14, Redis Sentinel is no longer experimental. +If you've used it with versions `< 8.14` before, please check the updated +documentation here. + +High Availability with [Redis] is possible using a **Master** x **Slave** +topology with a [Redis Sentinel][sentinel] service to watch and automatically +start the failover procedure. + +You can choose to install and manage Redis and Sentinel yourself, use +a hosted cloud solution or you can use the one that comes bundled with +Omnibus GitLab packages. + +> **Notes:** +- Redis requires authentication for High Availability. See [Redis Security](http://redis.io/topics/security) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. +- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation + before configuring Redis HA with GitLab to fully understand the topology and + architecture. +- This is the documentation for the Omnibus GitLab packages. For installations + from source, follow the [Redis HA source installation](redis_source.md) guide. +- Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only. + For configuring Sentinel with the Omnibus GitLab Community Edition and + installations from source, read the + [Available configuration setups](#available-configuration-setups) section + below. + +## Overview + +Before diving into the details of setting up Redis and Redis Sentinel for HA, +make sure you read this Overview section to better understand how the components +are tied together. + +You need at least `3` independent machines: physical, or VMs running into +distinct physical machines. It is essential that all master and slaves Redis +instances run in different machines. If you fail to provision the machines in +that specific way, any issue with the shared environment can bring your entire +setup down. + +It is OK to run a Sentinel along with a master or slave Redis instance. +No more than one Sentinel in the same machine though. + +You also need to take in consideration the underlying network topology, +making sure you have redundant connectivity between Redis / Sentinel and +GitLab instances, otherwise the networks will become a single point of +failure. + +Make sure that you read this document once as a whole before configuring the +components below. + +### High Availability with Sentinel + +>**Notes:** +- Starting with GitLab `8.11`, you can configure a list of Redis Sentinel + servers that will monitor a group of Redis servers to provide failover support. +- Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package + comes with Redis Sentinel daemon built-in. + +High Availability with Redis requires a few things: + +- Multiple Redis instances +- Run Redis in a **Master** x **Slave** topology +- Multiple Sentinel instances +- Application support and visibility to all Sentinel and Redis instances + +Redis Sentinel can handle the most important tasks in an HA environment and that's +to help keep servers online with minimal to no downtime. Redis Sentinel: + +- Monitors **Master** and **Slaves** instances to see if they are available +- Promotes a **Slave** to **Master** when the **Master** fails +- Demotes a **Master** to **Slave** when the failed **Master** comes back online + (to prevent data-partitioning) +- Can be queried by the application to always connect to the current **Master** + server + +When a **Master** fails to respond, it's the application's responsibility +(in our case GitLab) to handle timeout and reconnect (querying a **Sentinel** +for a new **Master**). -## Configure your own Redis server +To get a better understanding on how to correctly setup Sentinel, please read +the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as +failing to configure it correctly can lead to data loss or can bring your +whole cluster down, invalidating the failover effort. -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for Redis. For example, AWS offers a managed ElastiCache service -that runs Redis. +### Recommended setup -## Configure Redis using Omnibus +For a minimal setup, you will install the Omnibus GitLab package in `3` +**independent** machines, both with **Redis** and **Sentinel**: -If you don't want to bother setting up your own Redis server, you can use the -one bundled with Omnibus. In this case, you should disable all services except -Redis. +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel -1. Download/install GitLab Omnibus using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. - Be sure to change the `external_url` to match your eventual GitLab front-end - URL: +If you are not sure or don't understand why and where the amount of nodes come +from, read [Redis setup overview](#redis-setup-overview) and +[Sentinel setup overview](#sentinel-setup-overview). - ```ruby - external_url 'https://gitlab.example.com' - - # Disable all services except Redis - redis['enable'] = true - bootstrap['enable'] = false - nginx['enable'] = false - unicorn['enable'] = false - sidekiq['enable'] = false - postgresql['enable'] = false - gitlab_workhorse['enable'] = false - mailroom['enable'] = false - - # Redis configuration - redis['port'] = 6379 - redis['bind'] = '0.0.0.0' +For a recommended setup that can resist more failures, you will install +the Omnibus GitLab package in `5` **independent** machines, both with +**Redis** and **Sentinel**: - # If you wish to use Redis authentication (recommended) - redis['password'] = 'Redis Password' - ``` +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel -1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. +### Redis setup overview - > **Note**: This `reconfigure` step will result in some errors. - That's OK - don't be alarmed. +You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they +need to be each in a independent machine (see explanation above). -1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations - from running on upgrade. Only the primary GitLab application server should - handle migrations. +You can have additional Redis nodes, that will help survive a situation +where more nodes goes down. Whenever there is only `2` nodes online, a failover +will not be initiated. -## Experimental Redis Sentinel support +As an example, if you have `6` Redis nodes, a maximum of `3` can be +simultaneously down. -> [Introduced][ce-1877] in GitLab 8.11. +Please note that there are different requirements for Sentinel nodes. +If you host them in the same Redis machines, you may need to take +that restrictions into consideration when calculating the amount of +nodes to be provisioned. See [Sentinel setup overview](#sentinel-setup-overview) +documentation for more information. -Since GitLab 8.11, you can configure a list of Redis Sentinel servers that -will monitor a group of Redis servers to provide you with a standard failover -support. +All Redis nodes should be configured the same way and with similar server specs, as +in a failover situation, any **Slave** can be promoted as the new **Master** by +the Sentinel servers. -There is currently one exception to the Sentinel support: `mail_room`, the -component that processes incoming emails. It doesn't support Sentinel yet, but -we hope to integrate a future release that does support it. +The replication requires authentication, so you need to define a password to +protect all Redis nodes and the Sentinels. They will all share the same +password, and all instances must be able to talk to +each other over the network. -To get a better understanding on how to correctly setup Sentinel, please read -the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as -failing to configure it correctly can lead to data loss. +### Sentinel setup overview -The configuration consists of three parts: +Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel +detects that a Redis node is not responding, it will announce that to the +other Sentinels. They have to reach the **quorum**, that is the minimum amount +of Sentinels that agrees a node is down, in order to be able to start a failover. -- Redis setup -- Sentinel setup -- GitLab setup +Whenever the **quorum** is met, the **majority** of all known Sentinel nodes +need to be available and reachable, so that they can elect the Sentinel **leader** +who will take all the decisions to restore the service availability by: -Read carefully how to configure those components below. +- Promoting a new **Master** +- Reconfiguring the other **Slaves** and make them point to the new **Master** +- Announce the new **Master** to every other Sentinel peer +- Reconfigure the old **Master** and demote to **Slave** when it comes back online -### Redis setup +You must have at least `3` Redis Sentinel servers, and they need to +be each in a independent machine (that are believed to fail independently), +ideally in different geographical areas. -You must have at least 2 Redis servers: 1 Master, 1 or more Slaves. -They should be configured the same way and with similar server specs, as -in a failover situation, any Slave can be elected as the new Master by -the Sentinel servers. +You can configure them in the same machines where you've configured the other +Redis servers, but understand that if a whole node goes down, you loose both +a Sentinel and a Redis instance. -In a minimal setup, the only required change for the slaves in `redis.conf` -is the addition of a `slaveof` line pointing to the initial master. -You can increase the security by defining a `requirepass` configuration in -the master, and `masterauth` in slaves. +The number of sentinels should ideally always be an **odd** number, for the +consensus algorithm to be effective in the case of a failure. ---- +In a `3` nodes topology, you can only afford `1` Sentinel node going down. +Whenever the **majority** of the Sentinels goes down, the network partition +protection prevents destructive actions and a failover **will not be started**. -**Configuring your own Redis server** +Here are some examples: -1. Add to the slaves' `redis.conf`: +- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin. +- With `7` sentinels, a maximum of `3` nodes can go down. - ```conf - # IP and port of the master Redis server - slaveof 10.10.10.10 6379 - ``` +The **Leader** election can sometimes fail the voting round when **consensus** +is not achieved (see the odd number of nodes requirement above). In that case, +a new attempt will be made after the amount of time defined in +`sentinel['failover_timeout']` (in milliseconds). -1. Optionally, set up password authentication for increased security. - Add the following to master's `redis.conf`: +>**Note:** +We will see where `sentinel['failover_timeout']` is defined later. + +The `failover_timeout` variable has a lot of different use cases. According to +the official documentation: + +- The time needed to re-start a failover after a previous failover was + already tried against the same master by a given Sentinel, is two + times the failover timeout. + +- The time needed for a slave replicating to a wrong master according + to a Sentinel current configuration, to be forced to replicate + with the right master, is exactly the failover timeout (counting since + the moment a Sentinel detected the misconfiguration). + +- The time needed to cancel a failover that is already in progress but + did not produced any configuration change (SLAVEOF NO ONE yet not + acknowledged by the promoted slave). + +- The maximum time a failover in progress waits for all the slaves to be + reconfigured as slaves of the new master. However even after this time + the slaves will be reconfigured by the Sentinels anyway, but not with + the exact parallel-syncs progression as specified. + +### Available configuration setups + +Based on your infrastructure setup and how you have installed GitLab, there are +multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or +Redis Sentinel bundled with them so you only need to focus on configuration. +Pick the one that suits your needs. + +- [Installations from source][source]: You need to install Redis and Sentinel + yourself. Use the [Redis HA installation from source](redis_source.md) + documentation. +- [Omnibus GitLab **Community Edition** (CE) package][ce]: Redis is bundled, so you + can use the package with only the Redis service enabled as described in steps + 1 and 2 of this document (works for both master and slave setups). To install + and configure Sentinel, jump directly to the Sentinel section in the + [Redis HA installation from source](redis_source.md#step-3-configuring-the-redis-sentinel-instances) documentation. +- [Omnibus GitLab **Enterprise Edition** (EE) package][ee]: Both Redis and Sentinel + are bundled in the package, so you can use the EE package to setup the whole + Redis HA infrastructure (master, slave and Sentinel) which is described in + this document. +- If you have installed GitLab using the Omnibus GitLab packages (CE or EE), + but you want to use your own external Redis server, follow steps 1-3 in the + [Redis HA installation from source](redis_source.md) documentation, then go + straight to step 4 in this guide to + [set up the GitLab application](#step-4-configuring-the-gitlab-application). + +## Configuring Redis HA + +This is the section where we install and setup the new Redis instances. + +>**Notes:** +- We assume that you install GitLab and all HA components from scratch. If you + already have it installed and running, read how to + [switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha). +- Redis nodes (both master and slaves) will need the same password defined in + `redis['password']`. At any time during a failover the Sentinels can + reconfigure a node and change its status from master to slave and vice versa. + +### Prerequisites + +The prerequisites for a HA Redis setup are the following: + +1. Provision the minimum required number of instances as specified in the + [recommended setup](#recommended-setup) section. +1. **Do NOT** install Redis or Redis Sentinel in the same machines your + GitLab application is running on. You can however opt in to install Redis + and Sentinel in the same machine (each in independent ones is recommended + though). +1. All Redis nodes must be able to talk to each other and accept incoming + connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you + change the default ones). +1. The server that hosts the GitLab application must be able to access the + Redis nodes. +1. Protect the nodes from access from external networks ([Internet][it]), using + firewall. + +### Step 1. Configuring the master Redis instance + +1. SSH into the **master** Redis server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```conf - # Optional password authentication for increased security - requirepass "<password>" + ```ruby + # Enable the master role and disable all other services in the machine + # (you can still enable Sentinel). + redis_master_role['enable'] = true + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.1' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'redis-password-goes-here' ``` -1. Then add this line to all the slave servers' `redis.conf`: +1. Only the primary GitLab application server should handle migrations. To + prevent database migrations from running on upgrade, add the following + configuration to your `/etc/gitlab/gitlab.rb` file: - ```conf - masterauth "<password>" + ``` + gitlab_rails['auto_migrate'] = false ``` -1. Restart the Redis services for the changes to take effect. +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. ---- +### Step 2. Configuring the slave Redis instances -**Using Redis via Omnibus** +1. SSH into the **slave** Redis server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. -1. Edit `/etc/gitlab/gitlab.rb` of a master Redis machine (usualy a single machine): +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - ## Redis TCP support (will disable UNIX socket transport) - redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one + # Enable the slave role and disable all other services in the machine + # (you can still enable Sentinel). This will also set automatically + # `redis['master'] = false`. + redis_slave_role['enable'] = true + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.2' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. redis['port'] = 6379 - ## Master redis instance - redis['password'] = '<huge password string here>' - ``` + # The same password for Redeis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' -1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines): + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' - ```ruby - ## Redis TCP support (will disable UNIX socket transport) - redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one - redis['port'] = 6379 + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + ``` - ## Slave redis instance - redis['master_ip'] = '10.10.10.10' # IP of master Redis server - redis['master_port'] = 6379 # Port of master Redis server - redis['master_password'] = "<huge password string here>" +1. To prevent database migrations from running on upgrade, run: + + ``` + sudo touch /etc/gitlab/skip-auto-migrations ``` -1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` + Only the primary GitLab application server should handle migrations. + +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +1. Go through the steps again for all the other slave nodes. --- +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the Sentinels, and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +### Step 3. Configuring the Redis Sentinel instances + +>**Note:** +Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. The +following section assumes you are using Omnibus GitLab Enterprise Edition. +For the Omnibus Community Edition and installations from source, follow the +[Redis HA source install](redis_source.md) guide. + Now that the Redis servers are all set up, let's configure the Sentinel servers. -### Sentinel setup +If you are not sure if your Redis servers are working and replicating +correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) +and fix it before proceeding with Sentinel setup. -We don't provide yet an automated way to setup and run the Sentinel daemon -from Omnibus installation method. You must follow the instructions below and -run it by yourself. +You must have at least `3` Redis Sentinel servers, and they need to +be each in an independent machine. You can configure them in the same +machines where you've configured the other Redis servers. -The support for Sentinel in Ruby has some [caveats](https://github.com/redis/redis-rb/issues/531). -While you can give any name for the `master-group-name` part of the -configuration, as in this example: +With GitLab Enterprise Edition, you can use the Omnibus package to setup +multiple machines with the Sentinel daemon. -```conf -sentinel monitor <master-group-name> <ip> <port> <quorum> -``` +--- -,for it to work in Ruby, you have to use the "hostname" of the master Redis -server, otherwise you will get an error message like: -`Redis::CannotConnectError: No sentinels available.`. Read -[Sentinel troubleshooting](#sentinel-troubleshooting) for more information. +1. SSH into the server that will host Redis Sentinel. +1. **You can omit this step if the Sentinels will be hosted in the same node as + the other Redis instances.** -Here is an example configuration file (`sentinel.conf`) for a Sentinel node: + [Download/install](https://about.gitlab.com/downloads-ee) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. -```conf -port 26379 -sentinel monitor master-redis.example.com 10.10.10.10 6379 1 -sentinel down-after-milliseconds master-redis.example.com 10000 -sentinel config-epoch master-redis.example.com 0 -sentinel leader-epoch master-redis.example.com 0 -``` +1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the + Sentinels in the same node as the other Redis instances, some values might + be duplicate below): ---- + ```ruby + redis_sentinel_role['enable'] = true -The final part is to inform the main GitLab application server of the Redis -master and the new sentinels servers. + # Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' -### GitLab setup + # The same password for Redis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' -You can enable or disable sentinel support at any time in new or existing -installations. From the GitLab application perspective, all it requires is -the correct credentials for the master Redis and for a few Sentinel nodes. + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' -It doesn't require a list of all Sentinel nodes, as in case of a failure, -the application will need to query only one of them. + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 ->**Note:** -The following steps should be performed in the [GitLab application server](gitlab.md). + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel + sentinel['bind'] = '10.0.0.1' + + # Port that Sentinel listens on, uncomment to change to non default. Defaults + # to `26379`. + # sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to master failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the master. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the master being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + # sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## - The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + # sentinel['failover_timeout'] = 60000 + ``` -**For source based installations** +1. To prevent database migrations from running on upgrade, run: -1. Edit `/home/git/gitlab/config/resque.yml` following the example in - `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels - line, changing to the correct server credentials. -1. Restart GitLab for the changes to take effect. + ``` + sudo touch /etc/gitlab/skip-auto-migrations + ``` -**For Omnibus installations** + Only the primary GitLab application server should handle migrations. +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +1. Go through the steps again for all the other Sentinel nodes. + +### Step 4. Configuring the GitLab application + +The final part is to inform the main GitLab application server of the Redis +Sentinels servers and authentication credentials. + +You can enable or disable Sentinel support at any time in new or existing +installations. From the GitLab application perspective, all it requires is +the correct credentials for the Sentinel nodes. + +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at least one of the listed. + +>**Note:** +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels on it for a HA setup. + +1. SSH into the server where the GitLab application is installed. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: - ```ruby - gitlab-rails['redis_host'] = "master-redis.example.com" - gitlab-rails['redis_port'] = 6379 - gitlab-rails['redis_password'] = '<huge password string here>' - gitlab-rails['redis_sentinels'] = [ - {'host' => '10.10.10.1', 'port' => 26379}, - {'host' => '10.10.10.2', 'port' => 26379}, - {'host' => '10.10.10.3', 'port' => 26379} + ``` + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + ## The same password for Redis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' + + ## A list of sentinels with `host` and `port` + gitlab_rails['redis_sentinels'] = [ + {'host' => '10.0.0.1', 'port' => 26379}, + {'host' => '10.0.0.2', 'port' => 26379}, + {'host' => '10.0.0.3', 'port' => 26379} ] ``` -1. [Reconfigure] the GitLab for the changes to take effect. +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -### Sentinel troubleshooting +## Switching from an existing single-machine installation to Redis HA -If you get an error like: `Redis::CannotConnectError: No sentinels available.`, -there may be something wrong with your configuration files or it can be related -to [this issue][gh-531] ([pull request][gh-534] that should make things better). +If you already have a single-machine GitLab install running, you will need to +replicate from this machine first, before de-activating the Redis instance +inside it. + +Your single-machine install will be the initial **Master**, and the `3` others +should be configured as **Slave** pointing to this machine. + +After replication catches up, you will need to stop services in the +single-machine install, to rotate the **Master** to one of the new nodes. -It's a bit rigid the way you have to config `resque.yml` and `sentinel.conf`, -otherwise `redis-rb` will not work properly. +Make the required changes in configuration and restart the new nodes again. -The hostname ('my-primary-redis') of the primary Redis server (`sentinel.conf`) -**must** match the one configured in GitLab (`resque.yml` for source installations -or `gitlab-rails['redis_*']` in Omnibus) and it must be valid ex: +To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: -```conf -# sentinel.conf: -sentinel monitor my-primary-redis 10.10.10.10 6379 1 -sentinel down-after-milliseconds my-primary-redis 10000 -sentinel config-epoch my-primary-redis 0 -sentinel leader-epoch my-primary-redis 0 +```ruby +redis['enable'] = false ``` -```yaml -# resque.yaml -production: - url: redis://my-primary-redis:6378 - sentinels: - - - host: slave1 - port: 26380 # point to sentinel, not to redis port - - - host: slave2 - port: 26381 # point to sentinel, not to redis port +If you fail to replicate first, you may loose data (unprocessed background jobs). + +## Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels + +>**Note:** +Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For +different setups, read the +[available configuration setups](#available-configuration-setups) section. + +In this example we consider that all servers have an internal network +interface with IPs in the `10.0.0.x` range, and that they can connect +to each other using these IPs. + +In a real world usage, you would also setup firewall rules to prevent +unauthorized access from other machines and block traffic from the +outside (Internet). + +We will use the same `3` nodes with **Redis** + **Sentinel** topology +discussed in [Redis setup overview](#redis-setup-overview) and +[Sentinel setup overview](#sentinel-setup-overview) documentation. + +Here is a list and description of each **machine** and the assigned **IP**: + +* `10.0.0.1`: Redis Master + Sentinel 1 +* `10.0.0.2`: Redis Slave 1 + Sentinel 2 +* `10.0.0.3`: Redis Slave 2 + Sentinel 3 +* `10.0.0.4`: GitLab application + +Please note that after the initial configuration, if a failover is initiated +by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** +will change permanently (including in `redis.conf`) from one node to the other, +until a new failover is initiated again. + +The same thing will happen with `sentinel.conf` that will be overridden after the +initial execution, after any new sentinel node starts watching the **Master**, +or a failover promotes a different **Master** node. + +### Example configuration for Redis master and Sentinel 1 + +In `/etc/gitlab/gitlab.rb`: + +```ruby +redis_master_role['enable'] = true +redis_sentinel_role['enable'] = true +redis['bind'] = '10.0.0.1' +redis['port'] = 6379 +redis['password'] = 'redis-password-goes-here' +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default +sentinel['bind'] = '10.0.0.1' +# sentinel['port'] = 26379 # uncomment to change default port +sentinel['quorum'] = 2 +# sentinel['down_after_milliseconds'] = 10000 +# sentinel['failover_timeout'] = 60000 +``` + +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +### Example configuration for Redis slave 1 and Sentinel 2 + +In `/etc/gitlab/gitlab.rb`: + +```ruby +redis_slave_role['enable'] = true +redis_sentinel_role['enable'] = true +redis['bind'] = '10.0.0.2' +redis['port'] = 6379 +redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +sentinel['bind'] = '10.0.0.2' +# sentinel['port'] = 26379 # uncomment to change default port +sentinel['quorum'] = 2 +# sentinel['down_after_milliseconds'] = 10000 +# sentinel['failover_timeout'] = 60000 +``` + +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +### Example configuration for Redis slave 2 and Sentinel 3 + +In `/etc/gitlab/gitlab.rb`: + +```ruby +redis_slave_role['enable'] = true +redis_sentinel_role['enable'] = true +redis['bind'] = '10.0.0.3' +redis['port'] = 6379 +redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +sentinel['bind'] = '10.0.0.3' +# sentinel['port'] = 26379 # uncomment to change default port +sentinel['quorum'] = 2 +# sentinel['down_after_milliseconds'] = 10000 +# sentinel['failover_timeout'] = 60000 ``` -When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +### Example configuration for the GitLab application + +In `/etc/gitlab/gitlab.rb`: + +```ruby +redis['master_name'] = 'gitlab-redis' +redis['password'] = 'redis-password-goes-here' +gitlab_rails['redis_sentinels'] = [ + {'host' => '10.0.0.1', 'port' => 26379}, + {'host' => '10.0.0.2', 'port' => 26379}, + {'host' => '10.0.0.3', 'port' => 26379} +] +``` + +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +## Advanced configuration + +Omnibus GitLab configures some things behind the curtains to make the sysadmins' +lives easier. If you want to know what happens underneath keep reading. + +### Control running services + +In the previous example, we've used `redis_sentinel_role` and +`redis_master_role` which simplifies the amount of configuration changes. + +If you want more control, here is what each one sets for you automatically +when enabled: + +```ruby +## Redis Sentinel Role +redis_sentinel_role['enable'] = true + +# When Sentinel Role is enabled, the following services are also enabled +sentinel['enable'] = true + +# The following services are disabled +redis['enable'] = false +bootstrap['enable'] = false +nginx['enable'] = false +postgresql['enable'] = false +gitlab_rails['enable'] = false +mailroom['enable'] = false + +------- + +## Redis master/slave Role +redis_master_role['enable'] = true # enable only one of them +redis_slave_role['enable'] = true # enable only one of them + +# When Redis Master or Slave role are enabled, the following services are +# enabled/disabled. Note that if Redis and Sentinel roles are combined, both +# services will be enabled. + +# The following services are disabled +sentinel['enable'] = false +bootstrap['enable'] = false +nginx['enable'] = false +postgresql['enable'] = false +gitlab_rails['enable'] = false +mailroom['enable'] = false + +# For Redis Slave role, also change this setting from default 'true' to 'false': +redis['master'] = false +``` + +You can find the relevant attributes defined in [gitlab_rails.rb][omnifile]. + +## Troubleshooting + +There are a lot of moving parts that needs to be taken care carefully +in order for the HA setup to work as expected. + +Before proceeding with the troubleshooting below, check your firewall rules: + +- Redis machines + - Accept TCP connection in `6379` + - Connect to the other Redis machines via TCP in `6379` +- Sentinel machines + - Accept TCP connection in `26379` + - Connect to other Sentinel machines via TCP in `26379` + - Connect to the Redis machines via TCP in `6379` + +### Troubleshooting Redis replication + +You can check if everything is correct by connecting to each server using +`redis-cli` application, and sending the `INFO` command. + +If authentication was correctly defined, it should fail with: +`NOAUTH Authentication required` error. Try to authenticate with the +previous defined password with `AUTH redis-password-goes-here` and +try the `INFO` command again. + +Look for the `# Replication` section where you should see some important +information like the `role` of the server. + +When connected to a `master` redis, you will see the number of connected +`slaves`, and a list of each with connection details: + +``` +# Replication +role:master +connected_slaves:1 +slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1 +master_repl_offset:208037658 +repl_backlog_active:1 +repl_backlog_size:1048576 +repl_backlog_first_byte_offset:206989083 +repl_backlog_histlen:1048576 +``` + +When it's a `slave`, you will see details of the master connection and if +its `up` or `down`: + +``` +# Replication +role:slave +master_host:10.133.1.58 +master_port:6379 +master_link_status:up +master_last_io_seconds_ago:1 +master_sync_in_progress:0 +slave_repl_offset:208096498 +slave_priority:100 +slave_read_only:1 +connected_slaves:0 +master_repl_offset:0 +repl_backlog_active:0 +repl_backlog_size:1048576 +repl_backlog_first_byte_offset:0 +repl_backlog_histlen:0 +``` + +### Troubleshooting Sentinel + +If you get an error like: `Redis::CannotConnectError: No sentinels available.`, +there may be something wrong with your configuration files or it can be related +to [this issue][gh-531]. + +You must make sure you are defining the same value in `redis['master_name']` +and `redis['master_pasword']` as you defined for your sentinel node. + +The way the redis connector `redis-rb` works with sentinel is a bit +non-intuitive. We try to hide the complexity in omnibus, but it still requires +a few extra configs. --- @@ -273,7 +788,7 @@ To make sure your configuration is correct: sudo gitlab-rails console # For source installations - sudo -u git rails console RAILS_ENV=production + sudo -u git rails console production ``` 1. Run in the console: @@ -288,8 +803,8 @@ To make sure your configuration is correct: 1. To simulate a failover on master Redis, SSH into the Redis server and run: ```bash - # port must match your master redis port - redis-cli -h localhost -p 6379 DEBUG sleep 60 + # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one + redis-cli -h localhost -p 6379 DEBUG sleep 20 ``` 1. Then back in the Rails console from the first step, run: @@ -301,10 +816,26 @@ To make sure your configuration is correct: You should see a different port after a few seconds delay (the failover/reconnect time). ---- -Read more on high-availability configuration: +## Changelog + +Changes to Redis HA over time. + +**8.14** + +- Redis Sentinel support is production-ready and bundled in the Omnibus GitLab + Enterprise Edition package +- Documentation restructure for better readability + +**8.11** + +- Experimental Redis Sentinel support was added + +## Further reading + +Read more on High Availability: +1. [High Availability Overview](README.md) 1. [Configure the database](database.md) 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) @@ -315,3 +846,10 @@ Read more on high-availability configuration: [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [gh-531]: https://github.com/redis/redis-rb/issues/531 [gh-534]: https://github.com/redis/redis-rb/issues/534 +[redis]: http://redis.io/ +[sentinel]: http://redis.io/topics/sentinel +[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb +[source]: ../../install/installation.md +[ce]: https://about.gitlab.com/downloads +[ee]: https://about.gitlab.com/downloads-ee +[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md new file mode 100644 index 00000000000..3629772b8af --- /dev/null +++ b/doc/administration/high_availability/redis_source.md @@ -0,0 +1,366 @@ +# Configuring non-Omnibus Redis for GitLab HA + +This is the documentation for configuring a Highly Available Redis setup when +you have installed Redis all by yourself and not using the bundled one that +comes with the Omnibus packages. + +We cannot stress enough the importance of reading the +[Overview section](redis.md#overview) of the Omnibus Redis HA as it provides +some invaluable information to the configuration of Redis. Please proceed to +read it before going forward with this guide. + +We also highly recommend that you use the Omnibus GitLab packages, as we +optimize them specifically for GitLab, and we will take care of upgrading Redis +to the latest supported version. + +If you're not sure whether this guide is for you, please refer to +[Available configuration setups](redis.md#available-configuration-setups) in +the Omnibus Redis HA documentation. + +## Configuring your own Redis server + +This is the section where we install and setup the new Redis instances. + +### Prerequisites + +- All Redis servers in this guide must be configured to use a TCP connection + instead of a socket. To configure Redis to use TCP connections you need to + define both `bind` and `port` in the Redis config file. You can bind to all + interfaces (`0.0.0.0`) or specify the IP of the desired interface + (e.g., one from an internal network). +- Since Redis 3.2, you must define a password to receive external connections + (`requirepass`). +- If you are using Redis with Sentinel, you will also need to define the same + password for the slave password definition (`masterauth`) in the same instance. + +In addition, read the prerequisites as described in the +[Omnibus Redis HA document](redis.md#prerequisites) since they provide some +valuable information for the general setup. + +### Step 1. Configuring the master Redis instance + +Assuming that the Redis master instance IP is `10.0.0.1`: + +1. [Install Redis](../../install/installation.md#6-redis) +1. Edit `/etc/redis/redis.conf`: + + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 + + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 + + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` + +1. Restart the Redis service for the changes to take effect. + +### Step 2. Configuring the slave Redis instances + +Assuming that the Redis slave instance IP is `10.0.0.2`: + +1. [Install Redis](../../install/installation.md#6-redis) +1. Edit `/etc/redis/redis.conf`: + + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.2 + + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 + + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + + ## Define `slaveof` pointing to the Redis master instance with IP and port. + slaveof 10.0.0.1 6379 + ``` + +1. Restart the Redis service for the changes to take effect. +1. Go through the steps again for all the other slave nodes. + +### Step 3. Configuring the Redis Sentinel instances + +Sentinel is a special type of Redis server. It inherits most of the basic +configuration options you can define in `redis.conf`, with specific ones +starting with `sentinel` prefix. + +Assuming that the Redis Sentinel is installed on the same instance as Redis +master with IP `10.0.0.1` (some settings might overlap with the master): + +1. [Install Redis Sentinel](http://redis.io/topics/sentinel) +1. Edit `/etc/redis/sentinel.conf`: + + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 + + ## Define a `port` to force Sentinel to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 26379 + + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + + ## Define with `sentinel auth-pass` the same shared password you have + ## defined for both Redis master and slaves instances. + sentinel auth-pass gitlab-redis redis-password-goes-here + + ## Define with `sentinel monitor` the IP and port of the Redis + ## master node, and the quorum required to start a failover. + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + + ## Define with `sentinel down-after-milliseconds` the time in `ms` + ## that an unresponsive server will be considered down. + sentinel down-after-milliseconds gitlab-redis 10000 + + ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple + ## meanings: + ## + ## * The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## * The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## * The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## * The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + sentinel failover_timeout 30000 + ``` +1. Restart the Redis service for the changes to take effect. +1. Go through the steps again for all the other Sentinel nodes. + +### Step 4. Configuring the GitLab application + +You can enable or disable Sentinel support at any time in new or existing +installations. From the GitLab application perspective, all it requires is +the correct credentials for the Sentinel nodes. + +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at least one of listed ones. + +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels in the same machine for a HA +setup: + +1. Edit `/home/git/gitlab/config/resque.yml` following the example in + [resque.yml.example][resque], and uncomment the Sentinel lines, pointing to + the correct server credentials: + + ```yaml + # resque.yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels + +In this example we consider that all servers have an internal network +interface with IPs in the `10.0.0.x` range, and that they can connect +to each other using these IPs. + +In a real world usage, you would also setup firewall rules to prevent +unauthorized access from other machines, and block traffic from the +outside ([Internet][it]). + +For this example, **Sentinel 1** will be configured in the same machine as the +**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the +**Slave 1** and **Slave 2** respectively. + +Here is a list and description of each **machine** and the assigned **IP**: + +* `10.0.0.1`: Redis Master + Sentinel 1 +* `10.0.0.2`: Redis Slave 1 + Sentinel 2 +* `10.0.0.3`: Redis Slave 2 + Sentinel 3 +* `10.0.0.4`: GitLab application + +Please note that after the initial configuration, if a failover is initiated +by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** +will change permanently (including in `redis.conf`) from one node to the other, +until a new failover is initiated again. + +The same thing will happen with `sentinel.conf` that will be overridden after the +initial execution, after any new sentinel node starts watching the **Master**, +or a failover promotes a different **Master** node. + +### Example configuration for Redis master and Sentinel 1 + +1. In `/etc/redis/redis.conf`: + + ```conf + bind 10.0.0.1 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` + +1. In `/etc/redis/sentinel.conf`: + + ```conf + bind 10.0.0.1 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` + +1. Restart the Redis service for the changes to take effect. + +### Example configuration for Redis slave 1 and Sentinel 2 + +1. In `/etc/redis/redis.conf`: + + ```conf + bind 10.0.0.2 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` + +1. In `/etc/redis/sentinel.conf`: + + ```conf + bind 10.0.0.2 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` + +1. Restart the Redis service for the changes to take effect. + +### Example configuration for Redis slave 2 and Sentinel 3 + +1. In `/etc/redis/redis.conf`: + + ```conf + bind 10.0.0.3 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` + +1. In `/etc/redis/sentinel.conf`: + + ```conf + bind 10.0.0.3 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` + +1. Restart the Redis service for the changes to take effect. + +### Example configuration of the GitLab application + +1. Edit `/home/git/gitlab/config/resque.yml`: + + ```yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Troubleshooting + +We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained +in the documentation for Omnibus GitLab installations. Here we will list only +the things that are specific to a source installation. + +If you get an error in GitLab like `Redis::CannotConnectError: No sentinels available.`, +there may be something wrong with your configuration files or it can be related +to [this upstream issue][gh-531]. + +You must make sure that `resque.yml` and `sentinel.conf` are configured correctly, +otherwise `redis-rb` will not work properly. + +The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) +**must** be used as the hostname in GitLab (`resque.yml`): + +```conf +# sentinel.conf: +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel config-epoch gitlab-redis 0 +sentinel leader-epoch gitlab-redis 0 +``` + +```yaml +# resque.yaml +production: + url: redis://:myredispassword@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port +``` + +When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel). + +[gh-531]: https://github.com/redis/redis-rb/issues/531 +[downloads]: https://about.gitlab.com/downloads +[restart]: ../restart_gitlab.md#installations-from-source +[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png diff --git a/doc/administration/img/custom_hooks_error_msg.png b/doc/administration/img/custom_hooks_error_msg.png Binary files differindex 92e87e15fb3..1b3277bef16 100644 --- a/doc/administration/img/custom_hooks_error_msg.png +++ b/doc/administration/img/custom_hooks_error_msg.png diff --git a/doc/administration/img/high_availability/active-active-diagram.png b/doc/administration/img/high_availability/active-active-diagram.png Binary files differindex 81259e0ae93..4f5984b88fe 100644 --- a/doc/administration/img/high_availability/active-active-diagram.png +++ b/doc/administration/img/high_availability/active-active-diagram.png diff --git a/doc/administration/img/high_availability/active-passive-diagram.png b/doc/administration/img/high_availability/active-passive-diagram.png Binary files differindex f69ff1d0357..3b42ce5911c 100644 --- a/doc/administration/img/high_availability/active-passive-diagram.png +++ b/doc/administration/img/high_availability/active-passive-diagram.png diff --git a/doc/administration/img/housekeeping_settings.png b/doc/administration/img/housekeeping_settings.png Binary files differindex 6ebc6205635..acc4506993a 100644 --- a/doc/administration/img/housekeeping_settings.png +++ b/doc/administration/img/housekeeping_settings.png diff --git a/doc/administration/img/raketasks/check_repos_output.png b/doc/administration/img/raketasks/check_repos_output.png Binary files differnew file mode 100644 index 00000000000..7fda2ba0c0f --- /dev/null +++ b/doc/administration/img/raketasks/check_repos_output.png diff --git a/doc/administration/img/repository_storages_admin_ui.png b/doc/administration/img/repository_storages_admin_ui.png Binary files differindex 6481baca1ad..3e76c5b282c 100644 --- a/doc/administration/img/repository_storages_admin_ui.png +++ b/doc/administration/img/repository_storages_admin_ui.png diff --git a/doc/administration/logs.md b/doc/administration/logs.md index d757a3c2a66..4b8d5c5cc87 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,4 +1,4 @@ -## Log system +# Log system GitLab has an advanced log system where everything is logged so that you can analyze your instance using various system log files. In addition to @@ -9,10 +9,10 @@ documentation](http://docs.gitlab.com/ee/administration/audit_events.html) System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. -### production.log +## `production.log` This file lives in `/var/log/gitlab/gitlab-rails/production.log` for -omnibus package or in `/home/git/gitlab/log/production.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/production.log` for installations from source. (When Gitlab is running in an environment other than production, the corresponding logfile is shown here.) @@ -46,10 +46,10 @@ In this example we can see that server processed an HTTP request with URL 19:34:53 +0200. Also we can see that request was processed by `Projects::TreeController`. -### application.log +## `application.log` This file lives in `/var/log/gitlab/gitlab-rails/application.log` for -omnibus package or in `/home/git/gitlab/log/application.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/application.log` for installations from source. It helps you discover events happening in your instance such as user creation, @@ -63,10 +63,10 @@ October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk) was October 07, 2014 11:25: Project "project133" was removed ``` -### githost.log +## `githost.log` This file lives in `/var/log/gitlab/gitlab-rails/githost.log` for -omnibus package or in `/home/git/gitlab/log/githost.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/githost.log` for installations from source. GitLab has to interact with Git repositories but in some rare cases @@ -81,10 +81,10 @@ December 03, 2014 13:20 -> ERROR -> Command failed [1]: /usr/bin/git --git-dir=/ error: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git' ``` -### sidekiq.log +## `sidekiq.log` This file lives in `/var/log/gitlab/gitlab-rails/sidekiq.log` for -omnibus package or in `/home/git/gitlab/log/sidekiq.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq.log` for installations from source. GitLab uses background jobs for processing tasks which can take a long @@ -96,10 +96,10 @@ this file. For example: 2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"} ``` -### gitlab-shell.log +## `gitlab-shell.log` This file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log` for -omnibus package or in `/home/git/gitlab-shell/gitlab-shell.log` for +Omnibus GitLab packages or in `/home/git/gitlab-shell/gitlab-shell.log` for installations from source. GitLab shell is used by Gitlab for executing Git commands and provide @@ -110,10 +110,10 @@ I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. ``` -### unicorn\_stderr.log +## `unicorn\_stderr.log` This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for -omnibus package or in `/home/git/gitlab/log/unicorn_stderr.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for installations from source. Unicorn is a high-performance forking Web server which is used for @@ -136,3 +136,13 @@ I, [2015-02-13T07:16:01.530733 #9047] INFO -- : reaped #<Process::Status: pid 9 I, [2015-02-13T07:16:01.534501 #13379] INFO -- : worker=1 spawned pid=13379 I, [2015-02-13T07:16:01.534848 #13379] INFO -- : worker=1 ready ``` + +## `repocheck.log` + +This file lives in `/var/log/gitlab/gitlab-rails/repocheck.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/repocheck.log` for +installations from source. + +It logs information whenever a [repository check is run][repocheck] on a project. + +[repocheck]: repository_checks.md diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png b/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png Binary files differindex 7e34fad71ce..51eef90068d 100644 --- a/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png +++ b/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_import.png b/doc/administration/monitoring/performance/img/grafana_dashboard_import.png Binary files differindex f97624365c7..7761ea00522 100644 --- a/doc/administration/monitoring/performance/img/grafana_dashboard_import.png +++ b/doc/administration/monitoring/performance/img/grafana_dashboard_import.png diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png b/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png Binary files differindex 7d50e4c88c2..3e749eb8f9d 100644 --- a/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png +++ b/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_empty.png b/doc/administration/monitoring/performance/img/grafana_data_source_empty.png Binary files differindex aa39a53acae..33fcaaaef64 100644 --- a/doc/administration/monitoring/performance/img/grafana_data_source_empty.png +++ b/doc/administration/monitoring/performance/img/grafana_data_source_empty.png diff --git a/doc/administration/monitoring/performance/img/grafana_save_icon.png b/doc/administration/monitoring/performance/img/grafana_save_icon.png Binary files differindex c740e33cd1c..c18f2147e9d 100644 --- a/doc/administration/monitoring/performance/img/grafana_save_icon.png +++ b/doc/administration/monitoring/performance/img/grafana_save_icon.png diff --git a/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png b/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png Binary files differindex db396423e30..13bfd097b81 100644 --- a/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png +++ b/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png diff --git a/doc/administration/monitoring/performance/img/request_profile_result.png b/doc/administration/monitoring/performance/img/request_profile_result.png Binary files differindex 73e2fdcab67..8ebd74c2d3c 100644 --- a/doc/administration/monitoring/performance/img/request_profile_result.png +++ b/doc/administration/monitoring/performance/img/request_profile_result.png diff --git a/doc/administration/monitoring/performance/img/request_profiling_token.png b/doc/administration/monitoring/performance/img/request_profiling_token.png Binary files differindex 04d87567816..9160407e028 100644 --- a/doc/administration/monitoring/performance/img/request_profiling_token.png +++ b/doc/administration/monitoring/performance/img/request_profiling_token.png diff --git a/doc/administration/operations.md b/doc/administration/operations.md index 4b582d16b64..0daceb98d99 100644 --- a/doc/administration/operations.md +++ b/doc/administration/operations.md @@ -1,6 +1,7 @@ # GitLab operations - [Sidekiq MemoryKiller](operations/sidekiq_memory_killer.md) +- [Sidekiq Job throttling](operations/sidekiq_job_throttling.md) - [Cleaning up Redis sessions](operations/cleaning_up_redis_sessions.md) - [Understanding Unicorn and unicorn-worker-killer](operations/unicorn.md) - [Moving repositories to a new location](operations/moving_repositories.md) diff --git a/doc/administration/operations/img/sidekiq_job_throttling.png b/doc/administration/operations/img/sidekiq_job_throttling.png Binary files differnew file mode 100644 index 00000000000..dcf40b4bf17 --- /dev/null +++ b/doc/administration/operations/img/sidekiq_job_throttling.png diff --git a/doc/administration/operations/sidekiq_job_throttling.md b/doc/administration/operations/sidekiq_job_throttling.md new file mode 100644 index 00000000000..ddeaa22e288 --- /dev/null +++ b/doc/administration/operations/sidekiq_job_throttling.md @@ -0,0 +1,33 @@ +# Sidekiq Job throttling + +> Note: Introduced with GitLab 8.14 + +When your GitLab installation needs to handle tens of thousands of background +jobs, it can be convenient to throttle queues that do not need to be executed +immediately, e.g. long running jobs like Pipelines, thus allowing jobs that do +need to be executed immediately to have access to more resources. + +In order to accomplish this, you can limit the amount of workers that certain +slow running queues can have available. This is what we call Sidekiq Job +Throttling. Depending on your infrastructure, you might have different slow +running queues, which is why you can choose which queues you want to throttle +and by how much you want to throttle them. + +These settings are available in the Application Settings of your GitLab +installation. + + + +The throttle factor determines the maximum number of workers a queue can run on. +This value gets multiplied by `:concurrency` value set in the Sidekiq settings +and rounded up to the closest full integer. + +So, for example, you set the `:concurrency` to 25 and the `Throttling factor` to +0.1, the maximum workers assigned to the selected queues would be 3. + +```ruby +queue_limit = (factor * Sidekiq.options[:concurrency]).ceil +``` + +After enabling the job throttling, you will need to restart your GitLab +instance, in order for the changes to take effect.
\ No newline at end of file diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md new file mode 100644 index 00000000000..d1d2fed4861 --- /dev/null +++ b/doc/administration/raketasks/check.md @@ -0,0 +1,97 @@ +# Check Rake Tasks + +## Repository Integrity + +Even though Git is very resilient and tries to prevent data integrity issues, +there are times when things go wrong. The following Rake tasks intend to +help GitLab administrators diagnose problem repositories so they can be fixed. + +There are 3 things that are checked to determine integrity. + +1. Git repository file system check ([git fsck](https://git-scm.com/docs/git-fsck)). + This step verifies the connectivity and validity of objects in the repository. +1. Check for `config.lock` in the repository directory. +1. Check for any branch/references lock files in `refs/heads`. + +It's important to note that the existence of `config.lock` or reference locks +alone do not necessarily indicate a problem. Lock files are routinely created +and removed as Git and GitLab perform operations on the repository. They serve +to prevent data integrity issues. However, if a Git operation is interrupted these +locks may not be cleaned up properly. + +The following symptoms may indicate a problem with repository integrity. If users +experience these symptoms you may use the rake tasks described below to determine +exactly which repositories are causing the trouble. + +- Receiving an error when trying to push code - `remote: error: cannot lock ref` +- A 500 error when viewing the GitLab dashboard or when accessing a specific project. + +### Check all GitLab repositories + +This task loops through all repositories on the GitLab server and runs the +3 integrity checks described previously. + +**Omnibus Installation** + +``` +sudo gitlab-rake gitlab:repo:check +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake gitlab:repo:check RAILS_ENV=production +``` + +### Check repositories for a specific user + +This task checks all repositories that a specific user has access to. This is important +because sometimes you know which user is experiencing trouble but you don't know +which project might be the cause. + +If the rake task is executed without brackets at the end, you will be prompted +to enter a username. + +**Omnibus Installation** + +```bash +sudo gitlab-rake gitlab:user:check_repos +sudo gitlab-rake gitlab:user:check_repos[<username>] +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake gitlab:user:check_repos RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:user:check_repos[<username>] RAILS_ENV=production +``` + +Example output: + + + +## LDAP Check + +The LDAP check Rake task will test the bind_dn and password credentials +(if configured) and will list a sample of LDAP users. This task is also +executed as part of the `gitlab:check` task, but can run independently +using the command below. + +**Omnibus Installation** + +``` +sudo gitlab-rake gitlab:ldap:check +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production +``` + +By default, the task will return a sample of 100 LDAP users. Change this +limit by passing a number to the check task: + +```bash +rake gitlab:ldap:check[50] +``` diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 5a9a1582877..14cd7a03826 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -105,6 +105,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". gitlab_rails['incoming_email_mailbox_name'] = "inbox" + # The IDLE command timeout. + gitlab_rails['incoming_email_idle_timeout'] = 60 ``` ```ruby @@ -133,6 +135,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". gitlab_rails['incoming_email_mailbox_name'] = "inbox" + # The IDLE command timeout. + gitlab_rails['incoming_email_idle_timeout'] = 60 ``` 1. Reconfigure GitLab and restart mailroom for the changes to take effect: @@ -192,6 +196,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" + # The IDLE command timeout. + idle_timeout: 60 ``` ```yaml @@ -221,6 +227,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" + # The IDLE command timeout. + idle_timeout: 60 ``` 1. Enable `mail_room` in the init script at `/etc/default/gitlab`: @@ -277,6 +285,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" + # The IDLE command timeout. + idle_timeout: 60 ``` As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. |
