diff options
Diffstat (limited to 'doc/administration')
65 files changed, 1780 insertions, 717 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index a80ff330e03..02de2caf558 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -73,6 +73,8 @@ From there, you can see the following actions: - User was added to project and with which [permissions] - Permission changes of a user assigned to a project - User was removed from project +- Project export was downloaded +- Project repository was downloaded ### Instance events **(PREMIUM ONLY)** @@ -94,6 +96,7 @@ recorded: - Changed password - Ask for password reset - Grant OAuth access +- Started/stopped user impersonation It is possible to filter particular actions by choosing an audit data type from the filter drop-down. You can further filter by specific group, project or user diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 65d36612d85..18c415b5ff7 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -52,7 +52,7 @@ section. **Admin Area > Users**. You will find the option of the access level under the 'Access' section. -  +  1. Click **Save changes** or **Create user** for the changes to take effect. diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_1.png b/doc/administration/auth/img/google_secure_ldap_add_step_1.png Binary files differindex fd254443d75..bee9c602a14 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_1.png +++ b/doc/administration/auth/img/google_secure_ldap_add_step_1.png diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_2.png b/doc/administration/auth/img/google_secure_ldap_add_step_2.png Binary files differindex 611a21ae03c..b127410cb8c 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_2.png +++ b/doc/administration/auth/img/google_secure_ldap_add_step_2.png diff --git a/doc/administration/auth/img/google_secure_ldap_client_settings.png b/doc/administration/auth/img/google_secure_ldap_client_settings.png Binary files differindex 3c0b3f3d4bd..868e6645f56 100644 --- a/doc/administration/auth/img/google_secure_ldap_client_settings.png +++ b/doc/administration/auth/img/google_secure_ldap_client_settings.png diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 2f2ee8a27d3..34f3cfa353f 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -4,37 +4,27 @@ type: reference # LDAP Additions in GitLab EE **(STARTER ONLY)** -This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP -features specific to GitLab Enterprise Edition Starter, Premium and Ultimate. +This section documents LDAP features specific to to GitLab Enterprise Edition +[Starter](https://about.gitlab.com/pricing/#self-managed) and above. -## Overview - -[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) -stands for **Lightweight Directory Access Protocol**, which -is a standard application protocol for -accessing and maintaining distributed directory information services -over an Internet Protocol (IP) network. - -GitLab integrates with LDAP to support **user authentication**. This integration -works with most LDAP-compliant directory servers, including Microsoft Active -Directory, Apple Open Directory, Open LDAP, and 389 Server. -**GitLab Enterprise Edition** includes enhanced integration, including group -membership syncing. +For documentation relevant to both Community Edition and Enterprise Edition, +see the main [LDAP documentation](ldap.md). ## Use cases -- User sync: Once a day, GitLab will update users against LDAP +- User sync: Once a day, GitLab will update users against LDAP. - Group sync: Once an hour, GitLab will update group membership - based on LDAP group members + based on LDAP group members. -## Multiple LDAP servers +## Multiple LDAP servers **(STARTER ONLY)** With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers that your GitLab instance will connect to. -To add another LDAP server, you can start by duplicating the settings under -[the main configuration](ldap.md#configuration) and edit them to match the -additional LDAP server. +To add another LDAP server: + +1. Duplicating the settings under [the main configuration](ldap.md#configuration). +1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. This ID will be stored in the database so that GitLab can remember which LDAP @@ -47,19 +37,19 @@ users against LDAP. The process will execute the following access checks: -1. Ensure the user is still present in LDAP -1. If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This will only be checked if - `active_directory: true` is set in the LDAP configuration [^1] +- Ensure the user is still present in LDAP. +- If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration. [^1] The user will be set to `ldap_blocked` state in GitLab if the above conditions fail. This means the user will not be able to login or push/pull code. The process will also update the following user information: -1. Email address -1. If `sync_ssh_keys` is set, SSH public keys -1. If Kerberos is enabled, Kerberos identity +- Email address. +- If `sync_ssh_keys` is set, SSH public keys. +- If Kerberos is enabled, Kerberos identity. NOTE: **Note:** The LDAP sync process updates existing users while new users will @@ -72,9 +62,6 @@ first time GitLab will trigger a sync for groups the user should be a member of. That way they don't need to wait for the hourly sync to be granted access to their groups and projects. -In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can -also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). - A group sync process will run every hour on the hour, and `group_base` must be set in LDAP configuration for LDAP synchronizations based on group CN to work. This allows GitLab group membership to be automatically updated based on LDAP group members. @@ -120,13 +107,26 @@ following. 1. [Restart GitLab][restart] for the changes to take effect. ---- - To take advantage of group sync, group owners or maintainers will need to create an -LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP -groups and/or filters can be linked with a single GitLab group. When the link is -created, an access level/role is specified (Guest, Reporter, Developer, Maintainer, -or Owner). +LDAP group link in their group **Settings > LDAP Groups** page. + +Multiple LDAP groups and [filters](#filters-premium-only) can be linked with +a single GitLab group. When the link is created, an access level/role is +specified (Guest, Reporter, Developer, Maintainer, or Owner). + +### Filters **(PREMIUM ONLY)** + +In GitLab Premium, you can add an LDAP user filter for group synchronization. +Filters allow for complex logic without creating a special LDAP group. + +To sync GitLab group membership based on an LDAP filter: + +1. Open the **LDAP Synchronization** page for the GitLab group. +1. Select **LDAP user filter** as the **Sync method**. +1. Enter an LDAP user filter in the **LDAP user filter** field. + +The filter must comply with the +syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). ## Administrator sync @@ -190,10 +190,13 @@ group, as opposed to the full DN. ## Global group memberships lock "Lock memberships to LDAP synchronization" setting allows instance administrators -to lock down user abilities to invite new members to a group. When enabled following happens: +to lock down user abilities to invite new members to a group. + +When enabled, the following applies: -1. Only administrator can manage memberships of any group including access levels. -2. Users are not allowed to share project with other groups or invite members to a project created in a group. +- Only administrator can manage memberships of any group including access levels. +- Users are not allowed to share project with other groups or invite members to + a project created in a group. ## Adjusting LDAP user sync schedule @@ -333,10 +336,18 @@ administrative duties. ### Supported LDAP group types/attributes -GitLab supports LDAP groups that use member attributes `member`, `submember`, -`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at -least, LDAP groups with object class `groupOfNames`, `posixGroup`, and -`groupOfUniqueName`. Other object classes should work fine as long as members +GitLab supports LDAP groups that use member attributes: + +- `member` +- `submember` +- `uniquemember` +- `memberof` +- `memberuid`. + +This means group sync supports, at least, LDAP groups with object class: +`groupOfNames`, `posixGroup`, and `groupOfUniqueName`. + +Other object classes should work fine as long as members are defined as one of the mentioned attributes. This also means GitLab supports Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. Other LDAP servers should work, too. @@ -344,11 +355,12 @@ Other LDAP servers should work, too. Active Directory also supports nested groups. Group sync will recursively resolve membership if `active_directory: true` is set in the configuration file. -> **Note:** Nested group membership will only be resolved if the nested group - also falls within the configured `group_base`. For example, if GitLab sees a - nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but - the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` - will be ignored. +NOTE: **Note:** +Nested group membership will only be resolved if the nested group +also falls within the configured `group_base`. For example, if GitLab sees a +nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but +the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` +will be ignored. ### Queries @@ -403,7 +415,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server [^1]: In Active Directory, a user is marked as disabled/blocked if the user account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) - has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/ + has bit 2 set. See <https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/> for more information. ### User DN has changed @@ -423,10 +435,10 @@ things to check to debug the situation. - Ensure LDAP configuration has a `group_base` specified. This configuration is required for group sync to work properly. - Ensure the correct LDAP group link is added to the GitLab group. Check group - links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**. -- Check that the user has an LDAP identity + links by visiting the GitLab group, then **Settings dropdown > LDAP groups**. +- Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Navigate to **Admin area -> Users**. + 1. Navigate to **Admin area > Users**. 1. Search for the user 1. Open the user, by clicking on their name. Do not click 'Edit'. 1. Navigate to the **Identities** tab. There should be an LDAP identity with @@ -437,7 +449,7 @@ Often, the best way to learn more about why group sync is behaving a certain way is to enable debug logging. There is verbose output that details every step of the sync. -1. Start a Rails console +1. Start a Rails console: ```bash # For Omnibus installations @@ -446,6 +458,7 @@ step of the sync. # For installations from source sudo -u git -H bundle exec rails console production ``` + 1. Set the log level to debug (only for this session): ```ruby @@ -540,8 +553,9 @@ and more DNs may be added, or existing entries modified, based on additional LDAP group lookups. The very last occurrence of this entry should indicate exactly which users GitLab believes should be added to the group. -> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' - and 50 is 'Owner' +NOTE: **Note:** +10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' +and 50 is 'Owner'. ```bash Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index be05a4d63a7..186bf4c4825 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -7,28 +7,44 @@ type: reference # LDAP GitLab integrates with LDAP to support user authentication. -This integration works with most LDAP-compliant directory -servers, including Microsoft Active Directory, Apple Open Directory, Open LDAP, -and 389 Server. GitLab Enterprise Editions include enhanced integration, + +This integration works with most LDAP-compliant directory servers, including: + +- Microsoft Active Directory +- Apple Open Directory +- Open LDAP +- 389 Server. + +GitLab Enterprise Editions (EE) include enhanced integration, including group membership syncing as well as multiple LDAP servers support. -## GitLab EE +For more details about EE-specific LDAP features, see the +[LDAP Enterprise Edition documentation](ldap-ee.md). -The information on this page is relevant for both GitLab CE and EE. For more -details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](ldap-ee.md). **(STARTER ONLY)** +NOTE: **Note:** +The information on this page is relevant for both GitLab CE and EE. + +## Overview + +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which is a standard +application protocol for accessing and maintaining distributed directory +information services over an Internet Protocol (IP) network. ## Security -GitLab assumes that LDAP users are not able to change their LDAP 'mail', 'email' -or 'userPrincipalName' attribute. An LDAP user who is allowed to change their -email on the LDAP server can potentially -[take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) -on your GitLab server. +GitLab assumes that LDAP users: + +- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attribute. + An LDAP user who is allowed to change their email on the LDAP server can potentially + [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) + on your GitLab server. +- Have unique email addresses, otherwise it is possible for LDAP users with the same + email address to share the same GitLab account. We recommend against using LDAP integration if your LDAP users are -allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on -the LDAP server. +allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on +the LDAP server or share email addresses. ### User deletion @@ -64,9 +80,12 @@ NOTE: **Note**: In GitLab Enterprise Edition Starter, you can configure multiple LDAP servers to connect to one GitLab server. -For a complete guide on configuring LDAP with GitLab Community Edition, please check -the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). -For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** +For a complete guide on configuring LDAP with: + +- GitLab Community Edition, see + [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). +- Enterprise Editions, see + [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -384,7 +403,7 @@ production: Tip: If you want to limit access to the nested members of an Active Directory group, you can use the following syntax: -``` +```text (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` @@ -402,13 +421,13 @@ The `user_filter` DN can contain special characters. For example: - A comma: - ``` + ```text OU=GitLab, Inc,DC=gitlab,DC=com ``` - Open and close brackets: - ``` + ```text OU=Gitlab (Inc),DC=gitlab,DC=com ``` @@ -417,13 +436,13 @@ The `user_filter` DN can contain special characters. For example: - Escape commas with `\2C`. For example: - ``` + ```text OU=GitLab\2C Inc,DC=gitlab,DC=com ``` - Escape open and close brackets with `\28` and `\29`, respectively. For example: - ``` + ```text OU=Gitlab \28Inc\29,DC=gitlab,DC=com ``` @@ -507,11 +526,11 @@ timeout), the login is rejected and a message will be logged to ### Debug LDAP user filter with ldapsearch -This example uses ldapsearch and assumes you are using ActiveDirectory. The +This example uses `ldapsearch` and assumes you are using ActiveDirectory. The following query returns the login names of the users that will be allowed to log in to GitLab if you configure your own user_filter. -``` +```sh ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName ``` diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 78d040cda99..3445f916ef4 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -74,43 +74,38 @@ The OpenID Connect will provide you with a client details and secret for you to } ``` - > **Note:** - > - > - For more information on each configuration option refer to - the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) and - the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). + NOTE: **Note:** + For more information on each configuration option refer to the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) + and the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). 1. For the configuration above, change the values for the provider to match your OpenID Connect client setup. Use the following as a guide: - - `<your_oidc_label>` is the label that will be displayed on the login page. - - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. - If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. - - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. - - `client_auth_method` (optional) specifies the method used for authenticating the client with the OpenID Connect provider. - - Supported values are: - - `basic` - HTTP Basic Authentication - - `jwt_bearer` - JWT based authentication (private key and client secret signing) - - `mtls` - Mutual TLS or X.509 certificate validation - - Any other value will POST the client id and secret in the request body - - If not specified, defaults to `basic`. - - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. - If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. - - `client_options` are the OpenID Connect client-specific options. Specifically: - - - `identifier` is the client identifier as configured in the OpenID Connect service provider. - - `secret` is the client secret as configured in the OpenID Connect service provider. - - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. - - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. - - The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: - - - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. - - `token_endpoint` is the URL to the endpoint that provides Access Token. - - `userinfo_endpoint` is the URL to the endpoint that provides the user information. - - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. - -1. Save the configuration file. -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) - for the changes to take effect if you installed GitLab via Omnibus or from source respectively. + - `<your_oidc_label>` is the label that will be displayed on the login page. + - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. + If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. + - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. + - `client_auth_method` (optional) specifies the method used for authenticating the client with the OpenID Connect provider. + - Supported values are: + - `basic` - HTTP Basic Authentication + - `jwt_bearer` - JWT based authentication (private key and client secret signing) + - `mtls` - Mutual TLS or X.509 certificate validation + - Any other value will POST the client id and secret in the request body + - If not specified, defaults to `basic`. + - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. + If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. + - `client_options` are the OpenID Connect client-specific options. Specifically: + - `identifier` is the client identifier as configured in the OpenID Connect service provider. + - `secret` is the client secret as configured in the OpenID Connect service provider. + - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. + - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. + - The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: + - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. + - `token_endpoint` is the URL to the endpoint that provides Access Token. + - `userinfo_endpoint` is the URL to the endpoint that provides the user information. + - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. + +1. Save the configuration file. +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page, there should now be an OpenID Connect icon below the regular sign in form. Click the icon to begin the authentication process. The OpenID Connect provider will ask the user to diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index 3e3054af509..d43b3718bf9 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -27,8 +27,6 @@ but not recommended and out of the scope of this document. Read the [insecure Registry documentation][docker-insecure] if you want to implement this. ---- - **Installations from source** If you have installed GitLab from source: @@ -116,8 +114,6 @@ GitLab from source respectively. Be careful to choose a port different than the one that Registry listens to (`5000` by default), otherwise you will run into conflicts. ---- - **Omnibus GitLab installations** 1. Your `/etc/gitlab/gitlab.rb` should contain the Registry URL as well as the @@ -141,7 +137,14 @@ otherwise you will run into conflicts. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- +1. Validate using: + + ```sh + openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:443 > cacert.pem + ``` + +NOTE: **Note:** +If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. **Installations from source** @@ -158,8 +161,6 @@ otherwise you will run into conflicts. 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). ---- - Users should now be able to login to the Container Registry with their GitLab credentials using: @@ -171,14 +172,16 @@ docker login gitlab.example.com:4567 If the Registry is configured to use its own domain, you will need a TLS certificate for that specific domain (e.g., `registry.example.com`) or maybe -a wildcard certificate if hosted under a subdomain of your existing GitLab +a wildcard certificate if hosted under a subdomain of your existing GitLab domain (e.g., `registry.gitlab.example.com`). +NOTE: **Note:** +As well as manually generated SSL certificates (explained here), certificates automatically +generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html#host-services). + Let's assume that you want the container Registry to be accessible at `https://registry.gitlab.example.com`. ---- - **Omnibus GitLab installations** 1. Place your TLS certificate and key in @@ -210,8 +213,6 @@ look like: > registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/certificate.key" > ``` ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and @@ -226,8 +227,6 @@ look like: 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). ---- - Users should now be able to login to the Container Registry using their GitLab credentials: @@ -252,8 +251,6 @@ Registry application itself. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and @@ -272,8 +269,6 @@ If the Container Registry is enabled, then it will be available on all new projects. To disable this function and let the owners of a project to enable the Container Registry by themselves, follow the steps below. ---- - **Omnibus GitLab installations** 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -284,8 +279,6 @@ the Container Registry by themselves, follow the steps below. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `default_projects_features` @@ -321,8 +314,6 @@ This path is accessible to: > **Warning** You should confirm that all GitLab, Registry and web server users have access to this directory. ---- - **Omnibus GitLab installations** The default location where images are stored in Omnibus, is @@ -336,8 +327,6 @@ The default location where images are stored in Omnibus, is 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** The default location where images are stored in source installations, is @@ -446,8 +435,6 @@ In the examples below we set the Registry's port to `5001`. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open the configuration file of your Registry server and edit the @@ -565,8 +552,6 @@ To configure a notification endpoint in Omnibus: 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** Configuring the notification endpoint is done in your registry config YML file created @@ -640,8 +625,6 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **For installations from source** 1. Edit `config/gitlab.yml`: @@ -673,8 +656,6 @@ You can add a configuration option for backwards compatibility. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **For installations from source** 1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: @@ -701,6 +682,39 @@ To get around this, you can [change the group path](../user/group/index.md#chang branch name. Another option is to create a [push rule](../push_rules/push_rules.html) to prevent this at the instance level. +### Image push errors + +When getting errors or "retrying" loops in an attempt to push an image but `docker login` works fine, +there is likely an issue with the headers forwarded to the registry by NGINX. The default recommended +NGINX configurations should handle this, but it might occur in custom setups where the SSL is +offloaded to a third party reverse proxy. + +This problem was discussed in a [docker project issue][docker-image-push-issue] and a simple solution +would be to enable relative urls in the registry. + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['env'] = { + "REGISTRY_HTTP_RELATIVEURLS" => true + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**For installations from source** + +1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: + + ```yaml + http: + relativeurls: true + ``` + +1. Restart the registry for the changes to take affect. + [ce-18239]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18239 [docker-insecure-self-signed]: https://docs.docker.com/registry/insecure/#use-self-signed-certificates [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure @@ -719,3 +733,4 @@ this at the instance level. [new-domain]: #configure-container-registry-under-its-own-domain [notifications-config]: https://docs.docker.com/registry/notifications/ [registry-notifications-config]: https://docs.docker.com/registry/configuration/#notifications +[docker-image-push-issue]: https://github.com/docker/distribution/issues/970 diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 113514e1ee8..32462a95a1a 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -21,7 +21,7 @@ administrators can add custom git hooks to any GitLab project. ## Create a custom Git hook for a repository Server-side Git hooks are typically placed in the repository's `hooks` -subdirectory. In GitLab, hook directories are are symlinked to the gitlab-shell +subdirectory. In GitLab, hook directories are symlinked to the gitlab-shell `hooks` directory for ease of maintenance between gitlab-shell upgrades. Custom hooks are implemented differently, but the behavior is exactly the same once the hook is created. Follow the steps below to set up a custom hook for a diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 874b1f3c80d..37d7194af53 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -13,6 +13,7 @@ override certain values. Variable | Type | Description -------- | ---- | ----------- +`ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable) `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (e.g. `//mycdnsubdomain.fictional-cdn.com`) `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png Binary files differindex fd51523104b..0a7b841897b 100644 --- a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png Binary files differindex b2a6da69d3d..85759d903a4 100644 --- a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index d44e141b66b..ba95843b0b0 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,4 +1,4 @@ -# Disaster Recovery **(PREMIUM ONLY)** +# Disaster Recovery (Geo) **(PREMIUM ONLY)** Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 0e11dffa0d6..fd076bb79d8 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -17,7 +17,7 @@ You are encouraged to first read through all the steps before executing them in your testing/production environment. NOTE: **Note:** -**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. +**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. Any change that requires access to the **Admin Area** needs to be done in the **primary** node because the **secondary** node is a read-only replica. @@ -242,7 +242,7 @@ node's Geo Nodes dashboard in your browser.  If your installation isn't working properly, check the -[troubleshooting document]. +[troubleshooting document](troubleshooting.md). The two most obvious issues that can become apparent in the dashboard are: diff --git a/doc/administration/geo/replication/img/geo_architecture.png b/doc/administration/geo/replication/img/geo_architecture.png Binary files differindex d318cd5d0f4..aac63be41ff 100644 --- a/doc/administration/geo/replication/img/geo_architecture.png +++ b/doc/administration/geo/replication/img/geo_architecture.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index f0d329d5296..dbd466b906d 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -1,6 +1,12 @@ -# Geo Replication **(PREMIUM ONLY)** +# Replication (Geo) **(PREMIUM ONLY)** -Geo is the solution for widely distributed development teams. +> - Introduced in GitLab Enterprise Edition 8.9. +> - Using Geo in combination with +> [High Availability](../../high_availability/README.md) +> is considered **Generally Available** (GA) in +> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + +Replication with Geo is the solution for widely distributed development teams. ## Overview @@ -8,14 +14,8 @@ Fetching large repositories can take a long time for teams located far from a si Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development. -> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed). -> - Introduced in GitLab Enterprise Edition 8.9. -> - We recommend you use: -> - At least GitLab Enterprise Edition 10.0 for basic Geo features. -> - The latest version for a better experience. -> - Make sure that all nodes run the same GitLab version. -> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md). -> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. +NOTE: **Note:** +Check the [requirements](#requirements-for-running-geo) carefully before setting up Geo. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). @@ -111,6 +111,13 @@ The following are required to run Geo: - [Ubuntu](https://www.ubuntu.com) 16.04+ - PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ +- All nodes must run the same GitLab version. + +Additionally, check GitLab's [minimum requirements](../../../install/requirements.md), +and we recommend you use: + +- At least GitLab Enterprise Edition 10.0 for basic Geo features. +- The latest version for a better experience. ### Firewall rules @@ -239,35 +246,48 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. -### Limitations on replication - -Only the following items are replicated to the **secondary** node: - -- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata. -- Project repositories. -- Project wiki repositories. -- User uploads. For example, attachments to issues, merge requests, epics, and avatars. -- CI job artifacts and traces. +### Limitations on replication/verification + +The following table lists the GitLab features along with their replication +and verification status on a **secondary** node. + +You can keep track of the progress to include the missing items in: + +- [ee-893](https://gitlab.com/groups/gitlab-org/-/epics/893). +- [ee-1430](https://gitlab.com/groups/gitlab-org/-/epics/1430). + +| Feature | Replicated | Verified | +|-----------|------------|----------| +| All database content (e.g. snippets, epics, issues, merge requests, groups, and project metadata) | Yes | Yes | +| Project repository | Yes | Yes | +| Project wiki repository | Yes | Yes | +| Project designs repository | No | No | +| Uploads (e.g. attachments to issues, merge requests, epics, and avatars) | Yes | Yes, only on transfer, or manually (1) | +| LFS Objects | Yes | Yes, only on transfer, or manually (1) | +| CI job artifacts (other than traces) | Yes | No, only manually (1) | +| Archived traces | Yes | Yes, only on transfer, or manually (1) | +| Personal snippets | Yes | Yes | +| Version-controlled personal snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Project snippets | Yes | Yes | +| Version-controlled project snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Object pools for forked project deduplication | No | No | +| [Server-side Git Hooks](../../custom_hooks.md) | No | No | +| [Elasticsearch integration](../../../integration/elasticsearch.md) | No | No | +| [GitLab Pages](../../pages/index.md) | No | No | +| [Container Registry](../../container_registry.md) ([track progress](https://gitlab.com/gitlab-org/gitlab-ee/issues/2870)) | No | No | +| [NPM Registry](../../npm_registry.md) | No | No | +| [Maven Packages](../../maven_packages.md) | No | No | +| [External merge request diffs](../../merge_request_diffs.md) | No, if they are on-disk | No | +| Content in object storage ([track progress](https://gitlab.com/groups/gitlab-org/-/epics/1526)) | No | No | + +1. The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. DANGER: **DANGER** -Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**. - -### Examples of data not replicated - -Take special note that these examples of GitLab features are both: - -- Commonly used. -- **Not** replicated by Geo at present. - -Examples include: - -- [Elasticsearch integration](../../../integration/elasticsearch.md). -- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this. -- [GitLab Pages](../../pages/index.md). -- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/). - -CAUTION: **Caution:** -If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means. +Features not on this list, or with **No** in the **Replicated** column, +are not replicated on the **secondary** node. Failing over without manually +replicating data from those features will cause the data to be **lost**. +If you wish to use those features on a **secondary** node, or to execute a failover +successfully, you must replicate their data using some other means. ## Frequently Asked Questions diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 28abfff973d..fe1557fd8b5 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -230,7 +230,7 @@ sudo gitlab-ctl reconfigure This will increase the timeout to three hours (10800 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. -### Reseting Geo **secondary** node replication +### Resetting Geo **secondary** node replication If you get a **secondary** node in a broken state and want to reset the replication state, to start again from scratch, there are a few steps that can help you: @@ -524,6 +524,20 @@ If it doesn't exist or inadvertent changes have been made to it, run `sudo gitla If this path is mounted on a remote volume, please check your volume configuration and that it has correct permissions. +### An existing tracking database cannot be reused + +Geo cannot reuse an existing tracking database. + +It is safest to use a fresh secondary, or reset the whole secondary by following +[Resetting Geo secondary node replication](#resetting-geo-secondary-node-replication). + +If you are not concerned about possible orphaned directories and files, then you +can simply reset the existing tracking database with: + +```sh +sudo gitlab-rake geo:db:reset +``` + ### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node. This error refers to a problem with the database replica on a **secondary** node, diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 166ee94eca4..39174780e24 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -10,10 +10,23 @@ all you need to do is update GitLab itself: 1. Log into each node (**primary** and **secondary** nodes). 1. [Update GitLab][update]. -1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when - the tracking database is enabled. 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. +### Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + ## Upgrading to GitLab 12.1 By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see [the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure. @@ -251,7 +264,7 @@ Omnibus is the following: 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. 1. Install GitLab on the **secondary** server. -1. Re-run the [database replication process][database-replication]. +1. Re-run the [database replication process](database.md#step-3-initiate-the-replication-process). ## Special update notes for 9.0.x @@ -419,22 +432,7 @@ is prepended with the relevant node for better clarity: sudo gitlab-ctl start ``` -## Check status after updating - -Now that the update process is complete, you may want to check whether -everything is working correctly: - -1. Run the Geo raketask on all nodes, everything should be green: - - ```sh - sudo gitlab-rake gitlab:geo:check - ``` - -1. Check the **primary** node's Geo dashboard for any errors. -1. Test the data replication by pushing code to the **primary** node and see if it - is received by **secondary** nodes. - -## Update tracking database on **secondary** node +### Update tracking database on **secondary** node After updating a **secondary** node, you might need to run migrations on the tracking database. The tracking database was added in GitLab 9.1, diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 4407facfca9..878f0ef842d 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -2,127 +2,127 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that provides high-level RPC access to Git repositories. Without it, no other -components can read or write Git data. +components can read or write Git data. GitLab components that access Git +repositories (gitlab-rails, gitlab-shell, gitlab-workhorse, etc.) act as clients +to Gitaly. End users do not have direct access to Gitaly. -GitLab components that access Git repositories (gitlab-rails, -gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do -not have direct access to Gitaly. +In the rest of this page, Gitaly server is referred to the standalone node that +only runs Gitaly, and Gitaly client to the GitLab Rails node that runs all other +processes except Gitaly. ## Configuring Gitaly The Gitaly service itself is configured via a TOML configuration file. -This file is documented [in the gitaly +This file is documented [in the Gitaly repository](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md). -To change a Gitaly setting in Omnibus you can use -`gitaly['my_setting']` in `/etc/gitlab/gitlab.rb`. Changes will be applied -when you run `gitlab-ctl reconfigure`. +In case you want to change some of its settings: -```ruby -gitaly['prometheus_listen_addr'] = 'localhost:9236' -``` - -To change a Gitaly setting in installations from source you can edit -`/home/git/gitaly/config.toml`. Changes will be applied when you run -`service gitlab restart`. +**For Omnibus GitLab** -```toml -prometheus_listen_addr = "localhost:9236" -``` +1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -## Client-side GRPC logs +**For installations from source** -Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain useful information when -you are seeing Gitaly errors. You can control the log level of the -gRPC client with the `GRPC_LOG_LEVEL` environment variable. The -default level is `WARN`. +1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ## Running Gitaly on its own server -> This is an optional way to deploy Gitaly which can benefit GitLab +This is an optional way to deploy Gitaly which can benefit GitLab installations that are larger than a single machine. Most installations will be better served with the default configuration used by Omnibus and the GitLab source installation guide. Starting with GitLab 11.4, Gitaly is able to serve all Git requests without -needed a shared NFS mount for Git repository data. +requiring a shared NFS mount for Git repository data. Between 11.4 and 11.8 the exception was the [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). But since 11.8 the indexer uses Gitaly for data access as well. NFS can still be leveraged for redudancy on block level of the Git data. But only has to be mounted on the Gitaly server. -NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend -using EFS as it may impact GitLab's performance. Please review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) +Starting with GitLab 11.8, it is possible to use ElasticSearch in conjunction with +a Gitaly setup that isn't utilising NFS. In order to use ElasticSearch in this +scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) +needs to be enabled in your GitLab configuration. + +NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended +to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. ### Network architecture -- gitlab-rails shards repositories into "repository storages" -- `gitlab-rails/config/gitlab.yml` contains a map from storage names to - (Gitaly address, Gitaly token) pairs -- the `storage name` -\> `(Gitaly address, Gitaly token)` map in - `gitlab.yml` is the single source of truth for the Gitaly network - topology -- a (Gitaly address, Gitaly token) corresponds to a Gitaly server -- a Gitaly server hosts one or more storages -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients -- Gitaly clients are: unicorn, sidekiq, gitlab-workhorse, - gitlab-shell, Elasticsearch Indexer, and Gitaly itself -- special case: a Gitaly server must be able to make RPC calls **to - itself** via its own (Gitaly address, Gitaly token) pair as - specified in `gitlab-rails/config/gitlab.yml` -- Gitaly servers must not be exposed to the public internet +The following list depicts what the network architecture of Gitaly is: -Gitaly network traffic is unencrypted by default, but supports -[TLS](#tls-support). Authentication is done through a static token. +- GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md). +- `/config/gitlab.yml` contains a map from storage names to + `(Gitaly address, Gitaly token)` pairs. +- the `storage name` -\> `(Gitaly address, Gitaly token)` map in + `/config/gitlab.yml` is the single source of truth for the Gitaly network + topology. +- A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. +- A Gitaly server hosts one or more storages. +- A GitLab server can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly clients are: Unicorn, Sidekiq, gitlab-workhorse, + gitlab-shell, Elasticsearch Indexer, and Gitaly itself. +- A Gitaly server must be able to make RPC calls **to itself** via its own + `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. +- Gitaly servers must not be exposed to the public internet as Gitaly's network + traffic is unencrypted by default. The use of firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#tls-support). +- Authentication is done through a static token which is shared among the Gitaly + and GitLab Rails nodes. -NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to -restrict access to your Gitaly server. +Below we describe how to configure two Gitaly servers one at +`gitaly1.internal` and the other at `gitaly2.internal` +with secret token `abc123secret`. We assume +your GitLab installation has three repository storages: `default`, +`storage1` and `storage2`. -Below we describe how to configure a Gitaly server at address -`gitaly.internal:8075` with secret token `abc123secret`. We assume -your GitLab installation has two repository storages, `default` and -`storage1`. +### 1. Installation -### Installation +First install Gitaly on each Gitaly server using either +Omnibus GitLab or install it from source: -First install Gitaly using either Omnibus or from source. +- For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **_do not_** provide the `EXTERNAL_URL=` value. +- From source: [Install Gitaly](../../install/installation.md#install-gitaly). -Omnibus: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab -package you want using **steps 1 and 2** from the GitLab downloads page but -**_do not_** provide the `EXTERNAL_URL=` value. +### 2. Client side token configuration -Source: [Install Gitaly](../../install/installation.md#install-gitaly) +Configure a token on the instance that runs the GitLab Rails application. -### Client side token configuration +**For Omnibus GitLab** -Configure a token on the client side. +1. On the client node(s), edit `/etc/gitlab/gitlab.rb`: -Omnibus installations: + ```ruby + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -```ruby -# /etc/gitlab/gitlab.rb -gitlab_rails['gitaly_token'] = 'abc123secret' -``` +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -Source installations: +**For installations from source** -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - gitaly: - token: 'abc123secret' -``` +1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`: -You need to reconfigure (Omnibus) or restart (source) for these -changes to be picked up. + ```yaml + gitlab: + gitaly: + token: 'abc123secret' + ``` -### Gitaly server configuration +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -Next, on the Gitaly server, we need to configure storage paths, enable +### 3. Gitaly server configuration + +Next, on the Gitaly servers, you need to configure storage paths, enable the network listener and configure the token. NOTE: **Note:** if you want to reduce the risk of downtime when you enable @@ -137,131 +137,228 @@ the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gi from an existing GitLab server to the Gitaly server. Without this shared secret, Git operations in GitLab will result in an API error. -NOTE: **Note:** In most or all cases the storage paths below end in `/repositories` which is -different than `path` in `git_data_dirs` of Omnibus installations. Check the -directory layout on your Gitaly server to be sure. - -Omnibus installations: - -<!-- -updates to following example must also be made at -https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab ---> - -```ruby -# /etc/gitlab/gitlab.rb - -# Avoid running unnecessary services on the Gitaly server -postgresql['enable'] = false -redis['enable'] = false -nginx['enable'] = false -prometheus['enable'] = false -unicorn['enable'] = false -sidekiq['enable'] = false -gitlab_workhorse['enable'] = false - -# Prevent database connections during 'gitlab-ctl reconfigure' -gitlab_rails['rake_cache_clear'] = false -gitlab_rails['auto_migrate'] = false - -# Configure the gitlab-shell API callback URL. Without this, `git push` will -# fail. This can be your 'front door' GitLab URL or an internal load -# balancer. -# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. -gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - -# Make Gitaly accept connections on all network interfaces. You must use -# firewalls to restrict access to this address/port. -gitaly['listen_addr'] = "0.0.0.0:8075" -gitaly['auth_token'] = 'abc123secret' - -gitaly['storage'] = [ - { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' }, - { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' }, -] - -# To use TLS for Gitaly you need to add -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "path/to/cert.pem" -gitaly['key_path'] = "path/to/key.pem" -``` +NOTE: **Note:** +In most or all cases, the storage paths below end in `/repositories` which is +not that case with `path` in `git_data_dirs` of Omnibus GitLab installations. +Check the directory layout on your Gitaly server to be sure. + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + <!-- + updates to following example must also be made at + https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + # /etc/gitlab/gitlab.rb + + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + gitaly['listen_addr'] = "0.0.0.0:8075" + gitaly['auth_token'] = 'abc123secret' + + # To use TLS for Gitaly you need to add + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: + + For `gitaly1.internal`: + + ``` + gitaly['storage'] = [ + { 'name' => 'default' }, + { 'name' => 'storage1' }, + ] + ``` + + For `gitaly2.internal`: + + ``` + gitaly['storage'] = [ + { 'name' => 'storage2' }, + ] + ``` + + NOTE: **Note:** + In some cases, you'll have to set `path` for `gitaly['storage']` in the + format `'path' => '/mnt/gitlab/<storage name>/repositories'`. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. On the client node(s), edit `/home/git/gitaly/config.toml`: + + ```toml + listen_addr = '0.0.0.0:8075' + tls_listen_addr = '0.0.0.0:9999' + + [tls] + certificate_path = /path/to/cert.pem + key_path = /path/to/key.pem + + [auth] + token = 'abc123secret' + ``` + +1. Append the following to `/home/git/gitaly/config.toml` for each respective server: + + For `gitaly1.internal`: + + ```toml + [[storage]] + name = 'default' + + [[storage]] + name = 'storage1' + ``` + + For `gitaly2.internal`: + + ```toml + [[storage]] + name = 'storage2' + ``` + + NOTE: **Note:** + In some cases, you'll have to set `path` for each `[[storage]]` in the + format `path = '/mnt/gitlab/<storage name>/repositories'`. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +### 4. Converting clients to use the Gitaly server + +As the final step, you need to update the client machines to switch from using +their local Gitaly service to the new Gitaly server you just configured. This +is a risky step because if there is any sort of network, firewall, or name +resolution problem preventing your GitLab server from reaching the Gitaly server, +then all Gitaly requests will fail. -Source installations: +Additionally, you need to +[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). -```toml -# /home/git/gitaly/config.toml -listen_addr = '0.0.0.0:8075' -tls_listen_addr = '0.0.0.0:9999' +We assume that your `gitaly1.internal` Gitaly server can be reached at +`gitaly1.internal:8075` from your GitLab server, and that Gitaly server +can read and write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1`. -[tls] -certificate_path = /path/to/cert.pem -key_path = /path/to/key.pem +We assume also that your `gitaly2.internal` Gitaly server can be reached at +`gitaly2.internal:8075` from your GitLab server, and that Gitaly server +can read and write to `/mnt/gitlab/storage2`. -[auth] -token = 'abc123secret' +**For Omnibus GitLab** -[[storage]] -name = 'default' -path = '/mnt/gitlab/default/repositories' +1. Edit `/etc/gitlab/gitlab.rb`: -[[storage]] -name = 'storage1' -path = '/mnt/gitlab/storage1/repositories' -``` + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) -Again, reconfigure (Omnibus) or restart (source). + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -### Converting clients to use the Gitaly server + NOTE: **Note:** + In some cases, you'll have to set `path` for each `git_data_dirs` in the + format `'path' => '/mnt/gitlab/<storage name>'`. -Now as the final step update the client machines to switch from using -their local Gitaly service to the new Gitaly server you just -configured. This is a risky step because if there is any sort of -network, firewall, or name resolution problem preventing your GitLab -server from reaching the Gitaly server then all Gitaly requests will -fail. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Tail the logs to see the requests: -Additionally, you need to -[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). + ```sh + sudo gitlab-ctl tail gitaly + ``` -We assume that your Gitaly server can be reached at -`gitaly.internal:8075` from your GitLab server, and that Gitaly can read and -write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively. +**For installations from source** -Omnibus installations: +1. Edit `/home/git/gitlab/config/gitlab.yml`: -```ruby -# /etc/gitlab/gitlab.rb -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, -}) + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tcp://gitaly1.internal:8075 + storage1: + gitaly_address: tcp://gitaly1.internal:8075 + storage2: + gitaly_address: tcp://gitaly2.internal:8075 -gitlab_rails['gitaly_token'] = 'abc123secret' -``` + gitaly: + token: 'abc123secret' + ``` -Source installations: - -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - repositories: - storages: - default: - path: /mnt/gitlab/default/repositories - gitaly_address: tcp://gitaly.internal:8075 - storage1: - path: /mnt/gitlab/storage1/repositories - gitaly_address: tcp://gitaly.internal:8075 - - gitaly: - token: 'abc123secret' -``` + NOTE: **Note:** + In some cases, you'll have to set `path` for each of the `storages` in the + format `path: /mnt/gitlab/<storage name>/repositories`. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Tail the logs to see the requests: -Now reconfigure (Omnibus) or restart (source). When you tail the -Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or -`tail -f /home/git/gitlab/log/gitaly.log`) you should see requests -coming in. One sure way to trigger a Gitaly request is to clone a -repository from your GitLab server over HTTP. + ```sh + tail -f /home/git/gitlab/log/gitaly.log + ``` + +When you tail the Gitaly logs on your Gitaly server you should see requests +coming in. One sure way to trigger a Gitaly request is to clone a repository +from your GitLab server over HTTP. + +### Disabling the Gitaly service in a cluster environment + +If you are running Gitaly [as a remote +service](#running-gitaly-on-its-own-server) you may want to disable +the local Gitaly service that runs on your GitLab server by default. +Disabling Gitaly only makes sense when you run GitLab in a custom +cluster configuration, where different services run on different +machines. Disabling Gitaly on all machines in the cluster is not a +valid configuration. + +To disable Gitaly on a client node: + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['enable'] = false + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. Edit `/etc/default/gitlab`: + + ```shell + gitaly_enabled=false + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ## TLS support @@ -271,168 +368,131 @@ Gitaly supports TLS encryption. To be able to communicate with a Gitaly instance that listens for secure connections you will need to use `tls://` url scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. -The admin needs to bring their own certificate as we do not provide that automatically. -The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +You will need to bring your own certificates as this isn't provided automatically. +The certificate to be used needs to be installed on all Gitaly nodes and on all +client nodes that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -Note that it is possible to configure Gitaly servers with both an +NOTE: **Note:** +It is possible to configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if necessary. -To observe what type of connections are actually being used in a -production environment you can use the following Prometheus query: - -``` -sum(rate(gitaly_connections_total[5m])) by (type) -``` +To configure Gitaly with TLS: -### Example TLS configuration +**For Omnibus GitLab** -### Omnibus installations: +1. On the client nodes, edit `/etc/gitlab/gitlab.rb`: -#### On client nodes: + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) -```ruby -# /etc/gitlab/gitlab.rb -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, -}) + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -gitlab_rails['gitaly_token'] = 'abc123secret' -``` +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly server nodes, edit `/etc/gitlab/gitlab.rb`: -#### On Gitaly server nodes: + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` -```ruby -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "path/to/cert.pem" -gitaly['key_path'] = "path/to/key.pem" -``` +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Source installations: +**For installations from source** -#### On client nodes: +1. On the client nodes, edit `/home/git/gitlab/config/gitlab.yml`: -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - repositories: - storages: - default: - path: /mnt/gitlab/default/repositories - gitaly_address: tls://gitaly.internal:9999 - storage1: - path: /mnt/gitlab/storage1/repositories - gitaly_address: tls://gitaly.internal:9999 + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tls://gitaly1.internal:9999 + storage1: + gitaly_address: tls://gitaly1.internal:9999 + storage2: + gitaly_address: tls://gitaly2.internal:9999 - gitaly: - token: 'abc123secret' -``` - -#### On Gitaly server nodes: - -```toml -# /home/git/gitaly/config.toml -tls_listen_addr = '0.0.0.0:9999' + gitaly: + token: 'abc123secret' + ``` -[tls] -certificate_path = '/path/to/cert.pem' -key_path = '/path/to/key.pem' -``` - -## Gitaly-ruby + NOTE: **Note:** + In some cases, you'll have to set `path` for each of the `storages` in the + format `path: /mnt/gitlab/<storage name>/repositories`. -Gitaly was developed to replace Ruby application code in gitlab-ce/ee. -In order to save time and/or avoid the risk of rewriting existing -application logic, in some cases we chose to copy some application code -from gitlab-ce into Gitaly almost as-is. To be able to run that code, we -made gitaly-ruby, which is a sidecar process for the main Gitaly Go -process. Some examples of things that are implemented in gitaly-ruby are -RPC's that deal with wiki's, and RPC's that create commits on behalf of -a user, such as merge commits. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly server nodes, edit `/home/git/gitaly/config.toml`: -### Number of gitaly-ruby workers + ```toml + tls_listen_addr = '0.0.0.0:9999' -Gitaly-ruby has much less capacity than Gitaly itself. If your Gitaly -server has to handle a lot of request, the default setting of having -just 1 active gitaly-ruby sidecar might not be enough. If you see -ResourceExhausted errors from Gitaly it's very likely that you have not -enough gitaly-ruby capacity. + [tls] + certificate_path = '/path/to/cert.pem' + key_path = '/path/to/key.pem' + ``` -You can increase the number of gitaly-ruby processes on your Gitaly -server with the following settings. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -Omnibus: +To observe what type of connections are actually being used in a +production environment you can use the following Prometheus query: -```ruby -# /etc/gitlab/gitlab.rb -# Default is 2 workers. The minimum is 2; 1 worker is always reserved as -# a passive stand-by. -gitaly['ruby_num_workers'] = 4 ``` - -Source: - -```toml -# /home/git/gitaly/config.toml -[gitaly-ruby] -num_workers = 4 +sum(rate(gitaly_connections_total[5m])) by (type) ``` -### Observing gitaly-ruby traffic +## `gitaly-ruby` -Gitaly-ruby is a somewhat hidden, internal implementation detail of -Gitaly. There is not that much visibility into what goes on inside -gitaly-ruby processes. +Gitaly was developed to replace the Ruby application code in GitLab. +In order to save time and/or avoid the risk of rewriting existing +application logic, in some cases we chose to copy some application code +from GitLab into Gitaly almost as-is. To be able to run that code, +`gitaly-ruby` was created, which is a "sidecar" process for the main Gitaly Go +process. Some examples of things that are implemented in `gitaly-ruby` are +RPCs that deal with wikis, and RPCs that create commits on behalf of +a user, such as merge commits. -If you have Prometheus set up to scrape your Gitaly process, you can see -request rates and error codes for individual RPC's in gitaly-ruby by -querying `grpc_client_handled_total`. Strictly speaking this metric does -not differentiate between gitaly-ruby and other RPC's, but in practice -(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal -calls from the main Gitaly process to one of its gitaly-ruby sidecars. +### Number of `gitaly-ruby` workers -Assuming your `grpc_client_handled_total` counter only observes Gitaly, -the following query shows you RPC's are (most likely) internally -implemented as calls to gitaly-ruby. +`gitaly-ruby` has much less capacity than Gitaly itself. If your Gitaly +server has to handle a lot of requests, the default setting of having +just one active `gitaly-ruby` sidecar might not be enough. If you see +`ResourceExhausted` errors from Gitaly, it's very likely that you have not +enough `gitaly-ruby` capacity. -``` -sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 -``` +You can increase the number of `gitaly-ruby` processes on your Gitaly +server with the following settings. -## Disabling or enabling the Gitaly service in a cluster environment +**For Omnibus GitLab** -If you are running Gitaly [as a remote -service](#running-gitaly-on-its-own-server) you may want to disable -the local Gitaly service that runs on your GitLab server by default. +1. Edit `/etc/gitlab/gitlab.rb`: -> 'Disabling Gitaly' only makes sense when you run GitLab in a custom -cluster configuration, where different services run on different -machines. Disabling Gitaly on all machines in the cluster is not a -valid configuration. + ```ruby + # Default is 2 workers. The minimum is 2; 1 worker is always reserved as + # a passive stand-by. + gitaly['ruby_num_workers'] = 4 + ``` -If you are setting up a GitLab cluster where Gitaly does not need to -run on all machines, you can disable the Gitaly service in your -Omnibus installation, add the following line to `/etc/gitlab/gitlab.rb`: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -```ruby -gitaly['enable'] = false -``` - -When you run `gitlab-ctl reconfigure` the Gitaly service will be -disabled. +**For installations from source** -To disable the Gitaly service in a GitLab cluster where you installed -GitLab from source, add the following to `/etc/default/gitlab` on the -machine where you want to disable Gitaly. +1. Edit `/home/git/gitaly/config.toml`: -```shell -gitaly_enabled=false -``` + ```toml + [gitaly-ruby] + num_workers = 4 + ``` -When you run `service gitlab restart` Gitaly will be disabled on this -particular machine. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ## Eliminating NFS altogether @@ -440,19 +500,32 @@ If you are planning to use Gitaly without NFS for your storage needs and want to eliminate NFS from your environment altogether, there are a few things that you need to do: - 1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. - 1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md) - to eliminate the need for a shared authorized_keys file. - 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [live tracing](../job_traces.md#new-live-trace-architecture). - 1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). - 1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). - -NOTE: **Note:** One current feature of GitLab still requires a shared directory (NFS): [GitLab Pages](../../user/project/pages/index.md). +1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. +1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md) + to eliminate the need for a shared authorized_keys file. +1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [live tracing](../job_traces.md#new-live-trace-architecture). +1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). +1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). + +NOTE: **Note:** +One current feature of GitLab that still requires a shared directory (NFS) is +[GitLab Pages](../../user/project/pages/index.md). There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) to eliminate the need for NFS to support GitLab Pages. -## Troubleshooting Gitaly in production +## Troubleshooting Gitaly + +### Commits, pushes, and clones return a 401 + +``` +remote: GitLab: 401 Unauthorized +``` + +You will need to sync your `gitlab-secrets.json` file with your GitLab +app nodes. + +### `gitaly-debug` Since GitLab 11.6, Gitaly comes with a command-line tool called `gitaly-debug` that can be run on a Gitaly server to aid in @@ -462,3 +535,123 @@ Git clone speed for a specific repository on the server. For an up to date list of sub-commands see [the gitaly-debug README](https://gitlab.com/gitlab-org/gitaly/blob/master/cmd/gitaly-debug/README.md). + +### Client side GRPC logs + +Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC +client has its own log file which may contain useful information when +you are seeing Gitaly errors. You can control the log level of the +gRPC client with the `GRPC_LOG_LEVEL` environment variable. The +default level is `WARN`. + +### Observing `gitaly-ruby` traffic + +[`gitaly-ruby`](#gitaly-ruby) is an internal implementation detail of Gitaly, +so, there's not that much visibility into what goes on inside +`gitaly-ruby` processes. + +If you have Prometheus set up to scrape your Gitaly process, you can see +request rates and error codes for individual RPCs in `gitaly-ruby` by +querying `grpc_client_handled_total`. Strictly speaking, this metric does +not differentiate between `gitaly-ruby` and other RPCs, but in practice +(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal +calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. + +Assuming your `grpc_client_handled_total` counter only observes Gitaly, +the following query shows you RPCs are (most likely) internally +implemented as calls to `gitaly-ruby`: + +``` +sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 +``` + +### Repository changes fail with a `401 Unauthorized` error + +If you're running Gitaly on its own server and notice that users can +successfully clone and fetch repositories (via both SSH and HTTPS), but can't +push to them or make changes to the repository in the web UI without getting a +`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate +with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration). + +Confirm the following are all true: + +- When any user performs a `git push` to any repository on this Gitaly node, it + fails with the following error (note the `401 Unauthorized`): + + ```sh + remote: GitLab: 401 Unauthorized + To <REMOTE_URL> + ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) + error: failed to push some refs to '<REMOTE_URL>' + ``` + +- When any user adds or modifies a file from the repository using the GitLab + UI, it immediatley fails with a red `401 Unauthorized` banner. +- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) + successfully creates the project but doesn't create the README. +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.md#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors + when reaching the `/api/v4/internal/allowed` endpoint: + + ```sh + # api_json.log + { + "time": "2019-07-18T00:30:14.967Z", + "severity": "INFO", + "duration": 0.57, + "db": 0, + "view": 0.57, + "status": 401, + "method": "POST", + "path": "\/api\/v4\/internal\/allowed", + "params": [ + { + "key": "action", + "value": "git-receive-pack" + }, + { + "key": "changes", + "value": "REDACTED" + }, + { + "key": "gl_repository", + "value": "REDACTED" + }, + { + "key": "project", + "value": "\/path\/to\/project.git" + }, + { + "key": "protocol", + "value": "web" + }, + { + "key": "env", + "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" + }, + { + "key": "user_id", + "value": "2" + }, + { + "key": "secret_token", + "value": "[FILTERED]" + } + ], + "host": "gitlab.example.com", + "ip": "REDACTED", + "ua": "Ruby", + "route": "\/api\/:version\/internal\/allowed", + "queue_duration": 4.24, + "gitaly_calls": 0, + "gitaly_duration": 0, + "correlation_id": "XPUZqTukaP3" + } + + # nginx_access.log + [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" + ``` + +To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration) +on the Gitaly node matches the one on all other nodes. If it doesn't match, +update the secrets file on the Gitaly node to match the others, then +[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 41ef68f5b57..56665ba8b9a 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -164,25 +164,24 @@ contention due to certain workloads. #### Reference Architecture -- **Status:** Work-in-progress - **Supported Users (approximate):** 10,000 -- **Related Issues:** [gitlab-com/support/support-team-meta#1513](https://gitlab.com/gitlab-com/support/support-team-meta/issues/1513), - [gitlab-org/quality/team-tasks#110](https://gitlab.com/gitlab-org/quality/team-tasks/issues/110) - -The Support and Quality teams are in the process of building and performance testing -an environment that will support about 10,000 users. The specifications below -are a work-in-progress representation of the work so far. Quality will be -certifying this environment in FY20-Q2. The specifications may be adjusted -prior to certification based on performance testing. - -- 3 PostgreSQL - 4 CPU, 8GB RAM per node -- 1 PgBouncer - 2 CPU, 4GB RAM -- 2 Redis - 2 CPU, 8GB RAM per node -- 3 Consul/Sentinel - 2 CPU, 2GB RAM per node -- 4 Sidekiq - 4 CPU, 8GB RAM per node -- 5 GitLab application nodes - 20 CPU, 64GB RAM per node -- 1 Gitaly - 20 CPU, 64GB RAM -- 1 Monitoring node - 4 CPU, 8GB RAM +- **Known Issues:** While validating the reference architecture, slow endpoints were discovered and are being investigated. [gitlab-org/gitlab-ce/issues/64335](https://gitlab.com/gitlab-org/gitlab-ce/issues/64335) + +The Support and Quality teams built, performance tested, and validated an +environment that supports about 10,000 users. The specifications below are a +representation of the work so far. The specifications may be adjusted in the +future based on additional testing and iteration. + +NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + +- 3 PostgreSQL - 4 CPU, 16GiB memory per node +- 1 PgBouncer - 2 CPU, 4GiB memory +- 2 Redis - 2 CPU, 8GiB memory per node +- 3 Consul/Sentinel - 2 CPU, 2GiB memory per node +- 4 Sidekiq - 4 CPU, 16GiB memory per node +- 5 GitLab application nodes - 16 CPU, 64GiB memory per node +- 1 Gitaly - 16 CPU, 64GiB memory +- 1 Monitoring node - 2 CPU, 8GiB memory, 100GiB local storage ### Fully Distributed diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index f7a1f425b40..7c9e02d889e 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -327,6 +327,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' # Replace X with value of number of db nodes + 1 postgresql['max_wal_senders'] = X + postgresql['max_replication_slots'] = X # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) diff --git a/doc/administration/high_availability/img/fully-distributed.png b/doc/administration/high_availability/img/fully-distributed.png Binary files differindex ad23207134e..c3cd2bf24f0 100644 --- a/doc/administration/high_availability/img/fully-distributed.png +++ b/doc/administration/high_availability/img/fully-distributed.png diff --git a/doc/administration/high_availability/img/horizontal.png b/doc/administration/high_availability/img/horizontal.png Binary files differindex c3bd489d96f..75d08e1097a 100644 --- a/doc/administration/high_availability/img/horizontal.png +++ b/doc/administration/high_availability/img/horizontal.png diff --git a/doc/administration/high_availability/img/hybrid.png b/doc/administration/high_availability/img/hybrid.png Binary files differindex 7d4a56bf0ea..8dd9923e597 100644 --- a/doc/administration/high_availability/img/hybrid.png +++ b/doc/administration/high_availability/img/hybrid.png diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index 9e9f604317a..f11d27487d1 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -60,11 +60,21 @@ for details on managing SSL certificates and configuring Nginx. ### Basic ports -| LB Port | Backend Port | Protocol | -| ------- | ------------ | --------------- | -| 80 | 80 | HTTP [^1] | -| 443 | 443 | TCP or HTTPS [^1] [^2] | -| 22 | 22 | TCP | +| LB Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments.md#web-terminals) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](../integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. ### GitLab Pages Ports @@ -72,12 +82,19 @@ If you're using GitLab Pages with custom domain support you will need some additional port configurations. GitLab Pages requires a separate virtual IP address. Configure DNS to point the `pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the -[GitLab Pages documentation][gitlab-pages] for more information. +[GitLab Pages documentation](../pages/index.md) for more information. -| LB Port | Backend Port | Protocol | -| ------- | ------------ | -------- | -| 80 | Varies [^3] | HTTP | -| 443 | Varies [^3] | TCP [^4] | +| LB Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation](../pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. ### Alternate SSH Port @@ -86,7 +103,7 @@ it may be helpful to configure an alternate SSH hostname that allows users to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address compared to the other GitLab HTTP configuration above. -Configure DNS for an alternate SSH hostname such as altssh.gitlab.example.com. +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | LB Port | Backend Port | Protocol | | ------- | ------------ | -------- | @@ -101,24 +118,6 @@ Read more on high-availability configuration: 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) -[^1]: [Web terminal](../../ci/environments.md#web-terminals) support requires - your load balancer to correctly handle WebSocket connections. When using - HTTP or HTTPS proxying, this means your load balancer must be configured - to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the - [web terminal](../integration/terminal.md) integration guide for - more details. -[^2]: When using HTTPS protocol for port 443, you will need to add an SSL - certificate to the load balancers. If you wish to terminate SSL at the - GitLab application server instead, use TCP protocol. -[^3]: The backend port for GitLab Pages depends on the - `gitlab_pages['external_http']` and `gitlab_pages['external_https']` - setting. See [GitLab Pages documentation][gitlab-pages] for more details. -[^4]: Port 443 for GitLab Pages should always use the TCP protocol. Users can - configure custom domains with custom SSL, which would not be possible - if SSL was terminated at the load balancer. - -[gitlab-pages]: ../pages/index.md - <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md index b91a994d01e..b2750603c74 100644 --- a/doc/administration/high_availability/monitoring_node.md +++ b/doc/administration/high_availability/monitoring_node.md @@ -18,7 +18,7 @@ The steps below are the minimum necessary to configure a Monitoring node running Omnibus: 1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install) the Omnibus GitLab +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 294f0e969d5..274bd32299b 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -71,6 +71,11 @@ bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) that may be fixed in [more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). +NOTE: **Note** Red Hat Enterprise 7 [shipped a kernel +update](https://access.redhat.com/errata/RHSA-2019:2029) on August 6, +2019 that may have resolved this problem. The following instructions may +not be needed if the latest kernel is updated properly. + GitLab recommends all NFS users disable the NFS server delegation feature. To disable NFS server delegations on an Linux NFS server, do the following: diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 0b945bc6244..b99724d12a2 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -20,84 +20,85 @@ It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on i 1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - ```ruby - # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] - - # Configure Pgbouncer - pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - - # Configure Consul agent - consul['watchers'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # Replace CONSUL_PASSWORD_HASH with with a generated md5 value - # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value - pgbouncer['users'] = { - 'gitlab-consul': { - password: 'CONSUL_PASSWORD_HASH' - }, - 'pgbouncer': { - password: 'PGBOUNCER_PASSWORD_HASH' - } - } - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `pgbouncer_role` was introduced with GitLab 10.3 - -1. Run `gitlab-ctl reconfigure` + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure Pgbouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # Replace CONSUL_PASSWORD_HASH with with a generated md5 value + # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value + pgbouncer['users'] = { + 'gitlab-consul': { + password: 'CONSUL_PASSWORD_HASH' + }, + 'pgbouncer': { + password: 'PGBOUNCER_PASSWORD_HASH' + } + } + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + NOTE: **Note:** + `pgbouncer_role` was introduced with GitLab 10.3. + +1. Run `gitlab-ctl reconfigure` 1. Create a `.pgpass` file so Consul is able to reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: - ```sh - gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul - ``` + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` #### PGBouncer Checkpoint 1. Ensure the node is talking to the current master: - ```sh - gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD - ``` + ```sh + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` - If there is an error `psql: ERROR: Auth failed` after typing in the - password, ensure you previously generated the MD5 password hashes with the correct - format. The correct format is to concatenate the password and the username: - `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text - needed to generate an MD5 password hash for the `pgbouncer` user. + If there is an error `psql: ERROR: Auth failed` after typing in the + password, ensure you previously generated the MD5 password hashes with the correct + format. The correct format is to concatenate the password and the username: + `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text + needed to generate an MD5 password hash for the `pgbouncer` user. 1. Once the console prompt is available, run the following queries: - ```sh - show databases ; show clients ; - ``` - - The output should be similar to the following: + ```sh + show databases ; show clients ; + ``` - ``` - name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections - ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- - gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 - pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 - (2 rows) + The output should be similar to the following: - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls - ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- - C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | - (2 rows) - ``` + ``` + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` ### Running Pgbouncer as part of a non-HA GitLab installation diff --git a/doc/administration/img/custom_hooks_error_msg.png b/doc/administration/img/custom_hooks_error_msg.png Binary files differindex 4f25c471908..e7d5c157d08 100644 --- a/doc/administration/img/custom_hooks_error_msg.png +++ b/doc/administration/img/custom_hooks_error_msg.png diff --git a/doc/administration/index.md b/doc/administration/index.md index 00c8863f200..2b25e8efd23 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -2,7 +2,7 @@ description: 'Learn how to install, configure, update, and maintain your GitLab instance.' --- -# Administrator documentation **(CORE ONLY)** +# Administrator Docs **(CORE ONLY)** Learn how to administer your self-managed GitLab instance. @@ -185,3 +185,13 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. +- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab + Support team. +- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md): Tips to troubleshoot ElasticSearch. +- [Kubernetes troubleshooting](troubleshooting/kubernetes_cheat_sheet.md): Commands and tips useful + for troubleshooting Kubernetes-related issues. +- Useful links from the Support Team: + - [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html). + - [Repairing and recovering broken git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). + - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html). + - [Strace zine](https://wizardzines.com/zines/strace/). diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index c2ac063ce37..16a193550a1 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -72,12 +72,12 @@ our AsciiDoc snippets, wikis and repos using delimited blocks: - **Markdown** - ````markdown + ~~~markdown ```plantuml Bob -> Alice : hello Alice -> Bob : Go Away ``` - ```` + ~~~ - **AsciiDoc** diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 5490a59ceac..2b624f8de77 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -115,6 +115,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 5a2f389d298..9030c941cad 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -88,7 +88,7 @@ Introduced in GitLab 10.0, this file lives in It helps you see requests made directly to the API. For example: ```json -{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} +{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} ``` This entry above shows an access to an internal endpoint to check whether an @@ -151,22 +151,29 @@ etc. For example: {"severity":"ERROR","time":"2018-11-23T15:42:11.647Z","exception":"Kubeclient::HttpError","error_code":null,"service":"Clusters::Applications::InstallService","app_id":2,"project_ids":[19],"group_ids":[],"message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)"} ``` -## `githost.log` +## `git_json.log` -This file lives in `/var/log/gitlab/gitlab-rails/githost.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/githost.log` for +This file lives in `/var/log/gitlab/gitlab-rails/git_json.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/git_json.log` for installations from source. +NOTE: **Note:** +After 12.2, this file was renamed from `githost.log` to +`git_json.log` and stored in JSON format. + GitLab has to interact with Git repositories but in some rare cases something can go wrong and in this case you will know what exactly happened. This log file contains all failed requests from GitLab to Git repositories. In the majority of cases this file will be useful for developers only. For example: -``` -December 03, 2014 13:20 -> ERROR -> Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict - -error: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git' +```json +{ + "severity":"ERROR", + "time":"2019-07-19T22:16:12.528Z", + "correlation_id":"FeGxww5Hj64", + "message":"Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict\n\nerror: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'" +} ``` ## `audit_json.log` @@ -236,7 +243,7 @@ I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory User clone/fetch activity using ssh transport appears in this log as `executing git command <gitaly-upload-pack...`. -## `unicorn\_stderr.log` +## `unicorn_stderr.log` This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for @@ -277,16 +284,16 @@ Introduced in GitLab 11.3. This file lives in `/var/log/gitlab/gitlab-rails/impo Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for installations from source. -Currently it logs the progress of project imports from the Bitbucket Server -importer. Future importers may use this file. - -## `auth.log` +## `auth.log` Introduced in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/auth.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/auth.log` for installations from source. -It logs information whenever [Rack Attack] registers an abusive request. +This log records: + +- Information whenever [Rack Attack] registers an abusive request. +- Requests over the [Rate Limit] on raw endpoints. NOTE: **Note:** From [%12.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/62756), user id and username are available on this log. @@ -305,6 +312,12 @@ GraphQL queries are recorded in that file. For example: {"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration":7} ``` +## `migrations.log` + +Introduced in GitLab 12.3. This file lives in `/var/log/gitlab/gitlab-rails/migrations.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/migrations.log` for +installations from source. + ## Reconfigure Logs Reconfigure log files live in `/var/log/gitlab/reconfigure` for Omnibus GitLab @@ -324,3 +337,4 @@ installations from source. [repocheck]: repository_checks.md [Rack Attack]: ../security/rack_attack.md +[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 99cd9051778..0b065082ded 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -90,6 +90,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md new file mode 100644 index 00000000000..d445b68721d --- /dev/null +++ b/doc/administration/monitoring/gitlab_instance_administration_project/index.md @@ -0,0 +1,36 @@ +# GitLab instance administration project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/56883) in GitLab 12.2. + +GitLab has been adding the ability for administrators to see insights into the health of +their GitLab instance. In order to surface this experience in a native way, similar to how +you would interact with an application deployed via GitLab, a base project called +"GitLab Instance Administration" with +[internal visibility](../../../public_access/public_access.md#internal-projects) will be +added under a group called "GitLab Instance Administrators" specifically created for +visualizing and configuring the monitoring of your GitLab instance. + +All administrators at the time of creation of the project and group will be added +as maintainers of the group and project, and as an admin, you'll be able to add new +members to the group in order to give them maintainer access to the project. + +This project will be used for self-monitoring your GitLab instance. + +## Connection to Prometheus + +The project will be automatically configured to connect to the +[internal Prometheus](../prometheus/index.md) instance if the Prometheus +instance is present (should be the case if GitLab was installed via Omnibus +and you haven't disabled it). + +If that's not the case or if you have an external Prometheus instance or an HA setup, +you should +[configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). + +## Taking action on Prometheus alerts **(ULTIMATE)** + +You can [add a webhook](../../../user/project/integrations/prometheus.md#external-prometheus-instances) +to the Prometheus config in order for GitLab to receive notifications of any alerts. + +Once the webhook is setup, you can +[take action on incoming alerts](../../../user/project/integrations/prometheus.md#taking-action-on-incidents-ultimate). diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index fa0459b24ff..2b3daec42bd 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -2,6 +2,9 @@ Explore our features to monitor your GitLab instance: +- [GitLab self-monitoring](gitlab_instance_administration_project/index.md): The + GitLab instance administration project helps to monitor the GitLab instance and + take action on alerts. - [Performance monitoring](performance/index.md): GitLab Performance Monitoring makes it possible to measure a wide variety of statistics of your instance. - [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. - [GitHub imports](github_imports.md): Monitor the health and progress of GitLab's GitHub importer with various Prometheus metrics. diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 4dd0bbbe937..95be0d5fd88 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,6 +1,6 @@ # Grafana Configuration -[Grafana](https://grafana.org/) is a tool that allows you to visualize time +[Grafana](https://grafana.com/) is a tool that allows you to visualize time series metrics through graphs and dashboards. It supports several backend data stores, including InfluxDB. GitLab writes performance data to InfluxDB and Grafana will allow you to query to display useful graphs. @@ -118,6 +118,36 @@ If you have set up Grafana, you can enable a link to access it easily from the s 1. Click **Save changes**. 1. The new link will be available in the admin area under **Monitoring > Metrics Dashboard**. +## Security Update + +Users running GitLab version 12.0 or later should immediately upgrade to one of the following security releases due to a known vulnerability with the embedded Grafana dashboard: + +- 12.0.6 +- 12.1.6 + +After upgrading, the Grafana dashboard will be disabled and the location of your existing Grafana data will be changed from `/var/opt/gitlab/grafana/data/` to `/var/opt/gitlab/grafana/data.bak.#{Date.today}/`. + +To prevent the data from being relocated, you can run the following command prior to upgrading: + +```sh +echo "0" > /var/opt/gitlab/grafana/CVE_reset_status +``` + +To reinstate your old data, move it back into its original location: + +``` +sudo mv /var/opt/gitlab/grafana/data.bak.xxxx/ /var/opt/gitlab/grafana/data/ +``` + +However, you should **not** reinstate your old data _except_ under one of the following conditions: + +1. If you are certain that you changed your default admin password when you enabled Grafana +1. If you run GitLab in a private network, accessed only by trusted users, and your Grafana login page has not been exposed to the internet + +If you require access to your old Grafana data but do not meet one of these criteria, you may consider reinstating it temporarily, [exporting the dashboards](https://grafana.com/docs/reference/export_import/#exporting-a-dashboard) you need, then refreshing the data and [re-importing your dashboards](https://grafana.com/docs/reference/export_import/#importing-a-dashboard). Note that this poses a temporary vulnerability while your old Grafana data is in use, and the decision to do so should be weighed carefully with your need to access existing data and dashboards. + +For more information and further mitigation details, please refer to our [blog post on the security release](https://about.gitlab.com/2019/08/12/critical-security-release-gitlab-12-dot-1-dot-6-released/). + --- Read more on: diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex 2bf686f9017..d1187fd879a 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png Binary files differindex 7af6d401d1d..2c43201cbd0 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png Binary files differdeleted file mode 100644 index a55ce753101..00000000000 --- a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png Binary files differnew file mode 100644 index 00000000000..ecb2dffbf8d --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png Binary files differnew file mode 100644 index 00000000000..210f80a713d --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png Binary files differindex b3219b4fa94..6b571f4e85c 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png +++ b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png diff --git a/doc/administration/monitoring/performance/img/request_profile_result.png b/doc/administration/monitoring/performance/img/request_profile_result.png Binary files differindex 1b06e240fa0..9176a0b49fd 100644 --- a/doc/administration/monitoring/performance/img/request_profile_result.png +++ b/doc/administration/monitoring/performance/img/request_profile_result.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 4ee156fdc11..02f4b78bd60 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -8,16 +8,14 @@ activated, it looks as follows: It allows you to see (from left to right): - the current host serving the page -- the timing of the page (backend, frontend) - time taken and number of DB queries, click through for details of these queries  - time taken and number of [Gitaly] calls, click through for details of these calls  -- profile of the code used to generate the page, line by line. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). -  -- time taken and number of calls to Redis -- time taken and number of background jobs created by Sidekiq -- time taken and number of Ruby GC calls +- time taken and number of [Rugged] calls, click through for details of these calls +  +- time taken and number of Redis calls, click through for details of these calls +  On the far right is a request selector that allows you to view the same metrics (excluding the page timing and line profiler) for any requests made while the @@ -43,3 +41,4 @@ You can toggle the Bar using the same shortcut.  [Gitaly]: ../../gitaly/index.md +[Rugged]: ../../high_availability/nfs.md#improving-nfs-performance-with-gitlab diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 726882fbb87..c32edb60f9d 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -5,9 +5,9 @@ 1. Grab the profiling token from **Monitoring > Requests Profiles** admin page (highlighted in a blue in the image below).  -1. Pass the header `X-Profile-Token: <token>` to the request you want to profile. You can use: +1. Pass the header `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where `<mode>` can be `execution` or `memory`) to the request you want to profile. You can use: - Browser extensions. For example, [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension. - - `curl`. For example, `curl --header 'X-Profile-Token: <token>' https://gitlab.example.com/group/project`. + - `curl`. For example, `curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' https://gitlab.example.com/group/project`. 1. Once request is finished (which will take a little longer than usual), you can view the profiling output from **Monitoring > Requests Profiles** admin page.  diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 89501f20d99..ec26c0b2e7e 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -34,6 +34,9 @@ The following metrics are available: | filesystem_writable | Gauge | 9.4 | Whether or not the filesystem is writable | | filesystem_read_latency_seconds | Gauge | 9.4 | Read latency of a specific filesystem | | filesystem_readable | Gauge | 9.4 | Whether or not the filesystem is readable | +| gitlab_cache_misses_total | Counter | 10.2 | Cache read miss | +| gitlab_cache_operation_duration_seconds | Histogram | 10.2 | Cache access time | +| gitlab_cache_operations_total | Counter | 12.2 | Cache operations by controller/action | | http_requests_total | Counter | 9.4 | Rack request count | | http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | | pipelines_created_total | Counter | 9.4 | Counter of pipelines created | @@ -117,7 +120,6 @@ When Puma is used instead of Unicorn, following metrics are available: | puma_workers | Gauge | 12.0 | Total number of workers | | puma_running_workers | Gauge | 12.0 | Number of booted workers | | puma_stale_workers | Gauge | 12.0 | Number of old workers | -| puma_phase | Gauge | 12.0 | Phase number (increased during phased restarts) | | puma_running | Gauge | 12.0 | Number of running threads | | puma_queued_connections | Gauge | 12.0 | Number of connections in that worker's "todo" set waiting for a worker thread | | puma_active_connections | Gauge | 12.0 | Number of threads processing a request | diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index ea69378b249..e787af798bc 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -71,10 +71,10 @@ sudo service sshd reload Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo. -> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in +NOTE: **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in GitLab 11.11 and later. -> **Warning:** Do not disable writes until SSH is confirmed to be working +CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. In the case of lookup failures (which are common), the `authorized_keys` diff --git a/doc/administration/operations/img/sidekiq-cluster.png b/doc/administration/operations/img/sidekiq-cluster.png Binary files differindex 4eb1849010e..3899385eb8f 100644 --- a/doc/administration/operations/img/sidekiq-cluster.png +++ b/doc/administration/operations/img/sidekiq-cluster.png diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index ae67d7f08d6..8178cb243f3 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -69,7 +69,7 @@ unicorn['worker_memory_limit_min'] = "400 * 1 << 20" unicorn['worker_memory_limit_max'] = "650 * 1 << 20" ``` -Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MIN` +Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MAX` [environment variables](../environment_variables.md). This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. diff --git a/doc/administration/pages/img/lets_encrypt_integration_v12_1.png b/doc/administration/pages/img/lets_encrypt_integration_v12_1.png Binary files differindex 5ab63074e12..0f3ca25ce55 100644 --- a/doc/administration/pages/img/lets_encrypt_integration_v12_1.png +++ b/doc/administration/pages/img/lets_encrypt_integration_v12_1.png diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index b2cad6cf926..c77a1a9638f 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -24,8 +24,6 @@ SNI and exposes pages using HTTP2 by default. You are encouraged to read its [README][pages-readme] to fully understand how it works. ---- - In the case of [custom domains](#custom-domains) (but not [wildcard domains](#wildcard-domains)), the Pages daemon needs to listen on ports `80` and/or `443`. For that reason, there is some flexibility in the way @@ -92,8 +90,6 @@ since that is needed in all configurations. - [Wildcard DNS setup](#dns-configuration) ---- - URL scheme: `http://page.example.io` This is the minimum setup that you can use Pages with. It is the base for all @@ -152,13 +148,12 @@ The Pages daemon doesn't listen to the outside world. ### Wildcard domains with TLS support -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Wildcard TLS certificate -> -> --- -> -> URL scheme: `https://page.example.io` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Wildcard TLS certificate + +URL scheme: `https://page.example.io` Nginx will proxy all requests to the daemon. Pages daemon doesn't listen to the outside world. @@ -217,13 +212,12 @@ that without TLS certificates. ### Custom domains -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Secondary IP -> -> --- -> -> URL scheme: `http://page.example.io` and `http://domain.com` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Secondary IP + +URL scheme: `http://page.example.io` and `http://domain.com` In that case, the pages daemon is running, Nginx still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -282,14 +276,13 @@ world. Custom domains are supported, but no TLS. ### Custom domains with TLS support -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Wildcard TLS certificate -> - Secondary IP -> -> --- -> -> URL scheme: `https://page.example.io` and `https://domain.com` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Wildcard TLS certificate +- Secondary IP + +URL scheme: `https://page.example.io` and `https://domain.com` In that case, the pages daemon is running, Nginx still proxies requests to the daemon but the daemon is also able to receive requests from the outside diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index 4302667caf5..4cf3c607dae 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -52,7 +52,37 @@ as appropriate. The plugins file list is updated for each event, there is no need to restart GitLab to apply a new plugin. If a plugin executes with non-zero exit code or GitLab fails to execute it, a -message will be logged to `plugin.log`. +message will be logged to: + +- `gitlab-rails/plugin.log` in an Omnibus installation. +- `log/plugin.log` in a source installation. + +## Creating plugins + +Below is an example that will only response on the event `project_create` and +will inform the admins from the GitLab instance that a new project has been created. + +```ruby +# By using the embedded ruby version we eliminate the possibility that our chosen language +# would be unavailable from +#!/opt/gitlab/embedded/bin/ruby +require 'json' +require 'mail' + +# The incoming variables are in JSON format so we need to parse it first. +ARGS = JSON.parse(STDIN.read) + +# We only want to trigger this plugin on the event project_create +return unless ARGS['event_name'] == 'project_create' + +# We will inform our admins of our gitlab instance that a new project is created +Mail.deliver do + from 'info@gitlab_instance.com' + to 'admin@gitlab_instance.com' + subject "new project " + ARGS['name'] + body ARGS['owner_name'] + 'created project ' + ARGS['name'] +end +``` ## Validation diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 91fc0133d56..e880d76e756 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -28,6 +28,31 @@ limit by passing a number to the check task: rake gitlab:ldap:check[50] ``` +## Run a Group Sync + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.3. + +The following task will run a [group sync](../auth/ldap-ee.md#group-sync) immediately. This is valuable +when you'd like to update all configured group memberships against LDAP without +waiting for the next scheduled group sync to be run. + +NOTE: **NOTE:** +If you'd like to change the frequency at which a group sync is performed, +[adjust the cron schedule](../auth/ldap-ee.md#adjusting-ldap-group-sync-schedule) +instead. + +**Omnibus Installation** + +``` +sudo gitlab-rake gitlab:ldap:group_sync +``` + +**Source Installation** + +```bash +bundle exec rake gitlab:ldap:group_sync +``` + ## Rename a provider If you change the LDAP server ID in `gitlab.yml` or `gitlab.rb` you will need diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 2f83dd17d9f..1198f3414c5 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -105,8 +105,6 @@ sudo gitlab-rake gitlab:storage:legacy_projects sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production ``` ---- - To list projects using **Legacy** storage: **Omnibus Installation** @@ -138,8 +136,6 @@ sudo gitlab-rake gitlab:storage:hashed_projects sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production ``` ---- - To list projects using **Hashed** storage: **Omnibus Installation** @@ -170,8 +166,6 @@ sudo gitlab-rake gitlab:storage:legacy_attachments sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production ``` ---- - To list project attachments using **Legacy** storage: **Omnibus Installation** @@ -202,8 +196,6 @@ sudo gitlab-rake gitlab:storage:hashed_attachments sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production ``` ---- - To list project attachments using **Hashed** storage: **Omnibus Installation** diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index fd8ea8d3162..86e8b763f51 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -103,3 +103,13 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, S sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" ``` + +## Migrate legacy uploads out of deprecated paths + +> Introduced in GitLab 12.3. + +To migrate all uploads created by legacy uploaders, run: + +```shell +bundle exec rake gitlab:uploads:legacy:migrate +``` diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 7cf8f20a9dc..05a7cb006a3 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -33,8 +33,8 @@ in `repocheck.log`: - in the [admin panel](logs.md#repochecklog) - or on disk, see: - - `/var/log/gitlab/gitlab-rails` for Omnibus installations - - `/home/git/gitlab/log` for installations from source + - `/var/log/gitlab/gitlab-rails` for Omnibus installations + - `/home/git/gitlab/log` for installations from source If for some reason the periodic repository check caused a lot of false alarms you can choose to clear *all* repository check states by diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index ad3a9b19c3c..b1a870210a8 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -57,10 +57,10 @@ storage2: Now that you've read that big fat warning above, let's edit the configuration files and add the full paths of the alternative repository storage paths. In -the example below, we add two more mountpoints that are named `nfs` and `cephfs` +the example below, we add two more mountpoints that are named `nfs_1` and `nfs_2` respectively. -NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. **For installations from source** @@ -73,10 +73,10 @@ NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS storages: # You must have at least a 'default' storage path. default: path: /home/git/repositories - nfs: - path: /mnt/nfs/repositories - cephfs: - path: /mnt/cephfs/repositories + nfs_1: + path: /mnt/nfs1/repositories + nfs_2: + path: /mnt/nfs2/repositories ``` 1. [Restart GitLab][restart-gitlab] for the changes to take effect. @@ -96,8 +96,8 @@ working, you can remove the `repos_path` line. ```ruby git_data_dirs({ "default" => { "path" => "/var/opt/gitlab/git-data" }, - "nfs" => { "path" => "/mnt/nfs/git-data" }, - "cephfs" => { "path" => "/mnt/cephfs/git-data" } + "nfs_1" => { "path" => "/mnt/nfs1/git-data" }, + "nfs_2" => { "path" => "/mnt/nfs2/git-data" } }) ``` diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 9dea6074a3f..1669f8b128c 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -105,7 +105,7 @@ question. To start a migration, enable Hashed Storage for new projects: 1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -2. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. +1. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. Check if the change breaks any existing integration you may have that either runs on the same machine as your repositories are located, or may login to that machine @@ -133,7 +133,7 @@ Similar to the migration, to disable Hashed Storage for new projects: 1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -2. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. +1. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. To schedule a complete rollback, see the [rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 098d946a9fa..604dff5983d 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -209,7 +209,7 @@ ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -s 10 The output in `/tmp/unicorn.txt` may help diagnose the root cause. -# More information +## More information - [Debugging Stuck Ruby Processes](https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/) - [Cheatsheet of using gdb and ruby processes](gdb-stuck-ruby.txt) diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md new file mode 100644 index 00000000000..ab3b25f0e97 --- /dev/null +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -0,0 +1,27 @@ +--- +type: reference +--- + +# Diagnostics tools + +These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. +They are listed here for transparency, and they may be useful for users with experience +with troubleshooting GitLab. If you are currently having an issue with GitLab, you +may want to check your [support options](https://about.gitlab.com/support/) first, +before attempting to use these tools. + +## gitlabsos + +The [gitlabsos](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) utility +provides a unified method of gathering info and logs from GitLab and the system it's +running on. + +## strace-parser + +[strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze +and summarize raw strace data. + +## Pritaly + +[Pritaly](https://gitlab.com/wchandler/pritaly) takes Gitaly logs and colorizes output +or converts the logs to JSON. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md new file mode 100644 index 00000000000..c4a7ba01fae --- /dev/null +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -0,0 +1,345 @@ +# Troubleshooting ElasticSearch + +Troubleshooting ElasticSearch requires: + +- Knowledge of common terms. +- Establishing within which category the problem fits. + +## Common terminology + +- **Lucene**: A full-text search library written in Java. +- **Near Realtime (NRT)**: Refers to the slight latency from the time to index a + document to the time when it becomes searchable. +- **Cluster**: A collection of one or more nodes that work together to hold all + the data, providing indexing and search capabilities. +- **Node**: A single server that works as part of a cluster. +- **Index**: A collection of documents that have somewhat similar characteristics. +- **Document**: A basic unit of information that can be indexed. +- **Shards**: Fully-functional and independent subdivisions of indices. Each shard is actually + a Lucene index. +- **Replicas**: Failover mechanisms that duplicate indices. + +## Troubleshooting workflows + +The type of problem will determine what steps to take. The possible troubleshooting workflows are for: + +- Search results. +- Indexing. +- Integration. +- Performance. + +### Search Results workflow + +The following workflow is for ElasticSearch search results issues: + +```mermaid +graph TD; + B --> |No| B1 + B --> |Yes| B4 + B1 --> B2 + B2 --> B3 + B4 --> B5 + B5 --> |Yes| B6 + B5 --> |No| B7 + B7 --> B8 + B{Is GitLab using<br>ElasticSearch for<br>searching?} + B1[Check Admin Area > Integrations<br>to ensure the settings are correct] + B2[Perform a search via<br>the rails console] + B3[If all settings are correct<br>and it still doesn't show ElasticSearch<br>doing the searches, escalate<br>to GitLab support.] + B4[Perform<br>the same search via the<br>ElasticSearch API] + B5{Are the results<br>the same?} + B6[This means it is working as intended.<br>Speak with GitLab support<br>to confirm if the issue lies with<br>the filters.] + B7[Check the index status of the project<br>containing the missing search<br>results.] + B8(Indexing Troubleshooting) +``` + +### Indexing workflow + +The following workflow is for ElasticSearch indexing issues: + +```mermaid +graph TD; + C --> |Yes| C1 + C1 --> |Yes| C2 + C1 --> |No| C3 + C3 --> |Yes| C4 + C3 --> |No| C5 + C --> |No| C6 + C6 --> |No| C10 + C7 --> |GitLab| C8 + C7 --> |ElasticSearch| C9 + C6 --> |Yes| C7 + C10 --> |No| C12 + C10 --> |Yes| C11 + C12 --> |Yes| C13 + C12 --> |No| C14 + C14 --> |Yes| C15 + C14 --> |No| C16 + C{Is the problem with<br>creating an empty<br>index?} + C1{Does the gitlab-production<br>index exist on the<br>ElasticSearch instance?} + C2(Try to manually<br>delete the index on the<br>ElasticSearch instance and<br>retry creating an empty index.) + C3{Can indices be made<br>manually on the ElasticSearch<br>instance?} + C4(Retry the creation of an empty index) + C5(It is best to speak with an<br>ElasticSearch admin concerning the<br>instance's inability to create indices.) + C6{Is the indexer presenting<br>errors during indexing?} + C7{Is the error a GitLab<br>error or an ElasticSearch<br>error?} + C8[Escalate to<br>GitLab support] + C9[You will want<br>to speak with an<br>ElasticSearch admin.] + C10{Does the index status<br>show 100%?} + C11[Escalate to<br>GitLab support] + C12{Does re-indexing the project<br> present any GitLab errors?} + C13[Rectify the GitLab errors and<br>restart troubleshooting, or<br>escalate to GitLab support.] + C14{Does re-indexing the project<br>present errors on the <br>ElasticSearch instance?} + C15[It would be best<br>to speak with an<br>ElasticSearch admin.] + C16[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] +``` + +### Integration workflow + +The following workflow is for ElasticSearch integration issues: + +```mermaid +graph TD; + D --> |No| D1 + D --> |Yes| D2 + D2 --> |No| D3 + D2 --> |Yes| D4 + D4 --> |No| D5 + D4 --> |Yes| D6 + D{Is the error concerning<br>the beta indexer?} + D1[It would be best<br>to speak with an<br>ElasticSearch admin.] + D2{Is the ICU development<br>package installed?} + D3>This package is required.<br>Install the package<br>and retry.] + D4{Is the error stemming<br>from the indexer?} + D5[This would indicate an OS level<br> issue. It would be best to<br>contact your sysadmin.] + D6[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] +``` + +### Performance workflow + +The following workflow is for ElasticSearch performance issues: + +```mermaid +graph TD; + F --> |Yes| F1 + F --> |No| F2 + F2 --> |No| F3 + F2 --> |Yes| F4 + F4 --> F5 + F5 --> |No| F6 + F5 --> |Yes| F7 + F{Is the ElasticSearch instance<br>running on the same server<br>as the GitLab instance?} + F1(This is not advised and will cause issues.<br>We recommend moving the ElasticSearch<br>instance to a different server.) + F2{Does the ElasticSearch<br>server have at least 8<br>GB of RAM and 2 CPU<br>cores?} + F3(According to ElasticSearch, a non-prod<br>server needs these as a base requirement.<br>Production often requires more. We recommend<br>you increase the server specifications.) + F4(Obtain the <br>cluster health information) + F5(Does it show the<br>status as green?) + F6(We recommend you speak with<br>an ElasticSearch admin<br>about implementing sharding.) + F7(Escalate to<br>GitLab support.) +``` + +## Troubleshooting walkthrough + +Most ElasticSearch troubleshooting can be broken down into 4 categories: + +- [Troubleshooting search results](#troubleshooting-search-results) +- [Troubleshooting indexing](#troubleshooting-indexing) +- [Troubleshooting integration](#troubleshooting-integration) +- [Troubleshooting performance](#troubleshooting-performance) + +Generally speaking, if it does not fall into those four categories, it is either: + +- Something GitLab support needs to look into. +- Not a true ElasticSearch issue. + +Exercise caution. Issues that appear to be ElasticSearch problems can be OS-level issues. + +### Troubleshooting search results + +Troubleshooting search result issues is rather straight forward on ElasticSearch. + +The first step is to confirm GitLab is using ElasticSearch for the search function. +To do this: + +1. Confirm the integration is enabled in **Admin Area > Settings > Integrations**. +1. Confirm searches utilize ElasticSearch by accessing the rails console + (`sudo gitlab-rails console`) and running the following commands: + + ```rails + u = User.find_by_email('email_of_user_doing_search') + s = SearchService.new(u, {:search => 'search_term'}) + pp s.search_objects.class.name + ``` + +The ouput from the last command is the key here. If it shows: + +- `ActiveRecord::Relation`, **it is not** using ElasticSearch. +- `Kaminari::PaginatableArray`, **it is** using ElasticSearch. + +| Not using ElasticSearch | Using ElasticSearch | +|--------------------------|------------------------------| +| `ActiveRecord::Relation` | `Kaminari::PaginatableArray` | + +If all the settings look correct and it is still not using ElasticSearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. + +Moving past that, it is best to attempt the same search using the [ElasticSearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and compare the results from what you see in GitLab. + +If the results: + +- Sync up, then there is not a technical "issue" per se. Instead, it might be a problem + with the ElasticSearch filters we are using. This can be complicated, so it is best to + escalate to GitLab support to check these and guide you on the potential on whether or + not a feature request is needed. +- Do not match up, this indicates a problem with the documents generated from the + project. It is best to re-index that project and proceed with + [Troubleshooting indexing](#troubleshooting-indexing). + +### Troubleshooting indexing + +Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab +support or your ElasticSearch admin. + +The best place to start is to determine if the issue is with creating an empty index. +If it is, check on the ElasticSearch side to determine if the `gitlab-production` (the +name for the GitLab index) exists. If it exists, manually delete it on the ElasticSearch +side and attempt to recreate it from the +[`create_empty_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +rake task. + +If you still encounter issues, try creating an index manually on the ElasticSearch +instance. The details of the index aren't important here, as we want to test if indices +can be made. If the indices: + +- Cannot be made, speak with your ElasticSearch admin. +- Can be made, Escalate this to GitLab support. + +If the issue is not with creating an empty index, the next step is to check for errors +during the indexing of projects. If errors do occur, they will either stem from the indexing: + +- On the GitLab side. You need to rectify those. If they are not + something you are familiar with, contact GitLab support for guidance. +- Within the ElasticSearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your ElasticSearch admin. + +If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following rake tasks: + +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows the overall status) +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows specific projects that are not indexed) + +If: + +- Everything is showing at 100%, escalate to GitLab support. This could be a potential + bug/issue. +- You do see something not at 100%, attempt to reindex that project. To do this, + run `sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=<project ID> ID_TO=<project ID>`. + +If reindexing the project shows: + +- Errors on the GitLab side, escalate those to GitLab support. +- ElasticSearch errors or doesn't present any errors at all, reach out to your + ElasticSearch admin to check the instance. + +### Troubleshooting integration + +Troubleshooting integration tends to be pretty straight forward, as there really isn't +much to "integrate" here. + +If the issue is: + +- Not concerning the beta indexer, it is almost always an + ElasticSearch-side issue. This means you should reach out to your ElasticSearch admin + regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach + out to GitLab support. +- With the beta indexer, check if the ICU development package is installed. + This is a required package so make sure you install it. + +Beyond that, you will want to review the error. If it is: + +- Specifically from the indexer, this could be a bug/issue and should be escalated to + GitLab support. +- An OS issue, you will want to reach out to your systems administrator. + +### Troubleshooting performance + +Troubleshooting performance can be difficult on ElasticSearch. There is a ton of tuning +that *can* be done, but the majority of this falls on shoulders of a skilled +ElasticSearch administrator. + +Generally speaking, ensure: + +* The ElasticSearch server **is not** running on the same node as GitLab. +* The ElasticSearch server have enough RAM and CPU cores. +* That sharding **is** being used. + +Going into some more detail here, if ElasticSearch is running on the same server as GitLab, resource contention is **very** likely to occur. Ideally, ElasticSearch, which requires ample resources, should be running on its own server (maybe coupled with logstash and kibana). + +When it comes to ElasticSearch, RAM is the key resource. ElasticSearch themselves recommend: + +- **At least** 8 GB of RAM for a non-production instance. +- **At least** 16 GB of RAM for a production instance. +- Ideally, 64 GB of RAM. + +For CPU, ElasticSearch recommends at least 2 CPU cores, but ElasticSearch states common +setups use up to 8 cores. For more details on server specs, check out +[ElasticSearch's hardware guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). + +Beyond the obvious, sharding comes into play. Sharding is a core part of ElasticSearch. +It allows for horizontal scaling of indices, which is helpful when you are dealing with +a large amount of data. + +With the way GitLab does indexing, there is a **huge** amount of documents being +indexed. By utilizing sharding, you can speed up ElasticSearch's ability to locate +data, since each shard is a Lucene index. + +If you are not using sharding, you are likely to hit issues when you start using +ElasticSearch in a production environment. + +Keep in mind that an index with only one shard has **no scale factor** and will +likely encounter issues when called upon with some frequency. + +If you need to know how many shards, read +[ElasticSearch's documentation on capacity planning](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/capacity-planning.html), +as the answer is not straight forward. + +The easiest way to determine if sharding is in use is to check the output of the +[ElasticSearch Health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html): + +- Red means the cluster is down. +- Yellow means it is up with no sharding/replication. +- Green means it is healthy (up, sharding, replicating). + +For production use, it should always be green. + +Beyond these steps, you get into some of the more complicated things to check, +such as merges and caching. These can get complicated and it takes some time to +learn them, so it is best to escalate/pair with an ElasticSearch expert if you need to +dig further into these. + +Feel free to reach out to GitLab support, but this is likely to be something a skilled +ElasticSearch admin has more experience with. + +## Common issues + +All common issues [should be documented](../../integration/elasticsearch.md#troubleshooting). If not, +feel free to update that page with issues you encounter and solutions. + +## Replication + +Setting up ElasticSearch isn't too bad, but it can be a bit finnicky and time consuming. + +The eastiest method is to spin up a docker container with the required version and +bind ports 9200/9300 so it can be used. + +The following is an example of running a docker container of ElasticSearch v7.2.0: + +```bash +docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0 +docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.2.0 +``` + +From here, you can: + +- Grab the IP of the docker container (use `docker inspect <container_id>`) +- Use `<IP.add.re.ss:9200>` to communicate with it. + +This is a quick method to test out ElasticSearch, but by no means is this a +production solution. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md new file mode 100644 index 00000000000..238c522a0ee --- /dev/null +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -0,0 +1,251 @@ +--- +type: reference +--- + +# Kubernetes, GitLab and You + +This is a list of useful information regarding Kubernetes that the GitLab Support +Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone +can make use of the Support team's collected knowledge + +CAUTION: **Caution:** +These commands **can alter or break** your Kubernetes components so use these at your own risk. + +If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how +to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) +and they will assist you with any issues you are having. + +## Generic kubernetes commands + +- How to authorize to your GCP project (can be especially useful if you have projects + under different GCP accounts): + + ```bash + gcloud auth login + ``` + +- How to access Kubernetes dashboard: + + ```bash + # for minikube: + minikube dashboard —url + # for non-local installations if access via Kubectl is configured: + kubectl proxy + ``` + +- How to ssh to a Kubernetes node and enter the container as root + <https://github.com/kubernetes/kubernetes/issues/30656>: + + - For GCP, you may find the node name and run `gcloud compute ssh node-name`. + - List containers using `docker ps`. + - Enter container using `docker exec --user root -ti container-id bash`. + +- How to copy a file from local machine to a pod: + + ```bash + kubectl cp file-name pod-name:./destination-path + ``` + +- What to do with pods in `CrashLoopBackoff` status: + + - Check logs via Kubernetes dashboard. + - Check logs via Kubectl: + + ```bash + kubectl logs <unicorn pod> -c dependencies + ``` + +- How to tail all Kubernetes cluster events in real time: + + ```bash + kubectl get events -w --all-namespaces + ``` + +- How to get logs of the previously terminated pod instance: + + ```bash + kubectl logs <pod-name> --previous + ``` + + NOTE: **Note:** + No logs are kept in the containers/pods themselves, everything is written to stdout. + This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) + for details. + +## GitLab-specific kubernetes information + +- Minimal config that can be used to test a Kubernetes helm chart can be found + [here](https://gitlab.com/charts/gitlab/issues/620). + +- Tailing logs of a separate pod. An example for a unicorn pod: + + ```bash + kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn + ``` + +- It is not possible to get all the logs via Kubectl at once, like with `gitlab-ctl tail`, + but a number of third-party tools can be used to do it: + + - [Kubetail](https://github.com/johanhaleby/kubetail) + - [kail: kubernetes tail](https://github.com/boz/kail) + - [stern](https://github.com/wercker/stern) + +- Check all events in the `gitlab` namespace (the namespace name can be different if you + specified a different one when deploying the helm chart): + + ```bash + kubectl get events -w --namespace=gitlab + ``` + +- Most of the useful GitLab tools (console, rake tasks, etc) are found in the task-runner + pod. You may enter it and run commands inside or run them from the outside: + + ```bash + # find the pod + kubectl get pods | grep task-runner + + # enter it + kubectl exec -it <task-runner-pod-name> bash + + # open rails console + # rails console can be also called from other GitLab pods + /srv/gitlab/bin/rails console + + # source-style commands should also work + /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production + + # run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart + /usr/local/bin/gitlab-rake gitlab:check + + # open console without entering pod + kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console + + # check the status of DB migrations + kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status + ``` + + You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. + +- Troubleshooting **Operations > Kubernetes** integration: + + - Check the output of `kubectl get events -w --all-namespaces`. + - Check the logs of pods within `gitlab-managed-apps` namespace. + - On the side of GitLab check sidekiq log and kubernetes log. When GitLab is installed + via Helm Chart, `kubernetes.log` can be found inside the sidekiq pod. + +- How to get your initial admin password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: + + ```bash + # find the name of the secret containing the password + kubectl get secrets | grep initial-root + # decode it + kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo + ``` + +- How to connect to a GitLab Postgres database: + + ```bash + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p + ``` + +- How to get info about Helm installation status: + + ```bash + helm status name-of-installation + ``` + +- How to update GitLab installed using Helm Chart: + + ```bash + helm repo upgrade + + # get current values and redirect them to yaml file (analogue of gitlab.rb values) + helm get values <release name> > gitlab.yaml + + # run upgrade itself + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + + After <https://canary.gitlab.com/charts/gitlab/issues/780> is fixed, it should + be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#updating-gitlab-using-the-helm-chart) + for upgrades. + +- How to apply changes to GitLab config: + + - Modify the `gitlab.yaml` file. + - Run the following command to apply changes: + + ```bash + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + +## Installation of minimal GitLab config via Minukube on macOS + +This section is based on [Developing for Kubernetes with Minikube](https://gitlab.com/charts/gitlab/blob/master/doc/minikube/index.md) +and [Helm](https://gitlab.com/charts/gitlab/blob/master/doc/helm/index.md). Refer +to those documents for details. + +- Install Kubectl via Homebrew: + + ```bash + brew install kubernetes-cli + ``` + +- Install Minikube via Homebrew: + + ```bash + brew cask install minikube + ``` + +- Start Minikube and configure it. If Minikube cannot start, try running `minikube delete && minikube start` + and repeat the steps: + + ```bash + minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work + minikube addons enable ingress + minikube addons enable kube-dns + ``` + +- Install Helm via Homebrew and initialize it: + + ```bash + brew install kubernetes-helm + helm init --service-account tiller + ``` + +- Copy the file <https://gitlab.com/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml> + to your workstation. + +- Find the IP address in the output of `minikube ip` and update the yaml file with + this IP address. + +- Install the GitLab Helm Chart: + + ```bash + helm repo add gitlab https://charts.gitlab.io + helm install --name gitlab -f <path-to-yaml-file> gitlab/gitlab + ``` + + If you want to modify some GitLab settings, you can use the above-mentioned config + as a base and create your own yaml file. + +- Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. + The installation could take up to 20-30 minutes depending on the amount of resources + on your workstation. + +- When all the pods show either a `Running` or `Completed` status, get the GitLab password as + described in [Initial login](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#initial-login), + and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` + where `domain` is the value provided in the yaml file. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 51e267c865e..99f1c386183 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -78,6 +78,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -140,6 +141,22 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). +### Oracle Cloud S3 connection settings + +Note that Oracle Cloud S3 must be sure to use the following settings: + +| Setting | Value | +|---------|-------| +| `enable_signature_v4_streaming` | false | +| `path_style` | true | + +If `enable_signature_v4_streaming` is set to `true`, you may see the +following error: + +``` +STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported +``` + ### OpenStack compatible connection settings The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: |
