diff options
Diffstat (limited to 'doc/administration')
81 files changed, 5052 insertions, 0 deletions
diff --git a/doc/administration/customization/branded_login_page.md b/doc/administration/customization/branded_login_page.md new file mode 100644 index 00000000000..d4d9f5f7b5e --- /dev/null +++ b/doc/administration/customization/branded_login_page.md @@ -0,0 +1,19 @@ +# Changing the appearance of the login page + +GitLab Community Edition offers a way to put your company's identity on the login page of your GitLab server and make it a branded login page. + +By default, the page shows the GitLab logo and description. + + + +## Changing the appearance of the login page + +Navigate to the **Admin** area and go to the **Appearance** page. + +Fill in the required details like Title, Description and upload the company logo. + + + +After saving the page, your GitLab login page will have the details you filled in: + + diff --git a/doc/administration/customization/branded_login_page/appearance.png b/doc/administration/customization/branded_login_page/appearance.png Binary files differnew file mode 100644 index 00000000000..6bce1f0a287 --- /dev/null +++ b/doc/administration/customization/branded_login_page/appearance.png diff --git a/doc/administration/customization/branded_login_page/custom_sign_in.png b/doc/administration/customization/branded_login_page/custom_sign_in.png Binary files differnew file mode 100644 index 00000000000..d6020b029a2 --- /dev/null +++ b/doc/administration/customization/branded_login_page/custom_sign_in.png diff --git a/doc/administration/customization/branded_login_page/default_login_page.png b/doc/administration/customization/branded_login_page/default_login_page.png Binary files differnew file mode 100644 index 00000000000..795c7954d8e --- /dev/null +++ b/doc/administration/customization/branded_login_page/default_login_page.png diff --git a/doc/administration/customization/issue_closing.md b/doc/administration/customization/issue_closing.md new file mode 100644 index 00000000000..194b8e00299 --- /dev/null +++ b/doc/administration/customization/issue_closing.md @@ -0,0 +1,39 @@ +# Issue closing pattern + +When a commit or merge request resolves one or more issues, it is possible to automatically have these issues closed when the commit or merge request lands in the project's default branch. + +If a commit message or merge request description contains a sentence matching the regular expression below, all issues referenced from +the matched text will be closed. This happens when the commit is pushed to a project's default branch, or when a commit or merge request is merged into there. + +When not specified, the default `issue_closing_pattern` as shown below will be used: + +```bash +((?:[Cc]los(?:e[sd]?|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+) +``` + +Here, `%{issue_ref}` is a complex regular expression defined inside GitLab, that matches a reference to a local issue (`#123`), cross-project issue (`group/project#123`) or a link to an issue (`https://gitlab.example.com/group/project/issues/123`). + +For example: + +``` +git commit -m "Awesome commit message (Fix #20, Fixes #21 and Closes group/otherproject#22). This commit is also related to #17 and fixes #18, #19 and https://gitlab.example.com/group/otherproject/issues/23." +``` + +will close `#18`, `#19`, `#20`, and `#21` in the project this commit is pushed to, as well as `#22` and `#23` in group/otherproject. `#17` won't be closed as it does not match the pattern. It also works with multiline commit messages. + +Tip: you can test this closing pattern at [http://rubular.com][1]. Use this site +to test your own patterns. +Because Rubular doesn't understand `%{issue_ref}`, you can replace this by `#\d+` in testing, which matches only local issue references like `#123`. + +## Change the pattern + +For Omnibus installs you can change the default pattern in `/etc/gitlab/gitlab.rb`: + +``` +issue_closing_pattern: '((?:[Cc]los(?:e[sd]|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)' +``` + +For manual installs you can customize the pattern in [gitlab.yml][0] using the `issue_closing_pattern` key. + +[0]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example +[1]: http://rubular.com/r/Xmbexed1OJ diff --git a/doc/administration/customization/libravatar.md b/doc/administration/customization/libravatar.md new file mode 100644 index 00000000000..c46ce2ee203 --- /dev/null +++ b/doc/administration/customization/libravatar.md @@ -0,0 +1,82 @@ +# Use Libravatar service with GitLab + +GitLab by default supports [Gravatar](https://gravatar.com) avatar service. +Libravatar is a service which delivers your avatar (profile picture) to other websites and their API is +[heavily based on gravatar](https://wiki.libravatar.org/api/). + +This means that it is not complicated to switch to Libravatar avatar service or even self hosted Libravatar server. + +# Configuration + +In [gitlab.yml gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122) set +the configuration options as follows: + +## For HTTP + +```yml + gravatar: + enabled: true + # gravatar URLs: possible placeholders: %{hash} %{size} %{email} + plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + +## For HTTPS + +```yml + gravatar: + enabled: true + # gravatar URLs: possible placeholders: %{hash} %{size} %{email} + ssl_url: "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + +## Self-hosted + +If you are [running your own libravatar service](https://wiki.libravatar.org/running_your_own/) the URL will be different in the configuration +but the important part is to provide the same placeholders so GitLab can parse the URL correctly. + +For example, you host a service on `http://libravatar.example.com` the `plain_url` you need to supply in `gitlab.yml` is + +`http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` + + +## Omnibus-gitlab example + +In `/etc/gitlab/gitlab.rb`: + +#### For http + +```ruby +gitlab_rails['gravatar_enabled'] = true +gitlab_rails['gravatar_plain_url'] = "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + +#### For https + +```ruby +gitlab_rails['gravatar_enabled'] = true +gitlab_rails['gravatar_ssl_url'] = "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + + +Run `sudo gitlab-ctl reconfigure` for changes to take effect. + + +## Default URL for missing images + +[Libravatar supports different sets](https://wiki.libravatar.org/api/) of `missing images` for emails not found on the Libravatar service. + +In order to use a different set other than `identicon`, replace `&d=identicon` portion of the URL with another supported set. +For example, you can use `retro` set in which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` + + +## Usage examples + +#### For Microsoft Office 365 + +If your users are Office 365-users, the "GetPersonaPhoto" service can be used. Note that this service requires login, so this use case is +most useful in a corporate installation, where all users have access to Office 365. + +```ruby +gitlab_rails['gravatar_plain_url'] = 'http://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +gitlab_rails['gravatar_ssl_url'] = 'https://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +``` diff --git a/doc/administration/customization/welcome_message.md b/doc/administration/customization/welcome_message.md new file mode 100644 index 00000000000..a0cb234bea0 --- /dev/null +++ b/doc/administration/customization/welcome_message.md @@ -0,0 +1,12 @@ +# Customize the complete sign-in page + +Please see [Branded login page](branded_login_page.md) + +# Add a welcome message to the sign-in page (GitLab Community Edition) + +It is possible to add a markdown-formatted welcome message to your GitLab +sign-in page. Users of GitLab Enterprise Edition should use the [branded login +page feature](branded_login_page.md) instead. + +The welcome message (extra_sign_in_text) can now be set/changed in the Admin UI. +Admin area > Settings diff --git a/doc/administration/downgrade_ee_to_ce/README.md b/doc/administration/downgrade_ee_to_ce/README.md new file mode 100644 index 00000000000..3625c4191b8 --- /dev/null +++ b/doc/administration/downgrade_ee_to_ce/README.md @@ -0,0 +1,82 @@ +# Downgrading from EE to CE + +If you ever decide to downgrade your Enterprise Edition back to the Community +Edition, there are a few steps you need take before installing the CE package +on top of the current EE package, or, if you are in an installation from source, +before you change remotes and fetch the latest CE code. + +## Disable Enterprise-only features + +First thing to do is to disable the following features. + +### Authentication mechanisms + +Kerberos and Atlassian Crowd are only available on the Enterprise Edition, so +you should disable these mechanisms before downgrading and you should provide +alternative authentication methods to your users. + +### Git Annex + +Git Annex is also only available on the Enterprise Edition. This means that if +you have repositories that use Git Annex to store large files, these files will +no longer be easily available via Git. You should consider migrating these +repositories to use Git LFS before downgrading to the Community Edition. + +### Remove Jenkins CI Service entries from the database + +The `JenkinsService` class is only available on the Enterprise Edition codebase, +so if you downgrade to the Community Edition, you'll come across the following +error: + +``` +Completed 500 Internal Server Error in 497ms (ActiveRecord: 32.2ms) + +ActionView::Template::Error (The single-table inheritance mechanism failed to locate the subclass: 'JenkinsService'. This +error is raised because the column 'type' is reserved for storing the class in case of inheritance. Please rename this +column if you didn't intend it to be used for storing the inheritance class or overwrite Service.inheritance_column to +use another column for that information.) +``` + +All services are created automatically for every project you have, so in order +to avoid getting this error, you need to remove all instances of the +`JenkinsService` from your database: + +**Omnibus Installation** + +``` +$ sudo gitlab-rails runner "Service.where(type: 'JenkinsService').delete_all" +``` + +**Source Installation** + +``` +$ bundle exec rails runner "Service.where(type: 'JenkinsService').delete_all" production +``` + +## Downgrade to CE + +After performing the above mentioned steps, you are now ready to downgrade your +GitLab installation to the Community Edition. + +**Omnibus Installation** + +To downgrade an Omnibus installation, it is sufficient to install the Community +Edition package on top of the currently installed one. You can do this manually, +by directly [downloading the package](https://packages.gitlab.com/gitlab/gitlab-ce) +you need, or by adding our CE package repository and following the +[CE installation instructions](https://about.gitlab.com/downloads/). + +**Source Installation** + +To downgrade a source installation, you need to replace the current remote of +your GitLab installation with the Community Edition's remote, fetch the latest +changes, and checkout the latest stable branch: + +``` +$ git remote set-url origin git@gitlab.com:gitlab-org/gitlab-ce.git +$ git fetch --all +$ git checkout 8-x-stable +``` + +Remember to follow the correct [update guides](../update/README.md) to make +sure all dependencies are up to date. diff --git a/doc/administration/hooks/custom_hooks.md b/doc/administration/hooks/custom_hooks.md new file mode 100644 index 00000000000..820934f97f1 --- /dev/null +++ b/doc/administration/hooks/custom_hooks.md @@ -0,0 +1,41 @@ +# Custom Git Hooks + +**Note: Custom git hooks must be configured on the filesystem of the GitLab +server. Only GitLab server administrators will be able to complete these tasks. +Please explore [webhooks](../web_hooks/web_hooks.md) as an option if you do not have filesystem access. For a user configurable Git Hooks interface, please see [GitLab Enterprise Edition Git Hooks](http://docs.gitlab.com/ee/git_hooks/git_hooks.html).** + +Git natively supports hooks that are executed on different actions. +Examples of server-side git hooks include pre-receive, post-receive, and update. +See +[Git SCM Server-Side Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks) +for more information about each hook type. + +As of gitlab-shell version 2.2.0 (which requires GitLab 7.5+), GitLab +administrators can add custom git hooks to any GitLab project. + +## Setup + +Normally, git hooks are placed in the repository or project's `hooks` directory. +GitLab creates a symlink from each project's `hooks` directory to the +gitlab-shell `hooks` directory for ease of maintenance between gitlab-shell +upgrades. As such, custom hooks are implemented a little differently. Behavior +is exactly the same once the hook is created, though. Follow these steps to +set up a custom hook. + +1. Pick a project that needs a custom git hook. +1. On the GitLab server, navigate to the project's repository directory. +For an installation from source the path is usually +`/home/git/repositories/<group>/<project>.git`. For Omnibus installs the path is +usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. +1. Create a new directory in this location called `custom_hooks`. +1. Inside the new `custom_hooks` directory, create a file with a name matching +the hook type. For a pre-receive hook the file name should be `pre-receive` with +no extension. +1. Make the hook file executable and make sure it's owned by git. +1. Write the code to make the git hook function as expected. Hooks can be +in any language. Ensure the 'shebang' at the top properly reflects the language +type. For example, if the script is in Ruby the shebang will probably be +`#!/usr/bin/env ruby`. + +That's it! Assuming the hook code is properly implemented the hook will fire +as appropriate. diff --git a/doc/administration/incoming_email/README.md b/doc/administration/incoming_email/README.md new file mode 100644 index 00000000000..5a9a1582877 --- /dev/null +++ b/doc/administration/incoming_email/README.md @@ -0,0 +1,302 @@ +# Reply by email + +GitLab can be set up to allow users to comment on issues and merge requests by +replying to notification emails. + +## Requirement + +Reply by email requires an IMAP-enabled email account. GitLab allows you to use +three strategies for this feature: +- using email sub-addressing +- using a dedicated email address +- using a catch-all mailbox + +### Email sub-addressing + +**If your provider or server supports email sub-addressing, we recommend using it.** + +[Sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing) is +a feature where any email to `user+some_arbitrary_tag@example.com` will end up +in the mailbox for `user@example.com`, and is supported by providers such as +Gmail, Google Apps, Yahoo! Mail, Outlook.com and iCloud, as well as the Postfix +mail server which you can run on-premises. + +### Dedicated email address + +This solution is really simple to set up: you just have to create an email +address dedicated to receive your users' replies to GitLab notifications. + +### Catch-all mailbox + +A [catch-all mailbox](https://en.wikipedia.org/wiki/Catch-all) for a domain will +"catch all" the emails addressed to the domain that do not exist in the mail +server. + +## How it works? + +### 1. GitLab sends a notification email + +When GitLab sends a notification and Reply by email is enabled, the `Reply-To` +header is set to the address defined in your GitLab configuration, with the +`%{key}` placeholder (if present) replaced by a specific "reply key". In +addition, this "reply key" is also added to the `References` header. + +### 2. You reply to the notification email + +When you reply to the notification email, your email client will: + +- send the email to the `Reply-To` address it got from the notification email +- set the `In-Reply-To` header to the value of the `Message-ID` header from the + notification email +- set the `References` header to the value of the `Message-ID` plus the value of + the notification email's `References` header. + +### 3. GitLab receives your reply to the notification email + +When GitLab receives your reply, it will look for the "reply key" in the +following headers, in this order: + +1. the `To` header +1. the `References` header + +If it finds a reply key, it will be able to leave your reply as a comment on +the entity the notification was about (issue, merge request, commit...). + +For more details about the `Message-ID`, `In-Reply-To`, and `References headers`, +please consult [RFC 5322](https://tools.ietf.org/html/rfc5322#section-3.6.4). + +## Set it up + +If you want to use Gmail / Google Apps with Reply by email, make sure you have +[IMAP access enabled](https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665018) +and [allowed less secure apps to access the account](https://support.google.com/accounts/answer/6010255). + +To set up a basic Postfix mail server with IMAP access on Ubuntu, follow +[these instructions](./postfix.md). + +### Omnibus package installations + +1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the + feature and fill in the details for your specific IMAP server and email account: + + ```ruby + # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com + gitlab_rails['incoming_email_enabled'] = true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + gitlab_rails['incoming_email_address'] = "incoming+%{key}@gitlab.example.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + gitlab_rails['incoming_email_email'] = "incoming" + # Email account password + gitlab_rails['incoming_email_password'] = "[REDACTED]" + + # IMAP server host + gitlab_rails['incoming_email_host'] = "gitlab.example.com" + # IMAP server port + gitlab_rails['incoming_email_port'] = 143 + # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_ssl'] = false + # Whether the IMAP server uses StartTLS + gitlab_rails['incoming_email_start_tls'] = false + + # The mailbox where incoming mail will end up. Usually "inbox". + gitlab_rails['incoming_email_mailbox_name'] = "inbox" + ``` + + ```ruby + # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com + gitlab_rails['incoming_email_enabled'] = true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + gitlab_rails['incoming_email_address'] = "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + gitlab_rails['incoming_email_email'] = "gitlab-incoming@gmail.com" + # Email account password + gitlab_rails['incoming_email_password'] = "[REDACTED]" + + # IMAP server host + gitlab_rails['incoming_email_host'] = "imap.gmail.com" + # IMAP server port + gitlab_rails['incoming_email_port'] = 993 + # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_ssl'] = true + # Whether the IMAP server uses StartTLS + gitlab_rails['incoming_email_start_tls'] = false + + # The mailbox where incoming mail will end up. Usually "inbox". + gitlab_rails['incoming_email_mailbox_name'] = "inbox" + ``` + +1. Reconfigure GitLab and restart mailroom for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart mailroom + ``` + +1. Verify that everything is configured correctly: + + ```sh + sudo gitlab-rake gitlab:incoming_email:check + ``` + +1. Reply by email should now be working. + +### Installations from source + +1. Go to the GitLab installation directory: + + ```sh + cd /home/git/gitlab + ``` + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature + and fill in the details for your specific IMAP server and email account: + + ```sh + sudo editor config/gitlab.yml + ``` + + ```yaml + # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "incoming+%{key}@gitlab.example.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "incoming" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "gitlab.example.com" + # IMAP server port + port: 143 + # Whether the IMAP server uses SSL + ssl: false + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + ``` + + ```yaml + # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "gitlab-incoming@gmail.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "imap.gmail.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + ``` + +1. Enable `mail_room` in the init script at `/etc/default/gitlab`: + + ```sh + sudo mkdir -p /etc/default + echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab + ``` + +1. Restart GitLab: + + ```sh + sudo service gitlab restart + ``` + +1. Verify that everything is configured correctly: + + ```sh + sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production + ``` + +1. Reply by email should now be working. + +### Development + +1. Go to the GitLab installation directory. + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and fill in the details for your specific IMAP server and email account: + + ```yaml + # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "gitlab-incoming@gmail.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "imap.gmail.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + ``` + + As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. + +1. Uncomment the `mail_room` line in your `Procfile`: + + ```yaml + mail_room: bundle exec mail_room -q -c config/mail_room.yml + ``` + +1. Restart GitLab: + + ```sh + bundle exec foreman start + ``` + +1. Verify that everything is configured correctly: + + ```sh + bundle exec rake gitlab:incoming_email:check RAILS_ENV=development + ``` + +1. Reply by email should now be working. diff --git a/doc/administration/incoming_email/postfix.md b/doc/administration/incoming_email/postfix.md new file mode 100644 index 00000000000..787d21f7f8f --- /dev/null +++ b/doc/administration/incoming_email/postfix.md @@ -0,0 +1,321 @@ +# Set up Postfix for Reply by email + +This document will take you through the steps of setting up a basic Postfix mail server with IMAP authentication on Ubuntu, to be used with Reply by email. + +The instructions make the assumption that you will be using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. + +## Configure your server firewall + +1. Open up port 25 on your server so that people can send email into the server over SMTP. +2. If the mail server is different from the server running GitLab, open up port 143 on your server so that GitLab can read email from the server over IMAP. + +## Install packages + +1. Install the `postfix` package if it is not installed already: + + ```sh + sudo apt-get install postfix + ``` + + When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. + +1. Install the `mailutils` package. + + ```sh + sudo apt-get install mailutils + ``` + +## Create user + +1. Create a user for incoming email. + + ```sh + sudo useradd -m -s /bin/bash incoming + ``` + +1. Set a password for this user. + + ```sh + sudo passwd incoming + ``` + + Be sure not to forget this, you'll need it later. + +## Test the out-of-the-box setup + +1. Connect to the local SMTP server: + + ```sh + telnet localhost 25 + ``` + + You should see a prompt like this: + + ```sh + Trying 127.0.0.1... + Connected to localhost. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` + + If you get a `Connection refused` error instead, verify that `postfix` is running: + + ```sh + sudo postfix status + ``` + + If it is not, start it: + + ```sh + sudo postfix start + ``` + +1. Send the new `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + + ``` + ehlo localhost + mail from: root@localhost + rcpt to: incoming@localhost + data + Subject: Re: Some issue + + Sounds good! + . + quit + ``` + + _**Note:** The `.` is a literal period on its own line._ + + _**Note:** If you receive an error after entering `rcpt to: incoming@localhost` + then your Postfix `my_network` configuration is not correct. The error will + say 'Temporary lookup failure'. See + [Configure Postfix to receive email from the Internet](#configure-postfix-to-receive-email-from-the-internet)._ + +1. Check if the `incoming` user received the email: + + ```sh + su - incoming + mail + ``` + + You should see output like this: + + ``` + "/var/mail/incoming": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + +1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +## Configure Postfix to use Maildir-style mailboxes + +Courier, which we will install later to add IMAP authentication, requires mailboxes to have the Maildir format, rather than mbox. + +1. Configure Postfix to use Maildir-style mailboxes: + + ```sh + sudo postconf -e "home_mailbox = Maildir/" + ``` + +1. Restart Postfix: + + ```sh + sudo /etc/init.d/postfix restart + ``` + +1. Test the new setup: + + 1. Follow steps 1 and 2 of _[Test the out-of-the-box setup](#test-the-out-of-the-box-setup)_. + 1. Check if the `incoming` user received the email: + + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` + + You should see output like this: + + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + + _**Note:** If `mail` returns an error `Maildir: Is a directory` then your + version of `mail` doesn't support Maildir style mailboxes. Install + `heirloom-mailx` by running `sudo apt-get install heirloom-mailx`. Then, + try the above steps again, substituting `heirloom-mailx` for the `mail` + command._ + +1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +## Install the Courier IMAP server + +1. Install the `courier-imap` package: + + ```sh + sudo apt-get install courier-imap + ``` + +## Configure Postfix to receive email from the internet + +1. Let Postfix know about the domains that it should consider local: + + ```sh + sudo postconf -e "mydestination = gitlab.example.com, localhost.localdomain, localhost" + ``` + +1. Let Postfix know about the IPs that it should consider part of the LAN: + + We'll assume `192.168.1.0/24` is your local LAN. You can safely skip this step if you don't have other machines in the same local network. + + ```sh + sudo postconf -e "mynetworks = 127.0.0.0/8, 192.168.1.0/24" + ``` + +1. Configure Postfix to receive mail on all interfaces, which includes the internet: + + ```sh + sudo postconf -e "inet_interfaces = all" + ``` + +1. Configure Postfix to use the `+` delimiter for sub-addressing: + + ```sh + sudo postconf -e "recipient_delimiter = +" + ``` + +1. Restart Postfix: + + ```sh + sudo service postfix restart + ``` + +## Test the final setup + +1. Test SMTP under the new setup: + + 1. Connect to the SMTP server: + + ```sh + telnet gitlab.example.com 25 + ``` + + You should see a prompt like this: + + ```sh + Trying 123.123.123.123... + Connected to gitlab.example.com. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` + + If you get a `Connection refused` error instead, make sure your firewall is setup to allow inbound traffic on port 25. + + 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + + ``` + ehlo gitlab.example.com + mail from: root@gitlab.example.com + rcpt to: incoming@gitlab.example.com + data + Subject: Re: Some issue + + Sounds good! + . + quit + ``` + + (Note: The `.` is a literal period on its own line) + + 1. Check if the `incoming` user received the email: + + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` + + You should see output like this: + + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@gitlab.example.com 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + + 1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +1. Test IMAP under the new setup: + + 1. Connect to the IMAP server: + + ```sh + telnet gitlab.example.com 143 + ``` + + You should see a prompt like this: + + ```sh + Trying 123.123.123.123... + Connected to mail.example.gitlab.com. + Escape character is '^]'. + - OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. + ``` + + 1. Sign in as the `incoming` user to test IMAP, by entering the following into the IMAP prompt: + + ``` + a login incoming PASSWORD + ``` + + Replace PASSWORD with the password you set on the `incoming` user earlier. + + You should see output like this: + + ``` + a OK LOGIN Ok. + ``` + + 1. Disconnect from the IMAP server: + + ```sh + a logout + ``` + +## Done! + +If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [Reply by email](./README.md) guide to configure GitLab. + +--------- + +_This document was adapted from https://help.ubuntu.com/community/PostfixBasicSetupHowto, by contributors to the Ubuntu documentation wiki._ diff --git a/doc/administration/integration/README.md b/doc/administration/integration/README.md new file mode 100644 index 00000000000..fd330dd7a7d --- /dev/null +++ b/doc/administration/integration/README.md @@ -0,0 +1,61 @@ +# GitLab Integration + +GitLab integrates with multiple third-party services to allow external issue +trackers and external authentication. + +See the documentation below for details on how to configure these services. + +- [Jira](../project_services/jira.md) Integrate with the JIRA issue tracker +- [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. +- [LDAP](ldap.md) Set up sign in via LDAP +- [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, SAML, Crowd and Azure +- [SAML](saml.md) Configure GitLab as a SAML 2.0 Service Provider +- [CAS](cas.md) Configure GitLab to sign in using CAS +- [Slack](slack.md) Integrate with the Slack chat service +- [OAuth2 provider](oauth_provider.md) OAuth2 application creation +- [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages +- [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users +- [Akismet](akismet.md) Configure Akismet to stop spam + +GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. + +[jenkins]: http://docs.gitlab.com/ee/integration/jenkins.html + + +## Project services + +Integration with services such as Campfire, Flowdock, Gemnasium, HipChat, +Pivotal Tracker, and Slack are available in the form of a [Project Service][]. + +[Project Service]: ../project_services/project_services.md + +## SSL certificate errors + +When trying to integrate GitLab with services that are using self-signed certificates, +it is very likely that SSL certificate errors will occur on different parts of the +application, most likely Sidekiq. There are 2 approaches you can take to solve this: + +1. Add the root certificate to the trusted chain of the OS. +1. If using Omnibus, you can add the certificate to GitLab's trusted certificates. + +**OS main trusted chain** + +This [resource](http://kb.kerio.com/product/kerio-connect/server-configuration/ssl-certificates/adding-trusted-root-certificates-to-the-server-1605.html) +has all the information you need to add a certificate to the main trusted chain. + +This [answer](http://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu) +at SuperUser also has relevant information. + +**Omnibus Trusted Chain** + +It is enough to concatenate the certificate to the main trusted certificate: + +```bash +cat jira.pem >> /opt/gitlab/embedded/ssl/certs/cacert.pem +``` + +After that restart GitLab with: + +```bash +sudo gitlab-ctl restart +``` diff --git a/doc/administration/integration/akismet.md b/doc/administration/integration/akismet.md new file mode 100644 index 00000000000..5cc09bd536d --- /dev/null +++ b/doc/administration/integration/akismet.md @@ -0,0 +1,30 @@ +# Akismet + +GitLab leverages [Akismet](http://akismet.com) to protect against spam. Currently +GitLab uses Akismet to prevent users who are not members of a project from +creating spam via the GitLab API. Detected spam will be rejected, and +an entry in the "Spam Log" section in the Admin page will be created. + +Privacy note: GitLab submits the user's IP and user agent to Akismet. Note that +adding a user to a project will disable the Akismet check and prevent this +from happening. + +## Configuration + +To use Akismet: + +1. Go to the URL: https://akismet.com/account/ + +2. Sign-in or create a new account. + +3. Click on "Show" to reveal the API key. + +4. Go to Applications Settings on Admin Area (`admin/application_settings`) + +5. Check the `Enable Akismet` checkbox + +6. Fill in the API key from step 3. + +7. Save the configuration. + + diff --git a/doc/administration/integration/auth0.md b/doc/administration/integration/auth0.md new file mode 100644 index 00000000000..e5247082a89 --- /dev/null +++ b/doc/administration/integration/auth0.md @@ -0,0 +1,89 @@ +# Auth0 OmniAuth Provider + +To enable the Auth0 OmniAuth provider, you must create an Auth0 account, and an +application. + +1. Sign in to the [Auth0 Console](https://manage.auth0.com). If you need to +create an account, you can do so at the same link. + +1. Select "New App/API". + +1. Provide the Application Name ('GitLab' works fine). + +1. Once created, you should see the Quick Start options. Disregard them and +select 'Settings' above the Quick Start options. + +1. At the top of the Settings screen, you should see your Domain, Client ID and +Client Secret. Take note of these as you'll need to put them in the +configuration file. For example: + - Domain: `test1234.auth0.com` + - Client ID: `t6X8L2465bNePWLOvt9yi41i` + - Client Secret: `KbveM3nqfjwCbrhaUy_gDu2dss8TIlHIdzlyf33pB7dEK5u_NyQdp65O_o02hXs2` + +1. Fill in the Allowed Callback URLs: + - http://`YOUR_GITLAB_URL`/users/auth/auth0/callback (or) + - https://`YOUR_GITLAB_URL`/users/auth/auth0/callback + +1. Fill in the Allowed Origins (CORS): + - http://`YOUR_GITLAB_URL` (or) + - https://`YOUR_GITLAB_URL` + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) +for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "auth0", + "args" => { client_id: 'YOUR_AUTH0_CLIENT_ID'', + client_secret: 'YOUR_AUTH0_CLIENT_SECRET', + namespace: 'YOUR_AUTH0_DOMAIN' + } + } + ] + ``` + + For installations from source: + + ```yaml + - { name: 'auth0', + args: { + client_id: 'YOUR_AUTH0_CLIENT_ID', + client_secret: 'YOUR_AUTH0_CLIENT_SECRET', + namespace: 'YOUR_AUTH0_DOMAIN' + } + } + ``` + +1. Change `YOUR_AUTH0_CLIENT_ID` to the client ID from the Auth0 Console page +from step 5. + +1. Change `YOUR_AUTH0_CLIENT_SECRET` to the client secret from the Auth0 Console +page from step 5. + +1. Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) +for the changes to take effect. + +On the sign in page there should now be an Auth0 icon below the regular sign in +form. Click the icon to begin the authentication process. Auth0 will ask the +user to sign in and authorize the GitLab application. If everything goes well +the user will be returned to GitLab and will be signed in. diff --git a/doc/administration/integration/azure.md b/doc/administration/integration/azure.md new file mode 100644 index 00000000000..48dddf7df44 --- /dev/null +++ b/doc/administration/integration/azure.md @@ -0,0 +1,83 @@ +# Microsoft Azure OAuth2 OmniAuth Provider + +To enable the Microsoft Azure OAuth2 OmniAuth provider you must register your application with Azure. Azure will generate a client ID and secret key for you to use. + +1. Sign in to the [Azure Management Portal](https://manage.windowsazure.com>). + +1. Select "Active Directory" on the left and choose the directory you want to use to register GitLab. + +1. Select "Applications" at the top bar and click the "Add" button the bottom. + +1. Select "Add an application my organization is developing". + +1. Provide the project information and click the "Next" button. + - Name: 'GitLab' works just fine here. + - Type: 'WEB APPLICATION AND/OR WEB API' + +1. On the "App properties" page enter the needed URI's and click the "Complete" button. + - SIGN-IN URL: Enter the URL of your GitLab installation (e.g 'https://gitlab.mycompany.com/') + - APP ID URI: Enter the endpoint URL for Microsoft to use, just has to be unique (e.g 'https://mycompany.onmicrosoft.com/gitlab') + +1. Select "Configure" in the top menu. + +1. Add a "Reply URL" pointing to the Azure OAuth callback of your GitLab installation (e.g. https://gitlab.mycompany.com/users/auth/azure_oauth2/callback). + +1. Create a "Client secret" by selecting a duration, the secret will be generated as soon as you click the "Save" button in the bottom menu.. + +1. Note the "CLIENT ID" and the "CLIENT SECRET". + +1. Select "View endpoints" from the bottom menu. + +1. You will see lots of endpoint URLs in the form 'https://login.microsoftonline.com/TENANT ID/...', note down the TENANT ID part of one of those endpoints. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "azure_oauth2", + "args" => { + "client_id" => "CLIENT ID", + "client_secret" => "CLIENT SECRET", + "tenant_id" => "TENANT ID", + } + } + ] + ``` + + For installations from source: + + ``` + - { name: 'azure_oauth2', + args: { client_id: "CLIENT ID", + client_secret: "CLIENT SECRET", + tenant_id: "TENANT ID" } } + ``` + +1. Replace 'CLIENT ID', 'CLIENT SECRET' and 'TENANT ID' with the values you got above. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Microsoft icon below the regular sign in form. Click the icon to begin the authentication process. Microsoft will ask the user to sign in and authorize the GitLab application. If everything goes well the user will be returned to GitLab and will be signed in. diff --git a/doc/administration/integration/bitbucket.md b/doc/administration/integration/bitbucket.md new file mode 100644 index 00000000000..63432b04432 --- /dev/null +++ b/doc/administration/integration/bitbucket.md @@ -0,0 +1,140 @@ +# Integrate your server with Bitbucket + +Import projects from Bitbucket and login to your GitLab instance with your Bitbucket account. + +To enable the Bitbucket OmniAuth provider you must register your application with Bitbucket. +Bitbucket will generate an application ID and secret key for you to use. + +1. Sign in to Bitbucket. + +1. Navigate to your individual user settings or a team's settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or a team - that is entirely up to you. + +1. Select "OAuth" in the left menu. + +1. Select "Add consumer". + +1. Provide the required details. + - Name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. + - Application description: Fill this in if you wish. + - URL: The URL to your GitLab installation. 'https://gitlab.company.com' +1. Select "Save". + +1. You should now see a Key and Secret in the list of OAuth customers. + Keep this page open as you continue configuration. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "bitbucket", + "app_id" => "YOUR_KEY", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://bitbucket.org/" + } + ] + ``` + + For installation from source: + + ``` + - { name: 'bitbucket', app_id: 'YOUR_KEY', + app_secret: 'YOUR_APP_SECRET' } + ``` + +1. Change 'YOUR_APP_ID' to the key from the Bitbucket application page from step 7. + +1. Change 'YOUR_APP_SECRET' to the secret from the Bitbucket application page from step 7. + +1. Save the configuration file. + +1. If you're using the omnibus package, reconfigure GitLab (```gitlab-ctl reconfigure```). + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Bitbucket icon below the regular sign in form. +Click the icon to begin the authentication process. Bitbucket will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to GitLab and will be signed in. + +## Bitbucket project import + +To allow projects to be imported directly into GitLab, Bitbucket requires two extra setup steps compared to GitHub and GitLab.com. + +Bitbucket doesn't allow OAuth applications to clone repositories over HTTPS, and instead requires GitLab to use SSH and identify itself using your GitLab server's SSH key. + +### Step 1: Public key + +To be able to access repositories on Bitbucket, GitLab will automatically register your public key with Bitbucket as a deploy key for the repositories to be imported. Your public key needs to be at `~/.ssh/bitbucket_rsa.pub`, which will expand to `/home/git/.ssh/bitbucket_rsa.pub` in most configurations. + +If you have that file in place, you're all set and should see the "Import projects from Bitbucket" option enabled. If you don't, do the following: + +1. Create a new SSH key: + + ```sh + sudo -u git -H ssh-keygen + ``` + + When asked `Enter file in which to save the key` specify the correct path, eg. `/home/git/.ssh/bitbucket_rsa`. + Make sure to use an **empty passphrase**. + +1. Configure SSH client to use your new key: + + Open the SSH configuration file of the git user. + + ```sh + sudo editor /home/git/.ssh/config + ``` + + Add a host configuration for `bitbucket.org`. + + ```sh + Host bitbucket.org + IdentityFile ~/.ssh/bitbucket_rsa + User git + ``` + +### Step 2: Known hosts + +To allow GitLab to connect to Bitbucket over SSH, you need to add 'bitbucket.org' to your GitLab server's known SSH hosts. Take the following steps to do so: + +1. Manually connect to 'bitbucket.org' over SSH, while logged in as the `git` account that GitLab will use: + + ```sh + sudo -u git -H ssh bitbucket.org + ``` + +1. Verify the RSA key fingerprint you'll see in the response matches the one in the [Bitbucket documentation](https://confluence.atlassian.com/display/BITBUCKET/Use+the+SSH+protocol+with+Bitbucket#UsetheSSHprotocolwithBitbucket-KnownhostorBitbucket'spublickeyfingerprints) (the specific IP address doesn't matter): + + ```sh + The authenticity of host 'bitbucket.org (207.223.240.182)' can't be established. + RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40. + Are you sure you want to continue connecting (yes/no)? + ``` + +1. If the fingerprint matches, type `yes` to continue connecting and have 'bitbucket.org' be added to your known hosts. + +1. Your GitLab server is now able to connect to Bitbucket over SSH. + +1. Restart GitLab to allow it to find the new public key. + +You should now see the "Import projects from Bitbucket" option on the New Project page enabled. diff --git a/doc/administration/integration/cas.md b/doc/administration/integration/cas.md new file mode 100644 index 00000000000..e34e306f9ac --- /dev/null +++ b/doc/administration/integration/cas.md @@ -0,0 +1,65 @@ +# CAS OmniAuth Provider + +To enable the CAS OmniAuth provider you must register your application with your CAS instance. This requires the service URL GitLab will supply to CAS. It should be something like: `https://gitlab.example.com:443/users/auth/cas3/callback?url`. By default handling for SLO is enabled, you only need to configure CAS for backchannel logout. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name"=> "cas3", + "label"=> "cas", + "args"=> { + "url"=> 'CAS_SERVER', + "login_url"=> '/CAS_PATH/login', + "service_validate_url"=> '/CAS_PATH/p3/serviceValidate', + "logout_url"=> '/CAS_PATH/logout' + } + } + ] + ``` + + + For installations from source: + + ``` + - { name: 'cas3', + label: 'cas', + args: { + url: 'CAS_SERVER', + login_url: '/CAS_PATH/login', + service_validate_url: '/CAS_PATH/p3/serviceValidate', + logout_url: '/CAS_PATH/logout'} } + ``` + +1. Change 'CAS_PATH' to the root of your CAS instance (ie. `cas`). + +1. If your CAS instance does not use default TGC lifetimes, update the `cas3.session_duration` to at least the current TGC maximum lifetime. To explicitly disable SLO, regardless of CAS settings, set this to 0. + +1. Save the configuration file. + +1. Run `gitlab-ctl reconfigure` for the omnibus package. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a CAS tab in the sign in form. diff --git a/doc/administration/integration/crowd.md b/doc/administration/integration/crowd.md new file mode 100644 index 00000000000..40d93aef2a9 --- /dev/null +++ b/doc/administration/integration/crowd.md @@ -0,0 +1,58 @@ +# Crowd OmniAuth Provider + +To enable the Crowd OmniAuth provider you must register your application with Crowd. To configure Crowd integration you need an application name and password. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "crowd", + "args" => { + "crowd_server_url" => "CROWD", + "application_name" => "YOUR_APP_NAME", + "application_password" => "YOUR_APP_PASSWORD" + } + } + ] + ``` + + For installations from source: + + ``` + - { name: 'crowd', + args: { + crowd_server_url: 'CROWD SERVER URL', + application_name: 'YOUR_APP_NAME', + application_password: 'YOUR_APP_PASSWORD' } } + ``` + +1. Change 'YOUR_APP_NAME' to the application name from Crowd applications page. + +1. Change 'YOUR_APP_PASSWORD' to the application password you've set. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Crowd tab in the sign in form.
\ No newline at end of file diff --git a/doc/administration/integration/external-issue-tracker.md b/doc/administration/integration/external-issue-tracker.md new file mode 100644 index 00000000000..a2d7e922aad --- /dev/null +++ b/doc/administration/integration/external-issue-tracker.md @@ -0,0 +1,30 @@ +# External issue tracker + +GitLab has a great issue tracker but you can also use an external one such as +Jira or Redmine. Issue trackers are configurable per GitLab project and allow +you to do the following: + +- the **Issues** link on the GitLab project pages takes you to the appropriate + issue index of the external tracker +- clicking **New issue** on the project dashboard creates a new issue on the + external tracker + +## Configuration + +The configuration is done via a project's **Services**. + +### Project Service + +To enable an external issue tracker you must configure the appropriate **Service**. +Visit the links below for details: + +- [Redmine](../project_services/redmine.md) +- [Jira](../project_services/jira.md) + +### Service Template + +To save you the hassle from configuring each project's service individually, +GitLab provides the ability to set Service Templates which can then be +overridden in each project's settings. + +Read more on [Services Templates](../project_services/services_templates.md). diff --git a/doc/administration/integration/facebook.md b/doc/administration/integration/facebook.md new file mode 100644 index 00000000000..77bb75cbfca --- /dev/null +++ b/doc/administration/integration/facebook.md @@ -0,0 +1,97 @@ +# Facebook OAuth2 OmniAuth Provider + +To enable the Facebook OmniAuth provider you must register your application with Facebook. Facebook will generate an app ID and secret key for you to use. + +1. Sign in to the [Facebook Developer Platform](https://developers.facebook.com/). + +1. Choose "My Apps" > "Add a New App" + +1. Select the type "Website" + +1. Enter a name for your app. This can be anything. Consider something like "<Organization>'s GitLab" or "<Your Name>'s GitLab" or +something else descriptive. + +1. Choose "Create New Facebook App ID" + +1. Select a Category, for example "Productivity" + +1. Choose "Create App ID" + +1. Enter the address of your GitLab installation at the bottom of the package + +  + +1. Choose "Next" + +1. Choose "Skip Quick Start" in the upper right corner + +1. Choose "Settings" in the menu on the left + +1. Fill in a contact email for your app + +  + +1. Choose "Save Changes" + +1. Choose "Status & Review" in the menu on the left + +1. Change the switch on the right from No to Yes + +1. Choose "Confirm" when prompted to make the app public + +1. Choose "Dashboard" in the menu on the left + +1. Choose "Show" next to the hidden "App Secret" + +1. You should now see an app key and app secret (see screenshot). Keep this page open as you continue configuration. + +  + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "facebook", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET" + } + ] + ``` + + For installations from source: + + ``` + - { name: 'facebook', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET' } + ``` + +1. Change 'YOUR_APP_ID' to the API key from Facebook page in step 10. + +1. Change 'YOUR_APP_SECRET' to the API secret from the Facebook page in step 10. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Facebook icon below the regular sign in form. Click the icon to begin the authentication process. Facebook will ask the user to sign in and authorize the GitLab application. If everything goes well the user will be returned to GitLab and will be signed in. diff --git a/doc/administration/integration/github.md b/doc/administration/integration/github.md new file mode 100644 index 00000000000..e7497e475c9 --- /dev/null +++ b/doc/administration/integration/github.md @@ -0,0 +1,95 @@ +# Integrate your server with GitHub + +Import projects from GitHub and login to your GitLab instance with your GitHub account. + +To enable the GitHub OmniAuth provider you must register your application with GitHub. +GitHub will generate an application ID and secret key for you to use. + +1. Sign in to GitHub. + +1. Navigate to your individual user settings or an organization's settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or an organization - that is entirely up to you. + +1. Select "OAuth applications" in the left menu. + +1. If you already have applications listed, switch to the "Developer applications" tab. + +1. Select "Register new application". + +1. Provide the required details. + - Application name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. + - Homepage URL: The URL to your GitLab installation. 'https://gitlab.company.com' + - Application description: Fill this in if you wish. + - Default authorization callback URL is '${YOUR_DOMAIN}/import/github/callback' +1. Select "Register application". + +1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "github", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://github.com/", + "args" => { "scope" => "user:email" } + } + ] + ``` + + For installation from source: + + For GitHub.com: + + ``` + - { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'user:email' } } + ``` + + + For GitHub Enterprise: + + ``` + - { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + args: { scope: 'user:email' } } + ``` + + __Replace `https://github.example.com/` with your GitHub URL.__ + +1. Change 'YOUR_APP_ID' to the client ID from the GitHub application page from step 7. + +1. Change 'YOUR_APP_SECRET' to the client secret from the GitHub application page from step 7. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a GitHub icon below the regular sign in form. +Click the icon to begin the authentication process. GitHub will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to GitLab and will be signed in. diff --git a/doc/administration/integration/gitlab.md b/doc/administration/integration/gitlab.md new file mode 100644 index 00000000000..b215cc7c609 --- /dev/null +++ b/doc/administration/integration/gitlab.md @@ -0,0 +1,84 @@ +# Integrate your server with GitLab.com + +Import projects from GitLab.com and login to your GitLab instance with your GitLab.com account. + +To enable the GitLab.com OmniAuth provider you must register your application with GitLab.com. +GitLab.com will generate an application ID and secret key for you to use. + +1. Sign in to GitLab.com + +1. Navigate to your profile settings. + +1. Select "Applications" in the left menu. + +1. Select "New application". + +1. Provide the required details. + - Name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. + - Redirect URI: + + ``` + http://your-gitlab.example.com/import/gitlab/callback + http://your-gitlab.example.com/users/auth/gitlab/callback + ``` + + The first link is required for the importer and second for the authorization. + +1. Select "Submit". + +1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "gitlab", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "scope" => "api" } + } + ] + ``` + + For installations from source: + + ``` + - { name: 'gitlab', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'api' } } + ``` + +1. Change 'YOUR_APP_ID' to the Application ID from the GitLab.com application page. + +1. Change 'YOUR_APP_SECRET' to the secret from the GitLab.com application page. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a GitLab.com icon below the regular sign in form. +Click the icon to begin the authentication process. GitLab.com will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to your GitLab instance and will be signed in. diff --git a/doc/administration/integration/gmail_action_buttons_for_gitlab.md b/doc/administration/integration/gmail_action_buttons_for_gitlab.md new file mode 100644 index 00000000000..05a91d9bef9 --- /dev/null +++ b/doc/administration/integration/gmail_action_buttons_for_gitlab.md @@ -0,0 +1,22 @@ +# Gmail actions buttons for GitLab + +GitLab supports [Google actions in email](https://developers.google.com/gmail/markup/actions/actions-overview). + +If correctly setup, emails that require an action will be marked in Gmail. + + + +To get this functioning, you need to be registered with Google. +[See how to register with Google in this document.](https://developers.google.com/gmail/markup/registering-with-google) + +*This process has a lot of steps so make sure that you fulfill all requirements set by Google.* +*Your application will be rejected by Google if you fail to do so.* + +Pay close attention to: + +* Email account used by GitLab to send notification emails needs to have "Consistent history of sending a high volume of mail from your domain (order of hundred emails a day minimum to Gmail) for a few weeks at least". +* "A very very low rate of spam complaints from users." +* Emails must be authenticated via DKIM or SPF +* Before sending the final form("Gmail Schema Whitelist Request"), you must send a real email from your production server. This means that you will have to find a way to send this email from the email address you are registering. You can do this by, for example, forwarding the real email from the email address you are registering or going into the rails console on the GitLab server and triggering the email sending from there. + +You can check how it looks going through all the steps laid out in the "Registering with Google" doc in [this GitLab.com issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/1517). diff --git a/doc/administration/integration/google.md b/doc/administration/integration/google.md new file mode 100644 index 00000000000..82978b68a34 --- /dev/null +++ b/doc/administration/integration/google.md @@ -0,0 +1,89 @@ +# Google OAuth2 OmniAuth Provider + +To enable the Google OAuth2 OmniAuth provider you must register your application with Google. Google will generate a client ID and secret key for you to use. + +1. Sign in to the [Google Developers Console](https://console.developers.google.com/) with the Google account you want to use to register GitLab. + +1. Select "Create Project". + +1. Provide the project information + - Project name: 'GitLab' works just fine here. + - Project ID: Must be unique to all Google Developer registered applications. Google provides a randomly generated Project ID by default. You can use the randomly generated ID or choose a new one. +1. Refresh the page. You should now see your new project in the list. Click on the project. + +1. Select the "Google APIs" tab in the Overview. + +1. Select and enable the following Google APIs - listed under "Popular APIs" + - Enable `Contacts API` + - Enable `Google+ API` + +1. Select "Credentials" in the submenu. + +1. Select "Create New Client ID". + +1. Fill in the required information + - Application type: "Web Application" + - Authorized JavaScript origins: This isn't really used by GitLab but go ahead and put 'https://gitlab.example.com' here. + - Authorized redirect URI: 'https://gitlab.example.com/users/auth/google_oauth2/callback' +1. Under the heading "Client ID for web application" you should see a Client ID and Client secret (see screenshot). Keep this page open as you continue configuration.  + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "google_oauth2", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "access_type" => "offline", "approval_prompt" => '' } + } + ] + ``` + + For installations from source: + + ``` + - { name: 'google_oauth2', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { access_type: 'offline', approval_prompt: '' } } + ``` + +1. Change 'YOUR_APP_ID' to the client ID from the Google Developer page from step 10. + +1. Change 'YOUR_APP_SECRET' to the client secret from the Google Developer page from step 10. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Google icon below the regular sign in form. Click the icon to begin the authentication process. Google will ask the user to sign in and authorize the GitLab application. If everything goes well the user will be returned to GitLab and will be signed in. + +## Further Configuration + +This further configuration is not required for Google authentication to function but it is strongly recommended. Taking these steps will increase usability for users by providing a little more recognition and branding. + +At this point, when users first try to authenticate to your GitLab installation with Google they will see a generic application name on the prompt screen. The prompt informs the user that "Project Default Service Account" would like to access their account. "Project Default Service Account" isn't very recognizable and may confuse or cause users to be concerned. This is easily changeable. + +1. Select 'Consent screen' in the left menu. (See steps 1, 4 and 5 above for instructions on how to get here if you closed your window). +1. Scroll down until you find "Product Name". Change the product name to something more descriptive. +1. Add any additional information as you wish - homepage, logo, privacy policy, etc. None of this is required, but it may help your users. diff --git a/doc/administration/integration/img/akismet_settings.png b/doc/administration/integration/img/akismet_settings.png Binary files differnew file mode 100644 index 00000000000..ccdd3adb1c5 --- /dev/null +++ b/doc/administration/integration/img/akismet_settings.png diff --git a/doc/administration/integration/img/enabled-oauth-sign-in-sources.png b/doc/administration/integration/img/enabled-oauth-sign-in-sources.png Binary files differnew file mode 100644 index 00000000000..95f8bbdcd24 --- /dev/null +++ b/doc/administration/integration/img/enabled-oauth-sign-in-sources.png diff --git a/doc/administration/integration/img/facebook_api_keys.png b/doc/administration/integration/img/facebook_api_keys.png Binary files differnew file mode 100644 index 00000000000..d6c44ac0f11 --- /dev/null +++ b/doc/administration/integration/img/facebook_api_keys.png diff --git a/doc/administration/integration/img/facebook_app_settings.png b/doc/administration/integration/img/facebook_app_settings.png Binary files differnew file mode 100644 index 00000000000..30dd21e198a --- /dev/null +++ b/doc/administration/integration/img/facebook_app_settings.png diff --git a/doc/administration/integration/img/facebook_website_url.png b/doc/administration/integration/img/facebook_website_url.png Binary files differnew file mode 100644 index 00000000000..dc3088bb2fa --- /dev/null +++ b/doc/administration/integration/img/facebook_website_url.png diff --git a/doc/administration/integration/img/github_app.png b/doc/administration/integration/img/github_app.png Binary files differnew file mode 100644 index 00000000000..d890345ced9 --- /dev/null +++ b/doc/administration/integration/img/github_app.png diff --git a/doc/administration/integration/img/gitlab_app.png b/doc/administration/integration/img/gitlab_app.png Binary files differnew file mode 100644 index 00000000000..3f9391a821b --- /dev/null +++ b/doc/administration/integration/img/gitlab_app.png diff --git a/doc/administration/integration/img/gmail_action_buttons_for_gitlab.png b/doc/administration/integration/img/gmail_action_buttons_for_gitlab.png Binary files differnew file mode 100644 index 00000000000..b08f54d137b --- /dev/null +++ b/doc/administration/integration/img/gmail_action_buttons_for_gitlab.png diff --git a/doc/administration/integration/img/google_app.png b/doc/administration/integration/img/google_app.png Binary files differnew file mode 100644 index 00000000000..5a62ad35009 --- /dev/null +++ b/doc/administration/integration/img/google_app.png diff --git a/doc/administration/integration/img/oauth_provider_admin_application.png b/doc/administration/integration/img/oauth_provider_admin_application.png Binary files differnew file mode 100644 index 00000000000..a2d8e14c120 --- /dev/null +++ b/doc/administration/integration/img/oauth_provider_admin_application.png diff --git a/doc/administration/integration/img/oauth_provider_application_form.png b/doc/administration/integration/img/oauth_provider_application_form.png Binary files differnew file mode 100644 index 00000000000..3a676b22393 --- /dev/null +++ b/doc/administration/integration/img/oauth_provider_application_form.png diff --git a/doc/administration/integration/img/oauth_provider_application_id_secret.png b/doc/administration/integration/img/oauth_provider_application_id_secret.png Binary files differnew file mode 100644 index 00000000000..6d68df001af --- /dev/null +++ b/doc/administration/integration/img/oauth_provider_application_id_secret.png diff --git a/doc/administration/integration/img/oauth_provider_authorized_application.png b/doc/administration/integration/img/oauth_provider_authorized_application.png Binary files differnew file mode 100644 index 00000000000..efc3b807d71 --- /dev/null +++ b/doc/administration/integration/img/oauth_provider_authorized_application.png diff --git a/doc/administration/integration/img/oauth_provider_user_wide_applications.png b/doc/administration/integration/img/oauth_provider_user_wide_applications.png Binary files differnew file mode 100644 index 00000000000..45ad8a6d468 --- /dev/null +++ b/doc/administration/integration/img/oauth_provider_user_wide_applications.png diff --git a/doc/administration/integration/img/twitter_app_api_keys.png b/doc/administration/integration/img/twitter_app_api_keys.png Binary files differnew file mode 100644 index 00000000000..1076337172a --- /dev/null +++ b/doc/administration/integration/img/twitter_app_api_keys.png diff --git a/doc/administration/integration/img/twitter_app_details.png b/doc/administration/integration/img/twitter_app_details.png Binary files differnew file mode 100644 index 00000000000..b95e8af8a74 --- /dev/null +++ b/doc/administration/integration/img/twitter_app_details.png diff --git a/doc/administration/integration/jira.md b/doc/administration/integration/jira.md new file mode 100644 index 00000000000..78aa6634116 --- /dev/null +++ b/doc/administration/integration/jira.md @@ -0,0 +1,3 @@ +# GitLab JIRA integration + +This document was moved under [project_services/jira](../project_services/jira.md). diff --git a/doc/administration/integration/ldap.md b/doc/administration/integration/ldap.md new file mode 100644 index 00000000000..30f0c15dacc --- /dev/null +++ b/doc/administration/integration/ldap.md @@ -0,0 +1,3 @@ +# GitLab LDAP integration + +This document was moved under [`administration/auth/ldap`](../administration/auth/ldap.md). diff --git a/doc/administration/integration/oauth_provider.md b/doc/administration/integration/oauth_provider.md new file mode 100644 index 00000000000..5f8bb57365c --- /dev/null +++ b/doc/administration/integration/oauth_provider.md @@ -0,0 +1,80 @@ +# GitLab as OAuth2 authentication service provider + +This document is about using GitLab as an OAuth authentication service provider +to sign in to other services. + +If you want to use other OAuth authentication service providers to sign in to +GitLab, please see the [OAuth2 client documentation](../api/oauth2.md). + +## Introduction to OAuth + +[OAuth] provides to client applications a 'secure delegated access' to server +resources on behalf of a resource owner. In fact, OAuth allows an authorization +server to issue access tokens to third-party clients with the approval of the +resource owner, or the end-user. + +OAuth is mostly used as a Single Sign-On service (SSO), but you can find a +lot of different uses for this functionality. For example, you can allow users +to sign in to your application with their GitLab.com account, or GitLab.com +can be used for authentication to your GitLab instance +(see [GitLab OmniAuth](gitlab.md)). + +The 'GitLab Importer' feature is also using the OAuth protocol to give access +to repositories without sharing user credentials to your GitLab.com account. + +--- + +GitLab supports two ways of adding a new OAuth2 application to an instance. You +can either add an application as a regular user or add it in the admin area. +What this means is that GitLab can actually have instance-wide and a user-wide +applications. There is no difference between them except for the different +permission levels they are set (user/admin). + +## Adding an application through the profile + +In order to add a new application via your profile, navigate to +**Profile Settings > Applications** and select **New Application**. + + + +--- + +In the application form, enter a **Name** (arbitrary), and make sure to set up +correctly the **Redirect URI** which is the URL where users will be sent after +they authorize with GitLab. + + + +--- + +When you hit **Submit** you will be provided with the application ID and +the application secret which you can then use with your application that +connects to GitLab. + + + +--- + +## OAuth applications in the admin area + +To create an application that does not belong to a certain user, you can create +it from the admin area. + + + +--- + +## Authorized applications + +Every application you authorized to use your GitLab credentials will be shown +in the **Authorized applications** section under **Profile Settings > Applications**. + + + +--- + +As you can see, the default scope `api` is used, which is the only scope that +GitLab supports so far. At any time you can revoke any access by just clicking +**Revoke**. + +[oauth]: http://oauth.net/2/ "OAuth website" diff --git a/doc/administration/integration/omniauth.md b/doc/administration/integration/omniauth.md new file mode 100644 index 00000000000..820f40f81a9 --- /dev/null +++ b/doc/administration/integration/omniauth.md @@ -0,0 +1,208 @@ +# OmniAuth + +GitLab leverages OmniAuth to allow users to sign in using Twitter, GitHub, and +other popular services. + +Configuring OmniAuth does not prevent standard GitLab authentication or LDAP +(if configured) from continuing to work. Users can choose to sign in using any +of the configured mechanisms. + +- [Initial OmniAuth Configuration](#initial-omniauth-configuration) +- [Supported Providers](#supported-providers) +- [Enable OmniAuth for an Existing User](#enable-omniauth-for-an-existing-user) +- [OmniAuth configuration sample when using Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master#omniauth-google-twitter-github-login) +- [Enable or disable Sign In with an OmniAuth provider without disabling import sources](#enable-or-disable-sign-in-with-an-omniauth-provider-without-disabling-import-sources) + +## Supported Providers + +This is a list of the current supported OmniAuth providers. Before proceeding +on each provider's documentation, make sure to first read this document as it +contains some settings that are common for all providers. + +- [GitHub](github.md) +- [Bitbucket](bitbucket.md) +- [GitLab.com](gitlab.md) +- [Google](google.md) +- [Facebook](facebook.md) +- [Twitter](twitter.md) +- [Shibboleth](shibboleth.md) +- [SAML](saml.md) +- [Crowd](crowd.md) +- [Azure](azure.md) +- [Auth0](auth0.md) + +## Initial OmniAuth Configuration + +Before configuring individual OmniAuth providers there are a few global settings +that are in common for all providers that we need to consider. + +- Omniauth needs to be enabled, see details below for example. +- `allow_single_sign_on` allows you to specify the providers you want to allow to + automatically create an account. It defaults to `false`. If `false` users must + be created manually or they will not be able to sign in via OmniAuth. +- `block_auto_created_users` defaults to `true`. If `true` auto created users will + be blocked by default and will have to be unblocked by an administrator before + they are able to sign in. + +>**Note:** +If you set `block_auto_created_users` to `false`, make sure to only +define providers under `allow_single_sign_on` that you are able to control, like +SAML, Shibboleth, Crowd or Google, or set it to `false` otherwise any user on +the Internet will be able to successfully sign in to your GitLab without +administrative approval. + +To change these settings: + +* **For omnibus package** + + Open the configuration file: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + and change: + + ```ruby + gitlab_rails['omniauth_enabled'] = true + + # CAUTION! + # This allows users to login without having a user account first. Define the allowed providers + # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. + # User accounts will be created automatically when authentication was successful. + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml', 'twitter'] + gitlab_rails['omniauth_block_auto_created_users'] = true + ``` + +* **For installations from source** + + Open the configuration file: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + + and change the following section: + + ```yaml + ## OmniAuth settings + omniauth: + # Allow login via Twitter, Google, etc. using OmniAuth providers + enabled: true + + # CAUTION! + # This allows users to login without having a user account first. Define the allowed providers + # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. + # User accounts will be created automatically when authentication was successful. + allow_single_sign_on: ["saml", "twitter"] + + # Locks down those users until they have been cleared by the admin (default: true). + block_auto_created_users: true + ``` + +Now we can choose one or more of the Supported Providers listed above to continue +the configuration process. + +## Enable OmniAuth for an Existing User + +Existing users can enable OmniAuth for specific providers after the account is +created. For example, if the user originally signed in with LDAP, an OmniAuth +provider such as Twitter can be enabled. Follow the steps below to enable an +OmniAuth provider for an existing user. + +1. Sign in normally - whether standard sign in, LDAP, or another OmniAuth provider. +1. Go to profile settings (the silhouette icon in the top right corner). +1. Select the "Account" tab. +1. Under "Connected Accounts" select the desired OmniAuth provider, such as Twitter. +1. The user will be redirected to the provider. Once the user authorized GitLab + they will be redirected back to GitLab. + +The chosen OmniAuth provider is now active and can be used to sign in to GitLab from then on. + +## Configure OmniAuth Providers as External + +>**Note:** +This setting was introduced with version 8.7 of GitLab + +You can define which OmniAuth providers you want to be `external` so that all users +creating accounts via these providers will not be able to have access to internal +projects. You will need to use the full name of the provider, like `google_oauth2` +for Google. Refer to the examples for the full names of the supported providers. + +**For Omnibus installations** + +```ruby + gitlab_rails['omniauth_external_providers'] = ['twitter', 'google_oauth2'] +``` + +**For installations from source** + +```yaml + omniauth: + external_providers: ['twitter', 'google_oauth2'] +``` + +## Using Custom Omniauth Providers + +>**Note:** +The following information only applies for installations from source. + +GitLab uses [Omniauth](http://www.omniauth.org/) for authentication and already ships +with a few providers pre-installed (e.g. LDAP, GitHub, Twitter). But sometimes that +is not enough and you need to integrate with other authentication solutions. For +these cases you can use the Omniauth provider. + +### Steps + +These steps are fairly general and you will need to figure out the exact details +from the Omniauth provider's documentation. + +- Stop GitLab: + + sudo service gitlab stop + +- Add the gem to your [Gemfile](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile): + + gem "omniauth-your-auth-provider" + +- If you're using MySQL, install the new Omniauth provider gem by running the following command: + + sudo -u git -H bundle install --without development test postgres --path vendor/bundle --no-deployment + +- If you're using PostgreSQL, install the new Omniauth provider gem by running the following command: + + sudo -u git -H bundle install --without development test mysql --path vendor/bundle --no-deployment + + > These are the same commands you used in the [Install Gems section](#install-gems) with `--path vendor/bundle --no-deployment` instead of `--deployment`. + +- Start GitLab: + + sudo service gitlab start + +### Examples + +If you have successfully set up a provider that is not shipped with GitLab itself, +please let us know. + +You can help others by reporting successful configurations and probably share a +few insights or provide warnings for common errors or pitfalls by sharing your +experience [in the public Wiki](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Custom-omniauth-provider-configurations). + +While we can't officially support every possible authentication mechanism out there, +we'd like to at least help those with specific needs. + +## Enable or disable Sign In with an OmniAuth provider without disabling import sources + +>**Note:** +This setting was introduced with version 8.8 of GitLab. + +Administrators are able to enable or disable Sign In via some OmniAuth providers. + +>**Note:** +By default Sign In is enabled via all the OAuth Providers that have been configured in `config/gitlab.yml`. + +In order to enable/disable an OmniAuth provider, go to Admin Area -> Settings -> Sign-in Restrictions section -> Enabled OAuth Sign-In sources and select the providers you want to enable or disable. + + diff --git a/doc/administration/integration/recaptcha.md b/doc/administration/integration/recaptcha.md new file mode 100644 index 00000000000..a301d1a613c --- /dev/null +++ b/doc/administration/integration/recaptcha.md @@ -0,0 +1,23 @@ +# reCAPTCHA + +GitLab leverages [Google's reCAPTCHA](https://www.google.com/recaptcha/intro/index.html) +to protect against spam and abuse. GitLab displays the CAPTCHA form on the sign-up page +to confirm that a real user, not a bot, is attempting to create an account. + +## Configuration + +To use reCAPTCHA, first you must create a site and private key. + +1. Go to the URL: https://www.google.com/recaptcha/admin + +2. Fill out the form necessary to obtain reCAPTCHA keys. + +3. Login to your GitLab server, with administrator credentials. + +4. Go to Applications Settings on Admin Area (`admin/application_settings`) + +5. Fill all recaptcha fields with keys from previous steps + +6. Check the `Enable reCAPTCHA` checkbox + +7. Save the configuration. diff --git a/doc/administration/integration/saml.md b/doc/administration/integration/saml.md new file mode 100644 index 00000000000..8a7205caaa4 --- /dev/null +++ b/doc/administration/integration/saml.md @@ -0,0 +1,309 @@ +# SAML OmniAuth Provider + +GitLab can be configured to act as a SAML 2.0 Service Provider (SP). This allows +GitLab to consume assertions from a SAML 2.0 Identity Provider (IdP) such as +Microsoft ADFS to authenticate users. + +First configure SAML 2.0 support in GitLab, then register the GitLab application +in your SAML IdP: + +1. Make sure GitLab is configured with HTTPS. + See [Using HTTPS](../install/installation.md#using-https) for instructions. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) + for initial settings. + +1. To allow your users to use SAML to sign up without having to manually create + an account first, don't forget to add the following values to your configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] + gitlab_rails['omniauth_block_auto_created_users'] = false + ``` + + For installations from source: + + ```yaml + allow_single_sign_on: ["saml"] + block_auto_created_users: false + ``` + +1. You can also automatically link SAML users with existing GitLab users if their + email addresses match by adding the following setting: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_auto_link_saml_user'] = true + ``` + + For installations from source: + + ```yaml + auto_link_saml_user: true + ``` + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` + + For installations from source: + + ```yaml + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ``` + +1. Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint + of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab + installation to generate the correct value). + +1. Change the values of `idp_cert_fingerprint`, `idp_sso_target_url`, + `name_identifier_format` to match your IdP. Check + [the omniauth-saml documentation](https://github.com/omniauth/omniauth-saml) + for details on these options. + +1. Change the value of `issuer` to a unique name, which will identify the application + to the IdP. + +1. Restart GitLab for the changes to take effect. + +1. Register the GitLab SP in your SAML 2.0 IdP, using the application name specified + in `issuer`. + +To ease configuration, most IdP accept a metadata URL for the application to provide +configuration information to the IdP. To build the metadata URL for GitLab, append +`users/auth/saml/metadata` to the HTTPS URL of your GitLab installation, for instance: + ``` + https://gitlab.example.com/users/auth/saml/metadata + ``` + +At a minimum the IdP *must* provide a claim containing the user's email address, using +claim name `email` or `mail`. The email will be used to automatically generate the GitLab +username. GitLab will also use claims with name `name`, `first_name`, `last_name` +(see [the omniauth-saml gem](https://github.com/omniauth/omniauth-saml/blob/master/lib/omniauth/strategies/saml.rb) +for supported claims). + +On the sign in page there should now be a SAML button below the regular sign in form. +Click the icon to begin the authentication process. If everything goes well the user +will be returned to GitLab and will be signed in. + +## External Groups + +>**Note:** +This setting is only available on GitLab 8.7 and above. + +SAML login includes support for external groups. You can define in the SAML +settings which groups, to which your users belong in your IdP, you wish to be +marked as [external](../permissions/permissions.md). + +### Requirements + +First you need to tell GitLab where to look for group information. For this you +need to make sure that your IdP server sends a specific `AttributeStament` along +with the regular SAML response. Here is an example: + +```xml +<saml:AttributeStatement> + <saml:Attribute Name="Groups"> + <saml:AttributeValue xsi:type="xs:string">SecurityGroup</saml:AttributeValue> + <saml:AttributeValue xsi:type="xs:string">Developers</saml:AttributeValue> + <saml:AttributeValue xsi:type="xs:string">Designers</saml:AttributeValue> + </saml:Attribute> +</saml:AttributeStatement> +``` + +The name of the attribute can be anything you like, but it must contain the groups +to which a user belongs. In order to tell GitLab where to find these groups, you need +to add a `groups_attribute:` element to your SAML settings. You will also need to +tell GitLab which groups are external via the `external_groups:` element: + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + external_groups: ['Freelancers', 'Interns'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + +## Customization + +### `auto_sign_in_with_provider` + +You can add this setting to your GitLab configuration to automatically redirect you +to your SAML server for authentication, thus removing the need to click a button +before actually signing in. + +For omnibus package: + +```ruby +gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml' +``` + +For installations from source: + +```yaml +omniauth: + auto_sign_in_with_provider: saml +``` + +Please keep in mind that every sign in attempt will be redirected to the SAML server, +so you will not be able to sign in using local credentials. Make sure that at least one +of the SAML users has admin permissions. + +### `attribute_statements` + +>**Note:** +This setting is only available on GitLab 8.6 and above. +This setting should only be used to map attributes that are part of the +OmniAuth info hash schema. + +`attribute_statements` is used to map Attribute Names in a SAMLResponse to entries +in the OmniAuth [info hash](https://github.com/intridea/omniauth/wiki/Auth-Hash-Schema#schema-10-and-later). + +For example, if your SAMLResponse contains an Attribute called 'EmailAddress', +specify `{ email: ['EmailAddress'] }` to map the Attribute to the +corresponding key in the info hash. URI-named Attributes are also supported, e.g. +`{ email: ['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress'] }`. + +This setting allows you tell GitLab where to look for certain attributes required +to create an account. Like mentioned above, if your IdP sends the user's email +address as `EmailAddress` instead of `email`, let GitLab know by setting it on +your configuration: + +```yaml +args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient', + attribute_statements: { email: ['EmailAddress'] } +} +``` + +### `allowed_clock_drift` + +The clock of the Identity Provider may drift slightly ahead of your system clocks. +To allow for a small amount of clock drift you can use `allowed_clock_drift` within +your settings. Its value must be given in a number (and/or fraction) of seconds. +The value given is added to the current time at which the response is validated. + +```yaml +args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient', + attribute_statements: { email: ['EmailAddress'] }, + allowed_clock_drift: 1 # for one second clock drift +} +``` + +## Troubleshooting + +### 500 error after login + +If you see a "500 error" in GitLab when you are redirected back from the SAML sign in page, +this likely indicates that GitLab could not get the email address for the SAML user. + +Make sure the IdP provides a claim containing the user's email address, using claim name +`email` or `mail`. + +### Redirect back to login screen with no evident error + +If after signing in into your SAML server you are redirected back to the sign in page and +no error is displayed, check your `production.log` file. It will most likely contain the +message `Can't verify CSRF token authenticity`. This means that there is an error during +the SAML request, but this error never reaches GitLab due to the CSRF check. + +To bypass this you can add `skip_before_action :verify_authenticity_token` to the +`omniauth_callbacks_controller.rb` file. This will allow the error to hit GitLab, +where it can then be seen in the usual logs, or as a flash message in the login +screen. + +That file is located at `/opt/gitlab/embedded/service/gitlab-rails/app/controllers` +for Omnibus installations and by default on `/home/git/gitlab/app/controllers` for +installations from source. + +### Invalid audience + +This error means that the IdP doesn't recognize GitLab as a valid sender and +receiver of SAML requests. Make sure to add the GitLab callback URL to the approved +audiences of the IdP server. + +### Missing claims + +The IdP server needs to pass certain information in order for GitLab to either +create an account, or match the login information to an existing account. `email` +is the minimum amount of information that needs to be passed. If the IdP server +is not providing this information, all SAML requests will fail. + +Make sure this information is provided. + +### Key validation error, Digest mismatch or Fingerprint mismatch + +These errors all come from a similar place, the SAML certificate. SAML requests +need to be validated using a fingerprint, a certificate or a validator. + +For this you need take the following into account: + +- If no certificate is provided in the settings, a fingerprint or fingerprint + validator needs to be provided and the response from the server must contain + a certificate (`<ds:KeyInfo><ds:X509Data><ds:X509Certificate>`) +- If a certificate is provided in the settings, it is no longer necessary for + the request to contain one. In this case the fingerprint or fingerprint + validators are optional + +Make sure that one of the above described scenarios is valid, or the requests will +fail with one of the mentioned errors.
\ No newline at end of file diff --git a/doc/administration/integration/shibboleth.md b/doc/administration/integration/shibboleth.md new file mode 100644 index 00000000000..b6b2d4e5e88 --- /dev/null +++ b/doc/administration/integration/shibboleth.md @@ -0,0 +1,125 @@ +# Shibboleth OmniAuth Provider + +This documentation is for enabling shibboleth with omnibus-gitlab package. + +In order to enable Shibboleth support in gitlab we need to use Apache instead of Nginx (It may be possible to use Nginx, however I did not found way to easily configure Nginx that is bundled in omnibus-gitlab package). Apache uses mod_shib2 module for shibboleth authentication and can pass attributes as headers to omniauth-shibboleth provider. + + +To enable the Shibboleth OmniAuth provider you must: + +1. Configure Apache shibboleth module. Installation and configuration of module it self is out of scope of this document. +Check https://wiki.shibboleth.net/ for more info. + +1. You can find Apache config in gitlab-recipes (https://github.com/gitlabhq/gitlab-recipes/blob/master/web-server/apache/gitlab-ssl.conf) + +Following changes are needed to enable shibboleth: + +protect omniauth-shibboleth callback URL: +``` + <Location /users/auth/shibboleth/callback> + AuthType shibboleth + ShibRequestSetting requireSession 1 + ShibUseHeaders On + require valid-user + </Location> + + Alias /shibboleth-sp /usr/share/shibboleth + <Location /shibboleth-sp> + Satisfy any + </Location> + + <Location /Shibboleth.sso> + SetHandler shib + </Location> +``` +exclude shibboleth URLs from rewriting, add "RewriteCond %{REQUEST_URI} !/Shibboleth.sso" and "RewriteCond %{REQUEST_URI} !/shibboleth-sp", config should look like this: +``` + # Apache equivalent of Nginx try files + RewriteEngine on + RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA] + RequestHeader set X_FORWARDED_PROTO 'https' +``` + +1. Edit /etc/gitlab/gitlab.rb configuration file, your shibboleth attributes should be in form of "HTTP_ATTRIBUTE" and you should addjust them to your need and environment. Add any other configuration you need. + +File should look like this: +``` +external_url 'https://gitlab.example.com' +gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + +# disable Nginx +nginx['enable'] = false + +gitlab_rails['omniauth_allow_single_sign_on'] = true +gitlab_rails['omniauth_block_auto_created_users'] = false +gitlab_rails['omniauth_enabled'] = true +gitlab_rails['omniauth_providers'] = [ + { + "name" => 'shibboleth', + "args" => { + "shib_session_id_field" => "HTTP_SHIB_SESSION_ID", + "shib_application_id_field" => "HTTP_SHIB_APPLICATION_ID", + "uid_field" => 'HTTP_EPPN', + "name_field" => 'HTTP_CN', + "info_fields" => { "email" => 'HTTP_MAIL'} + } + } +] + +``` +1. Save changes and reconfigure gitlab: +``` +sudo gitlab-ctl reconfigure +``` + +On the sign in page there should now be a "Sign in with: Shibboleth" icon below the regular sign in form. Click the icon to begin the authentication process. You will be redirected to IdP server (Depends on your Shibboleth module configuration). If everything goes well the user will be returned to GitLab and will be signed in. + +## Apache 2.4 / GitLab 8.6 update +The order of the first 2 Location directives is important. If they are reversed, +you will not get a shibboleth session! + +``` + <Location /> + Require all granted + ProxyPassReverse http://127.0.0.1:8181 + ProxyPassReverse http://YOUR_SERVER_FQDN/ + </Location> + + <Location /users/auth/shibboleth/callback> + AuthType shibboleth + ShibRequestSetting requireSession 1 + ShibUseHeaders On + Require shib-session + </Location> + + Alias /shibboleth-sp /usr/share/shibboleth + + <Location /shibboleth-sp> + Require all granted + </Location> + + <Location /Shibboleth.sso> + SetHandler shib + </Location> + + RewriteEngine on + + #Don't escape encoded characters in api requests + RewriteCond %{REQUEST_URI} ^/api/v3/.* + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE] + + #Forward all requests to gitlab-workhorse except existing files + RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR] + RewriteCond %{REQUEST_URI} ^/uploads/.* + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA] + + RequestHeader set X_FORWARDED_PROTO 'https' + RequestHeader set X-Forwarded-Ssl on +```
\ No newline at end of file diff --git a/doc/administration/integration/slack.md b/doc/administration/integration/slack.md new file mode 100644 index 00000000000..f6ba80f46d5 --- /dev/null +++ b/doc/administration/integration/slack.md @@ -0,0 +1,41 @@ +# Slack integration + +## On Slack + +To enable Slack integration you must create an Incoming WebHooks integration on Slack: + +1. [Sign in to Slack](https://slack.com/signin) + +1. Visit [Incoming WebHooks](https://my.slack.com/services/new/incoming-webhook/) + +1. Choose the channel name you want to send notifications to. + +1. Click **Add Incoming WebHooks Integration** + - Optional step; You can change bot's name and avatar by clicking modifying the bot name or avatar under **Integration Settings**. + +1. Copy the **Webhook URL**, we'll need this later for GitLab. + + +## On GitLab + +After Slack is ready we need to setup GitLab. Here are the steps to achieve this. + +1. Sign in to GitLab + +1. Pick the repository you want. + +1. Navigate to Settings -> Services -> Slack + +1. Pick the triggers you want to activate + +1. Fill in your Slack details + - Webhook: Paste the Webhook URL from the step above + - Username: Fill this in if you want to change the username of the bot + - Channel: Fill this in if you want to change the channel where the messages will be posted + - Mark it as active + +1. Save your settings + +Have fun :) + +*P.S. You can set "branch,pushed,Compare changes" as highlight words on your Slack profile settings, so that you can be aware of new commits when somebody pushes them.* diff --git a/doc/administration/integration/twitter.md b/doc/administration/integration/twitter.md new file mode 100644 index 00000000000..4769f26b259 --- /dev/null +++ b/doc/administration/integration/twitter.md @@ -0,0 +1,79 @@ +# Twitter OAuth2 OmniAuth Provider + +To enable the Twitter OmniAuth provider you must register your application with Twitter. Twitter will generate a client ID and secret key for you to use. + +1. Sign in to [Twitter Application Management](https://apps.twitter.com/). + +1. Select "Create new app" + +1. Fill in the application details. + - Name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or + something else descriptive. + - Description: Create a description. + - Website: The URL to your GitLab installation. 'https://gitlab.example.com' + - Callback URL: 'https://gitlab.example.com/users/auth/twitter/callback' + - Agree to the "Developer Agreement". + +  +1. Select "Create your Twitter application." + +1. Select the "Settings" tab. + +1. Underneath the Callback URL check the box next to "Allow this application to be used to Sign in with Twitter." + +1. Select "Update settings" at the bottom to save changes. + +1. Select the "Keys and Access Tokens" tab. + +1. You should now see an API key and API secret (see screenshot). Keep this page open as you continue configuration. + +  + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "twitter", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET" + } + ] + ``` + + For installations from source: + + ``` + - { name: 'twitter', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET' } + ``` + +1. Change 'YOUR_APP_ID' to the API key from Twitter page in step 11. + +1. Change 'YOUR_APP_SECRET' to the API secret from the Twitter page in step 11. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Twitter icon below the regular sign in form. Click the icon to begin the authentication process. Twitter will ask the user to sign in and authorize the GitLab application. If everything goes well the user will be returned to GitLab and will be signed in. diff --git a/doc/administration/logs/logs.md b/doc/administration/logs/logs.md new file mode 100644 index 00000000000..a2eca62d691 --- /dev/null +++ b/doc/administration/logs/logs.md @@ -0,0 +1 @@ +This document was moved to [administration/logs.md](../administration/logs.md). diff --git a/doc/administration/migrate_ci_to_ce/README.md b/doc/administration/migrate_ci_to_ce/README.md new file mode 100644 index 00000000000..8f9ef054949 --- /dev/null +++ b/doc/administration/migrate_ci_to_ce/README.md @@ -0,0 +1,435 @@ +## Migrate GitLab CI to GitLab CE or EE + +Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise +Edition (EE), GitLab CI is no longer its own application, but is instead built +into the CE and EE applications. + +This guide will detail the process of migrating your CI installation and data +into your GitLab CE or EE installation. **You can only migrate CI data from +GitLab CI 8.0 to GitLab 8.0; migrating between other versions (e.g.7.14 to 8.1) +is not possible.** + +We recommend that you read through the entire migration process in this +document before beginning. + +### Overview + +In this document we assume you have a GitLab server and a GitLab CI server. It +does not matter if these are the same machine. + +The migration consists of three parts: updating GitLab and GitLab CI, moving +data, and redirecting traffic. + +Please note that CI builds triggered on your GitLab server in the time between +updating to 8.0 and finishing the migration will be lost. Your GitLab server +can be online for most of the procedure; the only GitLab downtime (if any) is +during the upgrade to 8.0. Your CI service will be offline from the moment you +upgrade to 8.0 until you finish the migration procedure. + +### Before upgrading + +If you have GitLab CI installed using omnibus-gitlab packages but **you don't want to migrate your existing data**: + +```bash +mv /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds.$(date +%s) +``` + +run `sudo gitlab-ctl reconfigure` and you can reach CI at `gitlab.example.com/ci`. + +If you want to migrate your existing data, continue reading. + +#### 0. Updating Omnibus from versions prior to 7.13 + +If you are updating from older versions you should first update to 7.14 and then to 8.0. +Otherwise it's pretty likely that you will encounter problems described in the [Troubleshooting](#troubleshooting). + +#### 1. Verify that backups work + +Make sure that the backup script on both servers can connect to the database. + +``` +# On your CI server: +# Omnibus +sudo chown gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds +sudo gitlab-ci-rake backup:create + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production +``` + +Also check on your GitLab server. + +``` +# On your GitLab server: +# Omnibus +sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads +``` + +If this fails you need to fix it before upgrading to 8.0. Also see +https://about.gitlab.com/getting-help/ + +#### 2. Check source and target database types + +Check what databases you use on your GitLab server and your CI server. + Look for the 'adapter:' line. If your CI server and your GitLab server use +the same database adapter no special care is needed. If your CI server uses +MySQL and your GitLab server uses PostgreSQL you need to pass a special option +during the 'Moving data' part. **If your CI server uses PostgreSQL and your +GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.** + +``` +# On your CI server: +# Omnibus +sudo gitlab-ci-rake env:info + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production +``` + +``` +# On your GitLab server: +# Omnibus +sudo gitlab-rake gitlab:env:info + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +#### 3. Storage planning + +Decide where to store CI build traces on GitLab server. GitLab CI uses + files on disk to store CI build traces. The default path for these build +traces is `/var/opt/gitlab/gitlab-ci/builds` (Omnibus) or +`/home/git/gitlab/builds` (Source). If you are storing your repository data in +a special location, or if you are using NFS, you should make sure that you +store build traces on the same storage as your Git repositories. + +### I. Upgrading + +From this point on, GitLab CI will be unavailable for your end users. + +#### 1. Upgrade GitLab to 8.0 + +First upgrade your GitLab server to version 8.0: +https://about.gitlab.com/update/ + +#### 2. Disable CI on the GitLab server during the migration + +After you update, go to the admin panel and temporarily disable CI. As + an administrator, go to **Admin Area** -> **Settings**, and under +**Continuous Integration** uncheck **Disable to prevent CI usage until rake +ci:migrate is run (8.0 only)**. + +#### 3. CI settings are now in GitLab + +If you want to use custom CI settings (e.g. change where builds are + stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or +`/home/git/gitlab/config/gitlab.yml` (Source). + +#### 4. Upgrade GitLab CI to 8.0 + +Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, + this may have already happened when you upgraded GitLab to 8.0. + +#### 5. Disable GitLab CI on the CI server + +Disable GitLab CI after upgrading to 8.0. + +``` +# On your CI server: +# Omnibus +sudo gitlab-ctl stop ci-unicorn +sudo gitlab-ctl stop ci-sidekiq + +# Source +sudo service gitlab_ci stop +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec whenever --clear-crontab RAILS_ENV=production +``` + +### II. Moving data + +#### 1. Database encryption key + +Move the database encryption key from your CI server to your GitLab + server. The command below will show you what you need to copy-paste to your +GitLab server. On Omnibus GitLab servers you will have to add a line to +`/etc/gitlab/gitlab.rb`. On GitLab servers installed from source you will have +to replace the contents of `/home/git/gitlab/config/secrets.yml`. + +``` +# On your CI server: +# Omnibus +sudo gitlab-ci-rake backup:show_secrets + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production +``` + +#### 2. SQL data and build traces + +Create your final CI data export. If you are converting from MySQL to + PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When +the command finishes it will print the path to your data export archive; you +will need this file later. + +``` +# On your CI server: +# Omnibus +sudo chown gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds +sudo gitlab-ci-rake backup:create + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production +``` + +#### 3. Copy data to the GitLab server + +If you were running GitLab and GitLab CI on the same server you can skip this +step. + +Copy your CI data archive to your GitLab server. There are many ways to do +this, below we use SSH agent forwarding and 'scp', which will be easy and fast +for most setups. You can also copy the data archive first from the CI server to +your laptop and then from your laptop to the GitLab server. + +``` +# Start from your laptop +ssh -A ci_admin@ci_server.example +# Now on the CI server +scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~ +``` + +#### 4. Move data to the GitLab backups folder + +Make the CI data archive discoverable for GitLab. We assume below that you +store backups in the default path, adjust the command if necessary. + +``` +# On your GitLab server: +# Omnibus +sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ + +# Source +sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ +``` + +#### 5. Import the CI data into GitLab. + +This step will delete any existing CI data on your GitLab server. There should +be no CI data yet because you turned CI on the GitLab server off earlier. + +``` +# On your GitLab server: +# Omnibus +sudo chown git:git /var/opt/gitlab/gitlab-ci/builds +sudo gitlab-rake ci:migrate + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production +``` + +#### 6. Restart GitLab + +``` +# On your GitLab server: +# Omnibus +sudo gitlab-ctl hup unicorn +sudo gitlab-ctl restart sidekiq + +# Source +sudo service gitlab reload +``` + +### III. Redirecting traffic + +If you were running GitLab CI with Omnibus packages and you were using the +internal NGINX configuration your CI service should now be available both at +`ci.example.com` (the old address) and `gitlab.example.com/ci`. **You are done!** + +If you installed GitLab CI from source we now need to configure a redirect in +NGINX so that existing CI runners can keep using the old CI server address, and +so that existing links to your CI server keep working. + +#### 1. Update Nginx configuration + +To ensure that your existing CI runners are able to communicate with the +migrated installation, and that existing build triggers still work, you'll need +to update your Nginx configuration to redirect requests for the old locations to +the new ones. + +Edit `/etc/nginx/sites-available/gitlab_ci` and paste: + +```nginx +# GITLAB CI +server { + listen 80 default_server; # e.g., listen 192.168.1.1:80; + server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; + + access_log /var/log/nginx/gitlab_ci_access.log; + error_log /var/log/nginx/gitlab_ci_error.log; + + # expose API to fix runners + location /api { + proxy_read_timeout 300; + proxy_connect_timeout 300; + proxy_redirect off; + proxy_set_header X-Real-IP $remote_addr; + + # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN + resolver 8.8.8.8 8.8.4.4; + proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # redirect all other CI requests + location / { + return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # adjust this to match the largest build log your runners might submit, + # set to 0 to disable limit + client_max_body_size 10m; +} +``` + +Make sure you substitute these placeholder values with your real ones: + +1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI + install (e.g., `ci.gitlab.com`). +1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab + CE (or EE) install (e.g., `gitlab.com`). + +**Make sure not to remove the `/ci$request_uri` part. This is required to +properly forward the requests.** + +You should also make sure that you can: + +1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server. +1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server. + +#### 2. Check Nginx configuration + + sudo nginx -t + +#### 3. Restart Nginx + + sudo /etc/init.d/nginx restart + +#### Restore from backup + +If something went wrong and you need to restore a backup, consult the [Backup +restoration](../raketasks/backup_restore.md) guide. + +### Troubleshooting + +#### show:secrets problem (Omnibus-only) +If you see errors like this: +``` +Missing `secret_key_base` or `db_key_base` for 'production' environment. The secrets will be generated and stored in `config/secrets.yml` +rake aborted! +Errno::EACCES: Permission denied @ rb_sysopen - config/secrets.yml +``` + +This can happen if you are updating from versions prior to 7.13 straight to 8.0. +The fix for this is to update to Omnibus 7.14 first and then update it to 8.0. + +#### Permission denied when accessing /var/opt/gitlab/gitlab-ci/builds +To fix that issue you have to change builds/ folder permission before doing final backup: +``` +sudo chown -R gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds +``` + +Then before executing `ci:migrate` you need to fix builds folder permission: +``` +sudo chown git:git /var/opt/gitlab/gitlab-ci/builds +``` + +#### Problems when importing CI database to GitLab +If you were migrating CI database from MySQL to PostgreSQL manually you can see errors during import about missing sequences: +``` +ALTER SEQUENCE +ERROR: relation "ci_builds_id_seq" does not exist +ERROR: relation "ci_commits_id_seq" does not exist +ERROR: relation "ci_events_id_seq" does not exist +ERROR: relation "ci_jobs_id_seq" does not exist +ERROR: relation "ci_projects_id_seq" does not exist +ERROR: relation "ci_runner_projects_id_seq" does not exist +ERROR: relation "ci_runners_id_seq" does not exist +ERROR: relation "ci_services_id_seq" does not exist +ERROR: relation "ci_taggings_id_seq" does not exist +ERROR: relation "ci_tags_id_seq" does not exist +CREATE TABLE +``` + +To fix that you need to apply this SQL statement before doing final backup: +``` +# Omnibus +gitlab-ci-rails dbconsole <<EOF +-- ALTER TABLES - DROP DEFAULTS +ALTER TABLE ONLY ci_application_settings ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_builds ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_commits ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_events ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_jobs ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_projects ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_runner_projects ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_runners ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_services ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_taggings ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_tags ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_trigger_requests ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_triggers ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_variables ALTER COLUMN id DROP DEFAULT; +ALTER TABLE ONLY ci_web_hooks ALTER COLUMN id DROP DEFAULT; + +-- ALTER SEQUENCES +ALTER SEQUENCE ci_application_settings_id_seq OWNED BY ci_application_settings.id; +ALTER SEQUENCE ci_builds_id_seq OWNED BY ci_builds.id; +ALTER SEQUENCE ci_commits_id_seq OWNED BY ci_commits.id; +ALTER SEQUENCE ci_events_id_seq OWNED BY ci_events.id; +ALTER SEQUENCE ci_jobs_id_seq OWNED BY ci_jobs.id; +ALTER SEQUENCE ci_projects_id_seq OWNED BY ci_projects.id; +ALTER SEQUENCE ci_runner_projects_id_seq OWNED BY ci_runner_projects.id; +ALTER SEQUENCE ci_runners_id_seq OWNED BY ci_runners.id; +ALTER SEQUENCE ci_services_id_seq OWNED BY ci_services.id; +ALTER SEQUENCE ci_taggings_id_seq OWNED BY ci_taggings.id; +ALTER SEQUENCE ci_tags_id_seq OWNED BY ci_tags.id; +ALTER SEQUENCE ci_trigger_requests_id_seq OWNED BY ci_trigger_requests.id; +ALTER SEQUENCE ci_triggers_id_seq OWNED BY ci_triggers.id; +ALTER SEQUENCE ci_variables_id_seq OWNED BY ci_variables.id; +ALTER SEQUENCE ci_web_hooks_id_seq OWNED BY ci_web_hooks.id; + +-- ALTER TABLES - RE-APPLY DEFAULTS +ALTER TABLE ONLY ci_application_settings ALTER COLUMN id SET DEFAULT nextval('ci_application_settings_id_seq'::regclass); +ALTER TABLE ONLY ci_builds ALTER COLUMN id SET DEFAULT nextval('ci_builds_id_seq'::regclass); +ALTER TABLE ONLY ci_commits ALTER COLUMN id SET DEFAULT nextval('ci_commits_id_seq'::regclass); +ALTER TABLE ONLY ci_events ALTER COLUMN id SET DEFAULT nextval('ci_events_id_seq'::regclass); +ALTER TABLE ONLY ci_jobs ALTER COLUMN id SET DEFAULT nextval('ci_jobs_id_seq'::regclass); +ALTER TABLE ONLY ci_projects ALTER COLUMN id SET DEFAULT nextval('ci_projects_id_seq'::regclass); +ALTER TABLE ONLY ci_runner_projects ALTER COLUMN id SET DEFAULT nextval('ci_runner_projects_id_seq'::regclass); +ALTER TABLE ONLY ci_runners ALTER COLUMN id SET DEFAULT nextval('ci_runners_id_seq'::regclass); +ALTER TABLE ONLY ci_services ALTER COLUMN id SET DEFAULT nextval('ci_services_id_seq'::regclass); +ALTER TABLE ONLY ci_taggings ALTER COLUMN id SET DEFAULT nextval('ci_taggings_id_seq'::regclass); +ALTER TABLE ONLY ci_tags ALTER COLUMN id SET DEFAULT nextval('ci_tags_id_seq'::regclass); +ALTER TABLE ONLY ci_trigger_requests ALTER COLUMN id SET DEFAULT nextval('ci_trigger_requests_id_seq'::regclass); +ALTER TABLE ONLY ci_triggers ALTER COLUMN id SET DEFAULT nextval('ci_triggers_id_seq'::regclass); +ALTER TABLE ONLY ci_variables ALTER COLUMN id SET DEFAULT nextval('ci_variables_id_seq'::regclass); +ALTER TABLE ONLY ci_web_hooks ALTER COLUMN id SET DEFAULT nextval('ci_web_hooks_id_seq'::regclass); +EOF + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rails dbconsole production <<EOF +... COPY SQL STATEMENTS FROM ABOVE ... +EOF +``` diff --git a/doc/administration/monitoring/health_check.md b/doc/administration/monitoring/health_check.md new file mode 100644 index 00000000000..0d17799372f --- /dev/null +++ b/doc/administration/monitoring/health_check.md @@ -0,0 +1,66 @@ +# Health Check + +>**Note:** This feature was [introduced][ce-3888] in GitLab 8.8. + +GitLab provides a health check endpoint for uptime monitoring on the `health_check` web +endpoint. The health check reports on the overall system status based on the status of +the database connection, the state of the database migrations, and the ability to write +and access the cache. This endpoint can be provided to uptime monitoring services like +[Pingdom][pingdom], [Nagios][nagios-health], and [NewRelic][newrelic-health]. + +## Access Token + +An access token needs to be provided while accessing the health check endpoint. The current +accepted token can be found on the `admin/health_check` page of your GitLab instance. + + + +The access token can be passed as a URL parameter: + +``` +https://gitlab.example.com/health_check.json?token=ACCESS_TOKEN +``` + +or as an HTTP header: + +```bash +curl -H "TOKEN: ACCESS_TOKEN" https://gitlab.example.com/health_check.json +``` + +## Using the Endpoint + +Once you have the access token, health information can be retrieved as plain text, JSON, +or XML using the `health_check` endpoint: + +- `https://gitlab.example.com/health_check?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check.json?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check.xml?token=ACCESS_TOKEN` + +You can also ask for the status of specific services: + +- `https://gitlab.example.com/health_check/cache.json?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check/database.json?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check/migrations.json?token=ACCESS_TOKEN` + +For example, the JSON output of the following health check: + +```bash +curl -H "TOKEN: ACCESS_TOKEN" https://gitlab.example.com/health_check.json +``` + +would be like: + +``` +{"healthy":true,"message":"success"} +``` + +## Status + +On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint +will return a valid successful HTTP status code, and a `success` message. Ideally your +uptime monitoring should look for the success message. + +[ce-3888]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3888 +[pingdom]: https://www.pingdom.com +[nagios-health]: https://nagios-plugins.org/doc/man/check_http.html +[newrelic-health]: https://docs.newrelic.com/docs/alerts/alert-policies/downtime-alerts/availability-monitoring diff --git a/doc/administration/monitoring/img/health_check_token.png b/doc/administration/monitoring/img/health_check_token.png Binary files differnew file mode 100644 index 00000000000..2daf8606b00 --- /dev/null +++ b/doc/administration/monitoring/img/health_check_token.png diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md new file mode 100644 index 00000000000..771584268d9 --- /dev/null +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -0,0 +1,40 @@ +# GitLab Configuration + +GitLab Performance Monitoring is disabled by default. To enable it and change any of its +settings, navigate to the Admin area in **Settings > Metrics** +(`/admin/application_settings`). + +The minimum required settings you need to set are the InfluxDB host and port. +Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the +changes. + +--- + + + +--- + +Finally, a restart of all GitLab processes is required for the changes to take +effect: + +```bash +# For Omnibus installations +sudo gitlab-ctl restart + +# For installations from source +sudo service gitlab restart +``` + +## Pending Migrations + +When any migrations are pending, the metrics are disabled until the migrations +have been performed. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md new file mode 100644 index 00000000000..168bd85c26a --- /dev/null +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -0,0 +1,149 @@ +# Grafana Configuration + +[Grafana](http://grafana.org/) is a tool that allows you to visualize time +series metrics through graphs and dashboards. It supports several backend +data stores, including InfluxDB. GitLab writes performance data to InfluxDB +and Grafana will allow you to query InfluxDB to display useful graphs. + +For the easiest installation and configuration, install Grafana on the same +server as InfluxDB. For larger installations, you may want to split out these +services. + +## Installation + +Grafana supplies package repositories (Yum/Apt) for easy installation. +See [Grafana installation documentation](http://docs.grafana.org/installation/) +for detailed steps. + +> **Note**: Before starting Grafana for the first time, set the admin user +and password in `/etc/grafana/grafana.ini`. Otherwise, the default password +will be `admin`. + +## Configuration + +Login as the admin user. Expand the menu by clicking the Grafana logo in the +top left corner. Choose 'Data Sources' from the menu. Then, click 'Add new' +in the top bar. + + + +Fill in the configuration details for the InfluxDB data source. Save and +Test Connection to ensure the configuration is correct. + +- **Name**: InfluxDB +- **Default**: Checked +- **Type**: InfluxDB 0.9.x (Even if you're using InfluxDB 0.10.x) +- **Url**: https://localhost:8086 (Or the remote URL if you've installed InfluxDB +on a separate server) +- **Access**: proxy +- **Database**: gitlab +- **User**: admin (Or the username configured when setting up InfluxDB) +- **Password**: The password configured when you set up InfluxDB + + + +## Apply retention policies and create continuous queries + +If you intend to import the GitLab provided Grafana dashboards, you will need +to copy and run a set of queries against InfluxDB to create the needed data +sets. + +On the InfluxDB server, run the following command, substituting your InfluxDB +user and password: + +```bash +influxdb --username admin -password super_secret +``` + +This will drop you in to an InfluxDB interactive session. Copy the entire +contents below and paste it in to the interactive session: + +``` +CREATE RETENTION POLICY default ON gitlab DURATION 1h REPLICATION 1 DEFAULT +CREATE RETENTION POLICY downsampled ON gitlab DURATION 7d REPLICATION 1 +CREATE CONTINUOUS QUERY grape_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_git_timings_per_action FROM gitlab."default".rails_method_calls WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.grape_markdown_render_timings_overall FROM gitlab."default".rails_transactions WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.grape_markdown_render_timings_per_action FROM gitlab."default".rails_transactions WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_markdown_timings_overall FROM gitlab."default".rails_method_calls WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_method_call_timings_per_action_and_method FROM gitlab."default".rails_method_calls WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY grape_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_method_call_timings_per_method FROM gitlab."default".rails_method_calls WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY grape_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.grape_transaction_counts_overall FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.grape_transaction_counts_per_action FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.grape_transaction_timings_overall FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.grape_transaction_timings_per_action FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_file_descriptor_counts ON gitlab BEGIN SELECT sum(value) AS count INTO gitlab.downsampled.rails_file_descriptor_counts FROM gitlab."default".rails_file_descriptors GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_gc_counts ON gitlab BEGIN SELECT sum(count) AS total, sum(minor_gc_count) AS minor, sum(major_gc_count) AS major INTO gitlab.downsampled.rails_gc_counts FROM gitlab."default".rails_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_gc_timings ON gitlab BEGIN SELECT mean(total_time) AS duration_mean, percentile(total_time, 95) AS duration_95th, percentile(total_time, 99) AS duration_99th INTO gitlab.downsampled.rails_gc_timings FROM gitlab."default".rails_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_git_timings_per_action FROM gitlab."default".rails_method_calls WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.rails_markdown_render_timings_overall FROM gitlab."default".rails_transactions WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.rails_markdown_render_timings_per_action FROM gitlab."default".rails_transactions WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_markdown_timings_overall FROM gitlab."default".rails_method_calls WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_memory_usage_overall ON gitlab BEGIN SELECT mean(value) AS memory_mean, percentile(value, 95) AS memory_95th, percentile(value, 99) AS memory_99th INTO gitlab.downsampled.rails_memory_usage_overall FROM gitlab."default".rails_memory_usage GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_method_call_timings_per_action_and_method FROM gitlab."default".rails_method_calls WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY rails_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_method_call_timings_per_method FROM gitlab."default".rails_method_calls WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY rails_object_counts_overall ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.rails_object_counts_overall FROM gitlab."default".rails_object_counts GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_object_counts_per_type ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.rails_object_counts_per_type FROM gitlab."default".rails_object_counts GROUP BY time(1m), type END; +CREATE CONTINUOUS QUERY rails_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.rails_transaction_counts_overall FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.rails_transaction_counts_per_action FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.rails_transaction_timings_overall FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.rails_transaction_timings_per_action FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_view_timings_per_action_and_view ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_view_timings_per_action_and_view FROM gitlab."default".rails_views WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action, view END; +CREATE CONTINUOUS QUERY sidekiq_file_descriptor_counts ON gitlab BEGIN SELECT sum(value) AS count INTO gitlab.downsampled.sidekiq_file_descriptor_counts FROM gitlab."default".sidekiq_file_descriptors GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_gc_counts ON gitlab BEGIN SELECT sum(count) AS total, sum(minor_gc_count) AS minor, sum(major_gc_count) AS major INTO gitlab.downsampled.sidekiq_gc_counts FROM gitlab."default".sidekiq_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_gc_timings ON gitlab BEGIN SELECT mean(total_time) AS duration_mean, percentile(total_time, 95) AS duration_95th, percentile(total_time, 99) AS duration_99th INTO gitlab.downsampled.sidekiq_gc_timings FROM gitlab."default".sidekiq_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_git_timings_per_action FROM gitlab."default".sidekiq_method_calls WHERE method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.sidekiq_markdown_render_timings_overall FROM gitlab."default".sidekiq_transactions WHERE (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.sidekiq_markdown_render_timings_per_action FROM gitlab."default".sidekiq_transactions WHERE (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_markdown_timings_overall FROM gitlab."default".sidekiq_method_calls WHERE method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_memory_usage_overall ON gitlab BEGIN SELECT mean(value) AS memory_mean, percentile(value, 95) AS memory_95th, percentile(value, 99) AS memory_99th INTO gitlab.downsampled.sidekiq_memory_usage_overall FROM gitlab."default".sidekiq_memory_usage GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_method_call_timings_per_action_and_method FROM gitlab."default".sidekiq_method_calls GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_method_call_timings_per_method FROM gitlab."default".sidekiq_method_calls GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY sidekiq_object_counts_overall ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.sidekiq_object_counts_overall FROM gitlab."default".sidekiq_object_counts GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_object_counts_per_type ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.sidekiq_object_counts_per_type FROM gitlab."default".sidekiq_object_counts GROUP BY time(1m), type END; +CREATE CONTINUOUS QUERY sidekiq_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.sidekiq_transaction_counts_overall FROM gitlab."default".sidekiq_transactions GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.sidekiq_transaction_counts_per_action FROM gitlab."default".sidekiq_transactions GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.sidekiq_transaction_timings_overall FROM gitlab."default".sidekiq_transactions GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.sidekiq_transaction_timings_per_action FROM gitlab."default".sidekiq_transactions GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_view_timings_per_action_and_view ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_view_timings_per_action_and_view FROM gitlab."default".sidekiq_views GROUP BY time(1m), action, view END; +CREATE CONTINUOUS QUERY web_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.web_transaction_counts_overall FROM gitlab."default".rails_transactions GROUP BY time(1m) END; +``` + +## Import Dashboards + +You can now import a set of default dashboards that will give you a good +start on displaying useful information. GitLab has published a set of default +[Grafana dashboards][grafana-dashboards] to get you started. Clone the +repository or download a zip/tarball, then follow these steps to import each +JSON file. + +Open the dashboard dropdown menu and click 'Import' + + + +Click 'Choose file' and browse to the location where you downloaded or cloned +the dashboard repository. Pick one of the JSON files to import. + + + +Once the dashboard is imported, be sure to click save icon in the top bar. If +you do not save the dashboard after importing it will be removed when you +navigate away. + + + +Repeat this process for each dashboard you wish to import. + +Alternatively you can automatically import all the dashboards into your Grafana +instance. See the README of the [Grafana dashboards][grafana-dashboards] +repository for more information on this process. + +[grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Installation/Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png b/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png Binary files differnew file mode 100644 index 00000000000..b4448c7a09f --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_import.png b/doc/administration/monitoring/performance/img/grafana_dashboard_import.png Binary files differnew file mode 100644 index 00000000000..5a2d3c0937a --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_dashboard_import.png diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png b/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png Binary files differnew file mode 100644 index 00000000000..7e2e111f570 --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_empty.png b/doc/administration/monitoring/performance/img/grafana_data_source_empty.png Binary files differnew file mode 100644 index 00000000000..11e27571e64 --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_data_source_empty.png diff --git a/doc/administration/monitoring/performance/img/grafana_save_icon.png b/doc/administration/monitoring/performance/img/grafana_save_icon.png Binary files differnew file mode 100644 index 00000000000..3d4265bee8e --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_save_icon.png diff --git a/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png b/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png Binary files differnew file mode 100644 index 00000000000..14d82b6ac98 --- /dev/null +++ b/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png diff --git a/doc/administration/monitoring/performance/influxdb_configuration.md b/doc/administration/monitoring/performance/influxdb_configuration.md new file mode 100644 index 00000000000..c30cd2950d8 --- /dev/null +++ b/doc/administration/monitoring/performance/influxdb_configuration.md @@ -0,0 +1,193 @@ +# InfluxDB Configuration + +The default settings provided by [InfluxDB] are not sufficient for a high traffic +GitLab environment. The settings discussed in this document are based on the +settings GitLab uses for GitLab.com, depending on your own needs you may need to +further adjust them. + +If you are intending to run InfluxDB on the same server as GitLab, make sure +you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. + +Unless you are going with a budget setup, it's advised to run it separately. + +## Requirements + +- InfluxDB 0.9.5 or newer +- A fairly modern version of Linux +- At least 4GB of RAM +- At least 10GB of storage for InfluxDB data + +Note that the RAM and storage requirements can differ greatly depending on the +amount of data received/stored. To limit the amount of stored data users can +look into [InfluxDB Retention Policies][influxdb-retention]. + +## Installation + +Installing InfluxDB is out of the scope of this document. Please refer to the +[InfluxDB documentation]. + +## InfluxDB Server Settings + +Since InfluxDB has many settings that users may wish to customize themselves +(e.g. what port to run InfluxDB on), we'll only cover the essentials. + +The configuration file in question is usually located at +`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, +InfluxDB needs to be restarted. + +### Storage Engine + +InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new +storage engine is available, called [TSM Tree]. All users **must** use the new +`tsm1` storage engine as this [will be the default engine][tsm1-commit] in +upcoming InfluxDB releases. + +Make sure you have the following in your configuration file: + +``` +[data] + dir = "/var/lib/influxdb/data" + engine = "tsm1" +``` + +### Admin Panel + +Production environments should have the InfluxDB admin panel **disabled**. This +feature can be disabled by adding the following to your InfluxDB configuration +file: + +``` +[admin] + enabled = false +``` + +### HTTP + +HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, +thus it should be enabled. When enabling make sure to _also_ enable +authentication: + +``` +[http] + enabled = true + auth-enabled = true +``` + +_**Note:** Before you enable authentication, you might want to [create an +admin user](#create-a-new-admin-user)._ + +### UDP + +GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling +UDP can be done using the following settings: + +``` +[[udp]] + enabled = true + bind-address = ":8089" + database = "gitlab" + batch-size = 1000 + batch-pending = 5 + batch-timeout = "1s" + read-buffer = 209715200 +``` + +This does the following: + +1. Enable UDP and bind it to port 8089 for all addresses. +2. Store any data received in the "gitlab" database. +3. Define a batch of points to be 1000 points in size and allow a maximum of + 5 batches _or_ flush them automatically after 1 second. +4. Define a UDP read buffer size of 200 MB. + +One of the most important settings here is the UDP read buffer size as if this +value is set too low, packets will be dropped. You must also make sure the OS +buffer size is set to the same value, the default value is almost never enough. + +To set the OS buffer size to 200 MB, on Linux you can run the following command: + +```bash +sysctl -w net.core.rmem_max=209715200 +``` + +To make this permanent, add the following to `/etc/sysctl.conf` and restart the +server: + +```bash +net.core.rmem_max=209715200 +``` + +It is **very important** to make sure the buffer sizes are large enough to +handle all data sent to InfluxDB as otherwise you _will_ lose data. The above +buffer sizes are based on the traffic for GitLab.com. Depending on the amount of +traffic, users may be able to use a smaller buffer size, but we highly recommend +using _at least_ 100 MB. + +When enabling UDP, users should take care to not expose the port to the public, +as doing so will allow anybody to write data into your InfluxDB database (as +[InfluxDB's UDP protocol][udp] doesn't support authentication). We recommend either +whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only +allowing traffic from members of said VLAN. + +## Create a new admin user + +If you want to [enable authentication](#http), you might want to [create an +admin user][influx-admin]: + +``` +influx -execute "CREATE USER jeff WITH PASSWORD '1234' WITH ALL PRIVILEGES" +``` + +## Create the `gitlab` database + +Once you get InfluxDB up and running, you need to create a database for GitLab. +Make sure you have changed the [storage engine](#storage-engine) to `tsm1` +before creating a database. + +_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled +[HTTP authentication](#http), remember to append the username (`-username <username>`) +and password (`-password <password>`) you set earlier to the commands below._ + +Run the following command to create a database named `gitlab`: + +```bash +influx -execute 'CREATE DATABASE gitlab' +``` + +The name **must** be `gitlab`, do not use any other name. + +Next, make sure that the database was successfully created: + +```bash +influx -execute 'SHOW DATABASES' +``` + +The output should be similar to: + +``` +name: databases +--------------- +name +_internal +gitlab +``` + +That's it! Now your GitLab instance should send data to InfluxDB. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md) + +[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management +[influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ +[influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ +[udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ +[tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d +[influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user diff --git a/doc/administration/monitoring/performance/influxdb_schema.md b/doc/administration/monitoring/performance/influxdb_schema.md new file mode 100644 index 00000000000..41861860b6d --- /dev/null +++ b/doc/administration/monitoring/performance/influxdb_schema.md @@ -0,0 +1,88 @@ +# InfluxDB Schema + +The following measurements are currently stored in InfluxDB: + +- `PROCESS_file_descriptors` +- `PROCESS_gc_statistics` +- `PROCESS_memory_usage` +- `PROCESS_method_calls` +- `PROCESS_object_counts` +- `PROCESS_transactions` +- `PROCESS_views` + +Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the +process type. In all series, any form of duration is stored in milliseconds. + +## PROCESS_file_descriptors + +This measurement contains the number of open file descriptors over time. The +value field `value` contains the number of descriptors. + +## PROCESS_gc_statistics + +This measurement contains Ruby garbage collection statistics such as the amount +of minor/major GC runs (relative to the last sampling interval), the time spent +in garbage collection cycles, and all fields/values returned by `GC.stat`. + +## PROCESS_memory_usage + +This measurement contains the process' memory usage (in bytes) over time. The +value field `value` contains the number of bytes. + +## PROCESS_method_calls + +This measurement contains the methods called during a transaction along with +their duration, and a name of the transaction action that invoked the method (if +available). The method call duration is stored in the value field `duration`, +while the method name is stored in the tag `method`. The tag `action` contains +the full name of the transaction action. Both the `method` and `action` fields +are in the following format: + +``` +ClassName#method_name +``` + +For example, a method called by the `show` method in the `UsersController` class +would have `action` set to `UsersController#show`. + +## PROCESS_object_counts + +This measurement is used to store retained Ruby objects (per class) and the +amount of retained objects. The number of objects is stored in the `count` value +field while the class name is stored in the `type` tag. + +## PROCESS_transactions + +This measurement is used to store basic transaction details such as the time it +took to complete a transaction, how much time was spent in SQL queries, etc. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The total duration of the transaction | +| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | +| `method_duration` | The total time spent in method calls | +| `sql_duration` | The total time spent in SQL queries | +| `view_duration` | The total time spent in views | + +## PROCESS_views + +This measurement is used to store view rendering timings for a transaction. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The rendering time of the view | +| `view` | The path of the view, relative to the application's root directory | + +The `action` tag contains the action name of the transaction that rendered the +view. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/introduction.md b/doc/administration/monitoring/performance/introduction.md new file mode 100644 index 00000000000..79904916b7e --- /dev/null +++ b/doc/administration/monitoring/performance/introduction.md @@ -0,0 +1,65 @@ +# GitLab Performance Monitoring + +GitLab comes with its own application performance measuring system as of GitLab +8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the +Community and Enterprise editions. + +Apart from this introduction, you are advised to read through the following +documents in order to understand and properly configure GitLab Performance Monitoring: + +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Install/Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md) + +## Introduction to GitLab Performance Monitoring + +GitLab Performance Monitoring makes it possible to measure a wide variety of statistics +including (but not limited to): + +- The time it took to complete a transaction (a web request or Sidekiq job). +- The time spent in running SQL queries and rendering HAML views. +- The time spent executing (instrumented) Ruby methods. +- Ruby object allocations, and retained objects in particular. +- System statistics such as the process' memory usage and open file descriptors. +- Ruby garbage collection statistics. + +Metrics data is written to [InfluxDB][influxdb] over [UDP][influxdb-udp]. Stored +data can be visualized using [Grafana][grafana] or any other application that +supports reading data from InfluxDB. Alternatively data can be queried using the +InfluxDB CLI. + +## Metric Types + +Two types of metrics are collected: + +1. Transaction specific metrics. +1. Sampled metrics, collected at a certain interval in a separate thread. + +### Transaction Metrics + +Transaction metrics are metrics that can be associated with a single +transaction. This includes statistics such as the transaction duration, timings +of any executed SQL queries, time spent rendering HAML views, etc. These metrics +are collected for every Rack request and Sidekiq job processed. + +### Sampled Metrics + +Sampled metrics are metrics that can't be associated with a single transaction. +Examples include garbage collection statistics and retained Ruby objects. These +metrics are collected at a regular interval. This interval is made up out of two +parts: + +1. A user defined interval. +1. A randomly generated offset added on top of the interval, the same offset + can't be used twice in a row. + +The actual interval can be anywhere between a half of the defined interval and a +half above the interval. For example, for a user defined interval of 15 seconds +the actual interval can be anywhere between 7.5 and 22.5. The interval is +re-generated for every sampling run instead of being generated once and re-used +for the duration of the process' lifetime. + +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[grafana]: http://grafana.org/ diff --git a/doc/administration/operations/README.md b/doc/administration/operations/README.md new file mode 100644 index 00000000000..6a35dab7b6c --- /dev/null +++ b/doc/administration/operations/README.md @@ -0,0 +1,5 @@ +# GitLab operations + +- [Sidekiq MemoryKiller](sidekiq_memory_killer.md) +- [Cleaning up Redis sessions](cleaning_up_redis_sessions.md) +- [Understanding Unicorn and unicorn-worker-killer](unicorn.md) diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md new file mode 100644 index 00000000000..93521e976d5 --- /dev/null +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -0,0 +1,52 @@ +# Cleaning up stale Redis sessions + +Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. +Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If +you have been running a large GitLab server (thousands of users) since before +GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis +database after you upgrade to GitLab 7.3. You can also perform a cleanup while +still running GitLab 7.2 or older, but in that case new stale sessions will +start building up again after you clean up. + +In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte +hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with +GitLab 7.3.0, the keys are +prefixed with 'session:gitlab:', so they would look like +'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to +remove the keys in the old format. + +First we define a shell function with the proper Redis connection details. + +``` +rcli() { + # This example works for Omnibus installations of GitLab 7.3 or newer. For an + # installation from source you will have to change the socket path and the + # path to redis-cli. + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" +} + +# test the new shell function; the response should be PONG +rcli ping +``` + +Now we do a search to see if there are any session keys in the old format for +us to clean up. + +``` +# returns the number of old-format session keys in Redis +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l +``` + +If the number is larger than zero, you can proceed to expire the keys from +Redis. If the number is zero there is nothing to clean up. + +``` +# Tell Redis to expire each matched key after 600 seconds. +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli +# This will print '(integer) 1' for each key that gets expired. +``` + +Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis +background save interval) your Redis database will be compacted. If you are +still using GitLab 7.2, users who are not clicking around in GitLab during the +10 minute expiry window will be signed out of GitLab. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md new file mode 100644 index 00000000000..54adb99386a --- /dev/null +++ b/doc/administration/operations/moving_repositories.md @@ -0,0 +1,180 @@ +# Moving repositories managed by GitLab + +Sometimes you need to move all repositories managed by GitLab to +another filesystem or another server. In this document we will look +at some of the ways you can copy all your repositories from +`/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`. + +We will look at three scenarios: the target directory is empty, the +target directory contains an outdated copy of the repositories, and +how to deal with thousands of repositories. + +**Each of the approaches we list can/will overwrite data in the +target directory `/mnt/gitlab/repositories`. Do not mix up the +source and the target.** + +## Target directory is empty: use a tar pipe + +If the target directory `/mnt/gitlab/repositories` is empty the +simplest thing to do is to use a tar pipe. This method has low +overhead and tar is almost always already installed on your system. +However, it is not possible to resume an interrupted tar pipe: if +that happens then all data must be copied again. + +``` +# As the git user +tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ + tar -C /mnt/gitlab/repositories -xf - +``` + +If you want to see progress, replace `-xf` with `-xvf`. + +### Tar pipe to another server + +You can also use a tar pipe to copy data to another server. If your +'git' user has SSH access to the newserver as 'git@newserver', you +can pipe the data through SSH. + +``` +# As the git user +tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ + ssh git@newserver tar -C /mnt/gitlab/repositories -xf - +``` + +If you want to compress the data before it goes over the network +(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`. + +## The target directory contains an outdated copy of the repositories: use rsync + +If the target directory already contains a partial / outdated copy +of the repositories it may be wasteful to copy all the data again +with tar. In this scenario it is better to use rsync. This utility +is either already installed on your system or easily installable +via apt, yum etc. + +``` +# As the 'git' user +rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ + /mnt/gitlab/repositories +``` + +The `/.` in the command above is very important, without it you can +easily get the wrong directory structure in the target directory. +If you want to see progress, replace `-a` with `-av`. + +### Single rsync to another server + +If the 'git' user on your source system has SSH access to the target +server you can send the repositories over the network with rsync. + +``` +# As the 'git' user +rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ + git@newserver:/mnt/gitlab/repositories +``` + +## Thousands of Git repositories: use one rsync per repository + +Every time you start an rsync job it has to inspect all files in +the source directory, all files in the target directory, and then +decide what files to copy or not. If the source or target directory +has many contents this startup phase of rsync can become a burden +for your GitLab server. In cases like this you can make rsync's +life easier by dividing its work in smaller pieces, and sync one +repository at a time. + +In addition to rsync we will use [GNU +Parallel](http://www.gnu.org/software/parallel/). This utility is +not included in GitLab so you need to install it yourself with apt +or yum. Also note that the GitLab scripts we used below were added +in GitLab 8.1. + +** This process does not clean up repositories at the target location that no +longer exist at the source. ** If you start using your GitLab instance with +`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos` +after switching to the new repository storage directory. + +### Parallel rsync for all repositories known to GitLab + +This will sync repositories with 10 rsync processes at a time. We keep +track of progress so that the transfer can be restarted if necessary. + +First we create a new directory, owned by 'git', to hold transfer +logs. We assume the directory is empty before we start the transfer +procedure, and that we are the only ones writing files in it. + +``` +# Omnibus +sudo mkdir /var/opt/gitlab/transfer-logs +sudo chown git:git /var/opt/gitlab/transfer-logs + +# Source +sudo -u git -H mkdir /home/git/transfer-logs +``` + +We seed the process with a list of the directories we want to copy. + +``` +# Omnibus +sudo -u git sh -c 'gitlab-rake gitlab:list_repos > /var/opt/gitlab/transfer-logs/all-repos-$(date +%s).txt' + +# Source +cd /home/git/gitlab +sudo -u git -H sh -c 'bundle exec rake gitlab:list_repos > /home/git/transfer-logs/all-repos-$(date +%s).txt' +``` + +Now we can start the transfer. The command below is idempotent, and +the number of jobs done by GNU Parallel should converge to zero. If it +does not some repositories listed in all-repos-1234.txt may have been +deleted/renamed before they could be copied. + +``` +# Omnibus +sudo -u git sh -c ' +cat /var/opt/gitlab/transfer-logs/* | sort | uniq -u |\ + /usr/bin/env JOBS=10 \ + /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ + /var/opt/gitlab/transfer-logs/success-$(date +%s).log \ + /var/opt/gitlab/git-data/repositories \ + /mnt/gitlab/repositories +' + +# Source +cd /home/git/gitlab +sudo -u git -H sh -c ' +cat /home/git/transfer-logs/* | sort | uniq -u |\ + /usr/bin/env JOBS=10 \ + bin/parallel-rsync-repos \ + /home/git/transfer-logs/success-$(date +%s).log \ + /home/git/repositories \ + /mnt/gitlab/repositories +` +``` + +### Parallel rsync only for repositories with recent activity + +Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. +Then you might only want to sync repositories that were changed via GitLab +_after_ that time. You can use the 'SINCE' variable to tell 'rake +gitlab:list_repos' to only print repositories with recent activity. + +``` +# Omnibus +sudo gitlab-rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ + sudo -u git \ + /usr/bin/env JOBS=10 \ + /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ + success-$(date +%s).log \ + /var/opt/gitlab/git-data/repositories \ + /mnt/gitlab/repositories + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ + sudo -u git -H \ + /usr/bin/env JOBS=10 \ + bin/parallel-rsync-repos \ + success-$(date +%s).log \ + /home/git/repositories \ + /mnt/gitlab/repositories +``` diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md new file mode 100644 index 00000000000..b5e78348989 --- /dev/null +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -0,0 +1,40 @@ +# Sidekiq MemoryKiller + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +restarts Unicorn worker processes in between requests when needed. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + +Unlike unicorn-worker-killer, which is enabled by default for all GitLab +installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default +_only_ for Omnibus packages. The reason for this is that the MemoryKiller +relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use Runit or an equivalent. + +With the default settings, the MemoryKiller will cause a Sidekiq restart no +more often than once every 15 minutes, with the restart causing about one +minute of delay for incoming background jobs. + +## Configuring the MemoryKiller + +The MemoryKiller is controlled using environment variables. + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is + greater than 0, then after each Sidekiq job, the MemoryKiller will check the + RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq + process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a + delayed shutdown is triggered. The default value for Omnibus packages is set + [in the omnibus-gitlab + repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When + a shutdown is triggered, the Sidekiq process will keep working normally for + another 15 minutes. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace + time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. + Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells + Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must + restart Sidekiq. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to `SIGKILL`. The name of + the final signal sent to the Sidekiq process when we want it to shut down. diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md new file mode 100644 index 00000000000..bad61151bda --- /dev/null +++ b/doc/administration/operations/unicorn.md @@ -0,0 +1,86 @@ +# Understanding Unicorn and unicorn-worker-killer + +## Unicorn + +GitLab uses [Unicorn](http://unicorn.bogomips.org/), a pre-forking Ruby web +server, to handle web requests (web browsers and Git HTTP clients). Unicorn is +a daemon written in Ruby and C that can load and run a Ruby on Rails +application; in our case the Rails application is GitLab Community Edition or +GitLab Enterprise Edition. + +Unicorn has a multi-process architecture to make better use of available CPU +cores (processes can run on different cores) and to have stronger fault +tolerance (most failures stay isolated in only one process and cannot take down +GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby +environment with the GitLab application code, and then spawns 'workers' which +inherit this clean initial environment. The 'master' never handles any +requests, that is left to the workers. The operating system network stack +queues incoming requests and distributes them among the workers. + +In a perfect world, the master would spawn its pool of workers once, and then +the workers handle incoming web requests one after another until the end of +time. In reality, worker processes can crash or time out: if the master notices +that a worker takes too long to handle a request it will terminate the worker +process with SIGKILL ('kill -9'). No matter how the worker process ended, the +master process will replace it with a new 'clean' process again. Unicorn is +designed to be able to replace 'crashed' workers without dropping user +requests. + +This is what a Unicorn worker timeout looks like in `unicorn_stderr.log`. The +master process has PID 56227 below. + +``` +[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing +[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10 +[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538 +[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready +``` + +### Tunables + +The main tunables for Unicorn are the number of worker processes and the +request timeout after which the Unicorn master terminates a worker process. +See the [omnibus-gitlab Unicorn settings +documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md) +if you want to adjust these settings. + +## unicorn-worker-killer + +GitLab has memory leaks. These memory leaks manifest themselves in long-running +processes, such as Unicorn workers. (The Unicorn master process is not known to +leak memory, probably because it does not handle user requests.) + +To make these memory leaks manageable, GitLab comes with the +[unicorn-worker-killer gem](https://github.com/kzk/unicorn-worker-killer). This +gem [monkey-patches](https://en.wikipedia.org/wiki/Monkey_patch) the Unicorn +workers to do a memory self-check after every 16 requests. If the memory of the +Unicorn worker exceeds a pre-set limit then the worker process exits. The +Unicorn master then automatically replaces the worker process. + +This is a robust way to handle memory leaks: Unicorn is designed to handle +workers that 'crash' so no user requests will be dropped. The +unicorn-worker-killer gem is designed to only terminate a worker process _in +between requests_, so no user requests are affected. + +This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. +You see that worker 4 (PID 125918) is inspecting itself and decides to exit. +The threshold memory value was 254802235 bytes, about 250MB. With GitLab this +threshold is a random value between 200 and 250 MB. The master process (PID +117565) then reaps the worker process and spawns a new 'worker 4' with PID +127549. + +``` +[2015-06-05T12:07:41.828374 #125918] WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes) +[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1) +[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4 +[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549 +[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready +``` + +One other thing that stands out in the log snippet above, taken from +GitLab.com, is that 'worker 4' was serving requests for only 23 seconds. This +is a normal value for our current GitLab.com setup and traffic. + +The high frequency of Unicorn memory restarts on some GitLab sites can be a +source of confusion for administrators. Usually they are a [red +herring](https://en.wikipedia.org/wiki/Red_herring). diff --git a/doc/administration/raketasks/README.md b/doc/administration/raketasks/README.md new file mode 100644 index 00000000000..a49c43b8ef2 --- /dev/null +++ b/doc/administration/raketasks/README.md @@ -0,0 +1,11 @@ +# Rake tasks + +- [Backup restore](backup_restore.md) +- [Check](check.md) +- [Cleanup](cleanup.md) +- [Features](features.md) +- [Maintenance](maintenance.md) and self-checks +- [User management](user_management.md) +- [Webhooks](web_hooks.md) +- [Import](import.md) of git repositories in bulk +- [Rebuild authorized_keys file](http://docs.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators diff --git a/doc/administration/raketasks/backup_hrz.png b/doc/administration/raketasks/backup_hrz.png Binary files differnew file mode 100644 index 00000000000..03e50df1d76 --- /dev/null +++ b/doc/administration/raketasks/backup_hrz.png diff --git a/doc/administration/raketasks/backup_restore.md b/doc/administration/raketasks/backup_restore.md new file mode 100644 index 00000000000..fa976134341 --- /dev/null +++ b/doc/administration/raketasks/backup_restore.md @@ -0,0 +1,438 @@ +# Backup restore + + + +## Create a backup of the GitLab system + +A backup creates an archive file that contains the database, all repositories and all attachments. +This archive will be saved in backup_path (see `config/gitlab.yml`). +The filename will be `[TIMESTAMP]_gitlab_backup.tar`. This timestamp can be used to restore an specific backup. +You can only restore a backup to exactly the same version of GitLab that you created it +on, for example 7.2.1. The best way to migrate your repositories from one server to +another is through backup restore. + +You need to keep a separate copy of `/etc/gitlab/gitlab-secrets.json` +(for omnibus packages) or `/home/git/gitlab/.secret` (for installations +from source). This file contains the database encryption key used +for two-factor authentication. If you restore a GitLab backup without +restoring the database encryption key, users who have two-factor +authentication enabled will lose access to your GitLab server. + +``` +# use this command if you've installed GitLab with the Omnibus package +sudo gitlab-rake gitlab:backup:create + +# if you've installed GitLab from source +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +Also you can choose what should be backed up by adding environment variable SKIP. Available options: db, +uploads (attachments), repositories, builds(CI build output logs), artifacts (CI build artifacts), lfs (LFS objects). +Use a comma to specify several options at the same time. + +``` +sudo gitlab-rake gitlab:backup:create SKIP=db,uploads +``` + +Example output: + +``` +Dumping database tables: +- Dumping table events... [DONE] +- Dumping table issues... [DONE] +- Dumping table keys... [DONE] +- Dumping table merge_requests... [DONE] +- Dumping table milestones... [DONE] +- Dumping table namespaces... [DONE] +- Dumping table notes... [DONE] +- Dumping table projects... [DONE] +- Dumping table protected_branches... [DONE] +- Dumping table schema_migrations... [DONE] +- Dumping table services... [DONE] +- Dumping table snippets... [DONE] +- Dumping table taggings... [DONE] +- Dumping table tags... [DONE] +- Dumping table users... [DONE] +- Dumping table users_projects... [DONE] +- Dumping table web_hooks... [DONE] +- Dumping table wikis... [DONE] +Dumping repositories: +- Dumping repository abcd... [DONE] +Creating backup archive: $TIMESTAMP_gitlab_backup.tar [DONE] +Deleting tmp directories...[DONE] +Deleting old backups... [SKIPPING] +``` + +## Upload backups to remote (cloud) storage + +Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates. +It uses the [Fog library](http://fog.io/) to perform the upload. +In the example below we use Amazon S3 for storage. +But Fog also lets you use [other storage providers](http://fog.io/storage/). + +For omnibus packages: + +```ruby +gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-west-1', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123' +} +gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' +``` + +For installations from source: + +```yaml + backup: + # snip + upload: + # Fog storage connection settings, see http://fog.io/storage/ . + connection: + provider: AWS + region: eu-west-1 + aws_access_key_id: AKIAKIAKI + aws_secret_access_key: 'secret123' + # The remote 'directory' to store your backups. For S3, this would be the bucket name. + remote_directory: 'my.s3.bucket' + # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional + # encryption: 'AES256' +``` + +If you are uploading your backups to S3 you will probably want to create a new +IAM user with restricted access rights. To give the upload user access only for +uploading backups create the following IAM profile, replacing `my.s3.bucket` +with the name of your bucket: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Stmt1412062044000", + "Effect": "Allow", + "Action": [ + "s3:AbortMultipartUpload", + "s3:GetBucketAcl", + "s3:GetBucketLocation", + "s3:GetObject", + "s3:GetObjectAcl", + "s3:ListBucketMultipartUploads", + "s3:PutObject", + "s3:PutObjectAcl" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket/*" + ] + }, + { + "Sid": "Stmt1412062097000", + "Effect": "Allow", + "Action": [ + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "*" + ] + }, + { + "Sid": "Stmt1412062128000", + "Effect": "Allow", + "Action": [ + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket" + ] + } + ] +} +``` + +### Uploading to locally mounted shares + +You may also send backups to a mounted share (`NFS` / `CIFS` / `SMB` / etc.) by +using the [`Local`](https://github.com/fog/fog-local#usage) storage provider. +The directory pointed to by the `local_root` key **must** be owned by the `git` +user **when mounted** (mounting with the `uid=` of the `git` user for `CIFS` and +`SMB`) or the user that you are executing the backup tasks under (for omnibus +packages, this is the `git` user). + +The `backup_upload_remote_directory` **must** be set in addition to the +`local_root` key. This is the sub directory inside the mounted directory that +backups will be copied to, and will be created if it does not exist. If the +directory that you want to copy the tarballs to is the root of your mounted +directory, just use `.` instead. + +For omnibus packages: + +```ruby +gitlab_rails['backup_upload_connection'] = { + :provider => 'Local', + :local_root => '/mnt/backups' +} + +# The directory inside the mounted folder to copy backups to +# Use '.' to store them in the root directory +gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' +``` + +For installations from source: + +```yaml + backup: + # snip + upload: + # Fog storage connection settings, see http://fog.io/storage/ . + connection: + provider: Local + local_root: '/mnt/backups' + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + remote_directory: 'gitlab_backups' +``` + +## Backup archive permissions + +The backup archives created by GitLab (123456_gitlab_backup.tar) will have owner/group git:git and 0600 permissions by default. +This is meant to avoid other system users reading GitLab's data. +If you need the backup archives to have different permissions you can use the 'archive_permissions' setting. + +``` +# In /etc/gitlab/gitlab.rb, for omnibus packages +gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable +``` + +``` +# In gitlab.yml, for installations from source: + backup: + archive_permissions: 0644 # Makes the backup archives world-readable +``` + +## Storing configuration files + +Please be informed that a backup does not store your configuration +files. One reason for this is that your database contains encrypted +information for two-factor authentication. Storing encrypted +information along with its key in the same place defeats the purpose +of using encryption in the first place! + +If you use an Omnibus package please see the [instructions in the readme to backup your configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#backup-and-restore-omnibus-gitlab-configuration). +If you have a cookbook installation there should be a copy of your configuration in Chef. +If you have an installation from source, please consider backing up your `.secret` file, `gitlab.yml` file, any SSL keys and certificates, and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + +At the very **minimum** you should backup `/etc/gitlab/gitlab-secrets.json` +(Omnibus) or `/home/git/gitlab/.secret` (source) to preserve your +database encryption key. + +## Restore a previously created backup + +You can only restore a backup to exactly the same version of GitLab that you created it on, for example 7.2.1. + +### Prerequisites + +You need to have a working GitLab installation before you can perform +a restore. This is mainly because the system user performing the +restore actions ('git') is usually not allowed to create or delete +the SQL database it needs to import data into ('gitlabhq_production'). +All existing data will be either erased (SQL) or moved to a separate +directory (repositories, uploads). + +If some or all of your GitLab users are using two-factor authentication +(2FA) then you must also make sure to restore +`/etc/gitlab/gitlab-secrets.json` (Omnibus) or `/home/git/gitlab/.secret` +(installations from source). Note that you need to run `gitlab-ctl +reconfigure` after changing `gitlab-secrets.json`. + +### Installation from source + +``` +# Stop processes that are connected to the database +sudo service gitlab stop + +bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +Options: + +``` +BACKUP=timestamp_of_backup (required if more than one backup exists) +force=yes (do not ask if the authorized_keys file should get regenerated) +``` + +Example output: + +``` +Unpacking backup... [DONE] +Restoring database tables: +-- create_table("events", {:force=>true}) + -> 0.2231s +[...] +- Loading fixture events...[DONE] +- Loading fixture issues...[DONE] +- Loading fixture keys...[SKIPPING] +- Loading fixture merge_requests...[DONE] +- Loading fixture milestones...[DONE] +- Loading fixture namespaces...[DONE] +- Loading fixture notes...[DONE] +- Loading fixture projects...[DONE] +- Loading fixture protected_branches...[SKIPPING] +- Loading fixture schema_migrations...[DONE] +- Loading fixture services...[SKIPPING] +- Loading fixture snippets...[SKIPPING] +- Loading fixture taggings...[SKIPPING] +- Loading fixture tags...[SKIPPING] +- Loading fixture users...[DONE] +- Loading fixture users_projects...[DONE] +- Loading fixture web_hooks...[SKIPPING] +- Loading fixture wikis...[SKIPPING] +Restoring repositories: +- Restoring repository abcd... [DONE] +Deleting tmp directories...[DONE] +``` + +### Omnibus installations + +This procedure assumes that: + +- You have installed the exact same version of GitLab Omnibus with which the + backup was created +- You have run `sudo gitlab-ctl reconfigure` at least once +- GitLab is running. If not, start it using `sudo gitlab-ctl start`. + +First make sure your backup tar file is in the backup directory described in the +`gitlab.rb` configuration `gitlab_rails['backup_path']`. The default is +`/var/opt/gitlab/backups`. + +```shell +sudo cp 1393513186_gitlab_backup.tar /var/opt/gitlab/backups/ +``` + +Stop the processes that are connected to the database. Leave the rest of GitLab +running: + +```shell +sudo gitlab-ctl stop unicorn +sudo gitlab-ctl stop sidekiq +# Verify +sudo gitlab-ctl status +``` + +Next, restore the backup, specifying the timestamp of the backup you wish to +restore: + +```shell +# This command will overwrite the contents of your GitLab database! +sudo gitlab-rake gitlab:backup:restore BACKUP=1393513186 +``` + +Restart and check GitLab: + +```shell +sudo gitlab-ctl start +sudo gitlab-rake gitlab:check SANITIZE=true +``` + +If there is a GitLab version mismatch between your backup tar file and the installed +version of GitLab, the restore command will abort with an error. Install the +[correct GitLab version](https://www.gitlab.com/downloads/archives/) and try again. + +## Configure cron to make daily backups + +### For installation from source: +``` +cd /home/git/gitlab +sudo -u git -H editor config/gitlab.yml # Enable keep_time in the backup section to automatically delete old backups +sudo -u git crontab -e # Edit the crontab for the git user +``` + +Add the following lines at the bottom: + +``` +# Create a full backup of the GitLab repositories and SQL database every day at 4am +0 4 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1 +``` + +The `CRON=1` environment setting tells the backup script to suppress all progress output if there are no errors. +This is recommended to reduce cron spam. + +### For omnibus installations + +To schedule a cron job that backs up your repositories and GitLab metadata, use the root user: + +``` +sudo su - +crontab -e +``` + +There, add the following line to schedule the backup for everyday at 2 AM: + +``` +0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1 +``` + +You may also want to set a limited lifetime for backups to prevent regular +backups using all your disk space. To do this add the following lines to +`/etc/gitlab/gitlab.rb` and reconfigure: + +``` +# limit backup lifetime to 7 days - 604800 seconds +gitlab_rails['backup_keep_time'] = 604800 +``` + +NOTE: This cron job does not [backup your omnibus-gitlab configuration](#backup-and-restore-omnibus-gitlab-configuration) or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + +## Alternative backup strategies + +If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. +In this case you can consider using filesystem snapshots as part of your backup strategy. + +Example: Amazon EBS + +> A GitLab server using omnibus-gitlab hosted on Amazon AWS. +> An EBS drive containing an ext4 filesystem is mounted at `/var/opt/gitlab`. +> In this case you could make an application backup by taking an EBS snapshot. +> The backup includes all repositories, uploads and Postgres data. + +Example: LVM snapshots + rsync + +> A GitLab server using omnibus-gitlab, with an LVM logical volume mounted at `/var/opt/gitlab`. +> Replicating the `/var/opt/gitlab` directory using rsync would not be reliable because too many files would change while rsync is running. +> Instead of rsync-ing `/var/opt/gitlab`, we create a temporary LVM snapshot, which we mount as a read-only filesystem at `/mnt/gitlab_backup`. +> Now we can have a longer running rsync job which will create a consistent replica on the remote server. +> The replica includes all repositories, uploads and Postgres data. + +If you are running GitLab on a virtualized server you can possibly also create VM snapshots of the entire GitLab server. +It is not uncommon however for a VM snapshot to require you to power down the server, so this approach is probably of limited practical use. + +## Troubleshooting + +### Restoring database backup using omnibus packages outputs warnings +If you are using backup restore procedures you might encounter the following warnings: + +``` +psql:/var/opt/gitlab/backups/db/database.sql:22: ERROR: must be owner of extension plpgsql +psql:/var/opt/gitlab/backups/db/database.sql:2931: WARNING: no privileges could be revoked for "public" (two occurrences) +psql:/var/opt/gitlab/backups/db/database.sql:2933: WARNING: no privileges were granted for "public" (two occurrences) + +``` + +Be advised that, backup is successfully restored in spite of these warnings. + +The rake task runs this as the `gitlab` user which does not have the superuser access to the database. When restore is initiated it will also run as `gitlab` user but it will also try to alter the objects it does not have access to. +Those objects have no influence on the database backup/restore but they give this annoying warning. + +For more information see similar questions on postgresql issue tracker[here](http://www.postgresql.org/message-id/201110220712.30886.adrian.klaver@gmail.com) and [here](http://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us) as well as [stack overflow](http://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql). + +## Note +This documentation is for GitLab CE. +We backup GitLab.com and make sure your data is secure, but you can't use these methods to export / backup your data yourself from GitLab.com. + +Issues are stored in the database. They can't be stored in Git itself. + +To migrate your repositories from one server to another with an up-to-date version of +GitLab, you can use the [import rake task](import.md) to do a mass import of the +repository. Note that if you do an import rake task, rather than a backup restore, you +will have all your repositories, but not any other data. diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md new file mode 100644 index 00000000000..3ff3fee6a40 --- /dev/null +++ b/doc/administration/raketasks/check.md @@ -0,0 +1,63 @@ +# Check Rake Tasks + +## Repository Integrity + +Even though Git is very resilient and tries to prevent data integrity issues, +there are times when things go wrong. The following Rake tasks intend to +help GitLab administrators diagnose problem repositories so they can be fixed. + +There are 3 things that are checked to determine integrity. + +1. Git repository file system check ([git fsck](https://git-scm.com/docs/git-fsck)). + This step verifies the connectivity and validity of objects in the repository. +1. Check for `config.lock` in the repository directory. +1. Check for any branch/references lock files in `refs/heads`. + +It's important to note that the existence of `config.lock` or reference locks +alone do not necessarily indicate a problem. Lock files are routinely created +and removed as Git and GitLab perform operations on the repository. They serve +to prevent data integrity issues. However, if a Git operation is interrupted these +locks may not be cleaned up properly. + +The following symptoms may indicate a problem with repository integrity. If users +experience these symptoms you may use the rake tasks described below to determine +exactly which repositories are causing the trouble. + +- Receiving an error when trying to push code - `remote: error: cannot lock ref` +- A 500 error when viewing the GitLab dashboard or when accessing a specific project. + +### Check all GitLab repositories + +This task loops through all repositories on the GitLab server and runs the +3 integrity checks described previously. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:repo:check + +# installation from source +bundle exec rake gitlab:repo:check RAILS_ENV=production +``` + +### Check repositories for a specific user + +This task checks all repositories that a specific user has access to. This is important +because sometimes you know which user is experiencing trouble but you don't know +which project might be the cause. + +If the rake task is executed without brackets at the end, you will be prompted +to enter a username. + +```bash +# omnibus-gitlab +sudo gitlab-rake gitlab:user:check_repos +sudo gitlab-rake gitlab:user:check_repos[<username>] + +# installation from source +bundle exec rake gitlab:user:check_repos RAILS_ENV=production +bundle exec rake gitlab:user:check_repos[<username>] RAILS_ENV=production +``` + +Example output: + + diff --git a/doc/administration/raketasks/check_repos_output.png b/doc/administration/raketasks/check_repos_output.png Binary files differnew file mode 100644 index 00000000000..916b1685101 --- /dev/null +++ b/doc/administration/raketasks/check_repos_output.png diff --git a/doc/administration/raketasks/cleanup.md b/doc/administration/raketasks/cleanup.md new file mode 100644 index 00000000000..8fbcbb983e9 --- /dev/null +++ b/doc/administration/raketasks/cleanup.md @@ -0,0 +1,24 @@ +# Cleanup + +## Remove garbage from filesystem. Important! Data loss! + +Remove namespaces(dirs) from `/home/git/repositories` if they don't exist in GitLab database. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:cleanup:dirs + +# installation from source +bundle exec rake gitlab:cleanup:dirs RAILS_ENV=production +``` + +Rename repositories from `/home/git/repositories` if they don't exist in GitLab database. +The repositories get a `+orphaned+TIMESTAMP` suffix so that they cannot block new repositories from being created. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:cleanup:repos + +# installation from source +bundle exec rake gitlab:cleanup:repos RAILS_ENV=production +``` diff --git a/doc/administration/raketasks/features.md b/doc/administration/raketasks/features.md new file mode 100644 index 00000000000..f9a46193547 --- /dev/null +++ b/doc/administration/raketasks/features.md @@ -0,0 +1,20 @@ +# Features + +## Enable usernames and namespaces for user projects + +This command will enable the namespaces feature introduced in v4.0. It will move every project in its namespace folder. + +Note: + +- Because the **repository location will change**, you will need to **update all your git URLs** to point to the new location. +- Username can be changed at [Profile / Account](/profile/account) + +**Example:** + +Old path: `git@example.org:myrepo.git` + +New path: `git@example.org:username/myrepo.git` or `git@example.org:groupname/myrepo.git` + +``` +bundle exec rake gitlab:enable_namespaces RAILS_ENV=production +``` diff --git a/doc/administration/raketasks/import.md b/doc/administration/raketasks/import.md new file mode 100644 index 00000000000..8a38937062e --- /dev/null +++ b/doc/administration/raketasks/import.md @@ -0,0 +1,68 @@ +# Import bare repositories into your GitLab instance + +## Notes + +- The owner of the project will be the first admin +- The groups will be created as needed +- The owner of the group will be the first admin +- Existing projects will be skipped + +## How to use + +### Create a new folder inside the git repositories path. This will be the name of the new group. + +- For omnibus-gitlab, it is located at: `/var/opt/gitlab/git-data/repositories` by default, unless you changed +it in the `/etc/gitlab/gitlab.rb` file. +- For installations from source, it is usually located at: `/home/git/repositories` or you can see where +your repositories are located by looking at `config/gitlab.yml` under the `gitlab_shell => repos_path` entry. + +New folder needs to have git user ownership and read/write/execute access for git user and its group: + +``` +sudo -u git mkdir /var/opt/gitlab/git-data/repositories/new_group +``` + +If you are using an installation from source, replace `/var/opt/gitlab/git-data` +with `/home/git`. + +### Copy your bare repositories inside this newly created folder: + +``` +sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repositories/new_group/ + +# Do this once when you are done copying git repositories +sudo chown -R git:git /var/opt/gitlab/git-data/repositories/new_group/ +``` + +`foo.git` needs to be owned by the git user and git users group. + +If you are using an installation from source, replace `/var/opt/gitlab/git-data` +with `/home/git`. + +### Run the command below depending on your type of installation: + +#### Omnibus Installation + +``` +$ sudo gitlab-rake gitlab:import:repos +``` + +#### Installation from source + +Before running this command you need to change the directory to where your GitLab installation is located: + +``` +$ cd /home/git/gitlab +$ sudo -u git -H bundle exec rake gitlab:import:repos RAILS_ENV=production +``` + +#### Example output + +``` +Processing abcd.git + * Created abcd (abcd.git) +Processing group/xyz.git + * Created Group group (2) + * Created xyz (group/xyz.git) +[...] +``` diff --git a/doc/administration/raketasks/list_repos.md b/doc/administration/raketasks/list_repos.md new file mode 100644 index 00000000000..476428eb4f5 --- /dev/null +++ b/doc/administration/raketasks/list_repos.md @@ -0,0 +1,30 @@ +# Listing repository directories + +You can print a list of all Git repositories on disk managed by +GitLab with the following command: + +``` +# Omnibus +sudo gitlab-rake gitlab:list_repos + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production +``` + +If you only want to list projects with recent activity you can pass +a date with the 'SINCE' environment variable. The time you specify +is parsed by the Rails [TimeZone#parse +function](http://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse). + +``` +# Omnibus +sudo gitlab-rake gitlab:list_repos SINCE='Sep 1 2015' + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production SINCE='Sep 1 2015' +``` + +Note that the projects listed are NOT sorted by activity; they use +the default ordering of the GitLab Rails application. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md new file mode 100644 index 00000000000..d9dce2af480 --- /dev/null +++ b/doc/administration/raketasks/maintenance.md @@ -0,0 +1,169 @@ +# Maintenance + +## Gather information about GitLab and the system it runs on + +This command gathers information about your GitLab installation and the System it runs on. These may be useful when asking for help or reporting issues. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:env:info + +# installation from source +bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +Example output: + +``` +System information +System: Debian 7.8 +Current User: git +Using RVM: no +Ruby Version: 2.1.5p273 +Gem Version: 2.4.3 +Bundler Version: 1.7.6 +Rake Version: 10.3.2 +Sidekiq Version: 2.17.8 + +GitLab information +Version: 7.7.1 +Revision: 41ab9e1 +Directory: /home/git/gitlab +DB Adapter: postgresql +URL: https://gitlab.example.com +HTTP Clone URL: https://gitlab.example.com/some-project.git +SSH Clone URL: git@gitlab.example.com:some-project.git +Using LDAP: no +Using Omniauth: no + +GitLab Shell +Version: 2.4.1 +Repositories: /home/git/repositories/ +Hooks: /home/git/gitlab-shell/hooks/ +Git: /usr/bin/git +``` + +## Check GitLab configuration + +Runs the following rake tasks: + +- `gitlab:gitlab_shell:check` +- `gitlab:sidekiq:check` +- `gitlab:app:check` + +It will check that each component was setup according to the installation guide and suggest fixes for issues found. + +You may also have a look at our [Trouble Shooting Guide](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Trouble-Shooting-Guide). + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:check + +# installation from source +bundle exec rake gitlab:check RAILS_ENV=production +``` + +NOTE: Use SANITIZE=true for gitlab:check if you want to omit project names from the output. + +Example output: + +``` +Checking Environment ... + +Git configured for git user? ... yes +Has python2? ... yes +python2 is supported version? ... yes + +Checking Environment ... Finished + +Checking GitLab Shell ... + +GitLab Shell version? ... OK (1.2.0) +Repo base directory exists? ... yes +Repo base directory is a symlink? ... no +Repo base owned by git:git? ... yes +Repo base access is drwxrws---? ... yes +post-receive hook up-to-date? ... yes +post-receive hooks in repos are links: ... yes + +Checking GitLab Shell ... Finished + +Checking Sidekiq ... + +Running? ... yes + +Checking Sidekiq ... Finished + +Checking GitLab ... + +Database config exists? ... yes +Database is SQLite ... no +All migrations up? ... yes +GitLab config exists? ... yes +GitLab config outdated? ... no +Log directory writable? ... yes +Tmp directory writable? ... yes +Init script exists? ... yes +Init script up-to-date? ... yes +Redis version >= 2.0.0? ... yes + +Checking GitLab ... Finished +``` + +## Rebuild authorized_keys file + +In some case it is necessary to rebuild the `authorized_keys` file. + +For Omnibus-packages: +``` +sudo gitlab-rake gitlab:shell:setup +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production +``` + +``` +This will rebuild an authorized_keys file. +You will lose any data stored in authorized_keys file. +Do you want to continue (yes/no)? yes +``` + +## Clear redis cache + +If for some reason the dashboard shows wrong information you might want to +clear Redis' cache. + +For Omnibus-packages: +``` +sudo gitlab-rake cache:clear +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +## Precompile the assets + +Sometimes during version upgrades you might end up with some wrong CSS or +missing some icons. In that case, try to precompile the assets again. + +Note that this only applies to source installations and does NOT apply to +omnibus packages. + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production +``` + +For omnibus versions, the unoptimized assets (JavaScript, CSS) are frozen at +the release of upstream GitLab. The omnibus version includes optimized versions +of those assets. Unless you are modifying the JavaScript / CSS code on your +production machine after installing the package, there should be no reason to redo +rake assets:precompile on the production machine. If you suspect that assets +have been corrupted, you should reinstall the omnibus package. diff --git a/doc/administration/raketasks/user_management.md b/doc/administration/raketasks/user_management.md new file mode 100644 index 00000000000..629d38efc53 --- /dev/null +++ b/doc/administration/raketasks/user_management.md @@ -0,0 +1,72 @@ +# User management + +## Add user as a developer to all projects + +```bash +# omnibus-gitlab +sudo gitlab-rake gitlab:import:user_to_projects[username@domain.tld] + +# installation from source +bundle exec rake gitlab:import:user_to_projects[username@domain.tld] RAILS_ENV=production +``` + +## Add all users to all projects + +Notes: + +- admin users are added as masters + +```bash +# omnibus-gitlab +sudo gitlab-rake gitlab:import:all_users_to_all_projects + +# installation from source +bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production +``` + +## Add user as a developer to all groups + +```bash +# omnibus-gitlab +sudo gitlab-rake gitlab:import:user_to_groups[username@domain.tld] + +# installation from source +bundle exec rake gitlab:import:user_to_groups[username@domain.tld] RAILS_ENV=production +``` + +## Add all users to all groups + +Notes: + +- admin users are added as owners so they can add additional users to the group + +```bash +# omnibus-gitlab +sudo gitlab-rake gitlab:import:all_users_to_all_groups + +# installation from source +bundle exec rake gitlab:import:all_users_to_all_groups RAILS_ENV=production +``` + +## Maintain tight control over the number of active users on your GitLab installation + +- Enable this setting to keep new users blocked until they have been cleared by the admin (default: false). + + +``` +block_auto_created_users: false +``` + +## Disable Two-factor Authentication (2FA) for all users + +This task will disable 2FA for all users that have it enabled. This can be +useful if GitLab's `.secret` file has been lost and users are unable to login, +for example. + +```bash +# omnibus-gitlab +sudo gitlab-rake gitlab:two_factor:disable_for_all_users + +# installation from source +bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production +``` diff --git a/doc/administration/raketasks/web_hooks.md b/doc/administration/raketasks/web_hooks.md new file mode 100644 index 00000000000..2ebf7c48f4e --- /dev/null +++ b/doc/administration/raketasks/web_hooks.md @@ -0,0 +1,45 @@ +# Webhooks + +## Add a webhook for **ALL** projects: + + # omnibus-gitlab + sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" + # source installations + bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production + +## Add a webhook for projects in a given **NAMESPACE**: + + # omnibus-gitlab + sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme + # source installations + bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production + +## Remove a webhook from **ALL** projects using: + + # omnibus-gitlab + sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" + # source installations + bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production + +## Remove a webhook from projects in a given **NAMESPACE**: + + # omnibus-gitlab + sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme + # source installations + bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production + +## List **ALL** webhooks: + + # omnibus-gitlab + sudo gitlab-rake gitlab:web_hook:list + # source installations + bundle exec rake gitlab:web_hook:list RAILS_ENV=production + +## List the webhooks from projects in a given **NAMESPACE**: + + # omnibus-gitlab + sudo gitlab-rake gitlab:web_hook:list NAMESPACE=/ + # source installations + bundle exec rake gitlab:web_hook:list NAMESPACE=/ RAILS_ENV=production + +> Note: `/` is the global namespace. |
