Adding documentation for Prometheus Service Discovery

Updating documentation to include
Prometheus Service Discovery. Added
monitoring node documentaiton as
well as documentation for the
individual service nodes.

6 files changed, 192 insertions, 2 deletions

diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md
index d9c80b1ec59..0c4f926c579 100644
--- a/doc/administration/high_availability/README.md
+++ b/doc/administration/high_availability/README.md
@@ -65,6 +65,7 @@ larger one.
 - 1 Redis node
 - 1 NFS/Gitaly storage server
 - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq)
+- 1 Monitoring node (Prometheus, Grafana)
 
 #### Installation Instructions
 
@@ -76,6 +77,7 @@ you can continue with the next step.
 1. [Redis](redis.md#redis-in-a-scaled-environment)
 1. [Gitaly](gitaly.md) (recommended) or [NFS](nfs.md)
 1. [GitLab application nodes](gitlab.md)
+1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md)
 
 ### Full Scaling
 
@@ -90,6 +92,7 @@ in size, indicating that there is contention or not enough resources.
 - 2 or more NFS/Gitaly storage servers
 - 2 or more Sidekiq nodes
 - 2 or more GitLab application nodes (Unicorn, Workhorse)
+- 1 Monitoring node (Prometheus, Grafana)
 
 ## High Availability Architecture Examples
 
@@ -133,6 +136,7 @@ the contention.
 - 3 Consul/Sentinel nodes
 - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer)
 - 1 NFS/Gitaly server
+- 1 Monitoring node (Prometheus, Grafana)
 
 ![Horizontal architecture diagram](img/horizontal.png)
 
@@ -192,6 +196,7 @@ with the added complexity of many more nodes to configure, manage and monitor.
 - 2 or more API nodes (All requests to `/api`)
 - 2 or more Web nodes (All other web requests)
 - 2 or more NFS/Gitaly servers
+- 1 Monitoring node (Prometheus, Grafana)
 
 ![Fully Distributed architecture diagram](img/fully-distributed.png)
 
@@ -205,4 +210,5 @@ separately:
    1. [NFS Client and Host setup](nfs_host_client_setup.md)
 1. [Configure the GitLab application servers](gitlab.md)
 1. [Configure the load balancers](load_balancer.md)
+1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md)
 
diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md
index 2c051b660ee..4db53353218 100644
--- a/doc/administration/high_availability/database.md
+++ b/doc/administration/high_availability/database.md
@@ -82,7 +82,8 @@ deploy the bundled PostgreSQL.
 1. Note the PostgreSQL node's IP address or hostname, port, and
    plain text password. These will be necessary when configuring the GitLab
    application servers later.
-
+1. [Enable monitoring](#enable-monitoring)
+   
 Advanced configuration options are supported and can be added if
 needed.
 
@@ -399,6 +400,7 @@ check the [Troubleshooting section](#troubleshooting) before proceeding.
    ```
 
 1. [Reconfigure GitLab] for the changes to take effect.
+1. [Enable Monitoring](#enable-monitoring)
 
 > Please note:
 >
@@ -1086,6 +1088,25 @@ the previous section:
       the `gitlab` database user
    1. [Reconfigure GitLab] for the changes to take effect
 
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+If you enable Monitoring, it must be enabled on **all** database servers.
+
+1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+   ```ruby
+   # Enable service discovery for Prometheus
+   consul['monitoring_service_discovery'] = true
+
+   # Set the network addresses that the exporters will listen on
+   node_exporter['listen_address'] = '0.0.0.0:9100'
+   postgres_exporter['listen_address'] = '0.0.0.0:9187'
+   ```
+
+1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
 ## Troubleshooting
 
 #### Consul and PostgreSQL changes not taking effect.
diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md
index 1d8e6c999cb..90e5f71d835 100644
--- a/doc/administration/high_availability/gitaly.md
+++ b/doc/administration/high_availability/gitaly.md
@@ -19,3 +19,28 @@ Continue configuration of other components by going back to:
 
 - [Scaled Architectures](README.md#scalable-architecture-examples)
 - [High Availability Architectures](README.md#high-availability-architecture-examples)
+
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+  1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+     ```ruby
+     # Enable service discovery for Prometheus
+     consul['enable'] = true
+     consul['monitoring_service_discovery'] =  true
+
+     # Replace placeholders
+     # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+     # with the addresses of the Consul server nodes
+     consul['configuration'] = {
+        retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+     }
+
+     # Set the network addresses that the exporters will listen on
+     node_exporter['listen_address'] = '0.0.0.0:9100' 
+     gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
+     ```
+
+  1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md
index 7e3ff741071..0e655e49922 100644
--- a/doc/administration/high_availability/gitlab.md
+++ b/doc/administration/high_availability/gitlab.md
@@ -76,6 +76,8 @@
     registry['gid'] = 9002
     ```
 
+1. [Enable monitoring](#enable-monitoring)
+
     > **Note:** To maintain uniformity of links across HA clusters, the `external_url`
     on the first application server as well as the additional application
     servers should point to the external url that users will use to access GitLab.
@@ -88,7 +90,8 @@
     [Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
     for more information.
     >
-    > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure.
+    > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure
+    of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure.
 
 ## First GitLab application server
 
@@ -129,6 +132,46 @@ need some extra configuration.
 
 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
 
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+If you enable Monitoring, it must be enabled on **all** GitLab servers.
+
+1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+   ```ruby
+   # Enable service discovery for Prometheus
+   consul['enable'] = true
+   consul['monitoring_service_discovery'] =  true
+
+   # Replace placeholders
+   # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+   # with the addresses of the Consul server nodes
+   consul['configuration'] = {
+      retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+   }
+
+   # Set the network addresses that the exporters will listen on
+   node_exporter['listen_address'] = '0.0.0.0:9100'
+   gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
+   sidekiq['listen_address'] = "0.0.0.0"
+   unicorn['listen'] = '0.0.0.0'
+
+   # Add the monitoring node's IP address to the monitoring whitelist and allow it to scrape the NGINX metrics
+   # Replace placeholder
+   # monitoring.gitlab.example.com
+   # with the addresses gathered for the monitoring node
+   gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com']
+   nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com']
+   ```
+
+1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
+> **Warning:** After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`,
+  it can take an extended period of time for unicorn to complete reloading after receiving a `HUP`.
+  For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401).
+
 ## Troubleshooting
 
 - `mount: wrong fs type, bad option, bad superblock on`
diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md
new file mode 100644
index 00000000000..d16bf7dc0f0
--- /dev/null
+++ b/doc/administration/high_availability/monitoring_node.md
@@ -0,0 +1,67 @@
+# Configuring a Monitoring node for Scaling and High Availability
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+## Standalone Monitoring node using GitLab Omnibus
+
+The GitLab Omnibus package can be used to configure a standalone Monitoring node running Prometheus and Grafana.
+The monitoring node is not highly available. See [Scaling and High Availability](README.md)
+for an overview of GitLab scaling and high availability options.
+
+The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with
+Omnibus:
+
+1. SSH into the Monitoring node.
+1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
+   package you want using **steps 1 and 2** from the GitLab downloads page.
+   - Do not complete any other steps on the download page.
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
+
+    ```ruby
+    external_url 'http://gitlab.example.com'
+
+    # Enable Prometheus
+    prometheus['enable'] = true
+    prometheus['listen_address'] = '0.0.0.0:9090'
+    prometheus['monitor_kubernetes'] = false
+
+    # Enable Grafana
+    grafana['enable'] = true
+    grafana['admin_password'] = 'toomanysecrets'
+
+    # Enable service discovery for Prometheus
+    consul['enable'] = true
+    consul['monitoring_service_discovery'] =  true
+
+    # Replace placeholders
+    # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+    # with the addresses of the Consul server nodes
+    consul['configuration'] = {
+       retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+    }
+
+    # Disable all other services
+    gitlab_rails['auto_migrate'] = false
+    alertmanager['enable'] = false
+    gitaly['enable'] = false
+    gitlab_monitor['enable'] = false
+    gitlab_workhorse['enable'] = false
+    nginx['enable'] = true
+    postgres_exporter['enable'] = false
+    postgresql['enable'] = false
+    redis['enable'] = false
+    redis_exporter['enable'] = false
+    sidekiq['enable'] = false
+    unicorn['enable'] = false
+    node_exporter['enable'] = false
+    ```
+
+1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
+## Migrating to Service Discovery
+
+Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] =  true`,
+ensure that `prometheus['scrape_configs']` is not set  in `/etc/gitlab/gitlab.rb`. Setting both
+`consul['monitoring_service_discovery'] =  true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb`
+will result in errors.
diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md
index 1aaa709fc8f..f61a8834af3 100644
--- a/doc/administration/high_availability/redis.md
+++ b/doc/administration/high_availability/redis.md
@@ -74,6 +74,7 @@ Omnibus:
 1. Note the Redis node's IP address or hostname, port, and
    Redis password. These will be necessary when configuring the GitLab
    application servers later.
+1. [Enable Monitoring](#enable-monitoring)   
 
 Advanced configuration options are supported and can be added if
 needed.
@@ -749,6 +750,33 @@ gitlab_rails['redis_sentinels'] = [
 
 [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
 
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+  If you enable Monitoring, it must be enabled on **all** Redis servers.
+
+  1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+     ```ruby
+     # Enable service discovery for Prometheus
+     consul['enable'] = true
+     consul['monitoring_service_discovery'] =  true
+
+     # Replace placeholders
+     # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+     # with the addresses of the Consul server nodes
+     consul['configuration'] = {
+        retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+     }
+
+     # Set the network addresses that the exporters will listen on
+     node_exporter['listen_address'] = '0.0.0.0:9100'
+     redis_exporter['listen_address'] = '0.0.0.0:9121'
+     ```
+
+  1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
 ## Advanced configuration
 
 Omnibus GitLab configures some things behind the curtains to make the sysadmins'