diff options
Diffstat (limited to 'doc/user/project/integrations/prometheus.md')
| -rw-r--r-- | doc/user/project/integrations/prometheus.md | 161 |
1 files changed, 135 insertions, 26 deletions
diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 6c40e5b9696..0d17ff51372 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -25,7 +25,7 @@ Once enabled, GitLab will automatically detect metrics from known services in th ### Managed Prometheus on Kubernetes -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/28916) in GitLab 10.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28916) in GitLab 10.5. GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), making monitoring of your apps easy. @@ -61,7 +61,7 @@ will help you to quickly create a deployment: 1. Navigate to your project's **CI/CD > Pipelines** page, and run a pipeline on any branch. 1. When the pipeline has run successfully, graphs will be available on the **Operations > Metrics** page. - + #### Using the Metrics Dashboard @@ -209,10 +209,19 @@ GitLab supports a limited set of [CI variables](../../../ci/variables/README.md) - `ci_project_namespace` - `ci_project_path` - `ci_environment_name` +- `__range` NOTE: **Note:** Variables for Prometheus queries must be lowercase. +###### __range + +The `__range` variable is useful in Prometheus +[range vector selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors). +Its value is the total number of seconds in the dashboard's time range. +For example, if the dashboard time range is set to 8 hours, the value of +`__range` is `28800s`. + ##### User-defined variables [Variables can be defined](#templating-templating-properties) in a custom dashboard YAML file. @@ -244,7 +253,7 @@ http://gitlab.com/<user>/<project>/-/environments/<environment_id>/metrics?dashb #### Editing additional metrics from the dashboard -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/208976) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208976) in GitLab 12.9. You can edit existing additional custom metrics by clicking the **{ellipsis_v}** **More actions** dropdown and selecting **Edit metric**. @@ -252,7 +261,7 @@ You can edit existing additional custom metrics by clicking the **{ellipsis_v}** ### Defining custom dashboards per project -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/59974) in GitLab 12.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59974) in GitLab 12.1. By default, all projects include a GitLab-defined Prometheus dashboard, which includes a few key metrics, but you can also define your own custom dashboards. @@ -308,8 +317,8 @@ supported and will not be available in the UI. #### Duplicating a GitLab-defined dashboard -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/37238) in GitLab 12.7. -> - From [GitLab 12.8 onwards](https://gitlab.com/gitlab-org/gitlab/issues/39505), custom metrics are also duplicated when you duplicate a dashboard. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37238) in GitLab 12.7. +> - From [GitLab 12.8 onwards](https://gitlab.com/gitlab-org/gitlab/-/issues/39505), custom metrics are also duplicated when you duplicate a dashboard. You can save a complete copy of a GitLab defined dashboard along with all custom metrics added to it. Resulting `.yml` file can be customized and adapted to your project. @@ -326,6 +335,42 @@ new branch. If you select your **default** branch, the new dashboard becomes immediately available. If you select another branch, this branch should be merged to your **default** branch first. +#### Dashboard YAML syntax validation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33202) in GitLab 13.1. + +To confirm your dashboard definition contains valid YAML syntax: + +1. Navigate to **{doc-text}** **Repository > Files**. +1. Navigate to your dashboard file in your repository. +1. Review the information pane about the file, displayed above the file contents. + +Files with valid syntax display **Metrics Dashboard YAML definition is valid**, +and files with invalid syntax display **Metrics Dashboard YAML definition is invalid**. + + + +When **Metrics Dashboard YAML definition is invalid** at least one of the following messages is displayed: + +1. `dashboard: can't be blank` [learn more](#dashboard-top-level-properties) +1. `panel_groups: can't be blank` [learn more](#dashboard-top-level-properties) +1. `group: can't be blank` [learn more](#panel-group-panel_groups-properties) +1. `panels: can't be blank` [learn more](#panel-group-panel_groups-properties) +1. `metrics: can't be blank` [learn more](#panel-panels-properties) +1. `title: can't be blank` [learn more](#panel-panels-properties) +1. `query: can't be blank` [learn more](#metrics-metrics-properties) +1. `query_range: can't be blank` [learn more](#metrics-metrics-properties) +1. `unit: can't be blank` [learn more](#metrics-metrics-properties) +1. `YAML syntax: The parsed YAML is too big` + + This is displayed when the YAML file is larger than 1 MB. + +1. `YAML syntax: Invalid configuration format` + + This is displayed when the YAML file is empty or does not contain valid YAML. + +Metrics Dashboard YAML definition validation information is also available as a [GraphQL API field](../../../api/graphql/reference/index.md#metricsdashboard) + #### Dashboard YAML properties Dashboards have several components: @@ -342,16 +387,27 @@ The following tables outline the details of expected properties. | ------ | ------ | ------ | ------ | | `dashboard` | string | yes | Heading for the dashboard. Only one dashboard should be defined per file. | | `panel_groups` | array | yes | The panel groups which should be on the dashboard. | -| `templating` | Hash | no | Top level key under which templating related options can be added. | +| `templating` | hash | no | Top level key under which templating related options can be added. | +| `links` | array | no | Add links to display on the dashboard. | ##### **Templating (`templating`) properties** | Property | Type | Required | Description | | -------- | ---- | -------- | ----------- | -| `variables` | Hash | no | Variables can be defined here. | +| `variables` | hash | yes | Variables can be defined here. | Read the documentation on [templating](#templating-variables-for-metrics-dashboards). +##### **Links (`links`) properties** + +| Property | Type | Required | Description | +| -------- | ---- | -------- | ----------- | +| `url` | string | yes | The address of the link. | +| `title` | string | no | Display title for the link. | +| `type` | string | no | Type of the link. Specifies the link type, can be: `grafana` | + +Read the documentation on [links](#add-related-links-to-custom-dashboards). + ##### **Panel group (`panel_groups`) properties** | Property | Type | Required | Description | @@ -371,6 +427,7 @@ Read the documentation on [templating](#templating-variables-for-metrics-dashboa | `max_value` | number | no | Denominator value used for calculating [percentile based results](#percentile-based-results) | | `weight` | number | no, defaults to order in file | Order to appear within the grouping. Lower number means higher priority, which will be higher on the page. Numbers do not need to be consecutive. | | `metrics` | array | yes | The metrics which should be displayed in the panel. Any number of metrics can be displayed when `type` is `area-chart` or `line-chart`, whereas only 3 can be displayed when `type` is `anomaly-chart`. | +| `links` | array | no | Add links to display on the chart's [context menu](#chart-context-menu). | ##### **Axis (`panels[].y_axis`) properties** @@ -472,7 +529,7 @@ Note the following properties:  -Starting in [version 12.8](https://gitlab.com/gitlab-org/gitlab/issues/202696), the y-axis values will automatically scale according to the data. Previously, it always started from 0. +Starting in [version 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/202696), the y-axis values will automatically scale according to the data. Previously, it always started from 0. ##### Anomaly chart @@ -574,7 +631,7 @@ Note the following properties: ##### Stacked column -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30583) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30583) in GitLab 12.8. To add a stacked column panel type to a dashboard, look at the following sample dashboard file: @@ -639,7 +696,7 @@ Note the following properties: ###### Percentile based results -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/201946) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201946) in GitLab 12.8. Query results sometimes need to be represented as a percentage value out of 100. You can use the `max_value` property at the root of the panel definition: @@ -662,7 +719,7 @@ For example, if you have a query value of `53.6`, adding `%` as the unit results ##### Heatmaps -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30581) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30581) in GitLab 12.5. To add a heatmap panel type to a dashboard, look at the following sample dashboard file: @@ -699,7 +756,7 @@ Templating variables can be used to make your metrics dashboard more versatile. [dashboard YAML](#dashboard-top-level-properties). Define your variables in the `variables` key, under `templating`. The value of the `variables` key should be a hash, and each key under `variables` -defines a templating variable on the dashboard. +defines a templating variable on the dashboard, and may contain alphanumeric and underscore characters. A variable can be used in a Prometheus query in the same dashboard using the syntax described [here](#using-variables). @@ -766,7 +823,7 @@ templating: ###### Full syntax -This example creates a variable called `variable1`, with a default value of `var1_option_2`. +This example creates a variable called `variable1`, with a default value of `value_option_2`. The label for the text box on the UI will be the value of the `label` key. The dashboard UI will display a dropdown with `Option 1` and `Option 2` as the choices. @@ -789,9 +846,50 @@ templating: default: true # (Optional) This option should be the default value of this variable. ``` +### Add related links to custom dashboards + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216385) in GitLab 13.1. + +You can embed links to other dashboards or external services in your custom +dashboard by adding **Related links** to your dashboard's YAML file. Related links +open in the same tab as the dashboard. Related links can be displayed in the +following locations on your dashboard: + +- At the top of your dashboard as the top level [`links` dashboard property](#dashboard-top-level-properties). +- In charts context menus as the [`links` property of a panel](#panel-panels-properties). + +Related links can contain the following attributes: + +- `url`: The full URL to the link. Required. +- `title`: A phrase describing the link. Optional. If this attribute is not set, + the full URL is used for the link title. +- `type`: A string declaring the type of link. Optional. If set to `grafana`, the + dashboard's time range values are converted to Grafana's time range format and + appended to the `url`. + +The dashboard's time range is appended to the `url` as URL parameters. + +The following example shows two related links (`GitLab.com` and `GitLab Documentation`) +added to a dashboard: + + + +#### Links Syntax + +```yaml +links: + - title: GitLab.com + url: https://gitlab.com + - title: GitLab Documentation + url: https://docs.gitlab.com + - title: Public Grafana playground dashboard + url: https://play.grafana.org/d/000000012/grafana-play-home?orgId=1 + type: grafana +``` + ### View and edit the source file of a custom dashboard -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/34779) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34779) in GitLab 12.5. When viewing a custom dashboard of a project, you can view the original `.yml` file by clicking on the **Edit dashboard** button. @@ -812,7 +910,7 @@ The options are: ### Dashboard Annotations -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/211330) in GitLab 12.10 (enabled by feature flag `metrics_dashboard_annotations`). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211330) in GitLab 12.10 (enabled by feature flag `metrics_dashboard_annotations`). > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/215224) in GitLab 13.0. You can use **Metrics Dashboard Annotations** to mark any important events on @@ -827,6 +925,14 @@ You can create annotations by making requests to the  +#### Retention policy + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211433) in GitLab 13.01. + +To avoid excessive storage space consumption by stale annotations, records attached +to time periods older than two weeks are removed daily. This recurring background +job runs at 1:00 a.m. local server time. + ### Expand panel > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3100) in GitLab 13.0. @@ -839,7 +945,7 @@ browser, or pressing the <kbd>Esc</kbd> key. ### View Logs **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/122013) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/122013) in GitLab 12.8. If you have [Logs](../clusters/kubernetes_pod_logs.md) enabled, you can navigate from the charts in the dashboard to view Logs by @@ -881,8 +987,8 @@ To remove the alert, click back on the alert icon for the desired metric, and cl #### External Prometheus instances ->- [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8. ->- [Moved](https://gitlab.com/gitlab-org/gitlab/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.10. +>- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8. +>- [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.10. For manually configured Prometheus servers, a notify endpoint is provided to use with Prometheus webhooks. If you have manual configuration enabled, an **Alerts** section is added to **Settings > Integrations > Prometheus**. This contains the *URL* and *Authorization Key*. The **Reset Key** button will invalidate the key and generate a new one. @@ -903,12 +1009,15 @@ receivers: In order for GitLab to associate your alerts with an [environment](../../../ci/environments/index.md), you need to configure a `gitlab_environment_name` label on the alerts you set up in Prometheus. The value of this should match the name of your Environment in GitLab. +NOTE: **Note** +In GitLab versions 13.1 and greater, you can configure your manually configured Prometheus server to use the [Generic alerts integration](generic_alerts.md). + ### Taking action on incidents **(ULTIMATE)** ->- [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. ->- [From GitLab Ultimate 12.5](https://gitlab.com/gitlab-org/gitlab/issues/13401), when GitLab receives a recovery alert, it will automatically close the associated issue. +>- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. +>- [From GitLab Ultimate 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/13401), when GitLab receives a recovery alert, it will automatically close the associated issue. -Alerts can be used to trigger actions, like opening an issue automatically (enabled by default since `12.1`). To configure the actions: +Alerts can be used to trigger actions, like opening an issue automatically (disabled by default since `13.1`). To configure the actions: 1. Navigate to your project's **Settings > Operations > Incidents**. 1. Enable the option to create issues. @@ -931,14 +1040,14 @@ When GitLab receives a **Recovery Alert**, it will automatically close the assoc To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template, which will apply to all incidents. To limit quick actions or other information to only specific types of alerts, use the `annotations/gitlab_incident_markdown` field. -Since [version 12.2](https://gitlab.com/gitlab-org/gitlab-foss/issues/63373), GitLab will tag each incident issue with the `incident` label automatically. If the label does not yet exist, it will be created automatically as well. +Since [version 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63373), GitLab will tag each incident issue with the `incident` label automatically. If the label does not yet exist, it will be created automatically as well. If the metric exceeds the threshold of the alert for over 5 minutes, an email will be sent to all [Maintainers and Owners](../../permissions.md#project-members-permissions) of the project. ## Determining the performance impact of a merge > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10408) in GitLab 9.2. -> - GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-foss/issues/27439) of the 30 minute averages. +> - GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27439) of the 30 minute averages. Developers can view the performance impact of their changes within the merge request workflow. @@ -1031,7 +1140,7 @@ For [manually configured Prometheus instances](#manual-configuration-of-promethe ### Embedding Cluster Health Charts **(ULTIMATE)** -> [Introduced](<https://gitlab.com/gitlab-org/gitlab/issues/40997>) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. +> [Introduced](<https://gitlab.com/gitlab-org/gitlab/-/issues/40997>) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. [Cluster Health Metrics](../clusters/index.md#monitoring-your-kubernetes-cluster-ultimate) can also be embedded in [GitLab-flavored Markdown](../../markdown.md). @@ -1075,7 +1184,7 @@ This will render like so: #### Embedding charts via integration with Grafana HTTP API -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/31376) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31376) in GitLab 12.5. Each project can support integration with one Grafana instance. This configuration allows a user to copy a link to a panel in Grafana, then paste it into a GitLab Markdown field. The chart will be rendered in the GitLab chart format. |