1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
|
---
stage: Monitor
group: Respond
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Manage incidents **(FREE)**
This page collects instructions for all the things you can do with [incidents](incidents.md) or in relation to them.
## Create an incident
You can create an incident manually or automatically.
### From the incidents list
> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3.
> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5.
> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8.
Prerequisites:
- You must have at least the Reporter role for the project.
To create an incident from the incidents list:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Monitor > Incidents**.
1. Select **Create incident**.
### From the issues list
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4.
Prerequisites:
- You must have at least the Reporter role for the project.
To create an incident from the issues list:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Issues > List**, and select **New issue**.
1. From the **Type** dropdown list, select **Incident**. Only fields relevant to
incidents are available on the page.
1. Select **Create issue**.
### From an alert
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1.
Create an incident issue when viewing an [alert](alerts.md).
The incident description is populated from the alert.
Prerequisites:
- You must have at least the Developer role for the project.
To create an incident from an alert:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Monitor > Alerts**.
1. Select your desired alert.
1. Select **Create incident**.
After an incident is created, to view it from the alert, select **View incident**.
When you [close an incident](#close-an-incident) linked to an alert, GitLab
[changes the alert's status](alerts.md#change-an-alerts-status) to **Resolved**.
You are then credited with the alert's status change.
### Automatically, when an alert is triggered **(ULTIMATE)**
In the project settings, you can turn on [creating an incident automatically](../metrics/alerts.md#trigger-actions-from-alerts)
whenever an alert is triggered.
### Using the PagerDuty webhook
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3.
> - [PagerDuty V3 Webhook](https://support.pagerduty.com/docs/webhooks) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383029) in GitLab 15.7.
You can set up a webhook with PagerDuty to automatically create a GitLab incident
for each PagerDuty incident. This configuration requires you to make changes
in both PagerDuty and GitLab.
Prerequisites:
- You must have at least the Maintainer role for the project.
To set up a webhook with PagerDuty:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Monitor**
1. Expand **Incidents**.
1. Select the **PagerDuty integration** tab.
1. Turn on the **Active** toggle.
1. Select **Save integration**.
1. Copy the value of **Webhook URL** for use in a later step.
1. To add the webhook URL to a PagerDuty webhook integration, follow the steps described in the [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks#manage-v3-webhook-subscriptions).
To confirm the integration is successful, trigger a test incident from PagerDuty to
check if a GitLab incident is created from the incident.
## View incidents list
To view the [incidents list](incidents.md#incidents-list):
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Monitor > Incidents**.
To view an incident's [details page](incidents.md#incident-details), select it from the list.
### Who can view an incident
Whether you can view an incident depends on the [project visibility level](../../user/public_access.md) and
the incident's confidentiality status:
- Public project and a non-confidential incident: You don't have to be a member of the project.
- Private project and non-confidential incident: You must have at least the Guest role for the project.
- Confidential incident (regardless of project visibility): You must have at least the Reporter role for the project.
## Assign to a user
Assign incidents to users that are actively responding.
Prerequisites:
- You must have at least the Reporter role for the project.
To assign a user:
1. In an incident, on the right sidebar, next to **Assignees**, select **Edit**.
1. From the dropdown list, select one or [multiple users](../../user/project/issues/multiple_assignees_for_issues.md) to add as **assignees**.
1. Select any area outside the dropdown list.
## Change severity
> Editing severity on incident details page was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4.
See [incident list](incidents.md#incidents-list) for a full description of the severity levels available.
Prerequisites:
- You must have at least the Reporter role for the project.
To change an incident's severity:
1. In an incident, on the right sidebar, next to **Severity**, select **Edit**.
1. From the dropdown list, select the new severity.
You can also change the severity using the `/severity` [quick action](../../user/project/quick_actions.md).
## Change status
> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default.
> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10.
> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1.
Prerequisites:
- You must have at least the Developer role for the project.
To change the status of an incident:
1. In an incident, on the right sidebar, next to **Status**, select **Edit**.
1. From the dropdown list, select the new severity.
**Triggered** is the default status for new incidents.
### As an on-call responder **(PREMIUM)**
On-call responders can respond to [incident pages](paging.md#escalating-an-incident)
by changing the status.
Changing the status has the following effects:
- To **Acknowledged**: limits on-call pages based on the project's [escalation policy](escalation_policies.md).
- To **Resolved**: silences all on-call pages for the incident.
- From **Resolved** to **Triggered**: restarts the incident escalating.
In GitLab 15.1 and earlier, changing the status of an [incident created from an alert](#from-an-alert)
also changes the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057),
the alert status is independent and does not change when the incident status changes.
## Change escalation policy **(PREMIUM)**
Prerequisites:
- You must have at least the Developer role for the project.
To change the escalation policy of an incident:
1. In an incident, on the right sidebar, next to **Escalation policy**, select **Edit**.
1. From the dropdown list, select the escalation policy.
By default, new incidents do not have an escalation policy selected.
Selecting an escalation policy [changes the incident status](#change-status) to **Triggered** and begins
[escalating the incident to on-call responders](paging.md#escalating-an-incident).
In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](#from-an-alert)
reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057),
the incident escalation policy is independent and can be changed.
## Embed metrics
You can embed metrics anywhere [GitLab Flavored Markdown](../../user/markdown.md) is
used, like descriptions or comments. Embedding
metrics helps you share them when discussing incidents or performance issues.
To embed metrics in a Markdown text box in GitLab,
[paste the link to the dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics).
You can embed both [GitLab-hosted metrics](../metrics/embed.md) (deprecated) and
[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue
templates.
## Close an incident
Prerequisites:
- You must have at least the Reporter role for the project.
To close an incident, in the upper-right corner, select **Close incident**.
When you close an incident that is linked to an [alert](alerts.md),
the linked alert's status changes to **Resolved**.
You are then credited with the alert's status change.
### Automatically close incidents via recovery alerts
> [Introduced for HTTP integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4.
Turn on closing an incident automatically when GitLab receives a recovery alert
from a HTTP or Prometheus webhook.
Prerequisites:
- You must have at least the Maintainer role for the project.
To configure the setting:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Monitor**.
1. Expand the **Incidents** section.
1. Select the **Automatically close associated incident** checkbox.
1. Select **Save changes**.
When GitLab receives a recovery alert, it closes the associated incident.
This action is recorded as a system note on the incident indicating that it
was closed automatically by the GitLab Alert bot.
## Other actions
Because incidents in GitLab are built on top of [issues](../../user/project/issues/index.md),
they have the following actions in common:
- [Add a to-do item](../../user/todos.md#create-a-to-do-item)
- [Add labels](../../user/project/labels.md#assign-and-unassign-labels)
- [Assign a milestone](../../user/project/milestones/index.md#assign-a-milestone-to-an-issue-or-merge-request)
- [Make an incident confidential](../../user/project/issues/confidential_issues.md)
- [Set a due date](../../user/project/issues/due_dates.md)
- [Toggle notifications](../../user/profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics)
- [Track time spent](../../user/project/time_tracking.md)
|
