Merge branch 'separate-sidekiq-queues' into 'master'

Use separate queues for all Sidekiq workers

## What does this MR do?

This MR updates all workers so that they (mostly) use their own Sidekiq queues. This in turn allows us to monitor queues more accurately and in the future impose queue specific throttles, limits, etc.

This is a critical part we need in 8.13, despite it being so close to release day.

See https://gitlab.com/gitlab-org/gitlab-ce/issues/23370 for more information.

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG.md) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if it does - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

https://gitlab.com/gitlab-org/gitlab-ce/issues/23370

See merge request !7006

2 files changed, 40 insertions, 1 deletions

diff --git a/doc/development/README.md b/doc/development/README.md
index 9706cb1de7f..fb6a8a5b095 100644
--- a/doc/development/README.md
+++ b/doc/development/README.md
@@ -14,7 +14,8 @@
 - [Testing standards and style guidelines](testing.md)
 - [UI guide](ui_guide.md) for building GitLab with existing CSS styles and elements
 - [Frontend guidelines](frontend.md)
-- [SQL guidelines](sql.md) for SQL guidelines
+- [SQL guidelines](sql.md) for working with SQL queries
+- [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers
 
 ## Process
 
diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md
new file mode 100644
index 00000000000..e3a20f29a09
--- /dev/null
+++ b/doc/development/sidekiq_style_guide.md
@@ -0,0 +1,38 @@
+# Sidekiq Style Guide
+
+This document outlines various guidelines that should be followed when adding or
+modifying Sidekiq workers.
+
+## Default Queue
+
+Use of the "default" queue is not allowed. Every worker should use a queue that
+matches the worker's purpose the closest. For example, workers that are to be
+executed periodically should use the "cronjob" queue.
+
+A list of all available queues can be found in `config/sidekiq_queues.yml`.
+
+## Dedicated Queues
+
+Most workers should use their own queue. To ease this process a worker can
+include the `DedicatedSidekiqQueue` concern as follows:
+
+```ruby
+class ProcessSomethingWorker
+  include Sidekiq::Worker
+  include DedicatedSidekiqQueue
+end
+```
+
+This will set the queue name based on the class' name, minus the `Worker`
+suffix. In the above example this would lead to the queue being
+`process_something`.
+
+In some cases multiple workers do use the same queue. For example, the various
+workers for updating CI pipelines all use the `pipeline` queue. Adding workers
+to existing queues should be done with care, as adding more workers can lead to
+slow jobs blocking work (even for different jobs) on the shared queue.
+
+## Tests
+
+Each Sidekiq worker must be tested using RSpec, just like any other class. These
+tests should be placed in `spec/workers`.