Enable DAG support by default

Closes #65457

See merge request gitlab-org/gitlab-ce!31814

This toggles the ci_dag_support flag to be on by default.
This relies on ci_dag_limit_needs to be present to reduce
amount of inter-dependencies between jobs

Extend PipelineProcessWorker to accept a list of builds

See merge request gitlab-org/gitlab-ce!31425

This changes used worker from `BuildProcessWorker`
to `PipelineProcessWorker` to make pipeline
processing much simpler. We process `pipeline_id`,
based on some triggers.

Currently, some of the jobs with `needs:`
would be processed in stages, it means
that `when:` for such jobs would not be
respected.

This changes the behavior to have a separate
execution paths for jobs with `needs:`.

This implements the support for `needs:` keyword
as part of GitLab CI. That makes some of the jobs
to be run out of order.

Adds Job specific variables to facilitate specifying variables when
running manual jobs.

Convert several occurrences of `map` + `flatten` to
`flat_map` where applicable.

Often live traces are removed even though the archived trace
doesn't exist. This commit checkes the existence strictly.