Merge branch 'refactor-builds-creation-service' into 'master'

Refactor pipeline creation service

## What does this MR do?
This refactors GitLab CI build processing: all builds for pipeline are pre-created when a pipeline object is created. 
The builds are created with a new introduced status `created`.
The builds are then automatically promoted to `pending` when a previous stage do succeed.
This significantly simplifies pipeline processing code solving a lot of problems of lazily initialisation of previous approach (builds were created on-demand).

## Why was this MR needed?
The previous mechanism had a lot of flows (shown in related issues) in how it work, but also in code design. Removing cross model-service-library dependencies.

The current approach moves a build creation to single place `CreatePipelineService` and removes a dynamic dependency on `config_processor` significantly simplifying a build creation and pipeline processing. Pipeline processing is implemented in `ProcessPipelineService`.

This also allows to easily extend GitLab with Manual Actions which is part of 8.10 direction issue.

## Migration problem
~~This MR removes the a on-demand creation of builds in pipelines.
 Pipelines that are running and are in mid-stage (some stages started, but not all) will not be fully evaluated after application restart. 
This happens, because the code responsible for on-demand creation is removed. 
There's no easy way to migrate existing pipelines, other than doing offline migration and putting pipeline processing in migration code (which seems to be a really bad idea).~~

To support old pipelines I added a lazy initialization of builds if none is found.

## What are the relevant issue numbers?
Fixes: https://gitlab.com/gitlab-org/gitlab-ce/issues/12839
Solves: https://gitlab.com/gitlab-org/gitlab-ce/issues/18644 https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/289
Allows to easily implement: https://gitlab.com/gitlab-org/gitlab-ce/issues/17010

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- Tests
  - [x] Added for this feature/bug
  - [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)


See merge request !5295

3 files changed, 12 insertions, 10 deletions

diff --git a/spec/lib/ci/charts_spec.rb b/spec/lib/ci/charts_spec.rb
index 034ea098193..2cd6b00dad6 100644
--- a/spec/lib/ci/charts_spec.rb
+++ b/spec/lib/ci/charts_spec.rb
@@ -5,6 +5,7 @@ describe Ci::Charts, lib: true do
     before do
       @pipeline = FactoryGirl.create(:ci_pipeline)
       FactoryGirl.create(:ci_build, pipeline: @pipeline)
+      @pipeline.reload_status!
     end
 
     it 'returns build times in minutes' do
diff --git a/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb b/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb
index 85374b8761d..be51d942af7 100644
--- a/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb
+++ b/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb
@@ -19,7 +19,7 @@ module Ci
         expect(config_processor.builds_for_stage_and_ref(type, "master").first).to eq({
           stage: "test",
           stage_idx: 1,
-          name: :rspec,
+          name: "rspec",
           commands: "pwd\nrspec",
           tag_list: [],
           options: {},
@@ -433,7 +433,7 @@ module Ci
         expect(config_processor.builds_for_stage_and_ref("test", "master").first).to eq({
           stage: "test",
           stage_idx: 1,
-          name: :rspec,
+          name: "rspec",
           commands: "pwd\nrspec",
           tag_list: [],
           options: {
@@ -461,7 +461,7 @@ module Ci
         expect(config_processor.builds_for_stage_and_ref("test", "master").first).to eq({
           stage: "test",
           stage_idx: 1,
-          name: :rspec,
+          name: "rspec",
           commands: "pwd\nrspec",
           tag_list: [],
           options: {
@@ -700,7 +700,7 @@ module Ci
         expect(config_processor.builds_for_stage_and_ref("test", "master").first).to eq({
           stage: "test",
           stage_idx: 1,
-          name: :rspec,
+          name: "rspec",
           commands: "pwd\nrspec",
           tag_list: [],
           options: {
@@ -837,7 +837,7 @@ module Ci
           expect(subject.first).to eq({
             stage: "test",
             stage_idx: 1,
-            name: :normal_job,
+            name: "normal_job",
             commands: "test",
             tag_list: [],
             options: {},
@@ -882,7 +882,7 @@ module Ci
           expect(subject.first).to eq({
             stage: "build",
             stage_idx: 0,
-            name: :job1,
+            name: "job1",
             commands: "execute-script-for-job",
             tag_list: [],
             options: {},
@@ -894,7 +894,7 @@ module Ci
           expect(subject.second).to eq({
             stage: "build",
             stage_idx: 0,
-            name: :job2,
+            name: "job2",
             commands: "execute-script-for-job",
             tag_list: [],
             options: {},
diff --git a/spec/lib/gitlab/badge/build_spec.rb b/spec/lib/gitlab/badge/build_spec.rb
index ef9d9e7fef4..bb8144d5122 100644
--- a/spec/lib/gitlab/badge/build_spec.rb
+++ b/spec/lib/gitlab/badge/build_spec.rb
@@ -96,9 +96,10 @@ describe Gitlab::Badge::Build do
   end
 
   def create_build(project, sha, branch)
-    pipeline = create(:ci_pipeline, project: project,
-                                    sha: sha,
-                                    ref: branch)
+    pipeline = create(:ci_empty_pipeline,
+                      project: project,
+                      sha: sha,
+                      ref: branch)
 
     create(:ci_build, pipeline: pipeline, stage: 'notify')
   end