Merge branch '21170-cycle-analytics' into 'master'

Cycle Analytics: first iteration

## What does this MR do?

- Implement the first iteration of the "Cycle Analytics" feature.

## What are the relevant issue numbers?

- Closes #21170 

## Screenshots

![cycle_analytics_screencast.gif](/uploads/d23c3c912caa6935fd47b53ca3a56b97/cycle_analytics.gif)

## Backend Tasks

- [x]  Implementation
    - [x]  Phases
        - [x]  Issue (Tracker)
        - [x]  Plan (Board)
        - [x]  Code (IDE)
        - [x]  Test (CI)
        - [x]  Review (MR)
        - [x]  Staging (CD)
        - [x]  Production (Total)
    - [x]  Make heuristics more modular
    - [x]  Scope to project
    - [x]  Date range (30 days, 90 days)
    - [x]  Access restriction
- [x]  Test
    - [x]  Find a better way to test these phases
    - [x]  Phases
        - [x]  Issue (Tracker)
        - [x]  Plan (Board)
        - [x]  Code (IDE)
        - [x]  Test (CI)
        - [x]  Review (MR)
        - [x]  Staging (CD)
        - [x]  Production (Total)
    - [x]  Test for "end case happens before start case"
    - [x]  Consolidate helper
- [x]  Miniboss review
- [x]  Performance testing with mock data
- [x]  Improve performance
    - [x]  Pre-calculate "merge requests closing issues
    - [x]  Pre-calculate everything else
- [x]  Test performance against 10k issues
- [x]  Test all pre-calculation code
    - [x]  Ci::Pipeline -> build start/finish
    - [x]  Ci::Pipeline#merge_requests
    - [x]  Issue -> record default metrics after save
    - [x]  MergeRequest -> record default metrics after save
    - [x]  Deployment -> Update "first_deployed_to_production_at" for MR metrics
    - [x]  Git Push -> Update "first commit mention" for issue metrics
    - [x]  Merge request create/update/refresh -> Update "merge requests closing issues"
- [x]  Remove `MergeRequestsClosingIssues` when necessary
- [x]  Changes to unblock Fatih
    - [x]  Add summary data
    - [x]  `stats` should be array
    - [x]  Let `stats` be `null` if all `stats` are null
- [x]  Indexes for "merge requests closing issues"
- [x]  Test summary data
- [x]  Scope everything to project
    - [x]  Find out why tests were passing
- [x]  Filter should include issues/MRs which have made it to production within the range
- [x]  Don't create duplicate `MergeRequestsClosingIssues`
- [x]  Fix tests
- [x]  MySQL median
- [x]  Assign to Douwe for review
- [x]  Fix conflicts
- [x]  Implement suggestions from Yorick's review
    - [x]  Test on PG
    - [x]  Test on MySQL
- [x]  Refactor
    - [x]  Cleanup
        - [x]  What happens if we have no data at all?
        - [x]  Extract common queries to methods / scopes
    - [x]  Remove unused queries
    - [x]  Downtime for foreign key migrations
    - [x]  Find a way around "if issue.metrics.present?" all over the place
    - [x]  Find a way around "if merge_request.metrics.present?" all over the place
    - [x]  Test migrations on a fresh database
        - [x]  MySQL
        - [x]  Pg
- [x]  Access issues
    - While the project is public and the visibility is set to "Everyone with access", you cannot visit the cycle analytics page when signed out.
- [x]  CHANGELOG
- [x]  Implement suggestions from Douwe's review
    - [x]  First set of comments
    - [x]  Second set of comments
    - [x]  Third set of comments
    - [x]  Fourth set of comments
- [x]  Make sure build is green
- [ ]  Make issue for "polish"
- [ ]  EE MR


See merge request !5986

6 files changed, 32 insertions, 1 deletions

diff --git a/app/services/create_deployment_service.rb b/app/services/create_deployment_service.rb
index e6667132e27..799ad3e1bd0 100644
--- a/app/services/create_deployment_service.rb
+++ b/app/services/create_deployment_service.rb
@@ -4,7 +4,7 @@ class CreateDeploymentService < BaseService
   def execute(deployable = nil)
     environment = find_or_create_environment
 
-    project.deployments.create(
+    deployment = project.deployments.create(
       environment: environment,
       ref: params[:ref],
       tag: params[:tag],
@@ -12,6 +12,10 @@ class CreateDeploymentService < BaseService
       user: current_user,
       deployable: deployable
     )
+
+    deployment.update_merge_request_metrics!
+
+    deployment
   end
 
   private
diff --git a/app/services/git_push_service.rb b/app/services/git_push_service.rb
index 948041063c0..c499427605a 100644
--- a/app/services/git_push_service.rb
+++ b/app/services/git_push_service.rb
@@ -134,6 +134,7 @@ class GitPushService < BaseService
       end
 
       commit.create_cross_references!(authors[commit], closed_issues)
+      update_issue_metrics(commit, authors)
     end
   end
 
@@ -186,4 +187,11 @@ class GitPushService < BaseService
   def branch_name
     @branch_name ||= Gitlab::Git.ref_name(params[:ref])
   end
+
+  def update_issue_metrics(commit, authors)
+    mentioned_issues = commit.all_references(authors[commit]).issues
+
+    Issue::Metrics.where(issue_id: mentioned_issues.map(&:id), first_mentioned_in_commit_at: nil).
+      update_all(first_mentioned_in_commit_at: commit.committed_date)
+  end
 end
diff --git a/app/services/issuable_base_service.rb b/app/services/issuable_base_service.rb
index 4c8d93999a7..fbce46769f7 100644
--- a/app/services/issuable_base_service.rb
+++ b/app/services/issuable_base_service.rb
@@ -157,6 +157,10 @@ class IssuableBaseService < BaseService
     # To be overridden by subclasses
   end
 
+  def after_update(issuable)
+    # To be overridden by subclasses
+  end
+
   def update_issuable(issuable, attributes)
     issuable.with_transaction_returning_status do
       issuable.update(attributes.merge(updated_by: current_user))
@@ -182,6 +186,7 @@ class IssuableBaseService < BaseService
       end
 
       handle_changes(issuable, old_labels: old_labels, old_mentioned_users: old_mentioned_users)
+      after_update(issuable)
       issuable.create_new_cross_references!(current_user)
       execute_hooks(issuable, 'update')
     end
diff --git a/app/services/merge_requests/create_service.rb b/app/services/merge_requests/create_service.rb
index 73247e62421..b0ae2dfe4ce 100644
--- a/app/services/merge_requests/create_service.rb
+++ b/app/services/merge_requests/create_service.rb
@@ -20,6 +20,7 @@ module MergeRequests
       event_service.open_mr(issuable, current_user)
       notification_service.new_merge_request(issuable, current_user)
       todo_service.new_merge_request(issuable, current_user)
+      issuable.cache_merge_request_closes_issues!(current_user)
     end
   end
 end
diff --git a/app/services/merge_requests/refresh_service.rb b/app/services/merge_requests/refresh_service.rb
index 5cedd6f11d9..22596b4014a 100644
--- a/app/services/merge_requests/refresh_service.rb
+++ b/app/services/merge_requests/refresh_service.rb
@@ -13,6 +13,7 @@ module MergeRequests
       reload_merge_requests
       reset_merge_when_build_succeeds
       mark_pending_todos_done
+      cache_merge_requests_closing_issues
 
       # Leave a system note if a branch was deleted/added
       if branch_added? || branch_removed?
@@ -141,6 +142,14 @@ module MergeRequests
       end
     end
 
+    # If the merge requests closes any issues, save this information in the
+    # `MergeRequestsClosingIssues` model (as a performance optimization).
+    def cache_merge_requests_closing_issues
+      @project.merge_requests.where(source_branch: @branch_name).each do |merge_request|
+        merge_request.cache_merge_request_closes_issues!(@current_user)
+      end
+    end
+
     def filter_merge_requests(merge_requests)
       merge_requests.uniq.select(&:source_project)
     end
diff --git a/app/services/merge_requests/update_service.rb b/app/services/merge_requests/update_service.rb
index 398ec47f0ea..f14f9e4b327 100644
--- a/app/services/merge_requests/update_service.rb
+++ b/app/services/merge_requests/update_service.rb
@@ -77,5 +77,9 @@ module MergeRequests
     def close_service
       MergeRequests::CloseService
     end
+
+    def after_update(issuable)
+      issuable.cache_merge_request_closes_issues!(current_user)
+    end
   end
 end