Merge branch 'new-diff-notes' into 'master'

New diff notes

Fixes #12732, #14731, #19375, #14783 

Builds on https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4110

To do:
- [x] Get it mostly working
- [x] Validate position validity
- [x] Fix: Don’t link to `#`
- [x] Fix: Base ref can be `nil`, potentially, when the MR has an oprhan source branch => Yep, doesn’t work. We need to store a `start_id`
- [x] Optimize: Fewer duplicate `git diff` compares
- [x] Optimize: Pass paths to `PositionTracer#diff` for faster diffs
- [x] Refactor: Use `head_id` in `MergeRequest`/`MergeRequestDiff` instead of `source_sha`
- [x] Refactor: Convert existing array-based diff refs to the DiffRefs model
- [x] Tweak: Use `note_type` in `Autosave` key
- [x] Tweak: Remove `line_code: note.line_code` from `link_to_reply_discussion`
- [x] Update: `SentNotifications` and reply-by-email receiver
- [x] Update: MR diff notification email
- [x] Update: API (MR, Commit note creation and entity)
- [x] Update: GitHub importer
- [x] Address any other TODO comments
- [x] Fix: Suppress "edited 4 minutes ago"
- [x] Write tests
  - [x] `LineMapper`
  - [x] `PositionTracer`
  - [x] `Position`
  - [x] `DiffPositionUpdateService`
  - [x] `DiffNote`
  - [x] `MergeRequests::RefreshService` / `MergeRequest#update_diff_notes_positions`
- [x] Make sure commits with diff notes don't get cleaned up, since this would prevent the diff notes from being rendered (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5062)

Future improvements:
- Display unresolved comments on files outside the diff, if the comment was added when that file _was_ part of the diff
- Allow commenting on sections between hunks, when expanding the diff using `...`
  - (We'd need to generate line code based on Position if we have it, even if it falls outside bounds of diff)
- `diff_hunk` on diff note API entity
- Show diff hunk in notification email
- Resolved line notes would have a boolean, and be inactive through `notes.any? { !active? || resolved? }`
- Multi line notes would store a number of positions, and do the right thing (™) in grouping and then rendering if the first item is multiline? => true
- Image diff notes could store x,y,width,height instead of old_line,new_line for similar grouping. Does it need a reference to say if it's on old or new? These can't have line_codes, clearly. Rendering would be interesting.
- Show commit line comments in the MR diff
- Comment on specific selected words
- Comment on file header
- Unfold top of discussion diff note
- New diff notes API for commits and MRs

/cc @rspeicher

See merge request !4101

15 files changed, 580 insertions, 83 deletions

diff --git a/lib/api/entities.rb b/lib/api/entities.rb
index 8cc4368b5c2..db877d2eeb0 100644
--- a/lib/api/entities.rb
+++ b/lib/api/entities.rb
@@ -240,9 +240,9 @@ module API
 
     class CommitNote < Grape::Entity
       expose :note
-      expose(:path) { |note| note.diff_file_path if note.legacy_diff_note? }
-      expose(:line) { |note| note.diff_new_line if note.legacy_diff_note? }
-      expose(:line_type) { |note| note.diff_line_type if note.legacy_diff_note? }
+      expose(:path) { |note| note.diff_file.try(:file_path) if note.diff_note? }
+      expose(:line) { |note| note.diff_line.try(:new_line) if note.diff_note? }
+      expose(:line_type) { |note| note.diff_line.try(:type) if note.diff_note? }
       expose :author, using: Entities::UserBasic
       expose :created_at
     end
diff --git a/lib/api/merge_requests.rb b/lib/api/merge_requests.rb
index 0e94efd4acd..4fcdf8968c9 100644
--- a/lib/api/merge_requests.rb
+++ b/lib/api/merge_requests.rb
@@ -233,8 +233,8 @@ module API
 
           render_api_error!('Branch cannot be merged', 406) unless merge_request.mergeable?
 
-          if params[:sha] && merge_request.source_sha != params[:sha]
-            render_api_error!("SHA does not match HEAD of source branch: #{merge_request.source_sha}", 409)
+          if params[:sha] && merge_request.diff_head_sha != params[:sha]
+            render_api_error!("SHA does not match HEAD of source branch: #{merge_request.diff_head_sha}", 409)
           end
 
           merge_params = {
diff --git a/lib/gitlab/diff/diff_refs.rb b/lib/gitlab/diff/diff_refs.rb
new file mode 100644
index 00000000000..8406ca4269c
--- /dev/null
+++ b/lib/gitlab/diff/diff_refs.rb
@@ -0,0 +1,36 @@
+module Gitlab
+  module Diff
+    class DiffRefs
+      attr_reader :base_sha
+      attr_reader :start_sha
+      attr_reader :head_sha
+
+      def initialize(base_sha:, start_sha: base_sha, head_sha:)
+        @base_sha = base_sha
+        @start_sha = start_sha
+        @head_sha = head_sha
+      end
+
+      def ==(other)
+        other.is_a?(self.class) &&
+          base_sha == other.base_sha &&
+          start_sha == other.start_sha &&
+          head_sha == other.head_sha
+      end
+
+      # There is only one case in which we will have `start_sha` and `head_sha`,
+      # but not `base_sha`, which is when a diff is generated between an
+      # orphaned branch and another branch, which means there _is_ no base, but
+      # we're still able to highlight it, and to create diff notes, which are
+      # the primary things `DiffRefs` are used for.
+      # `DiffRefs` are "complete" when they have `start_sha` and `head_sha`,
+      # because `base_sha` can always be derived from this, to return an actual
+      # sha, or `nil`.
+      # We have `base_sha` directly available on `DiffRefs` because it's faster#
+      # than having to look it up in the repo every time.
+      def complete?
+        start_sha && head_sha
+      end
+    end
+  end
+end
diff --git a/lib/gitlab/diff/file.rb b/lib/gitlab/diff/file.rb
index d2e85cabf72..b0c50edba59 100644
--- a/lib/gitlab/diff/file.rb
+++ b/lib/gitlab/diff/file.rb
@@ -1,47 +1,83 @@
 module Gitlab
   module Diff
     class File
-      attr_reader :diff, :diff_refs
+      attr_reader :diff, :repository, :diff_refs
 
       delegate :new_file, :deleted_file, :renamed_file,
-        :old_path, :new_path, to: :diff, prefix: false
+        :old_path, :new_path, :a_mode, :b_mode,
+        :submodule?, :too_large?, to: :diff, prefix: false
 
-      def initialize(diff, diff_refs)
+      def initialize(diff, repository:, diff_refs: nil)
         @diff = diff
+        @repository = repository
         @diff_refs = diff_refs
       end
 
+      def position(line)
+        return unless diff_refs
+
+        Position.new(
+          old_path: old_path,
+          new_path: new_path,
+          old_line: line.old_line,
+          new_line: line.new_line,
+          diff_refs: diff_refs
+        )
+      end
+
+      def line_code(line)
+        return if line.meta?
+
+        Gitlab::Diff::LineCode.generate(file_path, line.new_pos, line.old_pos)
+      end
+
+      def line_for_line_code(code)
+        diff_lines.find { |line| line_code(line) == code }
+      end
+
+      def line_for_position(pos)
+        diff_lines.find { |line| position(line) == pos }
+      end
+
+      def position_for_line_code(code)
+        line = line_for_line_code(code)
+        position(line) if line
+      end
+
+      def line_code_for_position(pos)
+        line = line_for_position(pos)
+        line_code(line) if line
+      end
+
+      def content_commit
+        return unless diff_refs
+
+        repository.commit(deleted_file ? old_ref : new_ref)
+      end
+
       def old_ref
-        diff_refs[0] if diff_refs
+        diff_refs.try(:base_sha)
       end
 
       def new_ref
-        diff_refs[1] if diff_refs
+        diff_refs.try(:head_sha)
       end
 
-      # Array of Gitlab::DIff::Line objects
+      # Array of Gitlab::Diff::Line objects
       def diff_lines
-        @lines ||= parser.parse(raw_diff.each_line).to_a
-      end
-
-      def too_large?
-        diff.too_large?
+        @lines ||= Gitlab::Diff::Parser.new.parse(raw_diff.each_line).to_a
       end
 
       def highlighted_diff_lines
-        Gitlab::Diff::Highlight.new(self).highlight
+        @highlighted_diff_lines ||= Gitlab::Diff::Highlight.new(self, repository: self.repository).highlight
       end
 
       def parallel_diff_lines
-        Gitlab::Diff::ParallelDiff.new(self).parallelize
+        @parallel_diff_lines ||= Gitlab::Diff::ParallelDiff.new(self).parallelize
       end
 
       def mode_changed?
-        !!(diff.a_mode && diff.b_mode && diff.a_mode != diff.b_mode)
-      end
-
-      def parser
-        Gitlab::Diff::Parser.new
+        a_mode && b_mode && a_mode != b_mode
       end
 
       def raw_diff
@@ -53,17 +89,15 @@ module Gitlab
       end
 
       def prev_line(index)
-        if index > 0
-          diff_lines[index - 1]
-        end
+        diff_lines[index - 1] if index > 0
+      end
+
+      def paths
+        [old_path, new_path].compact
       end
 
       def file_path
-        if diff.new_path.present?
-          diff.new_path
-        elsif diff.old_path.present?
-          diff.old_path
-        end
+        new_path.presence || old_path
       end
 
       def added_lines
@@ -73,6 +107,21 @@ module Gitlab
       def removed_lines
         diff_lines.count(&:removed?)
       end
+
+      def old_blob(commit = content_commit)
+        return unless commit
+
+        parent_id = commit.parent_id
+        return unless parent_id
+
+        repository.blob_at(parent_id, old_path)
+      end
+
+      def blob(commit = content_commit)
+        return unless commit
+
+        repository.blob_at(commit.id, file_path)
+      end
     end
   end
 end
diff --git a/lib/gitlab/diff/highlight.rb b/lib/gitlab/diff/highlight.rb
index 9429b3ff88d..649a265a02c 100644
--- a/lib/gitlab/diff/highlight.rb
+++ b/lib/gitlab/diff/highlight.rb
@@ -1,11 +1,13 @@
 module Gitlab
   module Diff
     class Highlight
-      attr_reader :diff_file, :diff_lines, :raw_lines
+      attr_reader :diff_file, :diff_lines, :raw_lines, :repository
 
       delegate :old_path, :new_path, :old_ref, :new_ref, to: :diff_file, prefix: :diff
 
-      def initialize(diff_lines)
+      def initialize(diff_lines, repository: nil)
+        @repository = repository
+
         if diff_lines.is_a?(Gitlab::Diff::File)
           @diff_file = diff_lines
           @diff_lines = @diff_file.diff_lines
@@ -19,7 +21,7 @@ module Gitlab
         @diff_lines.map.with_index do |diff_line, i|
           diff_line = diff_line.dup
           # ignore highlighting for "match" lines
-          next diff_line if diff_line.type == 'match' || diff_line.type == 'nonewline'
+          next diff_line if diff_line.meta?
 
           rich_line = highlight_line(diff_line) || diff_line.text
 
@@ -40,12 +42,12 @@ module Gitlab
 
         line_prefix = diff_line.text.match(/\A(.)/) ? $1 : ' '
 
-        case diff_line.type
-        when 'new', nil
-          rich_line = new_lines[diff_line.new_pos - 1]
-        when 'old'
-          rich_line = old_lines[diff_line.old_pos - 1]
-        end
+        rich_line =
+          if diff_line.unchanged? || diff_line.added?
+            new_lines[diff_line.new_pos - 1]
+          elsif diff_line.removed?
+            old_lines[diff_line.old_pos - 1]
+          end
 
         # Only update text if line is found. This will prevent
         # issues with submodules given the line only exists in diff content.
@@ -58,19 +60,12 @@ module Gitlab
 
       def old_lines
         return unless diff_file
-        @old_lines ||= Gitlab::Highlight.highlight_lines(*processing_args(:old))
+        @old_lines ||= Gitlab::Highlight.highlight_lines(self.repository, diff_old_ref, diff_old_path)
       end
 
       def new_lines
         return unless diff_file
-        @new_lines ||= Gitlab::Highlight.highlight_lines(*processing_args(:new))
-      end
-
-      def processing_args(version)
-        ref  = send("diff_#{version}_ref")
-        path = send("diff_#{version}_path")
-
-        [ref.project.repository, ref.id, path]
+        @new_lines ||= Gitlab::Highlight.highlight_lines(self.repository, diff_new_ref, diff_new_path)
       end
     end
   end
diff --git a/lib/gitlab/diff/line.rb b/lib/gitlab/diff/line.rb
index 03730b435ad..c6189d660c2 100644
--- a/lib/gitlab/diff/line.rb
+++ b/lib/gitlab/diff/line.rb
@@ -9,6 +9,18 @@ module Gitlab
         @old_pos, @new_pos = old_pos, new_pos
       end
 
+      def old_line
+        old_pos unless added? || meta?
+      end
+
+      def new_line
+        new_pos unless removed? || meta?
+      end
+
+      def unchanged?
+        type.nil?
+      end
+
       def added?
         type == 'new'
       end
@@ -16,6 +28,10 @@ module Gitlab
       def removed?
         type == 'old'
       end
+
+      def meta?
+        type == 'match' || type == 'nonewline'
+      end
     end
   end
 end
diff --git a/lib/gitlab/diff/line_mapper.rb b/lib/gitlab/diff/line_mapper.rb
new file mode 100644
index 00000000000..576a761423e
--- /dev/null
+++ b/lib/gitlab/diff/line_mapper.rb
@@ -0,0 +1,64 @@
+# When provided a diff for a specific file, maps old line numbers to new line
+# numbers and back, to find out where a specific line in a file was moved by the
+# changes.
+module Gitlab
+  module Diff
+    class LineMapper
+      attr_accessor :diff_file
+
+      def initialize(diff_file)
+        @diff_file = diff_file
+      end
+
+      # Find new line number for old line number.
+      def old_to_new(old_line)
+        map_line_number(old_line, from: :old_line, to: :new_line)
+      end
+
+      # Find old line number for new line number.
+      def new_to_old(new_line)
+        map_line_number(new_line, from: :new_line, to: :old_line)
+      end
+
+      private
+
+      def diff_lines
+        @diff_lines ||= @diff_file.diff_lines
+      end
+
+      # Find old/new line number based on its old/new counterpart line number.
+      def map_line_number(from_line, from:, to:)
+        # If no diff file could be found, the file wasn't changed, and the
+        # mapped line number is the same as the specified line number.
+        return from_line unless diff_file
+
+        # To find the mapped line number for the specified line number,
+        # we need to find:
+        # - The diff line with that exact line number, if it is in the diff context
+        # - The first diff line with a higher line number, if it falls between diff contexts
+        # - The last known diff line, if it falls after the last diff context
+        diff_line = diff_lines.find do |diff_line|
+          diff_from_line = diff_line.send(from)
+          diff_from_line && diff_from_line >= from_line
+        end
+        diff_line ||= diff_lines.last
+
+        # If no diff line could be found, the file wasn't changed, and the
+        # mapped line number is the same as the specified line number.
+        return from_line unless diff_line
+
+        diff_from_line = diff_line.send(from)
+        diff_to_line = diff_line.send(to)
+
+        # If the line was removed, there is no mapped line number.
+        return unless diff_to_line
+
+        # Because we may not have the diff line with the exact line number
+        # we were looking for, we need to adjust the mapped line number.
+        distance = diff_from_line - from_line
+
+        diff_to_line - distance
+      end
+    end
+  end
+end
diff --git a/lib/gitlab/diff/parallel_diff.rb b/lib/gitlab/diff/parallel_diff.rb
index 74f9b3c050a..1c1fc148123 100644
--- a/lib/gitlab/diff/parallel_diff.rb
+++ b/lib/gitlab/diff/parallel_diff.rb
@@ -15,17 +15,19 @@ module Gitlab
         highlighted_diff_lines.each do |line|
           full_line = line.text
           type = line.type
-          line_code = generate_line_code(diff_file.file_path, line)
+          line_code = diff_file.line_code(line)
           line_new = line.new_pos
           line_old = line.old_pos
+          position = diff_file.position(line)
 
           next_line = diff_file.next_line(line.index)
 
           if next_line
             next_line = highlighted_diff_lines[next_line.index]
-            next_line_code = generate_line_code(diff_file.file_path, next_line)
+            full_next_line = next_line.text
+            next_line_code = diff_file.line_code(next_line)
             next_type = next_line.type
-            next_line = next_line.text
+            next_position = diff_file.position(next_line)
           end
 
           case type
@@ -37,12 +39,14 @@ module Gitlab
                 number:     line_old,
                 text:       full_line,
                 line_code:  line_code,
+                position:   position
               },
               right: {
                 type:       type,
                 number:     line_new,
                 text:       full_line,
-                line_code:  line_code
+                line_code:  line_code,
+                position:   position
               }
             }
           when 'old'
@@ -55,12 +59,14 @@ module Gitlab
                   number:     line_old,
                   text:       full_line,
                   line_code:  line_code,
+                  position:   position
                 },
                 right: {
                   type:       next_type,
                   number:     line_new,
-                  text:       next_line,
-                  line_code:  next_line_code
+                  text:       full_next_line,
+                  line_code:  next_line_code,
+                  position:   next_position,
                 }
               }
               skip_next = true
@@ -73,12 +79,14 @@ module Gitlab
                   number:     line_old,
                   text:       full_line,
                   line_code:  line_code,
+                  position:   position
                 },
                 right: {
                   type:       next_type,
                   number:     nil,
                   text:       "",
-                  line_code:  nil
+                  line_code:  nil,
+                  position:   nil
                 }
               }
             end
@@ -95,12 +103,14 @@ module Gitlab
                   number:     nil,
                   text:       "",
                   line_code:  line_code,
+                  position:   position
                 },
                 right: {
                   type:       type,
                   number:     line_new,
                   text:       full_line,
-                  line_code:  line_code
+                  line_code:  line_code,
+                  position:   position
                 }
               }
             end
@@ -108,12 +118,6 @@ module Gitlab
         end
         lines
       end
-
-      private
-
-      def generate_line_code(file_path, line)
-        Gitlab::Diff::LineCode.generate(file_path, line.new_pos, line.old_pos)
-      end
     end
   end
 end
diff --git a/lib/gitlab/diff/position.rb b/lib/gitlab/diff/position.rb
new file mode 100644
index 00000000000..989fff8918e
--- /dev/null
+++ b/lib/gitlab/diff/position.rb
@@ -0,0 +1,155 @@
+# Defines a specific location, identified by paths and line numbers,
+# within a specific diff, identified by start, head and base commit ids.
+module Gitlab
+  module Diff
+    class Position
+      attr_reader :old_path
+      attr_reader :new_path
+      attr_reader :old_line
+      attr_reader :new_line
+      attr_reader :base_sha
+      attr_reader :start_sha
+      attr_reader :head_sha
+
+      def initialize(attrs = {})
+        @old_path = attrs[:old_path]
+        @new_path = attrs[:new_path]
+        @old_line = attrs[:old_line]
+        @new_line = attrs[:new_line]
+
+        if attrs[:diff_refs]
+          @base_sha  = attrs[:diff_refs].base_sha
+          @start_sha = attrs[:diff_refs].start_sha
+          @head_sha  = attrs[:diff_refs].head_sha
+        else
+          @base_sha  = attrs[:base_sha]
+          @start_sha = attrs[:start_sha]
+          @head_sha  = attrs[:head_sha]
+        end
+      end
+
+      # `Gitlab::Diff::Position` objects are stored as serialized attributes in
+      # `DiffNote`, which use YAML to encode and decode objects.
+      # `#init_with` and `#encode_with` can be used to customize the en/decoding
+      # behavior. In this case, we override these to prevent memoized instance
+      # variables like `@diff_file` and `@diff_line` from being serialized.
+      def init_with(coder)
+        initialize(coder['attributes'])
+
+        self
+      end
+
+      def encode_with(coder)
+        coder['attributes'] = self.to_h
+      end
+
+      def key
+        @key ||= [base_sha, start_sha, head_sha, Digest::SHA1.hexdigest(old_path || ""), Digest::SHA1.hexdigest(new_path || ""), old_line, new_line]
+      end
+
+      def ==(other)
+        other.is_a?(self.class) && key == other.key
+      end
+
+      def to_h
+        {
+          old_path: old_path,
+          new_path: new_path,
+          old_line: old_line,
+          new_line: new_line,
+          base_sha:  base_sha,
+          start_sha: start_sha,
+          head_sha:  head_sha
+        }
+      end
+
+      def inspect
+        %(#<#{self.class}:#{object_id} #{to_h}>)
+      end
+
+      def complete?
+        file_path.present? &&
+          (old_line || new_line) &&
+          diff_refs.complete?
+      end
+
+      def to_json
+        JSON.generate(self.to_h)
+      end
+
+      def type
+        if old_line && new_line
+          nil
+        elsif new_line
+          'new'
+        else
+          'old'
+        end
+      end
+
+      def unchanged?
+        type.nil?
+      end
+
+      def added?
+        type == 'new'
+      end
+
+      def removed?
+        type == 'old'
+      end
+
+      def paths
+        [old_path, new_path].compact.uniq
+      end
+
+      def file_path
+        new_path.presence || old_path
+      end
+
+      def diff_refs
+        @diff_refs ||= DiffRefs.new(base_sha: base_sha, start_sha: start_sha, head_sha: head_sha)
+      end
+
+      def diff_file(repository)
+        @diff_file ||= begin
+          if RequestStore.active?
+            key = {
+              project_id: repository.project.id,
+              start_sha: start_sha,
+              head_sha: head_sha,
+              path: file_path
+            }
+
+            RequestStore.fetch(key) { find_diff_file(repository) }
+          else
+            find_diff_file(repository)
+          end
+        end
+      end
+
+      def diff_line(repository)
+        @diff_line ||= diff_file(repository).line_for_position(self)
+      end
+
+      def line_code(repository)
+        @line_code ||= diff_file(repository).line_code_for_position(self)
+      end
+
+      private
+
+      def find_diff_file(repository)
+        diffs = Gitlab::Git::Compare.new(
+          repository.raw_repository,
+          start_sha,
+          head_sha
+        ).diffs(paths: paths)
+
+        diff = diffs.first
+        return unless diff
+
+        Gitlab::Diff::File.new(diff, repository: repository, diff_refs: diff_refs)
+      end
+    end
+  end
+end
diff --git a/lib/gitlab/diff/position_tracer.rb b/lib/gitlab/diff/position_tracer.rb
new file mode 100644
index 00000000000..4d04f867268
--- /dev/null
+++ b/lib/gitlab/diff/position_tracer.rb
@@ -0,0 +1,168 @@
+# Finds the diff position in the new diff that corresponds to the same location
+# specified by the provided position in the old diff.
+module Gitlab
+  module Diff
+    class PositionTracer
+      attr_accessor :repository
+      attr_accessor :old_diff_refs
+      attr_accessor :new_diff_refs
+      attr_accessor :paths
+
+      def initialize(repository:, old_diff_refs:, new_diff_refs:, paths: nil)
+        @repository = repository
+        @old_diff_refs = old_diff_refs
+        @new_diff_refs = new_diff_refs
+        @paths = paths
+      end
+
+      def trace(old_position)
+        return unless old_diff_refs.complete? && new_diff_refs.complete?
+        return unless old_position.diff_refs == old_diff_refs
+
+        # Suppose we have an MR with source branch `feature` and target branch `master`.
+        # When the MR was created, the head of `master` was commit A, and the
+        # head of `feature` was commit B, resulting in the original diff A->B.
+        # Since creation, `master` was updated to C.
+        # Now `feature` is being updated to D, and the newly generated MR diff is C->D.
+        # It is possible that C and D are direct decendants of A and B respectively,
+        # but this isn't necessarily the case as rebases and merges come into play.
+        #
+        # Suppose we have a diff note on the original diff A->B. Now that the MR
+        # is updated, we need to find out what line in C->D corresponds to the
+        # line the note was originally created on, so that we can update the diff note's
+        # records and continue to display it in the right place in the diffs.
+        # If we cannot find this line in the new diff, this means the diff note is now
+        # outdated, and we will display that fact to the user.
+        #
+        # In the new diff, the file the diff note was originally created on may
+        # have been renamed, deleted or even created, if the file existed in A and B,
+        # but was removed in C, and restored in D.
+        #
+        # Every diff note stores a Position object that defines a specific location,
+        # identified by paths and line numbers, within a specific diff, identified
+        # by start, head and base commit ids.
+        #
+        # For diff notes for diff A->B, the position looks like this:
+        # Position
+        #   base_sha - ID of commit A
+        #   head_sha - ID of commit B
+        #   old_path - path as of A (nil if file was newly created)
+        #   new_path - path as of B (nil if file was deleted)
+        #   old_line - line number as of A (nil if file was newly created)
+        #   new_line - line number as of B (nil if file was deleted)
+        #
+        # We can easily update `base_sha` and `head_sha` to hold the IDs of commits C and D,
+        # but need to find the paths and line numbers as of C and D.
+        #
+        # If the file was unchanged or newly created in A->B, the path as of D can be found
+        # by generating diff B->D ("head to head"), finding the diff file with
+        # `diff_file.old_path == position.new_path`, and taking `diff_file.new_path`.
+        # The path as of C can be found by taking diff C->D, finding the diff file
+        # with that same `new_path` and taking `diff_file.old_path`.
+        # The line number as of D can be found by using the LineMapper on diff B->D
+        # and providing the line number as of B.
+        # The line number as of C can be found by using the LineMapper on diff C->D
+        # and providing the line number as of D.
+        #
+        # If the file was deleted in A->B, the path as of C can be found
+        # by generating diff A->C ("base to base"), finding the diff file with
+        # `diff_file.old_path == position.old_path`, and taking `diff_file.new_path`.
+        # The path as of D can be found by taking diff C->D, finding the diff file
+        # with that same `old_path` and taking `diff_file.new_path`.
+        # The line number as of C can be found by using the LineMapper on diff A->C
+        # and providing the line number as of A.
+        # The line number as of D can be found by using the LineMapper on diff C->D
+        # and providing the line number as of C.
+
+        results = nil
+        results ||= trace_added_line(old_position)   if old_position.added?   || old_position.unchanged?
+        results ||= trace_removed_line(old_position) if old_position.removed? || old_position.unchanged?
+
+        return unless results
+
+        file_diff, old_line, new_line = results
+
+        Position.new(
+          old_path: file_diff.old_path,
+          new_path: file_diff.new_path,
+          head_sha: new_diff_refs.head_sha,
+          start_sha: new_diff_refs.start_sha,
+          base_sha: new_diff_refs.base_sha,
+          old_line: old_line,
+          new_line: new_line
+        )
+      end
+
+      private
+
+      def trace_added_line(old_position)
+        file_path = old_position.new_path
+
+        return unless diff_head_to_head
+
+        file_head_to_head = diff_head_to_head.find { |diff_file| diff_file.old_path == file_path }
+
+        file_path = file_head_to_head.new_path if file_head_to_head
+
+        new_line = LineMapper.new(file_head_to_head).old_to_new(old_position.new_line)
+
+        return unless new_line
+
+        file_diff = new_diffs.find { |diff_file| diff_file.new_path == file_path }
+        return unless file_diff
+
+        old_line = LineMapper.new(file_diff).new_to_old(new_line)
+
+        [file_diff, old_line, new_line]
+      end
+
+      def trace_removed_line(old_position)
+        file_path = old_position.old_path
+
+        return unless diff_base_to_base
+
+        file_base_to_base = diff_base_to_base.find { |diff_file| diff_file.old_path == file_path }
+
+        file_path = file_base_to_base.old_path if file_base_to_base
+
+        old_line = LineMapper.new(file_base_to_base).old_to_new(old_position.old_line)
+
+        return unless old_line
+
+        file_diff = new_diffs.find { |diff_file| diff_file.old_path == file_path }
+        return unless file_diff
+
+        new_line = LineMapper.new(file_diff).old_to_new(old_line)
+
+        [file_diff, old_line, new_line]
+      end
+
+      def diff_base_to_base
+        @diff_base_to_base ||= diff_files(old_diff_refs.base_sha || old_diff_refs.start_sha, new_diff_refs.base_sha || new_diff_refs.start_sha)
+      end
+
+      def diff_head_to_head
+        @diff_head_to_head ||= diff_files(old_diff_refs.head_sha, new_diff_refs.head_sha)
+      end
+
+      def new_diffs
+        @new_diffs ||= diff_files(new_diff_refs.start_sha, new_diff_refs.head_sha, use_base: true)
+      end
+
+      def diff_files(start_sha, head_sha, use_base: false)
+        base_sha = self.repository.merge_base(start_sha, head_sha) || start_sha
+
+        diffs = self.repository.raw_repository.diff(
+          use_base ? base_sha : start_sha,
+          head_sha,
+          {},
+          *paths
+        )
+
+        diffs.decorate! do |diff|
+          Gitlab::Diff::File.new(diff, repository: self.repository)
+        end
+      end
+    end
+  end
+end
diff --git a/lib/gitlab/email/message/repository_push.rb b/lib/gitlab/email/message/repository_push.rb
index 047c77c6fc2..97701b0cd42 100644
--- a/lib/gitlab/email/message/repository_push.rb
+++ b/lib/gitlab/email/message/repository_push.rb
@@ -33,11 +33,15 @@ module Gitlab
         end
 
         def commits
-          @commits ||= (Commit.decorate(compare.commits, project) if compare)
+          return unless compare
+
+          @commits ||= Commit.decorate(compare.commits, project)
         end
 
         def diffs
-          @diffs ||= (safe_diff_files(compare.diffs(max_files: 30), diff_refs) if compare)
+          return unless compare
+          
+          @diffs ||= safe_diff_files(compare.diffs(max_files: 30), diff_refs: diff_refs, repository: project.repository)
         end
 
         def diffs_count
diff --git a/lib/gitlab/email/receiver.rb b/lib/gitlab/email/receiver.rb
index 97ef9851d71..1c671a7487b 100644
--- a/lib/gitlab/email/receiver.rb
+++ b/lib/gitlab/email/receiver.rb
@@ -104,15 +104,7 @@ module Gitlab
       end
 
       def create_note(reply)
-        Notes::CreateService.new(
-          sent_notification.project,
-          sent_notification.recipient,
-          note:           reply,
-          noteable_type:  sent_notification.noteable_type,
-          noteable_id:    sent_notification.noteable_id,
-          commit_id:      sent_notification.commit_id,
-          line_code:      sent_notification.line_code
-        ).execute
+        sent_notification.create_note(reply)
       end
     end
   end
diff --git a/lib/gitlab/github_import/pull_request_formatter.rb b/lib/gitlab/github_import/pull_request_formatter.rb
index 498b00cb658..a4ea2210abd 100644
--- a/lib/gitlab/github_import/pull_request_formatter.rb
+++ b/lib/gitlab/github_import/pull_request_formatter.rb
@@ -11,10 +11,10 @@ module Gitlab
           description: description,
           source_project: source_branch_project,
           source_branch: source_branch_name,
-          head_source_sha: source_branch_sha,
+          source_branch_sha: source_branch_sha,
           target_project: target_branch_project,
           target_branch: target_branch_name,
-          base_target_sha: target_branch_sha,
+          target_branch_sha: target_branch_sha,
           state: state,
           milestone: milestone,
           author_id: author_id,
diff --git a/lib/gitlab/timeless.rb b/lib/gitlab/timeless.rb
new file mode 100644
index 00000000000..b290c716f97
--- /dev/null
+++ b/lib/gitlab/timeless.rb
@@ -0,0 +1,16 @@
+module Gitlab
+  module Timeless
+    def self.timeless(model, &block)
+      original_record_timestamps = model.record_timestamps
+      model.record_timestamps = false
+
+      if block.arity.abs == 1
+        block.call(model)
+      else
+        block.call
+      end
+    ensure
+      model.record_timestamps = original_record_timestamps
+    end
+  end
+end
diff --git a/lib/gitlab/workhorse.rb b/lib/gitlab/workhorse.rb
index ef1241f8600..bc0193a6c32 100644
--- a/lib/gitlab/workhorse.rb
+++ b/lib/gitlab/workhorse.rb
@@ -38,12 +38,10 @@ module Gitlab
       end
 
       def send_git_diff(repository, diff_refs)
-        from, to = diff_refs
-
         params = {
           'RepoPath'  => repository.path_to_repo,
-          'ShaFrom'   => from.sha,
-          'ShaTo'     => to.sha
+          'ShaFrom'   => diff_refs.start_sha,
+          'ShaTo'     => diff_refs.head_sha
         }
 
         [
@@ -52,11 +50,11 @@ module Gitlab
         ]
       end
 
-      def send_git_patch(repository, from, to)
+      def send_git_patch(repository, diff_refs)
         params = {
           'RepoPath'  => repository.path_to_repo,
-          'ShaFrom'   => from,
-          'ShaTo'     => to
+          'ShaFrom'   => diff_refs.start_sha,
+          'ShaTo'     => diff_refs.head_sha
         }
 
         [