<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/gitlab/gitlab-ce.git/app/models/cycle_analytics.rb, branch koding-setting-api</title>
<subtitle>gitlab.com: gitlab-org/gitlab-ce.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/'/>
<entry>
<title>Implement a second round of review comments from @DouweM.</title>
<updated>2016-09-20T19:17:37+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T19:17:37+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=918e589c2b29c18d9fe3a8e6c93a3f490c86beb1'/>
<id>918e589c2b29c18d9fe3a8e6c93a3f490c86beb1</id>
<content type='text'>
- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
  `Issue#closed_by_merge_requests`  associations. They were either
  shadowing or were too similar to existing methods. They are not being
  used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
  need it.
- Other minor refactorings.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
  `Issue#closed_by_merge_requests`  associations. They were either
  shadowing or were too similar to existing methods. They are not being
  used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
  need it.
- Other minor refactorings.
</pre>
</div>
</content>
</entry>
<entry>
<title>Implement (some) comments from @DouweM's review.</title>
<updated>2016-09-20T12:43:11+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T12:43:11+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=71d4bf721be6cd57ab3480bc5efb11a91d6e1891'/>
<id>71d4bf721be6cd57ab3480bc5efb11a91d6e1891</id>
<content type='text'>
- Move things common to `Issue` and `MergeRequest` into `Issuable`
- Move more database-specific functions into `Gitlab::Database`
- Indentation changes and other minor refactorings.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- Move things common to `Issue` and `MergeRequest` into `Issuable`
- Move more database-specific functions into `Gitlab::Database`
- Indentation changes and other minor refactorings.
</pre>
</div>
</content>
</entry>
<entry>
<title>Implement review comments from @yorickpeterse</title>
<updated>2016-09-20T10:35:25+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T09:36:54+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=8957293d9bd0d711db3af26182205c2fe4125194'/>
<id>8957293d9bd0d711db3af26182205c2fe4125194</id>
<content type='text'>
1. Change multiple updates to a single `update_all`

2. Use cascading deletes

3. Extract an average function for the database median.

4. Move database median to `lib/gitlab/database`

5. Use `delete_all` instead of `destroy_all`

6. Minor refactoring
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Change multiple updates to a single `update_all`

2. Use cascading deletes

3. Extract an average function for the database median.

4. Move database median to `lib/gitlab/database`

5. Use `delete_all` instead of `destroy_all`

6. Minor refactoring
</pre>
</div>
</content>
</entry>
<entry>
<title>Implement a database median strategy for MySQL.</title>
<updated>2016-09-20T07:53:14+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T07:03:37+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=4ff8d5d28d402284ad365630d9c85b9ae4479b04'/>
<id>4ff8d5d28d402284ad365630d9c85b9ae4479b04</id>
<content type='text'>
1. Dispatch between the two strategies automatically based on the
   current database type.

2. The MySQL version needs to run multiple statements, so the
   `cycle_analytics` model is modified to support this.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Dispatch between the two strategies automatically based on the
   current database type.

2. The MySQL version needs to run multiple statements, so the
   `cycle_analytics` model is modified to support this.
</pre>
</div>
</content>
</entry>
<entry>
<title>CycleAnalytics operates on merge requests that have been deployed.</title>
<updated>2016-09-20T07:53:14+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-20T05:44:40+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=cebfe053a85c8e376b35101686ba05225c6f4e65'/>
<id>cebfe053a85c8e376b35101686ba05225c6f4e65</id>
<content type='text'>
1. Look for merge requests (and issues that they close) that have been
   deployed to production in the last X days (where X is given by the
   `from` parameter).

2. Cycle analytics queries only operate on this fitered set of merge
   requests and issues.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Look for merge requests (and issues that they close) that have been
   deployed to production in the last X days (where X is given by the
   `from` parameter).

2. Cycle analytics queries only operate on this fitered set of merge
   requests and issues.
</pre>
</div>
</content>
</entry>
<entry>
<title>Make changes to the cycle analytics JSON endpoint.</title>
<updated>2016-09-19T09:30:55+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-19T09:30:55+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=4531e433ad33e67cb66abaa4626b5e608eb11f11'/>
<id>4531e433ad33e67cb66abaa4626b5e608eb11f11</id>
<content type='text'>
1. Add `summary` section.
2. `stats` is `null` if no stats are present.
3. `stats` and `summary` are both arrays.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Add `summary` section.
2. `stats` is `null` if no stats are present.
3. `stats` and `summary` are both arrays.
</pre>
</div>
</content>
</entry>
<entry>
<title>Move cycle analytics JSON generation to a helper.</title>
<updated>2016-09-17T06:46:48+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-17T06:38:23+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=edb38d69cc6cffef46b6c7b957ad97ce213173cf'/>
<id>edb38d69cc6cffef46b6c7b957ad97ce213173cf</id>
<content type='text'>
1. Use a new format, with each stage having a `title`, `description`,
   and `value.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Use a new format, with each stage having a `title`, `description`,
   and `value.
</pre>
</div>
</content>
</entry>
<entry>
<title>Move cycle analytics calculations to SQL.</title>
<updated>2016-09-17T06:46:48+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-16T06:04:49+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=7d69ff3ddf0fb83c6a1ec02f85b01b454080b647'/>
<id>7d69ff3ddf0fb83c6a1ec02f85b01b454080b647</id>
<content type='text'>
1. Use Arel for composable queries.
2. For a project with ~10k issues, the page loads in around 600ms.
   Previously, a project with ~5k issues would have a ~20s page load
   time.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. Use Arel for composable queries.
2. For a project with ~10k issues, the page loads in around 600ms.
   Previously, a project with ~5k issues would have a ~20s page load
   time.
</pre>
</div>
</content>
</entry>
<entry>
<title>Improve performance of the cycle analytics page.</title>
<updated>2016-09-15T09:23:02+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-15T08:59:36+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=ba25e2f1ac61b47940f939a2d9f1d0ad417e1de2'/>
<id>ba25e2f1ac61b47940f939a2d9f1d0ad417e1de2</id>
<content type='text'>
1. These changes bring down page load time for 100 issues from more than
   a minute to about 1.5 seconds.

2. This entire commit is composed of these types of performance
   enhancements:

     - Cache relevant data in `IssueMetrics` wherever possible.
     - Cache relevant data in `MergeRequestMetrics` wherever possible.
     - Preload metrics

3. Given these improvements, we now only need to make 4 SQL calls:

    - Load all issues
    - Load all merge requests
    - Load all metrics for the issues
    - Load all metrics for the merge requests

4. A list of all the data points that are now being pre-calculated:

    a. The first time an issue is mentioned in a commit

      - In `GitPushService`, find all issues mentioned by the given commit
        using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at`
        flag for each of them.

      - There seems to be a (pre-existing) bug here - files (and
        therefore commits) created using the Web CI don't have
        cross-references created, and issues are not closed even when
        the commit title is "Fixes #xx".

    b. The first time a merge request is deployed to production

      When a `Deployment` is created, find all merge requests that
      were merged in before the deployment, and set the
      `first_deployed_to_production_at` flag for each of them.

    c. The start / end time for a merge request pipeline

      Hook into the `Pipeline` state machine. When the `status` moves to
      `running`, find the merge requests whose tip commit matches the
      pipeline, and record the `latest_build_started_at` time for each
      of them. When the `status` moves to `success`, record the
      `latest_build_finished_at` time.

    d. The merge requests that close an issue

      - This was a big cause of the performance problems we were having
        with Cycle Analytics. We need to use `ReferenceExtractor` to make
        this calculation, which is slow when we have to run it on a large
        number of merge requests.

      - When a merge request is created, updated, or refreshed, find the
        issues it closes, and create an instance of
        `MergeRequestsClosingIssues`, which acts as a join model between
        merge requests and issues.

      - If a `MergeRequestsClosingIssues` instance links a merge request
        and an issue, that issue closes that merge request.

5. The `Queries` module was changed into a class, so we can cache the
   results of `issues` and `merge_requests_closing_issues` across
   various cycle analytics stages.

6. The code added in this commit is untested. Tests will be added in the
   next commit.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
1. These changes bring down page load time for 100 issues from more than
   a minute to about 1.5 seconds.

2. This entire commit is composed of these types of performance
   enhancements:

     - Cache relevant data in `IssueMetrics` wherever possible.
     - Cache relevant data in `MergeRequestMetrics` wherever possible.
     - Preload metrics

3. Given these improvements, we now only need to make 4 SQL calls:

    - Load all issues
    - Load all merge requests
    - Load all metrics for the issues
    - Load all metrics for the merge requests

4. A list of all the data points that are now being pre-calculated:

    a. The first time an issue is mentioned in a commit

      - In `GitPushService`, find all issues mentioned by the given commit
        using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at`
        flag for each of them.

      - There seems to be a (pre-existing) bug here - files (and
        therefore commits) created using the Web CI don't have
        cross-references created, and issues are not closed even when
        the commit title is "Fixes #xx".

    b. The first time a merge request is deployed to production

      When a `Deployment` is created, find all merge requests that
      were merged in before the deployment, and set the
      `first_deployed_to_production_at` flag for each of them.

    c. The start / end time for a merge request pipeline

      Hook into the `Pipeline` state machine. When the `status` moves to
      `running`, find the merge requests whose tip commit matches the
      pipeline, and record the `latest_build_started_at` time for each
      of them. When the `status` moves to `success`, record the
      `latest_build_finished_at` time.

    d. The merge requests that close an issue

      - This was a big cause of the performance problems we were having
        with Cycle Analytics. We need to use `ReferenceExtractor` to make
        this calculation, which is slow when we have to run it on a large
        number of merge requests.

      - When a merge request is created, updated, or refreshed, find the
        issues it closes, and create an instance of
        `MergeRequestsClosingIssues`, which acts as a join model between
        merge requests and issues.

      - If a `MergeRequestsClosingIssues` instance links a merge request
        and an issue, that issue closes that merge request.

5. The `Queries` module was changed into a class, so we can cache the
   results of `issues` and `merge_requests_closing_issues` across
   various cycle analytics stages.

6. The code added in this commit is untested. Tests will be added in the
   next commit.
</pre>
</div>
</content>
</entry>
<entry>
<title>Add a JSON version of the `CycleAnalytics` page.</title>
<updated>2016-09-08T09:33:38+00:00</updated>
<author>
<name>Timothy Andrew</name>
<email>mail@timothyandrew.net</email>
</author>
<published>2016-09-08T09:33:38+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=9901c3df41d37e5d5650e28a5287a8ddf0aa754e'/>
<id>9901c3df41d37e5d5650e28a5287a8ddf0aa754e</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
