<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/gitlab/gitlab-ce.git/db/migrate, branch scroll-code-blocks</title>
<subtitle>gitlab.com: gitlab-org/gitlab-ce.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/'/>
<entry>
<title>Fix bug where destroying a namespace would not always destroy projects</title>
<updated>2016-08-11T22:36:35+00:00</updated>
<author>
<name>Stan Hu</name>
<email>stanhu@gmail.com</email>
</author>
<published>2016-05-29T02:54:17+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=cb8a425ba42e9be23b8712ed29b1db2dcc6bd139'/>
<id>cb8a425ba42e9be23b8712ed29b1db2dcc6bd139</id>
<content type='text'>
There is a race condition in DestroyGroupService now that projects are deleted asynchronously:

1. User attempts to delete group
2. DestroyGroupService iterates through all projects and schedules a Sidekiq job to delete each Project
3. DestroyGroupService destroys the Group, leaving all its projects without a namespace
4. Projects::DestroyService runs later but the can?(current_user,
   :remove_project) is `false` because the user no longer has permission to
   destroy projects with no namespace.
5. This leaves the project in pending_delete state with no namespace/group.

Projects without a namespace or group also adds another problem: it's not possible to destroy the container
registry tags, since container_registry_path_with_namespace is the wrong value.

The fix is to destroy the group asynchronously and to run execute directly on Projects::DestroyService.

Closes #17893
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
There is a race condition in DestroyGroupService now that projects are deleted asynchronously:

1. User attempts to delete group
2. DestroyGroupService iterates through all projects and schedules a Sidekiq job to delete each Project
3. DestroyGroupService destroys the Group, leaving all its projects without a namespace
4. Projects::DestroyService runs later but the can?(current_user,
   :remove_project) is `false` because the user no longer has permission to
   destroy projects with no namespace.
5. This leaves the project in pending_delete state with no namespace/group.

Projects without a namespace or group also adds another problem: it's not possible to destroy the container
registry tags, since container_registry_path_with_namespace is the wrong value.

The fix is to destroy the group asynchronously and to run execute directly on Projects::DestroyService.

Closes #17893
</pre>
</div>
</content>
</entry>
<entry>
<title>Merge branch 'refactor-builds-creation-service' into 'master'</title>
<updated>2016-08-11T14:27:42+00:00</updated>
<author>
<name>Rémy Coutable</name>
<email>remy@rymai.me</email>
</author>
<published>2016-08-11T14:27:42+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=7e9b41896d8c2ffae36152401f35479b39297e78'/>
<id>7e9b41896d8c2ffae36152401f35479b39297e78</id>
<content type='text'>

Refactor pipeline creation service

## What does this MR do?
This refactors GitLab CI build processing: all builds for pipeline are pre-created when a pipeline object is created. 
The builds are created with a new introduced status `created`.
The builds are then automatically promoted to `pending` when a previous stage do succeed.
This significantly simplifies pipeline processing code solving a lot of problems of lazily initialisation of previous approach (builds were created on-demand).

## Why was this MR needed?
The previous mechanism had a lot of flows (shown in related issues) in how it work, but also in code design. Removing cross model-service-library dependencies.

The current approach moves a build creation to single place `CreatePipelineService` and removes a dynamic dependency on `config_processor` significantly simplifying a build creation and pipeline processing. Pipeline processing is implemented in `ProcessPipelineService`.

This also allows to easily extend GitLab with Manual Actions which is part of 8.10 direction issue.

## Migration problem
~~This MR removes the a on-demand creation of builds in pipelines.
 Pipelines that are running and are in mid-stage (some stages started, but not all) will not be fully evaluated after application restart. 
This happens, because the code responsible for on-demand creation is removed. 
There's no easy way to migrate existing pipelines, other than doing offline migration and putting pipeline processing in migration code (which seems to be a really bad idea).~~

To support old pipelines I added a lazy initialization of builds if none is found.

## What are the relevant issue numbers?
Fixes: https://gitlab.com/gitlab-org/gitlab-ce/issues/12839
Solves: https://gitlab.com/gitlab-org/gitlab-ce/issues/18644 https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/289
Allows to easily implement: https://gitlab.com/gitlab-org/gitlab-ce/issues/17010

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- Tests
  - [x] Added for this feature/bug
  - [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)


See merge request !5295</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>

Refactor pipeline creation service

## What does this MR do?
This refactors GitLab CI build processing: all builds for pipeline are pre-created when a pipeline object is created. 
The builds are created with a new introduced status `created`.
The builds are then automatically promoted to `pending` when a previous stage do succeed.
This significantly simplifies pipeline processing code solving a lot of problems of lazily initialisation of previous approach (builds were created on-demand).

## Why was this MR needed?
The previous mechanism had a lot of flows (shown in related issues) in how it work, but also in code design. Removing cross model-service-library dependencies.

The current approach moves a build creation to single place `CreatePipelineService` and removes a dynamic dependency on `config_processor` significantly simplifying a build creation and pipeline processing. Pipeline processing is implemented in `ProcessPipelineService`.

This also allows to easily extend GitLab with Manual Actions which is part of 8.10 direction issue.

## Migration problem
~~This MR removes the a on-demand creation of builds in pipelines.
 Pipelines that are running and are in mid-stage (some stages started, but not all) will not be fully evaluated after application restart. 
This happens, because the code responsible for on-demand creation is removed. 
There's no easy way to migrate existing pipelines, other than doing offline migration and putting pipeline processing in migration code (which seems to be a really bad idea).~~

To support old pipelines I added a lazy initialization of builds if none is found.

## What are the relevant issue numbers?
Fixes: https://gitlab.com/gitlab-org/gitlab-ce/issues/12839
Solves: https://gitlab.com/gitlab-org/gitlab-ce/issues/18644 https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/289
Allows to easily implement: https://gitlab.com/gitlab-org/gitlab-ce/issues/17010

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- Tests
  - [x] Added for this feature/bug
  - [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)


See merge request !5295</pre>
</div>
</content>
</entry>
<entry>
<title>Pre-create all builds for Pipeline when a trigger is received</title>
<updated>2016-08-11T13:22:35+00:00</updated>
<author>
<name>Kamil Trzcinski</name>
<email>ayufan@ayufan.eu</email>
</author>
<published>2016-08-11T13:22:35+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=39203f1adfc6fee3eca50f0cab99ffc597865200'/>
<id>39203f1adfc6fee3eca50f0cab99ffc597865200</id>
<content type='text'>
This change simplifies a Pipeline processing by introducing a special new status: created.
This status is used for all builds that are created for a pipeline.
We are then processing next stages and queueing some of the builds (created -&gt; pending) or skipping them (created -&gt; skipped).
This makes it possible to simplify and solve a few ordering problems with how previously builds were scheduled.
This also allows us to visualise a full pipeline (with created builds).

This also removes an after_touch used for updating a pipeline state parameters.
Right now in various places we explicitly call a reload_status! on pipeline to force it to be updated and saved.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This change simplifies a Pipeline processing by introducing a special new status: created.
This status is used for all builds that are created for a pipeline.
We are then processing next stages and queueing some of the builds (created -&gt; pending) or skipping them (created -&gt; skipped).
This makes it possible to simplify and solve a few ordering problems with how previously builds were scheduled.
This also allows us to visualise a full pipeline (with created builds).

This also removes an after_touch used for updating a pipeline state parameters.
Right now in various places we explicitly call a reload_status! on pipeline to force it to be updated and saved.
</pre>
</div>
</content>
</entry>
<entry>
<title>Remove various redundant indexes</title>
<updated>2016-08-11T11:05:51+00:00</updated>
<author>
<name>Yorick Peterse</name>
<email>yorickpeterse@gmail.com</email>
</author>
<published>2016-08-10T15:26:52+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=7d39bc879d6dc4c0111b3c4ecb2d879034d8d36f'/>
<id>7d39bc879d6dc4c0111b3c4ecb2d879034d8d36f</id>
<content type='text'>
One can see which indexes are used in PostgreSQL by running the
following query:

    SELECT relname as table_name, indexrelname as index_name, idx_scan, idx_tup_read, idx_tup_fetch, pg_size_pretty(pg_relation_size(indexrelname::regclass))
    FROM pg_stat_all_indexes
    WHERE schemaname = 'public'
    AND "idx_scan" = 0
    ORDER BY pg_relation_size(indexrelname::regclass) desc;

Using this query I built a list of indexes that could be potentially
removed. After checking every single one by hand to make sure they
really aren't used I only found 1 index that _would_ be used. This was a
GitLab GEO index (EE) specific that's currently not used simply because
the table is empty.

Apart from this one index all indexes could be removed. The migration
also takes care of 6 composite indexes that can be replaced with a
single column index, which in most cases was already present.

For more information see gitlab-org/gitlab-ce#20767.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
One can see which indexes are used in PostgreSQL by running the
following query:

    SELECT relname as table_name, indexrelname as index_name, idx_scan, idx_tup_read, idx_tup_fetch, pg_size_pretty(pg_relation_size(indexrelname::regclass))
    FROM pg_stat_all_indexes
    WHERE schemaname = 'public'
    AND "idx_scan" = 0
    ORDER BY pg_relation_size(indexrelname::regclass) desc;

Using this query I built a list of indexes that could be potentially
removed. After checking every single one by hand to make sure they
really aren't used I only found 1 index that _would_ be used. This was a
GitLab GEO index (EE) specific that's currently not used simply because
the table is empty.

Apart from this one index all indexes could be removed. The migration
also takes care of 6 composite indexes that can be replaced with a
single column index, which in most cases was already present.

For more information see gitlab-org/gitlab-ce#20767.
</pre>
</div>
</content>
</entry>
<entry>
<title>Remove trigram indexes for "ci_runners"</title>
<updated>2016-08-10T11:06:15+00:00</updated>
<author>
<name>Yorick Peterse</name>
<email>yorickpeterse@gmail.com</email>
</author>
<published>2016-08-10T10:29:06+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=17dd3e89d5a43761e85d7af30ad083db6ca5834b'/>
<id>17dd3e89d5a43761e85d7af30ad083db6ca5834b</id>
<content type='text'>
These indexes are only used when you search for runners in the admin
interface. This operation is so rarely used that it does not make sense
to slow down every update in order to update the GIN trigram indexes.

Removing these indexes should speed up queries such as those used for
updating the last contact time of CI runners. Locally the timings of
this query were reduced from ~50 ms to ~25 ms:

    UPDATE ci_runners SET updated_at = now(), contacted_at = now();
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
These indexes are only used when you search for runners in the admin
interface. This operation is so rarely used that it does not make sense
to slow down every update in order to update the GIN trigram indexes.

Removing these indexes should speed up queries such as those used for
updating the last contact time of CI runners. Locally the timings of
this query were reduced from ~50 ms to ~25 ms:

    UPDATE ci_runners SET updated_at = now(), contacted_at = now();
</pre>
</div>
</content>
</entry>
<entry>
<title>Merge branch '20568-fix-member-data-again' into 'master'</title>
<updated>2016-08-04T20:03:14+00:00</updated>
<author>
<name>Douwe Maan</name>
<email>douwe@gitlab.com</email>
</author>
<published>2016-08-04T20:03:14+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=1aba0668ee0c98b5274517692c73958801cdb285'/>
<id>1aba0668ee0c98b5274517692c73958801cdb285</id>
<content type='text'>

Add a data migration to fix some missing timestamps in the members table (again)

## What does this MR do?

Repeats an earlier migration to fix historic bad data in the members table (missing created_at and updated_at fields)

## Are there points in the code the reviewer needs to double check?

I'm expecting the WHERE clauses to be fast enough, and to return few enough rows, that the migration doesn't need to use batches, but I'm not too familiar with the size of these tables in the wild, so perhaps that's a poor assumption. 

## Why was this MR needed?

8.10 introduced a dependency on the  `members.created_at` field in the project and namespace member view. If bad data is present, viewing the list of members now results in an NoMethodError and a 500 response from GitLab. Although the previous migration should have fixed all bad rows, we have evidence that it didn't in at least one case, despite the migration claiming to have run in the past.

## What are the relevant issue numbers?

#20568 

## Screenshots (if relevant)

## Does this MR meet the acceptance criteria?

- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

Closes #20568

See merge request !5670</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>

Add a data migration to fix some missing timestamps in the members table (again)

## What does this MR do?

Repeats an earlier migration to fix historic bad data in the members table (missing created_at and updated_at fields)

## Are there points in the code the reviewer needs to double check?

I'm expecting the WHERE clauses to be fast enough, and to return few enough rows, that the migration doesn't need to use batches, but I'm not too familiar with the size of these tables in the wild, so perhaps that's a poor assumption. 

## Why was this MR needed?

8.10 introduced a dependency on the  `members.created_at` field in the project and namespace member view. If bad data is present, viewing the list of members now results in an NoMethodError and a 500 response from GitLab. Although the previous migration should have fixed all bad rows, we have evidence that it didn't in at least one case, despite the migration claiming to have run in the past.

## What are the relevant issue numbers?

#20568 

## Screenshots (if relevant)

## Does this MR meet the acceptance criteria?

- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

Closes #20568

See merge request !5670</pre>
</div>
</content>
</entry>
<entry>
<title>Add missing DOWNTIME constant to the AddTimestampsToMembersAgain migration</title>
<updated>2016-08-04T16:01:47+00:00</updated>
<author>
<name>Nick Thomas</name>
<email>nick@gitlab.com</email>
</author>
<published>2016-08-04T16:01:47+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=98260f36cf697665509b89288c4208007d8ad6ce'/>
<id>98260f36cf697665509b89288c4208007d8ad6ce</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Merge branch '20609-fix-protected-branch-access-levels-migration' into 'master'</title>
<updated>2016-08-04T15:49:33+00:00</updated>
<author>
<name>Douwe Maan</name>
<email>douwe@gitlab.com</email>
</author>
<published>2016-08-04T15:49:33+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=96abb19206c6910c8c25f1db77c663baff023d35'/>
<id>96abb19206c6910c8c25f1db77c663baff023d35</id>
<content type='text'>

Migrate protected branch access levels to match constants in `Gitlab::Access`

Closes #20606 

Context: https://gitlab.com/gitlab-org/gitlab-ce/issues/20606#note_13561628

See merge request !5658</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>

Migrate protected branch access levels to match constants in `Gitlab::Access`

Closes #20606 

Context: https://gitlab.com/gitlab-org/gitlab-ce/issues/20606#note_13561628

See merge request !5658</pre>
</div>
</content>
</entry>
<entry>
<title>Merge branch 'fix-add-column-with-default-null' into 'master'</title>
<updated>2016-08-04T15:48:52+00:00</updated>
<author>
<name>Douwe Maan</name>
<email>douwe@gitlab.com</email>
</author>
<published>2016-08-04T15:48:52+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=bd156bb8fcc8210b1f727bb608cf781870950947'/>
<id>bd156bb8fcc8210b1f727bb608cf781870950947</id>
<content type='text'>

Fix `#down` for two protected branches-related migrations.

- The migrations called `add_column_with_default` with a `null` option, which the Rails `add_column` method accepts. This fails because `add_column_with_default` expects an `allow_null` option instead.

- The migrations have been fixed to use `allow_null`.

See merge request !5660</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>

Fix `#down` for two protected branches-related migrations.

- The migrations called `add_column_with_default` with a `null` option, which the Rails `add_column` method accepts. This fails because `add_column_with_default` expects an `allow_null` option instead.

- The migrations have been fixed to use `allow_null`.

See merge request !5660</pre>
</div>
</content>
</entry>
<entry>
<title>Add a data migration to fix some missing timestamps in the members table (again)</title>
<updated>2016-08-04T14:27:53+00:00</updated>
<author>
<name>Nick Thomas</name>
<email>nick@gitlab.com</email>
</author>
<published>2016-08-04T14:27:53+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/gitlab/gitlab-ce.git/commit/?id=f301d547c2a3ca2a3f1fd07f9ddc4eb451e70244'/>
<id>f301d547c2a3ca2a3f1fd07f9ddc4eb451e70244</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
