diff options
| author | Tomasz Maczukin <tomasz@maczukin.pl> | 2015-12-21 13:27:34 +0100 |
|---|---|---|
| committer | Tomasz Maczukin <tomasz@maczukin.pl> | 2015-12-21 13:27:34 +0100 |
| commit | 3cfd892f382d3784f614fea75f929c44fe838559 (patch) | |
| tree | cb9aa9ab48ee02e8f00e8506fecc182d1e66b9ea /doc | |
| parent | 85ad95be741848fbf15a01789f065e001326cefa (diff) | |
| parent | 4b4cbf0ce4925e22a635e4432e7ac8602199fa5b (diff) | |
| download | gitlab-ce-3cfd892f382d3784f614fea75f929c44fe838559.tar.gz | |
Merge branch 'master' into fix/visibility-level-setting-in-forked-projects
* master: (723 commits)
Bump Rack Attack to v4.3.1 for security fix
Remove duplicate entry in the changelog
Remove extra spaces after branchname
Fix merge-request-reopen button title
Add branch and tag operation to tree dropdown
Use gitlab-shell 2.6.9
Clarify Windows shell executor artifact upload support
Fix feature specs: we always show the build status if ci_commit is present
Do not display project group/name when issue and MR are in same project
Don't create CI status for refs that doesn't have .gitlab-ci.yml, even if the builds are enabled
Use gitlab-workhorse 0.5.1
Fix ci_projects migration by using the value only from latest row [ci skip]
Revert sidebar position for issue and merge request
Add info on using private Docker registries in CI [ci skip]
Upgrade Poltergeist to 1.8.1. #4131
Fix ux issue with "This issue will be closed automatically" message
Move MR Builds tab next to Commits
Api support for requesting starred projects for user
Fix Rubocop complain.
Fix merge widget JS for buttons
...
Conflicts:
app/models/project.rb
Diffstat (limited to 'doc')
74 files changed, 2448 insertions, 500 deletions
diff --git a/doc/README.md b/doc/README.md index 0f6866475f7..8bac00f2f23 100644 --- a/doc/README.md +++ b/doc/README.md @@ -7,6 +7,7 @@ - [GitLab Basics](gitlab-basics/README.md) Find step by step how to start working on your commandline and on GitLab. - [Importing to GitLab](workflow/importing/README.md). - [Markdown](markdown/markdown.md) GitLab's advanced formatting system. +- [Migrating from SVN](migration/README.md) Convert a SVN repository to Git and GitLab - [Permissions](permissions/permissions.md) Learn what each role in a project (guest/reporter/developer/master/owner) can do. - [Profile Settings](profile/README.md) - [Project Services](project_services/project_services.md) Integrate a project with external services, such as CI and chat. @@ -24,9 +25,21 @@ - [Using Docker Images](ci/docker/using_docker_images.md) - [Using Docker Build](ci/docker/using_docker_build.md) - [Using Variables](ci/variables/README.md) +- [Using SSH keys](ci/ssh_keys/README.md) - [User permissions](ci/permissions/README.md) - [API](ci/api/README.md) +### CI Languages + ++ [Testing PHP](ci/languages/php.md) + +### CI Services + ++ [Using MySQL](ci/services/mysql.md) ++ [Using PostgreSQL](ci/services/postgres.md) ++ [Using Redis](ci/services/redis.md) ++ [Using Other Services](ci/docker/using_docker_images.md#how-to-use-other-images-as-services) + ### CI Examples - [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md) @@ -50,6 +63,7 @@ - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. +- [Git LFS configuration](workflow/lfs/lfs_administration.md) ## Contributor documentation diff --git a/doc/api/README.md b/doc/api/README.md index 6b8528de50c..25a31b235cc 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -10,6 +10,7 @@ - [Repositories](repositories.md) - [Repository Files](repository_files.md) - [Commits](commits.md) +- [Tags](tags.md) - [Branches](branches.md) - [Merge Requests](merge_requests.md) - [Issues](issues.md) diff --git a/doc/api/groups.md b/doc/api/groups.md index 0b9f6406d8d..808675d8605 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -1,6 +1,6 @@ # Groups
-## List project groups
+## List groups
Get a list of groups. (As user: my groups, as admin: all groups)
@@ -21,6 +21,70 @@ GET /groups You can search for groups by name or path, see below.
+
+## List a group's projects
+
+Get a list of projects in this group.
+
+```
+GET /groups/:id/projects
+```
+
+Parameters:
+
+- `archived` (optional) - if passed, limit by archived status
+- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at`
+- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc`
+- `search` (optional) - Return list of authorized projects according to a search criteria
+- `ci_enabled_first` - Return projects ordered by ci_enabled flag. Projects with enabled GitLab CI go first
+
+```json
+[
+ {
+ "id": 4,
+ "description": null,
+ "default_branch": "master",
+ "public": false,
+ "visibility_level": 0,
+ "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git",
+ "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git",
+ "web_url": "http://example.com/diaspora/diaspora-client",
+ "tag_list": [
+ "example",
+ "disapora client"
+ ],
+ "owner": {
+ "id": 3,
+ "name": "Diaspora",
+ "created_at": "2013-09-30T13: 46: 02Z"
+ },
+ "name": "Diaspora Client",
+ "name_with_namespace": "Diaspora / Diaspora Client",
+ "path": "diaspora-client",
+ "path_with_namespace": "diaspora/diaspora-client",
+ "issues_enabled": true,
+ "merge_requests_enabled": true,
+ "builds_enabled": true,
+ "wiki_enabled": true,
+ "snippets_enabled": false,
+ "created_at": "2013-09-30T13: 46: 02Z",
+ "last_activity_at": "2013-09-30T13: 46: 02Z",
+ "creator_id": 3,
+ "namespace": {
+ "created_at": "2013-09-30T13: 46: 02Z",
+ "description": "",
+ "id": 3,
+ "name": "Diaspora",
+ "owner_id": 1,
+ "path": "diaspora",
+ "updated_at": "2013-09-30T13: 46: 02Z"
+ },
+ "archived": false,
+ "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png"
+ }
+]
+```
+
## Details of a group
Get all details of a group.
@@ -186,7 +250,7 @@ To get more (up to 100), pass the following as an argument to the API call: /groups?per_page=100
```
-And to switch pages add:
+And to switch pages add:
```
/groups?per_page=100&page=2
-```
\ No newline at end of file +```
diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 2f17d4ae06b..366a1f8abec 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -3,8 +3,9 @@ ## List merge requests Get all merge requests for this project. -The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). -The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. +The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). +The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. With GitLab 8.2 the return fields `upvotes` and +`downvotes` are deprecated and always return `0`. ``` GET /projects/:id/merge_requests @@ -31,6 +32,8 @@ Parameters: "project_id": 3, "title": "test1", "state": "opened", + "upvotes": 0, + "downvotes": 0, "author": { "id": 1, "username": "admin", @@ -55,7 +58,7 @@ Parameters: ## Get single MR -Shows information about a single merge request. +Shows information about a single merge request. With GitLab 8.2 the return fields `upvotes` and `downvotes` are deprecated and always return `0`. ``` GET /projects/:id/merge_request/:merge_request_id @@ -75,6 +78,8 @@ Parameters: "project_id": 3, "title": "test1", "state": "merged", + "upvotes": 0, + "downvotes": 0, "author": { "id": 1, "username": "admin", @@ -96,9 +101,48 @@ Parameters: } ``` +## Get single MR commits + +Get a list of merge request commits. + +``` +GET /projects/:id/merge_request/:merge_request_id/commits +``` + +Parameters: + +- `id` (required) - The ID of a project +- `merge_request_id` (required) - The ID of MR + + +```json +[ + { + "id": "ed899a2f4b50b4370feeea94676502b42383c746", + "short_id": "ed899a2f4b5", + "title": "Replace sanitize with escape once", + "author_name": "Dmitriy Zaporozhets", + "author_email": "dzaporozhets@sphereconsultinginc.com", + "created_at": "2012-09-20T11:50:22+03:00", + "message": "Replace sanitize with escape once" + }, + { + "id": "6104942438c14ec7bd21c6cd5bd995272b3faff6", + "short_id": "6104942438c", + "title": "Sanitize for network graph", + "author_name": "randx", + "author_email": "dmitriy.zaporozhets@gmail.com", + "created_at": "2012-09-20T09:06:12+03:00", + "message": "Sanitize for network graph" + } +] +``` + ## Get single MR changes -Shows information about the merge request including its files and changes +Shows information about the merge request including its files and changes. +With GitLab 8.2 the return fields `upvotes` and `downvotes` are deprecated and +always return `0`. ``` GET /projects/:id/merge_request/:merge_request_id/changes @@ -122,6 +166,8 @@ Parameters: "updated_at": "2015-02-02T20:08:49.959Z", "target_branch": "secret_token", "source_branch": "version-1-9", + "upvotes": 0, + "downvotes": 0, "author": { "name": "Chad Hamill", "username": "jarrett", @@ -150,7 +196,7 @@ Parameters: "updated_at": "2015-02-02T19:49:26.013Z", "due_date": null }, - "files": [ + "changes": [ { "old_path": "VERSION", "new_path": "VERSION", @@ -167,7 +213,8 @@ Parameters: ## Create MR -Creates a new merge request. +Creates a new merge request. With GitLab 8.2 the return fields `upvotes` and ` +downvotes` are deprecated and always return `0`. ``` POST /projects/:id/merge_requests @@ -192,6 +239,8 @@ Parameters: "project_id": 3, "title": "test1", "state": "opened", + "upvotes": 0, + "downvotes": 0, "author": { "id": 1, "username": "admin", @@ -217,7 +266,8 @@ If an error occurs, an error number and a message explaining the reason is retur ## Update MR -Updates an existing merge request. You can change the target branch, title, or even close the MR. +Updates an existing merge request. You can change the target branch, title, or even close the MR. With GitLab 8.2 the return fields `upvotes` and `downvotes` +are deprecated and always return `0`. ``` PUT /projects/:id/merge_request/:merge_request_id @@ -242,6 +292,8 @@ Parameters: "title": "test1", "description": "description1", "state": "opened", + "upvotes": 0, + "downvotes": 0, "author": { "id": 1, "username": "admin", @@ -266,7 +318,8 @@ If an error occurs, an error number and a message explaining the reason is retur ## Accept MR -Merge changes submitted with MR using this API. +Merge changes submitted with MR using this API. With GitLab 8.2 the return +fields `upvotes` and `downvotes` are deprecated and always return `0`. If merge success you get `200 OK`. @@ -282,9 +335,57 @@ PUT /projects/:id/merge_request/:merge_request_id/merge Parameters: -- `id` (required) - The ID of a project -- `merge_request_id` (required) - ID of MR -- `merge_commit_message` (optional) - Custom merge commit message +- `id` (required) - The ID of a project +- `merge_request_id` (required) - ID of MR +- `merge_commit_message` (optional) - Custom merge commit message +- `should_remove_source_branch` (optional) - if `true` removes the source branch +- `merged_when_build_succeeds` (optional) - if `true` the MR is merge when the build succeeds + +```json +{ + "id": 1, + "target_branch": "master", + "source_branch": "test1", + "project_id": 3, + "title": "test1", + "state": "merged", + "upvotes": 0, + "downvotes": 0, + "author": { + "id": 1, + "username": "admin", + "email": "admin@example.com", + "name": "Administrator", + "state": "active", + "created_at": "2012-04-29T08:46:00Z" + }, + "assignee": { + "id": 1, + "username": "admin", + "email": "admin@example.com", + "name": "Administrator", + "state": "active", + "created_at": "2012-04-29T08:46:00Z" + } +} +``` + +## Cancel Merge When Build Succeeds + +If successful you'll get `200 OK`. + +If you don't have permissions to accept this merge request - you'll get a 401 + +If the merge request is already merged or closed - you get 405 and error message 'Method Not Allowed' + +In case the merge request is not set to be merged when the build succeeds, you'll also get a 406 error. +``` +PUT /projects/:id/merge_request/:merge_request_id/cancel_merge_when_build_succeeds +``` +Parameters: + +- `id` (required) - The ID of a project +- `merge_request_id` (required) - ID of MR ```json { @@ -294,6 +395,8 @@ Parameters: "project_id": 3, "title": "test1", "state": "merged", + "upvotes": 0, + "downvotes": 0, "author": { "id": 1, "username": "admin", diff --git a/doc/api/notes.md b/doc/api/notes.md index bcba1f62151..4d7ef288df8 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -6,7 +6,8 @@ Notes are comments on snippets, issues or merge requests. ### List project issue notes -Gets a list of all notes for a single issue. +Gets a list of all notes for a single issue. With GitLab 8.2 the return fields +`upvote` and `downvote` are deprecated and always return `false`. ``` GET /projects/:id/issues/:issue_id/notes @@ -32,7 +33,11 @@ Parameters: "created_at": "2013-09-30T13:46:01Z" }, "created_at": "2013-10-02T09:22:45Z", - "system": true + "system": true, + "upvote": false, + "downvote": false, + "noteable_id": 377, + "noteable_type": "Issue" }, { "id": 305, @@ -47,7 +52,11 @@ Parameters: "created_at": "2013-09-30T13:46:01Z" }, "created_at": "2013-10-02T09:56:03Z", - "system": false + "system": true, + "upvote": false, + "downvote": false, + "noteable_id": 121, + "noteable_type": "Issue" } ] ``` @@ -214,7 +223,12 @@ Parameters: "state": "active", "created_at": "2013-09-30T13:46:01Z" }, - "created_at": "2013-10-02T08:57:14Z" + "created_at": "2013-10-02T08:57:14Z", + "system": false, + "upvote": false, + "downvote": false, + "noteable_id": 2, + "noteable_type": "MergeRequest" } ``` diff --git a/doc/api/projects.md b/doc/api/projects.md index 755cc6525c2..658e65c6f01 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -32,7 +32,6 @@ Parameters: - `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` - `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` - `search` (optional) - Return list of authorized projects according to a search criteria -- `ci_enabled_first` - Return projects ordered by ci_enabled flag. Projects with enabled GitLab CI go first ```json [ @@ -59,6 +58,7 @@ Parameters: "path": "diaspora-client", "path_with_namespace": "diaspora/diaspora-client", "issues_enabled": true, + "open_issues_count": 1, "merge_requests_enabled": true, "builds_enabled": true, "wiki_enabled": true, @@ -101,6 +101,7 @@ Parameters: "path": "puppet", "path_with_namespace": "brightbox/puppet", "issues_enabled": true, + "open_issues_count": 1, "merge_requests_enabled": true, "builds_enabled": true, "wiki_enabled": true, @@ -137,7 +138,21 @@ Parameters: - `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` - `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` - `search` (optional) - Return list of authorized projects according to a search criteria -- `ci_enabled_first` - Return projects ordered by ci_enabled flag. Projects with enabled GitLab CI go first + +### List starred projects + +Get a list of projects which are starred by the authenticated user. + +``` +GET /projects/starred +``` + +Parameters: + +- `archived` (optional) - if passed, limit by archived status +- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` +- `search` (optional) - Return list of authorized projects according to a search criteria ### List ALL projects @@ -153,7 +168,6 @@ Parameters: - `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` - `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` - `search` (optional) - Return list of authorized projects according to a search criteria -- `ci_enabled_first` - Return projects ordered by ci_enabled flag. Projects with enabled GitLab CI go first ### Get single project @@ -192,6 +206,7 @@ Parameters: "path": "diaspora-project-site", "path_with_namespace": "diaspora/diaspora-project-site", "issues_enabled": true, + "open_issues_count": 1, "merge_requests_enabled": true, "builds_enabled": true, "wiki_enabled": true, @@ -245,9 +260,17 @@ Parameters: "target_id": 830, "target_type": "Issue", "author_id": 1, - "author_username": "john", "data": null, - "target_title": "Public project search field" + "target_title": "Public project search field", + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/u/root" + }, + "author_username": "root" }, { "title": null, @@ -256,6 +279,14 @@ Parameters: "target_id": null, "target_type": null, "author_id": 1, + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/u/root" + }, "author_username": "john", "data": { "before": "50d4420237a9de7be1304607147aec22e4a14af7", @@ -292,9 +323,56 @@ Parameters: "target_id": 840, "target_type": "Issue", "author_id": 1, - "author_username": "john", "data": null, - "target_title": "Finish & merge Code search PR" + "target_title": "Finish & merge Code search PR", + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/u/root" + }, + "author_username": "root" + }, + { + "title": null, + "project_id": 15, + "action_name": "commented on", + "target_id": 1312, + "target_type": "Note", + "author_id": 1, + "data": null, + "target_title": null, + "created_at": "2015-12-04T10:33:58.089Z", + "note": { + "id": 1312, + "body": "What an awesome day!", + "attachment": null, + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/u/root" + }, + "created_at": "2015-12-04T10:33:56.698Z", + "system": false, + "upvote": false, + "downvote": false, + "noteable_id": 377, + "noteable_type": "Issue" + }, + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/u/root" + }, + "author_username": "root" } ] ``` @@ -335,7 +413,6 @@ Parameters: - `user_id` (required) - user_id of owner - `name` (required) - new project name - `description` (optional) - short project description -- `default_branch` (optional) - 'master' by default - `issues_enabled` (optional) - `merge_requests_enabled` (optional) - `builds_enabled` (optional) diff --git a/doc/api/settings.md b/doc/api/settings.md index d1b93a09c02..96867c67915 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -57,7 +57,7 @@ Parameters: - `default_project_visibility` - what visibility level new project receives - `default_snippet_visibility` - what visibility level new snippet receives - `restricted_signup_domains` - force people to use only corporate emails for signup -- `user_oauth_applications` - allow users to create oauth applicaitons +- `user_oauth_applications` - allow users to create oauth applications - `after_sign_out_path` - where redirect user after logout All parameters are optional. You can send only one that you want to change. diff --git a/doc/api/tags.md b/doc/api/tags.md index b5b90cf6b82..085d387e824 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -29,7 +29,7 @@ Parameters: ] }, "release": { - "tag": "1.0.0", + "tag_name": "1.0.0", "description": "Amazing release. Wow" }, "name": "v1.0.0", @@ -70,7 +70,7 @@ Parameters: ] }, "release": { - "tag": "1.0.0", + "tag_name": "1.0.0", "description": "Amazing release. Wow" }, "name": "v1.0.0", @@ -84,23 +84,48 @@ It returns 200 if the operation succeed. In case of an error, 405 with an explaining error message is returned. -## New release +## Create a new release -Add release notes to the existing git tag +Add release notes to the existing git tag. It returns 201 if the release is +created successfully. If the tag does not exist, 404 is returned. If there +already exists a release for the given tag, 409 is returned. ``` -PUT /projects/:id/repository/:tag/release +POST /projects/:id/repository/tags/:tag_name/release ``` Parameters: - `id` (required) - The ID of a project -- `tag` (required) - The name of a tag +- `tag_name` (required) - The name of a tag - `description` (required) - Release notes with markdown support ```json { - "tag": "1.0.0", + "tag_name": "1.0.0", "description": "Amazing release. Wow" } ``` + +## Update a release + +Updates the release notes of a given release. It returns 200 if the release is +successfully updated. If the tag or the release does not exist, it returns 404 +with a proper error message. + +``` +PUT /projects/:id/repository/tags/:tag_name/release +``` + +Parameters: + +- `id` (required) - The ID of a project +- `tag_name` (required) - The name of a tag +- `description` (required) - Release notes with markdown support + +```json +{ + "tag_name": "1.0.0", + "description": "Amazing release. Wow" +} +```
\ No newline at end of file diff --git a/doc/ci/README.md b/doc/ci/README.md index 97325069ceb..5d9d7a81db3 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -9,6 +9,18 @@ + [Using Docker Images](docker/using_docker_images.md) + [Using Docker Build](docker/using_docker_build.md) + [Using Variables](variables/README.md) ++ [Using SSH keys](ssh_keys/README.md) + +### Languages + ++ [Testing PHP](languages/php.md) + +### Services + ++ [Using MySQL](services/mysql.md) ++ [Using PostgreSQL](services/postgres.md) ++ [Using Redis](services/redis.md) ++ [Using Other Services](docker/using_docker_images.md#how-to-use-other-images-as-services) ### Examples diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index ef8a7ec1e86..31458d61674 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -1,19 +1,29 @@ # Using Docker Images -GitLab CI can use [Docker Engine](https://www.docker.com/) to build projects. -Docker is an open-source project that allows to use predefined images to run applications -in independent "containers" that are run within a single Linux instance. -[Docker Hub](https://registry.hub.docker.com/) have rich database of built images that can be used to build applications. +GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use +[Docker Engine](https://www.docker.com/) to test and build any application. -Docker when used with GitLab CI runs each build in separate and isolated container using predefined image and always from scratch. -It makes it easier to have simple and reproducible build environment that can also be run on your workstation. -This allows you to test all commands from your shell, rather than having to test them on a CI server. +Docker is an open-source project that allows you to use predefined images to +run applications in independent "containers" that are run within a single Linux +instance. [Docker Hub][hub] has a rich database of pre-built images that can be +used to test and build your applications. -### Register Docker runner -To use GitLab Runner with Docker you need to register new runner to use `docker` executor: +Docker, when used with GitLab CI, runs each build in a separate and isolated +container using the predefined image that is set up in +[`.gitlab-ci.yml`](../yaml/README.md). + +This makes it easier to have a simple and reproducible build environment that +can also run on your workstation. The added benefit is that you can test all +the commands that we will explore later from your shell, rather than having to +test them on a dedicated CI server. + +## Register docker runner + +To use GitLab Runner with docker you need to register a new runner to use the +`docker` executor: ```bash -gitlab-ci-multi-runner register \ +gitlab-runner register \ --url "https://gitlab.com/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "docker-ruby-2.1" \ @@ -23,101 +33,79 @@ gitlab-ci-multi-runner register \ --docker-mysql latest ``` -**The registered runner will use `ruby:2.1` image and will run two services (`postgres:latest` and `mysql:latest`) that will be accessible for time of the build.** +The registered runner will use the `ruby:2.1` docker image and will run two +services, `postgres:latest` and `mysql:latest`, both of which will be +accessible during the build process. -### What is image? -The image is the name of any repository that is present in local Docker Engine or any repository that can be found at [Docker Hub](https://registry.hub.docker.com/). -For more information about the image and Docker Hub please read the [Docker Fundamentals](https://docs.docker.com/introduction/understanding-docker/). +## What is image -### What is service? -Service is just another image that is run for time of your build and is linked to your build. This allows you to access the service image during build time. -The service image can run any application, but most common use case is to run some database container, ie.: `mysql`. -It's easier and faster to use existing image, run it as additional container than install `mysql` every time project is built. +The `image` keyword is the name of the docker image that is present in the +local Docker Engine (list all images with `docker images`) or any image that +can be found at [Docker Hub][hub]. For more information about images and Docker +Hub please read the [Docker Fundamentals][] documentation. -#### How is service linked to the build? -There's good document that describes how Docker linking works: [Linking containers together](https://docs.docker.com/userguide/dockerlinks/). -To summarize: if you add `mysql` as service to your application, the image will be used to create container that is linked to build container. -The service container for MySQL will be accessible under hostname `mysql`. -So, **to access your database service you have to connect to host: `mysql` instead of socket or `localhost`**. +In short, with `image` we refer to the docker image, which will be used to +create a container on which your build will run. -### How to use other images as services? -You are not limited to have only database services. -You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). -Look for `[runners.docker]` section: -``` -[runners.docker] - image = "ruby:2.1" - services = ["mysql:latest", "postgres:latest"] -``` +## What is service -For example you need `wordpress` instance to test some API integration with `Wordpress`. -You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). -This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in: -``` -[runners.docker] - image = "ruby:2.1" - services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"] -``` +The `services` keyword defines just another docker image that is run during +your build and is linked to the docker image that the `image` keyword defines. +This allows you to access the service image during build time. -Next time when you run your application the `tutum/wordpress` will be started -and you will have access to it from your build container under hostname: `tutum_wordpress`. +The service image can run any application, but the most common use case is to +run a database container, eg. `mysql`. It's easier and faster to use an +existing image and run it as an additional container than install `mysql` every +time the project is built. -Alias hostname for the service is made from the image name: -1. Everything after `:` is stripped, -2. '/' is replaced to `_`. +You can see some widely used services examples in the relevant documentation of +[CI services examples](../services/README.md). -### Configuring services -Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment. +### How is service linked to the build -GitLab Runner 0.5.0 and up passes all YAML-defined variables to created service containers. +To better understand how the container linking works, read +[Linking containers together](https://docs.docker.com/userguide/dockerlinks/). -1. To configure database name for [postgres](https://registry.hub.docker.com/u/library/postgres/) service, -you need to set POSTGRES_DB. +To summarize, if you add `mysql` as service to your application, the image will +then be used to create a container that is linked to the build container. - ```yaml - services: - - postgres - - variables: - POSTGRES_DB: gitlab - ``` +The service container for MySQL will be accessible under the hostname `mysql`. +So, in order to access your database service you have to connect to the host +named `mysql` instead of a socket or `localhost`. -1. To use [mysql](https://registry.hub.docker.com/u/library/mysql/) service with empty password for time of build, -you need to set MYSQL_ALLOW_EMPTY_PASSWORD. +## Overwrite image and services - ```yaml - services: - - mysql - - variables: - MYSQL_ALLOW_EMPTY_PASSWORD: "yes" - ``` +See [How to use other images as services](#how-to-use-other-images-as-services). -For other possible configuration variables check the -https://registry.hub.docker.com/u/library/mysql/ or https://registry.hub.docker.com/u/library/postgres/ -or README page for any other Docker image. +## How to use other images as services -**Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.** +You are not limited to have only database services. You can add as many +services you need to `.gitlab-ci.yml` or manually modify `config.toml`. +Any image found at [Docker Hub][hub] can be used as a service. -### Overwrite image and services -It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`. -If you add to your YAML the `image` and the `services` these parameters -be used instead of the ones that were specified during runner's registration. -``` +## Define image and services from `.gitlab-ci.yml` + +You can simply define an image that will be used for all jobs and a list of +services that you want to use during build time. + +```yaml image: ruby:2.2 + services: - postgres:9.3 -before_install: + +before_script: - bundle install - + test: script: - bundle exec rake spec ``` -It's possible to define image and service per-job: -``` -before_install: +It is also possible to define different images and services per job: + +```yaml +before_script: - bundle install test:2.1: @@ -135,34 +123,91 @@ test:2.2: - bundle exec rake spec ``` -#### How to enable overwriting? -To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). -You can do that by hand modifying runner configuration: `config.toml`. -Please go to section where is `[runners.docker]` definition for your runner. -Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`: +## Define image and services in `config.toml` + +Look for the `[runners.docker]` section: + ``` [runners.docker] image = "ruby:2.1" - allowed_images = ["ruby:*", "python:*"] - allowed_services = ["mysql:*", "redis:*"] + services = ["mysql:latest", "postgres:latest"] ``` -This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. -You will be able to pick only `ruby` and `python` images. -The same rule can be applied to limit services. -If you are courageous enough, you can make it fully open and accept everything: +The image and services defined this way will be added to all builds run by +that runner. + +## Define an image from a private Docker registry + +Starting with GitLab Runner 0.6.0, you are able to define images located to +private registries that could also require authentication. + +All you have to do is be explicit on the image definition in `.gitlab-ci.yml`. + +```yaml +image: my.registry.tld:5000/namepace/image:tag ``` -[runners.docker] - image = "ruby:2.1" - allowed_images = ["*", "*/*"] - allowed_services = ["*", "*/*"] + +In the example above, GitLab Runner will look at `my.registry.tld:5000` for the +image `namespace/image:tag`. + +If the repository is private you need to authenticate your GitLab Runner in the +registry. Learn how to do that on +[GitLab Runner's documentation][runner-priv-reg]. + +## Accessing the services + +Let's say that you need a Wordpress instance to test some API integration with +your application. + +You can then use for example the [tutum/wordpress][] image in your +`.gitlab-ci.yml`: + +```yaml +services: +- tutum/wordpress:latest ``` -**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.** +When the build is run, `tutum/wordpress` will be started and you will have +access to it from your build container under the hostname `tutum__wordpress`. + +The alias hostname for the service is made from the image name following these +rules: + +1. Everything after `:` is stripped +2. Backslash (`/`) is replaced with double underscores (`__`) + +## Configuring services + +Many services accept environment variables which allow you to easily change +database names or set account names depending on the environment. + +GitLab Runner 0.5.0 and up passes all YAML-defined variables to the created +service containers. + +For all possible configuration variables check the documentation of each image +provided in their corresponding Docker hub page. + +*Note: All variables will be passed to all services containers. It's not +designed to distinguish which variable should go where.* + +### PostgreSQL service example + +See the specific documentation for +[using PostgreSQL as a service](../services/postgres.md). + +### MySQL service example + +See the specific documentation for +[using MySQL as a service](../services/mysql.md). + +## How Docker integration works + +Below is a high level overview of the steps performed by docker during build +time. -### How Docker integration works 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. -1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example). +1. Create cache container to store all volumes as defined in `config.toml` and + `Dockerfile` of build image (`ruby:2.1` as in above example). 1. Create build container and link any service container to build container. 1. Start build container and send build script to the container. 1. Run build script. @@ -171,33 +216,64 @@ If you are courageous enough, you can make it fully open and accept everything: 1. Check exit status of build script. 1. Remove build container and all created service containers. -### How to debug a build locally -1. Create a file with build script: +## How to debug a build locally + +*Note: The following commands are run without root privileges. You should be +able to run docker with your regular user account.* + +First start with creating a file named `build script`: + ```bash -$ cat <<EOF > build_script +cat <<EOF > build_script git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner cd /builds/gitlab-org/gitlab-ci-multi-runner -make <- or any other build step +make EOF ``` -1. Create service containers: +Here we use as an example the GitLab Runner repository which contains a +Makefile, so running `make` will execute the commands defined in the Makefile. +Your mileage may vary, so instead of `make` you could run the command which +is specific to your project. + +Then create some service containers: + ``` -$ docker run -d -n service-mysql mysql:latest -$ docker run -d -n service-postgres postgres:latest +docker run -d -n service-mysql mysql:latest +docker run -d -n service-postgres postgres:latest ``` -This will create two service containers (MySQL and PostgreSQL). -1. Create a build container and execute script in its context: +This will create two service containers, named `service-mysql` and +`service-postgres` which use the latest MySQL and PostgreSQL images +respectively. They will both run in the background (`-d`). + +Finally, create a build container by executing the `build_script` file we +created earlier: + ``` -$ cat build_script | docker run -n build -i -l mysql:service-mysql -l postgres:service-postgres ruby:2.1 /bin/bash +docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script ``` -This will create build container that has two service containers linked. -The build_script is piped using STDIN to bash interpreter which executes the build script in container. -1. At the end remove all containers: +The above command will create a container named `build` that is spawned from +the `ruby:2.1` image and has two services linked to it. The `build_script` is +piped using STDIN to the bash interpreter which in turn executes the +`build_script` in the `build` container. + +When you finish testing and no longer need the containers, you can remove them +with: + ``` docker rm -f -v build service-mysql service-postgres ``` -This will forcefully (the `-f` switch) remove build container and service containers -and all volumes (the `-v` switch) that were created with the container creation. + +This will forcefully (`-f`) remove the `build` container, the two service +containers as well as all volumes (`-v`) that were created with the container +creation. + +[Docker Fundamentals]: https://docs.docker.com/engine/introduction/understanding-docker/ +[hub]: https://hub.docker.com/ +[linking-containers]: https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/ +[tutum/wordpress]: https://registry.hub.docker.com/u/tutum/wordpress/ +[postgres-hub]: https://registry.hub.docker.com/u/library/postgres/ +[mysql-hub]: https://registry.hub.docker.com/u/library/mysql/ +[runner-priv-reg]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry diff --git a/doc/ci/img/builds_tab.png b/doc/ci/img/builds_tab.png Binary files differnew file mode 100644 index 00000000000..d088b8b329d --- /dev/null +++ b/doc/ci/img/builds_tab.png diff --git a/doc/ci/languages/README.md b/doc/ci/languages/README.md new file mode 100644 index 00000000000..54b2343e08b --- /dev/null +++ b/doc/ci/languages/README.md @@ -0,0 +1,7 @@ +### Languages + +This is a list of languages you can test with GitLab CI. Each section has +comprehensive documentation and comes with a test repository hosted on +GitLab.com + ++ [Testing PHP](php.md) diff --git a/doc/ci/languages/php.md b/doc/ci/languages/php.md new file mode 100644 index 00000000000..dacb67fa3ff --- /dev/null +++ b/doc/ci/languages/php.md @@ -0,0 +1,284 @@ +# Testing PHP projects + +This guide covers basic building instructions for PHP projects. + +There are covered two cases: testing using the Docker executor and testing +using the Shell executor. + +## Test PHP projects using the Docker executor + +While it is possible to test PHP apps on any system, this would require manual +configuration from the developer. To overcome this we will be using the +official [PHP docker image][php-hub] that can be found in Docker Hub. + +This will allow us to test PHP projects against different versions of PHP. +However, not everything is plug 'n' play, you still need to onfigure some +things manually. + +As with every build, you need to create a valid `.gitlab-ci.yml` describing the +build environment. + +Let's first specify the PHP image that will be used for the build process +(you can read more about what an image means in the Runner's lingo reading +about [Using Docker images](../docker/using_docker_images.md#what-is-image)). + +Start by adding the image to your `.gitlab-ci.yml`: + +```yaml +image: php:5.6 +``` + +The official images are great, but they lack a few useful tools for testing. +We need to first prepare the build environment. A way to overcome this is to +create a script which installs all prerequisites prior the actual testing is +done. + +Let's create a `ci/docker_install.sh` file in the root directory of our +repository with the following content: + +```bash +#!/bin/bash + +# We need to install dependencies only for Docker +[[ ! -e /.dockerinit ]] && exit 0 + +set -xe + +# Install git (the php image doesn't have it) which is required by composer +apt-get update -yqq +apt-get install git -yqq + +# Install phpunit, the tool that we will use for testing +curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar +chmod +x /usr/local/bin/phpunit + +# Install mysql driver +# Here you can install any other extension that you need +docker-php-ext-install pdo_mysql +``` + +You might wonder what `docker-php-ext-install` is. In short, it is a script +provided by the official php docker image that you can use to easilly install +extensions. For more information read the the documentation at +<https://hub.docker.com/_/php/>. + +Now that we created the script that contains all prerequisites for our build +environment, let's add it in `.gitlab-ci.yml`: + +```yaml +... + +before_script: +- bash ci/docker_install.sh > /dev/null + +... +``` + +Last step, run the actual tests using `phpunit`: + +```yaml +... + +test:app: + script: + - phpunit --configuration phpunit_myapp.xml + +... +``` + +Finally, commit your files and push them to GitLab to see your build succeeding +(or failing). + +The final `.gitlab-ci.yml` should look similar to this: + +```yaml +# Select image from https://hub.docker.com/_/php/ +image: php:5.6 + +before_script: +# Install dependencies +- ci/docker_install.sh > /dev/null + +test:app: + script: + - phpunit --configuration phpunit_myapp.xml +``` + +### Test against different PHP versions in Docker builds + +Testing against multiple versions of PHP is super easy. Just add another job +with a different docker image version and the runner will do the rest: + +```yaml +before_script: +# Install dependencies +- ci/docker_install.sh > /dev/null + +# We test PHP5.6 +test:5.6: + image: php:5.6 + script: + - phpunit --configuration phpunit_myapp.xml + +# We test PHP7.0 (good luck with that) +test:7.0: + image: php:7.0 + script: + - phpunit --configuration phpunit_myapp.xml +``` + +### Custom PHP configuration in Docker builds + +There are times where you will need to customise your PHP environment by +putting your `.ini` file into `/usr/local/etc/php/conf.d/`. For that purpose +add a `before_script` action: + +```yaml +before_script: +- cp my_php.ini /usr/local/etc/php/conf.d/test.ini +``` + +Of course, `my_php.ini` must be present in the root directory of your repository. + +## Test PHP projects using the Shell executor + +The shell executor runs your builds in a terminal session on your server. +Thus, in order to test your projects you first need to make sure that all +dependencies are installed. + +For example, in a VM running Debian 8 we first update the cache, then we +install `phpunit` and `php5-mysql`: + +```bash +sudo apt-get update -y +sudo apt-get install -y phpunit php5-mysql +``` + +Next, add the following snippet to your `.gitlab-ci.yml`: + +```yaml +test:app: + script: + - phpunit --configuration phpunit_myapp.xml +``` + +Finally, push to GitLab and let the tests begin! + +### Test against different PHP versions in Shell builds + +The [phpenv][] project allows you to easily manage different versions of PHP +each with its own config. This is specially usefull when testing PHP projects +with the Shell executor. + +You will have to install it on your build machine under the `gitlab-runner` +user following [the upstream installation guide][phpenv-installation]. + +Using phpenv also allows to easily configure the PHP environment with: + +``` +phpenv config-add my_config.ini +``` + +*__Important note:__ It seems `phpenv/phpenv` + [is abandoned](https://github.com/phpenv/phpenv/issues/57). There is a fork + at [madumlao/phpenv](https://github.com/madumlao/phpenv) that tries to bring + the project back to life. [CHH/phpenv](https://github.com/CHH/phpenv) also + seems like a good alternative. Picking any of the mentioned tools will work + with the basic phpenv commands. Guiding you to choose the right phpenv is out + of the scope of this tutorial.* + +### Install custom extensions + +Since this is a pretty bare installation of the PHP environment, you may need +some extensions that are not currently present on the build machine. + +To install additional extensions simply execute: + +```bash +pecl install <extension> +``` + +It's not advised to add this to `.gitlab-ci.yml`. You should execute this +command once, only to setup the build environment. + +## Extend your tests + +### Using atoum + +Instead of PHPUnit, you can use any other tool to run unit tests. For example +you can use [atoum](https://github.com/atoum/atoum): + +```yaml +before_script: +- wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar + +test:atoum: + script: + - php mageekguy.atoum.phar +``` + +### Using Composer + +The majority of the PHP projects use Composer for managing their PHP packages. +In order to execute Composer before running your tests, simply add the +following in your `.gitlab-ci.yml`: + +```yaml +... + +# Composer stores all downloaded packages in the vendor/ directory. +# Do not use the following if the vendor/ directory is commited to +# your git repository. +cache: + paths: + - vendor/ + +before_script: +# Install composer dependencies +- curl -sS https://getcomposer.org/installer | php +- php composer.phar install + +... +``` + +## Access private packages / dependencies + +If your test suite needs to access a private repository, you need to configure +[the SSH keys](../ssh_keys/README.md) in order to be able to clone it. + +## Use databases or other services + +Most of the time you will need a running database in order for your tests to +run. If you are using the Docker executor you can leverage Docker's ability to +link to other containers. In GitLab Runner lingo, this can be achieved by +defining a `service`. + +This functionality is covered in [the CI services](../services/README.md) +documentation. + +## Testing things locally + +With GitLab Runner 1.0 you can also test any changes locally. From your +terminal execute: + +```bash +# Check using docker executor +gitlab-runner exec docker test:app + +# Check using shell executor +gitlab-runner exec shell test:app +``` + +## Example project + +We have set up an [Example PHP Project][php-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[php-hub]: https://hub.docker.com/_/php/ +[phpenv]: https://github.com/phpenv/phpenv +[phpenv-installation]: https://github.com/phpenv/phpenv#installation +[php-example-repo]: https://gitlab.com/gitlab-examples/php diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index d69064a91fd..a9b36139de9 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -1,44 +1,62 @@ # Quick Start -To start building projects with GitLab CI a few steps needs to be done. +Starting from version 8.0, GitLab Continuous Integration (CI) is fully +integrated into GitLab itself and is enabled by default on all projects. -## 1. Install GitLab and CI +This guide assumes that you: -First you need to have a working GitLab and GitLab CI instance. +- have a working GitLab instance of version 8.0 or higher or are using + [GitLab.com](https://gitlab.com/users/sign_in) +- have a project in GitLab that you would like to use CI for -You can omit this step if you use [GitLab.com](https://GitLab.com/). +In brief, the steps needed to have a working CI can be summed up to: -## 2. Create repository on GitLab +1. Create a new project +1. Add `.gitlab-ci.yml` to the git repository and push to GitLab +1. Configure a Runner -Once you login on your GitLab add a new repository where you will store your source code. -Push your application to that repository. +From there on, on every push to your git repository the build will be +automagically started by the Runner and will appear under the project's +`/builds` page. -## 3. Add project to CI +Now, let's break it down to pieces and work on solving the GitLab CI puzzle. -The next part is to login to GitLab CI. -Point your browser to the URL you have set GitLab or use [gitlab.com/ci](https://gitlab.com/ci/). +## Creating a `.gitlab-ci.yml` file -On the first screen you will see a list of GitLab's projects that you have access to: +Before you create `.gitlab-ci.yml` let's first explain in brief what this is +all about. - +### What is `.gitlab-ci.yml` -Click **Add Project to CI**. -This will create project in CI and authorize GitLab CI to fetch sources from GitLab. +The `.gitlab-ci.yml` file is where you configure what CI does with your project. +It lives in the root of your repository. -> GitLab CI creates unique token that is used to configure GitLab CI service in GitLab. -> This token allows to access GitLab's repository and configures GitLab to trigger GitLab CI webhook on **Push events** and **Tag push events**. -> You can see that token by going to Project's Settings > Services > GitLab CI. -> You will see there token, the same token is assigned in GitLab CI settings of project. +On any push to your repository, GitLab will look for the `.gitlab-ci.yml` +file and start builds on _Runners_ according to the contents of the file, +for that commit. -## 4. Create project's configuration - .gitlab-ci.yml +Because `.gitlab-ci.yml` is in the repository, it is version controlled, +old versions still build succesfully, forks can easily make use of CI, +branches can have separate builds and you have a single source of truth for CI. +You can read more about the reasons why we are using `.gitlab-ci.yml` +[in our blog about it][blog-ci]. -The next: You have to define how your project will be built. -GitLab CI uses [YAML](https://en.wikipedia.org/wiki/YAML) file to store build configuration. -You need to create `.gitlab-ci.yml` in root directory of your repository: +**Note:** `.gitlab-ci.yml` is a [YAML](https://en.wikipedia.org/wiki/YAML) file +so you have to pay extra attention to the identation. Always use spaces, not +tabs. + +### Creating a simple `.gitlab-ci.yml` file + +You need to create a file named `.gitlab-ci.yml` in the root directory of your +repository. Below is an example for a Ruby on Rails project. ```yaml before_script: - - bundle install + - apt-get update -qq && apt-get install -y -qq sqlite3 libsqlite3-dev nodejs + - ruby -v + - which ruby + - gem install bundler --no-ri --no-rdoc + - bundle install --jobs $(nproc) "${FLAGS[@]}" rspec: script: @@ -49,71 +67,131 @@ rubocop: - bundle exec rubocop ``` -This is the simplest possible build configuration that will work for most Ruby applications: -1. Define two jobs `rspec` and `rubocop` with two different commands to be executed. -1. Before every job execute commands defined by `before_script`. +This is the simplest possible build configuration that will work for most Ruby +applications: + +1. Define two jobs `rspec` and `rubocop` (the names are arbitrary) with + different commands to be executed. +1. Before every job, the commands defined by `before_script` are executed. -The `.gitlab-ci.yml` defines set of jobs with constrains how and when they should be run. -The jobs are defined as top-level elements with name and always have to contain the `script`. -Jobs are used to create builds, which are then picked by [runners](../runners/README.md) and executed within environment of the runner. -What is important that each job is run independently from each other. +The `.gitlab-ci.yml` file defines sets of jobs with constraints of how and when +they should be run. The jobs are defined as top-level elements with a name (in +our case `rspec` and `rubocop`) and always have to contain the `script` keyword. +Jobs are used to create builds, which are then picked by +[Runners](../runners/README.md) and executed within the environment of the Runner. -For more information and complete `.gitlab-ci.yml` syntax, please check the [Configuring project (.gitlab-ci.yml)](../yaml/README.md). +What is important is that each job is run independently from each other. -## 5. Add file and push .gitlab-ci.yml to repository +If you want to check whether your `.gitlab-ci.yml` file is valid, there is a +Lint tool under the page `/ci/lint` of your GitLab instance. You can also find +the link under **Settings > CI settings** in your project. -Once you created `.gitlab-ci.yml` you should add it to git repository and push it to GitLab. +For more information and a complete `.gitlab-ci.yml` syntax, please check +[the documentation on .gitlab-ci.yml](../yaml/README.md). + +### Push `.gitlab-ci.yml` to GitLab + +Once you've created `.gitlab-ci.yml`, you should add it to your git repository +and push it to GitLab. ```bash git add .gitlab-ci.yml -git commit +git commit -m "Add .gitlab-ci.yml" git push origin master ``` -If you refresh the project's page on GitLab CI you will notice a one new commit: +Now if you go to the **Builds** page you will see that the builds are pending. + +You can also go to the **Commits** page and notice the little clock icon next +to the commit SHA. + + + +Clicking on the clock icon you will be directed to the builds page for that +specific commit. + + + +Notice that there are two jobs pending which are named after what we wrote in +`.gitlab-ci.yml`. The red triangle indicates that there is no Runner configured +yet for these builds. + +The next step is to configure a Runner so that it picks the pending jobs. + +## Configuring a Runner + +In GitLab, Runners run the builds that you define in `.gitlab-ci.yml`. +A Runner can be a virtual machine, a VPS, a bare-metal machine, a docker +container or even a cluster of containers. GitLab and the Runners communicate +through an API, so the only needed requirement is that the machine on which the +Runner is configured to has Internet access. + +A Runner can be specific to a certain project or serve multiple projects in +GitLab. If it serves all projects it's called a _Shared Runner_. + +Find more information about different Runners in the +[Runners](../runners/README.md) documentation. + +You can find whether any Runners are assigned to your project by going to +**Settings > Runners**. Setting up a Runner is easy and straightforward. The +official Runner supported by GitLab is written in Go and can be found at +<https://gitlab.com/gitlab-org/gitlab-ci-multi-runner>. + +In order to have a functional Runner you need to follow two steps: + +1. [Install it][runner-install] +2. [Configure it](../runners/README.md#registering-a-specific-runner) + +Follow the links above to set up your own Runner or use a Shared Runner as +described in the next section. + +For other types of unofficial Runners written in other languages, see the +[instructions for the various GitLab Runners](https://about.gitlab.com/gitlab-ci/#gitlab-runner). + +Once the Runner has been set up, you should see it on the Runners page of your +project, following **Settings > Runners**. - + -However the commit has status **pending** which means that commit was not yet picked by runner. +### Shared Runners -## 6. Configure runner +If you use [GitLab.com](https://gitlab.com/) you can use **Shared Runners** +provided by GitLab Inc. -In GitLab CI, Runners run your builds. -A runner is a machine (can be virtual, bare-metal or VPS) that picks up builds through the coordinator API of GitLab CI. +These are special virtual machines that run on GitLab's infrastructure and can +build any project. -A runner can be specific to a certain project or serve any project in GitLab CI. -A runner that serves all projects is called a shared runner. -More information about different runner types can be found in [Configuring runner](../runners/README.md). +To enable **Shared Runners** you have to go to your project's +**Settings > Runners** and click **Enable shared runners**. -To check if you have runners assigned to your project go to **Runners**. You will find there information how to setup project specific runner: +[Read more on Shared Runners](../runners/README.md). -1. Install GitLab Runner software. Checkout the [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner) section to install it. -1. Specify following URL during runner setup: https://gitlab.com/ci/ -1. Use the following registration token during setup: TOKEN +## Seeing the status of your build -If you do it correctly your runner should be shown under **Runners activated for this project**: +After configuring the Runner succesfully, you should see the status of your +last commit change from _pending_ to either _running_, _success_ or _failed_. - +You can view all builds, by going to the **Builds** page in your project. -### Shared runners + -If you use [gitlab.com/ci](https://gitlab.com/ci/) you can use **Shared runners** provided by GitLab Inc. -These are special virtual machines that are run on GitLab's infrastructure that can build any project. -To enable **Shared runners** you have to go to **Runners** and click **Enable shared runners** for this project. +By clicking on a Build ID, you will be able to see the log of that build. +This is important to diagnose why a build failed or acted differently than +you expected. -## 7. Check status of commit + -If everything went OK and you go to commit, the status of the commit should change from **pending** to either **running**, **success** or **failed**. +You are also able to view the status of any commit in the various pages in +GitLab, such as **Commits** and **Merge Requests**. - +## Next steps -You can click **Build ID** to view build log for specific job. +Awesome! You started using CI in GitLab! -## 8. Congratulations! +Next you can look into doing more with the CI. Many people are using GitLab +to package, containerize, test and deploy software. -You managed to build your first project using GitLab CI. -You may need to tune your `.gitlab-ci.yml` file to implement build plan for your project. -A few examples how it can be done you can find on [Examples](../examples/README.md) page. +Visit our various languages examples at <https://gitlab.com/groups/gitlab-examples>. -GitLab CI also offers **the Lint** tool to verify validity of your `.gitlab-ci.yml` which can be useful to troubleshoot potential problems. -The Lint is available from project's settings or by adding `/lint` to GitLab CI url. +[runner-install]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/tree/master#installation +[blog-ci]: https://about.gitlab.com/2015/05/06/why-were-replacing-gitlab-ci-jobs-with-gitlab-ci-dot-yml/ diff --git a/doc/ci/quick_start/build_status.png b/doc/ci/quick_start/build_status.png Binary files differdeleted file mode 100644 index 333259e6acd..00000000000 --- a/doc/ci/quick_start/build_status.png +++ /dev/null diff --git a/doc/ci/quick_start/commit_status.png b/doc/ci/quick_start/commit_status.png Binary files differdeleted file mode 100644 index 725b79e6f91..00000000000 --- a/doc/ci/quick_start/commit_status.png +++ /dev/null diff --git a/doc/ci/quick_start/img/build_log.png b/doc/ci/quick_start/img/build_log.png Binary files differnew file mode 100644 index 00000000000..89e6cd40cb6 --- /dev/null +++ b/doc/ci/quick_start/img/build_log.png diff --git a/doc/ci/quick_start/img/builds_status.png b/doc/ci/quick_start/img/builds_status.png Binary files differnew file mode 100644 index 00000000000..b8e6c2a361a --- /dev/null +++ b/doc/ci/quick_start/img/builds_status.png diff --git a/doc/ci/quick_start/img/new_commit.png b/doc/ci/quick_start/img/new_commit.png Binary files differnew file mode 100644 index 00000000000..3d3c9d5c0bd --- /dev/null +++ b/doc/ci/quick_start/img/new_commit.png diff --git a/doc/ci/quick_start/img/runners_activated.png b/doc/ci/quick_start/img/runners_activated.png Binary files differnew file mode 100644 index 00000000000..eafcfd6ecd5 --- /dev/null +++ b/doc/ci/quick_start/img/runners_activated.png diff --git a/doc/ci/quick_start/img/single_commit_status_pending.png b/doc/ci/quick_start/img/single_commit_status_pending.png Binary files differnew file mode 100644 index 00000000000..23b3bb5acfc --- /dev/null +++ b/doc/ci/quick_start/img/single_commit_status_pending.png diff --git a/doc/ci/quick_start/img/status_pending.png b/doc/ci/quick_start/img/status_pending.png Binary files differnew file mode 100644 index 00000000000..a049ec2a5ba --- /dev/null +++ b/doc/ci/quick_start/img/status_pending.png diff --git a/doc/ci/quick_start/new_commit.png b/doc/ci/quick_start/new_commit.png Binary files differdeleted file mode 100644 index 3839e893c17..00000000000 --- a/doc/ci/quick_start/new_commit.png +++ /dev/null diff --git a/doc/ci/quick_start/projects.png b/doc/ci/quick_start/projects.png Binary files differdeleted file mode 100644 index 0b3430a69db..00000000000 --- a/doc/ci/quick_start/projects.png +++ /dev/null diff --git a/doc/ci/quick_start/runners.png b/doc/ci/quick_start/runners.png Binary files differdeleted file mode 100644 index 25b4046bc00..00000000000 --- a/doc/ci/quick_start/runners.png +++ /dev/null diff --git a/doc/ci/quick_start/runners_activated.png b/doc/ci/quick_start/runners_activated.png Binary files differdeleted file mode 100644 index c934bd12f41..00000000000 --- a/doc/ci/quick_start/runners_activated.png +++ /dev/null diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md new file mode 100644 index 00000000000..1ebb0a4a250 --- /dev/null +++ b/doc/ci/services/README.md @@ -0,0 +1,9 @@ +## GitLab CI Services + +GitLab CI uses the `services` keyword to define what docker containers should be +linked with your base image. Below is a list of examples you may use. + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) ++ [Using Other Services](../docker/using_docker_images.md#how-to-use-other-images-as-services) diff --git a/doc/ci/services/docker-services.md b/doc/ci/services/docker-services.md new file mode 100644 index 00000000000..df36ebaf7d4 --- /dev/null +++ b/doc/ci/services/docker-services.md @@ -0,0 +1,5 @@ +## GitLab CI Services + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md new file mode 100644 index 00000000000..c66d77122b2 --- /dev/null +++ b/doc/ci/services/mysql.md @@ -0,0 +1,118 @@ +# Using MySQL + +As many applications depend on MySQL as their database, you will eventually +need it in order for your tests to run. Below you are guided how to do this +with the Docker and Shell executors of GitLab Runner. + +## Use MySQL with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - mysql:latest + +variables: + # Configure mysql environment variables (https://hub.docker.com/_/mysql/) + MYSQL_DATABASE: el_duderino + MYSQL_ROOT_PASSWORD: mysql_strong_password +``` + +And then configure your application to use the database, for example: + +```yaml +Host: mysql +User: root +Password: mysql_strong_password +Database: el_duderino +``` + +If you are wondering why we used `mysql` for the `Host`, read more at +[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build). + +You can also use any other docker image available on [Docker Hub][hub-mysql]. +For example, to use MySQL 5.5 the service becomes `mysql:5.5`. + +The `mysql` image can accept some environment variables. For more details +check the documentation on [Docker Hub][hub-mysql]. + +## Use MySQL with the Shell executor + +You can also use MySQL on manually configured servers that are using +GitLab Runner with the Shell executor. + +First install the MySQL server: + +```bash +sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev +``` + +Pick a MySQL root password (can be anything), and type it twice when asked. + +*Note: As a security measure you can run `mysql_secure_installation` to +remove anonymous users, drop the test database and disable remote logins with +the root user.* + +The next step is to create a user, so login to MySQL as root: + +```bash +mysql -u root -p +``` + +Then create a user (in our case `runner`) which will be used by your +application. Change `$password` in the command below to a real strong password. + +*Note: Do not type `mysql>`, this is part of the MySQL prompt.* + +```bash +mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password'; +``` + +Create the database: + +```bash +mysql> CREATE DATABASE IF NOT EXISTS `el_duderino` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; +``` + +Grant the necessary permissions on the database: + +```bash +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `el_duderino`.* TO 'runner'@'localhost'; +``` + +If all went well you can now quit the database session: + +```bash +mysql> \q +``` + +Now, try to connect to the newly created database to check that everything is +in place: + +```bash +mysql -u runner -p -D el_duderino +``` + +As a final step, configure your application to use the database, for example: + +```bash +Host: localhost +User: runner +Password: $password +Database: el_duderino +``` + +## Example project + +We have set up an [Example MySQL Project][mysql-example-repo] for your +convenience that runs on [GitLab.com](https://gitlab.com) using our publicly +available [shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-mysql]: https://hub.docker.com/_/mysql/ +[mysql-example-repo]: https://gitlab.com/gitlab-examples/mysql diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md new file mode 100644 index 00000000000..17d21dbda1c --- /dev/null +++ b/doc/ci/services/postgres.md @@ -0,0 +1,114 @@ +# Using PostgreSQL + +As many applications depend on PostgreSQL as their database, you will +eventually need it in order for your tests to run. Below you are guided how to +do this with the Docker and Shell executors of GitLab Runner. + +## Use PostgreSQL with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - postgres:latest + +variables: + POSTGRES_DB: nice_marmot + POSTGRES_USER: runner + POSTGRES_PASSWORD: "" +``` + +And then configure your application to use the database, for example: + +```yaml +Host: postgres +User: runner +Password: +Database: nice_marmot +``` + +If you are wondering why we used `postgres` for the `Host`, read more at +[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build). + +You can also use any other docker image available on [Docker Hub][hub-pg]. +For example, to use PostgreSQL 9.3 the service becomes `postgres:9.3`. + +The `postgres` image can accept some environment variables. For more details +check the documentation on [Docker Hub][hub-pg]. + +## Use PostgreSQL with the Shell executor + +You can also use PostgreSQL on manually configured servers that are using +GitLab Runner with the Shell executor. + +First install the PostgreSQL server: + +```bash +sudo apt-get install -y postgresql postgresql-client libpq-dev +``` + +The next step is to create a user, so login to PostgreSQL: + +```bash +sudo -u postgres psql -d template1 +``` + +Then create a user (in our case `runner`) which will be used by your +application. Change `$password` in the command below to a real strong password. + +*__Note:__ Do not type `template1=#`, this is part of the PostgreSQL prompt.* + +```bash +template1=# CREATE USER runner WITH PASSWORD '$password' CREATEDB; +``` + +*__Note:__ Notice that we created the user with the privilege to be able to +create databases (`CREATEDB`). In the following steps we will create a database +explicitly for that user but having that privilege can be useful if in your +testing framework you have tools that drop and create databases.* + +Create the database and grant all privileges on it for the user `runner`: + +```bash +template1=# CREATE DATABASE nice_marmot OWNER runner; +``` + +If all went well you can now quit the database session: + +```bash +template1=# \q +``` + +Now, try to connect to the newly created database with the user `runner` to +check that everything is in place. + +```bash +psql -U runner -h localhost -d nice_marmot -W +``` + +*__Note:__ We are explicitly telling `psql` to connect to localhost in order +to use the md5 authentication. If you omit this step you will be denied access.* + +Finally, configure your application to use the database, for example: + +```yaml +Host: localhost +User: runner +Password: $password +Database: nice_marmot +``` + +## Example project + +We have set up an [Example PostgreSQL Project][postgres-example-repo] for your +convenience that runs on [GitLab.com](https://gitlab.com) using our publicly +available [shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-pg]: https://hub.docker.com/_/postgres/ +[postgres-example-repo]: https://gitlab.com/gitlab-examples/postgres diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md new file mode 100644 index 00000000000..b281e8f9f60 --- /dev/null +++ b/doc/ci/services/redis.md @@ -0,0 +1,69 @@ +# Using Redis + +As many applications depend on Redis as their key-value store, you will +eventually need it in order for your tests to run. Below you are guided how to +do this with the Docker and Shell executors of GitLab Runner. + +## Use Redis with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - redis:latest +``` + +Then you need to configure your application to use the Redis database, for +example: + +```yaml +Host: redis +``` + +And that's it. Redis will now be available to be used within your testing +framework. + +You can also use any other docker image available on [Docker Hub][hub-redis]. +For example, to use Redis 2.8 the service becomes `redis:2.8`. + +## Use Redis with the Shell executor + +Redis can also be used on manually configured servers that are using GitLab +Runner with the Shell executor. + +In your build machine install the Redis server: + +```bash +sudo apt-get install redis-server +``` + +Verify that you can connect to the server with the `gitlab-runner` user: + +```bash +# Try connecting the the Redis server +sudo -u gitlab-runner -H redis-cli + +# Quit the session +127.0.0.1:6379> quit +``` + +Finally, configure your application to use the database, for example: + +```yaml +Host: localhost +``` + +## Example project + +We have set up an [Example Redis Project][redis-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-redis]: https://hub.docker.com/_/redis/ +[redis-example-repo]: https://gitlab.com/gitlab-examples/redis diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md new file mode 100644 index 00000000000..210f9c3e849 --- /dev/null +++ b/doc/ci/ssh_keys/README.md @@ -0,0 +1,109 @@ +# Using SSH keys + +GitLab currently doesn't have built-in support for managing SSH keys in a build +environment. + +The SSH keys can be useful when: + +1. You want to checkout internal submodules +2. You want to download private packages using your package manager (eg. bundler) +3. You want to deploy your application to eg. Heroku or your own server +4. You want to execute SSH commands from the build server to the remote server +5. You want to rsync files from your build server to the remote server + +If anything of the above rings a bell, then you most likely need an SSH key. + +## Inject keys in your build server + +The most widely supported method is to inject an SSH key into your build +environment by extending your `.gitlab-ci.yml`. + +This is the universal solution which works with any type of executor +(docker, shell, etc.). + +### How it works + +1. Create a new SSH key pair with [ssh-keygen][] +2. Add the private key as a **Secret Variable** to the project +3. Run the [ssh-agent][] during build to load the private key. + +## SSH keys when using the Docker executor + +You will first need to create an SSH key pair. For more information, follow the +instructions to [generate an SSH key](../ssh/README.md). + +Then, create a new **Secret Variable** in your project settings on GitLab +following **Settings > Variables**. As **Key** add the name `SSH_PRIVATE_KEY` +and in the **Value** field paste the content of your _private_ key that you +created earlier. + +Next you need to modify your `.gitlab-ci.yml` with a `before_script` action. +Add it to the top: + +``` +before_script: + # Install ssh-agent if not already installed, it is required by Docker. + # (change apt-get to yum if you use a CentOS-based image) + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + + # Run ssh-agent (inside the build environment) + - eval $(ssh-agent -s) + + # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store + - ssh-add <(echo "$SSH_PRIVATE_KEY") + + # For Docker builds disable host key checking. Be aware that by adding that + # you are suspectible to man-in-the-middle attacks. + # WARNING: Use this only with the Docker executor, if you use it with shell + # you will overwrite your user's SSH config. + - mkdir -p ~/.ssh + - '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config` +``` + +As a final step, add the _public_ key from the one you created earlier to the +services that you want to have an access to from within the build environment. +If you are accessing a private GitLab repository you need to add it as a +[deploy key](../ssh/README.md#deploy-keys). + +That's it! You can now have access to private servers or repositories in your +build environment. + +## SSH keys when using the Shell executor + +If you are using the Shell executor and not Docker, it is easier to set up an +SSH key. + +You can generate the SSH key from the machine that GitLab Runner is installed +on, and use that key for all projects that are run on this machine. + +First, you need to login to the server that runs your builds. + +Then from the terminal login as the `gitlab-runner` user and generate the SSH +key pair as described in the [SSH keys documentation](../ssh/README.md). + +As a final step, add the _public_ key from the one you created earlier to the +services that you want to have an access to from within the build environment. +If you are accessing a private GitLab repository you need to add it as a +[deploy key](../ssh/README.md#deploy-keys). + +Once done, try to login to the remote server in order to accept the fingerprint: + +```bash +ssh <address-of-my-server> +``` + +For accessing repositories on GitLab.com, the `<address-of-my-server>` would be +`git@gitlab.com`. + +## Example project + +We have set up an [Example SSH Project][ssh-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[ssh-keygen]: http://linux.die.net/man/1/ssh-keygen +[ssh-agent]: http://linux.die.net/man/1/ssh-agent +[ssh-example-repo]: https://gitlab.com/gitlab-examples/ssh-private-key/ diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 022afb70042..b99ea25a3fe 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -27,7 +27,6 @@ The API_TOKEN will take the Secure Variable value: `SECURE`. | **CI_BUILD_TAG** | 0.5 | The commit tag name. Present only when building tags. | | **CI_BUILD_NAME** | 0.5 | The name of the build as defined in `.gitlab-ci.yml` | | **CI_BUILD_STAGE** | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| **CI_BUILD_BEFORE_SHA** | all | The first commit that were included in push request | | **CI_BUILD_REF_NAME** | all | The branch or tag name for which project is built | | **CI_BUILD_ID** | all | The unique id of the current build that GitLab CI uses internally | | **CI_BUILD_REPO** | all | The URL to clone the Git repository | @@ -40,7 +39,6 @@ The API_TOKEN will take the Secure Variable value: `SECURE`. Example values: ```bash -export CI_BUILD_BEFORE_SHA="9df57456fa9de2a6d335ca5edf9750ed812b9df0" export CI_BUILD_ID="50" export CI_BUILD_REF="1ecfd275763eff1d6b4844ea3168962458c9f27a" export CI_BUILD_REF_NAME="master" diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 3e6071a2ee7..fd0d49de4e4 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,9 +1,12 @@ # Configuration of your builds with .gitlab-ci.yml -From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (**.gitlab-ci.yml**) for the project configuration. -It is placed in the root of your repository and contains definitions of how your project should be built. -The YAML file defines a set of jobs with constraints stating when they should be run. -The jobs are defined as top-level elements with a name and always have to contain the `script` clause: +From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) +file (`.gitlab-ci.yml`) for the project configuration. It is placed in the root +of your repository and contains definitions of how your project should be built. + +The YAML file defines a set of jobs with constraints stating when they should +be run. The jobs are defined as top-level elements with a name and always have +to contain the `script` clause: ```yaml job1: @@ -13,15 +16,21 @@ job2: script: "execute-script-for-job2" ``` -The above example is the simplest possible CI configuration with two separate jobs, -where each of the jobs executes a different command. -Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. +The above example is the simplest possible CI configuration with two separate +jobs, where each of the jobs executes a different command. + +Of course a command can execute code directly (`./configure;make;make install`) +or run a script (`test.sh`) in the repository. -Jobs are used to create builds, which are then picked up by [runners](../runners/README.md) and executed within the environment of the runner. -What is important, is that each job is run independently from each other. +Jobs are used to create builds, which are then picked up by +[runners](../runners/README.md) and executed within the environment of the +runner. What is important, is that each job is run independently from each +other. ## .gitlab-ci.yml -The YAML syntax allows for using more complex job specifications than in the above example: + +The YAML syntax allows for using more complex job specifications than in the +above example: ```yaml image: ruby:2.1 @@ -46,26 +55,31 @@ job1: - docker ``` -There are a few `keywords` that can't be used as job names: +There are a few reserved `keywords` that **cannot** be used as job names: -| keyword | required | description | +| Keyword | Required | Description | |---------------|----------|-------------| -| image | optional | Use docker image, covered in [Use Docker](../docker/README.md) | -| services | optional | Use docker services, covered in [Use Docker](../docker/README.md) | -| stages | optional | Define build stages | -| types | optional | Alias for `stages` | -| before_script | optional | Define commands prepended for each job's script | -| variables | optional | Define build variables | -| cache | optional | Define list of files that should be cached between subsequent runs | +| image | no | Use docker image, covered in [Use Docker](../docker/README.md) | +| services | no | Use docker services, covered in [Use Docker](../docker/README.md) | +| stages | no | Define build stages | +| types | no | Alias for `stages` | +| before_script | no | Define commands that run before each job's script | +| variables | no | Define build variables | +| cache | no | Define list of files that should be cached between subsequent runs | ### image and services -This allows to specify a custom Docker image and a list of services that can be used for time of the build. -The configuration of this feature is covered in separate document: [Use Docker](../docker/README.md). + +This allows to specify a custom Docker image and a list of services that can be +used for time of the build. The configuration of this feature is covered in +separate document: [Use Docker](../docker/README.md). ### before_script -`before_script` is used to define the command that should be run before all builds, including deploy builds. This can be an array or a multiline string. + +`before_script` is used to define the command that should be run before all +builds, including deploy builds. This can be an array or a multi-line string. ### stages + `stages` is used to define build stages that can be used by jobs. The specification of `stages` allows for having flexible multi stage pipelines. @@ -75,7 +89,8 @@ The ordering of elements in `stages` defines the ordering of builds' execution: 1. Builds of next stage are run after success. Let's consider the following example, which defines 3 stages: -``` + +```yaml stages: - build - test @@ -86,21 +101,26 @@ stages: 1. If all jobs of `build` succeeds, the `test` jobs are executed in parallel. 1. If all jobs of `test` succeeds, the `deploy` jobs are executed in parallel. 1. If all jobs of `deploy` succeeds, the commit is marked as `success`. -1. If any of the previous jobs fails, the commit is marked as `failed` and no jobs of further stage are executed. +1. If any of the previous jobs fails, the commit is marked as `failed` and no + jobs of further stage are executed. There are also two edge cases worth mentioning: -1. If no `stages` is defined in `.gitlab-ci.yml`, then by default the `build`, `test` and `deploy` are allowed to be used as job's stage by default. +1. If no `stages` is defined in `.gitlab-ci.yml`, then by default the `build`, + `test` and `deploy` are allowed to be used as job's stage by default. 2. If a job doesn't specify `stage`, the job is assigned the `test` stage. ### types + Alias for [stages](#stages). ### variables -**This feature requires `gitlab-runner` with version equal or greater than 0.5.0.** -GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build environment. -The variables are stored in repository and are meant to store non-sensitive project configuration, ie. RAILS_ENV or DATABASE_URL. +_**Note:** Introduced in GitLab Runner v0.5.0._ + +GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build +environment. The variables are stored in the git repository and are meant to +store non-sensitive project configuration, for example: ```yaml variables: @@ -109,15 +129,23 @@ variables: These variables can be later used in all executed commands and scripts. -The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. +The YAML-defined variables are also set to all created service containers, +thus allowing to fine tune them. ### cache -`cache` is used to specify list of files and directories which should be cached between builds. -**The global setting allows to specify default cached files for all jobs.** +`cache` is used to specify a list of files and directories which should be +cached between builds. Caches are stored according to the branch/ref and the +job name. They are not currently shared between different job names or between +branches/refs, which means that caching will benefit you if you push subsequent +commits to an existing feature branch. + +If `cache` is defined outside the scope of the jobs, it means it is set +globally and all jobs will use its definition. To cache all git untracked files and files in `binaries`: -``` + +```yaml cache: untracked: true paths: @@ -125,9 +153,10 @@ cache: ``` ## Jobs -`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. -Each job has to have a unique `job_name`, which is not one of the keywords mentioned above. -A job is defined by a list of parameters that define the build behaviour. + +`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job +must have a unique name, which is not one of the Keywords mentioned above. +A job is defined by a list of parameters that define the build behavior. ```yaml job_name: @@ -145,21 +174,22 @@ job_name: allow_failure: true ``` -| keyword | required | description | +| Keyword | Required | Description | |---------------|----------|-------------| -| script | required | Defines a shell script which is executed by runner | -| stage | optional (default: test) | Defines a build stage | -| type | optional | Alias for `stage` | -| only | optional | Defines a list of git refs for which build is created | -| except | optional | Defines a list of git refs for which build is not created | -| tags | optional | Defines a list of tags which are used to select runner | -| allow_failure | optional | Allow build to fail. Failed build doesn't contribute to commit status | -| when | optional | Define when to run build. Can be `on_success`, `on_failure` or `always` | -| artifacts | optional | Define list build artifacts | -| cache | optional | Define list of files that should be cached between subsequent runs | +| script | yes | Defines a shell script which is executed by runner | +| stage | no (default: `test`) | Defines a build stage | +| type | no | Alias for `stage` | +| only | no | Defines a list of git refs for which build is created | +| except | no | Defines a list of git refs for which build is not created | +| tags | no | Defines a list of tags which are used to select runner | +| allow_failure | no | Allow build to fail. Failed build doesn't contribute to commit status | +| when | no | Define when to run build. Can be `on_success`, `on_failure` or `always` | +| artifacts | no | Define list build artifacts | +| cache | no | Define list of files that should be cached between subsequent runs | ### script -`script` is a shell script which is executed by runner. The shell script is prepended with `before_script`. + +`script` is a shell script which is executed by the runner. For example: ```yaml job: @@ -167,6 +197,7 @@ job: ``` This parameter can also contain several commands using an array: + ```yaml job: script: @@ -175,31 +206,45 @@ job: ``` ### stage -`stage` allows to group build into different stages. Builds of the same `stage` are executed in `parallel`. -For more info about the use of `stage` please check the [stages](#stages). + +`stage` allows to group build into different stages. Builds of the same `stage` +are executed in `parallel`. For more info about the use of `stage` please check +[stages](#stages). ### only and except -This are two parameters that allow for setting a refs policy to limit when jobs are built: -1. `only` defines the names of branches and tags for which job will be built. -2. `except` defines the names of branches and tags for which the job wil **not** be built. -There are a few rules that apply to usage of refs policy: +`only` and `except` are two parameters that set a refs policy to limit when +jobs are built: + +1. `only` defines the names of branches and tags for which the job will be + built. +2. `except` defines the names of branches and tags for which the job will + **not** be built. -1. `only` and `except` are inclusive. If both `only` and `except` are defined in job specification the ref is filtered by `only` and `except`. -1. `only` and `except` allow for using the regexp expressions. -1. `only` and `except` allow for using special keywords: `branches` and `tags`. -These names can be used for example to exclude all tags and all branches. +There are a few rules that apply to the usage of refs policy: + +* `only` and `except` are inclusive. If both `only` and `except` are defined + in a job specification, the ref is filtered by `only` and `except`. +* `only` and `except` allow the use of regular expressions. +* `only` and `except` allow the use of special keywords: `branches` and `tags`. +* `only` and `except` allow to specify a repository path to filter jobs for + forks. + +In the example below, `job` will run only for refs that start with `issue-`, +whereas all branches will be skipped. ```yaml job: + # use regexp only: - - /^issue-.*$/ # use regexp + - /^issue-.*$/ + # use special keyword except: - - branches # use special keyword + - branches ``` -1. `only` and `except` allow for specify repository path to filter jobs for forks. -The repository path can be used to have jobs executed only for parent repository. +The repository path can be used to have jobs executed only for the parent +repository and not forks: ```yaml job: @@ -208,33 +253,47 @@ job: except: - master@gitlab-org/gitlab-ce ``` -The above will run `job` for all branches on `gitlab-org/gitlab-ce`, except master . + +The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, +except master. ### tags -`tags` is used to select specific runners from the list of all runners that are allowed to run this project. -During registration of a runner, you can specify the runner's tags, ie.: `ruby`, `postgres`, `development`. -`tags` allow you to run builds with runners that have the specified tags assigned: +`tags` is used to select specific runners from the list of all runners that are +allowed to run this project. -``` +During the registration of a runner, you can specify the runner's tags, for +example `ruby`, `postgres`, `development`. + +`tags` allow you to run builds with runners that have the specified tags +assigned to them: + +```yaml job: tags: - ruby - postgres ``` -The above specification will make sure that `job` is built by a runner that have `ruby` AND `postgres` tags defined. +The specification above, will make sure that `job` is built by a runner that +has both `ruby` AND `postgres` tags defined. ### when -`when` is used to implement jobs that are run in case of failure or despite the failure. + +`when` is used to implement jobs that are run in case of failure or despite the +failure. `when` can be set to one of the following values: -1. `on_success` - execute build only when all builds from prior stages succeeded. This is the default. -1. `on_failure` - execute build only when at least one build from prior stages failed. +1. `on_success` - execute build only when all builds from prior stages + succeeded. This is the default. +1. `on_failure` - execute build only when at least one build from prior stages + failed. 1. `always` - execute build despite the status of builds from prior stages. -``` +For example: + +```yaml stages: - build - cleanup_build @@ -242,28 +301,28 @@ stages: - deploy - cleanup -build: +build_job: stage: build script: - make build -cleanup_build: +cleanup_build_job: stage: cleanup_build script: - cleanup build when failed when: on_failure -test: +test_job: stage: test script: - make test -deploy: +deploy_job: stage: deploy script: - make deploy -cleanup: +cleanup_job: stage: cleanup script: - cleanup after builds @@ -271,43 +330,56 @@ cleanup: ``` The above script will: -1. Execute `cleanup_build` only when the `build` failed, -2. Always execute `cleanup` as the last step in pipeline. + +1. Execute `cleanup_build_job` only when `build_job` fails +2. Always execute `cleanup_job` as the last step in pipeline. ### artifacts -`artifacts` is used to specify list of files and directories which should be attached to build after success. -1. Send all files in `binaries` and `.config`: -``` +_**Note:** Introduced in GitLab Runner v0.7.0. Also, the Windows shell executor + does not currently support artifact uploads._ + +`artifacts` is used to specify list of files and directories which should be +attached to build after success. Below are some examples. + +Send all files in `binaries` and `.config`: + +```yaml artifacts: paths: - binaries/ - .config ``` -2. Send all git untracked files: -``` +Send all git untracked files: + +```yaml artifacts: untracked: true ``` -3. Send all git untracked files and files in `binaries`: -``` +Send all git untracked files and files in `binaries`: + +```yaml artifacts: untracked: true paths: - binaries/ ``` -The artifacts will be send after the build success to GitLab and will be accessible in GitLab interface to download. - -This feature requires GitLab Runner v0.7.0 or higher. +The artifacts will be send after a successful build success to GitLab, and will +be accessible in the GitLab UI to download. ### cache -`cache` is used to specify list of files and directories which should be cached between builds. -1. Cache all files in `binaries` and `.config`: -``` +_**Note:** Introduced in GitLab Runner v0.7.0._ + +`cache` is used to specify list of files and directories which should be cached +between builds. Below are some examples: + +Cache all files in `binaries` and `.config`: + +```yaml rspec: script: test cache: @@ -316,16 +388,18 @@ rspec: - .config ``` -2. Cache all git untracked files: -``` +Cache all git untracked files: + +```yaml rspec: script: test cache: untracked: true ``` -3. Cache all git untracked files and files in `binaries`: -``` +Cache all git untracked files and files in `binaries`: + +```yaml rspec: script: test cache: @@ -334,9 +408,10 @@ rspec: - binaries/ ``` -4. Locally defined cache overwrites globally defined options. This will cache only `binaries/`: +Locally defined cache overwrites globally defined options. This will cache only +`binaries/`: -``` +```yaml cache: paths: - my/files @@ -348,15 +423,15 @@ rspec: - binaries/ ``` -The cache is provided on best effort basis, so don't expect that cache will be present. -For implementation details please check GitLab Runner. - -This feature requires GitLab Runner v0.7.0 or higher. - +The cache is provided on best effort basis, so don't expect that cache will be +always present. For implementation details please check GitLab Runner. ## Validate the .gitlab-ci.yml + Each instance of GitLab CI has an embedded debug tool called Lint. -You can find the link to the Lint in the project's settings page or use short url `/lint`. +You can find the link under `/ci/lint` of your gitlab instance. ## Skipping builds -There is one more way to skip all builds, if your commit message contains tag [ci skip]. In this case, commit will be created but builds will be skipped + +If your commit message contains `[ci skip]`, the commit will be created but the +builds will be skipped. diff --git a/doc/customization/issue_closing.md b/doc/customization/issue_closing.md index 64f128f5a63..00edfc97ed9 100644 --- a/doc/customization/issue_closing.md +++ b/doc/customization/issue_closing.md @@ -1,38 +1,39 @@ # Issue closing pattern -Here's how to close multiple issues in one commit message: +When a commit or merge request resolves one or more issues, it is possible to automatically have these issues closed when the commit or merge request lands in the project's default branch. -If a commit message matches the regular expression below, all issues referenced from -the matched text will be closed. This happens when the commit is pushed or merged -into the default branch of a project. +If a commit message or merge request description contains a sentence matching the regular expression below, all issues referenced from +the matched text will be closed. This happens when the commit is pushed to a project's default branch, or when a commit or merge request is merged into there. -When not specified, the default issue_closing_pattern as shown below will be used: +When not specified, the default `issue_closing_pattern` as shown below will be used: ```bash -((?:[Cc]los(?:e[sd]?|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?#\d+(?:(?:, *| +and +)?))+) +((?:[Cc]los(?:e[sd]?|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+) ``` +Here, `%{issue_ref}` is a complex regular expression defined inside GitLab, that matches a reference to a local issue (`#123`), cross-project issue (`group/project#123`) or a link to an issue (`https://gitlab.example.com/group/project/issues/123`). + For example: ``` -git commit -m "Awesome commit message (Fix #20, Fixes #21 and Closes #22). This commit is also related to #17 and fixes #18, #19 and #23." +git commit -m "Awesome commit message (Fix #20, Fixes #21 and Closes group/otherproject#2). This commit is also related to #17 and fixes #18, #19 and https://gitlab.example.com/group/otherproject/issues/23." ``` -will close `#20`, `#21`, `#22`, `#18`, `#19` and `#23`, but `#17` won't be closed -as it does not match the pattern. It also works with multiline commit messages. +will close `#18`, `#19`, `#20`, and `#21` in the project this commit is pushed to, as well as `#22` and `#23` in group/otherproject. `#17` won't be closed as it does not match the pattern. It also works with multiline commit messages. Tip: you can test this closing pattern at [http://rubular.com][1]. Use this site to test your own patterns. +Because Rubular doesn't understand `%{issue_ref}`, you can replace this by `#\d+` in testing, which matches only local issue references like `#123`. ## Change the pattern For Omnibus installs you can change the default pattern in `/etc/gitlab/gitlab.rb`: ``` -issue_closing_pattern: '((?:[Cc]los(?:e[sd]|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?#\d+(?:(?:, *| +and +)?))+)' +issue_closing_pattern: '((?:[Cc]los(?:e[sd]|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)' ``` -For manual installs you can customize the pattern in [gitlab.yml][0]. +For manual installs you can customize the pattern in [gitlab.yml][0] using the `issue_closing_pattern` key. -[0]: https://gitlab.com/gitlab-org/gitlab-ce/blob/40c3675372320febf5264061c9bcd63db2dfd13c/config/gitlab.yml.example#L65 -[1]: http://rubular.com/r/Xmbexed1OJ
\ No newline at end of file +[0]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example +[1]: http://rubular.com/r/Xmbexed1OJ diff --git a/doc/development/db_dump.md b/doc/development/db_dump.md index 21f1b3edecd..e4ff72aa349 100644 --- a/doc/development/db_dump.md +++ b/doc/development/db_dump.md @@ -1,4 +1,4 @@ -# Importing a database dump into a staging enviroment +# Importing a database dump into a staging environment Sometimes it is useful to import the database from a production environment into a staging environment for testing. The procedure below assumes you have diff --git a/doc/install/installation.md b/doc/install/installation.md index 52ae30af805..81edd8da2b8 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -38,6 +38,7 @@ The GitLab installation consists of setting up the following components: 1. Packages / Dependencies 1. Ruby +1. Go 1. System Users 1. Database 1. Redis @@ -62,7 +63,7 @@ up-to-date and install it. Install the required packages (needed to compile Ruby and native extensions to Ruby gems): - sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server redis-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake nodejs + sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake nodejs If you want to use Kerberos for user authentication, then install libkrb5-dev: @@ -174,33 +175,53 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da ## 6. Redis - sudo apt-get install redis-server +As of this writing, most Debian/Ubuntu distributions ship with Redis 2.2 or +2.4. GitLab requires at least Redis 2.8. - # Configure redis to use sockets - sudo cp /etc/redis/redis.conf /etc/redis/redis.conf.orig +Ubuntu users [can use a PPA](https://launchpad.net/~chris-lea/+archive/ubuntu/redis-server) +to install a recent version of Redis. - # Disable Redis listening on TCP by setting 'port' to 0 - sed 's/^port .*/port 0/' /etc/redis/redis.conf.orig | sudo tee /etc/redis/redis.conf +The following instructions cover building and installing Redis from scratch: - # Enable Redis socket for default Debian / Ubuntu path - echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf - # Grant permission to the socket to all members of the redis group - echo 'unixsocketperm 770' | sudo tee -a /etc/redis/redis.conf +```sh +# Build Redis +wget http://download.redis.io/releases/redis-2.8.23.tar.gz +tar xzf redis-2.8.23.tar.gz +cd redis-2.8.23 +make - # Create the directory which contains the socket - mkdir /var/run/redis - chown redis:redis /var/run/redis - chmod 755 /var/run/redis - # Persist the directory which contains the socket, if applicable - if [ -d /etc/tmpfiles.d ]; then - echo 'd /var/run/redis 0755 redis redis 10d -' | sudo tee -a /etc/tmpfiles.d/redis.conf - fi +# Install Redis +cd utils +sudo ./install_server.sh - # Activate the changes to redis.conf - sudo service redis-server restart +# Configure redis to use sockets +sudo cp /etc/redis/redis.conf /etc/redis/redis.conf.orig - # Add git to the redis group - sudo usermod -aG redis git +# Disable Redis listening on TCP by setting 'port' to 0 +sed 's/^port .*/port 0/' /etc/redis/redis.conf.orig | sudo tee /etc/redis/redis.conf + +# Enable Redis socket for default Debian / Ubuntu path +echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf + +# Grant permission to the socket to all members of the redis group +echo 'unixsocketperm 770' | sudo tee -a /etc/redis/redis.conf + +# Create the directory which contains the socket +mkdir /var/run/redis +chown redis:redis /var/run/redis +chmod 755 /var/run/redis + +# Persist the directory which contains the socket, if applicable +if [ -d /etc/tmpfiles.d ]; then + echo 'd /var/run/redis 0755 redis redis 10d -' | sudo tee -a /etc/tmpfiles.d/redis.conf +fi + +# Activate the changes to redis.conf +sudo service redis_6379 start + +# Add git to the redis group +sudo usermod -aG redis git +``` ## 7. GitLab @@ -210,9 +231,9 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 8-2-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 8-3-stable gitlab -**Note:** You can change `8-2-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `8-3-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It @@ -312,7 +333,7 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da GitLab Shell is an SSH access and repository management software developed specially for GitLab. # Run the installation task for gitlab-shell (replace `REDIS_URL` if needed): - sudo -u git -H bundle exec rake gitlab:shell:install[v2.6.7] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:shell:install REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production # By default, the gitlab-shell config is generated from your main GitLab config. # You can review (and modify) the gitlab-shell config as follows: @@ -320,19 +341,19 @@ GitLab Shell is an SSH access and repository management software developed speci **Note:** If you want to use HTTPS, see [Using HTTPS](#using-https) for the additional steps. -**Note:** Make sure your hostname can be resolved on the machine itself by either a proper DNS record or an additional line in /etc/hosts ("127.0.0.1 hostname"). This might be necessary for example if you set up gitlab behind a reverse proxy. If the hostname cannot be resolved, the final installation check will fail with "Check GitLab API access: FAILED. code: 401" and pushing commits will be rejected with "[remote rejected] master -> master (hook declined)". +**Note:** Make sure your hostname can be resolved on the machine itself by either a proper DNS record or an additional line in /etc/hosts ("127.0.0.1 hostname"). This might be necessary for example if you set up GitLab behind a reverse proxy. If the hostname cannot be resolved, the final installation check will fail with "Check GitLab API access: FAILED. code: 401" and pushing commits will be rejected with "[remote rejected] master -> master (hook declined)". ### Install gitlab-workhorse cd /home/git sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-workhorse.git cd gitlab-workhorse - sudo -u git -H git checkout 0.4.1 + sudo -u git -H git checkout 0.5.1 sudo -u git -H make ### Initialize Database and Activate Advanced Features - # Go to Gitlab installation folder + # Go to GitLab installation folder cd /home/git/gitlab diff --git a/doc/integration/bitbucket.md b/doc/integration/bitbucket.md index 6a0fa4ce015..63432b04432 100644 --- a/doc/integration/bitbucket.md +++ b/doc/integration/bitbucket.md @@ -30,7 +30,7 @@ Bitbucket will generate an application ID and secret key for you to use. sudo editor /etc/gitlab/gitlab.rb ``` - For instalations from source: + For installations from source: ```sh cd /home/git/gitlab diff --git a/doc/integration/crowd.md b/doc/integration/crowd.md index 2ecc8795ac1..40d93aef2a9 100644 --- a/doc/integration/crowd.md +++ b/doc/integration/crowd.md @@ -10,7 +10,7 @@ To enable the Crowd OmniAuth provider you must register your application with Cr sudo editor /etc/gitlab/gitlab.rb ``` - For instalations from source: + For installations from source: ```sh cd /home/git/gitlab diff --git a/doc/integration/github.md b/doc/integration/github.md index b64501c2aaa..a789d2c814f 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -32,7 +32,7 @@ GitHub will generate an application ID and secret key for you to use. sudo editor /etc/gitlab/gitlab.rb ``` - For instalations from source: + For installations from source: ```sh cd /home/git/gitlab diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index 216f1f11a9b..80e3c0142a0 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -38,7 +38,7 @@ GitLab.com will generate an application ID and secret key for you to use. sudo editor /etc/gitlab/gitlab.rb ``` - For instalations from source: + For installations from source: ```sh cd /home/git/gitlab diff --git a/doc/integration/google.md b/doc/integration/google.md index e1c14c7c948..91e9b2495cc 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -35,7 +35,7 @@ To enable the Google OAuth2 OmniAuth provider you must register your application sudo editor /etc/gitlab/gitlab.rb ``` - For instalations from source: + For installations from source: ```sh cd /home/git/gitlab diff --git a/doc/integration/ldap.md b/doc/integration/ldap.md index 7e2920b8865..845f588f913 100644 --- a/doc/integration/ldap.md +++ b/doc/integration/ldap.md @@ -13,6 +13,12 @@ An LDAP user who is allowed to change their email on the LDAP server can [take o We recommend against using GitLab LDAP integration if your LDAP users are allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on the LDAP server. +If a user is deleted from the LDAP server, they will be blocked in GitLab as well. +Users will be immediately blocked from logging in. However, there is an LDAP check +cache time of one hour. The means users that are already logged in or are using Git +over SSH will still be able to access GitLab for up to one hour. Manually block +the user in the GitLab Admin area to immediately block all access. + ## Configuring GitLab for LDAP integration To enable GitLab LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml`. @@ -192,4 +198,4 @@ Not supported by GitLab's configuration options. When setting `method: ssl`, the underlying authentication method used by `omniauth-ldap` is `simple_tls`. This method establishes TLS encryption with the LDAP server before any LDAP-protocol data is exchanged but no validation of -the LDAP server's SSL certificate is performed.
\ No newline at end of file +the LDAP server's SSL certificate is performed. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 4aa6dbe758a..1632e42f701 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -14,7 +14,7 @@ First configure SAML 2.0 support in GitLab, then register the GitLab application sudo editor /etc/gitlab/gitlab.rb ``` - For instalations from source: + For installations from source: ```sh cd /home/git/gitlab @@ -38,7 +38,8 @@ First configure SAML 2.0 support in GitLab, then register the GitLab application idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - } + }, + "label" => "Company Login" # optional label for SAML login button, defaults to "Saml" } ] ``` @@ -79,4 +80,4 @@ On the sign in page there should now be a SAML button below the regular sign in If you see a "500 error" in GitLab when you are redirected back from the SAML sign in page, this likely indicates that GitLab could not get the email address for the SAML user. -Make sure the IdP provides a claim containing the user's email address, using claim name 'email' or 'mail'. The email will be used to automatically generate the GitLab username. +Make sure the IdP provides a claim containing the user's email address, using claim name 'email' or 'mail'. The email will be used to automatically generate the GitLab username.
\ No newline at end of file diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index 1350c8f693c..52ed4a22339 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -37,7 +37,7 @@ To enable the Twitter OmniAuth provider you must register your application with sudo editor /etc/gitlab/gitlab.rb ``` - For instalations from source: + For installations from source: ```sh cd /home/git/gitlab diff --git a/doc/operations/moving_repositories.md b/doc/operations/moving_repositories.md new file mode 100644 index 00000000000..39086b7a251 --- /dev/null +++ b/doc/operations/moving_repositories.md @@ -0,0 +1,180 @@ +# Moving repositories managed by GitLab + +Sometimes you need to move all repositories managed by GitLab to +another filesystem or another server. In this document we will look +at some of the ways you can copy all your repositories from +`/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`. + +We will look at three scenarios: the target directory is empty, the +target directory contains an outdated copy of the repositories, and +how to deal with thousands of repositories. + +**Each of the approaches we list can/will overwrite data in the +target directory `/mnt/gitlab/repositories`. Do not mix up the +source and the target.** + +## Target directory is empty: use a tar pipe + +If the target directory `/mnt/gitlab/repositories` is empty the +simplest thing to do is to use a tar pipe. This method has low +overhead and tar is almost always already installed on your system. +However, it is not possible to resume an interrupted tar pipe: if +that happens then all data must be copied again. + +``` +# As the git user +tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ + tar -C /mnt/gitlab/repositories -xf - +``` + +If you want to see progress, replace `-xf` with `-xvf`. + +### Tar pipe to another server + +You can also use a tar pipe to copy data to another server. If your +'git' user has SSH access to the newserver as 'git@newserver', you +can pipe the data through SSH. + +``` +# As the git user +tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ + ssh git@newserver tar -C /mnt/gitlab/repositories -xf - +``` + +If you want to compress the data before it goes over the network +(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`. + +## The target directory contains an outdated copy of the repositories: use rsync + +If the target directory already contains a partial / outdated copy +of the repositories it may be wasteful to copy all the data again +with tar. In this scenario it is better to use rsync. This utility +is either already installed on your system or easily installable +via apt, yum etc. + +``` +# As the 'git' user +rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ + /mnt/gitlab/repositories +``` + +The `/.` in the command above is very important, without it you can +easily get the wrong directory structure in the target directory. +If you want to see progress, replace `-a` with `-av`. + +### Single rsync to another server + +If the 'git' user on your source system has SSH access to the target +server you can send the repositories over the network with rsync. + +``` +# As the 'git' user +rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ + git@newserver:/mnt/gitlab/repositories +``` + +## Thousands of Git repositories: use one rsync per repository + +Every time you start an rsync job it has to inspect all files in +the source directory, all files in the target directory, and then +decide what files to copy or not. If the source or target directory +has many contents this startup phase of rsync can become a burden +for your GitLab server. In cases like this you can make rsync's +life easier by dividing its work in smaller pieces, and sync one +repository at a time. + +In addition to rsync we will use [GNU +Parallel](http://www.gnu.org/software/parallel/). This utility is +not included in GitLab so you need to install it yourself with apt +or yum. Also note that the GitLab scripts we used below were added +in GitLab 8.1. + +** This process does not clean up repositories at the target location that no +longer exist at the source. ** If you start using your GitLab instance with +`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos` +after switching to the new repository storage directory. + +### Parallel rsync for all repositories known to GitLab + +This will sync repositories with 10 rsync processes at a time. We keep +track of progress so that the transfer can be restarted if necessary. + +First we create a new directory, owned by 'git', to hold transfer +logs. We assume the directory is empty before we start the transfer +procedure, and that we are the only ones writing files in it. + +``` +# Omnibus +sudo mkdir /var/opt/gitlab/transfer-logs +sudo chown git:git /var/opt/gitlab/transfer-logs + +# Source +sudo -u git -H mkdir /home/git/transfer-logs +``` + +We seed the process with a list of the directories we want to copy. + +``` +# Omnibus +sudo -u git sh -c 'gitlab-rake gitlab:list_repos > /var/opt/gitlab/transfer-logs/all-repos-$(date +%s).txt' + +# Source +cd /home/git/gitlab +sudo -u git -H sh -c 'bundle exec rake gitlab:list_repos > /home/git/transfer-logs/all-repos-$(date +%s).txt' +``` + +Now we can start the transfer. The command below is idempotent, and +the number of jobs done by GNU Parallel should converge to zero. If it +does not some repositories listed in all-repos-1234.txt may have been +deleted/renamed before they could be copied. + +``` +# Omnibus +sudo -u git sh -c ' +cat /var/opt/gitlab/transfer-logs/* | sort | uniq -u |\ + /usr/bin/env JOBS=10 \ + /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ + /var/opt/gitlab/transfer-logs/succes-$(date +%s).log \ + /var/opt/gitlab/git-data/repositories \ + /mnt/gitlab/repositories +' + +# Source +cd /home/git/gitlab +sudo -u git -H sh -c ' +cat /home/git/transfer-logs/* | sort | uniq -u |\ + /usr/bin/env JOBS=10 \ + bin/parallel-rsync-repos \ + /home/git/transfer-logs/succes-$(date +%s).log \ + /home/git/repositories \ + /mnt/gitlab/repositories +` +``` + +### Parallel rsync only for repositories with recent activity + +Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. +Then you might only want to sync repositories that were changed via GitLab +_after_ that time. You can use the 'SINCE' variable to tell 'rake +gitlab:list_repos' to only print repositories with recent activity. + +``` +# Omnibus +sudo gitlab-rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ + sudo -u git \ + /usr/bin/env JOBS=10 \ + /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ + succes-$(date +%s).log \ + /var/opt/gitlab/git-data/repositories \ + /mnt/gitlab/repositories + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ + sudo -u git -H \ + /usr/bin/env JOBS=10 \ + bin/parallel-rsync-repos \ + succes-$(date +%s).log \ + /home/git/repositories \ + /mnt/gitlab/repositories +``` diff --git a/doc/permissions/permissions.md b/doc/permissions/permissions.md index 8d4c2ceab7d..bcd00cfc6bf 100644 --- a/doc/permissions/permissions.md +++ b/doc/permissions/permissions.md @@ -6,6 +6,9 @@ If a user is both in a project group and in the project itself, the highest perm If a user is a GitLab administrator they receive all permissions. +On public projects the Guest role is not enforced. +All users will be able to create issues, leave comments, and pull or download the project code. + To add or import a user, you can follow the [project users and members documentation](doc/workflow/add-user/add-user.md). @@ -15,8 +18,8 @@ documentation](doc/workflow/add-user/add-user.md). |---------------------------------------|---------|------------|-------------|----------|--------| | Create new issue | ✓ | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ | ✓ | ✓ | ✓ | ✓ | -| Pull project code | ✓ | ✓ | ✓ | ✓ | ✓ | -| Download project | ✓ | ✓ | ✓ | ✓ | ✓ | +| Pull project code | | ✓ | ✓ | ✓ | ✓ | +| Download project | | ✓ | ✓ | ✓ | ✓ | | Create code snippets | | ✓ | ✓ | ✓ | ✓ | | Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | | Manage labels | | ✓ | ✓ | ✓ | ✓ | diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index bd439f7c6f3..6e22ea7b72a 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -1,44 +1,59 @@ # Public access -GitLab allows you to open selected projects to be accessed **publicly** or **internally**. +GitLab allows you to change your projects' visibility in order be accessed +**publicly** or **internally**. -Projects with either of these visibility levels will be listed in the [public access directory](/public). +Projects with either of these visibility levels will be listed in the +public access directory (`/public` under your GitLab instance). +Here is the [GitLab.com example](https://gitlab.com/public). Internal projects will only be available to authenticated users. -## Public projects +## Visibility of projects + +### Public projects Public projects can be cloned **without any** authentication. -It will also be listed on the [public access directory](/public). +They will also be listed on the public access directory (`/public`). -**Any logged in user** will have [Guest](../permissions/permissions) permissions on the repository. +**Any logged in user** will have [Guest](../permissions/permissions) +permissions on the repository. -## Internal projects +### Internal projects Internal projects can be cloned by any logged in user. -It will also be listed on the [public access directory](/public) for logged in users. +They will also be listed on the public access directory (`/public`) for logged +in users. -Any logged in user will have [Guest](../permissions/permissions) permissions on the repository. +Any logged in user will have [Guest](../permissions/permissions) permissions on +the repository. -## How to change project visibility +### How to change project visibility -1. Go to your project dashboard -1. Click on the "Edit" tab -1. Change "Visibility Level" +1. Go to your project's **Settings** +1. Change "Visibility Level" to either Public, Internal or Private ## Visibility of users -The public page of users, located at `/u/username` is visible if either: +The public page of a user, located at `/u/username`, is always visible whether +you are logged in or not. + +When visiting the public page of a user, you can only see the projects which +you are privileged to. -- You are logged in. -- You are logged out, and the target user is authorized to (is Guest, Reporter, etc.) at least one public project. +## Visibility of groups -Otherwise, you will be redirected to the sign in page. +The public page of a group, located at `/groups/groupname`, is always visible +to everyone. -When visiting the public page of an user, you will only see listed projects which you can view yourself. +Logged out users will be able to see the description and the avatar of the +group as well as all public projects belonging to that group. ## Restricting the use of public or internal projects -In the Admin area under Settings you can disable public projects or public and internal projects for the entire GitLab installation to prevent people making code public by accident. The restricted visibility settings do not apply to admin users. +In the Admin area under **Settings** (`/admin/application_settings`), you can +restrict the use of visibility levels for users when they create a project or a +snippet. This is useful to prevent people exposing their repositories to public +by accident. The restricted visibility settings do not apply to admin users. diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index a8dc5c24df2..cc8a22cd003 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -1,10 +1,11 @@ # Rake tasks - [Backup restore](backup_restore.md) +- [Check](check.md) - [Cleanup](cleanup.md) - [Features](features.md) - [Maintenance](maintenance.md) and self-checks - [User management](user_management.md) - [Web hooks](web_hooks.md) - [Import](import.md) of git repositories in bulk -- [Rebuild authorized_keys file](http://doc.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators
\ No newline at end of file +- [Rebuild authorized_keys file](http://doc.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 1a5442cdac7..093450a6de3 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -29,7 +29,7 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` Also you can choose what should be backed up by adding environment variable SKIP. Available options: db, -uploads (attachments), repositories, builds(CI build output logs), artifacts (CI build artifacts). +uploads (attachments), repositories, builds(CI build output logs), artifacts (CI build artifacts), lfs (LFS objects). Use a comma to specify several options at the same time. ``` @@ -274,9 +274,6 @@ sudo gitlab-rake gitlab:backup:restore BACKUP=1393513186 # Start GitLab sudo gitlab-ctl start -# Create satellites -sudo gitlab-rake gitlab:satellites:create - # Check GitLab sudo gitlab-rake gitlab:check SANITIZE=true ``` @@ -360,8 +357,8 @@ If you are using backup restore procedures you might encounter the following war ``` psql:/var/opt/gitlab/backups/db/database.sql:22: ERROR: must be owner of extension plpgsql -psql:/var/opt/gitlab/backups/db/database.sql:2931: WARNING: no privileges could be revoked for "public" (two occurences) -psql:/var/opt/gitlab/backups/db/database.sql:2933: WARNING: no privileges were granted for "public" (two occurences) +psql:/var/opt/gitlab/backups/db/database.sql:2931: WARNING: no privileges could be revoked for "public" (two occurrences) +psql:/var/opt/gitlab/backups/db/database.sql:2933: WARNING: no privileges were granted for "public" (two occurrences) ``` diff --git a/doc/raketasks/check.md b/doc/raketasks/check.md new file mode 100644 index 00000000000..3ff3fee6a40 --- /dev/null +++ b/doc/raketasks/check.md @@ -0,0 +1,63 @@ +# Check Rake Tasks + +## Repository Integrity + +Even though Git is very resilient and tries to prevent data integrity issues, +there are times when things go wrong. The following Rake tasks intend to +help GitLab administrators diagnose problem repositories so they can be fixed. + +There are 3 things that are checked to determine integrity. + +1. Git repository file system check ([git fsck](https://git-scm.com/docs/git-fsck)). + This step verifies the connectivity and validity of objects in the repository. +1. Check for `config.lock` in the repository directory. +1. Check for any branch/references lock files in `refs/heads`. + +It's important to note that the existence of `config.lock` or reference locks +alone do not necessarily indicate a problem. Lock files are routinely created +and removed as Git and GitLab perform operations on the repository. They serve +to prevent data integrity issues. However, if a Git operation is interrupted these +locks may not be cleaned up properly. + +The following symptoms may indicate a problem with repository integrity. If users +experience these symptoms you may use the rake tasks described below to determine +exactly which repositories are causing the trouble. + +- Receiving an error when trying to push code - `remote: error: cannot lock ref` +- A 500 error when viewing the GitLab dashboard or when accessing a specific project. + +### Check all GitLab repositories + +This task loops through all repositories on the GitLab server and runs the +3 integrity checks described previously. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:repo:check + +# installation from source +bundle exec rake gitlab:repo:check RAILS_ENV=production +``` + +### Check repositories for a specific user + +This task checks all repositories that a specific user has access to. This is important +because sometimes you know which user is experiencing trouble but you don't know +which project might be the cause. + +If the rake task is executed without brackets at the end, you will be prompted +to enter a username. + +```bash +# omnibus-gitlab +sudo gitlab-rake gitlab:user:check_repos +sudo gitlab-rake gitlab:user:check_repos[<username>] + +# installation from source +bundle exec rake gitlab:user:check_repos RAILS_ENV=production +bundle exec rake gitlab:user:check_repos[<username>] RAILS_ENV=production +``` + +Example output: + + diff --git a/doc/raketasks/check_repos_output.png b/doc/raketasks/check_repos_output.png Binary files differnew file mode 100644 index 00000000000..916b1685101 --- /dev/null +++ b/doc/raketasks/check_repos_output.png diff --git a/doc/raketasks/list_repos.md b/doc/raketasks/list_repos.md new file mode 100644 index 00000000000..476428eb4f5 --- /dev/null +++ b/doc/raketasks/list_repos.md @@ -0,0 +1,30 @@ +# Listing repository directories + +You can print a list of all Git repositories on disk managed by +GitLab with the following command: + +``` +# Omnibus +sudo gitlab-rake gitlab:list_repos + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production +``` + +If you only want to list projects with recent activity you can pass +a date with the 'SINCE' environment variable. The time you specify +is parsed by the Rails [TimeZone#parse +function](http://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse). + +``` +# Omnibus +sudo gitlab-rake gitlab:list_repos SINCE='Sep 1 2015' + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production SINCE='Sep 1 2015' +``` + +Note that the projects listed are NOT sorted by activity; they use +the default ordering of the GitLab Rails application. diff --git a/doc/release/README.md b/doc/release/README.md index 1342b90f3b3..52eca7c02a6 100644 --- a/doc/release/README.md +++ b/doc/release/README.md @@ -1,4 +1,8 @@ -GitLab has the following updates: +## Release cycle + +Since 2011 a minor or major version of GitLab is released on the 22nd of every month. Patch and security releases are published when needed. New features are detailed on the [blog](https://about.gitlab.com/blog/) and in the [changelog](CHANGELOG). Features that will likely be in the next releases can be found on the [direction page](https://about.gitlab.com/direction/). + +## Release process documentation - [Monthly release](monthly.md), every month on the 22nd. - [Patch release](patch.md), if there are serious regressions. diff --git a/doc/release/monthly.md b/doc/release/monthly.md index c9ab87671d2..907c19e65a0 100644 --- a/doc/release/monthly.md +++ b/doc/release/monthly.md @@ -159,7 +159,7 @@ Please do not raise issues directly in this issue but link to issues that might The decision to create a patch release or not is with the release manager who is assigned to this issue. The release manager will comment here about the plans for patch releases. -Assign the issue to the release manager and at mention all members of gitlab core team. If there are any known bugs in the release add them immediately. +Assign the issue to the release manager and at mention all members of GitLab core team. If there are any known bugs in the release add them immediately. ## Tweet about RC1 @@ -176,7 +176,7 @@ Tweet about the RC release: 1. Also check the CI changelog 1. Add a proposed tweet text to the blog post WIP MR description. 1. Create a WIP MR for the blog post -1. Make sure merge request title starts with `WIP` so it can not be accidently merged until ready. +1. Make sure merge request title starts with `WIP` so it can not be accidentally merged until ready. 1. Ask Dmitriy (or a team member with OS X) to add screenshots to the WIP MR. 1. Decide with core team who will be the MVP user. 1. Create WIP MR for adding MVP to MVP page on website diff --git a/doc/release/patch.md b/doc/release/patch.md index 6aa11b283df..3022e375aca 100644 --- a/doc/release/patch.md +++ b/doc/release/patch.md @@ -1,21 +1,46 @@ # Things to do when doing a patch release -NOTE: This is a guide for GitLab developers. If you are trying to install GitLab see the latest stable [installation guide](install/installation.md) and if you are trying to upgrade, see the [upgrade guides](update). +NOTE: This is a guide for GitLab developers. If you are trying to install GitLab +see the latest stable [installation guide](install/installation.md) and if you +are trying to upgrade, see the [upgrade guides](update). ## When to do a patch release -Do a patch release when there is a critical regression that needs to be addresses before the next monthly release. - -Otherwise include it in the monthly release and note there was a regression fix in the release announcement. +Patch releases are done as-needed in order to fix regressions in the current +major release that cannot or should not wait until the next major release. +What's included and when to release is at the discretion of the release manager. ## Release Procedure +### Create a patch issue + +Create an issue in the GitLab CE project. Name it "Release x.y.z", tag it with +the `release` label, and assign it to the milestone of the corresponding major +release. + +Use the following template: + +``` +- Picked into respective `stable` branches: +- [ ] Merge `x-y-stable` into `x-y-stable-ee` +- [ ] release-tools: `x.y.z` +- gitlab-omnibus + - [ ] `x.y.z+ee.0` + - [ ] `x.y.z+ce.0` +- [ ] Deploy +- [ ] Add patch notice to [x.y regressions]() +- [ ] [Blog post]() +- [ ] [Tweet]() +- [ ] Add entry to version.gitlab.com +``` + +Update the issue with links to merge requests that need to be/have been picked +into the `stable` branches. + ### Preparation 1. Verify that the issue can be reproduced 1. Note in the 'GitLab X.X regressions' that you will create a patch -1. Create an issue on private GitLab development server -1. Name the issue "Release X.X.X CE and X.X.X EE", this will make searching easier 1. Fix the issue on a feature branch, do this on the private GitLab development server 1. If it is a security issue, then assign it to the release manager and apply a 'security' label 1. Consider creating and testing workarounds @@ -25,7 +50,6 @@ Otherwise include it in the monthly release and note there was a regression fix 1. For EE, update the CHANGELOG-EE if it is EE specific fix. Otherwise, merge the stable CE branch and add to CHANGELOG-EE "Merge community edition changes for version X.X.X" 1. Merge CE stable branch into EE stable branch - ### Bump version Get release tools @@ -54,4 +78,4 @@ bundle exec rake release["x.x.x"] 1. Note in the 'GitLab X.X regressions' issue that the patch was published (CE only) 1. Create the 'x.y.0' version on version.gitlab.com 1. [Create new AMIs](https://dev.gitlab.org/gitlab/AMI/blob/master/README.md) -1. Create a new patch release issue for the next potential release
\ No newline at end of file +1. Create a new patch release issue for the next potential release diff --git a/doc/security/README.md b/doc/security/README.md index 473f3632dcd..fba6013d9c1 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -4,4 +4,5 @@ - [Rack attack](rack_attack.md) - [Web Hooks and insecure internal web services](webhooks.md) - [Information exclusivity](information_exclusivity.md) -- [Reset your root password](reset_root_password.md)
\ No newline at end of file +- [Reset your root password](reset_root_password.md) +- [User File Uploads](user_file_uploads.md) diff --git a/doc/security/user_file_uploads.md b/doc/security/user_file_uploads.md new file mode 100644 index 00000000000..98493d33b00 --- /dev/null +++ b/doc/security/user_file_uploads.md @@ -0,0 +1,11 @@ +# User File Uploads + +Images attached to issues, merge requests or comments do not require authentication +to be viewed if someone knows the direct URL. This direct URL contains a random +32-character ID that prevents unauthorized people from guessing the URL to an +image containing sensitive information. We don't enable authentication because +these images need to be visible in the body of notification emails, which are +often read from email clients that are not authenticated with GitLab, like +Outlook, Apple Mail, or the Mail app on your mobile device. + +Note that non-image attachments do require authentication to be viewed. diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 9753504ac8b..fe5b45dd432 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -15,7 +15,8 @@ Note: It is a best practice to use a password for an SSH key, but it is not required and you can skip creating a password by pressing enter. Note that the password you choose here can't be altered or retrieved. -To generate a new SSH key, use the following commandGitLab```bash +To generate a new SSH key, use the following command: +```bash ssh-keygen -t rsa -C "$your_email" ``` This command will prompt you for a location and filename to store the key @@ -108,4 +109,4 @@ Note in the gitlab.com example above a username was specified to override the de Due to the wide variety of SSH clients and their very large number of configuration options, further explanation of these topics is beyond the scope of this document. Public SSH keys need to be unique, as they will bind to your account. Your SSH key is the only identifier you'll -have when pushing code via SSH. That's why it needs to uniquely map to a single user. +have when pushing code via SSH. That's why it needs to uniquely map to a single user.
\ No newline at end of file diff --git a/doc/update/8.1-to-8.2.md b/doc/update/8.1-to-8.2.md index 261031442ac..46dfa2232b4 100644 --- a/doc/update/8.1-to-8.2.md +++ b/doc/update/8.1-to-8.2.md @@ -68,7 +68,7 @@ sudo -u git -H git checkout 8-2-stable-ee ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.6.7 +sudo -u git -H git checkout v2.6.8 ``` ### 5. Replace gitlab-git-http-server with gitlab-workhorse @@ -81,15 +81,14 @@ from GitLab 8.1. cd /home/git sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-workhorse.git cd gitlab-workhorse -sudo -u git -H git checkout 0.4.1 +sudo -u git -H git checkout 0.4.2 sudo -u git -H make ``` -Update the GitLab init script and 'default' file. +Update the GitLab 'default' file. ``` cd /home/git/gitlab -sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab test -e /etc/default/gitlab && \ sudo sed -i.pre-8.2 's/^\([^=]*\)gitlab_git_http_server/\1gitlab_workhorse/' /etc/default/gitlab ``` @@ -143,7 +142,7 @@ git diff origin/8-1-stable:lib/support/nginx/gitlab origin/8-2-stable:lib/suppor If you are using Apache instead of NGINX please see the updated [Apache templates]. Also note that because Apache does not support upstreams behind Unix sockets you -will need to let gitlab-git-http-server listen on a TCP port. You can do this +will need to let gitlab-workhorse listen on a TCP port. You can do this via [/etc/default/gitlab]. [Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache diff --git a/doc/update/8.2-to-8.3.md b/doc/update/8.2-to-8.3.md new file mode 100644 index 00000000000..c4661dc16af --- /dev/null +++ b/doc/update/8.2-to-8.3.md @@ -0,0 +1,202 @@ +# From 8.2 to 8.3 + +**NOTE:** GitLab 8.0 introduced several significant changes related to +installation and configuration which *are not duplicated here*. Be sure you're +already running a working version of at least 8.0 before proceeding with this +guide. + +### 0. Double-check your Git version + +**This notice applies only to /usr/local/bin/git** + +If you compiled Git from source on your GitLab server then please double-check +that you are using a version that protects against CVE-2014-9390. For six +months after this vulnerability became known the GitLab installation guide +still contained instructions that would install the outdated, 'vulnerable' Git +version 2.1.2. + +Run the following command to get your current Git version: + +```sh +/usr/local/bin/git --version +``` + +If you see 'No such file or directory' then you did not install Git according +to the outdated instructions from the GitLab installation guide and you can go +to the next step 'Stop server' below. + +If you see a version string then it should be v1.8.5.6, v1.9.5, v2.0.5, v2.1.4, +v2.2.1 or newer. You can use the [instructions in the GitLab source installation +guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md#1-packages-dependencies) +to install a newer version of Git. + +### 1. Stop server + + sudo service gitlab stop + +### 2. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 8-3-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 8-3-stable-ee +``` + +### 4. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch --all +sudo -u git -H git checkout v2.6.9 +``` + +### 5. Update gitlab-workhorse + +Install and compile gitlab-workhorse. This requires [Go 1.5](https://golang.org/dl) +which should already be on your system from GitLab 8.1. + +```bash +cd /home/git/gitlab-workhorse +sudo -u git -H git fetch --all +sudo -u git -H git checkout 0.5.1 +sudo -u git -H make +``` + +### 6. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 7. Update configuration files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them manually to your current `gitlab.yml`: + +```sh +git diff origin/8-2-stable:config/gitlab.yml.example origin/8-3-stable:config/gitlab.yml.example +``` + +#### Nginx configuration + +GitLab 8.3 introduces major changes in the NGINX configuration. +Because all HTTP requests pass through gitlab-workhorse now a lot of +directives need to be removed from NGINX. During future upgrades there +should be much less changes in the NGINX configuration because of +this. + +View changes between the previous recommended Nginx configuration and the +current one: + +```sh +# For HTTPS configurations +git diff origin/8-2-stable:lib/support/nginx/gitlab-ssl origin/8-3-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/8-2-stable:lib/support/nginx/gitlab origin/8-3-stable:lib/support/nginx/gitlab +``` + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-3-stable/lib/support/init.d/gitlab.default.example#L34 + +#### Init script + +We updated the init script for GitLab in order to pass new +configuration options to gitlab-workhorse. We let gitlab-workhorse +connect to the Rails application via a Unix domain socket and we tell +it where the 'public' directory of GitLab is. + +``` +cd /home/git/gitlab +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 8. Use Redis v2.8.0+ + +Previous versions of GitLab allowed Redis versions >= 2.0 to be used, but +GitLab 8.3 uses Sidekiq 4.0, which requires Redis 2.8. You can check your Redis version +with the following command: + + redis-cli info | grep redis_version + +If you need to upgrade, see the [installation guide for Redis](https://gitlab.com/gitlab-org/gitlab-ce/blob/8-3-stable/doc/install/installation.md#6-redis). + +### 9. Start application + + sudo service gitlab start + sudo service nginx restart + +### 10. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (8.2) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 8.1 to 8.2](8.1-to-8.2.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +## Troubleshooting + +### "You appear to have cloned an empty repository." + +See the [7.14 to 8.0 update guide](7.14-to-8.0.md#troubleshooting). diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 593722eb01f..c19ee49f9e0 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -6,7 +6,8 @@ For example from 7.14.0 to 7.14.3, also see the [semantic versioning specificati ### 0. Backup It's useful to make a backup just in case things go south: -(With MySQL, this may require granting "LOCK TABLES" privileges to the GitLab user on the database version) +(With MySQL, this may require granting "LOCK TABLES" privileges to the GitLab +user on the database version) ```bash cd /home/git/gitlab @@ -15,19 +16,23 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ### 1. Stop server - sudo service gitlab stop +```bash +sudo service gitlab stop +``` ### 2. Get latest code for the stable branch +In the commands below, replace `LATEST_TAG` with the latest GitLab tag you want +to update to, for example `v8.0.3`. Use `git tag -l 'v*.[0-9]' --sort='v:refname'` +to see a list of all tags. Make sure to update patch versions only (check your +current version with `cat VERSION`). + ```bash cd /home/git/gitlab sudo -u git -H git fetch --all sudo -u git -H git checkout -- Gemfile.lock db/schema.rb sudo -u git -H git checkout LATEST_TAG -b LATEST_TAG ``` -Replace `LATEST_TAG` with the latest GitLab tag you want to update to, for example `v8.0.3`. -Use `git tag -l 'v*.[0-9]' --sort='v:refname'` to see a list of all tags. -Make sure to update patch versions only (check your current version with `cat VERSION`) ### 3. Update gitlab-shell to the corresponding version @@ -37,12 +42,20 @@ sudo -u git -H git fetch sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` -b v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` ``` -### 4. Install libs, migrations, etc. +### 4. Update gitlab-workhorse to the corresponding version + +```bash +cd /home/git/gitlab-workhorse +sudo -u git -H git fetch +sudo -u git -H git checkout `cat /home/git/gitlab/GITLAB_WORKHORSE_VERSION` -b `cat /home/git/gitlab/GITLAB_WORKHORSE_VERSION` +``` + +### 5. Install libs, migrations, etc. ```bash cd /home/git/gitlab -#PostgreSQL +# PostgreSQL sudo -u git -H bundle install --without development test mysql --deployment # MySQL @@ -52,19 +65,25 @@ sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production ``` -### 5. Start application +### 6. Start application - sudo service gitlab start - sudo service nginx restart +```bash +sudo service gitlab start +sudo service nginx restart +``` -### 6. Check application status +### 7. Check application status Check if GitLab and its environment are configured correctly: - sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +```bash +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` To make sure you didn't miss anything run a more thorough check with: - sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +```bash +sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +``` If all items are green, then congratulations upgrade complete! diff --git a/doc/web_hooks/web_hooks.md b/doc/web_hooks/web_hooks.md index 7d838187a26..6420d65cf1b 100644 --- a/doc/web_hooks/web_hooks.md +++ b/doc/web_hooks/web_hooks.md @@ -57,6 +57,9 @@ X-Gitlab-Event: Push Hook "name": "Jordi Mallach", "email": "jordi@softcatala.org" } + "added": ["CHANGELOG"], + "modified": ["app/controller/application.rb"], + "removed": [] }, { "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", @@ -66,13 +69,14 @@ X-Gitlab-Event: Push Hook "author": { "name": "GitLab dev user", "email": "gitlabdev@dv6700.(none)" - } + }, + "added": ["CHANGELOG"], + "modified": ["app/controller/application.rb"], + "removed": [] } ], - "total_commits_count": 4, - "added": ["CHANGELOG"], - "modified": ["app/controller/application.rb"], - "removed": [] + "total_commits_count": 4 + } ``` @@ -184,7 +188,7 @@ X-Gitlab-Event: Note Hook { "object_kind": "note", "user": { - "name": "Adminstrator", + "name": "Administrator", "username": "root", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" }, @@ -337,7 +341,7 @@ X-Gitlab-Event: Note Hook { "object_kind": "note", "user": { - "name": "Adminstrator", + "name": "Administrator", "username": "root", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" }, diff --git a/doc/workflow/README.md b/doc/workflow/README.md index c1a4f64981f..3651b55f438 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -17,3 +17,6 @@ - [Milestones](milestones.md) - [Merge Requests](merge_requests.md) - ["Work In Progress" Merge Requests](wip_merge_requests.md) +- [Merge When Build Succeeds](merge_when_build_succeeds.md) +- [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md) +- [Importing from SVN, GitHub, BitBucket, etc](importing/README.md) diff --git a/doc/workflow/award_emoji.png b/doc/workflow/award_emoji.png Binary files differnew file mode 100644 index 00000000000..fb26ee04393 --- /dev/null +++ b/doc/workflow/award_emoji.png diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index 31495bce76e..8965e5b3654 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -244,13 +244,12 @@ Developing software happen in small messy steps and it is OK to have your histor You can use tools to view the network graphs of commits and understand the messy history that created your code. If you rebase code the history is incorrect, and there is no way for tools to remedy this because they can't deal with changing commit identifiers. -## Voting on merge requests +## Award emojis on issues and merge requests - + -It is common to voice approval or disapproval by using +1 or -1 emoticons. -In GitLab the +1 and -1 are aggregated and shown at the top of the merge request. -As a rule of thumb anything that doesn't have two times more +1's than -1's is suspect and should not be merged yet. +It is common to voice approval or disapproval by using +1 or -1. In GitLab you +can use emojis to give a virtual high five on issues and merge requests. ## Pushing and removing branches diff --git a/doc/workflow/importing/README.md b/doc/workflow/importing/README.md index 7ccf06fbd60..18e5d950866 100644 --- a/doc/workflow/importing/README.md +++ b/doc/workflow/importing/README.md @@ -1,13 +1,17 @@ # Migrating projects to a GitLab instance
1. [Bitbucket](import_projects_from_bitbucket.md)
-2. [GitHub](import_projects_from_github.md)
-3. [GitLab.com](import_projects_from_gitlab_com.md)
-4. [FogBugz](import_projects_from_fogbugz.md)
-4. [SVN](migrating_from_svn.md)
+1. [GitHub](import_projects_from_github.md)
+1. [GitLab.com](import_projects_from_gitlab_com.md)
+1. [FogBugz](import_projects_from_fogbugz.md)
+1. [SVN](migrating_from_svn.md)
-### Note
-* If you'd like to migrate from a self-hosted GitLab instance to GitLab.com, you can copy your repos by changing the remote and pushing to the new server; but issues and merge requests can't be imported.
+In addition to the specific migration documentation above, you can import any
+Git repository via HTTP from the New Project page. Be aware that if the
+repository is too large the import can timeout.
+
+### Migrating from self-hosted GitLab to GitLab.com
+
+You can copy your repos by changing the remote and pushing to the new server;
+but issues and merge requests can't be imported.
-* You can import any Git repository via HTTP from the New Project page.
-If the repository is too large, it can timeout.
diff --git a/doc/workflow/importing/migrating_from_svn.md b/doc/workflow/importing/migrating_from_svn.md index 1938ccd0c26..b355a91b5a6 100644 --- a/doc/workflow/importing/migrating_from_svn.md +++ b/doc/workflow/importing/migrating_from_svn.md @@ -1,17 +1,78 @@ # Migrating from SVN to GitLab -SVN stands for Subversion and is a version control system (VCS). -Git is a distributed version control system. +Subversion (SVN) is a central version control system (VCS) while +Git is a distributed version control system. There are some major differences +between the two, for more information consult your favorite search engine. -There are some major differences between the two, for more information consult your favorite search engine. +If you are currently using an SVN repository, you can migrate the repository +to Git and GitLab. We recommend a hard cut over - run the migration command once +and then have all developers start using the new GitLab repository immediately. +Otherwise, it's hard to keep changing in sync in both directions. The conversion +process should be run on a local workstation. -Git has tools for migrating SVN repositories to git, namely `git svn`. You can read more about this at -[git documentation pages](https://git-scm.com/book/en/Git-and-Other-Systems-Git-and-Subversion). +Install `svn2git`. On all systems you can install as a Ruby gem if you already +have Ruby and Git installed. -Apart from the [official git documentation](https://git-scm.com/book/en/Git-and-Other-Systems-Migrating-to-Git) there is also -user created step by step guide for migrating from SVN to GitLab. +```bash +sudo gem install svn2git +``` -[Benjamin New](https://github.com/leftclickben) wrote [a guide that shows how to do a migration](https://gist.github.com/leftclickben/322b7a3042cbe97ed2af). Mirrors can be found [here](https://gitlab.com/snippets/2168) and [here](https://gist.github.com/maxlazio/f1b593b0d00aa966e9ca). +On Debian-based Linux distributions you can install the native packages: + +```bash +sudo apt-get install git-core git-svn ruby +``` + +Optionally, prepare an authors file so `svn2git` can map SVN authors to Git authors. +If you choose not to create the authors file then commits will not be attributed +to the correct GitLab user. Some users may not consider this a big issue while +others will want to ensure they complete this step. If you choose to map authors +you will be required to map every author that is present on changes in the SVN +repository. If you don't, the conversion will fail and you will have to update +the author file accordingly. The following command will search through the +repository and output a list of authors. + +```bash +svn log --quiet | grep -E "r[0-9]+ \| .+ \|" | cut -d'|' -f2 | sed 's/ //g' | sort | uniq +``` + +Use the output from the last command to construct the authors file. +Create a file called `authors.txt` and add one mapping per line. + +``` +janedoe = Jane Doe <janedoe@example.com> +johndoe = John Doe <johndoe@example.com> +``` + +If your SVN repository is in the standard format (trunk, branches, tags, +not nested) the conversion is simple. For a non-standard repository see +[svn2git documentation](https://github.com/nirvdrum/svn2git). The following +command will checkout the repository and do the conversion in the current +working directory. Be sure to create a new directory for each repository before +running the `svn2git` command. The conversion process will take some time. + +```bash +svn2git https://svn.example.com/path/to/repo --authors /path/to/authors.txt +``` + +If your SVN repository requires a username and password add the +`--username <username>` and `--password <password` flags to the above command. +`svn2git` also supports excluding certain file paths, branches, tags, etc. See +[svn2git documentation](https://github.com/nirvdrum/svn2git) or run +`svn2git --help` for full documentation on all of the available options. + +Create a new GitLab project, where you will eventually push your converted code. +Copy the SSH or HTTP(S) repository URL from the project page. Add the GitLab +repository as a Git remote and push all the changes. This will push all commits, +branches and tags. + +```bash +git remote add origin git@gitlab.com:<group>/<project>.git +git push --all origin +``` ## Contribute to this guide -We welcome all contributions that would expand this guide with instructions on how to migrate from SVN and other version control systems. +We welcome all contributions that would expand this guide with instructions on +how to migrate from SVN and other version control systems. + + diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md new file mode 100644 index 00000000000..5076b2697a3 --- /dev/null +++ b/doc/workflow/lfs/lfs_administration.md @@ -0,0 +1,41 @@ +# GitLab Git LFS Administration + +Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](manage_large_binaries_with_git_lfs.md). + +## Requirements + +* Git LFS is supported in GitLab starting with version 8.2. +* Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up. + +## Configuration + +Git LFS objects can be large in size. By default, they are stored on the server GitLab is installed on. + +There are two configuration options to help GitLab server administrators: + +* Enabling/disabling Git LFS support +* Changing the location of LFS object storage + +### Omnibus packages + +In `/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_rails['lfs_enabled'] = false +gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" +``` + +### Installations from source + +In `config/gitlab.yml`: + +```yaml + lfs: + enabled: false + storage_path: /mnt/storage/lfs-objects +``` + +## Known limitations + +* Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets) is not supported +* Currently, removing LFS objects from GitLab Git LFS storage is not supported diff --git a/doc/workflow/git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index e1064051fe8..b59e92cb317 100644 --- a/doc/workflow/git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -7,55 +7,27 @@ GitLab already supports [managing large files with git annex](http://doc.gitlab. environments it is not always convenient to use different commands to differentiate between the large files and regular ones. Git LFS makes this simpler for the end user by removing the requirement to learn new commands. -<!-- more --> ## How it works Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication to authorize client requests. Once the request is authorized, Git LFS client receives instructions from where to fetch or where to push the large file. -## Requirements - -* Git LFS is supported in GitLab starting with version 8.2 -* Git LFS [client](https://git-lfs.github.com) version 0.6.0 and up - -## GitLab and Git LFS - -### Configuration - -Git LFS objects can be large in size. By default, they are stored on the server GitLab is installed on. - -There are two configuration options to help GitLab server administrators: +## GitLab server configuration -* Enabling/disabling Git LFS support -* Changing the location of LFS object storage +Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md). -#### Omnibus packages - -In `/etc/gitlab/gitlab.rb`: - -```ruby -gitlab_rails['lfs_enabled'] = false -gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" -``` - -#### Installations from source - -In `config/gitlab.yml`: +## Requirements -```yaml - lfs: - enabled: false - storage_path: /mnt/storage/lfs-objects -``` +* Git LFS is supported in GitLab starting with version 8.2 +* [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up ## Known limitations -* Git LFS v1 original API is not supported since it was deprecated early in LFS development, starting with Git LFS version 0.6.0 +* Git LFS v1 original API is not supported since it was deprecated early in LFS development * When SSH is set as a remote, Git LFS objects still go through HTTPS * Any Git LFS request will ask for HTTPS credentials to be provided so good Git credentials store is recommended -* Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets) is not supported -* Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the url to Git config manually (see #troubleshooting-tips) +* Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the URL to Git config manually (see #troubleshooting) ## Using Git LFS @@ -77,13 +49,17 @@ git commit -am "Added Debian iso" # commit the file meta data git push origin master # sync the git repo and large file to the GitLab server ``` -Downloading a single large file is also very simple: +Cloning the repository works the same as before. Git automatically detects the LFS-tracked files and clones them via HTTP. If you performed the git clone command with a SSH URL, you have to enter your GitLab credentials for HTTP authentication. ```bash git clone git@gitlab.example.com:group/project.git -git lfs fetch debian.iso # download the large file ``` +If you already cloned the repository and you want to get the latest LFS object that are on the remote repository, eg. from branch `master`: + +```bash +git lfs fetch master +``` ## Troubleshooting @@ -91,17 +67,31 @@ git lfs fetch debian.iso # download the large file There are a couple of reasons why this error can occur: -* Wrong version of LFS client used: +* You don't have permissions to access certain LFS object + +Check if you have permissions to push to the project or fetch from the project. -Check the version of Git LFS on the client machine with `git lfs version`. Only version 0.6.0 and newer are supported. +* Project is not allowed to access the LFS object -* Project is using deprecated LFS API +LFS object you are trying to push to the project or fetch from the project is not available to the project anymore. +Probably the object was removed from the server. -Check the Git config of the project for traces of deprecated API with `git lfs -l`. If `batch = false` is set in the config, remove the line and try using Git LFS client newer than 0.6.0. +* Local git repository is using deprecated LFS API ### Invalid status for <url> : 501 -When attempting to push a LFS object to a GitLab server that doesn't have Git LFS support enabled, server will return status `error 501`. Check with your GitLab administrator why Git LFS is not enabled on the server. See [Configuration section](#configuration) for instructions on how to enable LFS support. +Git LFS will log the failures into a log file. +To view this log file, while in project directory: + +```bash +git lfs logs last +``` + +If the status `error 501` is shown, it is because: + +* Git LFS support is not enabled on the GitLab server. Check with your GitLab administrator why Git LFS is not enabled on the server. See [LFS administration documentation](lfs_administration.md) for instructions on how to enable LFS support. + +* Git LFS client version is not supported by GitLab server. Check your Git LFS version with `git lfs version`. Check the Git config of the project for traces of deprecated API with `git lfs -l`. If `batch = false` is set in the config, remove the line and try to update your Git LFS client. Only version 1.0.1 and newer are supported. ### getsockopt: connection refused @@ -131,6 +121,6 @@ git config --global credential.helper 'cache --timeout=3600' This will remember the credentials for an hour after which Git operations will require re-authentication. -If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. For Windows, `wincred` is available. +If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases). -More details about various methods of storing the user credentials can be found on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage)
\ No newline at end of file +More details about various methods of storing the user credentials can be found on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
\ No newline at end of file diff --git a/doc/workflow/merge_when_build_succeeds.md b/doc/workflow/merge_when_build_succeeds.md new file mode 100644 index 00000000000..75e1fdff2b2 --- /dev/null +++ b/doc/workflow/merge_when_build_succeeds.md @@ -0,0 +1,15 @@ +# Merge When Build Succeeds + +When reviewing a merge request that looks ready to merge but still has one or more CI builds running, you can set it to be merged automatically when all builds succeed. This way, you don't have to wait for the builds to finish and remember to merge the request manually. + + + +When you hit the "Merge When Build Succeeds" button, the status of the merge request will be updated to represent the impending merge. If you cannot wait for the build to succeed and want to merge immediately, this option is available in the dropdown menu on the right of the main button. + +Both team developers and the author of the merge request have the option to cancel the automatic merge if they find a reason why it shouldn't be merged after all. + + + +When the build succeeds, the merge request will automatically be merged. When the build fails, the author gets a chance to retry any failed builds, or to push new commits to fix the failure. + +When the builds are retried and succeed on the second try, the merge request will automatically be merged after all. When the merge request is updated with new commits, the automatic merge is automatically canceled to allow the new changes to be reviewed. diff --git a/doc/workflow/merge_when_build_succeeds/enable.png b/doc/workflow/merge_when_build_succeeds/enable.png Binary files differnew file mode 100644 index 00000000000..633efa1246f --- /dev/null +++ b/doc/workflow/merge_when_build_succeeds/enable.png diff --git a/doc/workflow/merge_when_build_succeeds/status.png b/doc/workflow/merge_when_build_succeeds/status.png Binary files differnew file mode 100644 index 00000000000..c856c7d14dc --- /dev/null +++ b/doc/workflow/merge_when_build_succeeds/status.png diff --git a/doc/workflow/voting_slider.png b/doc/workflow/voting_slider.png Binary files differdeleted file mode 100644 index 4c660ef9593..00000000000 --- a/doc/workflow/voting_slider.png +++ /dev/null |
