diff options
| author | Dennis Tang <dennis@dennistang.net> | 2018-08-20 20:59:11 +0000 |
|---|---|---|
| committer | Dennis Tang <dennis@dennistang.net> | 2018-08-20 20:59:11 +0000 |
| commit | fb28b3b375f54a7c14a395eee366cd2509a309be (patch) | |
| tree | 859f7f3c28cf9012958ae61a834c02e8a0785cd6 /doc | |
| parent | e8c69b9581436e7bae9a5ff882948efdd66f7e3c (diff) | |
| parent | 5874b2af0ba3710faf2d247e5a63e867e2b5de7a (diff) | |
| download | gitlab-ce-44704-improve-project-overview-ui-buttons.tar.gz | |
Merge branch '44704-improve-project-overview-ui' into '44704-improve-project-overview-ui-buttons'44704-improve-project-overview-ui-buttons
# Conflicts:
# app/assets/stylesheets/pages/projects.scss
Diffstat (limited to 'doc')
78 files changed, 1885 insertions, 262 deletions
diff --git a/doc/README.md b/doc/README.md index a814c787f94..4248f62c08c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -133,6 +133,7 @@ scales to run your tests faster. - [GitLab CI/CD](ci/README.md): Explore the features and capabilities of Continuous Integration, Continuous Delivery, and Continuous Deployment with GitLab. - [Review Apps](ci/review_apps/index.md): Preview changes to your app right from a merge request. - [Pipeline Graphs](ci/pipelines.md#pipeline-graphs) +- [JUnit test reports](ci/junit_test_reports.md) ### Package diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 637d44d2823..b74554a5598 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -25,11 +25,11 @@ for each GitLab application server in your environment. options. Here is an example snippet to add to `/etc/fstab`: ``` - 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 ``` 1. Create the shared directories. These may be different depending on your NFS diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 387c3fb6a5b..cd2284f5f2a 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -55,14 +55,14 @@ Below is an example of an NFS mount point defined in `/etc/fstab` we use on GitLab.com: ``` -10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 +10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 ``` Notice several options that you should consider using: | Setting | Description | | ------- | ----------- | -| `nobootwait` | Don't halt boot process waiting for this mount to become available +| `nofail` | Don't halt boot process waiting for this mount to become available | `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. ## A single NFS mount diff --git a/doc/administration/index.md b/doc/administration/index.md index 112d14652af..030a2f95e23 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -93,7 +93,6 @@ created in snippets, wikis, and repos. - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming emails. -- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. [reply by email]: reply_by_email.md [issues by email]: ../user/project/issues/create_new_issue.md#new-issue-via-email @@ -137,7 +136,6 @@ created in snippets, wikis, and repos. - [Monitoring uptime](../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - [IP whitelist](monitoring/ip_whitelist.md): Monitor endpoints that provide health check information when probed. - [Monitoring GitHub imports](monitoring/github_imports.md): GitLab's GitHub Importer displays Prometheus metrics to monitor the health and progress of the importer. -- [Conversational Development (ConvDev) Index](../user/admin_area/monitoring/convdev.md): Provides an overview of your entire instance's feature usage. ### Performance Monitoring diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index 24d1a3fd151..6e2f67f61bc 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -3,10 +3,6 @@ Job traces are sent by GitLab Runner while it's processing a job. You can see traces in job pages, pipelines, email notifications, etc. -There isn't a way to automatically expire old job logs, but it's safe to remove -them if they're taking up too much space. If you remove the logs manually, the -job output in the UI will be empty. - ## Data flow In general, there are two states in job traces: "live trace" and "archived trace". @@ -57,11 +53,55 @@ To change the location where the job logs will be stored, follow the steps below ## Uploading traces to object storage -An archived trace is considered as a [job artifact](job_artifacts.md). -Therefore, when you [set up an object storage](job_artifacts.md#object-storage-settings), +Archived traces are considered as [job artifacts](job_artifacts.md). +Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings), job traces are automatically migrated to it along with the other job artifacts. -See [Data flow](#data-flow) to learn about the process. +See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process. + +## How to archive legacy job trace files + +Legacy job traces, which were created before GitLab 10.5, were not archived regularly. +It's the same state with the "2: overwriting" in the above [Data flow](#data-flow). +To archive those legacy job traces, please follow the instruction below. + +1. Execute the following command + + ```bash + gitlab-rake gitlab:traces:archive + ``` + + After you executed this task, GitLab instance queues up Sidekiq jobs (asynchronous processes) + for migrating job trace files from local storage to object storage. + It could take time to complete the all migration jobs. You can check the progress by the following command + + ```bash + sudo gitlab-rails console + ``` + + ```bash + [1] pry(main)> Sidekiq::Stats.new.queues['pipeline_background:archive_trace'] + => 100 + ``` + + If the count becomes zero, the archiving processes are done + +## How to migrate archived job traces to object storage + +If job traces have already been archived into local storage, and you want to migrate those traces to object storage, please follow the instruction below. + +1. Ensure [Object storage integration for Job Artifacts](job_artifacts.md#object-storage-settings) is enabled +1. Execute the following command + + ```bash + gitlab-rake gitlab:traces:migrate + ``` + +## How to remove job traces + +There isn't a way to automatically expire old job logs, but it's safe to remove +them if they're taking up too much space. If you remove the logs manually, the +job output in the UI will be empty. ## New live trace architecture diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 8968afba01b..9edccd25ced 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -163,3 +163,20 @@ Such a restriction can currently be hacked in by e.g. providing a custom `AuthorizedKeysCommand` which checks if the discovered key-ID returned from `gitlab-shell-authorized-keys-check` is a deploy key or not (all non-deploy keys should be refused). + +## Disabling the global warning about users lacking SSH keys + +By default GitLab will show a "You won't be able to pull or push +project code via SSH" warning to users who have not uploaded an SSH +key to their profile. + +This is counterproductive when using SSH certificates, since users +aren't expected to upload their own keys. + +To disable this warning globally, go to "Application settings -> +Account and limit settings" and disable the "Show user add SSH key +message" setting. + +This setting was added specifically for use with SSH certificates, but +can be turned off without using them if you'd like to hide the warning +for some other reason. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 88221db78f1..bd758c49eba 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -42,7 +42,7 @@ Registry, etc. ## Hashed Storage > **Warning:** Hashed storage is in **Beta**. For the latest updates, check the -> associated [issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821) +> associated [issue](https://gitlab.com/gitlab-com/infrastructure/issues/3542) > and please report any problems you encounter. Hashed Storage is the new storage behavior we are rolling out with 10.0. Instead diff --git a/doc/api/boards.md b/doc/api/boards.md index 246de50323e..5f006f4f012 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -144,7 +144,7 @@ Example response: ## List board lists Get a list of the board's lists. -Does not include `backlog` and `closed` lists +Does not include `open` and `closed` lists ``` GET /projects/:id/boards/:board_id/lists diff --git a/doc/api/events.md b/doc/api/events.md index f4d26c4de1c..1b6c4d437dd 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -255,7 +255,7 @@ Example response: Get a list of visible events for a particular project. ``` -GET /:project_id/events +GET /projects/:project_id/events ``` Parameters: diff --git a/doc/api/group_boards.md b/doc/api/group_boards.md index 45a8544d6b1..373904e50c4 100644 --- a/doc/api/group_boards.md +++ b/doc/api/group_boards.md @@ -119,7 +119,7 @@ Example response: ## List board lists Get a list of the board's lists. -Does not include `backlog` and `closed` lists +Does not include `open` and `closed` lists ``` GET /groups/:id/boards/:board_id/lists diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 9a950097675..4bf65a8fafd 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -33,7 +33,6 @@ Example of response }, "coverage": null, "created_at": "2015-12-24T15:51:21.727Z", - "artifacts_file": null, "finished_at": "2015-12-24T17:54:24.921Z", "artifacts_expire_at": "2016-01-23T17:54:24.921Z", "id": 6, @@ -45,6 +44,7 @@ Example of response "status": "pending" }, "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": "2015-12-24T17:54:24.729Z", @@ -82,6 +82,12 @@ Example of response "filename": "artifacts.zip", "size": 1000 }, + "artifacts": [ + {"file_type": "archive", "size": 1000, "filename": "artifacts.zip", "file_format": "zip"}, + {"file_type": "metadata", "size": 186, "filename": "metadata.gz", "file_format": "gzip"}, + {"file_type": "trace", "size": 1500, "filename": "job.log", "file_format": "raw"}, + {"file_type": "junit", "size": 750, "filename": "junit.xml.gz", "file_format": "gzip"} + ], "finished_at": "2015-12-24T17:54:27.895Z", "artifacts_expire_at": "2016-01-23T17:54:27.895Z", "id": 7, @@ -93,6 +99,7 @@ Example of response "status": "pending" }, "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": "2015-12-24T17:54:27.722Z", @@ -151,7 +158,6 @@ Example of response }, "coverage": null, "created_at": "2015-12-24T15:51:21.727Z", - "artifacts_file": null, "finished_at": "2015-12-24T17:54:24.921Z", "artifacts_expire_at": "2016-01-23T17:54:24.921Z", "id": 6, @@ -163,6 +169,7 @@ Example of response "status": "pending" }, "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": "2015-12-24T17:54:24.729Z", @@ -200,6 +207,12 @@ Example of response "filename": "artifacts.zip", "size": 1000 }, + "artifacts": [ + {"file_type": "archive", "size": 1000, "filename": "artifacts.zip", "file_format": "zip"}, + {"file_type": "metadata", "size": 186, "filename": "metadata.gz", "file_format": "gzip"}, + {"file_type": "trace", "size": 1500, "filename": "job.log", "file_format": "raw"}, + {"file_type": "junit", "size": 750, "filename": "junit.xml.gz", "file_format": "gzip"} + ], "finished_at": "2015-12-24T17:54:27.895Z", "artifacts_expire_at": "2016-01-23T17:54:27.895Z", "id": 7, @@ -211,6 +224,7 @@ Example of response "status": "pending" }, "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": "2015-12-24T17:54:27.722Z", @@ -267,7 +281,6 @@ Example of response }, "coverage": null, "created_at": "2015-12-24T15:51:21.880Z", - "artifacts_file": null, "finished_at": "2015-12-24T17:54:31.198Z", "artifacts_expire_at": "2016-01-23T17:54:31.198Z", "id": 8, @@ -279,6 +292,7 @@ Example of response "status": "pending" }, "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": "2015-12-24T17:54:30.733Z", @@ -458,11 +472,11 @@ Example of response }, "coverage": null, "created_at": "2016-01-11T10:13:33.506Z", - "artifacts_file": null, "finished_at": "2016-01-11T10:14:09.526Z", "id": 42, "name": "rubocop", "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": null, @@ -505,11 +519,11 @@ Example of response }, "coverage": null, "created_at": "2016-01-11T10:13:33.506Z", - "artifacts_file": null, "finished_at": null, "id": 42, "name": "rubocop", "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": null, @@ -559,6 +573,7 @@ Example of response "id": 42, "name": "rubocop", "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "created_at": "2016-01-11T10:13:33.506Z", @@ -610,6 +625,7 @@ Example response: "id": 42, "name": "rubocop", "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "created_at": "2016-01-11T10:13:33.506Z", @@ -654,11 +670,11 @@ Example of response }, "coverage": null, "created_at": "2016-01-11T10:13:33.506Z", - "artifacts_file": null, "finished_at": null, "id": 42, "name": "rubocop", "ref": "master", + "artifacts": [], "runner": null, "stage": "test", "started_at": null, diff --git a/doc/api/notification_settings.md b/doc/api/notification_settings.md index 682b90361bd..165b9a11c7a 100644 --- a/doc/api/notification_settings.md +++ b/doc/api/notification_settings.md @@ -15,7 +15,7 @@ mention custom ``` -If the `custom` level is used, specific email events can be controlled. Notification email events are defined in the `NotificationSetting::EMAIL_EVENTS` model variable. Currently, these events are recognized: +If the `custom` level is used, specific email events can be controlled. Available events are returned by `NotificationSetting.email_events`. Currently, these events are recognized: ``` new_note diff --git a/doc/api/projects.md b/doc/api/projects.md index f360b49c293..bda4164ee92 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -258,8 +258,7 @@ GET /projects?custom_attributes[key]=value&custom_attributes[other_key]=other_va ## List user projects -Get a list of visible projects for the given user. When accessed without -authentication, only public projects are returned. +Get a list of visible projects owned by the given user. When accessed without authentication, only public projects are returned. ``` GET /users/:user_id/projects diff --git a/doc/api/repositories.md b/doc/api/repositories.md index cb816bbd712..a4fdeca162e 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -204,3 +204,39 @@ Response: "deletions": 244 }] ``` + +## Merge Base + +Get the common ancestor for 2 refs (commit SHAs, branch names or tags). + +``` +GET /projects/:id/repository/merge_base +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `refs` | array | yes | The refs to find the common ancestor of, for now only 2 refs are supported | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/5/repository/merge_base?refs[]=304d257dcb821665ab5110318fc58a007bd104ed&refs[]=0031876facac3f2b2702a0e53a26e89939a42209" +``` + +Example response: + +```json +{ + "id": "1a0b36b3cdad1d2ee32457c102a8c0b7056fa863", + "short_id": "1a0b36b3", + "title": "Initial commit", + "created_at": "2014-02-27T08:03:18.000Z", + "parent_ids": [], + "message": "Initial commit\n", + "author_name": "Dmitriy Zaporozhets", + "author_email": "dmitriy.zaporozhets@gmail.com", + "authored_date": "2014-02-27T08:03:18.000Z", + "committer_name": "Dmitriy Zaporozhets", + "committer_email": "dmitriy.zaporozhets@gmail.com", + "committed_date": "2014-02-27T08:03:18.000Z" +} +``` diff --git a/doc/api/services.md b/doc/api/services.md index efa173180bb..8c59b232b6d 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -401,48 +401,6 @@ Get Flowdock service settings for a project. GET /projects/:id/services/flowdock ``` -## Gemnasium - -Gemnasium monitors your project dependencies and alerts you about updates and security vulnerabilities. - -CAUTION: **Warning:** -Gemnasium service integration has been deprecated in GitLab 11.0. Gemnasium has been -[acquired by GitLab](https://about.gitlab.com/press/releases/2018-01-30-gemnasium-acquisition.html) -in January 2018 and since May 15, 2018, the service provided by Gemnasium is no longer available. -You can [migrate from Gemnasium to GitLab](https://docs.gitlab.com/ee/user/project/import/gemnasium.html) -to keep monitoring your dependencies. - -### Create/Edit Gemnasium service - -Set Gemnasium service for a project. - -``` -PUT /projects/:id/services/gemnasium -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `api_key` | string | true | Your personal API KEY on gemnasium.com | -| `token` | string | true | The project's slug on gemnasium.com | - -### Delete Gemnasium service - -Delete Gemnasium service for a project. - -``` -DELETE /projects/:id/services/gemnasium -``` - -### Get Gemnasium service settings - -Get Gemnasium service settings for a project. - -``` -GET /projects/:id/services/gemnasium -``` - ## Hangouts Chat Google GSuite team collaboration tool. diff --git a/doc/api/settings.md b/doc/api/settings.md index 68fc56b1fa3..b480d62e16a 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -56,7 +56,8 @@ Example response: "enforce_terms": true, "terms": "Hello world!", "performance_bar_allowed_group_id": 42, - "instance_statistics_visibility_private": false + "instance_statistics_visibility_private": false, + "user_show_add_ssh_key_message": true } ``` @@ -161,6 +162,8 @@ PUT /application/settings | `enforce_terms` | boolean | no | Enforce application ToS to all users | | `terms` | text | yes (if `enforce_terms` is true) | Markdown content for the ToS | | `instance_statistics_visibility_private` | boolean | no | When set to `true` Instance statistics will only be available to admins | +| `user_show_add_ssh_key_message` | boolean | no | When set to `false` disable the "You won't be able to pull or push ++project code via SSH" warning shown to users with no uploaded SSH key | ```bash curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/application/settings?signup_enabled=false&default_project_visibility=internal @@ -206,6 +209,7 @@ Example response: "enforce_terms": true, "terms": "Hello world!", "performance_bar_allowed_group_id": 42, - "instance_statistics_visibility_private": false + "instance_statistics_visibility_private": false, + "user_show_add_ssh_key_message": true } ``` diff --git a/doc/api/system_hooks.md b/doc/api/system_hooks.md index dd424470b67..7b8db6cfa8f 100644 --- a/doc/api/system_hooks.md +++ b/doc/api/system_hooks.md @@ -34,6 +34,7 @@ Example response: "push_events":true, "tag_push_events":false, "merge_requests_events": true, + "repository_update_events": true, "enable_ssl_verification":true } ] @@ -56,6 +57,7 @@ POST /hooks | `push_events` | boolean | no | When true, the hook will fire on push events | | `tag_push_events` | boolean | no | When true, the hook will fire on new tags being pushed | | `merge_requests_events` | boolean | no | Trigger hook on merge requests events | +| `repository_update_events` | boolean | no | Trigger hook on repository update events | | `enable_ssl_verification` | boolean | no | Do SSL verification when triggering the hook | Example request: @@ -75,6 +77,7 @@ Example response: "push_events":true, "tag_push_events":false, "merge_requests_events": true, + "repository_update_events": true, "enable_ssl_verification":true } ] @@ -127,4 +130,4 @@ Example request: ```bash curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/hooks/2 -``` +```
\ No newline at end of file diff --git a/doc/ci/README.md b/doc/ci/README.md index 7666219acb0..d782d64e971 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -76,6 +76,8 @@ learn how to leverage its potential even more. - [Trigger pipelines on a schedule](../user/project/pipelines/schedules.md) - [Kubernetes clusters](../user/project/clusters/index.md) - Integrate one or more Kubernetes clusters to your project +- [Interactive web terminal](interactive_web_terminal/index.md) - Open an interactive + web terminal to debug the running jobs ## GitLab CI/CD for Docker diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md index b0e01d74f7e..8ae80b2bc02 100644 --- a/doc/ci/docker/README.md +++ b/doc/ci/docker/README.md @@ -6,3 +6,4 @@ comments: false - [Using Docker Images](using_docker_images.md) - [Using Docker Build](using_docker_build.md) +- [Using kaniko](using_kaniko.md) diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md new file mode 100644 index 00000000000..7d4f28e1f47 --- /dev/null +++ b/doc/ci/docker/using_kaniko.md @@ -0,0 +1,60 @@ +# Building images with kaniko and GitLab CI/CD + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45512) in GitLab 11.2. +Requires GitLab Runner 11.2 and above. + +[kaniko](https://github.com/GoogleContainerTools/kaniko) is a tool to build +container images from a Dockerfile, inside a container or Kubernetes cluster. + +kaniko solves two problems with using the +[docker-in-docker build](using_docker_build.md#use-docker-in-docker-executor) method: + +1. Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) + in order to function, which is a significant security concern. +1. Docker-in-docker generally incurs a performance penalty and can be quite slow. + +## Requirements + +In order to utilize kaniko with GitLab, a [GitLab Runner](https://docs.gitlab.com/runner/) +using either the [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html), +[Docker](https://docs.gitlab.com/runner/executors/docker.html), or +[Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html) +executors is required. + +## Building a Docker image with kaniko + +When building an image with kaniko and GitLab CI/CD, you should be aware of a +few important details: + +- The kaniko debug image is recommended (`gcr.io/kaniko-project/executor:debug`) + because it has a shell, and a shell is required for an image to be used with + GitLab CI/CD. +- The entrypoint will need to be [overridden](using_docker_images.md#overriding-the-entrypoint-of-an-image), + otherwise the build script will not run. +- A Docker `config.json` file needs to be created with the authentication + information for the desired container registry. + +--- + +In the following example, kaniko is used to build a Docker image and then push +it to [GitLab Container Registry](../../user/project/container_registry.md). +The job will run only when a tag is pushed. A `config.json` file is created under +`/root/.docker` with the needed GitLab Container Registry credentials taken from the +[environment variables](../variables/README.md#predefined-variables-environment-variables) +GitLab CI/CD provides. In the last step, kaniko uses the `Dockerfile` under the +root directory of the project, builds the Docker image and pushes it to the +project's Container Registry while tagging it with the Git tag: + +```yaml +build: + stage: build + image: + name: gcr.io/kaniko-project/executor:debug + entrypoint: [""] + script: + - mkdir -p /root/.docker + - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /root/.docker/config.json + - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG + only: + - tags +``` diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 811f4d1f07a..8eb96ae10b2 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -43,6 +43,10 @@ There's also a collection of repositories with [example projects](https://gitlab - [Using `dpl` as deployment tool](deployment/README.md) - [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) +## Test Reports + +[Collect test reports in Verify stage](../junit_test_reports.md). + ## Code Quality analysis **(Starter)** [Analyze your project's Code Quality](code_quality.md). diff --git a/doc/ci/img/junit_test_report.png b/doc/ci/img/junit_test_report.png Binary files differnew file mode 100644 index 00000000000..ad098eb457f --- /dev/null +++ b/doc/ci/img/junit_test_report.png diff --git a/doc/ci/interactive_web_terminal/img/finished_job_with_terminal_open.png b/doc/ci/interactive_web_terminal/img/finished_job_with_terminal_open.png Binary files differnew file mode 100644 index 00000000000..199268a1486 --- /dev/null +++ b/doc/ci/interactive_web_terminal/img/finished_job_with_terminal_open.png diff --git a/doc/ci/interactive_web_terminal/img/interactive_web_terminal_page.png b/doc/ci/interactive_web_terminal/img/interactive_web_terminal_page.png Binary files differnew file mode 100644 index 00000000000..b59c1b6bc43 --- /dev/null +++ b/doc/ci/interactive_web_terminal/img/interactive_web_terminal_page.png diff --git a/doc/ci/interactive_web_terminal/img/interactive_web_terminal_running_job.png b/doc/ci/interactive_web_terminal/img/interactive_web_terminal_running_job.png Binary files differnew file mode 100644 index 00000000000..f92c6df07a1 --- /dev/null +++ b/doc/ci/interactive_web_terminal/img/interactive_web_terminal_running_job.png diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md new file mode 100644 index 00000000000..507aceb27fa --- /dev/null +++ b/doc/ci/interactive_web_terminal/index.md @@ -0,0 +1,52 @@ +# Getting started with interactive web terminals + +> Introduced in GitLab 11.3. + +CAUTION: **Warning:** +Interactive web terminals are in beta, so they might not work properly and +lack features. For more information [follow issue #25990](https://gitlab.com/gitlab-org/gitlab-ce/issues/25990). + +Interactive web terminals give the user access to a terminal in GitLab for +running one-of commands for their CI pipeline. + +NOTE: **Note:** +This is not available for the shared Runners on GitLab.com. +To make use of this feature, you need to provide your +[own Runner](https://docs.gitlab.com/runner/install/) and properly +[configure it](#configuration). + +## Configuration + +Two things need to be configured for the interactive web terminal to work: + +- The Runner needs to have [`[session_server]` configured + properly][session-server] +- Web terminals need to be + [enabled](../../administration/integration/terminal.md#enabling-and-disabling-terminal-support) + +## Debugging a running job + +NOTE: **Note:** Not all executors are +[supported](https://docs.gitlab.com/runner/executors/#compatibility-chart). + +Sometimes, when a job is running, things don't go as you would expect, and it +would be helpful if one can have a shell to aid debugging. When a job is +running, on the right panel you can see a button `debug` that will open the terminal +for the current job. + + + +When clicked, a new tab will open to the terminal page where you can access +the terminal and type commands like a normal shell. + + + +If you have the terminal open and the job has finished with its tasks, the +terminal will block the job from finishing for the duration configured in +[`[session_server].terminal_max_retention_time`][session-server] until you +close the terminal window. + + + +[session-server]: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md new file mode 100644 index 00000000000..5ae8ecaafa6 --- /dev/null +++ b/doc/ci/junit_test_reports.md @@ -0,0 +1,102 @@ +# JUnit test reports + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45318) in GitLab 11.2. +Requires GitLab Runner 11.2 and above. + +## Overview + +It is very common that a [CI/CD pipeline](pipelines.md) contains a +test job that will verify your code. +If the tests fail, the pipeline fails and users get notified. The person that +works on the merge request will have to check the job logs and see where the +tests failed so that they can fix them. + +You can configure your job to use JUnit test reports, and GitLab will display a +report on the merge request so that it's easier and faster to identify the +failure without having to check the entire log. + +## Use cases + +Consider the following workflow: + +1. Your `master` branch is rock solid, your project is using GitLab CI/CD and + your pipelines indicate that there isn't anything broken. +1. Someone from you team submits a merge request, a test fails and the pipeline + gets the known red icon. To investigate more, you have to go through the job + logs to figure out the cause of the failed test, which usually contain + thousands of lines. +1. You configure the JUnit test reports and immediately GitLab collects and + exposes them in the merge request. No more searching in the job logs. +1. Your development and debugging workflow becomes easier, faster and efficient. + +## How it works + +First, GitLab Runner uploads all JUnit XML files as artifacts to GitLab. Then, +when you visit a merge request, GitLab starts comparing the head and base branch's +JUnit test reports, where: + +- The base branch is the target branch (usually `master`). +- The head branch is the source branch (the latest pipeline in each merge request). + +The reports panel has a summary showing how many tests failed and how many were fixed. +If no comparison can be done because data for the base branch is not available, +the panel will just show the list of failed tests for head. + +There are three types of results: + +1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch +1. **Existing failures:** Test cases which failed on base branch and failed on head branch +1. **Resolved failures:** Test cases which failed on base branch and passed on head branch + +Each entry in the panel will show the test name and its type from the list +above. Clicking on the test name will open a modal window with details of its +execution time and the error output. + + + +## How to set it up + +NOTE: **Note:** +For a list of supported languages on JUnit tests, check the +[Wikipedia article](https://en.wikipedia.org/wiki/JUnit#Ports). + +To enable the JUnit reports in merge requests, you need to add +[`artifacts:reports:junit`](yaml/README.md#artifacts-reports-junit) +in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. + +In the following examples, the job in the `test` stage runs and GitLab +collects the JUnit test report from each job. After each job is executed, the +XML reports are stored in GitLab as artifacts and their results are shown in the +merge request widget. + +### Ruby example + +Use the following job in `.gitlab-ci.yml`: + +```yaml +## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec +ruby: + stage: test + script: + - bundle install + - rspec spec/lib/ --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +### Go example + +Use the following job in `.gitlab-ci.yml`: + +```yaml +## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go +golang: + stage: test + script: + - go get -u github.com/jstemmer/go-junit-report + - go test -v 2>&1 | go-junit-report > report.xml + artifacts: + reports: + junit: report.xml +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 95d705d3a3d..abba748db8b 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1075,8 +1075,10 @@ keep artifacts forever. After their expiry, artifacts are deleted hourly by default (via a cron job), and are not accessible anymore. -The value of `expire_in` is an elapsed time. Examples of parsable values: +The value of `expire_in` is an elapsed time in seconds, unless a unit is +provided. Examples of parsable values: +- '42' - '3 mins 4 sec' - '2 hrs 20 min' - '2h20min' @@ -1092,6 +1094,52 @@ job: expire_in: 1 week ``` +### `artifacts:reports` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in +GitLab 11.2. Requires GitLab Runner 11.2 and above. + +The `reports` keyword is used for collecting test reports from jobs and +exposing them in GitLab's UI (merge requests, pipeline views). Read how to use +this with [JUnit reports](#artifacts-reports-junit). + +NOTE: **Note:** +The test reports are collected regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](#artifacts-expire_in) to set up an expiration +date for their artifacts. + +#### `artifacts:reports:junit` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in +GitLab 11.2. Requires GitLab Runner 11.2 and above. + +The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +as artifacts. Although JUnit was originally developed in Java, there are many +[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other +languages like Javascript, Python, Ruby, etc. + +Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: + +```yaml +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +The collected JUnit reports will be uploaded to GitLab as an artifact and will +be automatically [shown in merge requests](../junit_test_reports.md). + +NOTE: **Note:** +In case the JUnit tool you use exports to multiple XML files, you can specify +multiple test report paths within a single job +(`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`) and they will be automatically +concatenated into a single file. + ## `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. diff --git a/doc/development/README.md b/doc/development/README.md index fed3903c771..ee9a9852205 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -55,7 +55,13 @@ description: 'Learn how to contribute to GitLab.' - [Merge request performance guidelines](merge_request_performance_guidelines.md) for ensuring merge requests do not negatively impact GitLab performance -## Databases guides +## Database guides + +### Tooling + +- [Understanding EXPLAIN plans](understanding_explain_plans.md) +- [explain.depesz.com](https://explain.depesz.com/) for visualising the output + of `EXPLAIN` ### Migrations diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index d4cce79b067..57ae318e821 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -18,23 +18,16 @@ To better understand the priority by which UX tackles issues, see the [UX sectio Once an issue has been worked on and is ready for development, a UXer removes the ~"UX" label and applies the ~"UX ready" label to that issue. -The UX team has a special type label called ~"design artifact". This label indicates that the final output -for an issue is a UX solution/design. The solution will be developed by frontend and/or backend in a subsequent milestone. -Any issue labeled ~"design artifact" should not also be labeled ~"frontend" or ~"backend" since no development is -needed until the solution has been decided. +There is a special type label called ~"product discovery". It represents a discovery issue intended for UX, PM, FE, and BE to discuss the problem and potential solutions. The final output for this issue could be a doc of requirements, a design artifact, or even a prototype. The solution will be developed in a subsequent milestone. -~"design artifact" issues are like any other issue and should contain a milestone label, ~"Deliverable" or ~"Stretch", when scheduled in the current milestone. +~"product discovery" issues are like any other issue and should contain a milestone label, ~"Deliverable" or ~"Stretch", when scheduled in the current milestone. -To prevent the misunderstanding that a feature will be be delivered in the -assigned milestone, when only UX design is planned for that milestone, the -Product Manager should create a separate issue for the ~"design artifact", -assign the ~UX, ~"design artifact" and ~"Deliverable" labels, add a milestone -and use a title that makes it clear that the scheduled issue is design only -(e.g. `Design exploration for XYZ`). +The initial issue should be about the problem we are solving. If a separate [product discovery issue](#product-discovery-issues) is needed for additional research and design work, it will be created by a PM or UX person. Assign the ~UX, ~"product discovery" and ~"Deliverable" labels, add a milestone and use a title that makes it clear that the scheduled issue is product discovery +(e.g. `Product discovery for XYZ`). -When the ~"design artifact" issue has been completed, the UXer removes the ~UX +When the ~"product discovery" issue has been completed, the UXer removes the ~UX label, adds the ~"UX ready" label and closes the issue. This indicates the -design artifact is complete. The UXer will also copy the designs to related +UX work for the issue is complete. The UXer will also copy any designs to related issues for implementation in an upcoming milestone. ## Style guides diff --git a/doc/development/diffs.md b/doc/development/diffs.md index 55fc16e0b33..d06339480b1 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -1,6 +1,6 @@ -# Working with Merge Request diffs +# Working with diffs -Currently we rely on different sources to present merge request diffs, these include: +Currently we rely on different sources to present diffs, these include: - Rugged gem - Gitaly service @@ -11,6 +11,8 @@ We're constantly moving Rugged calls to Gitaly and the progress can be followed ## Architecture overview +### Merge request diffs + When refreshing a Merge Request (pushing to a source branch, force-pushing to target branch, or if the target branch now contains any commits from the MR) we fetch the comparison information using `Gitlab::Git::Compare`, which fetches `base` and `head` data using Gitaly and diff between them through `Gitlab::Git::Diff.between` (which uses _Gitaly_ if it's enabled, otherwise _Rugged_). @@ -32,6 +34,17 @@ In order to present diffs information on the Merge Request diffs page, we: 3. If the diff file is cacheable (text-based), it's cached on Redis using `Gitlab::Diff::FileCollection::MergeRequestDiff` +### Note diffs + +When commenting on a diff (any comparison), we persist a truncated diff version +on `NoteDiffFile` (which is associated with the actual `DiffNote`). So instead +of hitting the repository every time we need the diff of the file, we: + +1. Check whether we have the `NoteDiffFile#diff` persisted and use it +2. Otherwise, if it's a current MR revision, use the persisted +`MergeRequestDiffFile#diff` +3. In the last scenario, go the the repository and fetch the diff + ## Diff limits As explained above, we limit single diff files and the size of the whole diff. There are scenarios where we collapse the diff file, diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index f5cdd310f6f..f46c171d9f1 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -25,52 +25,23 @@ them to review it for you. We use the [monthly release blog post](https://about.gitlab.com/handbook/marketing/blog/release-posts/#monthly-releases) as a changelog checklist to ensure everything is documented. -Whenever you submit a merge request for the documentation, use the documentation MR description template. +Whenever you submit a merge request for the documentation, use the +"Documentation" MR description template. If you're changing documentation +location, use the MR description template called "Change documentation +location" instead. -Please check the [documentation workflow](https://about.gitlab.com/handbook/product/technical-writing/workflow/) before getting started. +## Documentation workflow -## Documentation structure - -- Overview and use cases: what it is, why it is necessary, why one would use it -- Requirements: what do we need to get started -- Tutorial: how to set it up, how to use it - -Always link a new document from its topic-related index, otherwise, it will -not be included it in the documentation site search. - -_Note: to be extended._ - -### Feature overview and use cases - -Every major feature (regardless if present in GitLab Community or Enterprise editions) -should present, at the beginning of the document, two main sections: **overview** and -**use cases**. Every GitLab EE-only feature should also contain these sections. +Please read through the [documentation workflow](workflow.md) before getting started. -**Overview**: as the name suggests, the goal here is to provide an overview of the feature. -Describe what is it, what it does, why it is important/cool/nice-to-have, -what problem it solves, and what you can do with this feature that you couldn't -do before. - -**Use cases**: provide at least two, ideally three, use cases for every major feature. -You should answer this question: what can you do with this feature/change? Use cases -are examples of how this feature or change can be used in real life. - -Examples: -- CE and EE: [Issues](../user/project/issues/index.md#use-cases) -- CE and EE: [Merge Requests](../user/project/merge_requests/index.md#overview) -- EE-only: [Geo](https://docs.gitlab.com/ee/gitlab-geo/README.html#overview) -- EE-only: [Jenkins integration](https://docs.gitlab.com/ee/integration/jenkins.md#overview) - -Note that if you don't have anything to add between the doc title (`<h1>`) and -the header `## Overview`, you can omit the header, but keep the content of the -overview there. +## Documentation structure -> **Overview** and **use cases** are required to **every** Enterprise Edition feature, -and for every **major** feature present in Community Edition. +Follow through the [documentation structure guide](structure.md) for learning +how to structure GitLab docs. ## Markdown and styles -Currently GitLab docs use Redcarpet as [markdown](../user/markdown.md) engine, but there's an [open discussion](https://gitlab.com/gitlab-com/gitlab-docs/issues/50) for implementing Kramdown in the near future. +Currently GitLab docs use Redcarpet as [markdown](../../user/markdown.md) engine, but there's an [open discussion](https://gitlab.com/gitlab-com/gitlab-docs/issues/50) for implementing Kramdown in the near future. All the docs follow the [documentation style guidelines](styleguide.md). @@ -84,9 +55,18 @@ In order to have a [solid site structure](https://searchengineland.com/seo-benef all docs should be linked. Every new document should be cross-linked to its related documentation, and linked from its topic-related index, when existent. The directories `/workflow/`, `/gitlab-basics/`, `/university/`, and `/articles/` have -been deprecated and the majority their docs have been moved to their correct location +been **deprecated** and the majority their docs have been moved to their correct location in small iterations. Please don't create new docs in these folders. +### Documentation files + +- When you create a new directory, always start with an `index.md` file. +Do not use another file name and **do not** create `README.md` files +- **Do not** use special chars and spaces, or capital letters in file names, +directory names, branch names, and anything that generates a path. +- Max screenshot size: 100KB +- We do not support videos (yet) + ### Location and naming documents The documentation hierarchy can be vastly improved by providing a better layout @@ -116,7 +96,7 @@ The table below shows what kind of documentation goes where. --- -**General rules:** +**General rules & best practices:** 1. The correct naming and location of a new document, is a combination of the relative URL of the document in question and the GitLab Map design @@ -203,7 +183,7 @@ Things to note: documentation, sometimes it might be useful to search a path deeper. - The `*.md` extension is not used when a document is linked to GitLab's built-in help page, that's why we omit it in `git grep`. -- Use the checklist on the documentation MR description template. +- Use the checklist on the "Change documentation location" MR description template. #### Alternative redirection method @@ -514,7 +494,7 @@ Suppose there's a process to go from point A to point B in 5 steps: `(A) 1 > 2 > A **guide** can be understood as a description of certain processes to achieve a particular objective. A guide brings you from A to B describing the characteristics of that process, but not necessarily going over each step. It can mention, for example, steps 2 and 3, but does not necessarily explain how to accomplish them. -- Live example: "[Static sites and GitLab Pages domains (Part 1)](../user/project/pages/getting_started_part_one.md) to [Creating and Tweaking GitLab CI/CD for GitLab Pages (Part 4)](../../user/project/pages/getting_started_part_four.md)" +- Live example: "[Static sites and GitLab Pages domains (Part 1)](../../user/project/pages/getting_started_part_one.md) to [Creating and Tweaking GitLab CI/CD for GitLab Pages (Part 4)](../../user/project/pages/getting_started_part_four.md)" A **tutorial** requires a clear **step-by-step** guidance to achieve a singular objective. It brings you from A to B, describing precisely all the necessary steps involved in that process, showing each of the 5 steps to go from A to B. It does not only describes steps 2 and 3, but also shows you how to accomplish them. diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md new file mode 100644 index 00000000000..1002836096a --- /dev/null +++ b/doc/development/documentation/structure.md @@ -0,0 +1,149 @@ +--- +description: Learn the how to correctly structure GitLab documentation. +--- + +# Documentation structure + +For consistency throughout the documentation, it's important to maintain the same +structure among the docs. + +Before getting started, read through the following docs: + +- [Contributing to GitLab documentation](index.md#contributing-to-docs) +- [Merge requests for GitLab documentation](index.md#merge-requests-for-gitlab-documentation) +- [Branch naming for docs-only changes](index.md#branch-naming) +- [Documentation directory structure](index.md#documentation-directory-structure) +- [Documentation style guidelines](styleguide.md) +- [Documentation workflow](workflow.md) + +## Documentation blurb + +Every document should include the following content in the following sequence: + +- **Feature name**: defines an intuitive name for the feature that clearly +states what it is and is consistent with any relevant UI text. +- **Feature overview** and description: describe what it is, what it does, and in what context it should be used. +- **Use cases**: describes real use case scenarios for that feature. +- **Requirements**: describes what software and/or configuration is required to be able to +use the feature and, if applicable, prerequisite knowledge for being able to follow/implement the tutorial. +For example, familiarity with GitLab CI/CD, an account on a third-party service, dependencies installed, etc. +Link each one to its most relevant resource; i.e., where the reader can go to begin to fullfil that requirement. +(Another doc page, a third party application's site, etc.) +- **Instructions**: clearly describes the steps to use the feature, leaving no gaps. +- **Troubleshooting** guide (recommended but not required): if you know beforehand what issues +one might have when setting it up, or when something is changed, or on upgrading, it's +important to describe those too. Think of things that may go wrong and include them in the +docs. This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. Answering them beforehand only makes your +document better and more approachable. + +For additional details, see the subsections below, as well as the [Documentation template for new docs](#Documentation-template-for-new-docs). + +### Feature overview and use cases + +Every major feature (regardless if present in GitLab Community or Enterprise editions) +should present, at the beginning of the document, two main sections: **overview** and +**use cases**. Every GitLab EE-only feature should also contain these sections. + +**Overview**: as the name suggests, the goal here is to provide an overview of the feature. +Describe what is it, what it does, why it is important/cool/nice-to-have, +what problem it solves, and what you can do with this feature that you couldn't +do before. + +**Use cases**: provide at least two, ideally three, use cases for every major feature. +You should answer this question: what can you do with this feature/change? Use cases +are examples of how this feature or change can be used in real life. + +Examples: +- CE and EE: [Issues](../user/project/issues/index.md#use-cases) +- CE and EE: [Merge Requests](../user/project/merge_requests/index.md#overview) +- EE-only: [Geo](https://docs.gitlab.com/ee/gitlab-geo/README.html#overview) +- EE-only: [Jenkins integration](https://docs.gitlab.com/ee/integration/jenkins.md#overview) + +Note that if you don't have anything to add between the doc title (`<h1>`) and +the header `## Overview`, you can omit the header, but keep the content of the +overview there. + +> **Overview** and **use cases** are required to **every** Enterprise Edition feature, +and for every **major** feature present in Community Edition. + +### Discoverability + +Your new document will be discoverable by the user only if: + +- Crosslinked from the higher-level index (e.g., Issue Boards docs +should be linked from Issues; Prometheus docs should be linked from +Monitoring; CI/CD tutorials should be linked from CI/CD examples). + - When referencing other GitLab products and features, link to their +respective docs; when referencing third-party products or technologies, +link out to their external sites, documentation, and resources. +- The headings are clear. E.g., "App testing" is a bad heading, "Testing +an application with GitLab CI/CD" is much better. Think of something +someone will search for and use these keywords in the headings. + +## Documentation template for new docs + +To start a new document, respect the file tree and file name guidelines, +as well as the style guidelines. Use the following template: + +```md +--- +description: "short document description." # Up to ~200 chars long. They will be displayed in Google Search Snippets. +--- + +# Feature Name **[TIER]** (1) + +> [Introduced](link_to_issue_or_mr) in GitLab Tier X.Y (2). + +A short description for the feature (can be the same used in the frontmatter's +`description`). + +## Overview + +To write the feature overview, you should consider answering the following questions: + +- What is it? +- Who is it for? +- What is the context in which it is used and are there any prerequisites/requirements? +- What can the user do with it? (Be sure to consider multiple audiences, like GitLab admin and developer-user.) +- What are the benefits to using it over any alternatives? + +## Use cases + +Describe one to three use cases for that feature. Give real-life examples. + +## Requirements + +State any requirements, if any, for using the feature and/or following along with the tutorial. + +The only assumption that is redundant and doesn't need to be mentioned is having an account +on GitLab. + +## Instructions + +("Instructions" is not necessarily the name of the heading) + +- Write a step-by-step guide, with no gaps between the steps. +- Start with an h2 (`##`), break complex steps into small steps using +subheadings h3 > h4 > h5 > h6. _Never skip the hierarchy level, such +as h2 > h4_, as it will break the TOC and may affect the breadcrumbs. +- Use short and descriptive headings (up to ~50 chars). You can use one +single heading `## How it works` for the instructions when the feature +is simple and the document is short. +- Be clear, concise, and stick to the goal of the doc: explain how to +use that feature. +- Use inclusive language and avoid jargons, as well as uncommon and +fancy words. The docs should be clear and very easy to understand. +- Write in the 3rd person (use "we", "you", "us", "one", instead of "I" or "me"). +- Always provide internal and external reference links. +- Always link the doc from its higher-level index. + +<!-- ## Troubleshooting + +Add a troubleshooting guide when possible/applicable. --> +``` + +Notes: + +- (1): Apply the [tier badges](styleguide.md#product-badges) accordingly +- (2): Apply the correct format for the [GitLab version introducing the feature](styleguide.md#gitlab-versions-and-tiers) diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index ad49c77aac8..6c60a517b6d 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -10,6 +10,22 @@ GitLab documentation. Check the Check the GitLab handbook for the [writing styles guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines). +## Files + +- [Directory structure](index.md#location-and-naming-documents): place the docs +in the correct location +- [Documentation files](index.md#documentation-files): name the files accordingly +- [Markdown](../../user/markdown.md): use the GitLab Flavored Markdown in the +documentation + +NOTE: **Note:** +**Do not** use capital letters, spaces, or special chars in file names, +branch names, directory names, headings, or in anything that generates a path. + +NOTE: **Note:** +**Do not** create new `README.md` files, name them `index.md` instead. There's +a test that will fail if it spots a new `README.md` file. + ## Text - Split up long lines (wrap text), this makes it much easier to review and edit. Only @@ -61,7 +77,8 @@ For punctuation rules, please refer to the [GitLab UX guide](https://design.gitl - Add **only one H1** in each document, by adding `#` at the beginning of it (when using markdown). The `h1` will be the document `<title>`. -- For subheadings, use `##`, `###` and so on +- Start with an h2 (`##`), and respect the order h2 > h3 > h4 > h5 > h6. + Never skip the hierarchy level, such as h2 > h4 - Avoid putting numbers in headings. Numbers shift, hence documentation anchor links shift too, which eventually leads to dead links. If you think it is compelling to add numbers in headings, make sure to at least discuss it with @@ -115,10 +132,7 @@ needs to expand the tab to find the settings you're referring to the `.md` document that you're working on is located. Always prepend their names with the name of the document that they will be included in. For example, if there is a document called `twitter.md`, then a valid image name - could be `twitter_login_screen.png`. [**Exception**: images for - [articles](index.md#technical-articles) should be - put in a directory called `img` underneath `/articles/article_title/img/`, therefore, - there's no need to prepend the document name to their filenames.] + could be `twitter_login_screen.png`. - Images should have a specific, non-generic name that will differentiate them. - Keep all file names in lower case. - Consider using PNG images instead of JPEG. @@ -126,6 +140,8 @@ needs to expand the tab to find the settings you're referring to - Compress gifs with <https://ezgif.com/optimize> or similar tool. - Images should be used (only when necessary) to _illustrate_ the description of a process, not to _replace_ it. +- Max image size: 100KB (gifs included). +- The GitLab docs do not support videos yet. Inside the document: diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md new file mode 100644 index 00000000000..339ec80f889 --- /dev/null +++ b/doc/development/documentation/workflow.md @@ -0,0 +1,186 @@ +--- +description: Learn the process of shipping documentation for GitLab. +--- + +# Documentation process at GitLab + +At GitLab, developers contribute new or updated documentation along with their code, but product managers and technical writers also have essential roles in the process. + +- Product Managers (PMs): in the issue for all new and updated features, +PMs include specific documentation requirements that the developer who is +writing or updating the docs must meet, along with feature descriptions +and use cases. They call out any specific areas where collaborating with +a technical writer is recommended, and usually act as the first reviewer +of the docs. +- Developers: author documentation and merge it on time (up to a week after +the feature freeze). +- Technical Writers: review each issue to ensure PM's requirements are complete, +help developers with any questions throughout the process, and act as the final +reviewer of all new and updated docs content before it's merged. + +## Requirements + +Documentation must be delivered whenever: + +- A new feature is shipped +- There are changes to the UI +- A process, workflow, or previously documented feature is changed + +Documentation is not required when a feature is changed on the backend +only and does not directly affect the way that any regular user or +administrator would interact with GitLab. + +NOTE: **Note:** +When refactoring documentation in needed, it should be submitted it in its own MR. +**Do not** join new features' MRs with refactoring existing docs, as they might have +different priorities. + +NOTE: **Note:** +[Smaller MRs are better](https://gitlab.com/gitlab-com/blog-posts/issues/185#note_4401010)! Do not mix subjects, and ship the smallest MR possible. + +### Documentation review process + +The docs shipped by the developer should be reviewed by the PM (for accuracy) and a Technical Writer (for clarity and structure). + +#### Documentation updates that require Technical Writer review + +Every documentation change that meets the criteria below must be reviewed by a Technical Writer +to ensure clarity and discoverability, and avoid redundancy, bad file locations, typos, broken links, etc. +Within the GitLab issue or MR, ping the relevant technical writer for the subject area. If you're not sure who that is, +ping any of them or all of them (`@gl\-docsteam`). + +A Technical Writer must review documentation updates that involve: + +- Docs introducing new features +- Changing documentation location +- Refactoring existing documentation +- Creating new documentation files + +If you need any help to choose the correct place for a doc, discuss a documentation +idea or outline, or request any other help, ping a Technical Writer on your issue, MR, +or on Slack in `#docs`. + +#### Skip the PM's review + +When there's a non-significant change to the docs, you can skip the review +of the PM. Add the same labels as you would for a regular doc change and +assign the correct milestone. In these cases, assign a Technical Writer +for approval/merge, or mention `@gl\-docsteam` in case you don't know +which Tech Writer to assign for. + +#### Skip the entire review + +When the MR only contains corrections to the content (typos, grammar, +broken links, etc), it can be merged without the PM's and Tech Writer's review. + +## Documentation structure + +Read through the [documentation structure](structure.md) docs for an overview. + +## Documentation workflow + +To follow a consistent workflow every month, documentation changes +involve the Product Managers, the developer who shipped the feature, +and the Technical Writing team. Each role is described below. + +### 1. Product Manager's role in the documentation process + +The Product Manager (PM) should add to the feature issue: + +- Feature name, overview/description, and use cases, for the [documentation blurb](structure.md#documentation-blurb) +- The documentation requirements for the developer working on the docs + - What new page, new subsection of an existing page, or other update to an existing page/subsection is needed. + - Just one page/section/update or multiple (perhaps there's an end user and admin change needing docs, or we need to update a previously recommended workflow, or we want to link the new feature from various places; consider and mention all ways documentation should be affected + - Suggested title of any page or subsection, if applicable +- Label the issue with `Documentation`, `Deliverable`, `docs:P1`, and assign + the correct milestone + +### 2. Developer's role in the documentation process + +As a developer, or as a community contributor, you should ship the documentation +with the feature, as in GitLab the documentation is part of the product. + +The docs can either be shipped along with the MR introducing the code, or, +alternatively, created from a follow-up issue and MR. + +The docs should be shipped **by the feature freeze date**. Justified +exceptions are accepted, as long as the [following process](#documentation-shipped-late) +and the missed-deliverable due date (the 14th of each month) are both respected. + +#### Documentation shipped in the feature MR + +The developer should add to the feature MR the documentation containing: + +- The [documentation blurb](structure.md#documentation-blurb): copy the +feature name, overview/description, and use cases from the feature issue +- Instructions: write how to use the feature, step by step, with no gaps. +- [Crosslink for discoverability](structure.md#discoverability): link with +internal docs and external resources (if applicable) +- Index: link the new doc or the new heading from the higher-level index +for [discoverability](#discoverability) +- [Screenshots](styleguide.md#images): when necessary, add screenshots for: + - Illustrating a step of the process + - Indicating the location of a navigation menu +- Label the MR with `Documentation`, `Deliverable`, `docs-P1`, and assign +the correct milestone +- Assign the PM for review +- When done, mention the `@gl\-docsteam` in the MR asking for review +- **Due date**: feature freeze date and time + +#### Documentation shipped in a follow-up MR + +If the docs aren't being shipped within the feature MR: + +- Create a new issue mentioning "docs" or "documentation" in the title (use the Documentation issue description template) +- Label the issue with: `Documentation`, `Deliverable`, `docs-P1`, `<product-label>` +(product label == CI/CD, Pages, Prometheus, etc) +- Add the correct milestone +- Create a new MR for shipping the docs changes and follow the same +process [described above](#documentation-shipped-in-the-feature-mr) +- Use the MR description template called "Documentation" +- Add the same labels and milestone as you did for the issue +- Assign the PM for review +- When done, mention the `@gl\-docsteam` in the MR asking for review +- **Due date**: feature freeze date and time + +#### Documentation shipped late + +Shipping late means that you are affecting the whole feature workflow +as well as other teams' priorities (PMs, tech writers, release managers, +release post reviewers), so every effort should be made to avoid this. + +If you did not ship the docs within the feature freeze, proceed as +[described above](#documentation-shipped-in-a-follow-up-mr) and, +besides the regular labels, include the labels `Pick into X.Y` and +`missed-deliverable` in the issue and the MR, and assign them the correct +milestone. + +The **due date** for **merging** `missed-deliverable` MRs is on the +**14th** of each month. + +### 3. Technical Writer's role in the documentation process + +- **Planning** + - Once an issue contains a Documentation label and the current milestone, a +technical writer reviews the Product Manager's documentation requirements + - Once the documentation requirements are approved, the technical writer can +work with the developer to discuss any documentation questions and plans/outlines, as needed. + +- **Review** - A technical writer must review the documentation for: + - Clarity + - Relevance (make sure the content is appropriate given the impact of the feature) + - Location (make sure the doc is in the correct dir and has the correct name) + - Syntax, typos, and broken links + - Improvements to the content + - Accordance to the [docs style guide](styleguide.md) + +<!-- TBA: issue and MR description templates as part of the process --> + +<!-- +## New features vs feature updates + +- TBA: + - Describe the difference between new features and feature updates + - Creating a new doc vs updating an existing doc +--> + diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 32de741c9fe..1cd873b6fe3 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -258,6 +258,31 @@ end [`extend ::Gitlab::Utils::Override`]: utilities.md#override +##### Overriding CE class methods + +The same applies to class methods, except we want to use +`ActiveSupport::Concern` and put `extend ::Gitlab::Utils::Override` +within the block of `class_methods`. Here's an example: + +```ruby +module EE + module Groups + module GroupMembersController + extend ActiveSupport::Concern + + class_methods do + extend ::Gitlab::Utils::Override + + override :admin_not_required_endpoints + def admin_not_required_endpoints + super.concat(%i[update override]) + end + end + end + end +end +``` + #### Use self-descriptive wrapper methods When it's not possible/logical to modify the implementation of a @@ -665,6 +690,9 @@ module EE extend ActiveSupport::Concern class_methods do + extend ::Gitlab::Utils::Override + + override :update_params_at_least_one_of def update_params_at_least_one_of super.push(*%i[ squash diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 5d1f657015c..09ea8c05be6 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -20,7 +20,40 @@ dynamic (querying the DB etc.). Once defined in `lib/feature.rb`, you will be able to activate a feature for a given feature group via the [`feature_group` param of the features API](../api/features.md#set-or-create-a-feature) +For GitLab.com, team members have access to feature flags through chatops. Only +percentage gates are supported at this time. Setting a feature to be used 50% of +the time, you should execute `/chatops run feature set my_feature_flag 50`. + ## Feature flags for user applications GitLab does not yet support the use of feature flags in deployed user applications. -You can follow the progress on that [in the issue on our issue tracker](https://gitlab.com/gitlab-org/gitlab-ee/issues/779).
\ No newline at end of file +You can follow the progress on that [in the issue on our issue tracker](https://gitlab.com/gitlab-org/gitlab-ee/issues/779). + +## Developing with feature flags + +In general, it's better to have a group- or user-based gate, and you should prefer +it over the use of percentage gates. This would make debugging easier, as you +filter for example logs and errors based on actors too. Futhermore, this allows +for enabling for the `gitlab-org` group first, while the rest of the users +aren't impacted. + +```ruby +# Good +Feature.enabled?(:feature_flag, project) + +# Avoid, if possible +Feature.enabled?(:feature_flag) +``` + +To use feature gates based on actors, the model needs to respond to +`flipper_id`. For example, to enable for the Foo model: + +```ruby +class Foo < ActiveRecord::Base + include FeatureGate +end +``` + +Features that are developed and are intended to be merged behind a feature flag +should not include a changelog entry. The entry should be added in the merge +request removing the feature flags. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index a211effdfa7..6f31e5b82e5 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -182,6 +182,34 @@ class MyMigration < ActiveRecord::Migration end ``` +## Adding foreign-key constraints + +When adding a foreign-key constraint to either an existing or new +column remember to also add a index on the column. + +This is _required_ if the foreign-key constraint specifies +`ON DELETE CASCADE` or `ON DELETE SET NULL` behavior. On a cascading +delete, the [corresponding record needs to be retrieved using an +index](https://www.cybertec-postgresql.com/en/postgresql-indexes-and-foreign-keys/) +(otherwise, we'd need to scan the whole table) for subsequent update or +deletion. + +Here's an example where we add a new column with a foreign key +constraint. Note it includes `index: true` to create an index for it. + +```ruby +class Migration < ActiveRecord::Migration + + def change + add_reference :model, :other_model, index: true, foreign_key: { on_delete: :cascade } + end +end +``` + +When adding a foreign-key constraint to an existing column, we +have to employ `add_concurrent_foreign_key` and `add_concurrent_index` +instead of `add_reference`. + ## Adding Columns With Default Values When adding columns with default values you must use the method diff --git a/doc/development/new_fe_guide/development/testing.md b/doc/development/new_fe_guide/development/testing.md index e1e13474b75..53dfe6774e9 100644 --- a/doc/development/new_fe_guide/development/testing.md +++ b/doc/development/new_fe_guide/development/testing.md @@ -133,3 +133,27 @@ afterEach(() => { vm.$destroy(); }); ``` +## Testing with older browsers + +Some regressions only affect a specific browser version. We can install and test in particular browsers with either Firefox or Browserstack using the following steps: + + +### Browserstack + +[Browserstack](https://www.browserstack.com/) allows you to test more than 1200 mobile devices and browsers. +You can use it directly through the [live app](https://www.browserstack.com/live) or you can install the [chrome extension](https://chrome.google.com/webstore/detail/browserstack/nkihdmlheodkdfojglpcjjmioefjahjb) for easy access. +You can find the credentials on 1Password, under `frontendteam@gitlab.com`. + +### Firefox + +#### macOS +You can download any older version of Firefox from the releases FTP server, https://ftp.mozilla.org/pub/firefox/releases/ + +1. From the website, select a version, in this case `50.0.1`. +2. Go to the mac folder. +3. Select your preferred language, you will find the dmg package inside, download it. +4. Drag and drop the application to any other folder but the `Applications` folder. +5. Rename the application to something like `Firefox_Old`. +6. Move the application to the `Applications` folder. +7. Open up a terminal and run `/Applications/Firefox_Old.app/Contents/MacOS/firefox-bin -profilemanager` to create a new profile specific to that Firefox version. +8. Once the profile has been created, quit the app, and run it again like normal. You now have a working older Firefox version. diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md new file mode 100644 index 00000000000..3f2843cba6e --- /dev/null +++ b/doc/development/testing_guide/smoke.md @@ -0,0 +1,16 @@ +# Smoke Tests + +It is imperative in any testing suite that we have Smoke Tests. In short, smoke tests are will run quick sanity +end-to-end functional tests from GitLab QA and are designed to run against the specified environment to ensure that +basic functionality is working. + +Currently, our suite consists of this basic functionality coverage: + +- User Login (Standard Auth) +- Project Creation +- Issue Creation +- Merge Request Creation + +--- + +[Return to Testing documentation](index.md) diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 07ced36f0c1..4403072e96f 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -120,6 +120,14 @@ running feature tests (i.e. using Capybara) against it. The actual test scenarios and steps are [part of GitLab Rails] so that they're always in-sync with the codebase. +### Smoke tests + +Smoke tests are quick tests that may be run at any time (especially after the pre-deployment migrations). + +Much like feature tests - these tests run against the UI and ensure that basic functionality is working. + +> See [Smoke Tests](smoke.md) for more information. + Read a separate document about [end-to-end tests](end_to_end_tests.md) to learn more. diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md new file mode 100644 index 00000000000..adf8795a5e3 --- /dev/null +++ b/doc/development/understanding_explain_plans.md @@ -0,0 +1,676 @@ +# Understanding EXPLAIN plans + +PostgreSQL allows you to obtain query plans using the `EXPLAIN` command. This +command can be invaluable when trying to determine how a query will perform. +You can use this command directly in your SQL query, as long as the query starts +with it: + +```sql +EXPLAIN +SELECT COUNT(*) +FROM projects +WHERE visibility_level IN (0, 20); +``` + +When running this on GitLab.com, we are presented with the following output: + +``` +Aggregate (cost=922411.76..922411.77 rows=1 width=8) + -> Seq Scan on projects (cost=0.00..908044.47 rows=5746914 width=0) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) +``` + +When using _just_ `EXPLAIN`, PostgreSQL won't actually execute our query, +instead it produces an _estimated_ execution plan based on the available +statistics. This means the actual plan can differ quite a bit. Fortunately, +PostgreSQL provides us with the option to execute the query as well. To do so, +we need to use `EXPLAIN ANALYZE` instead of just `EXPLAIN`: + +```sql +EXPLAIN ANALYZE +SELECT COUNT(*) +FROM projects +WHERE visibility_level IN (0, 20); +``` + +This will produce: + +``` +Aggregate (cost=922420.60..922420.61 rows=1 width=8) (actual time=3428.535..3428.535 rows=1 loops=1) + -> Seq Scan on projects (cost=0.00..908053.18 rows=5746969 width=0) (actual time=0.041..2987.606 rows=5746940 loops=1) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) + Rows Removed by Filter: 65677 +Planning time: 2.861 ms +Execution time: 3428.596 ms +``` + +As we can see this plan is quite different, and includes a lot more data. Let's +discuss this step by step. + +Because `EXPLAIN ANALYZE` executes the query, care should be taken when using a +query that will write data or might time out. If the query modifies data, +consider wrapping it in a transaction that rolls back automatically like so: + +```sql +BEGIN; +EXPLAIN ANALYZE +DELETE FROM users WHERE id = 1; +ROLLBACK; +``` + +The `EXPLAIN` command also takes additional options, such as `BUFFERS`: + +```sql +EXPLAIN (ANALYZE, BUFFERS) +SELECT COUNT(*) +FROM projects +WHERE visibility_level IN (0, 20); +``` + +This will then produce: + +``` +Aggregate (cost=922420.60..922420.61 rows=1 width=8) (actual time=3428.535..3428.535 rows=1 loops=1) + Buffers: shared hit=208846 + -> Seq Scan on projects (cost=0.00..908053.18 rows=5746969 width=0) (actual time=0.041..2987.606 rows=5746940 loops=1) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) + Rows Removed by Filter: 65677 + Buffers: shared hit=208846 +Planning time: 2.861 ms +Execution time: 3428.596 ms +``` + +For more information, refer to the official [EXPLAIN +documentation](https://www.postgresql.org/docs/current/static/sql-explain.html). + +## Nodes + +Every query plan consists of nodes. Nodes can be nested, and are executed from +the inside out. This means that the innermost node is executed before an outer +node. This can be best thought of as nested function calls, returning their +results as they unwind. For example, a plan starting with an `Aggregate` +followed by a `Nested Loop`, followed by an `Index Only scan` can be thought of +as the following Ruby code: + +```ruby +aggregate( + nested_loop( + index_only_scan() + index_only_scan() + ) +) +``` + +Nodes are indicated using a `->` followed by the type of node taken. For +example: + +``` +Aggregate (cost=922411.76..922411.77 rows=1 width=8) + -> Seq Scan on projects (cost=0.00..908044.47 rows=5746914 width=0) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) +``` + +Here the first node executed is `Seq scan on projects`. The `Filter:` is an +additional filter applied to the results of the node. A filter is very similar +to Ruby's `Array#select`: it takes the input rows, applies the filter, and +produces a new list of rows. Once the node is done, we perform the `Aggregate` +above it. + +Nested nodes will look like this: + +``` +Aggregate (cost=176.97..176.98 rows=1 width=8) (actual time=0.252..0.252 rows=1 loops=1) + Buffers: shared hit=155 + -> Nested Loop (cost=0.86..176.75 rows=87 width=0) (actual time=0.035..0.249 rows=36 loops=1) + Buffers: shared hit=155 + -> Index Only Scan using users_pkey on users users_1 (cost=0.43..4.95 rows=87 width=4) (actual time=0.029..0.123 rows=36 loops=1) + Index Cond: (id < 100) + Heap Fetches: 0 + -> Index Only Scan using users_pkey on users (cost=0.43..1.96 rows=1 width=4) (actual time=0.003..0.003 rows=1 loops=36) + Index Cond: (id = users_1.id) + Heap Fetches: 0 +Planning time: 2.585 ms +Execution time: 0.310 ms +``` + +Here we first perform two separate "Index Only" scans, followed by performing a +"Nested Loop" on the result of these two scans. + +## Node statistics + +Each node in a plan has a set of associated statistics, such as the cost, the +number of rows produced, the number of loops performed, and more. For example: + +``` +Seq Scan on projects (cost=0.00..908044.47 rows=5746914 width=0) +``` + +Here we can see that our cost ranges from `0.00..908044.47` (we'll cover this in +a moment), and we estimate (since we're using `EXPLAIN` and not `EXPLAIN +ANALYZE`) a total of 5,746,914 rows to be produced by this node. The `width` +statistics describes the estimated width of each row, in bytes. + +The `costs` field specifies how expensive a node was. The cost is measured in +arbitrary units determined by the query planner's cost parameters. What +influences the costs depends on a variety of settings, such as `seq_page_cost`, +`cpu_tuple_cost`, and various others. +The format of the costs field is as follows: + +``` +STARTUP COST..TOTAL COST +``` + +The startup cost states how expensive it was to start the node, with the total +cost describing how expensive the entire node was. In general: the greater the +values, the more expensive the node. + +When using `EXPLAIN ANALYZE`, these statistics will also include the actual time +(in milliseconds) spent, and other runtime statistics (e.g. the actual number of +produced rows): + +``` +Seq Scan on projects (cost=0.00..908053.18 rows=5746969 width=0) (actual time=0.041..2987.606 rows=5746940 loops=1) +``` + +Here we can see we estimated 5,746,969 rows to be returned, but in reality we +returned 5,746,940 rows. We can also see that _just_ this sequential scan took +2.98 seconds to run. + +Using `EXPLAIN (ANALYZE, BUFFERS)` will also give us information about the +number of rows removed by a filter, the number of buffers used, and more. For +example: + +``` +Seq Scan on projects (cost=0.00..908053.18 rows=5746969 width=0) (actual time=0.041..2987.606 rows=5746940 loops=1) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) + Rows Removed by Filter: 65677 + Buffers: shared hit=208846 +``` + +Here we can see that our filter has to remove 65,677 rows, and that we use +208,846 buffers. Each buffer in PostgreSQL is 8 KB (8192 bytes), meaning our +above node uses *1.6 GB of buffers*. That's a lot! + +## Node types + +There are quite a few different types of nodes, so we only cover some of the +more common ones here. + +A full list of all the available nodes and their descriptions can be found in +the [PostgreSQL source file +"plannodes.h"](https://github.com/postgres/postgres/blob/master/src/include/nodes/plannodes.h) + +### Seq Scan + +A sequential scan over (a chunk of) a database table. This is like using +`Array#each`, but on a database table. Sequential scans can be quite slow when +retrieving lots of rows, so it's best to avoid these for large tables. + +### Index Only Scan + +A scan on an index that did not require fetching anything from the table. In +certain cases an index only scan may still fetch data from the table, in this +case the node will include a `Heap Fetches:` statistic. + +### Index Scan + +A scan on an index that required retrieving some data from the table. + +### Bitmap Index Scan and Bitmap Heap scan + +Bitmap scans fall between sequential scans and index scans. These are typically +used when we would read too much data from an index scan, but too little to +perform a sequential scan. A bitmap scan uses what is known as a [bitmap +index](https://en.wikipedia.org/wiki/Bitmap_index) to perform its work. + +The [source code of PostgreSQL](https://github.com/postgres/postgres/blob/1c2cb2744bf3d8ad751cd5cf3b347f10f48492b3/src/include/nodes/plannodes.h#L446-L457) +states the following on bitmap scans: + +> Bitmap Index Scan delivers a bitmap of potential tuple locations; it does not +> access the heap itself. The bitmap is used by an ancestor Bitmap Heap Scan +> node, possibly after passing through intermediate Bitmap And and/or Bitmap Or +> nodes to combine it with the results of other Bitmap Index Scans. + +### Limit + +Applies a `LIMIT` on the input rows. + +### Sort + +Sorts the input rows as specified using an `ORDER BY` statement. + +### Nested Loop + +A nested loop will execute its child nodes for every row produced by a node that +precedes it. For example: + +``` +-> Nested Loop (cost=0.86..176.75 rows=87 width=0) (actual time=0.035..0.249 rows=36 loops=1) + Buffers: shared hit=155 + -> Index Only Scan using users_pkey on users users_1 (cost=0.43..4.95 rows=87 width=4) (actual time=0.029..0.123 rows=36 loops=1) + Index Cond: (id < 100) + Heap Fetches: 0 + -> Index Only Scan using users_pkey on users (cost=0.43..1.96 rows=1 width=4) (actual time=0.003..0.003 rows=1 loops=36) + Index Cond: (id = users_1.id) + Heap Fetches: 0 +``` + +Here the first child node (`Index Only Scan using users_pkey on users users_1`) +produces 36 rows, and is executed once (`rows=36 loops=1`). The next node +produces 1 row (`rows=1`), but is repeated 36 times (`loops=36`). This is +because the previous node produced 36 rows. + +This means that nested loops can quickly slow the query down if the various +child nodes keep producing many rows. + +## Optimising queries + +With that out of the way, let's see how we can optimise a query. Let's use the +following query as an example: + +```sql +SELECT COUNT(*) +FROM users +WHERE twitter != ''; +``` + +This query simply counts the number of users that have a Twitter profile set. +Let's run this using `EXPLAIN (ANALYZE, BUFFERS)`: + +```sql +EXPLAIN (ANALYZE, BUFFERS) +SELECT COUNT(*) +FROM users +WHERE twitter != ''; +``` + +This will produce the following plan: + +``` +Aggregate (cost=845110.21..845110.22 rows=1 width=8) (actual time=1271.157..1271.158 rows=1 loops=1) + Buffers: shared hit=202662 + -> Seq Scan on users (cost=0.00..844969.99 rows=56087 width=0) (actual time=0.019..1265.883 rows=51833 loops=1) + Filter: ((twitter)::text <> ''::text) + Rows Removed by Filter: 2487813 + Buffers: shared hit=202662 +Planning time: 0.390 ms +Execution time: 1271.180 ms +``` + +From this query plan we can see the following: + +1. We need to perform a sequential scan on the `users` table. +1. This sequential scan filters out 2,487,813 rows using a `Filter`. +1. We use 202,622 buffers, which equals 1.58 GB of memory. +1. It takes us 1.2 seconds to do all of this. + +Considering we are just counting users, that's quite expensive! + +Before we start making any changes, let's see if there are any existing indexes +on the `users` table that we might be able to use. We can obtain this +information by running `\d users` in a `psql` console, then scrolling down to +the `Indexes:` section: + +``` +Indexes: + "users_pkey" PRIMARY KEY, btree (id) + "users_confirmation_token_key" UNIQUE CONSTRAINT, btree (confirmation_token) + "users_email_key" UNIQUE CONSTRAINT, btree (email) + "users_reset_password_token_key" UNIQUE CONSTRAINT, btree (reset_password_token) + "index_on_users_lower_email" btree (lower(email::text)) + "index_on_users_lower_username" btree (lower(username::text)) + "index_on_users_name_lower" btree (lower(name::text)) + "index_users_on_admin" btree (admin) + "index_users_on_created_at" btree (created_at) + "index_users_on_email_trigram" gin (email gin_trgm_ops) + "index_users_on_feed_token" btree (feed_token) + "index_users_on_ghost" btree (ghost) + "index_users_on_incoming_email_token" btree (incoming_email_token) + "index_users_on_name" btree (name) + "index_users_on_name_trigram" gin (name gin_trgm_ops) + "index_users_on_state" btree (state) + "index_users_on_state_and_internal_attrs" btree (state) WHERE ghost <> true AND support_bot <> true + "index_users_on_support_bot" btree (support_bot) + "index_users_on_username" btree (username) + "index_users_on_username_trigram" gin (username gin_trgm_ops) +``` + +Here we can see there is no index on the `twitter` column, which means +PostgreSQL has to perform a sequential scan in this case. Let's try to fix this +by adding the following index: + +```sql +CREATE INDEX CONCURRENTLY twitter_test ON users (twitter); +``` + +If we now re-run our query using `EXPLAIN (ANALYZE, BUFFERS)` we get the +following plan: + +``` +Aggregate (cost=61002.82..61002.83 rows=1 width=8) (actual time=297.311..297.312 rows=1 loops=1) + Buffers: shared hit=51854 dirtied=19 + -> Index Only Scan using twitter_test on users (cost=0.43..60873.13 rows=51877 width=0) (actual time=279.184..293.532 rows=51833 loops=1) + Filter: ((twitter)::text <> ''::text) + Rows Removed by Filter: 2487830 + Heap Fetches: 26037 + Buffers: shared hit=51854 dirtied=19 +Planning time: 0.191 ms +Execution time: 297.334 ms +``` + +Now it takes just under 300 milliseconds to get our data, instead of 1.2 +seconds. However, we still use 51,854 buffers, which is about 400 MB of memory. +300 milliseconds is also quite slow for such a simple query. To understand why +this query is still expensive, let's take a look at the following: + +``` +Index Only Scan using twitter_test on users (cost=0.43..60873.13 rows=51877 width=0) (actual time=279.184..293.532 rows=51833 loops=1) + Filter: ((twitter)::text <> ''::text) + Rows Removed by Filter: 2487830 +``` + +We start with an index only scan on our index, but we somehow still apply a +`Filter` that filters out 2,487,830 rows. Why is that? Well, let's look at how +we created the index: + +```sql +CREATE INDEX CONCURRENTLY twitter_test ON users (twitter); +``` + +We simply told PostgreSQL to index all possible values of the `twitter` column, +even empty strings. Our query in turn uses `WHERE twitter != ''`. This means +that the index does improve things, as we don't need to do a sequential scan, +but we may still encounter empty strings. This means PostgreSQL _has_ to apply a +Filter on the index results to get rid of those values. + +Fortunately, we can improve this even further using "partial indexes". Partial +indexes are indexes with a `WHERE` condition that is applied when indexing data. +For example: + +```sql +CREATE INDEX CONCURRENTLY some_index ON users (email) WHERE id < 100 +``` + +This index would only index the `email` value of rows that match `WHERE id < +100`. We can use partial indexes to change our Twitter index to the following: + +```sql +CREATE INDEX CONCURRENTLY twitter_test ON users (twitter) WHERE twitter != ''; +``` + +Once created, if we run our query again we will be given the following plan: + +``` +Aggregate (cost=1608.26..1608.27 rows=1 width=8) (actual time=19.821..19.821 rows=1 loops=1) + Buffers: shared hit=44036 + -> Index Only Scan using twitter_test on users (cost=0.41..1479.71 rows=51420 width=0) (actual time=0.023..15.514 rows=51833 loops=1) + Heap Fetches: 1208 + Buffers: shared hit=44036 +Planning time: 0.123 ms +Execution time: 19.848 ms +``` + +That's _a lot_ better! Now it only takes 20 milliseconds to get the data, and we +only use about 344 MB of buffers (instead of the original 1.58 GB). The reason +this works is that now PostgreSQL no longer needs to apply a `Filter`, as the +index only contains `twitter` values that are not empty. + +Keep in mind that you shouldn't just add partial indexes every time you want to +optimise a query. Every index has to be updated for every write, and they may +require quite a bit of space, depending on the amount of indexed data. As a +result, first check if there are any existing indexes you may be able to reuse. +If there aren't any, check if you can perhaps slightly change an existing one to +fit both the existing and new queries. Only add a new index if none of the +existing indexes can be used in any way. + +## Queries that can't be optimised + +Now that we have seen how to optimise a query, let's look at another query that +we might not be able to optimise: + +```sql +EXPLAIN (ANALYZE, BUFFERS) +SELECT COUNT(*) +FROM projects +WHERE visibility_level IN (0, 20); +``` + +The output of `EXPLAIN (ANALYZE, BUFFERS)` is as follows: + +``` +Aggregate (cost=922420.60..922420.61 rows=1 width=8) (actual time=3428.535..3428.535 rows=1 loops=1) + Buffers: shared hit=208846 + -> Seq Scan on projects (cost=0.00..908053.18 rows=5746969 width=0) (actual time=0.041..2987.606 rows=5746940 loops=1) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) + Rows Removed by Filter: 65677 + Buffers: shared hit=208846 +Planning time: 2.861 ms +Execution time: 3428.596 ms +``` + +Looking at the output we see the following Filter: + +``` +Filter: (visibility_level = ANY ('{0,20}'::integer[])) +Rows Removed by Filter: 65677 +``` + +Looking at the number of rows removed by the filter, we may be tempted to add an +index on `projects.visibility_level` to somehow turn this Sequential scan + +filter into an index-only scan. + +Unfortunately, doing so is unlikely to improve anything. Contrary to what some +might believe, an index being present _does not guarantee_ that PostgreSQL will +actually use it. For example, when doing a `SELECT * FROM projects` it is much +cheaper to just scan the entire table, instead of using an index and then +fetching data from the table. In such cases PostgreSQL may decide to not use an +index. + +Second, let's think for a moment what our query does: it gets all projects with +visibility level 0 or 20. In the above plan we can see this produces quite a lot +of rows (5,745,940), but how much is that relative to the total? Let's find out +by running the following query: + +```sql +SELECT visibility_level, count(*) AS amount +FROM projects +GROUP BY visibility_level +ORDER BY visibility_level ASC; +``` + +For GitLab.com this produces: + +``` + visibility_level | amount +------------------+--------- + 0 | 5071325 + 10 | 65678 + 20 | 674801 +``` + +Here the total number of projects is 5,811,804, and 5,746,126 of those are of +level 0 or 20. That's 98% of the entire table! + +So no matter what we do, this query will retrieve 98% of the entire table. Since +most time is spent doing exactly that, there isn't really much we can do to +improve this query, other than _not_ running it at all. + +What is important here is that while some may recommend to straight up add an +index the moment you see a sequential scan, it is _much more important_ to first +understand what your query does, how much data it retrieves, and so on. After +all, you can not optimise something you do not understand. + +### Cardinality and selectivity + +Earlier we saw that our query had to retrieve 98% of the rows in the table. +There are two terms commonly used for databases: cardinality, and selectivity. +Cardinality refers to the number of unique values in a particular column in a +table. + +Selectivity is the number of unique values produced by an operation (e.g. an +index scan or filter), relative to the total number of rows. The higher the +selectivity, the more likely PostgreSQL is able to use an index. + +In the above example, there are only 3 unique values: 0, 10, and 20. This means +the cardinality is 3. The selectivity in turn is also very low: 0.0000003% (2 / +5,811,804), because our `Filter` only filters using two values (`0` and `20`). +With such a low selectivity value it's not surprising that PostgreSQL decides +using an index is not worth it, because it would produce almost no unique rows. + +## Rewriting queries + +So the above query can't really be optimised as-is, or at least not much. But +what if we slightly change the purpose of it? What if instead of retrieving all +projects with `visibility_level` 0 or 20, we retrieve those that a user +interacted with somehow? + +Fortunately, GitLab has an answer for this, and it's a table called +`user_interacted_projects`. This table has the following schema: + +``` +Table "public.user_interacted_projects" + Column | Type | Modifiers +------------+---------+----------- + user_id | integer | not null + project_id | integer | not null +Indexes: + "index_user_interacted_projects_on_project_id_and_user_id" UNIQUE, btree (project_id, user_id) + "index_user_interacted_projects_on_user_id" btree (user_id) +Foreign-key constraints: + "fk_rails_0894651f08" FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE + "fk_rails_722ceba4f7" FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE +``` + +Let's rewrite our query to JOIN this table onto our projects, and get the +projects for a specific user: + +```sql +EXPLAIN ANALYZE +SELECT COUNT(*) +FROM projects +INNER JOIN user_interacted_projects ON user_interacted_projects.project_id = projects.id +WHERE projects.visibility_level IN (0, 20) +AND user_interacted_projects.user_id = 1; +``` + +What we do here is the following: + +1. Get our projects. +1. INNER JOIN `user_interacted_projects`, meaning we're only left with rows in + `projects` that have a corresponding row in `user_interacted_projects`. +1. Limit this to the projects with `visibility_level` of 0 or 20, and to + projects that the user with ID 1 interacted with. + +If we run this query we get the following plan: + +``` + Aggregate (cost=871.03..871.04 rows=1 width=8) (actual time=9.763..9.763 rows=1 loops=1) + -> Nested Loop (cost=0.86..870.52 rows=203 width=0) (actual time=1.072..9.748 rows=143 loops=1) + -> Index Scan using index_user_interacted_projects_on_user_id on user_interacted_projects (cost=0.43..160.71 rows=205 width=4) (actual time=0.939..2.508 rows=145 loops=1) + Index Cond: (user_id = 1) + -> Index Scan using projects_pkey on projects (cost=0.43..3.45 rows=1 width=4) (actual time=0.049..0.050 rows=1 loops=145) + Index Cond: (id = user_interacted_projects.project_id) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) + Rows Removed by Filter: 0 + Planning time: 2.614 ms + Execution time: 9.809 ms +``` + +Here it only took us just under 10 milliseconds to get the data. We can also see +we're retrieving far fewer projects: + +``` +Index Scan using projects_pkey on projects (cost=0.43..3.45 rows=1 width=4) (actual time=0.049..0.050 rows=1 loops=145) + Index Cond: (id = user_interacted_projects.project_id) + Filter: (visibility_level = ANY ('{0,20}'::integer[])) + Rows Removed by Filter: 0 +``` + +Here we see we perform 145 loops (`loops=145`), with every loop producing 1 row +(`rows=1`). This is much less than before, and our query performs much better! + +If we look at the plan we also see our costs are very low: + +``` +Index Scan using projects_pkey on projects (cost=0.43..3.45 rows=1 width=4) (actual time=0.049..0.050 rows=1 loops=145) +``` + +Here our cost is only 3.45, and it only takes us 0.050 milliseconds to do so. +The next index scan is a bit more expensive: + +``` +Index Scan using index_user_interacted_projects_on_user_id on user_interacted_projects (cost=0.43..160.71 rows=205 width=4) (actual time=0.939..2.508 rows=145 loops=1) +``` + +Here the cost is 160.71 (`cost=0.43..160.71`), taking about 2.5 milliseconds +(based on the output of `actual time=....`). + +The most expensive part here is the "Nested Loop" that acts upon the result of +these two index scans: + +``` +Nested Loop (cost=0.86..870.52 rows=203 width=0) (actual time=1.072..9.748 rows=143 loops=1) +``` + +Here we had to perform 870.52 disk page fetches for 203 rows, 9.748 +milliseconds, producing 143 rows in a single loop. + +The key takeaway here is that sometimes you have to rewrite (parts of) a query +to make it better. Sometimes that means having to slightly change your feature +to accommodate for better performance. + +## What makes a bad plan + +This is a bit of a difficult question to answer, because the definition of "bad" +is relative to the problem you are trying to solve. However, some patterns are +best avoided in most cases, such as: + +* Sequential scans on large tables +* Filters that remove a lot of rows +* Performing a certain step (e.g. an index scan) that requires _a lot_ of + buffers (e.g. more than 512 MB for GitLab.com). + +As a general guideline, aim for a query that: + +1. Takes no more than 10 milliseconds. Our target time spent in SQL per request + is around 100 milliseconds, so every query should be as fast as possible. +1. Does not use an excessive number of buffers, relative to the workload. For + example, retrieving ten rows shouldn't require 1 GB of buffers. +1. Does not spend a long amount of time performing disk IO operations. The + setting `track_io_timing` must be enabled for this data to be included in the + output of `EXPLAIN ANALYZE`. +1. Applies a `LIMIT` when retrieving rows without aggregating them, such as + `SELECT * FROM users`. +1. Doesn't use a `Filter` to filter out too many rows, especially if the query + does not use a `LIMIT` to limit the number of returned rows. Filters can + usually be removed by adding a (partial) index. + +These are _guidelines_ and not hard requirements, as different needs may require +different queries. The only _rule_ is that you _must always measure_ your query +(preferably using a production-like database) using `EXPLAIN (ANALYZE, BUFFERS)` +and related tools such as: + +* <https://explain.depesz.com/> +* <http://tatiyants.com/postgres-query-plan-visualization/> + +GitLab employees can also use our chatops solution, available in Slack using the +`/chatops` slash command. You can use chatops to get a query plan by running the +following: + +``` +/chatops run explain SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) +``` + +Visualising the plan using <https://explain.depesz.com/> is also supported: + +``` +/chatops run explain --visual SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) +``` + +Quoting the query is not necessary. + +For more information about the available options, run: + +``` +/chatops run explain --help +``` diff --git a/doc/install/installation.md b/doc/install/installation.md index ea01d88d85f..a310f12b29e 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -153,7 +153,7 @@ page](https://golang.org/dl). # Remove former Go installation folder sudo rm -rf /usr/local/go - + curl --remote-name --progress https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz echo 'fa1b0e45d3b647c252f51f5e1204aba049cde4af177ef9f2181f43004f901035 go1.10.3.linux-amd64.tar.gz' | shasum -a256 -c - && \ sudo tar -C /usr/local -xzf go1.10.3.linux-amd64.tar.gz @@ -494,11 +494,11 @@ Make GitLab start on boot: ### Install Gitaly # Fetch Gitaly source with Git and compile with Go - sudo -u git -H bundle exec rake "gitlab:gitaly:install[/home/git/gitaly]" RAILS_ENV=production + sudo -u git -H bundle exec rake "gitlab:gitaly:install[/home/git/gitaly,/home/git/repositories]" RAILS_ENV=production You can specify a different Git repository by providing it as an extra parameter: - sudo -u git -H bundle exec rake "gitlab:gitaly:install[/home/git/gitaly,https://example.com/gitaly.git]" RAILS_ENV=production + sudo -u git -H bundle exec rake "gitlab:gitaly:install[/home/git/gitaly,/home/git/repositories,https://example.com/gitaly.git]" RAILS_ENV=production Next, make sure gitaly configured: diff --git a/doc/install/kubernetes/gitlab_omnibus.md b/doc/install/kubernetes/gitlab_omnibus.md index 9aee6b9dc74..c2c8a7a92fd 100644 --- a/doc/install/kubernetes/gitlab_omnibus.md +++ b/doc/install/kubernetes/gitlab_omnibus.md @@ -120,7 +120,7 @@ gitlabConfigStorageSize: 1Gi Ingress routing and SSL are automatically configured within this Chart. An NGINX ingress is provisioned and configured, and will route traffic to any service. SSL certificates are automatically created and configured by [kube-lego](https://github.com/kubernetes/charts/tree/master/stable/kube-lego). > **Note:** -Let's Encrypt limits a single TLD to five certificate requests within a single week. This means that common DNS wildcard services like [nip.io](http://nip.io) and [nip.io](http://nip.io) are unlikely to work. +Let's Encrypt limits a single TLD to five certificate requests within a single week. This means that common DNS wildcard services like [nip.io](http://nip.io) are unlikely to work. ## Installing GitLab using the Helm Chart > **Note:** diff --git a/doc/integration/README.md b/doc/integration/README.md index 54e78bdef54..8a93d4cb84b 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -30,7 +30,7 @@ Bitbucket.org account ## Project services -Integration with services such as Campfire, Flowdock, Gemnasium, HipChat, +Integration with services such as Campfire, Flowdock, HipChat, Pivotal Tracker, and Slack are available in the form of a [Project Service][]. [Project Service]: ../user/project/integrations/project_services.md diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index f5574506595..0474182e324 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -233,6 +233,11 @@ in **Admin Area > Settings > Continuous Integration and Deployment**. Doing that all the projects that haven't explicitly set an option will have Auto DevOps enabled by default. +NOTE: **Note:** +There is also a feature flag to enable Auto DevOps to a percentage of projects +which can be enabled from the console with +`Feature.get(:force_autodevops_on_by_default).enable_percentage_of_actors(10)`. + ### Deployment strategy > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/38542) in GitLab 11.0. diff --git a/doc/user/admin_area/img/cohorts.png b/doc/user/admin_area/img/cohorts.png Binary files differdeleted file mode 100644 index 8bae7faff07..00000000000 --- a/doc/user/admin_area/img/cohorts.png +++ /dev/null diff --git a/doc/user/admin_area/monitoring/convdev.md b/doc/user/admin_area/monitoring/convdev.md index a98602c4d70..6ad8a5a7ff0 100644 --- a/doc/user/admin_area/monitoring/convdev.md +++ b/doc/user/admin_area/monitoring/convdev.md @@ -1,29 +1,5 @@ -# Conversational Development Index +--- +redirect_to: '../../instance_statistics/convdev.md' +--- -> [Introduced][ce-30469] in GitLab 9.3. - -Conversational Development Index (ConvDev) gives you an overview of your entire -instance's feature usage, from idea to production. It looks at your usage in the -past 30 days, averaged over the number of active users in that time period. It also -provides a lead score per feature, which is calculated based on GitLab's analysis -of top performing instances, based on [usage ping data][ping] that GitLab has -collected. Your score is compared to the lead score, expressed as a percentage. -The overall index score is an average over all your feature scores. - - - -The page also provides helpful links to articles and GitLab docs, to help you -improve your scores. - -Your GitLab instance's usage ping must be activated in order to use this feature. -Usage ping data is aggregated on GitLab's servers for analysis. Your usage -information is **not sent** to any other GitLab instances. - -If you have just started using GitLab, it may take a few weeks for data to be -collected before this feature is available. - -This feature is accessible only to a system admin, at -**Admin area > Overview > ConvDev Index**. - -[ce-30469]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30469 -[ping]: ../settings/usage_statistics.md#usage-ping +This document was moved to [another location](../../instance_statistics/convdev.md). diff --git a/doc/user/admin_area/monitoring/img/convdev_index.png b/doc/user/admin_area/monitoring/img/convdev_index.png Binary files differdeleted file mode 100644 index 1bf1d6a83c9..00000000000 --- a/doc/user/admin_area/monitoring/img/convdev_index.png +++ /dev/null diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index eb6f915f3f4..76d9a4ceb03 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -21,8 +21,9 @@ that this setting is set for each job. The default expiration time of the [job artifacts][art-yml] can be set in the Admin area of your GitLab instance. The syntax of duration is described in [artifacts:expire_in][duration-syntax]. The default is `30 days`. Note that -this setting is set for each job. Set it to 0 if you don't want default -expiration. +this setting is set for each job. Set it to `0` if you don't want default +expiration. The default unit is in seconds. + 1. Go to **Admin area > Settings** (`/admin/application_settings`). diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index 381efdf5d67..b7427592e10 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -6,7 +6,7 @@ to perform various actions. All statistics are opt-out, you can enable/disable them from the admin panel under **Admin area > Settings > Usage statistics**. -## Version check +## Version check **[CORE ONLY]** If enabled, version check will inform you if a new version is available and the importance of it through a status. This is shown on the help page (i.e. `/help`) @@ -29,7 +29,7 @@ secure. If you disable version check, this information will not be collected. Enable or disable the version check at **Admin area > Settings > Usage statistics**. -## Usage ping +## Usage ping **[CORE ONLY]** > [Introduced][ee-557] in GitLab Enterprise Edition 8.10. More statistics [were added][ee-735] in GitLab Enterprise Edition @@ -67,6 +67,15 @@ production: &base usage_ping_enabled: false ``` +## Instance statistics visibility **[CORE ONLY]** + +Once usage ping is enabled, GitLab will gather data from other instances and +will be able to show [usage statistics](../../instance_statistics/index.md) +of your instance to your users. + +This can be restricted to admins by selecting "Only admins" in the Instance +Statistics visibility section under **Admin area > Settings > Usage statistics**. + [ee-557]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/557 [ee-735]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/735 [ce-23361]: https://gitlab.com/gitlab-org/gitlab-ce/issues/23361 diff --git a/doc/user/admin_area/user_cohorts.md b/doc/user/admin_area/user_cohorts.md index e25e7a8bbc3..21e61e2ec44 100644 --- a/doc/user/admin_area/user_cohorts.md +++ b/doc/user/admin_area/user_cohorts.md @@ -1,37 +1,5 @@ -# Cohorts +--- +redirect_to: '../instance_statistics/user_cohorts.md' +--- -> **Notes:** -> [Introduced][ce-23361] in GitLab 9.1. - -As a benefit of having the [usage ping active](settings/usage_statistics.md), -GitLab lets you analyze the users' activities of your GitLab installation. -Under `/admin/cohorts`, when the usage ping is active, GitLab will show the -monthly cohorts of new users and their activities over time. - -## Overview - -How do we read the user cohorts table? Let's take an example with the following -user cohorts. - - - -For the cohort of June 2016, 163 users have been added on this server and have -been active since this month. One month later, in July 2016, out of -these 163 users, 155 users (or 95% of the June cohort) are still active. Two -months later, 139 users (or 85%) are still active. 9 months later, we can see -that only 6% of this cohort are still active. - -The Inactive users column shows the number of users who have been added during -the month, but who have never actually had any activity in the instance. - -How do we measure the activity of users? GitLab considers a user active if: - -* the user signs in -* the user has Git activity (whether push or pull). - -## Setup - -1. [Activate the usage ping](settings/usage_statistics.md) -2. Go to `/admin/cohorts` to see the user cohorts of the server - -[ce-23361]: https://gitlab.com/gitlab-org/gitlab-ce/issues/23361 +This document was moved to [another location](../instance_statistics/user_cohorts.md). diff --git a/doc/user/index.md b/doc/user/index.md index 90f0e2285c3..649c0b664a5 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -172,3 +172,7 @@ Automate GitLab via [API](../api/README.md). ## Git and GitLab Learn what is [Git](../topics/git/index.md) and its best practices. + +## Instance statistics + +See [various statistics](instance_statistics/index.md) of your GitLab instance. diff --git a/doc/user/instance_statistics/convdev.md b/doc/user/instance_statistics/convdev.md new file mode 100644 index 00000000000..d2795e092fc --- /dev/null +++ b/doc/user/instance_statistics/convdev.md @@ -0,0 +1,26 @@ +# Conversational Development Index + +> [Introduced][ce-30469] in GitLab 9.3. + +Conversational Development Index (ConvDev) gives you an overview of your entire +instance's feature usage, from idea to production. It looks at your usage in the +past 30 days, averaged over the number of active users in that time period. It also +provides a lead score per feature, which is calculated based on GitLab's analysis +of top performing instances, based on [usage ping data][ping] that GitLab has +collected. Your score is compared to the lead score, expressed as a percentage. +The overall index score is an average over all your feature scores. + + + +The page also provides helpful links to articles and GitLab docs, to help you +improve your scores. + +Your GitLab instance's usage ping must be activated in order to use this feature. +Usage ping data is aggregated on GitLab's servers for analysis. Your usage +information is **not sent** to any other GitLab instances. + +If you have just started using GitLab, it may take a few weeks for data to be +collected before this feature is available. + +[ce-30469]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30469 +[ping]: ../admin_area/settings/usage_statistics.md#usage-ping diff --git a/doc/user/instance_statistics/img/cohorts.png b/doc/user/instance_statistics/img/cohorts.png Binary files differnew file mode 100644 index 00000000000..12e839e7cd2 --- /dev/null +++ b/doc/user/instance_statistics/img/cohorts.png diff --git a/doc/user/instance_statistics/img/convdev_index.png b/doc/user/instance_statistics/img/convdev_index.png Binary files differnew file mode 100644 index 00000000000..191295c918b --- /dev/null +++ b/doc/user/instance_statistics/img/convdev_index.png diff --git a/doc/user/instance_statistics/img/instance_statistics_button.png b/doc/user/instance_statistics/img/instance_statistics_button.png Binary files differnew file mode 100644 index 00000000000..6104321b1a6 --- /dev/null +++ b/doc/user/instance_statistics/img/instance_statistics_button.png diff --git a/doc/user/instance_statistics/index.md b/doc/user/instance_statistics/index.md new file mode 100644 index 00000000000..a4eca89b7fe --- /dev/null +++ b/doc/user/instance_statistics/index.md @@ -0,0 +1,19 @@ +# Instance statistics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41416) +in GitLab 11.2. + +Instance statistics gives users or admins access to instance-wide analytics. +They are accessible to all users by default (GitLab admins can restrict its +visibility in the [admin area](../admin_area/settings/usage_statistics.md)), +and can be accessed via the top bar. + + + +For the statistics to show up, [usage ping must be enabled](../admin_area/settings/usage_statistics.md#usage-ping) +by an admin in the admin settings area. + +There are two kinds of statistics: + +- [Conversational Development (ConvDev) Index](convdev.md): Provides an overview of your entire instance's feature usage. +- [User Cohorts](user_cohorts.md): Display the monthly cohorts of new users and their activities over time. diff --git a/doc/user/instance_statistics/user_cohorts.md b/doc/user/instance_statistics/user_cohorts.md new file mode 100644 index 00000000000..70d5912dc4e --- /dev/null +++ b/doc/user/instance_statistics/user_cohorts.md @@ -0,0 +1,27 @@ +# Cohorts + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/23361) +in GitLab 9.1. + +As a benefit of having the [usage ping active](../admin_area/settings/usage_statistics.md), +GitLab lets you analyze the users' activities over time of your GitLab installation. + +## Overview + +How do we read the user cohorts table? Let's take an example with the following +user cohorts. + + + +For the cohort of Jan 2018, 15 users have been added on this server and have +been active since this month. One month later, in Feb 2018, all 15 users are +still active. 6 months later (Month 6, July), we can see 10 users from this cohort +are active, or 66% of the original cohort of 15 that joined in January. + +The Inactive users column shows the number of users who have been added during +the month, but who have never actually had any activity in the instance. + +How do we measure the activity of users? GitLab considers a user active if: + +* the user signs in +* the user has Git activity (whether push or pull). diff --git a/doc/user/markdown.md b/doc/user/markdown.md index bd199b55a61..6203561265b 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -234,6 +234,13 @@ https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: + Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. + + On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) to get full native emoji support. + + Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. + + Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: :zap: You can use emoji anywhere GFM is supported. :v: @@ -244,6 +251,14 @@ If you are new to this, don't be :fearful:. You can easily join the emoji :famil Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: +Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. + +On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) to get full native emoji support. + +Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. + + + ### Special GitLab References GFM recognizes special references. @@ -259,6 +274,7 @@ GFM will recognize the following: | `@user_name` | specific user | | `@group_name` | specific group | | `@all` | entire team | +| `namespace/project>` | project | | `#12345` | issue | | `!123` | merge request | | `$123` | snippet | diff --git a/doc/user/project/img/issue_board.png b/doc/user/project/img/issue_board.png Binary files differindex 50e051e25a0..925b969eebe 100644 --- a/doc/user/project/img/issue_board.png +++ b/doc/user/project/img/issue_board.png diff --git a/doc/user/project/import/img/manifest_upload.png b/doc/user/project/import/img/manifest_upload.png Binary files differdeleted file mode 100644 index d6bf4b157dd..00000000000 --- a/doc/user/project/import/img/manifest_upload.png +++ /dev/null diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index b55435e5b4f..4ea35a30bbf 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -11,7 +11,7 @@ 1. [From SVN](svn.md) 1. [From TFS](tfs.md) 1. [From repo by URL](repo_by_url.md) -1. [By uploading a manifest file](manifest.md) +1. [By uploading a manifest file (AOSP)](manifest.md) In addition to the specific migration documentation above, you can import any Git repository via HTTP from the New Project page. Be aware that if the diff --git a/doc/user/project/import/manifest.md b/doc/user/project/import/manifest.md index 06171f11e12..24bf6541a9d 100644 --- a/doc/user/project/import/manifest.md +++ b/doc/user/project/import/manifest.md @@ -1,36 +1,31 @@ # Import multiple repositories by uploading a manifest file -GitLab allows you to import all the required git repositories -based a manifest file like the one used by the [Android repository](https://android.googlesource.com/platform/manifest/+/2d6f081a3b05d8ef7a2b1b52b0d536b2b74feab4/default.xml). -This feature can be very handy when you need to import a project with many repositories like Android Open Source Project (AOSP). +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28811) in +GitLab 11.2. +GitLab allows you to import all the required Git repositories +based on a manifest file like the one used by the +[Android repository](https://android.googlesource.com/platform/manifest/+/2d6f081a3b05d8ef7a2b1b52b0d536b2b74feab4/default.xml). +This feature can be very handy when you need to import a project with many +repositories like the Android Open Source Project (AOSP). ->**Note:** -This feature requires [subgroups](../../group/subgroups/index.md) to be supported by your database. +## Requirements -You can do it by following next steps: +GitLab must be using PostgreSQL for its database, since +[subgroups](../../group/subgroups/index.md) are needed for the manifest import +to work. -1. From your GitLab dashboard click **New project** -1. Switch to the **Import project** tab -1. Click on the **Manifest file** button -1. Provide GitLab with a manifest xml file -1. Select a group you want to import to (you need to create a group first if you don't have one) -1. Click **List available repositories** -1. You will be redirected to the import status page with projects list based on manifest file -1. Check the list and click 'Import all repositories' to start import. - - +Read more about the [database requirements](../../../install/requirements.md#database). - +## Manifest format -### Manifest format +A manifest must be an XML file. There must be one `remote` tag with a `review` +attribute that contains a URL to a Git server, and each `project` tag must have +a `name` and `path` attribute. GitLab will then build the URL to the repository +by combining the URL from the `remote` tag with a project name. +A path attribute will be used to represent the project path in GitLab. -A manifest must be an XML file. There must be one `remote` tag with `review` attribute -that contains a URL to a git server. Each `project` tag must have `name` and `path` attribute. -GitLab will build URL to the repository by combining URL from `remote` tag with a project name. -A path attribute will be used to represent project path in GitLab system. - -Below is a valid example of manifest file. +Below is a valid example of a manifest file: ```xml <manifest> @@ -41,9 +36,24 @@ Below is a valid example of manifest file. </manifest> ``` -As result next projects will be created: +As a result, the following projects will be created: | GitLab | Import URL | |---|---| -| https://gitlab/YOUR_GROUP/build/make | https://android-review.googlesource.com/platform/build | -| https://gitlab/YOUR_GROUP/build/blueprint | https://android-review.googlesource.com/platform/build/blueprint | +| https://gitlab.com/YOUR_GROUP/build/make | https://android-review.googlesource.com/platform/build | +| https://gitlab.com/YOUR_GROUP/build/blueprint | https://android-review.googlesource.com/platform/build/blueprint | + +## Importing the repositories + +You can start the import with: + +1. From your GitLab dashboard click **New project** +1. Switch to the **Import project** tab +1. Click on the **Manifest file** button +1. Provide GitLab with a manifest xml file +1. Select a group you want to import to (you need to create a group first if you don't have one) +1. Click **List available repositories**. At this point, you will be redirected + to the import status page with projects list based on the manifest file. +1. Check the list and click **Import all repositories** to start the import. + +  diff --git a/doc/user/project/integrations/hangouts_chat.md b/doc/user/project/integrations/hangouts_chat.md index 6ab44420a10..47525617d95 100644 --- a/doc/user/project/integrations/hangouts_chat.md +++ b/doc/user/project/integrations/hangouts_chat.md @@ -1,5 +1,7 @@ # Hangouts Chat service +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/43756) in GitLab 11.2. + The Hangouts Chat service sends notifications from GitLab to the room for which the webhook was created. ## On Hangouts Chat diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index 05ee1b4e6d7..efb0381d7aa 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -34,7 +34,6 @@ Click on the service links to see further configuration instructions and details | [Emails on push](emails_on_push.md) | Email the commits and diff of each push to a list of recipients | | External Wiki | Replaces the link to the internal wiki with a link to an external wiki | | Flowdock | Flowdock is a collaboration web app for technical teams | -| Gemnasium _(Has been deprecated in GitLab 11.0)_ | Gemnasium monitors your project dependencies and alerts you about updates and security vulnerabilities | | [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | | [HipChat](hipchat.md) | Private group chat and IM | | [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 8e486318980..770b1810da1 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -1102,6 +1102,7 @@ X-Gitlab-Event: Build Hook "build_finished_at": null, "build_duration": null, "build_allow_failure": false, + "build_failure_reason": "script_failure", "project_id": 380, "project_name": "gitlab-org/gitlab-test", "user": { @@ -1122,7 +1123,6 @@ X-Gitlab-Event: Build Hook }, "repository": { "name": "gitlab_test", - "git_ssh_url": "git@192.168.64.1:gitlab-org/gitlab-test.git", "description": "Atque in sunt eos similique dolores voluptatem.", "homepage": "http://192.168.64.1:3005/gitlab-org/gitlab-test", "git_ssh_url": "git@192.168.64.1:gitlab-org/gitlab-test.git", diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 49b49271cff..0e847be79c2 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -119,10 +119,10 @@ Issue Board, that is, create or delete lists and drag issues from one list to an ## Issue Board terminology - **Issue Board** - Each board represents a unique view for your issues. It can have multiple lists with each list consisting of issues represented by cards. -- **List** - A column on the issue board that displays issues matching certain attributes. In addition to the default lists of 'Backlog' and 'Closed' issue, each additional list will show issues matching your chosen label or assignee. On the top of that list you can see the number of issues that belong to it. +- **List** - A column on the issue board that displays issues matching certain attributes. In addition to the default lists of 'Open' and 'Closed' issue, each additional list will show issues matching your chosen label or assignee. On the top of that list you can see the number of issues that belong to it. - **Label list**: a list based on a label. It shows all opened issues with that label. - **Assignee list**: a list which includes all issues assigned to a user. - - **Backlog** (default): shows all open issues that do not belong to one of the other lists. Always appears as the leftmost list. + - **Open** (default): shows all open issues that do not belong to one of the other lists. Always appears as the leftmost list. - **Closed** (default): shows all closed issues. Always appears as the rightmost list. - **Card** - A box in the list that represents an individual issue. The information you can see on a card consists of the issue number, the issue title, the assignee, and the labels associated with the issue. You can drag cards from one list to another to change their label or assignee from that of the source list to that of the destination list. @@ -353,9 +353,9 @@ To remove an assignee list, just as with a label list, click the trash icon. When dragging issues between lists, different behavior occurs depending on the source list and the target list. -| | To Backlog | To Closed | To label `B` list | To assignee `Bob` list | +| | To Open | To Closed | To label `B` list | To assignee `Bob` list | | --- | --- | --- | --- | --- | -| From Backlog | - | Issue closed | `B` added | `Bob` assigned | +| From Open | - | Issue closed | `B` added | `Bob` assigned | | From Closed | Issue reopened | - | Issue reopened<br/>`B` added | Issue reopened<br/>`Bob` assigned | | From label `A` list | `A` removed | Issue closed | `A` removed<br/>`B` added | `Bob` assigned | | From assignee `Alice` list | `Alice` unassigned | Issue closed | `B` added | `Alice` unassigned<br/>`Bob` assigned | diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 86ecf33ed31..43ca498d006 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -43,8 +43,7 @@ A. Consider you are a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team -1. You work on the implementation optimizing code with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** -1. You build and test your changes with GitLab CI/CD +1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD 1. You request the approval from your manager 1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#manual-actions) for GitLab CI/CD diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index 61af1d2ab27..e1eede8bbed 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -1,5 +1,5 @@ --- -last_updated: 2018-02-16 +last_updated: 2018-08-16 author: Marcia Ramos author_gitlab: marcia level: beginner @@ -28,7 +28,7 @@ Let's start from the beginning with [DNS records](#dns-records). If you already know how they work and want to skip the introduction to DNS, you may be interested in skipping it until the [TL;DR](#tl-dr) section below. -## DNS Records +### DNS Records A Domain Name System (DNS) web service routes visitors to websites by translating domain names (such as `www.example.com`) into the @@ -64,22 +64,28 @@ for the most popular hosting services: If your hosting service is not listed above, you can just try to search the web for `how to add dns record on <my hosting service>`. -### DNS A record +#### DNS A record In case you want to point a root domain (`example.com`) to your GitLab Pages site, deployed to `namespace.gitlab.io`, you need to log into your domain's admin control panel and add a DNS `A` record pointing your domain to Pages' server IP address. For projects on -GitLab.com, this IP is `52.167.214.135`. For projects living in +GitLab.com, this IP is `35.185.44.232`. For projects living in other GitLab instances (CE or EE), please contact your sysadmin asking for this information (which IP address is Pages server running on your instance). **Practical Example:** - + -### DNS CNAME record +NOTE: **Note:** +Note that if you use your root domain for your GitLab Pages website **only**, and if +your domain registrar supports this feature, you can add a DNS apex `CNAME` +record instead of an `A` record. The main advantage of doing so is that when GitLab Pages +IP on GitLab.com changes for whatever reason, you don't need to update your `A` record. + +#### DNS CNAME record In case you want to point a subdomain (`hello-world.example.com`) to your GitLab Pages site initially deployed to `namespace.gitlab.io`, @@ -112,14 +118,14 @@ If the domain has multiple uses (e.g., you host email on it as well): | From | DNS Record | To | | ---- | ---------- | -- | -| domain.com | A | 52.167.214.135 | +| domain.com | A | 35.185.44.232 | | domain.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | If the domain is dedicated to GitLab Pages use and no other services run on it: | From | DNS Record | To | | ---- | ---------- | -- | -| subdomain.domain.com | CNAME | gitlab.io | +| subdomain.domain.com | CNAME | namespace.gitlab.io | | _gitlab-pages-verification-code.subdomain.domain.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | > **Notes**: @@ -129,9 +135,11 @@ If the domain is dedicated to GitLab Pages use and no other services run on it: > - **Do not** add any special chars after the default Pages domain. E.g., **do not** point your `subdomain.domain.com` to `namespace.gitlab.io.` or `namespace.gitlab.io/`. -> - GitLab Pages IP on GitLab.com [has been changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) from `104.208.235.32` to `52.167.214.135`. +> - GitLab Pages IP on GitLab.com [was changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) in 2017 +> - GitLab Pages IP on GitLab.com [has been changed](https://about.gitlab.com/2018/07/19/gcp-move-update/#gitlab-pages-and-custom-domains) +from `52.167.214.135` to `35.185.44.232` in 2018 -## Add your custom domain to GitLab Pages settings +### Add your custom domain to GitLab Pages settings Once you've set the DNS record, you'll need navigate to your project's **Setting > Pages** and click **+ New domain** to add your custom domain to @@ -165,6 +173,18 @@ will fail and attempts to visit your domain will respond with a 404. Read through the [general documentation on GitLab Pages](introduction.md#add-a-custom-domain-to-your-pages-website) to learn more about adding custom domains to GitLab Pages sites. +### Redirecting `www.domain.com` to `domain.com` with Cloudflare + +If you use Cloudflare, you can redirect `www` to `domain.com` without the need of adding both +`www.domain.com` and `domain.com` to GitLab. This happens due to a [Cloudflare feature that creates +a 301 redirect as a "page rule"](https://gitlab.com/gitlab-org/gitlab-ce/issues/48848#note_87314849) for redirecting `www.domain.com` to `domain.com`. In this case, +you can use the following setup: + +- In Cloudflare, create a DNS `A` record pointing `domain.com` to `35.185.44.232` +- In GitLab, add the domain to GitLab Pages +- In Cloudflare, create a DNS `TXT` record to verify your domain +- In Cloudflare, create a DNS `CNAME` record poiting `www` to `domain.com` + ## SSL/TLS Certificates Every GitLab Pages project on GitLab.com will be available under diff --git a/doc/user/project/pages/img/dns_add_new_a_record_example_updated.png b/doc/user/project/pages/img/dns_add_new_a_record_example_updated.png Binary files differdeleted file mode 100644 index 2661a497b91..00000000000 --- a/doc/user/project/pages/img/dns_add_new_a_record_example_updated.png +++ /dev/null diff --git a/doc/user/project/pages/img/dns_add_new_a_record_example_updated_2018.png b/doc/user/project/pages/img/dns_add_new_a_record_example_updated_2018.png Binary files differnew file mode 100644 index 00000000000..fa72df66587 --- /dev/null +++ b/doc/user/project/pages/img/dns_add_new_a_record_example_updated_2018.png diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 0ef8eddad20..8fdfd2a6f4d 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -1,7 +1,7 @@ # GitLab quick actions -Quick actions are textual shortcuts for common actions on issues or merge -requests that are usually done by clicking buttons or dropdowns in GitLab's UI. +Quick actions are textual shortcuts for common actions on issues, merge requests +or commits that are usually done by clicking buttons or dropdowns in GitLab's UI. You can enter these commands while creating a new issue or merge request, and in comments. Each command should be on a separate line in order to be properly detected and executed. The commands are removed from the issue, merge request or @@ -39,7 +39,8 @@ do. | `/board_move ~column` | Move issue to column on the board | | `/duplicate #issue` | Closes this issue and marks it as a duplicate of another issue | | `/move path/to/project` | Moves issue to another project | +| `/tag v1.2.3 <message>` | Tags a commit with a given tag name and optional message | | `/tableflip` | Append the comment with `(╯°□°)╯︵ ┻━┻` | | `/shrug` | Append the comment with `¯\＿(ツ)＿/¯` | | <code>/copy_metadata #issue | !merge_request</code> | Copy labels and milestone from other issue or merge request | -| `/confidential` | Makes the issue confidential |
\ No newline at end of file +| `/confidential` | Makes the issue confidential | diff --git a/doc/user/project/repository/img/repository_languages.png b/doc/user/project/repository/img/repository_languages.png Binary files differnew file mode 100644 index 00000000000..d9fb1278e06 --- /dev/null +++ b/doc/user/project/repository/img/repository_languages.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 704c1777e62..1c3915a5fdd 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -155,6 +155,16 @@ The repository graph displays visually the Git flow strategy used in that reposi Find it under your project's **Repository > Graph**. +## Repository Languages + +For the default branch of each repository, GitLab will determine what programming languages +were used and display this on the projects pages. + + + +Not all files are detected, among others; documentation, +vendored code, and most markup languages are excluded. + ## Compare Select branches to compare and view the changes inline: diff --git a/doc/user/project/web_ide/img/admin_clientside_evaluation.png b/doc/user/project/web_ide/img/admin_clientside_evaluation.png Binary files differnew file mode 100644 index 00000000000..a930490398b --- /dev/null +++ b/doc/user/project/web_ide/img/admin_clientside_evaluation.png diff --git a/doc/user/project/web_ide/img/clientside_evaluation.png b/doc/user/project/web_ide/img/clientside_evaluation.png Binary files differnew file mode 100644 index 00000000000..bd04d3d644b --- /dev/null +++ b/doc/user/project/web_ide/img/clientside_evaluation.png diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 511ac2d7e79..16969b2c527 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -72,5 +72,39 @@ leaving the Web IDE. Click the dropdown in the top of the sidebar to open a list of branches. You will need to commit or discard all your changes before switching to a different branch. +## Client Side Evaluation + +> [Introduced in](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19764) [GitLab Core][ce] 11.2. + +The Web IDE can be used to preview JavaScript projects right in the browser. +This feature uses CodeSandbox to compile and bundle the JavaScript used to +preview the web application. On public projects, an `Open in CodeSandbox` +button is visible which will transfer the contents of the project into a +CodeSandbox project to share with others. +**Note** this button is not visible on private or internal projects. + + + +### Enabling Client Side Evaluation + +The Client Side Evaluation feature needs to be enabled in the GitLab instances +admin settings. Client Side Evaluation is enabled for all projects on +GitLab.com + + + +Once it has been enabled in application settings, projects with a +`package.json` file and a `main` entry point can be previewed inside of the Web +IDE. An example `package.json` is below. + +```json +{ + "main": "index.js", + "dependencies": { + "vue": "latest" + } +} +``` + [ce]: https://about.gitlab.com/pricing/ [ee]: https://about.gitlab.com/pricing/ |
