diff options
Diffstat (limited to 'doc')
103 files changed, 2599 insertions, 488 deletions
diff --git a/doc/README.md b/doc/README.md index 7c33a708dc7..11d52001440 100644 --- a/doc/README.md +++ b/doc/README.md @@ -13,13 +13,14 @@ GitLab offers the most scalable Git-based fully integrated platform for software - **GitLab Community Edition (CE)** is an [opensource product](https://gitlab.com/gitlab-org/gitlab-ce/), self-hosted, free to use. Every feature available in GitLab CE is also available on GitLab Enterprise Edition (Starter and Premium) and GitLab.com. - **GitLab Enterprise Edition (EE)** is an [opencore product](https://gitlab.com/gitlab-org/gitlab-ee/), -self-hosted, fully featured solution of GitLab, available under distinct [subscriptions](https://about.gitlab.com/products/): **GitLab Enterprise Edition Starter (EES)** and **GitLab Enterprise Edition Premium (EEP)**. +self-hosted, fully featured solution of GitLab, available under distinct [subscriptions](https://about.gitlab.com/products/): **GitLab Enterprise Edition Starter (EES)**, **GitLab Enterprise Edition Premium (EEP)**, and **GitLab Enterprise Edition Ultimate (EEU)**. - **GitLab.com**: SaaS GitLab solution, with [free and paid subscriptions](https://about.gitlab.com/gitlab-com/). GitLab.com is hosted by GitLab, Inc., and administrated by GitLab (users don't have access to admin settings). > **GitLab EE** contains all features available in **GitLab CE**, plus premium features available in each version: **Enterprise Edition Starter** -(**EES**) and **Enterprise Edition Premium** (**EEP**). Everything available in -**EES** is also available in **EEP**. +(**EES**), **Enterprise Edition Premium** (**EEP**), and **Enterprise Edition Ultimate** +(**EEU**). Everything available in **EES** is also available in **EEP**. Every feature +available in **EEP** is also available in **EEU**. ---- @@ -32,8 +33,8 @@ Shortcuts to GitLab's most visited docs: | [Using Docker images](ci/docker/using_docker_images.md) | [GitLab Pages](user/project/pages/index.md) | - [User documentation](user/index.md) -- [Administrator documentation](#administrator-documentation) -- [Technical Articles](articles/index.md) +- [Administrator documentation](administration/index.md) +- [Contributor documentation](#contributor-documentation) ## Getting started with GitLab @@ -133,81 +134,24 @@ Manage your [repositories](user/project/repository/index.md) from the UI (user i ## Administrator documentation -Learn how to administer your GitLab instance. Regular users don't -have access to GitLab administration tools and settings. +[Administration documentation](administration/index.md) applies to admin users of GitLab +self-hosted instances: -### Install, update, upgrade, migrate +- GitLab Community Edition +- GitLab [Enterprise Editions](https://about.gitlab.com/gitlab-ee/) + - Enterprise Edition Starter (EES) + - Enterprise Edition Premium (EEP) + - Enterprise Edition Ultimate (EEU) -- [Install](install/README.md): Requirements, directory structures and installation from source. -- [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate [Mattermost](https://about.mattermost.com/) with your GitLab installation. -- [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md): If you have an old GitLab installation (older than 8.0), follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. -- [Restart GitLab](administration/restart_gitlab.md): Learn how to restart GitLab and its components. -- [Update](update/README.md): Update guides to upgrade your installation. - -### User permissions - -- [Access restrictions](user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab -- [Authentication/Authorization](topics/authentication/index.md#gitlab-administrators): Enforce 2FA, configure external authentication with LDAP, SAML, CAS and additional Omniauth providers. - -### Features - -- [Container Registry](administration/container_registry.md): Configure Docker Registry with GitLab. -- [Custom Git hooks](administration/custom_hooks.md): Custom Git hooks (on the filesystem) for when webhooks aren't enough. -- [Git LFS configuration](workflow/lfs/lfs_administration.md): Learn how to use LFS under GitLab. -- [GitLab Pages configuration](administration/pages/index.md): Configure GitLab Pages. -- [High Availability](administration/high_availability/README.md): Configure multiple servers for scaling or high availability. -- [User cohorts](user/admin_area/user_cohorts.md): View user activity over time. -- [Web terminals](administration/integration/terminal.md): Provide terminal access to environments from within GitLab. -- GitLab CI - - [CI admin settings](user/admin_area/settings/continuous_integration.md): Define max artifacts size and expiration time. - -### Integrations - -- [Integrations](integration/README.md): How to integrate with systems such as JIRA, Redmine, Twitter. -- [Mattermost](user/project/integrations/mattermost.md): Set up GitLab with Mattermost. - -### Monitoring - -- [GitLab performance monitoring with InfluxDB](administration/monitoring/performance/introduction.md): Configure GitLab and InfluxDB for measuring performance metrics. -- [GitLab performance monitoring with Prometheus](administration/monitoring/prometheus/index.md): Configure GitLab and Prometheus for measuring performance metrics. -- [Monitoring uptime](user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - -### Performance - -- [Housekeeping](administration/housekeeping.md): Keep your Git repository tidy and fast. -- [Operations](administration/operations.md): Keeping GitLab up and running. -- [Polling](administration/polling.md): Configure how often the GitLab UI polls for updates. -- [Request Profiling](administration/monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. -- [Performance Bar](administration/monitoring/performance/performance_bar.md): Get performance information for the current page. - -### Customization - -- [Adjust your instance's timezone](workflow/timezone.md): Customize the default time zone of GitLab. -- [Environment variables](administration/environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. -- [Header logo](customization/branded_page_and_email_header.md): Change the logo on the overall page and email header. -- [Issue closing pattern](administration/issue_closing_pattern.md): Customize how to close an issue from commit messages. -- [Libravatar](customization/libravatar.md): Use Libravatar instead of Gravatar for user avatars. -- [Welcome message](customization/welcome_message.md): Add a custom welcome message to the sign-in page. - -### Admin tools - -- [Gitaly](administration/gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service -- [Raketasks](raketasks/README.md): Backups, maintenance, automatic webhook setup and the importing of projects. - - [Backup and restore](raketasks/backup_restore.md): Backup and restore your GitLab instance. -- [Reply by email](administration/reply_by_email.md): Allow users to comment on issues and merge requests by replying to notification emails. -- [Repository checks](administration/repository_checks.md): Periodic Git repository checks. -- [Repository storage paths](administration/repository_storage_paths.md): Manage the paths used to store repositories. -- [Security](security/README.md): Learn what you can do to further secure your GitLab instance. -- [System hooks](system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. - -### Troubleshooting - -- [Debugging tips](administration/troubleshooting/debug.md): Tips to debug problems when things go wrong -- [Log system](administration/logs.md): Where to look for logs. -- [Sidekiq Troubleshooting](administration/troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. +Learn how to install, configure, update, upgrade, integrate, and maintain your own instance. +Regular users don't have access to GitLab administration tools and settings. ## Contributor documentation +GitLab Community Edition is [opensource](https://gitlab.com/gitlab-org/gitlab-ce/) +and Enterprise Editions are [opencore](https://gitlab.com/gitlab-org/gitlab-ee/). +Learn how to contribute to GitLab: + - [Development](development/README.md): All styleguides and explanations how to contribute. - [Legal](legal/README.md): Contributor license agreements. - [Writing documentation](development/writing_documentation.md): Contributing to GitLab Docs. diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index ad903aef896..6b5a0f139c5 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -30,6 +30,12 @@ immediately block all access. >**Note**: GitLab EE supports a configurable sync time, with a default of one hour. +## Git password authentication + +LDAP-enabled users can always authenticate with Git using their GitLab username +or email and LDAP password, even if password authentication for Git is disabled +in the application settings. + ## Configuration To enable LDAP integration you need to add your LDAP server settings in diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 4d3be0ab8f6..a88e67bfeb5 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -53,7 +53,9 @@ or in different cloud availability zones. > **Note:** GitLab recommends against choosing this HA method because of the complexity of managing DRBD and crafting automatic failover. This is - *compatible* with GitLab, but not officially *supported*. + *compatible* with GitLab, but not officially *supported*. If you are + an EE customer, support will help you with GitLab related problems, but if the + root cause is identified as DRBD, we will not troubleshoot further. Components/Servers Required: 2 servers/virtual machines (one active/one passive) diff --git a/doc/administration/index.md b/doc/administration/index.md new file mode 100644 index 00000000000..c8d28d8485a --- /dev/null +++ b/doc/administration/index.md @@ -0,0 +1,121 @@ +# Administrator documentation + +Learn how to administer your GitLab instance (Community Edition and +[Enterprise Editions](https://about.gitlab.com/gitlab-ee/)). +Regular users don't have access to GitLab administration tools and settings. + +GitLab.com is administered by GitLab, Inc., therefore, only GitLab team members have +access to its admin configurations. If you're a GitLab.com user, please check the +[user documentation](../user/index.html). + +## Installing and maintaining GitLab + +Learn how to install, configure, update, and maintain your GitLab instance. + +### Installing GitLab + +- [Install](../install/README.md): Requirements, directory structures, and installation methods. +- [High Availability](high_availability/README.md): Configure multiple servers for scaling or high availability. + +### Configuring GitLab + +- [Adjust your instance's timezone](../workflow/timezone.md): Customize the default time zone of GitLab. +- [Header logo](../customization/branded_page_and_email_header.md): Change the logo on all pages and email headers. +- [Welcome message](../customization/welcome_message.md): Add a custom welcome message to the sign-in page. +- [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. +- [Security](../security/README.md): Learn what you can do to further secure your GitLab instance. +- [Usage statistics, version check, and usage ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. +- [Polling](polling.md): Configure how often the GitLab UI polls for updates. +- [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. +- [GitLab Pages configuration for installations from the source](pages/source.md): Enable and configure GitLab Pages on +[source installations](../install/installation.md#installation-from-source). +- [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. + +### Maintaining GitLab + +- [Raketasks](../raketasks/README.md): Perform various tasks for maintenance, backups, automatic webhooks setup, etc. + - [Backup and restore](../raketasks/backup_restore.md): Backup and restore your GitLab instance. +- [Operations](operations.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq Job throttling, Sidekiq MemoryKiller, Unicorn). +- [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components. + +#### Updating GitLab + +- [GitLab versions and maintenance policy](../policy/maintenance.md): Understand GitLab versions and releases (Major, Minor, Patch, Security), as well as update recommendations. +- [Update GitLab](../update/README.md): Update guides to upgrade your installation to a new version. +- [Downtimeless updates](../update/README.md#upgrading-without-downtime): Upgrade to a newer major, minor, or patch version of GitLab without taking your GitLab instance offline. +- [Migrate your GitLab CI/CD data to another version of GitLab](../migrate_ci_to_ce/README.md): If you have an old GitLab installation (older than 8.0), follow this guide to migrate your existing GitLab CI/CD data to another version of GitLab. + +### Upgrading or downgrading GitLab + +- [Upgrade from GitLab CE to GitLab EE](../update/README.md#upgrading-between-editions): learn how to upgrade GitLab Community Edition to GitLab Enterprise Editions. +- [Downgrade from GitLab EE to GitLab CE](../downgrade_ee_to_ce/README.md): Learn how to downgrade GitLab Enterprise Editions to Community Edition. + +### GitLab platform integrations + +- [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://about.mattermost.com/), an open source, private cloud workplace for web messaging. +- [PlantUML](integration/plantuml.md): Create simple diagrams in AsciiDoc and Markdown documents +created in snippets, wikis, and repos. +- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab's CI/CD [environments](../ci/environments.md#web-terminals). + +## User settings and permissions + +- [Libravatar](../customization/libravatar.md): Use Libravatar instead of Gravatar for user avatars. +- [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. +- [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). +- [Authentication/Authorization](../topics/authentication/index.md#gitlab-administrators): Enforce 2FA, configure external authentication with LDAP, SAML, CAS and additional Omniauth providers. +- [Reply by email](reply_by_email.md): Allow users to comment on issues and merge requests by replying to notification emails. + - [Postfix for Reply by email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail +server with IMAP authentication on Ubuntu, to be used with Reply by email. +- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. + +## Project settings + +- [Container Registry](container_registry.md): Configure Container Registry with GitLab. +- [Issue closing pattern](issue_closing_pattern.md): Customize how to close an issue from commit messages. +- [Gitaly](gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service. +- [Default labels](../user/admin_area/labels.html): Create labels that will be automatically added to every new project. + +### Repository settings + +- [Repository checks](repository_checks.md): Periodic Git repository checks. +- [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. +- [Repository storage rake tasks](raketasks/storage.md): A collection of rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. + +## Continuous Integration settings + +- [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance. +- [GitLab CI/CD admin settings](../user/admin_area/settings/continuous_integration.md): Define max artifacts size and expiration time. +- [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). +- [Artifacts size and expiration](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size): Define maximum artifacts limits and expiration date. +- [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance. +- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for Shared Runners. +- [Enable/disable Auto DevOps](../topics/autodevops/index.md#enabling-auto-devops): Enable or disable Auto DevOps for your instance. + +## Git configuration options + +- [Custom Git hooks](custom_hooks.md): Custom Git hooks (on the filesystem) for when webhooks aren't enough. +- [Git LFS configuration](../workflow/lfs/lfs_administration.md): Learn how to configure LFS for GitLab. +- [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast. + +## Monitoring GitLab + +- [Monitoring uptime](../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. + - [IP whitelist](monitoring/ip_whitelist.md): Monitor endpoints that provide health check information when probed. +- [Monitoring GitHub imports](monitoring/github_imports.md): GitLab's GitHub Importer displays Prometheus metrics to monitor the health and progress of the importer. +- [Conversational Development (ConvDev) Index](../user/admin_area/monitoring/convdev.md): Provides an overview of your entire instance's feature usage. + +### Performance Monitoring + +- [GitLab Performance Monitoring](monitoring/performance/gitlab_configuration.md): Enable GitLab Performance Monitoring. +- [GitLab performance monitoring with InfluxDB](monitoring/performance/introduction.md): Configure GitLab and InfluxDB for measuring performance metrics. + - [InfluxDB Schema](monitoring/performance/influxdb_schema.md): Measurements stored in InfluxDB. +- [GitLab performance monitoring with Prometheus](monitoring/prometheus/index.md): Configure GitLab and Prometheus for measuring performance metrics. +- [GitLab performance monitoring with Grafana](monitoring/prometheus/index.md): Configure GitLab to visualize time series metrics through graphs and dashboards. +- [Request Profiling](monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. +- [Performance Bar](monitoring/performance/performance_bar.md): Get performance information for the current page. + +## Troubleshooting + +- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong +- [Log system](logs.md): Where to look for logs. +- [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 59305774c4a..7f1a0486749 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -157,6 +157,45 @@ steps below. 1. Save the file and [restart GitLab][] for the changes to take effect. +## Validation for dependencies + +> Introduced in GitLab 10.3. + +To disable [the dependencies validation](../ci/yaml/README.md#when-a-dependent-job-will-fail), +you can flip the feature flag from a Rails console. + +--- + +**In Omnibus installations:** + +1. Enter the Rails console: + + ```sh + sudo gitlab-rails console + ``` + +1. Flip the switch and disable it: + + ```ruby + Feature.enable('ci_disable_validates_dependencies') + ``` +--- + +**In installations from source:** + +1. Enter the Rails console: + + ```sh + cd /home/git/gitlab + RAILS_ENV=production sudo -u git -H bundle exec rails console + ``` + +1. Flip the switch and disable it: + + ```ruby + Feature.enable('ci_disable_validates_dependencies') + ``` + ## Set the maximum file size of the artifacts Provided the artifacts are enabled, you can change the maximum file size of the diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md new file mode 100644 index 00000000000..5592e0a9e9a --- /dev/null +++ b/doc/administration/monitoring/github_imports.md @@ -0,0 +1,101 @@ +# Monitoring GitHub imports + +>**Note:** +Available since [GitLab 10.2][14731]. + +The GitHub importer exposes various Prometheus metrics that you can use to +monitor the health and progress of the importer. + +## Import Duration Times + +| Name | Type | +|------------------------------------------|-----------| +| `github_importer_total_duration_seconds` | histogram | + +This metric tracks the total time spent (in seconds) importing a project (from +project creation until the import process finishes), for every imported project. + +The name of the project is stored in the `project` label in the format +`namespace/name` (e.g. `gitlab-org/gitlab-ce`). + +## Number of imported projects + +| Name | Type | +|-------------------------------------|---------| +| `github_importer_imported_projects` | counter | + +This metric tracks the total number of projects imported over time. This metric +does not expose any labels. + +## Number of GitHub API calls + +| Name | Type | +|---------------------------------|---------| +| `github_importer_request_count` | counter | + +This metric tracks the total number of GitHub API calls performed over time, for +all projects. This metric does not expose any labels. + +## Rate limit errors + +| Name | Type | +|-----------------------------------|---------| +| `github_importer_rate_limit_hits` | counter | + +This metric tracks the number of times we hit the GitHub rate limit, for all +projects. This metric does not expose any labels. + +## Number of imported issues + +| Name | Type | +|-----------------------------------|---------| +| `github_importer_imported_issues` | counter | + +This metric tracks the number of imported issues across all projects. + +The name of the project is stored in the `project` label in the format +`namespace/name` (e.g. `gitlab-org/gitlab-ce`). + +## Number of imported pull requests + +| Name | Type | +|------------------------------------------|---------| +| `github_importer_imported_pull_requests` | counter | + +This metric tracks the number of imported pull requests across all projects. + +The name of the project is stored in the `project` label in the format +`namespace/name` (e.g. `gitlab-org/gitlab-ce`). + +## Number of imported comments + +| Name | Type | +|----------------------------------|---------| +| `github_importer_imported_notes` | counter | + +This metric tracks the number of imported comments across all projects. + +The name of the project is stored in the `project` label in the format +`namespace/name` (e.g. `gitlab-org/gitlab-ce`). + +## Number of imported pull request review comments + +| Name | Type | +|---------------------------------------|---------| +| `github_importer_imported_diff_notes` | counter | + +This metric tracks the number of imported comments across all projects. + +The name of the project is stored in the `project` label in the format +`namespace/name` (e.g. `gitlab-org/gitlab-ce`). + +## Number of imported repositories + +| Name | Type | +|-----------------------------------------|---------| +| `github_importer_imported_repositories` | counter | + +This metric tracks the number of imported repositories across all projects. This +metric does not expose any labels. + +[14731]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14731 diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 0c63b0b59a7..7d47aaac299 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -58,6 +58,9 @@ Before proceeding with the Pages configuration, you will need to: so that your users don't have to bring their own. 1. (Only for custom domains) Have a **secondary IP**. +NOTE: **Note:** +If your GitLab instance and the Pages daemon are deployed in a private network or behind a firewall, your GitLab Pages websites will only be accessible to devices/users that have access to the private network. + ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 5b6ee354887..136192191f9 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -58,7 +58,9 @@ Runs the following rake tasks: It will check that each component was setup according to the installation guide and suggest fixes for issues found. -You may also have a look at our [Trouble Shooting Guide](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Trouble-Shooting-Guide). +You may also have a look at our Trouble Shooting Guides: +- [Trouble Shooting Guide (GitLab)](http://docs.gitlab.com/ee/README.html#troubleshooting) +- [Trouble Shooting Guide (Omnibus Gitlab)](http://docs.gitlab.com/omnibus/README.html#troubleshooting) **Omnibus Installation** diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index bac8fa4bd9d..6ec5baeb6e3 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,10 +1,43 @@ # Repository Storage Rake Tasks This is a collection of rake tasks you can use to help you list and migrate -existing projects from Legacy storage to the new Hashed storage type. +existing projects and attachments associated with it from Legacy storage to +the new Hashed storage type. You can read more about the storage types [here][storage-types]. +## Migrate existing projects to Hashed storage + +Before migrating your existing projects, you should +[enable hashed storage][storage-migration] for the new projects as well. + +This task will schedule all your existing projects and attachments associated with it to be migrated to the +**Hashed** storage type: + +**Omnibus Installation** + +```bash +gitlab-rake gitlab:storage:migrate_to_hashed +``` + +**Source Installation** + +```bash +rake gitlab:storage:migrate_to_hashed + +``` + +You can monitor the progress in the _Admin > Monitoring > Background jobs_ screen. +There is a specific Queue you can watch to see how long it will take to finish: **project_migrate_hashed_storage** + +After it reaches zero, you can confirm every project has been migrated by running the commands bellow. +If you find it necessary, you can run this migration script again to schedule missing projects. + +Any error or warning will be logged in the sidekiq's log file. + +You only need the `gitlab:storage:migrate_to_hashed` rake task to migrate your repositories, but we have additional +commands below that helps you inspect projects and attachments in both legacy and hashed storage. + ## List projects on Legacy storage To have a simple summary of projects using **Legacy** storage: @@ -73,35 +106,73 @@ rake gitlab:storage:list_hashed_projects ``` -## Migrate existing projects to Hashed storage +## List attachments on Legacy storage -Before migrating your existing projects, you should -[enable hashed storage][storage-migration] for the new projects as well. +To have a simple summary of project attachments using **Legacy** storage: -This task will schedule all your existing projects to be migrated to the -**Hashed** storage type: +**Omnibus Installation** + +```bash +gitlab-rake gitlab:storage:legacy_attachments +``` + +**Source Installation** + +```bash +rake gitlab:storage:legacy_attachments + +``` + +------ + +To list project attachments using **Legacy** storage: **Omnibus Installation** ```bash -gitlab-rake gitlab:storage:migrate_to_hashed +gitlab-rake gitlab:storage:list_legacy_attachments ``` **Source Installation** ```bash -rake gitlab:storage:migrate_to_hashed +rake gitlab:storage:list_legacy_attachments ``` -You can monitor the progress in the _Admin > Monitoring > Background jobs_ screen. -There is a specific Queue you can watch to see how long it will take to finish: **project_migrate_hashed_storage** +## List attachments on Hashed storage -After it reaches zero, you can confirm every project has been migrated by running the commands above. -If you find it necessary, you can run this migration script again to schedule missing projects. +To have a simple summary of project attachments using **Hashed** storage: + +**Omnibus Installation** + +```bash +gitlab-rake gitlab:storage:hashed_attachments +``` -Any error or warning will be logged in the sidekiq log file. +**Source Installation** + +```bash +rake gitlab:storage:hashed_attachments + +``` + +------ + +To list project attachments using **Hashed** storage: + +**Omnibus Installation** +```bash +gitlab-rake gitlab:storage:list_hashed_attachments +``` + +**Source Installation** + +```bash +rake gitlab:storage:list_hashed_attachments + +``` [storage-types]: ../repository_storage_types.md [storage-migration]: ../repository_storage_types.md#how-to-migrate-to-hashed-storage diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index bc9b6253f1a..21184fed6e9 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -9,7 +9,7 @@ mapping structure from the projects URLs: * Project's repository: `#{namespace}/#{project_name}.git` * Project's wiki: `#{namespace}/#{project_name}.wiki.git` - + This structure made simple to migrate from existing solutions to GitLab and easy for Administrators to find where the repository is stored. @@ -27,7 +27,7 @@ of load in big installations, and can be even worst if they are using any type o Last, for GitLab Geo, this storage type means we have to synchronize the disk state, replicate renames in the correct order or we may end-up with wrong repository or missing data temporarily. -This pattern also exists in other objects stored in GitLab, like issue Attachments, GitLab Pages artifacts, +This pattern also exists in other objects stored in GitLab, like issue Attachments, GitLab Pages artifacts, Docker Containers for the integrated Registry, etc. ## Hashed Storage @@ -62,9 +62,9 @@ you will never mistakenly restore a repository in the wrong project (considering ### How to migrate to Hashed Storage -In GitLab, go to **Admin > Settings**, find the **Repository Storage** section and select +In GitLab, go to **Admin > Settings**, find the **Repository Storage** section and select "_Create new projects using hashed storage paths_". - + To migrate your existing projects to the new storage type, check the specific [rake tasks]. [ce-28283]: https://gitlab.com/gitlab-org/gitlab-ce/issues/28283 @@ -79,14 +79,14 @@ coverage status below. Note that things stored in an S3 compatible endpoint will not have the downsides mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, which is true for CI Cache and LFS Objects. -| Storable Object | Legacy Storage | Hashed Storage | S3 Compatible | GitLab Version | -| ----------------| -------------- | -------------- | ------------- | -------------- | +| Storable Object | Legacy Storage | Hashed Storage | S3 Compatible | GitLab Version | +| --------------- | -------------- | -------------- | ------------- | -------------- | | Repository | Yes | Yes | - | 10.0 | | Attachments | Yes | Yes | - | 10.2 | -| Avatars | Yes | No | - | - | +| Avatars | Yes | No | - | - | | Pages | Yes | No | - | - | | Docker Registry | Yes | No | - | - | -| CI Build Logs | No | No | - | - | -| CI Artifacts | No | No | - | - | +| CI Build Logs | No | No | - | - | +| CI Artifacts | No | No | Yes (EEP) | - | | CI Cache | No | No | Yes | - | -| LFS Objects | Yes | No | Yes (EEP) | - | +| LFS Objects | Yes | No | Yes (EEP) | - | diff --git a/doc/administration/repository_storages.md b/doc/administration/repository_storages.md index 9d41ba77f34..cf6de15743f 100644 --- a/doc/administration/repository_storages.md +++ b/doc/administration/repository_storages.md @@ -1,3 +1 @@ -# Repository storages - -This document was moved to a [new location](repository_storage_paths.md). +This document was moved to [another location](repository_storage_paths.md). diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index be538ea250a..83a714810c1 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -163,6 +163,34 @@ separate Rails process to debug the issue: 1. In a new window, run `top`. It should show this ruby process using 100% CPU. Write down the PID. 1. Follow step 2 from the previous section on using gdb. +### GitLab: API is not accessible + +This often occurs when gitlab-shell attempts to request authorization via the +internal API (e.g., `http://localhost:8080/api/v4/internal/allowed`), and +something in the check fails. There are many reasons why this may happen: + +1. Timeout connecting to a database (e.g., PostgreSQL or Redis) +1. Error in Git hooks or push rules +1. Error accessing the repository (e.g., stale NFS handles) + +To diagnose this problem, try to reproduce the problem and then see if there +is a unicorn worker that is spinning via `top`. Try to use the `gdb` +techniques above. In addition, using `strace` may help isolate issues: + +```shell +strace -tt -T -f -s 1024 -p <PID of unicorn worker> -o /tmp/unicorn.txt +``` + +If you cannot isolate which Unicorn worker is the issue, try to run `strace` +on all the Unicorn workers to see where the `/internal/allowed` endpoint gets +stuck: + +```shell +ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -s 1024 -o /tmp/unicorn.txt +``` + +The output in `/tmp/unicorn.txt` may help diagnose the root cause. + # More information * [Debugging Stuck Ruby Processes](https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/) diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index b71f8fabbc8..9d157720ad2 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -11,7 +11,7 @@ troubleshooting steps that will help you diagnose the bottleneck. debug steps with GitLab Support so the backtraces can be analyzed by our team. It may reveal a bug or necessary improvement in GitLab. -> **Note:** In any of the backtraces, be weary of suspecting cases where every +> **Note:** In any of the backtraces, be wary of suspecting cases where every thread appears to be waiting in the database, Redis, or waiting to acquire a mutex. This **may** mean there's contention in the database, for example, but look for one thread that is different than the rest. This other thread diff --git a/doc/api/environments.md b/doc/api/environments.md index e8deb3e07e9..6e20781f51a 100644 --- a/doc/api/environments.md +++ b/doc/api/environments.md @@ -36,7 +36,7 @@ Creates a new environment with the given name and external_url. It returns `201` if the environment was successfully created, `400` for wrong parameters. ``` -POST /projects/:id/environment +POST /projects/:id/environments ``` | Attribute | Type | Required | Description | diff --git a/doc/api/groups.md b/doc/api/groups.md index 16db9c2f259..c1b5737c247 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -9,13 +9,13 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `skip_groups` | array of integers | no | Skip the group IDs passes | -| `all_available` | boolean | no | Show all the groups you have access to | -| `search` | string | no | Return list of authorized groups matching the search criteria | +| `skip_groups` | array of integers | no | Skip the group IDs passed | +| `all_available` | boolean | no | Show all the groups you have access to (defaults to `false` for authenticated users) | +| `search` | string | no | Return the list of authorized groups matching the search criteria | | `order_by` | string | no | Order groups by `name` or `path`. Default is `name` | | `sort` | string | no | Order groups in `asc` or `desc` order. Default is `asc` | | `statistics` | boolean | no | Include group statistics (admins only) | -| `owned` | boolean | no | Limit by groups owned by the current user | +| `owned` | boolean | no | Limit to groups owned by the current user | ``` GET /groups @@ -80,6 +80,49 @@ You can filter by [custom attributes](custom_attributes.md) with: GET /groups?custom_attributes[key]=value&custom_attributes[other_key]=other_value ``` +## List a groups's subgroups + +> [Introduced][ce-15142] in GitLab 10.3. + +Get a list of visible direct subgroups in this group. +When accessed without authentication, only public groups are returned. + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) of the parent group | +| `skip_groups` | array of integers | no | Skip the group IDs passed | +| `all_available` | boolean | no | Show all the groups you have access to (defaults to `false` for authenticated users) | +| `search` | string | no | Return the list of authorized groups matching the search criteria | +| `order_by` | string | no | Order groups by `name` or `path`. Default is `name` | +| `sort` | string | no | Order groups in `asc` or `desc` order. Default is `asc` | +| `statistics` | boolean | no | Include group statistics (admins only) | +| `owned` | boolean | no | Limit to groups owned by the current user | + +``` +GET /groups/:id/subgroups +``` + +```json +[ + { + "id": 1, + "name": "Foobar Group", + "path": "foo-bar", + "description": "An interesting group", + "visibility": "public", + "lfs_enabled": true, + "avatar_url": "http://gitlab.example.com/uploads/group/avatar/1/foo.jpg", + "web_url": "http://gitlab.example.com/groups/foo-bar", + "request_access_enabled": false, + "full_name": "Foobar Group", + "full_path": "foo-bar", + "parent_id": 123 + } +] +``` + ## List a group's projects Get a list of projects in this group. When accessed without authentication, only @@ -472,3 +515,5 @@ And to switch pages add: ``` /groups?per_page=100&page=2 ``` + +[ce-15142]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15142 diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 6de460f2778..b2e4b6d0955 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -58,6 +58,8 @@ Parameters: "project_id": 3, "title": "test1", "state": "opened", + "created_at": "2017-04-29T08:46:00Z", + "updated_at": "2017-04-29T08:46:00Z", "upvotes": 0, "downvotes": 0, "author": { @@ -170,6 +172,8 @@ Parameters: "project_id": 3, "title": "test1", "state": "opened", + "created_at": "2017-04-29T08:46:00Z", + "updated_at": "2017-04-29T08:46:00Z", "upvotes": 0, "downvotes": 0, "author": { @@ -246,6 +250,8 @@ Parameters: "project_id": 3, "title": "test1", "state": "merged", + "created_at": "2017-04-29T08:46:00Z", + "updated_at": "2017-04-29T08:46:00Z", "upvotes": 0, "downvotes": 0, "author": { diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index 5c0bebbaeb0..25cae5ce1f9 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -89,3 +89,55 @@ Example response: } ] ``` + +## Get namespace by ID + +Get a namespace by ID. + +``` +GET /namespaces/:id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or path of the namespace | + +Example request: + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/namespaces/2 +``` + +Example response: + +```json +{ + "id": 2, + "name": "group1", + "path": "group1", + "kind": "group", + "full_path": "group1", + "parent_id": "null", + "members_count_with_descendants": 2 +} +``` + +Example request: + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/namespaces/group1 +``` + +Example response: + +```json +{ + "id": 2, + "name": "group1", + "path": "group1", + "kind": "group", + "full_path": "group1", + "parent_id": "null", + "members_count_with_descendants": 2 +} +``` diff --git a/doc/api/notes.md b/doc/api/notes.md index e627369e17b..d02ef84d0bd 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -10,12 +10,15 @@ Gets a list of all notes for a single issue. ``` GET /projects/:id/issues/:issue_iid/notes +GET /projects/:id/issues/:issue_iid/notes?sort=asc&order_by=updated_at ``` -Parameters: - -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user -- `issue_iid` (required) - The IID of an issue +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `issue_iid` | integer | yes | The IID of an issue +| `sort` | string | no | Return issue notes sorted in `asc` or `desc` order. Default is `desc` +| `order_by` | string | no | Return issue notes ordered by `created_at` or `updated_at` fields. Default is `created_at` ```json [ @@ -133,12 +136,15 @@ Gets a list of all notes for a single snippet. Snippet notes are comments users ``` GET /projects/:id/snippets/:snippet_id/notes +GET /projects/:id/snippets/:snippet_id/notes?sort=asc&order_by=updated_at ``` -Parameters: - -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user -- `snippet_id` (required) - The ID of a project snippet +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `snippet_id` | integer | yes | The ID of a project snippet +| `sort` | string | no | Return snippet notes sorted in `asc` or `desc` order. Default is `desc` +| `order_by` | string | no | Return snippet notes ordered by `created_at` or `updated_at` fields. Default is `created_at` ### Get single snippet note @@ -231,12 +237,15 @@ Gets a list of all notes for a single merge request. ``` GET /projects/:id/merge_requests/:merge_request_iid/notes +GET /projects/:id/merge_requests/:merge_request_iid/notes?sort=asc&order_by=updated_at ``` -Parameters: - -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user -- `merge_request_iid` (required) - The IID of a project merge request +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `merge_request_iid` | integer | yes | The IID of a project merge request +| `sort` | string | no | Return merge request notes sorted in `asc` or `desc` order. Default is `desc` +| `order_by` | string | no | Return merge request notes ordered by `created_at` or `updated_at` fields. Default is `created_at` ### Get single merge request note diff --git a/doc/api/pages_domains.md b/doc/api/pages_domains.md index 51962595e33..50685f335f7 100644 --- a/doc/api/pages_domains.md +++ b/doc/api/pages_domains.md @@ -4,6 +4,31 @@ Endpoints for connecting custom domain(s) and TLS certificates in [GitLab Pages] The GitLab Pages feature must be enabled to use these endpoints. Find out more about [administering](../administration/pages/index.md) and [using](../user/project/pages/index.md) the feature. +## List all pages domains + +Get a list of all pages domains. The user must have admin permissions. + +```http +GET /pages/domains +``` + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/pages/domains +``` + +```json +[ + { + "domain": "ssl.domain.example", + "url": "https://ssl.domain.example", + "certificate": { + "expired": false, + "expiration": "2020-04-12T14:32:00.000Z" + } + } +] +``` + ## List pages domains Get a list of project pages domains. The user must have permissions to view pages domains. diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index 10faa95d7e8..950ead52560 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -4,7 +4,7 @@ **Valid access levels** -The access levels are defined in the `ProtectedBranchAccess::ALLOWED_ACCESS_LEVELS` constant. Currently, these levels are recognized: +The access levels are defined in the `ProtectedRefAccess::ALLOWED_ACCESS_LEVELS` constant. Currently, these levels are recognized: ``` 0 => No access 30 => Developer access @@ -136,7 +136,7 @@ DELETE /projects/:id/protected_branches/:name ``` ```bash -curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v4/projects/5/protected_branches/*-stable' +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v4/projects/5/protected_branches/*-stable' ``` | Attribute | Type | Required | Description | diff --git a/doc/api/runners.md b/doc/api/runners.md index 6304a496f94..015b09a745e 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -215,6 +215,91 @@ DELETE /runners/:id curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/runners/6" ``` +## List runner's jobs + +List jobs that are being processed or were processed by specified Runner. + +``` +GET /runners/:id/jobs +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a runner | +| `status` | string | no | Status of the job; one of: `running`, `success`, `failed`, `canceled` | + +``` +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/runners/1/jobs?status=running" +``` + +Example response: + +```json +[ + { + "id": 2, + "status": "running", + "stage": "test", + "name": "test", + "ref": "master", + "tag": false, + "coverage": null, + "created_at": "2017-11-16T08:50:29.000Z", + "started_at": "2017-11-16T08:51:29.000Z", + "finished_at": "2017-11-16T08:53:29.000Z", + "duration": 120, + "user": { + "id": 1, + "name": "John Doe2", + "username": "user2", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/c922747a93b40d1ea88262bf1aebee62?s=80&d=identicon", + "web_url": "http://localhost/user2", + "created_at": "2017-11-16T18:38:46.000Z", + "bio": null, + "location": null, + "skype": "", + "linkedin": "", + "twitter": "", + "website_url": "", + "organization": null + }, + "commit": { + "id": "97de212e80737a608d939f648d959671fb0a0142", + "short_id": "97de212e", + "title": "Update configuration\r", + "created_at": "2017-11-16T08:50:28.000Z", + "parent_ids": [ + "1b12f15a11fc6e62177bef08f47bc7b5ce50b141", + "498214de67004b1da3d820901307bed2a68a8ef6" + ], + "message": "See merge request !123", + "author_name": "John Doe2", + "author_email": "user2@example.org", + "authored_date": "2017-11-16T08:50:27.000Z", + "committer_name": "John Doe2", + "committer_email": "user2@example.org", + "committed_date": "2017-11-16T08:50:27.000Z" + }, + "pipeline": { + "id": 2, + "sha": "97de212e80737a608d939f648d959671fb0a0142", + "ref": "master", + "status": "running" + }, + "project": { + "id": 1, + "description": null, + "name": "project1", + "name_with_namespace": "John Doe2 / project1", + "path": "project1", + "path_with_namespace": "namespace1/project1", + "created_at": "2017-11-16T18:38:46.620Z" + } + } +] +``` + ## List project's runners List all runners (specific and shared) available in the project. Shared runners diff --git a/doc/api/services.md b/doc/api/services.md index e642ec964de..08df26db3ec 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -490,6 +490,41 @@ Remove all previously JIRA settings from a project. DELETE /projects/:id/services/jira ``` +## Kubernetes + +Kubernetes / Openshift integration + +### Create/Edit Kubernetes service + +Set Kubernetes service for a project. + +``` +PUT /projects/:id/services/kubernetes +``` + +Parameters: + +- `namespace` (**required**) - The Kubernetes namespace to use +- `api_url` (**required**) - The URL to the Kubernetes cluster API, e.g., https://kubernetes.example.com +- `token` (**required**) - The service token to authenticate against the Kubernetes cluster with +- `ca_pem` (optional) - A custom certificate authority bundle to verify the Kubernetes cluster with (PEM format) + +### Delete Kubernetes service + +Delete Kubernetes service for a project. + +``` +DELETE /projects/:id/services/kubernetes +``` + +### Get Kubernetes service settings + +Get Kubernetes service settings for a project. + +``` +GET /projects/:id/services/kubernetes +``` + ## Slack slash commands Ability to receive slash commands from a Slack chat instance. @@ -572,7 +607,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `token` | string | yes | The Mattermost token | - +| `username` | string | no | The username to use to post the message | ### Delete Mattermost slash command service diff --git a/doc/api/settings.md b/doc/api/settings.md index 4e24e4bbfc3..0e4758cda2d 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -25,7 +25,7 @@ Example response: "id" : 1, "default_branch_protection" : 2, "restricted_visibility_levels" : [], - "password_authentication_enabled" : true, + "password_authentication_enabled_for_web" : true, "after_sign_out_path" : null, "max_attachment_size" : 10, "user_oauth_applications" : true, @@ -70,16 +70,15 @@ PUT /application/settings | `akismet_api_key` | string | no | API key for akismet spam protection | | `akismet_enabled` | boolean | no | Enable or disable akismet spam protection | | `circuitbreaker_access_retries | integer | no | The number of attempts GitLab will make to access a storage. | -| `circuitbreaker_backoff_threshold | integer | no | The number of failures after which GitLab will start temporarily disabling access to a storage shard on a host. | +| `circuitbreaker_check_interval` | integer | no | Number of seconds in between storage checks. | | `circuitbreaker_failure_count_threshold` | integer | no | The number of failures of after which GitLab will completely prevent access to the storage. | | `circuitbreaker_failure_reset_time` | integer | no | Time in seconds GitLab will keep storage failure information. When no failures occur during this time, the failure information is reset. | -| `circuitbreaker_failure_wait_time` | integer | no | Time in seconds GitLab will block access to a failing storage to allow it to recover. | | `circuitbreaker_storage_timeout` | integer | no | Seconds to wait for a storage access attempt | | `clientside_sentry_dsn` | string | no | Required if `clientside_sentry_dsn` is enabled | | `clientside_sentry_enabled` | boolean | no | Enable Sentry error reporting for the client side | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts | -| `default_branch_protection` | integer | no | Determine if developers can push to master. Can take `0` _(not protected, both developers and masters can push new commits, force push or delete the branch)_, `1` _(partially protected, developers can push new commits, but cannot force push or delete the branch, masters can do anything)_ or `2` _(fully protected, developers cannot push new commits, force push or delete the branch, masters can do anything)_ as a parameter. Default is `2`. | +| `default_branch_protection` | integer | no | Determine if developers can push to master. Can take `0` _(not protected, both developers and masters can push new commits, force push, or delete the branch)_, `1` _(partially protected, developers and masters can push new commits, but cannot force push or delete the branch)_ or `2` _(fully protected, developers cannot push new commits, but masters can; no-one can force push or delete the branch)_ as a parameter. Default is `2`. | | `default_group_visibility` | string | no | What visibility level new groups receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_project_visibility` | string | no | What visibility level new projects receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_projects_limit` | integer | no | Project limit per user. Default is `100000` | @@ -117,7 +116,8 @@ PUT /application/settings | `metrics_port` | integer | no | The UDP port to use for connecting to InfluxDB | | `metrics_sample_interval` | integer | yes (if `metrics_enabled` is `true`) | The sampling interval in seconds. | | `metrics_timeout` | integer | yes (if `metrics_enabled` is `true`) | The amount of seconds after which InfluxDB will time out. | -| `password_authentication_enabled` | boolean | no | Enable authentication via a GitLab account password. Default is `true`. | +| `password_authentication_enabled_for_web` | boolean | no | Enable authentication for the web interface via a GitLab account password. Default is `true`. | +| `password_authentication_enabled_for_git` | boolean | no | Enable authentication for Git over HTTP(S) via a GitLab account password. Default is `true`. | | `performance_bar_allowed_group_id` | string | no | The group that is allowed to enable the performance bar | | `performance_bar_enabled` | boolean | no | Allow enabling the performance bar | | `plantuml_enabled` | boolean | no | Enable PlantUML integration. Default is `false`. | @@ -165,7 +165,7 @@ Example response: "id": 1, "default_projects_limit": 100000, "signup_enabled": true, - "password_authentication_enabled": true, + "password_authentication_enabled_for_web": true, "gravatar_enabled": true, "sign_in_text": "", "created_at": "2015-06-12T15:51:55.432Z", diff --git a/doc/api/users.md b/doc/api/users.md index aa711090af1..478d747a50d 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -297,6 +297,7 @@ Parameters: - `location` (optional) - User's location - `admin` (optional) - User is admin - true or false (default) - `can_create_group` (optional) - User can create groups - true or false +- `skip_reconfirmation` (optional) - Skip reconfirmation - true or false (default) - `external` (optional) - Flags the user as external - true or false(default) - `avatar` (optional) - Image file for user's avatar diff --git a/doc/articles/index.md b/doc/articles/index.md index 798d4cbf4ff..862fe0868a6 100644 --- a/doc/articles/index.md +++ b/doc/articles/index.md @@ -26,6 +26,7 @@ Build, test, and deploy the software you develop with [GitLab CI/CD](../ci/READM | Article title | Category | Publishing date | | :------------ | :------: | --------------: | +| [Autoscaling GitLab Runners on AWS](runner_autoscale_aws/index.md) | Admin guide | 2017-11-24 | | [How to test and deploy Laravel/PHP applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md) | Tutorial | 2017-08-31 | | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md) | Tutorial | 2017-08-15 | | [Making CI Easier with GitLab](https://about.gitlab.com/2017/07/13/making-ci-easier-with-gitlab/) | Concepts | 2017-07-13 | diff --git a/doc/articles/runner_autoscale_aws/index.md b/doc/articles/runner_autoscale_aws/index.md new file mode 100644 index 00000000000..9d4c4a57ce5 --- /dev/null +++ b/doc/articles/runner_autoscale_aws/index.md @@ -0,0 +1,410 @@ +--- +last_updated: 2017-11-24 +--- + +> **[Article Type](../../development/writing_documentation.html#types-of-technical-articles):** Admin guide || +> **Level:** intermediary || +> **Author:** [Achilleas Pipinellis](https://gitlab.com/axil) || +> **Publication date:** 2017/11/24 + +# Autoscaling GitLab Runner on AWS + +One of the biggest advantages of GitLab Runner is its ability to automatically +spin up and down VMs to make sure your builds get processed immediately. It's a +great feature, and if used correctly, it can be extremely useful in situations +where you don't use your Runners 24/7 and want to have a cost-effective and +scalable solution. + +## Introduction + +In this tutorial, we'll explore how to properly configure a GitLab Runner in +AWS that will serve as the bastion where it will spawn new Docker machines on +demand. + +In addition, we'll make use of [Amazon's EC2 Spot instances][spot] which will +greatly reduce the costs of the Runner instances while still using quite +powerful autoscaling machines. + +## Prerequisites + +NOTE: **Note:** +A familiarity with Amazon Web Services (AWS) is required as this is where most +of the configuration will take place. + +Your GitLab instance is going to need to talk to the Runners over the network, +and that is something you need think about when configuring any AWS security +groups or when setting up your DNS configuration. + +For example, you can keep the EC2 resources segmented away from public traffic +in a different VPC to better strengthen your network security. Your environment +is likely different, so consider what works best for your situation. + +### AWS security groups + +Docker Machine will attempt to use a +[default security group](https://docs.docker.com/machine/drivers/aws/#security-group) +with rules for port `2376`, which is required for communication with the Docker +daemon. Instead of relying on Docker, you can create a security group with the +rules you need and provide that in the Runner options as we will +[see below](#the-runners-machine-section). This way, you can customize it to your +liking ahead of time based on your networking environment. + +### AWS credentials + +You'll need an [AWS Access Key](https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html) +tied to a user with permission to scale (EC2) and update the cache (via S3). +Create a new user with [policies](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-policies-for-amazon-ec2.html) +for EC2 (AmazonEC2FullAccess) and S3 (AmazonS3FullAccess). To be more secure, +you can disable console login for that user. Keep the tab open or copy paste the +security credentials in an editor as we'll use them later during the +[Runner configuration](#the-runners-machine-section). + +## Prepare the bastion instance + +The first step is to install GitLab Runner in an EC2 instance that will serve +as the bastion that spawns new machines. This doesn't have to be a powerful +machine since it will not run any jobs itself, a `t2.micro` instance will do. +This machine will be a dedicated host since we need it always up and running, +thus it will be the only standard cost. + +NOTE: **Note:** +For the bastion instance, choose a distribution that both Docker and GitLab +Runner support, for example either Ubuntu, Debian, CentOS or RHEL will work fine. + +Install the prerequisites: + +1. Log in to your server +1. [Install GitLab Runner from the official GitLab repository](https://docs.gitlab.com/runner/install/linux-repository.html) +1. [Install Docker](https://docs.docker.com/engine/installation/#server) +1. [Install Docker Machine](https://docs.docker.com/machine/install-machine/) + +Now that the Runner is installed, it's time to register it. + +## Registering the GitLab Runner + +Before configuring the GitLab Runner, you need to first register it, so that +it connects with your GitLab instance: + +1. [Obtain a Runner token](../../ci/runners/README.md) +1. [Register the Runner](https://docs.gitlab.com/runner/register/index.html#gnu-linux) +1. When asked the executor type, enter `docker+machine` + +You can now move on to the most important part, configuring the GitLab Runner. + +TIP: **Tip:** +If you want every user in your instance to be able to use the autoscaled Runners, +register the Runner as a shared one. + +## Configuring the GitLab Runner + +Now that the Runner is registered, you need to edit its configuration file and +add the required options for the AWS machine driver. + +Let's first break it down to pieces. + +### The global section + +In the global section, you can define the limit of the jobs that can be run +concurrently across all Runners (`concurrent`). This heavily depends on your +needs, like how many users your Runners will accommodate, how much time your +builds take, etc. You can start with something low like `10`, and increase or +decrease its value going forward. + +The `check_interval` option defines how often the Runner should check GitLab +for new jobs, in seconds. + +Example: + +```toml +concurrent = 10 +check_interval = 0 +``` + +[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) +about all the options you can use. + +### The `runners` section + +From the `[[runners]]` section, the most important part is the `executor` which +must be set to `docker+machine`. Most of those settings are taken care of when +you register the Runner for the first time. + +`limit` sets the maximum number of machines (running and idle) that this Runner +will spawn. For more info check the [relationship between `limit`, `concurrent` +and `IdleCount`](https://docs.gitlab.com/runner/configuration/autoscale.html#how-concurrent-limit-and-idlecount-generate-the-upper-limit-of-running-machines). + +Example: + +```toml +[[runners]] + name = "gitlab-aws-autoscaler" + url = "<URL of your GitLab instance>" + token = "<Runner's token>" + executor = "docker+machine" + limit = 20 +``` + +[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +about all the options you can use under `[[runners]]`. + +### The `runners.docker` section + +In the `[runners.docker]` section you can define the default Docker image to +be used by the child Runners if it's not defined in [`.gitlab-ci.yml`](../../ci/yaml/README.md). +By using `privileged = true`, all Runners will be able to run +[Docker in Docker](../../ci/docker/using_docker_build.md#use-docker-in-docker-executor) +which is useful if you plan to build your own Docker images via GitLab CI/CD. + +Next, we use `disable_cache = true` to disable the Docker executor's inner +cache mechanism since we will use the distributed cache mode as described +in the following section. + +Example: + +```toml + [runners.docker] + image = "alpine" + privileged = true + disable_cache = true +``` + +[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-docker-section) +about all the options you can use under `[runners.docker]`. + +### The `runners.cache` section + +To speed up your jobs, GitLab Runner provides a cache mechanism where selected +directories and/or files are saved and shared between subsequent jobs. +While not required for this setup, it is recommended to use the distributed cache +mechanism that GitLab Runner provides. Since new instances will be created on +demand, it is essential to have a common place where the cache is stored. + +In the following example, we use Amazon S3: + +```toml + [runners.cache] + Type = "s3" + ServerAddress = "s3.amazonaws.com" + AccessKey = "<your AWS Access Key ID>" + SecretKey = "<your AWS Secret Access Key>" + BucketName = "<the bucket where your cache should be kept>" + BucketLocation = "us-east-1" + Shared = true +``` + +Here's some more info to further explore the cache mechanism: + +- [Reference for `runners.cache`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-cache-section) +- [Deploying and using a cache server for GitLab Runner](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) +- [How cache works](../../ci/yaml/README.md#cache) + +### The `runners.machine` section + +This is the most important part of the configuration and it's the one that +tells GitLab Runner how and when to spawn new or remove old Docker Machine +instances. + +We will focus on the AWS machine options, for the rest of the settings read +about the: + +- [Autoscaling algorithm and the parameters it's based on](https://docs.gitlab.com/runner/configuration/autoscale.html#autoscaling-algorithm-and-parameters) - depends on the needs of your organization +- [Off peak time configuration](https://docs.gitlab.com/runner/configuration/autoscale.html#off-peak-time-mode-configuration) - useful when there are regular time periods in your organization when no work is done, for example weekends + +Here's an example of the `runners.machine` section: + +```toml + [runners.machine] + IdleCount = 1 + IdleTime = 1800 + MaxBuilds = 10 + OffPeakPeriods = [ + "* * 0-9,18-23 * * mon-fri *", + "* * * * * sat,sun *" + ] + OffPeakIdleCount = 0 + OffPeakIdleTime = 1200 + MachineDriver = "amazonec2" + MachineName = "gitlab-docker-machine-%s" + MachineOptions = [ + "amazonec2-access-key=XXXX", + "amazonec2-secret-key=XXXX", + "amazonec2-region=us-central-1", + "amazonec2-vpc-id=vpc-xxxxx", + "amazonec2-subnet-id=subnet-xxxxx", + "amazonec2-use-private-address=true", + "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", + "amazonec2-security-group=docker-machine-scaler", + "amazonec2-instance-type=m4.2xlarge", + ] +``` + +The Docker Machine driver is set to `amazonec2` and the machine name has a +standard prefix followed by `%s` (required) that is replaced by the ID of the +child Runner: `gitlab-docker-machine-%s`. + +Now, depending on your AWS infrastructure, there are many options you can set up +under `MachineOptions`. Below you can see the most common ones. + +| Machine option | Description | +| -------------- | ----------- | +| `amazonec2-access-key=XXXX` | The AWS access key of the user that has permissions to create EC2 instances, see [AWS credentials](#aws-credentials). | +| `amazonec2-secret-key=XXXX` | The AWS secret key of the user that has permissions to create EC2 instances, see [AWS credentials](#aws-credentials). | +| `amazonec2-region=eu-central-1` | The region to use when launching the instance. You can omit this entirely and the default `us-east-1` will be used. | +| `amazonec2-vpc-id=vpc-xxxxx` | Your [VPC ID](https://docs.docker.com/machine/drivers/aws/#vpc-id) to launch the instance in. | +| `amazonec2-subnet-id=subnet-xxxx` | The AWS VPC subnet ID. | +| `amazonec2-use-private-address=true` | Use the private IP address of Docker Machines, but still create a public IP address. Useful to keep the traffic internal and avoid extra costs.| +| `amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true` | AWS extra tag key-value pairs, useful to identify the instances on the AWS console. The "Name" tag is set to the machine name by default. We set the "runner-manager-name" to match the Runner name set in `[[runners]]`, so that we can filter all the EC2 instances created by a specific manager setup. Read more about [using tags in AWS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). | +| `amazonec2-security-group=docker-machine-scaler` | AWS VPC security group name, see [AWS security groups](#aws-security-groups). | +| `amazonec2-instance-type=m4.2xlarge` | The instance type that the child Runners will run on. | + +TIP: **Tip:** +Under `MachineOptions` you can add anything that the [AWS Docker Machine driver +supports](https://docs.docker.com/machine/drivers/aws/#options). You are highly +encouraged to read Docker's docs as your infrastructure setup may warrant +different options to be applied. + +NOTE: **Note:** +The child instances will use by default Ubuntu 16.04 unless you choose a +different AMI ID by setting `amazonec2-ami`. + +NOTE: **Note:** +If you specify `amazonec2-private-address-only=true` as one of the machine +options, your EC2 instance won't get assigned a public IP. This is ok if your +VPC is configured correctly with an Internet Gateway (IGW) and routing is fine, +but it’s something to consider if you've got a more complex configuration. Read +more in [Docker docs about VPC connectivity](https://docs.docker.com/machine/drivers/aws/#vpc-connectivity). + +[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-machine-section) +about all the options you can use under `[runners.machine]`. + +### Getting it all together + +Here's the full example of `/etc/gitlab-runner/config.toml`: + +```toml +concurrent = 10 +check_interval = 0 + +[[runners]] + name = "gitlab-aws-autoscaler" + url = "<URL of your GitLab instance>" + token = "<Runner's token>" + executor = "docker+machine" + limit = 20 + [runners.docker] + image = "alpine" + privileged = true + disable_cache = true + [runners.cache] + Type = "s3" + ServerAddress = "s3.amazonaws.com" + AccessKey = "<your AWS Access Key ID>" + SecretKey = "<your AWS Secret Access Key>" + BucketName = "<the bucket where your cache should be kept>" + BucketLocation = "us-east-1" + Shared = true + [runners.machine] + IdleCount = 1 + IdleTime = 1800 + MaxBuilds = 100 + OffPeakPeriods = [ + "* * 0-9,18-23 * * mon-fri *", + "* * * * * sat,sun *" + ] + OffPeakIdleCount = 0 + OffPeakIdleTime = 1200 + MachineDriver = "amazonec2" + MachineName = "gitlab-docker-machine-%s" + MachineOptions = [ + "amazonec2-access-key=XXXX", + "amazonec2-secret-key=XXXX", + "amazonec2-region=us-central-1", + "amazonec2-vpc-id=vpc-xxxxx", + "amazonec2-subnet-id=subnet-xxxxx", + "amazonec2-use-private-address=true", + "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", + "amazonec2-security-group=docker-machine-scaler", + "amazonec2-instance-type=m4.2xlarge", + ] +``` + +## Cutting down costs with Amazon EC2 Spot instances + +As [described by][spot] Amazon: + +> +Amazon EC2 Spot instances allow you to bid on spare Amazon EC2 computing capacity. +Since Spot instances are often available at a discount compared to On-Demand +pricing, you can significantly reduce the cost of running your applications, +grow your application’s compute capacity and throughput for the same budget, +and enable new types of cloud computing applications. + +In addition to the [`runners.machine`](#the-runners-machine-section) options +you picked above, in `/etc/gitlab-runner/config.toml` under the `MachineOptions` +section, add the following: + +```toml + MachineOptions = [ + "amazonec2-request-spot-instance=true", + "amazonec2-spot-price=0.03", + "amazonec2-block-duration-minutes=60" + ] +``` + +With this configuration, Docker Machines are created on Spot instances with a +maximum bid price of $0.03 per hour and the duration of the Spot instance is +capped at 60 minutes. The `0.03` number mentioned above is just an example, so +be sure to check on the current pricing based on the region you picked. + +To learn more about Amazon EC2 Spot instances, visit the following links: + +- https://aws.amazon.com/ec2/spot/ +- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html +- https://aws.amazon.com/blogs/aws/focusing-on-spot-instances-lets-talk-about-best-practices/ + +### Caveats of Spot instances + +While Spot instances is a great way to use unused resources and minimize the +costs of your infrastructure, you must be aware of the implications. + +Running CI jobs on Spot instances may increase the failure rates because of the +Spot instances pricing model. If the price exceeds your bid, the existing Spot +instances will be immediately terminated and all your jobs on that host will fail. + +As a consequence, the auto-scale Runner would fail to create new machines while +it will continue to request new instances. This eventually will make 60 requests +and then AWS won't accept any more. Then once the Spot price is acceptable, you +are locked out for a bit because the call amount limit is exceeded. + +If you encounter that case, you can use the following command in the bastion +machine to see the Docker Machines state: + +```sh +docker-machine ls -q --filter state=Error --format "{{.NAME}}" +``` + +NOTE: **Note:** +There are some issues regarding making GitLab Runner gracefully handle Spot +price changes, and there are reports of `docker-machine` attempting to +continually remove a Docker Machine. GitLab has provided patches for both cases +in the upstream project. For more information, see issues +[#2771](https://gitlab.com/gitlab-org/gitlab-runner/issues/2771) and +[#2772](https://gitlab.com/gitlab-org/gitlab-runner/issues/2772). + +## Conclusion + +In this guide we learned how to install and configure a GitLab Runner in +autoscale mode on AWS. + +Using the autoscale feature of GitLab Runner can save you both time and money. +Using the Spot instances that AWS provides can save you even more, but you must +be aware of the implications. As long as your bid is high enough, there shouldn't +be an issue. + +You can read the following use cases from which this tutorial was (heavily) +influenced: + +- [HumanGeo - Scaling GitLab CI](http://blog.thehumangeo.com/gitlab-autoscale-runners.html) +- [subtrakt Health - Autoscale GitLab CI Runners and save 90% on EC2 costs](https://substrakthealth.com/news/gitlab-ci-cost-savings/) + +[spot]: https://aws.amazon.com/ec2/spot/ diff --git a/doc/ci/autodeploy/quick_start_guide.md b/doc/ci/autodeploy/quick_start_guide.md index f76c2a2cf31..cc6c9ec0e0a 100644 --- a/doc/ci/autodeploy/quick_start_guide.md +++ b/doc/ci/autodeploy/quick_start_guide.md @@ -11,11 +11,11 @@ We made a minimal [Ruby application](https://gitlab.com/gitlab-examples/minimal- Let’s start by forking our sample application. Go to [the project page](https://gitlab.com/gitlab-examples/minimal-ruby-app) and press the `Fork` button. Soon you should have a project under your namespace with the necessary files. -## Setup your own cluster on Google Container Engine +## Setup your own cluster on Google Kubernetes Engine If you do not already have a Google Cloud account, create one at https://console.cloud.google.com. -Visit the [`Container Engine`](https://console.cloud.google.com/kubernetes/list) tab and create a new cluster. You can change the name and leave the rest of the default settings. Once you have your cluster running, you need to connect to the cluster by following the Google interface. +Visit the [`Kubernetes Engine`](https://console.cloud.google.com/kubernetes/list) tab and create a new cluster. You can change the name and leave the rest of the default settings. Once you have your cluster running, you need to connect to the cluster by following the Google interface. ## Connect to Kubernetes cluster diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index c83d3f6f248..286f3dee665 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -8,7 +8,7 @@ with the use of [SSH keys](ssh_keys/README.md). - With GitLab 8.12 onward, your permissions are used to evaluate what a CI job can access. More information about how this system works can be found in the - [Jobs permissions model](../user/permissions.md#jobs-permissions). + [Jobs permissions model](../user/permissions.md#job-permissions). - The HTTP(S) Git protocol [must be enabled][gitpro] in your GitLab instance. ## Configuring the `.gitmodules` file diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 6ad70707594..32464cbb259 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -124,7 +124,7 @@ stages: 1. First, all jobs of `build` are executed in parallel. 1. If all jobs of `build` succeed, the `test` jobs are executed in parallel. 1. If all jobs of `test` succeed, the `deploy` jobs are executed in parallel. -1. If all jobs of `deploy` succeed, the commit is marked as `success`. +1. If all jobs of `deploy` succeed, the commit is marked as `passed`. 1. If any of the previous jobs fails, the commit is marked as `failed` and no jobs of further stage are executed. @@ -1153,6 +1153,20 @@ deploy: script: make deploy ``` +#### When a dependent job will fail + +> Introduced in GitLab 10.3. + +If the artifacts of the job that is set as a dependency have been +[expired](#artifacts-expire_in) or +[erased](../../user/project/pipelines/job_artifacts.md#erasing-artifacts), then +the dependent job will fail. + +NOTE: **Note:** +You can ask your administrator to +[flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) +and bring back the old behavior. + ### before_script and after_script It's possible to overwrite the globally defined `before_script` and `after_script`: diff --git a/doc/customization/new_project_page.md b/doc/customization/new_project_page.md new file mode 100644 index 00000000000..148bf9512c6 --- /dev/null +++ b/doc/customization/new_project_page.md @@ -0,0 +1,20 @@ +# Customizing the new project page + +It is possible to add a markdown-formatted message to your GitLab +new project page. + +By default, the new project page shows a sidebar with general information: + + + +## Changing the appearance of the new project page + +Navigate to the **Admin** area and go to the **Appearance** page. + +Fill in your project guidelines: + + + +After saving the page, your new project page will show the guidelines in the sidebar, below the general information: + + diff --git a/doc/customization/new_project_page/appearance_settings.png b/doc/customization/new_project_page/appearance_settings.png Binary files differnew file mode 100644 index 00000000000..08eea684e14 --- /dev/null +++ b/doc/customization/new_project_page/appearance_settings.png diff --git a/doc/customization/new_project_page/custom_new_project_page.png b/doc/customization/new_project_page/custom_new_project_page.png Binary files differnew file mode 100644 index 00000000000..662c715f193 --- /dev/null +++ b/doc/customization/new_project_page/custom_new_project_page.png diff --git a/doc/customization/new_project_page/default_new_project_page.png b/doc/customization/new_project_page/default_new_project_page.png Binary files differnew file mode 100644 index 00000000000..4a0bcf09903 --- /dev/null +++ b/doc/customization/new_project_page/default_new_project_page.png diff --git a/doc/development/README.md b/doc/development/README.md index 0cafc112b6b..6892838be7f 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -22,6 +22,7 @@ comments: false - [UX guide](ux_guide/index.md) for building GitLab with existing CSS styles and elements - [Frontend guidelines](fe_guide/index.md) +- [Emoji guide](fe_guide/emojis.md) ## Backend guides @@ -37,6 +38,7 @@ comments: false - [Gotchas](gotchas.md) to avoid - [Issue and merge requests state models](object_state_models.md) - [How to dump production data to staging](db_dump.md) +- [Working with the GitHub importer](github_importer.md) ## Performance guides @@ -69,6 +71,7 @@ comments: false - [Iterating tables in batches](iterating_tables_in_batches.md) - [Ordering table columns](ordering_table_columns.md) - [Verifying database capabilities](verifying_database_capabilities.md) +- [Database Debugging and Troubleshooting](database_debugging.md) ## Testing guides diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 5452b0e7a2f..fd2b9d0e908 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -68,10 +68,10 @@ BackgroundMigrationWorker.perform_async('BackgroundMigrationClassName', [arg1, a ``` Usually it's better to enqueue jobs in bulk, for this you can use -`BackgroundMigrationWorker.perform_bulk`: +`BackgroundMigrationWorker.bulk_perform_async`: ```ruby -BackgroundMigrationWorker.perform_bulk( +BackgroundMigrationWorker.bulk_perform_async( [['BackgroundMigrationClassName', [1]], ['BackgroundMigrationClassName', [2]]] ) @@ -85,13 +85,13 @@ updates. Removals in turn can be handled by simply defining foreign keys with cascading deletes. If you would like to schedule jobs in bulk with a delay, you can use -`BackgroundMigrationWorker.perform_bulk_in`: +`BackgroundMigrationWorker.bulk_perform_in`: ```ruby jobs = [['BackgroundMigrationClassName', [1]], ['BackgroundMigrationClassName', [2]]] -BackgroundMigrationWorker.perform_bulk_in(5.minutes, jobs) +BackgroundMigrationWorker.bulk_perform_in(5.minutes, jobs) ``` ## Cleaning Up @@ -201,7 +201,7 @@ class ScheduleExtractServicesUrl < ActiveRecord::Migration ['ExtractServicesUrl', [id]] end - BackgroundMigrationWorker.perform_bulk(jobs) + BackgroundMigrationWorker.bulk_perform_async(jobs) end end diff --git a/doc/development/changelog.md b/doc/development/changelog.md index f869938fe11..48cffc0dd18 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -80,7 +80,7 @@ changes. The first example focuses on _how_ we fixed something, not on _what_ it fixes. The rewritten version clearly describes the _end benefit_ to the user (fewer 500 -errors), and _when_ (searching commits with ElasticSearch). +errors), and _when_ (searching commits with Elasticsearch). Use your best judgement and try to put yourself in the mindset of someone reading the compiled changelog. Does this entry add value? Does it offer context diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md new file mode 100644 index 00000000000..50eb8005b44 --- /dev/null +++ b/doc/development/database_debugging.md @@ -0,0 +1,55 @@ +# Database Debugging and Troubleshooting + +This section is to help give some copy-pasta you can use as a reference when you +run into some head-banging database problems. + +An easy first step is to search for your error in Slack or google "GitLab <my error>". + +--- + +Available `RAILS_ENV` + + - `production` (generally not for your main GDK db, but you may need this for e.g. omnibus) + - `development` (this is your main GDK db) + - `test` (used for tests like rspec and spinach) + + +## Nuke everything and start over + +If you just want to delete everything and start over with an empty DB (~1 minute): + + - `bundle exec rake db:reset RAILS_ENV=development` + +If you just want to delete everything and start over with dummy data (~40 minutes). This also does `db:reset` and runs DB-specific migrations: + + - `bundle exec rake dev:setup RAILS_ENV=development` + +If your test DB is giving you problems, it is safe to nuke it because it doesn't contain important data: + + - `bundle exec rake db:reset RAILS_ENV=test` + +## Migration wrangling + + - `bundle exec rake db:migrate RAILS_ENV=development`: Execute any pending migrations that you may have picked up from a MR + - `bundle exec rake db:migrate:status RAILS_ENV=development`: Check if all migrations are `up` or `down` + - `bundle exec rake db:migrate:down VERSION=20170926203418 RAILS_ENV=development`: Tear down a migration + - `bundle exec rake db:migrate:up VERSION=20170926203418 RAILS_ENV=development`: Setup a migration + - `bundle exec rake db:migrate:redo VERSION=20170926203418 RAILS_ENV=development`: Re-run a specific migration + + +## Manually access the database + +Access the database via one of these commands (they all get you to the same place) + +``` +gdk psql -d gitlabhq_development +bundle exec rails dbconsole RAILS_ENV=development +bundle exec rails db RAILS_ENV=development +``` + + - `\q`: Quit/exit + - `\dt`: List all tables + - `\d+ issues`: List columns for `issues` table + - `CREATE TABLE board_labels();`: Create a table called `board_labels` + - `SELECT * FROM schema_migrations WHERE version = '20170926203418';`: Check if a migration was run + - `DELETE FROM schema_migrations WHERE version = '20170926203418';`: Manually remove a migration diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md index 0e4ffbd7910..db13e0e6249 100644 --- a/doc/development/doc_styleguide.md +++ b/doc/development/doc_styleguide.md @@ -170,12 +170,6 @@ You can combine one or more of the following: = link_to 'Help page', help_page_path('user/permissions'), class: 'btn btn-info' ``` -1. **Underlining a link.** - - ```haml - = link_to 'Help page', help_page_path('user/permissions'), class: 'underlined-link' - ``` - 1. **Using links inline of some text.** ```haml @@ -303,10 +297,10 @@ GitLab.com or http://docs.gitlab.com. Make sure this is discussed with the Documentation team beforehand. If you indeed need to change a document's location, do NOT remove the old -document, but rather put a text in it that points to the new location, like: +document, but rather replace all of its contents with a new line: ``` -This document was moved to [path/to/new_doc.md](path/to/new_doc.md). +This document was moved to [another location](path/to/new_doc.md). ``` where `path/to/new_doc.md` is the relative path to the root directory `doc/`. @@ -320,7 +314,7 @@ For example, if you were to move `doc/workflow/lfs/lfs_administration.md` to 1. Replace the contents of `doc/workflow/lfs/lfs_administration.md` with: ``` - This document was moved to [administration/lfs.md](../../administration/lfs.md). + This document was moved to [another location](../../administration/lfs.md). ``` 1. Find and replace any occurrences of the old location with the new one. diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md new file mode 100644 index 00000000000..962fe3dcec9 --- /dev/null +++ b/doc/development/fe_guide/axios.md @@ -0,0 +1,68 @@ +# Axios +We use [axios][axios] to communicate with the server in Vue applications and most new code. + +In order to guarantee all defaults are set you *should not use `axios` directly*, you should import `axios` from `axios_utils`. + +## CSRF token +All our request require a CSRF token. +To guarantee this token is set, we are importing [axios][axios], setting the token, and exporting `axios` . + +This exported module should be used instead of directly using `axios` to ensure the token is set. + +## Usage +```javascript + import axios from '~/lib/utils/axios_utils'; + + axios.get(url) + .then((response) => { + // `data` is the response that was provided by the server + const data = response.data; + + // `headers` the headers that the server responded with + // All header names are lower cased + const paginationData = response.headers; + }) + .catch(() => { + //handle the error + }); +``` + +## Mock axios response on tests + +To help us mock the responses we need we use [axios-mock-adapter][axios-mock-adapter] + + +```javascript + import axios from '~/lib/utils/axios_utils'; + import MockAdapter from 'axios-mock-adapter'; + + let mock; + beforeEach(() => { + // This sets the mock adapter on the default instance + mock = new MockAdapter(axios); + // Mock any GET request to /users + // arguments for reply are (status, data, headers) + mock.onGet('/users').reply(200, { + users: [ + { id: 1, name: 'John Smith' } + ] + }); + }); + + afterEach(() => { + mock.reset(); + }); +``` + +### Mock poll requests on tests with axios + +Because polling function requires an header object, we need to always include an object as the third argument: + +```javascript + mock.onGet('/users').reply(200, { foo: 'bar' }, {}); +``` + +[axios]: https://github.com/axios/axios +[axios-instance]: #creating-an-instance +[axios-interceptors]: https://github.com/axios/axios#interceptors +[axios-mock-adapter]: https://github.com/ctimmerm/axios-mock-adapter diff --git a/doc/development/fe_guide/dropdowns.md b/doc/development/fe_guide/dropdowns.md new file mode 100644 index 00000000000..e1660ac5caa --- /dev/null +++ b/doc/development/fe_guide/dropdowns.md @@ -0,0 +1,38 @@ +# Dropdowns + + +## How to style a bootstrap dropdown +1. Use the HTML structure provided by the [docs][bootstrap-dropdowns] +1. Add a specific class to the top level `.dropdown` element + + + ```Haml + .dropdown.my-dropdown + %button{ type: 'button', data: { toggle: 'dropdown' }, 'aria-haspopup': true, 'aria-expanded': false } + %span.dropdown-toggle-text + Toggle Dropdown + = icon('chevron-down') + + %ul.dropdown-menu + %li + %a + item! + ``` + + Or use the helpers + ```Haml + .dropdown.my-dropdown + = dropdown_toggle('Toogle!', { toggle: 'dropdown' }) + = dropdown_content + %li + %a + item! + ``` + +1. Include the mixin in CSS + + ```SCSS + @include new-style-dropdown('.my-dropdown '); + ``` + +[bootstrap-dropdowns]: https://getbootstrap.com/docs/3.3/javascript/#dropdowns diff --git a/doc/development/fe_guide/emojis.md b/doc/development/fe_guide/emojis.md new file mode 100644 index 00000000000..38794c47965 --- /dev/null +++ b/doc/development/fe_guide/emojis.md @@ -0,0 +1,27 @@ +# Emojis + +GitLab supports native unicode emojis and fallsback to image-based emojis selectively +when your platform does not support it. + +# How to update Emojis + + 1. Update the `gemojione` gem + 1. Update `fixtures/emojis/index.json` from [Gemojione](https://github.com/jonathanwiesel/gemojione/blob/master/config/index.json). + In the future, we could grab the file directly from the gem. + We should probably make a PR on the Gemojione project to get access to + all emojis after being parsed or just a raw path to the `json` file itself. + 1. Ensure [`emoji-unicode-version`](https://www.npmjs.com/package/emoji-unicode-version) + is up to date with the latest version. + 1. Run `bundle exec rake gemojione:aliases` + 1. Run `bundle exec rake gemojione:digests` + 1. Run `bundle exec rake gemojione:sprite` + 1. Ensure new sprite sheets generated for 1x and 2x + - `app/assets/images/emoji.png` + - `app/assets/images/emoji@2x.png` + 1. Ensure you see new individual images copied into `app/assets/images/emoji/` + 1. Ensure you can see the new emojis and their aliases in the GFM Autocomplete + 1. Ensure you can see the new emojis and their aliases in the award emoji menu + 1. You might need to add new emoji unicode support checks and rules for platforms + that do not support a certain emoji and we need to fallback to an image. + See `app/assets/javascripts/emoji/support/is_emoji_unicode_supported.js` + and `app/assets/javascripts/emoji/support/unicode_support_map.js` diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index a76e978bd26..b288ee95722 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -4,15 +4,17 @@ We are using SVG Icons in GitLab with a SVG Sprite, due to this the icons are on ### Usage in HAML/Rails -To use a sprite Icon in HAML or Rails we use a specific helper function : +To use a sprite Icon in HAML or Rails we use a specific helper function : `sprite_icon(icon_name, size: nil, css_class: '')` -**icon_name** Use the icon_name that you can find in the SVG Sprite (Overview is available under `/assets/sprite.symbol.html`). +**icon_name** Use the icon_name that you can find in the SVG Sprite ([Overview is available here](http://gitlab-org.gitlab.io/gitlab-svgs/)`). + **size (optional)** Use one of the following sizes : 16,24,32,48,72 (this will be translated into a `s16` class) + **css_class (optional)** If you want to add additional css classes -**Example** +**Example** `= sprite_icon('issues', size: 72, css_class: 'icon-danger')` @@ -20,16 +22,34 @@ To use a sprite Icon in HAML or Rails we use a specific helper function : `<svg class="s72 icon-danger"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/assets/icons.svg#issues"></use></svg>` +### Usage in Vue + +We have a special Vue component for our sprite icons in `\vue_shared\components\icon.vue`. + +Sample usage : + +`<icon + name="retry" + :size="32" + css-classes="top" + />` + +**name** Name of the Icon in the SVG Sprite ([Overview is available here](http://gitlab-org.gitlab.io/gitlab-svgs/)`). + +**size (optional)** Number value for the size which is then mapped to a specific CSS class (Available Sizes: 8,12,16,18,24,32,48,72 are mapped to `sXX` css classes) + +**css-classes (optional)** Additional CSS Classes to add to the svg tag. + ### Usage in HTML/JS -Please use the following function inside JS to render an icon : +Please use the following function inside JS to render an icon : `gl.utils.spriteIcon(iconName)` ## Adding a new icon to the sprite All Icons and Illustrations are managed in the [gitlab-svgs](https://gitlab.com/gitlab-org/gitlab-svgs) repository which is added as a dev-dependency. -To upgrade to a new SVG Sprite version run `yarn upgrade https://gitlab.com/gitlab-org/gitlab-svgs` and then run `yarn run svg`. This task will copy the svg sprite and all illustrations in the correct folders. +To upgrade to a new SVG Sprite version run `yarn upgrade @gitlab-org/gitlab-svgs` and then run `yarn run svg`. This task will copy the svg sprite and all illustrations in the correct folders. The updated files should be tracked in Git as those are referenced. # SVG Illustrations diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 8f956681693..72cb557d054 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -71,12 +71,14 @@ Vue specific design patterns and practices. --- -## [Vue Resource](vue_resource.md) -Vue resource specific practices and gotchas. +## [Axios](axios.md) +Axios specific practices and gotchas. ## [Icons](icons.md) How we use SVG for our Icons. +## [Dropdowns](dropdowns.md) +How we use dropdowns. --- ## Style Guides diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index f88f0753687..6e9f18dd1c3 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -178,16 +178,13 @@ itself, please read this guide: [State Management][state-management] The Service is a class used only to communicate with the server. It does not store or manipulate any data. It is not aware of the store or the components. -We use [vue-resource][vue-resource-repo] to communicate with the server. -Refer to [vue resource](vue_resource.md) for more details. +We use [axios][axios] to communicate with the server. +Refer to [axios](axios.md) for more details. -Vue Resource should only be imported in the service file. +Axios instance should only be imported in the service file. ```javascript - import Vue from 'vue'; - import VueResource from 'vue-resource'; - - Vue.use(VueResource); + import axios from 'javascripts/lib/utils/axios_utils'; ``` ### End Result @@ -230,15 +227,14 @@ export default class Store { } // service.js -import Vue from 'vue'; -import VueResource from 'vue-resource'; -import 'vue_shared/vue_resource_interceptor'; - -Vue.use(VueResource); +import axios from 'javascripts/lib/utils/axios_utils' export default class Service { constructor(options) { - this.todos = Vue.resource(endpoint.todosEndpoint); + this.todos = axios.create({ + baseURL: endpoint.todosEndpoint + }); + } getTodos() { @@ -477,50 +473,8 @@ The main return value of a Vue component is the rendered output. In order to tes need to test the rendered output. [Vue][vue-test] guide's to unit test show us exactly that: ### Stubbing API responses -[Vue Resource Interceptors][vue-resource-interceptor] allow us to add a interceptor with -the response we need: - - ```javascript - // Mock the service to return data - const interceptor = (request, next) => { - next(request.respondWith(JSON.stringify([{ - title: 'This is a todo', - body: 'This is the text' - }]), { - status: 200, - })); - }; +Refer to [mock axios](axios.md#mock-axios-response-on-tests) - beforeEach(() => { - Vue.http.interceptors.push(interceptor); - }); - - afterEach(() => { - Vue.http.interceptors = _.without(Vue.http.interceptors, interceptor); - }); - - it('should do something', (done) => { - setTimeout(() => { - // Test received data - done(); - }, 0); - }); - ``` - -1. Headers interceptor -Refer to [this section](vue.md#headers) - -1. Use `$.mount()` to mount the component - -```javascript -// bad -new Component({ - el: document.createElement('div') -}); - -// good -new Component().$mount(); -``` ## Vuex To manage the state of an application you may use [Vuex][vuex-docs]. @@ -721,7 +675,6 @@ describe('component', () => { [component-system]: https://vuejs.org/v2/guide/#Composing-with-Components [state-management]: https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch [one-way-data-flow]: https://vuejs.org/v2/guide/components.html#One-Way-Data-Flow -[vue-resource-interceptor]: https://github.com/pagekit/vue-resource/blob/develop/docs/http.md#interceptors [vue-test]: https://vuejs.org/v2/guide/unit-testing.html [issue-boards-service]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/boards/services/board_service.js.es6 [flux]: https://facebook.github.io/flux @@ -729,3 +682,6 @@ describe('component', () => { [vuex-structure]: https://vuex.vuejs.org/en/structure.html [vuex-mutations]: https://vuex.vuejs.org/en/mutations.html [vuex-testing]: https://vuex.vuejs.org/en/testing.html +[axios]: https://github.com/axios/axios +[axios-interceptors]: https://github.com/axios/axios#interceptors + diff --git a/doc/development/fe_guide/vue_resource.md b/doc/development/fe_guide/vue_resource.md deleted file mode 100644 index c376c5c32bf..00000000000 --- a/doc/development/fe_guide/vue_resource.md +++ /dev/null @@ -1,72 +0,0 @@ -# Vue Resouce -In Vue applications we use [vue-resource][vue-resource-repo] to communicate with the server. - -## HTTP Status Codes - -### `.json()` -When making a request to the server, you will most likely need to access the body of the response. -Use `.json()` to convert. Because `.json()` returns a Promise the follwoing structure should be used: - - ```javascript - service.get('url') - .then(resp => resp.json()) - .then((data) => { - this.store.storeData(data); - }) - .catch(() => new Flash('Something went wrong')); - ``` - - -When using `Poll` (`app/assets/javascripts/lib/utils/poll.js`), the `successCallback` needs to handle `.json()` as a Promise: - ```javascript - successCallback: (response) => { - return response.json().then((data) => { - // handle the response - }); - } - ``` - -### 204 -Some endpoints - usually `delete` endpoints - return `204` as the success response. -When handling `204 - No Content` responses, we cannot use `.json()` since it tries to parse the non-existant body content. - -When handling `204` responses, do not use `.json`, otherwise the promise will throw an error and will enter the `catch` statement: - -```javascript - Vue.http.delete('path') - .then(() => { - // success! - }) - .catch(() => { - // handle error - }) -``` - -## Headers -Headers are being parsed into a plain object in an interceptor. -In Vue-resource 1.x `headers` object was changed into an `Headers` object. In order to not change all old code, an interceptor was added. - -If you need to write a unit test that takes the headers in consideration, you need to include an interceptor to parse the headers after your test interceptor. -You can see an example in `spec/javascripts/environments/environment_spec.js`: - ```javascript - import { headersInterceptor } from './helpers/vue_resource_helper'; - - beforeEach(() => { - Vue.http.interceptors.push(myInterceptor); - Vue.http.interceptors.push(headersInterceptor); - }); - - afterEach(() => { - Vue.http.interceptors = _.without(Vue.http.interceptors, myInterceptor); - Vue.http.interceptors = _.without(Vue.http.interceptors, headersInterceptor); - }); - ``` - -## CSRF token -We use a Vue Resource interceptor to manage the CSRF token. -`app/assets/javascripts/vue_shared/vue_resource_interceptor.js` holds all our common interceptors. -Note: You don't need to load `app/assets/javascripts/vue_shared/vue_resource_interceptor.js` -since it's already being loaded by `common_vue.js`. - - -[vue-resource-repo]: https://github.com/pagekit/vue-resource diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md new file mode 100644 index 00000000000..cf00e24e11a --- /dev/null +++ b/doc/development/file_storage.md @@ -0,0 +1,49 @@ +# File Storage in GitLab + +We use the [CarrierWave] gem to handle file upload, store and retrieval. + +There are many places where file uploading is used, according to contexts: + +* System + - Instance Logo (logo visible in sign in/sign up pages) + - Header Logo (one displayed in the navigation bar) +* Group + - Group avatars +* User + - User avatars + - User snippet attachments +* Project + - Project avatars + - Issues/MR Markdown attachments + - Issues/MR Legacy Markdown attachments + - CI Build Artifacts + - LFS Objects + + +## Disk storage + +GitLab started saving everything on local disk. While directory location changed from previous versions, +they are still not 100% standardized. You can see them below: + +| Description | In DB? | Relative path | Uploader class | model_type | +| ------------------------------------- | ------ | ----------------------------------------------------------- | ---------------------- | ---------- | +| Instance logo | yes | uploads/-/system/appearance/logo/:id/:filename | `AttachmentUploader` | Appearance | +| Header logo | yes | uploads/-/system/appearance/header_logo/:id/:filename | `AttachmentUploader` | Appearance | +| Group avatars | yes | uploads/-/system/group/avatar/:id/:filename | `AvatarUploader` | Group | +| User avatars | yes | uploads/-/system/user/avatar/:id/:filename | `AvatarUploader` | User | +| User snippet attachments | yes | uploads/-/system/personal_snippet/:id/:random_hex/:filename | `PersonalFileUploader` | Snippet | +| Project avatars | yes | uploads/-/system/project/avatar/:id/:filename | `AvatarUploader` | Project | +| Issues/MR Markdown attachments | yes | uploads/:project_path_with_namespace/:random_hex/:filename | `FileUploader` | Project | +| Issues/MR Legacy Markdown attachments | no | uploads/-/system/note/attachment/:id/:filename | `AttachmentUploader` | Note | +| CI Artifacts (CE) | yes | shared/artifacts/:year_:month/:project_id/:id | `ArtifactUploader` | Ci::Build | +| LFS Objects (CE) | yes | shared/lfs-objects/:hex/:hex/:object_hash | `LfsObjectUploader` | LfsObject | + +CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader` +while in EE they inherit the `ObjectStoreUploader` and store files in and S3 API compatible object store. + +In the case of Issues/MR Markdown attachments, there is a different approach using the [Hashed Storage] layout, +instead of basing the path into a mutable variable `:project_path_with_namespace`, it's possible to use the +hash of the project ID instead, if project migrates to the new approach (introduced in 10.2). + +[CarrierWave]: https://github.com/carrierwaveuploader/carrierwave +[Hashed Storage]: ../administration/repository_storage_types.md diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md new file mode 100644 index 00000000000..0d558583bb8 --- /dev/null +++ b/doc/development/github_importer.md @@ -0,0 +1,209 @@ +# Working with the GitHub importer + +In GitLab 10.2 a new version of the GitHub importer was introduced. This new +importer performs its work in parallel using Sidekiq, greatly reducing the time +necessary to import GitHub projects into a GitLab instance. + +The GitHub importer offers two different types of importers: a sequential +importer and a parallel importer. The Rake task `import:github` uses the +sequential importer, while everything else uses the parallel importer. The +difference between these two importers is quite simple: the sequential importer +does all work in a single thread, making it more useful for debugging purposes +or Rake tasks. The parallel importer on the other hand uses Sidekiq. + +## Requirements + +* GitLab CE 10.2.0 or newer. +* Sidekiq workers that process the `github_importer` and + `github_importer_advance_stage` queues (this is enabled by default). +* Octokit (used for interacting with the GitHub API) + +## Code structure + +The importer's codebase is broken up into the following directories: + +* `lib/gitlab/github_import`: this directory contains most of the code such as + the classes used for importing resources. +* `app/workers/gitlab/github_import`: this directory contains the Sidekiq + workers. +* `app/workers/concerns/gitlab/github_import`: this directory contains a few + modules reused by the various Sidekiq workers. + +## Architecture overview + +When a GitHub project is imported we schedule and execute a job for the +`RepositoryImportworker` worker as all other importers. However, unlike other +importers we don't immediately perform the work necessary. Instead work is +divided into separate stages, with each stage consisting out of a set of Sidekiq +jobs that are executed. Between every stage a job is scheduled that periodically +checks if all work of the current stage is completed, advancing the import +process to the next stage when this is the case. The worker handling this is +called `Gitlab::GithubImport::AdvanceStageWorker`. + +## Stages + +### 1. RepositoryImportWorker + +This worker will kick off the import process by simply scheduling a job for the +next worker. + +### 2. Stage::ImportRepositoryWorker + +This worker will import the repository and wiki, scheduling the next stage when +done. + +### 3. Stage::ImportBaseDataWorker + +This worker will import base data such as labels, milestones, and releases. This +work is done in a single thread since it can be performed fast enough that we +don't need to perform this work in parallel. + +### 4. Stage::ImportPullRequestsWorker + +This worker will import all pull requests. For every pull request a job for the +`Gitlab::GithubImport::ImportPullRequestWorker` worker is scheduled. + +### 5. Stage::ImportIssuesAndDiffNotesWorker + +This worker will import all issues and pull request comments. For every issue we +schedule a job for the `Gitlab::GithubImport::ImportIssueWorker` worker. For +pull request comments we instead schedule jobs for the +`Gitlab::GithubImport::DiffNoteImporter` worker. + +This worker processes both issues and diff notes in parallel so we don't need to +schedule a separate stage and wait for the previous one to complete. + +Issues are imported separately from pull requests because only the "issues" API +includes labels for both issue and pull requests. Importing issues and setting +label links in the same worker removes the need for performing a separate crawl +through the API data, reducing the number of API calls necessary to import a +project. + +### 6. Stage::ImportNotesWorker + +This worker imports regular comments for both issues and pull requests. For +every comment we schedule a job for the +`Gitlab::GithubImport::ImportNoteWorker` worker. + +Regular comments have to be imported at the end since the GitHub API used +returns comments for both issues and pull requests. This means we have to wait +for all issues and pull requests to be imported before we can import regular +comments. + +### 7. Stage::FinishImportWorker + +This worker will wrap up the import process by performing some housekeeping +(such as flushing any caches) and by marking the import as completed. + +## Advancing stages + +Advancing stages is done in one of two ways: + +1. Scheduling the worker for the next stage directly. +2. Scheduling a job for `Gitlab::GithubImport::AdvanceStageWorker` which will + advance the stage when all work of the current stage has been completed. + +The first approach should only be used by workers that perform all their work in +a single thread, while `AdvanceStageWorker` should be used for everything else. + +The way `AdvanceStageWorker` works is fairly simple. When scheduling a job it +will be given a project ID, a list of Redis keys, and the name of the next +stage. The Redis keys (produced by `Gitlab::JobWaiter`) are used to check if the +currently running stage has been completed or not. If the stage has not yet been +completed `AdvanceStageWorker` will reschedule itself. Once a stage finishes +`AdvanceStageworker` will refresh the import JID (more on this below) and +schedule the worker of the next stage. + +To reduce the number of `AdvanceStageWorker` jobs scheduled this worker will +briefly wait for jobs to complete before deciding what the next action should +be. For small projects this may slow down the import process a bit, but it will +also reduce pressure on the system as a whole. + +## Refreshing import JIDs + +GitLab includes a worker called `StuckImportJobsWorker` that will periodically +run and mark project imports as failed if they have been running for more than +15 hours. For GitHub projects this poses a bit of a problem: importing large +projects could take several hours depending on how often we hit the GitHub rate +limit (more on this below), but we don't want `StuckImportJobsWorker` to mark +our import as failed because of this. + +To prevent this from happening we periodically refresh the expiration time of +the import process. This works by storing the JID of the import job in the +database, then refreshing this JID's TTL at various stages throughout the import +process. This is done by calling `Project#refresh_import_jid_expiration`. By +refreshing this TTL we can ensure our import does not get marked as failed so +long we're still performing work. + +## GitHub rate limit + +GitHub has a rate limit of 5 000 API calls per hour. The number of requests +necessary to import a project is largely dominated by the number of unique users +involved in a project (e.g. issue authors). Other data such as issue pages +and comments typically only requires a few dozen requests to import. This is +because we need the Email address of users in order to map them to GitLab users. + +We handle this by doing the following: + +1. Once we hit the rate limit all jobs will automatically reschedule themselves + in such a way that they are not executed until the rate limit has been reset. +2. We cache the mapping of GitHub users to GitLab users in Redis. + +More information on user caching can be found below. + +## Caching user lookups + +When mapping GitHub users to GitLab users we need to (in the worst case) +perform: + +1. One API call to get the user's Email address. +2. Two database queries to see if a corresponding GitLab user exists. One query + will try to find the user based on the GitHub user ID, while the second query + is used to find the user using their GitHub Email address. + +Because this process is quite expensive we cache the result of these lookups in +Redis. For every user looked up we store three keys: + +1. A Redis key mapping GitHub usernames to their Email addresses. +2. A Redis key mapping a GitHub Email addresses to a GitLab user ID. +3. A Redis key mapping a GitHub user ID to GitLab user ID. + +There are two types of lookups we cache: + +1. A positive lookup, meaning we found a GitLab user ID. +2. A negative lookup, meaning we didn't find a GitLab user ID. Caching this + prevents us from performing the same work for users that we know don't exist + in our GitLab database. + +The expiration time of these keys is 24 hours. When retrieving the cache of a +positive lookups we refresh the TTL automatically. The TTL of false lookups is +never refreshed. + +Because of this caching layer it's possible newly registered GitLab accounts +won't be linked to their corresponding GitHub accounts. This however will sort +itself out once the cached keys expire. + +The user cache lookup is shared across projects. This means that the more +projects get imported the fewer GitHub API calls will be needed. + +The code for this resides in: + +* `lib/gitlab/github_import/user_finder.rb` +* `lib/gitlab/github_import/caching.rb` + +## Mapping labels and milestones + +To reduce pressure on the database we do not query it when setting labels and +milestones on issues and merge requests. Instead we cache this data when we +import labels and milestones, then we reuse this cache when assigning them to +issues/merge requests. Similar to the user lookups these cache keys are expired +automatically after 24 hours of not being used. + +Unlike the user lookup caches these label and milestone caches are scoped to the +project that is being imported. + +The code for this resides in: + +* `lib/gitlab/github_import/label_finder.rb` +* `lib/gitlab/github_import/milestone_finder.rb` +* `lib/gitlab/github_import/caching.rb` diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 7c38260406d..4b65a0f4a35 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -110,7 +110,7 @@ You can mark that content for translation with: In JavaScript we added the `__()` (double underscore parenthesis) function for translations. -### Updating the PO files with the new content +## Updating the PO files with the new content Now that the new content is marked for translation, we need to update the PO files with the following command: @@ -119,23 +119,20 @@ files with the following command: bundle exec rake gettext:find ``` -This command will update the `locale/**/gitlab.edit.po` file with the -new content that the parser has found. +This command will update the `locale/gitlab.pot` file with the newly externalized +strings and remove any strings that aren't used anymore. You should check this +file in. Once the changes are on master, they will be picked up by +[Crowdin](http://translate.gitlab.com) and be presented for translation. -New translations will be added with their default content and will be marked -fuzzy. To use the translation, look for the `#, fuzzy` mention in `gitlab.edit.po` -and remove it. +The command also updates the translation files for each language: `locale/*/gitlab.po` +These changes can be discarded, the languange files will be updated by Crowdin +automatically. -We need to make sure we remove the `fuzzy` translations before generating the -`locale/**/gitlab.po` file. When they aren't removed, the resulting `.po` will -be treated as a binary file which could overwrite translations that were merged -before the new translations. +Discard all of them at once like this: -When we are just preparing a page to be translated, but not actually adding any -translations. There's no need to generate `.po` files. - -Translations that aren't used in the source code anymore will be marked with -`~#`; these can be removed to keep our translation files clutter-free. +```sh +git checkout locale/*/gitlab.po +``` ### Validating PO files diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 902b1c74a42..274923c2d43 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -4,11 +4,11 @@ GitLab CE is licensed under the terms of the MIT License. GitLab EE is licensed ## Automated Testing -In order to comply with the terms the libraries we use are licensed under, we have to make sure to check new gems for compatible licenses whenever they're added. To automate this process, we use the [license_finder][license_finder] gem by Pivotal. It runs every time a new commit is pushed and verifies that all gems in the bundle use a license that doesn't conflict with the licensing of either GitLab Community Edition or GitLab Enterprise Edition. +In order to comply with the terms the libraries we use are licensed under, we have to make sure to check new gems for compatible licenses whenever they're added. To automate this process, we use the [license_finder][license_finder] gem by Pivotal. It runs every time a new commit is pushed and verifies that all gems and node modules in the bundle use a license that doesn't conflict with the licensing of either GitLab Community Edition or GitLab Enterprise Edition. -There are some limitations with the automated testing, however. CSS and JavaScript libraries, as well as any Ruby libraries not included by way of Bundler, must be verified manually and independently. Take care whenever one such library is used, as automated tests won't catch problematic licenses from them. +There are some limitations with the automated testing, however. CSS, JavaScript, or Ruby libraries which are not included by way of Bundler, NPM, or Yarn (for instance those manually copied into our source tree in the `vendor` directory), must be verified manually and independently. Take care whenever one such library is used, as automated tests won't catch problematic licenses from them. -Some gems may not include their license information in their `gemspec` file. These won't be detected by License Finder, and will have to be verified manually. +Some gems may not include their license information in their `gemspec` file, and some node modules may not include their license information in their `package.json` file. These won't be detected by License Finder, and will have to be verified manually. ### License Finder commands diff --git a/doc/development/limit_ee_conflicts.md b/doc/development/limit_ee_conflicts.md index 899be9eae4b..ba82babb38a 100644 --- a/doc/development/limit_ee_conflicts.md +++ b/doc/development/limit_ee_conflicts.md @@ -336,6 +336,12 @@ Blocks of code that are EE-specific should be moved to partials as much as possible to avoid conflicts with big chunks of HAML code that that are not fun to resolve when you add the indentation in the equation. +### Assets + +#### gitlab-svgs + +Conflicts in `app/assets/images/icons.json` or `app/assets/images/icons.svg` can be resolved simply by regenerating those assets with [`yarn run svg`](https://gitlab.com/gitlab-org/gitlab-svgs). + --- [Return to Development documentation](README.md) diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 9b8ab5da74e..a235dd74909 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -198,7 +198,43 @@ end Keep in mind that this operation can easily take 10-15 minutes to complete on larger installations (e.g. GitLab.com). As a result you should only add default -values if absolutely necessary. +values if absolutely necessary. There is a RuboCop cop that will fail if this +method is used on some tables that are very large on GitLab.com, which would +cause other issues. + +## Updating an existing column + +To update an existing column to a particular value, you can use +`update_column_in_batches` (`add_column_with_default` uses this internally to +fill in the default value). This will split the updates into batches, so we +don't update too many rows at in a single statement. + +This updates the column `foo` in the `projects` table to 10, where `some_column` +is `'hello'`: + +```ruby +update_column_in_batches(:projects, :foo, 10) do |table, query| + query.where(table[:some_column].eq('hello')) +end +``` + +To perform a computed update, the value can be wrapped in `Arel.sql`, so Arel +treats it as an SQL literal. The below example is the same as the one above, but +the value is set to the product of the `bar` and `baz` columns: + +```ruby +update_value = Arel.sql('bar * baz') + +update_column_in_batches(:projects, :foo, update_value) do |table, query| + query.where(table[:some_column].eq('hello')) +end +``` + +Like `add_column_with_default`, there is a RuboCop cop to detect usage of this +on large tables. In the case of `update_column_in_batches`, it may be acceptable +to run on a large table, as long as it is only updating a small subset of the +rows in the table, but do not ignore that without validating on the GitLab.com +staging environment - or asking someone else to do so for you - beforehand. ## Integer column type diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index e0127aaed4c..12e90101139 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -22,6 +22,52 @@ As an example you might create 5 issues in between counts, which would cause the > **Note:** In some cases the query count might change slightly between runs for unrelated reasons. In this case you might need to test `exceed_query_limit(control_count + acceptable_change)`, but this should be avoided if possible. +## Finding the source of the query + +It may be useful to identify the source of the queries by looking at the call backtrace. +To enable this, run the specs with the `QUERY_RECORDER_DEBUG` environment variable set. For example: + +``` +QUERY_RECORDER_DEBUG=1 bundle exec rspec spec/requests/api/projects_spec.rb +``` + +This will log calls to QueryRecorder into the `test.log`. For example: + +``` +QueryRecorder SQL: SELECT COUNT(*) FROM "issues" WHERE "issues"."deleted_at" IS NULL AND "issues"."project_id" = $1 AND ("issues"."state" IN ('opened')) AND "issues"."confidential" = $2 + --> /home/user/gitlab/gdk/gitlab/spec/support/query_recorder.rb:19:in `callback' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:127:in `finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `block in finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `each' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:36:in `finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:25:in `instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_cache' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in `execute_and_clear' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `block in select_all' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:83:in `cache_sql' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `select_all' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:270:in `execute_simple_calculation' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:227:in `perform_calculation' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:133:in `calculate' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:48:in `count' + --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:20:in `uncached_count' + --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:12:in `block in count' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `block in fetch' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `block in instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:166:in `instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:584:in `save_block_result_to_cache' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `fetch' + --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:12:in `count' + --> /home/user/gitlab/gdk/gitlab/app/models/project.rb:1296:in `open_issues_count' +``` + ## See also - [Bullet](profiling.md#Bullet) For finding `N+1` query problems diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index bfd80aab6a4..ceff57276d2 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -122,6 +122,15 @@ they can be easily inspected. bundle exec rake services:doc ``` +## Updating Emoji Aliases + +To update the Emoji aliases file (used for Emoji autocomplete) you must run the +following: + +``` +bundle exec rake gemojione:aliases +``` + ## Updating Emoji Digests To update the Emoji digests file (used for Emoji autocomplete) you must run the @@ -131,6 +140,7 @@ following: bundle exec rake gemojione:digests ``` + This will update the file `fixtures/emojis/digests.json` based on the currently available Emoji. @@ -153,7 +163,7 @@ Starting a project from a template needs this project to be exported. On a up to date master branch with run: ``` -gdk run db +gdk run # In a new terminal window bundle exec rake gitlab:update_project_templates git checkout -b update-project-templates diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 1e9fdbc65e2..085fb8f902c 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -3,6 +3,12 @@ This document outlines various guidelines that should be followed when adding or modifying Sidekiq workers. +## ApplicationWorker + +All workers should include `ApplicationWorker` instead of `Sidekiq::Worker`, +which adds some convenience methods and automatically sets the queue based on +the worker's name. + ## Default Queue Use of the "default" queue is not allowed. Every worker should use a queue that @@ -13,19 +19,10 @@ A list of all available queues can be found in `config/sidekiq_queues.yml`. ## Dedicated Queues -Most workers should use their own queue. To ease this process a worker can -include the `DedicatedSidekiqQueue` concern as follows: - -```ruby -class ProcessSomethingWorker - include Sidekiq::Worker - include DedicatedSidekiqQueue -end -``` - -This will set the queue name based on the class' name, minus the `Worker` -suffix. In the above example this would lead to the queue being -`process_something`. +Most workers should use their own queue, which is automatically set based on the +worker class name. For a worker named `ProcessSomethingWorker`, the queue name +would be `process_something`. If you're not sure what a worker's queue name is, +you can find it using `SomeWorker.queue`. In some cases multiple workers do use the same queue. For example, the various workers for updating CI pipelines all use the `pipeline` queue. Adding workers @@ -39,7 +36,7 @@ tests should be placed in `spec/workers`. ## Removing or renaming queues -Try to avoid renaming or removing queues in minor and patch releases. -During online update instance can have pending jobs and removing the queue can -lead to those jobs being stuck forever. If you can't write migration for those -Sidekiq jobs, please consider doing rename or remove queue in major release only.
\ No newline at end of file +Try to avoid renaming or removing queues in minor and patch releases. +During online update instance can have pending jobs and removing the queue can +lead to those jobs being stuck forever. If you can't write migration for those +Sidekiq jobs, please consider doing rename or remove queue in major release only. diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index c4830322fa8..05e0a64af18 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -37,7 +37,7 @@ when using the migration helper method `Gitlab::Database::MigrationHelpers#add_column_with_default`. This method works similar to `add_column` except it updates existing rows in batches without blocking access to the table being modified. See ["Adding Columns With Default -Values"](migration_style_guide.html#adding-columns-with-default-values) for more +Values"](migration_style_guide.md#adding-columns-with-default-values) for more information on how to use this method. ## Dropping Columns diff --git a/doc/development/writing_documentation.md b/doc/development/writing_documentation.md index 68ba3dd2da3..b6def7ef541 100644 --- a/doc/development/writing_documentation.md +++ b/doc/development/writing_documentation.md @@ -152,12 +152,23 @@ CE and EE. ## Previewing the changes live If you want to preview the doc changes of your merge request live, you can use -the manual `review-docs-deploy` job in your merge request. +the manual `review-docs-deploy` job in your merge request. You will need at +least Master permissions to be able to run it and is currently enabled for the +following projects: + +- https://gitlab.com/gitlab-org/gitlab-ce +- https://gitlab.com/gitlab-org/gitlab-ee + +NOTE: **Note:** +You will need to push a branch to those repositories, it doesn't work for forks. TIP: **Tip:** If your branch contains only documentation changes, you can use [special branch names](#testing) to avoid long running pipelines. +In the mini pipeline graph, you should see an `>>` icon. Clicking on it will +reveal the `review-docs-deploy` job. Hit the play button for the job to start. +  This job will: diff --git a/doc/install/installation.md b/doc/install/installation.md index 2a004152d5e..570b0d5b22f 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -299,9 +299,9 @@ sudo usermod -aG redis git ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 10-1-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 10-2-stable gitlab -**Note:** You can change `10-1-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `10-2-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It @@ -367,6 +367,9 @@ sudo usermod -aG redis git # Enable packfile bitmaps sudo -u git -H git config --global repack.writeBitmaps true + + # Enable push options + sudo -u git -H git config --global receive.advertisePushOptions true # Configure Redis connection settings sudo -u git -H cp config/resque.yml.example config/resque.yml @@ -513,8 +516,7 @@ Check if GitLab and its environment are configured correctly: ### Compile GetText PO files - sudo -u git -H bundle exec rake gettext:pack RAILS_ENV=production - sudo -u git -H bundle exec rake gettext:po_to_json RAILS_ENV=production + sudo -u git -H bundle exec rake gettext:compile RAILS_ENV=production ### Compile Assets diff --git a/doc/install/kubernetes/gitlab_chart.md b/doc/install/kubernetes/gitlab_chart.md index fa564d83785..96968c1e3ab 100644 --- a/doc/install/kubernetes/gitlab_chart.md +++ b/doc/install/kubernetes/gitlab_chart.md @@ -1,7 +1,7 @@ # GitLab Helm Chart > **Note**: * This chart is deprecated, and is being replaced by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). For more information on available charts, please see our [overview](index.md#chart-overview). -* These charts have been tested on Google Container Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). +* These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). For more information on available GitLab Helm Charts, please see our [overview](index.md#chart-overview). @@ -243,7 +243,7 @@ controller. For `nginx-ingress` you can check the on how to add the annotation to the `controller.service.annotations` array. >**Note:** -When using the `nginx-ingress` controller on Google Container Engine (GKE), and using the `external-traffic` annotation, +When using the `nginx-ingress` controller on Google Kubernetes Engine (GKE), and using the `external-traffic` annotation, you will need to additionally set the `controller.kind` to be DaemonSet. Otherwise only pods running on the same node as the nginx controller will be able to reach GitLab. This may result in pods within your cluster not being able to reach GitLab. See the [Kubernetes documentation](https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer) and diff --git a/doc/install/kubernetes/gitlab_omnibus.md b/doc/install/kubernetes/gitlab_omnibus.md index 6659c3cf7b2..5a5f8d67ff5 100644 --- a/doc/install/kubernetes/gitlab_omnibus.md +++ b/doc/install/kubernetes/gitlab_omnibus.md @@ -1,7 +1,7 @@ # GitLab-Omnibus Helm Chart > **Note:** * This Helm chart is in beta, and will be deprecated by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). -* These charts have been tested on Google Container Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). +* These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). This work is based partially on: https://github.com/lwolf/kubernetes-gitlab/. GitLab would like to thank Sergey Nuzhdin for his work. @@ -72,7 +72,7 @@ Other common configuration options: - `baseIP`: the desired [external IP address](#external-ip-recommended) - `gitlab`: Choose the [desired edition](https://about.gitlab.com/products), either `ee` or `ce`. `ce` is the default. - `gitlabEELicense`: For Enterprise Edition, the [license](https://docs.gitlab.com/ee/user/admin_area/license.html) can be installed directly via the Chart -- `provider`: Optimizes the deployment for a cloud provider. The default is `gke` for [Google Container Engine](https://cloud.google.com/container-engine/), with `acs` also supported for the [Azure Container Service](https://azure.microsoft.com/en-us/services/container-service/). +- `provider`: Optimizes the deployment for a cloud provider. The default is `gke` for [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/), with `acs` also supported for the [Azure Container Service](https://azure.microsoft.com/en-us/services/container-service/). For additional configuration options, consult the [values.yaml](https://gitlab.com/charts/charts.gitlab.io/blob/master/charts/gitlab-omnibus/values.yaml). diff --git a/doc/install/kubernetes/gitlab_runner_chart.md b/doc/install/kubernetes/gitlab_runner_chart.md index 5e0d7493b61..ca9c95aeced 100644 --- a/doc/install/kubernetes/gitlab_runner_chart.md +++ b/doc/install/kubernetes/gitlab_runner_chart.md @@ -1,6 +1,6 @@ # GitLab Runner Helm Chart > **Note:** -These charts have been tested on Google Container Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). +These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). The `gitlab-runner` Helm chart deploys a GitLab Runner instance into your Kubernetes cluster. diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index dd350820c18..0932e1eee3a 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -1,5 +1,5 @@ # Installing GitLab on Kubernetes -> **Note**: These charts have been tested on Google Container Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). +> **Note**: These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). The easiest method to deploy GitLab on [Kubernetes](https://kubernetes.io/) is to take advantage of GitLab's Helm charts. [Helm] is a package diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 7bf126eec5d..baecf9455b0 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -80,13 +80,13 @@ errors during usage. - 256GB RAM supports up to 32,000 users - More users? Run it on [multiple application servers](https://about.gitlab.com/high-availability/) -We recommend having at least 2GB of swap on your server, even if you currently have +We recommend having at least [2GB of swap on your server](https://askubuntu.com/a/505344/310789), even if you currently have enough available RAM. Having swap will help reduce the chance of errors occurring if your available memory changes. We also recommend [configuring the kernel's swappiness setting](https://askubuntu.com/a/103916) to a low value like `10` to make the most of your RAM while still having the swap available when needed. -Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as top or htop) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about how many you need of those. +Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as `top` or `htop`) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about how many you need of those. ## Database @@ -146,7 +146,7 @@ So for a machine with 2 cores, 3 unicorn workers is ideal. For all machines that have 2GB and up we recommend a minimum of three unicorn workers. If you have a 1GB machine we recommend to configure only two Unicorn workers to prevent excessive swapping. -To change the Unicorn workers when you have the Omnibus package please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). +To change the Unicorn workers when you have the Omnibus package (which defaults to the recommendation above) please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). ## Redis and Sidekiq diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index 372e1909330..075feaeead9 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -22,6 +22,7 @@ Visit the links below for details: - [Redmine](../user/project/integrations/redmine.md) - [Jira](../user/project/integrations/jira.md) - [Bugzilla](../user/project/integrations/bugzilla.md) +- [Custom Issue Tracker](../user/project/integrations/custom_issue_tracker.md) ### Service Template diff --git a/doc/integration/google.md b/doc/integration/google.md index 727ca13ebcf..07a700f7b64 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -35,7 +35,7 @@ In Google's side: 1. You should now be able to see a Client ID and Client secret. Note them down or keep this page open as you will need them later. -1. From the **Dashboard** select **ENABLE APIS AND SERVICES > Google Cloud APIs > Container Engine API > Enable** +1. From the **Dashboard** select **ENABLE APIS AND SERVICES > Compute > Google Container Engine API > Enable** On your GitLab server: diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md index aa52b5415cf..36a8844e953 100644 --- a/doc/integration/slash_commands.md +++ b/doc/integration/slash_commands.md @@ -17,6 +17,9 @@ Taking the trigger term as `project-name`, the commands are: | `/project-name issue search <query>` | Shows up to 5 issues matching `<query>` | | `/project-name deploy <from> to <to>` | Deploy from the `<from>` environment to the `<to>` environment | +Note that if you are using the [GitLab Slack application](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html) for +your GitLab.com projects, you need to [add the `gitlab` keyword at the beginning of the command](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html#usage). + ## Issue commands It is possible to create new issue, display issue details and search up to 5 issues. diff --git a/doc/topics/autodevops/img/auto_devops_settings.png b/doc/topics/autodevops/img/auto_devops_settings.png Binary files differnew file mode 100644 index 00000000000..067c9da3fdc --- /dev/null +++ b/doc/topics/autodevops/img/auto_devops_settings.png diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 1cfdabac248..d100b431721 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -121,7 +121,7 @@ Google Cloud. ## Enabling Auto DevOps -NOTE: **Note:** +**Note:** If you haven't done already, read the [prerequisites](#prerequisites) to make full use of Auto DevOps. If this is your fist time, we recommend you follow the [quick start guide](#quick-start). @@ -133,6 +133,8 @@ full use of Auto DevOps. If this is your fist time, we recommend you follow the that will be used by Kubernetes to deploy your application 1. Hit **Save changes** for the changes to take effect + + Now that it's enabled, there are a few more steps depending on whether your project has a `.gitlab-ci.yml` or not: @@ -321,7 +323,7 @@ Auto DevOps uses [Helm](https://helm.sh/) to deploy your application to Kubernet You can override the Helm chart used by bundling up a chart into your project repo or by specifying a project variable: -- **Bundled chart** - If your project has a `./charts` directory with a `Chart.yaml` +- **Bundled chart** - If your project has a `./chart` directory with a `Chart.yaml` file in it, Auto DevOps will detect the chart and use it instead of the [default one](https://gitlab.com/charts/charts.gitlab.io/tree/master/charts/auto-deploy-app). This can be a great way to control exactly how your application is deployed. diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index ffe05519d7b..4858735ee86 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -23,12 +23,12 @@ page](https://gitlab.com/auto-devops-examples/minimal-ruby-app) and press the **Fork** button. Soon you should have a project under your namespace with the necessary files. -## Setup your own cluster on Google Container Engine +## Setup your own cluster on Google Kubernetes Engine If you do not already have a Google Cloud account, create one at https://console.cloud.google.com. -Visit the [**Container Engine**](https://console.cloud.google.com/kubernetes/list) +Visit the [**Kubernetes Engine**](https://console.cloud.google.com/kubernetes/list) tab and create a new cluster. You can change the name and leave the rest of the default settings. Once you have your cluster running, you need to connect to the cluster by following the Google interface. diff --git a/doc/university/glossary/README.md b/doc/university/glossary/README.md index c6a91c8d5c2..fbe7353c6ca 100644 --- a/doc/university/glossary/README.md +++ b/doc/university/glossary/README.md @@ -23,6 +23,10 @@ A Microsoft-based [directory service](https://msdn.microsoft.com/en-us/library/b Building and [delivering software](http://agilemethodology.org/) in phases/parts rather than trying to build everything at once then delivering to the user/client. The latter is known as the WaterFall model. +### Amazon RDS + +External reference: <http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html> + ### Application Lifecycle Management (ALM) The entire product lifecycle management process for an application, from requirements management, development, and testing until deployment. GitLab has [advantages](https://docs.google.com/presentation/d/1vCU-NbZWz8NTNK8Vu3y4zGMAHb5DpC8PE5mHtw1PWfI/edit#slide=id.g72f2e4906_2_288) over both legacy and modern ALM tools. @@ -68,7 +72,7 @@ A branch is a parallel version of a repository. This allows you to work on the r Having your own logo on [your GitLab instance login page](https://docs.gitlab.com/ee/customization/branded_login_page.html) instead of the GitLab logo. -### Job triggers +### Job triggers (Build Triggers) These protect your code base against breaks, for instance when a team is working on the same project. Learn about [setting up](https://docs.gitlab.com/ce/ci/triggers/README.html) job triggers. ### CEPH @@ -109,15 +113,15 @@ Atlassian's product for collaboration on documents and projects. ### Continuous Delivery -A [software engineering approach](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which continuous integration, automated testing, and automated deployment capabilities allow software to be developed and deployed rapidly, reliably and repeatedly with minimal human intervention. Still, the deployment to production is defined strategically and triggered manually. +A [software engineering approach](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which continuous integration, automated testing, and automated deployment capabilities allow software to be developed and deployed rapidly, reliably and repeatedly with minimal human intervention. Still, the deployment to production is defined strategically and triggered manually. [Amazon moves toward continuous delivery](https://www.youtube.com/watch?v=esEFaY0FDKc) ### Continuous Deployment -A [software development practice](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which every code change goes through the entire pipeline and is put into production automatically, resulting in many production deployments every day. It does everything that Continuous Delivery does, but the process is fully automated, there's no human intervention at all. +A [software development practice](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which every code change goes through the entire pipeline and is put into production automatically, resulting in many production deployments every day. It does everything that Continuous Delivery does, but the process is fully automated, there's no human intervention at all. [The difference between Continuous Delivery and Continuous Integration.](https://www.youtube.com/watch?v=igwFj8PPSnw) ### Continuous Integration -A [software development practice](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which you build and test software every time a developer pushes code to the application, and it happens several times a day. +A [software development practice](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which you build and test software every time a developer pushes code to the application, and it happens several times a day. [Thoughtworks discusses continuous integration.](https://www.thoughtworks.com/continuous-integration) ### Contributor @@ -127,6 +131,10 @@ Term used for a person contributing to an open source project. A [natural evolution](https://about.gitlab.com/2016/09/14/gitlab-live-event-recap/) of software development that carries a conversation across functional groups throughout the development process, enabling developers to track the full path of development in a cohesive and intuitive way. ConvDev accelerates the development lifecycle by fostering collaboration and knowledge sharing from idea to production. +### Cycle Analytics + +See <https://gitlab.com/gitlab-org/gitlab-ce/issues/22458> + ### Cycle Time The time it takes to move from [idea to production](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab). @@ -135,6 +143,10 @@ The time it takes to move from [idea to production](https://about.gitlab.com/201 Atlassian product for High Availability. +### Dependencies + +As in "specify [dependencies](https://gitlab.com/gitlab-org/gitlab-ce/issues/14728) between stages." + ### Deploy Keys A [SSH key](https://docs.gitlab.com/ce/gitlab-basics/create-your-ssh-keys.html)stored on your server that grants access to a single GitLab repository. This is used by a GitLab runner to clone a project's code so that tests can be run against the checked out code. @@ -151,26 +163,44 @@ The intersection of software engineering, quality assurance, and technology oper The difference between two commits, or saved changes. This will also be shown visually after the changes. -#### Directory +### Directory A folder used for storing multiple files. ### Docker Container Registry -A [feature](https://docs.gitlab.com/ce/user/project/container_registry.html) of GitLab projects. Containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in. +A [feature](https://docs.gitlab.com/ce/user/project/container_registry.html) of [GitLab projects](https://about.gitlab.com/2016/05/23/gitlab-container-registry/). Containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in. -### Dynamic Environment +### Dynamic Environment (review apps) -### ElasticSearch +### EC2 Instance + +### Elasticsearch Elasticsearch is a flexible, scalable and powerful search service. When [enabled](https://gitlab.com/help/integration/elasticsearch.md), it helps keep GitLab's search fast when dealing with a huge amount of data. ### Emacs +External reference: <https://www.masteringemacs.org/article/mastering-key-bindings-emacs> + +### First Byte + +External reference: <https://en.wikipedia.org/wiki/Time_To_First_Byte> + +First Byte (sometimes referred to as time to first byte or [TTFB](https://en.wikipedia.org/wiki/Time_To_First_Byte)) measures the time between making a request and receiving the first byte of information in return. As a result, First Byte encompasses everything that is the backend as well as network transit issues. It differs from [_Speed Index_](#speed-index) mostly by frontend related issues which are included in Speed Index such as javascript loading, page rendering, and so on. + ### Fork Your [own copy](https://docs.gitlab.com/ce/workflow/forking_workflow.html) of a repository that allows you to make changes to the repository without affecting the original. +### Funnel, or: TOFU, MOFU, BOFU + +External reference: [Blog post](https://www.weidert.com/whole_brain_marketing_blog/bid/113688/ToFu-MoFu-BoFu-Serving-Up-The-Right-Content-for-Lead-Nurturing) + +TOFU: top of funnel +MOFU: middle of funnel +BOFU: bottom of funnel + ### Gerrit A code review [tool](https://www.gerritcodereview.com/) built on top of Git. @@ -183,6 +213,8 @@ A [git attributes file](https://git-scm.com/docs/gitattributes) is a simple text [Scripts](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) you can use to trigger actions at certain points. +Difference between a [webhook](#webhooks) and a git hook: a git hook is local to its repo (usually) while a webhook is not (it can make API or http calls). So for example if you want your linter to fire before you commit, you can set that up with a git hook. If the linter fails, the commit does not go through. A git hook _can_ be configured to go beyond its repo, e.g. by having it make an API call. + ### GitHost.io A single-tenant solution that provides GitLab CE or EE as a managed service. GitLab Inc. is responsible for installing, updating, hosting, and backing up customers' own private and secure GitLab instance. @@ -211,9 +243,20 @@ Our free SaaS for public and private repositories. Allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational version. It [can be used](https://docs.gitlab.com/ee/gitlab-geo/README.html) for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. +### GitLab High Availability + +### GitLab Master Plan + +Related blog post: <https://about.gitlab.com/2016/09/13/gitlab-master-plan/>. + ### GitLab Pages + These allow you to [create websites](https://gitlab.com/help/pages/README.md) for your GitLab projects, groups, or user account. +### GitLab Runner + +Related project: <https://gitlab.com/gitlab-org/gitlab-runner> + ### Gitolite An [access layer](https://git-scm.com/book/en/v1/Git-on-the-Server-Gitolite) that sits on top of Git. Users are granted access to repos via a simple config file. As an admin, you only need the users' public SSH key and a username. @@ -226,6 +269,10 @@ A web-based hosting service for projects using Git. It was acquired by GitLab an An open source programming [language](https://golang.org/). +### Gogs + +External reference: <https://gogs.io/> + ### GUI/ Git GUI A portable [graphical interface](https://git-scm.com/docs/git-gui) to Git that allows users to make changes to their repository by making new commits, amending existing ones, creating branches, performing local merges, and fetching/pushing to remote repositories. @@ -256,7 +303,7 @@ A [tool](https://docs.gitlab.com/ee/integration/external-issue-tracker.html) use ### Jenkins -An Open Source CI tool written using the Java programming language. [Jenkins](https://jenkins-ci.org/) does the same job as GitLab CI, Bamboo, and Travis CI. It is extremely popular. +An Open Source CI tool written using the Java programming language. [Jenkins](https://jenkins-ci.org/) does the same job as GitLab CI, Bamboo, and Travis CI. It is extremely popular. Related [documentation](https://docs.gitlab.com/ee/integration/jenkins.html). ### Jira @@ -290,6 +337,10 @@ GitLab [integrates](https://docs.gitlab.com/ce/administration/auth/ldap.html) wi Allows you to synchronize the members of a GitLab group with one or more LDAP groups. +### Lint + +Static code analysis for our various file types. For example, we use [scss-lint](https://github.com/brigade/scss-lint) to ensure that a consistent code styling is respected. Similar tools: rubocop / eslint. + ### Load Balancer A [device](https://en.wikipedia.org/wiki/Load_balancing_(computing)) that distributes network or application traffic across multiple servers. @@ -330,6 +381,10 @@ Takes changes from one branch, and [applies them](https://git-scm.com/docs/git-m [Arises](https://about.gitlab.com/2016/09/06/resolving-merge-conflicts-from-the-gitlab-ui/) when a merge can't be performed cleanly between two versions of the same file. +#### Merge Request + +[Takes changes](https://docs.gitlab.com/ce/gitlab-basics/add-merge-request.html) from one branch, and applies them into another branch. + ### Meteor A [platform](https://www.meteor.com) for building javascript apps. @@ -350,6 +405,14 @@ A type of software license. It lets people do anything with your code with prope A free disaster recovery [software](https://help.ubuntu.com/community/MondoMindi). +#### Mount + +External reference: + +As stated on the [wikipedia page](https://en.wikipedia.org/wiki/Mount_(Unix)), "Mounting makes file systems, files, directories, devices and special files available for use and available to the user." + +For example, we have NFS servers where the _git files_ reside. In order for a worker node to "see" or "use" the git files, the NFS server needs to be _mounted_ on the worker; that is, the worker needs to know that the NFS server exists and how to connect to it. Think of it as getting a shared drive to show up in your Finder (on Mac) or Explorer (on Windows). + ### MySQL A relational [database](http://www.mysql.com/) owned by Oracle. Currently only supported if you are using EE. @@ -360,7 +423,7 @@ A set of symbols that are used to organize objects of various kinds so that thes ### Nginx -A web [server](https://www.nginx.com/resources/wiki/) (pronounced "engine x"). It can act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache. +A web [server](https://www.nginx.com/resources/wiki/) (pronounced "engine x"). [It can act]((https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md) as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache. ### OAuth @@ -380,7 +443,11 @@ GitLab's [business model](https://about.gitlab.com/2016/07/20/gitlab-is-open-cor ### Open Source Software -Software for which the original source code is freely [available](https://opensource.org/docs/osd) and may be redistributed and modified. GitLab prioritizes open source [stewardship](https://about.gitlab.com/2016/01/11/being-a-good-open-source-steward/). +Software for which the original source code is freely [available](https://opensource.org/docs/osd) and may be redistributed and modified. GitLab prioritizes open source [stewardship](https://about.gitlab.com/2016/01/11/being-a-good-open-source-steward/). Including to providing access to the source code, open source software must comply with a number of criteria, among them free distribution and no discrimination against persons, groups, or fields of endeavor. + +#### Open Source Stewardship + +[Related blog post](https://about.gitlab.com/2016/01/11/being-a-good-open-source-steward/). ### Owner @@ -430,6 +497,8 @@ A popular DevOps [automation tool](https://puppet.com/product/how-puppet-works). Git [command](https://git-scm.com/docs/git-push) to send commits from the local repository to the remote repository. Read about [advanced push rules](https://gitlab.com/help/pages/README.md) in GitLab. +### Raketasks + ### RE Read Only Permissions to see a file and its contents, but not change it. @@ -438,10 +507,24 @@ Permissions to see a file and its contents, but not change it. In addition to the merge, the [rebase](https://git-scm.com/book/en/v2/Git-Branching-Rebasing) is a main way to integrate changes from one branch into another. +### Regression + +A regression is something that used to work one way in the last release and then we made a **breaking change** and it no longer works the same way. + +_or_ + +A regression is defined as a change that results in a negative impact on the functionality of an existing feature due to recent changes, i.e. the latest release. + +### Remote mirroring + ### (Git) Repository A directory where Git [has been initiatlized](https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository) to start version controlling your files. The history of your work is stored here. A remote repository is not on your machine, but usually online (like on GitLab.com, for instance). The main remote repository is usually called "Origin." +##### Remote repository + +A [repository](https://about.gitlab.com/2015/05/18/simple-words-for-a-gitlab-newbie/) that is not-on-your-machine, so it's anything that is not your computer. Usually, it is online, GitLab.com for instance. The main remote repository is usually called “Origin”. + ### Requirements management Gives your distributed teams a single shared repository to collaborate and share requirements, understand their relationship to tests, and evaluate linked defects. It includes multiple, preconfigured requirement types. @@ -474,7 +557,11 @@ Software that is hosted centrally and accessed on-demand (i.e. whenever you want This term is often used by people when they mean "Version Control." -## Scrum +#### SCLAU + +Abbreviation for SQO Count [Large And Up](https://about.gitlab.com/handbook/sales/#market-segmentation). This is the number of opportunities in large and strategic organizations passed from marketing to sales. + +### Scrum An Agile [framework](https://www.scrum.org/Resources/What-is-Scrum) designed to typically help complete complex software projects. It's made up of several parts: product requirements backlog, sprint planning, sprint (development), sprint review, and retrospec (analyzing the sprint). The goal is to end up with potentially shippable products. @@ -484,7 +571,9 @@ The board used to track the status and progress of each of the sprint backlog it ### Shell -Terminal on Mac OSX, GitBash on Windows, or Linux Terminal on Linux. You [use git]() and make changes to GitLab projects in your shell. You [use git](https://docs.gitlab.com/ce/gitlab-basics/start-using-git.html) and make changes to GitLab projects in your shell. +Terminal on Mac OSX, GitBash on Windows, or Linux Terminal on Linux. You [use git](https://docs.gitlab.com/ce/gitlab-basics/start-using-git.html) and make changes to GitLab projects in your shell. You [use git](https://docs.gitlab.com/ce/gitlab-basics/start-using-git.html) and make changes to GitLab projects in your shell. + +### Shell command runner ### Single-tenant @@ -494,6 +583,8 @@ The tenant purchases their own copy of the software and the software can be cust Real time messaging app for teams that is used internally by GitLab team members. GitLab users can enable [Slack integration](https://docs.gitlab.com/ce/project_services/slack.html) to trigger push, issue, and merge request events among others. +### Slash commands + ### Slave Servers Also known as secondary servers, these help to spread the load over multiple machines. They also provide backups when the master/primary server crashes. @@ -502,6 +593,10 @@ Also known as secondary servers, these help to spread the load over multiple mac Program code as typed by a computer programmer (i.e. it has not yet been compiled/translated by the computer to machine language). +### Speed Index + +[Speed Index](https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index) is "the average time at which visible parts of the page are displayed". + ### SSH Key A unique identifier of a computer. It is used to identify computers without the need for a password (e.g., On GitLab I have [added the ssh key](https://docs.gitlab.com/ce/gitlab-basics/create-your-ssh-keys.html) of all my work machines so that the GitLab instance knows that it can accept code pushes and pulls from this trusted machines whose keys are I have added.) @@ -542,6 +637,16 @@ An open source version control system. Read about [migrating from SVN](https://d [Represents](https://docs.gitlab.com/ce/api/tags.html) a version of a particular branch at a moment in time. +### Tenancy + +#### Multi-tenant + +A [multi-tenant](http://whatis.techtarget.com/definition/multi-tenancy) GitLab instance can have any number of customers - such as companies or groups of users using it. GitLab.com is an example of a multi-tenant GitLab instance. + +#### Single-tenant + +A [single-tenant](http://searchcloudapplications.techtarget.com/definition/single-tenancy) GitLab instance has only one customer - such as a company - using it. On premise GitLab instances are almost exclusively single-tenant. + ### Tool Stack The set of tools used in a process to achieve a common outcome (e.g. set of tools used in Application Lifecycle Management). @@ -550,9 +655,17 @@ The set of tools used in a process to achieve a common outcome (e.g. set of tool An open source project management and bug tracking web [application](https://trac.edgewall.org/). +### True-Up licensing model + +### Ubuntu + ### Untracked files -New files that Git has not [been told](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository) to track previously. +New files that Git has not [been told](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository) to track previously. Add them by using the command "git add [file path]" + +### Upstream repository vs. GitLab repository + +[External conversation](https://news.ycombinator.com/item?id=12487112) ### User @@ -560,11 +673,11 @@ Anyone interacting with the software. ### Version Control Software (VCS) -Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later. VCS [has evolved](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit#slide=id.gd69537a19_0_32) from local version control systems, to centralized version control systems, to the present distributed version control systems like Git, Mercurial, Bazaar, and Darcs. +Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later. VCS [has evolved](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit#slide=id.gd69537a19_0_32) from local version control systems, to centralized version control systems, to the present [distributed version control systems](https://en.wikipedia.org/wiki/Distributed_version_control) like Git, Mercurial, Bazaar, and Darcs. If any server dies, and these systems were collaborating via it, any of the client repositories can be copied back up to the server to restore it. ### Virtual Private Cloud (VPC) -An on demand configurable pool of shared computing resources allocated within a public cloud environment, providing some isolation between the different users using the resources. GitLab users need to create a new Amazon VPC in order to [setup High Availability](https://docs.gitlab.com/ce/university/high-availability/aws/). +A [VPC](https://docs.gitlab.com/ce/university/glossary/README.html#virtual-private-cloud-vpc) is an on demand configurable pool of shared computing resources allocated within a public cloud environment, providing some isolation between the different users using the resources. GitLab users need to create a new Amazon VPC in order to [setup High Availability](https://docs.gitlab.com/ce/university/high-availability/aws/). ### Virtual private server (VPS) @@ -572,7 +685,7 @@ A [virtual machine](https://en.wikipedia.org/wiki/Virtual_private_server) sold a ### VM Instance -In object-oriented programming, an [instance](http://stackoverflow.com/questions/20461907/what-is-meaning-of-instance-in-programming) is a specific realization of any object. An object may be varied in a number of ways. Each realized variation of that object is an instance. Therefore, a VM instance is an instance of a virtual machine, which is an emulation of a computer system. +In object-oriented programming, an [instance](http://stackoverflow.com/questions/20461907/what-is-meaning-of-instance-in-programming) is a specific realization of any [object](https://cloud.google.com/compute/docs/instances/). An object may be varied in a number of ways. Each realized variation of that object is an instance. Therefore, a VM instance is an instance of a virtual machine, which is an emulation of a computer system. ### Waterfall @@ -586,6 +699,10 @@ A way for for an app to [provide](https://docs.gitlab.com/ce/user/project/integr A [website/system](http://www.wiki.com/) that allows for collaborative editing of its content by the users. In programming, wikis usually contain documentation of how to use the software. +### Working area + +Files that have been modified but are not committed. Check them by using the command "git status". + ### Working Tree [Consists of files](http://stackoverflow.com/questions/3689838/difference-between-head-working-tree-index-in-git) that you are currently working on. diff --git a/doc/update/10.1-to-10.2.md b/doc/update/10.1-to-10.2.md new file mode 100644 index 00000000000..9e0d8f79522 --- /dev/null +++ b/doc/update/10.1-to-10.2.md @@ -0,0 +1,360 @@ +--- +comments: false +--- + +# From 10.1 to 10.2 + +Make sure you view this update guide from the tag (version) of GitLab you would +like to install. In most cases this should be the highest numbered production +tag (without rc in it). You can select the tag in the version dropdown at the +top left corner of GitLab (below the menu bar). + +If the highest number stable branch is unclear please check the +[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation +guide links by version. + +### 1. Stop server + +```bash +sudo service gitlab stop +``` + +### 2. Backup + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Update Ruby + +NOTE: GitLab 9.0 and higher only support Ruby 2.3.x and dropped support for Ruby 2.1.x. Be +sure to upgrade your interpreter if necessary. + +You can check which version you are running with `ruby -v`. + +Download and compile Ruby: + +```bash +mkdir /tmp/ruby && cd /tmp/ruby +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.5.tar.gz +echo '3247e217d6745c27ef23bdc77b6abdb4b57a118f ruby-2.3.5.tar.gz' | shasum -c - && tar xzf ruby-2.3.5.tar.gz +cd ruby-2.3.5 +./configure --disable-install-rdoc +make +sudo make install +``` + +Install Bundler: + +```bash +sudo gem install bundler --no-ri --no-rdoc +``` + +### 4. Update Node + +GitLab now runs [webpack](http://webpack.js.org) to compile frontend assets and +it has a minimum requirement of node v4.3.0. + +You can check which version you are running with `node -v`. If you are running +a version older than `v4.3.0` you will need to update to a newer version. You +can find instructions to install from community maintained packages or compile +from source at the nodejs.org website. + +<https://nodejs.org/en/download/> + + +Since 8.17, GitLab requires the use of yarn `>= v0.17.0` to manage +JavaScript dependencies. + +```bash +curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - +echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list +sudo apt-get update +sudo apt-get install yarn +``` + +More information can be found on the [yarn website](https://yarnpkg.com/en/docs/install). + +### 5. Update Go + +NOTE: GitLab 9.2 and higher only supports Go 1.8.3 and dropped support for Go +1.5.x through 1.7.x. Be sure to upgrade your installation if necessary. + +You can check which version you are running with `go version`. + +Download and install Go: + +```bash +# Remove former Go installation folder +sudo rm -rf /usr/local/go + +curl --remote-name --progress https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz +echo '1862f4c3d3907e59b04a757cfda0ea7aa9ef39274af99a784f5be843c80c6772 go1.8.3.linux-amd64.tar.gz' | shasum -a256 -c - && \ + sudo tar -C /usr/local -xzf go1.8.3.linux-amd64.tar.gz +sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ +rm go1.8.3.linux-amd64.tar.gz +``` + +### 6. Get latest code + +```bash +cd /home/git/gitlab + +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +sudo -u git -H git checkout -- locale +``` + +For GitLab Community Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 10-2-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 10-2-stable-ee +``` + +### 7. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell + +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) +sudo -u git -H bin/compile +``` + +### 8. Update gitlab-workhorse + +Install and compile gitlab-workhorse. GitLab-Workhorse uses +[GNU Make](https://www.gnu.org/software/make/). +If you are not using Linux you may have to run `gmake` instead of +`make` below. + +```bash +cd /home/git/gitlab-workhorse + +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) +sudo -u git -H make +``` + +### 9. Update Gitaly + +#### New Gitaly configuration options required + +In order to function Gitaly needs some additional configuration information. Below we assume you installed Gitaly in `/home/git/gitaly` and GitLab Shell in `/home/git/gitlab-shell`. + +```shell +echo ' +[gitaly-ruby] +dir = "/home/git/gitaly/ruby" + +[gitlab-shell] +dir = "/home/git/gitlab-shell" +' | sudo -u git tee -a /home/git/gitaly/config.toml +``` + +#### Check Gitaly configuration + +Due to a bug in the `rake gitlab:gitaly:install` script your Gitaly +configuration file may contain syntax errors. The block name +`[[storages]]`, which may occur more than once in your `config.toml` +file, should be `[[storage]]` instead. + +```shell +sudo -u git -H sed -i.pre-10.1 's/\[\[storages\]\]/[[storage]]/' /home/git/gitaly/config.toml +``` + +#### Compile Gitaly + +```shell +cd /home/git/gitaly +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) +sudo -u git -H make +``` + +### 10. Update MySQL permissions + +If you are using MySQL you need to grant the GitLab user the necessary +permissions on the database: + +```bash +mysql -u root -p -e "GRANT TRIGGER ON \`gitlabhq_production\`.* TO 'git'@'localhost';" +``` + +If you use MySQL with replication, or just have MySQL configured with binary logging, +you will need to also run the following on all of your MySQL servers: + +```bash +mysql -u root -p -e "SET GLOBAL log_bin_trust_function_creators = 1;" +``` + +You can make this setting permanent by adding it to your `my.cnf`: + +``` +log_bin_trust_function_creators=1 +``` + +### 11. Update configuration files + +#### New configuration options for `gitlab.yml` + +There might be configuration options available for [`gitlab.yml`][yaml]. View them with the command below and apply them manually to your current `gitlab.yml`: + +```sh +cd /home/git/gitlab + +git diff origin/10-1-stable:config/gitlab.yml.example origin/10-2-stable:config/gitlab.yml.example +``` + +#### Nginx configuration + +Ensure you're still up-to-date with the latest NGINX configuration changes: + +```sh +cd /home/git/gitlab + +# For HTTPS configurations +git diff origin/10-1-stable:lib/support/nginx/gitlab-ssl origin/10-2-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/10-1-stable:lib/support/nginx/gitlab origin/10-2-stable:lib/support/nginx/gitlab +``` + +If you are using Strict-Transport-Security in your installation to continue using it you must enable it in your Nginx +configuration as GitLab application no longer handles setting it. + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-2-stable/lib/support/init.d/gitlab.default.example#L38 + +#### SMTP configuration + +If you're installing from source and use SMTP to deliver mail, you will need to add the following line +to config/initializers/smtp_settings.rb: + +```ruby +ActionMailer::Base.delivery_method = :smtp +``` + +See [smtp_settings.rb.sample] as an example. + +[smtp_settings.rb.sample]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-2-stable/config/initializers/smtp_settings.rb.sample#L13 + +#### Init script + +There might be new configuration options available for [`gitlab.default.example`][gl-example]. View them with the command below and apply them manually to your current `/etc/default/gitlab`: + +```sh +cd /home/git/gitlab + +git diff origin/10-1-stable:lib/support/init.d/gitlab.default.example origin/10-2-stable:lib/support/init.d/gitlab.default.example +``` + +Ensure you're still up-to-date with the latest init script changes: + +```bash +cd /home/git/gitlab + +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +For Ubuntu 16.04.1 LTS: + +```bash +sudo systemctl daemon-reload +``` + +### 12. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Optional: clean up old gems +sudo -u git -H bundle clean + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Compile GetText PO files + +sudo -u git -H bundle exec rake gettext:compile RAILS_ENV=production + +# Update node dependencies and recompile assets +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production + +# Clean up cache +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +**MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). + +### 13. Start application + +```bash +sudo service gitlab start +sudo service nginx restart +``` + +### 14. Check application status + +Check if GitLab and its environment are configured correctly: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +To make sure you didn't miss anything run a more thorough check: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +``` + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (10.0) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 9.5 to 10.0](9.5-to-10.0.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +[yaml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-2-stable/config/gitlab.yml.example +[gl-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-2-stable/lib/support/init.d/gitlab.default.example diff --git a/doc/user/discussions/img/image_resolved_discussion.png b/doc/user/discussions/img/image_resolved_discussion.png Binary files differindex ed00b5c77fe..ed00b5c77fe 100755..100644 --- a/doc/user/discussions/img/image_resolved_discussion.png +++ b/doc/user/discussions/img/image_resolved_discussion.png diff --git a/doc/user/discussions/img/onion_skin_view.png b/doc/user/discussions/img/onion_skin_view.png Binary files differindex 91c3b396844..91c3b396844 100755..100644 --- a/doc/user/discussions/img/onion_skin_view.png +++ b/doc/user/discussions/img/onion_skin_view.png diff --git a/doc/user/discussions/img/swipe_view.png b/doc/user/discussions/img/swipe_view.png Binary files differindex 82d6e52173c..82d6e52173c 100755..100644 --- a/doc/user/discussions/img/swipe_view.png +++ b/doc/user/discussions/img/swipe_view.png diff --git a/doc/user/discussions/img/two_up_view.png b/doc/user/discussions/img/two_up_view.png Binary files differindex d9e90708e87..d9e90708e87 100755..100644 --- a/doc/user/discussions/img/two_up_view.png +++ b/doc/user/discussions/img/two_up_view.png diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 454988b9b80..a671c92640a 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -41,7 +41,7 @@ Line-breaks, or softreturns, are rendered if you end a line with two or more spa Sugar is sweet -Roses are red +Roses are red Violets are blue Sugar is sweet @@ -368,6 +368,40 @@ _Be advised that KaTeX only supports a [subset][katex-subset] of LaTeX._ >**Note:** This also works for the asciidoctor `:stem: latexmath`. For details see the [asciidoctor user manual][asciidoctor-manual]. +### Mermaid + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15107) in +GitLab 10.3. + +> If this is not rendered correctly, see +https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#mermaid + +It is possible to generate diagrams and flowcharts from text using [Mermaid][mermaid]. + +In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block. + +Example: + + ```mermaid + graph TD; + A-->B; + A-->C; + B-->D; + C-->D; + ``` + +Becomes: + +```mermaid +graph TD; + A-->B; + A-->C; + B-->D; + C-->D; +``` + +For details see the [Mermaid official page][mermaid]. + ## Standard Markdown ### Headers @@ -666,7 +700,7 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also a separate paragraph, but... This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. -This line is also a separate paragraph, and... +This line is also a separate paragraph, and... This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) spaces. @@ -679,7 +713,7 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also a separate paragraph, but... This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. -This line is also a separate paragraph, and... +This line is also a separate paragraph, and... This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) spaces. @@ -814,6 +848,7 @@ A link starting with a `/` is relative to the wiki root. [^2]: This is my awesome footnote. [markdown.md]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md +[mermaid]: https://mermaidjs.github.io/ "Mermaid website" [rouge]: http://rouge.jneen.net/ "Rouge website" [redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" [katex]: https://github.com/Khan/KaTeX "KaTeX website" diff --git a/doc/user/permissions.md b/doc/user/permissions.md index c03700a3501..4fa83388d0c 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -23,25 +23,26 @@ The following table depicts the various user permission levels in a project. |---------------------------------------|---------|------------|-------------|----------|--------| | Create new issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | Create confidential issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| View confidential issues | (✓) [^2] | ✓ | ✓ | ✓ | ✓ | +| View confidential issues | (✓) [^2] | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | Lock discussions (issues and merge requests) | | | | ✓ | ✓ | | See a list of jobs | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | -| See a job log | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | +| See a job log | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | Download and browse job artifacts | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | View wiki pages | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | Pull project code | [^1] | ✓ | ✓ | ✓ | ✓ | | Download project | [^1] | ✓ | ✓ | ✓ | ✓ | +| Assign issues and merge requests | | ✓ | ✓ | ✓ | ✓ | +| Label issues and merge requests | | ✓ | ✓ | ✓ | ✓ | | Create code snippets | | ✓ | ✓ | ✓ | ✓ | | Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | | Manage labels | | ✓ | ✓ | ✓ | ✓ | | See a commit status | | ✓ | ✓ | ✓ | ✓ | | See a container registry | | ✓ | ✓ | ✓ | ✓ | | See environments | | ✓ | ✓ | ✓ | ✓ | +| See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | | Create new environments | | | ✓ | ✓ | ✓ | -| Use environment terminals | | | | ✓ | ✓ | | Stop environments | | | ✓ | ✓ | ✓ | -| See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | | Manage/Accept merge requests | | | ✓ | ✓ | ✓ | | Create new merge request | | | ✓ | ✓ | ✓ | | Create new branches | | | ✓ | ✓ | ✓ | @@ -55,10 +56,11 @@ The following table depicts the various user permission levels in a project. | Update a container registry | | | ✓ | ✓ | ✓ | | Remove a container registry image | | | ✓ | ✓ | ✓ | | Create/edit/delete project milestones | | | ✓ | ✓ | ✓ | +| Use environment terminals | | | | ✓ | ✓ | | Add new team members | | | | ✓ | ✓ | | Push to protected branches | | | | ✓ | ✓ | | Enable/disable branch protection | | | | ✓ | ✓ | -| Turn on/off protected branch push for devs| | | | ✓ | ✓ | +| Turn on/off protected branch push for devs| | | | ✓ | ✓ | | Enable/disable tag protections | | | | ✓ | ✓ | | Rewrite/remove Git tags | | | | ✓ | ✓ | | Edit project | | | | ✓ | ✓ | @@ -69,14 +71,15 @@ The following table depicts the various user permission levels in a project. | Manage variables | | | | ✓ | ✓ | | Manage pages | | | | ✓ | ✓ | | Manage pages domains and certificates | | | | ✓ | ✓ | +| Manage clusters | | | | ✓ | ✓ | +| Edit comments (posted by any user) | | | | ✓ | ✓ | | Switch visibility level | | | | | ✓ | | Transfer project to another namespace | | | | | ✓ | | Remove project | | | | | ✓ | | Delete issues | | | | | ✓ | +| Remove pages | | | | | ✓ | | Force push to protected branches [^4] | | | | | | | Remove protected branches [^4] | | | | | | -| Remove pages | | | | | ✓ | -| Manage clusters | | | | ✓ | ✓ | ## Project features permissions @@ -197,6 +200,7 @@ instance and project. In addition, all admins can use the admin interface under |---------------------------------------|-----------------|-------------|----------|--------| | See commits and jobs | ✓ | ✓ | ✓ | ✓ | | Retry or cancel job | | ✓ | ✓ | ✓ | +| Erase job artifacts and trace | | ✓ [^7] | ✓ | ✓ | | Remove project | | | ✓ | ✓ | | Create project | | | ✓ | ✓ | | Change project configuration | | | ✓ | ✓ | @@ -261,5 +265,6 @@ only. [^4]: Not allowed for Guest, Reporter, Developer, Master, or Owner [^5]: Only if user is not external one. [^6]: Only if user is a member of the project. +[^7]: Only if the build was triggered by the user [ce-18994]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18994 [new-mod]: project/new_ci_build_permissions_model.md diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index f2ad42f21fd..022d6317555 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -55,9 +55,10 @@ You have 6 options here that you can use for your default dashboard view: The project home page content setting allows you to choose what content you want to see on a project’s home page. -You can choose between 2 options: +You can choose between 3 options: - Show the files and the readme (default) +- Show the readme - Show the project’s activity [rouge]: http://rouge.jneen.net/ "Rouge website" diff --git a/doc/user/project/clusters/img/cluster-applications.png b/doc/user/project/clusters/img/cluster-applications.png Binary files differdeleted file mode 100644 index 7c82d19297e..00000000000 --- a/doc/user/project/clusters/img/cluster-applications.png +++ /dev/null diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 27b4b49c207..e2924c66e70 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -1,14 +1,15 @@ -# Connecting GitLab with GKE +# Connecting GitLab with a Kubernetes cluster > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/35954) in 10.1. CAUTION: **Warning:** The Cluster integration is currently in **Beta**. -Connect your project to Google Container Engine (GKE) in a few steps. - With a cluster associated to your project, you can use Review Apps, deploy your -applications, run your pipelines, and much more in an easy way. +applications, run your pipelines, and much more, in an easy way. + +Connect your project to Google Kubernetes Engine (GKE) or your own Kubernetes +cluster in a few steps. NOTE: **Note:** The Cluster integration will eventually supersede the @@ -30,36 +31,58 @@ prerequisites must be met: - You must have Master [permissions] in order to be able to access the **Cluster** page. -If all of the above requirements are met, you can proceed to add a new cluster. +If all of the above requirements are met, you can proceed to add a new GKE +cluster. ## Adding a cluster NOTE: **Note:** You need Master [permissions] and above to add a cluster. +There are two options when adding a new cluster; either use Google Kubernetes +Engine (GKE) or provide the credentials to your own Kubernetes cluster. + To add a new cluster: -1. Navigate to your project's **CI/CD > Cluster** page. -1. Connect your Google account if you haven't done already by clicking the - "Sign-in with Google" button. -1. Fill in the requested values: - - **Cluster name** (required) - The name you wish to give the cluster. - - **GCP project ID** (required) - The ID of the project you created in your GCP - console that will host the Kubernetes cluster. This must **not** be confused - with the project name. Learn more about [Google Cloud Platform projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects). - - **Zone** - The zone under which the cluster will be created. Read more about - [the available zones](https://cloud.google.com/compute/docs/regions-zones/). - - **Number of nodes** - The number of nodes you wish the cluster to have. - - **Machine type** - The machine type of the Virtual Machine instance that - the cluster will be based on. Read more about [the available machine types](https://cloud.google.com/compute/docs/machine-types). - - **Project namespace** - The unique namespace for this project. By default you - don't have to fill it in; by leaving it blank, GitLab will create one for you. -1. Click the **Create cluster** button. - -After a few moments your cluster should be created. If something goes wrong, +1. Navigate to your project's **CI/CD > Cluster** page +1. If you want to let GitLab create a cluster on GKE for you, go through the + following steps, otherwise skip to the next one. + 1. Click on **Create with GKE** + 1. Connect your Google account if you haven't done already by clicking the + **Sign in with Google** button + 1. Fill in the requested values: + - **Cluster name** (required) - The name you wish to give the cluster. + - **GCP project ID** (required) - The ID of the project you created in your GCP + console that will host the Kubernetes cluster. This must **not** be confused + with the project name. Learn more about [Google Cloud Platform projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects). + - **Zone** - The [zone](https://cloud.google.com/compute/docs/regions-zones/) + under which the cluster will be created. + - **Number of nodes** - The number of nodes you wish the cluster to have. + - **Machine type** - The [machine type](https://cloud.google.com/compute/docs/machine-types) + of the Virtual Machine instance that the cluster will be based on. + - **Project namespace** - The unique namespace for this project. By default you + don't have to fill it in; by leaving it blank, GitLab will create one for you. +1. If you want to use your own existing Kubernetes cluster, click on + **Add an existing cluster** and fill in the details as described in the + [Kubernetes integration](../integrations/kubernetes.md) documentation. +1. Finally, click the **Create cluster** button + +After a few moments, your cluster should be created. If something goes wrong, you will be notified. -Now, you can proceed to [enable the Cluster integration](#enabling-or-disabling-the-cluster-integration). +You can now proceed to install some pre-defined applications and then +enable the Cluster integration. + +## Installing applications + +GitLab provides a one-click install for various applications which will be +added directly to your configured cluster. Those applications are needed for +[Review Apps](../../../ci/review_apps/index.md) and [deployments](../../../ci/environments.md). + +| Application | GitLab version | Description | +| ----------- | :------------: | ----------- | +| [Helm Tiller](https://docs.helm.sh/) | 10.2+ | Helm is a package manager for Kubernetes and is required to install all the other applications. It will be automatically installed as a dependency when you try to install a different app. It is installed in its own pod inside the cluster which can run the `helm` CLI in a safe environment. | +| [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps](../../../topics/autodevops/index.md) or deploy your own web apps. | ## Enabling or disabling the Cluster integration @@ -88,12 +111,3 @@ To remove the Cluster integration from your project, simply click on the and [add a cluster](#adding-a-cluster) again. [permissions]: ../../permissions.md - -## Installing applications - -GitLab provides a one-click install for -[Helm Tiller](https://docs.helm.sh/) and -[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) -which will be added directly to your configured cluster. - - diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 6423beefc77..72def9d1d1d 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -24,6 +24,8 @@ constrains of a Sidekiq worker. - the milestones (GitLab 8.7+) - the labels (GitLab 8.7+) - the release note descriptions (GitLab 8.12+) + - the pull request review comments (GitLab 10.2+) + - the regular issue and pull request comments - References to pull requests and issues are preserved (GitLab 8.7+) - Repository public access is retained. If a repository is private in GitHub it will be created as private in GitLab as well. @@ -43,10 +45,13 @@ the case the namespace is taken, the repository will be imported under the user' namespace that started the import process. The importer will also import branches on forks of projects related to open pull -requests. These branches will be imported with a naming scheume similar to +requests. These branches will be imported with a naming scheme similar to GH-SHA-Username/Pull-Request-number/fork-name/branch. This may lead to a discrepency in branches compared to the GitHub Repository. +For a more technical description and an overview of the architecture you can +refer to [Working with the GitHub importer][gh-import-dev-docs]. + ## Importing your GitHub repositories The importer page is visible when you create a new project. @@ -121,7 +126,29 @@ If you want, you can import all your GitHub projects in one go by hitting You can also choose a different name for the project and a different namespace, if you have the privileges to do so. +## Making the import process go faster + +For large projects it may take a while to import all data. To reduce the time +necessary you can increase the number of Sidekiq workers that process the +following queues: + +* `github_importer` +* `github_importer_advance_stage` + +For an optimal experience we recommend having at least 4 Sidekiq processes (each +running a number of threads equal to the number of CPU cores) that _only_ +process these queues. We also recommend that these processes run on separate +servers. For 4 servers with 8 cores this means you can import up to 32 objects +(e.g. issues) in parallel. + +Reducing the time spent in cloning a repository can be done by increasing +network throughput, CPU capacity, and disk performance (e.g. by using high +performance SSDs) of the disks that store the Git repositories (for your GitLab +instance). Increasing the number of Sidekiq workers will _not_ reduce the time +spent cloning repositories. + [gh-import]: ../../../integration/github.md "GitHub integration" [gh-rake]: ../../../administration/raketasks/github_import.md "GitHub rake task" [gh-integration]: #authorize-access-to-your-repositories-using-the-github-integration [gh-token]: #authorize-access-to-your-repositories-using-a-personal-access-token +[gh-import-dev-docs]: ../../../development/github_importer.md "Working with the GitHub importer" diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 97d0d529886..5d91aef5735 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -64,7 +64,7 @@ common actions on issues or merge requests - [Pipeline settings](pipelines/settings.md): Set up Git strategy (choose the default way your repository is fetched from GitLab in a job), timeout (defines the maximum amount of time in minutes that a job is able run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline's visibility, and much more - [GKE cluster integration](clusters/index.md): Connecting your GitLab project - with Google Container Engine + with Google Kubernetes Engine - [GitLab Pages](pages/index.md): Build, test, and deploy your static website with GitLab Pages diff --git a/doc/user/project/integrations/custom_issue_tracker.md b/doc/user/project/integrations/custom_issue_tracker.md new file mode 100644 index 00000000000..757522c2ae3 --- /dev/null +++ b/doc/user/project/integrations/custom_issue_tracker.md @@ -0,0 +1,20 @@ +# Custom Issue Tracker Service + +To enable the Custom Issue Tracker integration in a project, navigate to the +[Integrations page](project_services.md#accessing-the-project-services), click +the **Customer Issue Tracker** service, and fill in the required details on the page as described +in the table below. + +| Field | Description | +| ----- | ----------- | +| `title` | A title for the issue tracker (to differentiate between instances, for example) | +| `description` | A name for the issue tracker (to differentiate between instances, for example) | +| `project_url` | Currently unused. Will be changed in a future release. | +| `issues_url` | The URL to the issue in the issue tracker project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. For example, `https://customissuetracker.com/project-name/:id`. | +| `new_issue_url` | Currently unused. Will be changed in a future release. | + + +## Referencing issues + +Issues are referenced with `#<ID>`, where `<ID>` is a number (example `#143`). +So with the example above, `#143` would refer to `https://customissuetracker.com/project-name/143`.
\ No newline at end of file diff --git a/doc/user/project/integrations/img/issue_configuration.png b/doc/user/project/integrations/img/issue_configuration.png Binary files differnew file mode 100644 index 00000000000..2049d60fdd2 --- /dev/null +++ b/doc/user/project/integrations/img/issue_configuration.png diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 518683965e8..a6673fa2a00 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -13,8 +13,8 @@ integration services must be enabled. | Name | Query | | ---- | ----- | -| Average Memory Usage (MB) | (sum(container_memory_usage_bytes{container_name!="POD",%{environment_filter}}) / count(container_memory_usage_bytes{container_name!="POD",%{environment_filter}})) /1024/1024 | -| Average CPU Utilization (%) | sum(rate(container_cpu_usage_seconds_total{container_name!="POD",%{environment_filter}}[2m])) by (cpu) * 100 | +| Average Memory Usage (MB) | (sum(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job))) / count(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job)) /1024/1024 | +| Average CPU Utilization (%) | sum(avg(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="%{ci_environment_slug}"}[2m])) without (job)) * 100 | ## Configuring Prometheus to monitor for Kubernetes node metrics diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index cf92465da53..f530b6cb649 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -1,26 +1,29 @@ # Redmine Service -To enable the Redmine integration in a project, navigate to the +1. To enable the Redmine integration in a project, navigate to the [Integrations page](project_services.md#accessing-the-project-services), click the **Redmine** service, and fill in the required details on the page as described in the table below. -| Field | Description | -| ----- | ----------- | -| `description` | A name for the issue tracker (to differentiate between instances, for example) | -| `project_url` | The URL to the project in Redmine which is being linked to this GitLab project | -| `issues_url` | The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | -| `new_issue_url` | This is the URL to create a new issue in Redmine for the project linked to this GitLab project | + | Field | Description | + | ----- | ----------- | + | `description` | A name for the issue tracker (to differentiate between instances, for example) | + | `project_url` | The URL to the project in Redmine which is being linked to this GitLab project | + | `issues_url` | The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | + | `new_issue_url` | This is the URL to create a new issue in Redmine for the project linked to this GitLab project | -Once you have configured and enabled Redmine: + Once you have configured and enabled Redmine: + - the **Issues** link on the GitLab project pages takes you to the appropriate + Redmine issue index + - clicking **New issue** on the project dashboard creates a new Redmine issue -- the **Issues** link on the GitLab project pages takes you to the appropriate - Redmine issue index -- clicking **New issue** on the project dashboard creates a new Redmine issue + As an example, below is a configuration for a project named gitlab-ci. -As an example, below is a configuration for a project named gitlab-ci. +  - +2. To disable the internal issue tracking system in a project, navigate to the General page, expand [Permissions](../settings/index.md#sharing-and-permissions), and slide the Issues switch invalid. + +  ## Referencing issues in Redmine diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 5896f8f72a0..eafdd28071d 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -731,8 +731,6 @@ X-Gitlab-Event: Merge Request Hook "title": "MS-Viewport", "created_at": "2013-12-03T17:23:34Z", "updated_at": "2013-12-03T17:23:34Z", - "st_commits": null, - "st_diffs": null, "milestone_id": null, "state": "opened", "merge_status": "unchecked", diff --git a/doc/user/project/issues/automatic_issue_closing.md b/doc/user/project/issues/automatic_issue_closing.md index 402a2a3c727..b9607243c8a 100644 --- a/doc/user/project/issues/automatic_issue_closing.md +++ b/doc/user/project/issues/automatic_issue_closing.md @@ -12,8 +12,9 @@ in the project's default branch. If a commit message or merge request description contains a sentence matching a certain regular expression, all issues referenced from the matched text will -be closed. This happens when the commit is pushed to a project's **default** -branch, or when a commit or merge request is merged into it. +be closed. This happens when the commit is pushed to a project's +[**default** branch](../repository/branches/index.md#default-branch), or when a +commit or merge request is merged into it. ## Default closing pattern value diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index b8dd96087f1..43713855e26 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -21,7 +21,7 @@ want to add. --- -Select the user and the [permission level](../../user/permissions.md) +Select the user and the [permission level](../../permissions.md) that you'd like to give the user. Note that you can select more than one user.  diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 4b2e042251b..d76ea259301 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -27,7 +27,7 @@ With GitLab merge requests, you can: - [Resolve merge conflicts from the UI](#resolve-conflicts) - Enable [fast-forward merge requests](#fast-forward-merge-requests) - Enable [semi-linear history merge requests](#semi-linear-history-merge-requests) as another security layer to guarantee the pipeline is passing in the target branch - +- [Create new merge requests by email](#create_by_email) With **[GitLab Enterprise Edition][ee]**, you can also: @@ -132,6 +132,14 @@ those conflicts in the GitLab UI. [Learn more about resolving merge conflicts in the UI.](resolve_conflicts.md) +## Create new merge requests by email + +You can create a new merge request by sending an email to a user-specific email +address. The address can be obtained on the merge requests page by clicking on +a **Email a new merge request to this project** button. The subject will be +used as the source branch name for the new merge request and the target branch +will be the default branch for the project. + ## Revert changes GitLab implements Git's powerful feature to revert any commit with introducing diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index 271adee7da1..17dcd152363 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -230,7 +230,7 @@ test: - docker run $CI_REGISTRY/group/other-project:latest ``` -[job permissions]: ../permissions.md#jobs-permissions +[job permissions]: ../permissions.md#job-permissions [comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302 [ext]: ../permissions.md#external-users [gitsub]: ../../ci/git_submodules.md diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index 453e10184f0..1e19f422d94 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -62,7 +62,7 @@ which is highly recommendable and much faster than hardcoding. If you set up a GitLab Pages project on GitLab.com, it will automatically be accessible under a -[subdomain of `namespace.pages.io`](introduction.md#gitlab-pages-on-gitlab-com). +[subdomain of `namespace.gitlab.io`](introduction.md#gitlab-pages-on-gitlab-com). The `namespace` is defined by your username on GitLab.com, or the group name you created this project under. diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index abe6b4cbd8e..8404d789de6 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -1,49 +1,78 @@ -# GitLab Pages documentation - -With GitLab Pages you can create static websites for your GitLab projects, -groups, or user accounts. You can use any static website generator: Jekyll, -Middleman, Hexo, Hugo, Pelican, you name it! Connect as many customs domains -as you like and bring your own TLS certificate to secure them. - -Here's some info we've gathered to get you started. - -## General info - -- [Product webpage](https://pages.gitlab.io) -- ["We're bringing GitLab Pages to CE"](https://about.gitlab.com/2016/12/24/were-bringing-gitlab-pages-to-community-edition/) -- [Pages group - templates](https://gitlab.com/pages) -- [General user documentation](introduction.md) -- [Admin documentation - Set GitLab Pages on your own GitLab instance](../../../administration/pages/index.md) -- ["We are changing the IP of GitLab Pages on GitLab.com"](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) - -## Getting started - -- **GitLab Pages from A to Z** - - [Part 1: Static sites and GitLab Pages domains](getting_started_part_one.md) - - [Part 2: Quick start guide - Setting up GitLab Pages](getting_started_part_two.md) - - [Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](getting_started_part_three.md) - - [Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages](getting_started_part_four.md) -- **Static Site Generators - Blog posts series** - - [SSGs part 1: Static vs dynamic websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) - - [SSGs part 2: Modern static site generators](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) - - [SSGs part 3: Build any SSG site with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) -- **Secure GitLab Pages custom domain with SSL/TLS certificates** - - [Let's Encrypt](https://about.gitlab.com/2016/04/11/tutorial-securing-your-gitlab-pages-with-tls-and-letsencrypt/) - - [CloudFlare](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) - - [StartSSL](https://about.gitlab.com/2016/06/24/secure-gitlab-pages-with-startssl/) -- **General** - - [Hosting on GitLab.com with GitLab Pages](https://about.gitlab.com/2016/04/07/gitlab-pages-setup/) a comprehensive step-by-step guide - - [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/) - -## Video tutorials - -- [How to publish a website with GitLab Pages on GitLab.com: from a forked project](https://youtu.be/TWqh9MtT4Bg) -- [How to Enable GitLab Pages for GitLab CE and EE (for Admins only)](https://youtu.be/dD8c7WNcc6s) +# GitLab Pages + +With GitLab Pages you can host your website at no cost. + +Your files live in a GitLab project's [repository](../repository/index.md), +from which you can deploy [static websites](#explore-gitlab-pages). +GitLab Pages supports all static site generators (SSGs). + +## Getting Started + +Follow the steps below to get your website live. They shouldn't take more than +5 minutes to complete: + +- 1. [Fork](../../../gitlab-basics/fork-project.md#how-to-fork-a-project) an [example project](https://gitlab.com/pages) +- 2. Change a file to trigger a GitLab CI/CD pipeline +- 3. Visit your project's **Settings > Pages** to see your **website link**, and click on it. Bam! Your website is live. + +_Further steps (optional):_ + +- 4. Remove the [fork relationship](getting_started_part_two.md#fork-a-project-to-get-started-from) (_You don't need the relationship unless you intent to contribute back to the example project you forked from_). +- 5. Make it a [user/group website](getting_started_part_one.md#user-and-group-websites) + +**Watch a video with the steps above: https://www.youtube.com/watch?v=TWqh9MtT4Bg** + +_Advanced options:_ + +- [Use a custom domain](getting_started_part_three.md#adding-your-custom-domain-to-gitlab-pages) +- Apply [SSL/TLS certification](getting_started_part_three.md#ssl-tls-certificates) to your custom domain + +## Explore GitLab Pages + +With GitLab Pages you can create [static websites](getting_started_part_one.md#what-you-need-to-know-before-getting-started) +for your GitLab projects, groups, or user accounts. You can use any static +website generator: Jekyll, Middleman, Hexo, Hugo, Pelican, you name it! +Connect as many custom domains as you like and bring your own TLS certificate +to secure them. + +Read the following tutorials to know more about: + +- [Static websites and GitLab Pages domains](getting_started_part_one.md) +- [Forking projects and creating new ones from scratch, URLs and baseurls](getting_started_part_two.md) +- [Custom domains and subdomains, DNS records, SSL/TLS certificates](getting_started_part_three.md) +- [How to create your own `.gitlab-ci.yml` for your site](getting_started_part_four.md) +- [Technical aspects, custom 404 pages, limitations](introduction.md) +- [Hosting on GitLab.com with GitLab Pages](https://about.gitlab.com/2016/04/07/gitlab-pages-setup/) (outdated) + +_Blog posts series about Static Site Generators (SSGs):_ + +- [SSGs part 1: Static vs dynamic websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) +- [SSGs part 2: Modern static site generators](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) +- [SSGs part 3: Build any SSG site with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) + +_Blog posts for securing GitLab Pages custom domains with SSL/TLS certificates:_ + +- [CloudFlare](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) +- [Let's Encrypt](https://about.gitlab.com/2016/04/11/tutorial-securing-your-gitlab-pages-with-tls-and-letsencrypt/) (outdated) +- [StartSSL](https://about.gitlab.com/2016/06/24/secure-gitlab-pages-with-startssl/) (deprecated) ## Advanced use -- **Blog Posts** - - [GitLab CI: Run jobs sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/) - - [GitLab CI: Deployment & environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - - [Building a new GitLab docs site with Nanoc, GitLab CI, and GitLab Pages](https://about.gitlab.com/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/) - - [Publish code coverage reports with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/) +- [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/) +- [GitLab CI: Run jobs sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/) +- [GitLab CI: Deployment & environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) +- [Building a new GitLab docs site with Nanoc, GitLab CI, and GitLab Pages](https://about.gitlab.com/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/) +- [Publish code coverage reports with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/) + +## Admin GitLab Pages for CE and EE + +Enable and configure GitLab Pages on your own instance (GitLab Community Edition and Enterprise Editions) with +the [admin guide](../../../administration/pages/index.md). + +**Watch the video: https://www.youtube.com/watch?v=dD8c7WNcc6s** + +## More information about GitLab Pages + +- For an overview, visit the [feature webpage](https://about.gitlab.com/features/pages/) +- Announcement (2016-12-24): ["We're bringing GitLab Pages to CE"](https://about.gitlab.com/2016/12/24/were-bringing-gitlab-pages-to-community-edition/) +- Announcement (2017-03-06): ["We are changing the IP of GitLab Pages on GitLab.com"](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) diff --git a/doc/user/project/pipelines/job_artifacts.md b/doc/user/project/pipelines/job_artifacts.md index f9a268fb789..402989f4508 100644 --- a/doc/user/project/pipelines/job_artifacts.md +++ b/doc/user/project/pipelines/job_artifacts.md @@ -44,7 +44,7 @@ the artifacts will be kept forever. For more examples on artifacts, follow the [artifacts reference in `.gitlab-ci.yml`](../../../ci/yaml/README.md#artifacts). -## Browsing job artifacts +## Browsing artifacts >**Note:** With GitLab 9.2, PDFs, images, videos and other formats can be previewed @@ -77,7 +77,7 @@ one HTML file that you can view directly online when --- -## Downloading job artifacts +## Downloading artifacts If you need to download the whole archive, there are buttons in various places inside GitLab that make that possible. @@ -102,7 +102,7 @@ inside GitLab that make that possible.  -## Downloading the latest job artifacts +## Downloading the latest artifacts It is possible to download the latest artifacts of a job via a well known URL so you can use it for scripting purposes. @@ -163,6 +163,18 @@ information in the UI.  +## Erasing artifacts + +DANGER: **Warning:** +This is a destructive action that leads to data loss. Use with caution. + +If you have at least Developer [permissions](../../permissions.md#gitlab-ci-cd-permissions) +on the project, you can erase a single job via the UI which will also remove the +artifacts and the job's trace. + +1. Navigate to a job's page. +1. Click the trash icon at the top right of the job's trace. +1. Confirm the deletion. [expiry date]: ../../../ci/yaml/README.md#artifacts-expire_in [ce-14399]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14399 diff --git a/doc/user/project/pipelines/schedules.md b/doc/user/project/pipelines/schedules.md index 9ad15a12c3c..2101e3b1d58 100644 --- a/doc/user/project/pipelines/schedules.md +++ b/doc/user/project/pipelines/schedules.md @@ -5,7 +5,7 @@ - In 9.2, the feature was [renamed to Pipeline Schedule][ce-10853]. - Cron notation is parsed by [Rufus-Scheduler](https://github.com/jmettraux/rufus-scheduler). -Pipeline schedules can be used to run pipelines only once, or for example every +Pipeline schedules can be used to run a pipeline at specific intervals, for example every month on the 22nd for a certain branch. ## Using Pipeline schedules @@ -44,7 +44,7 @@ GitLab CI so that they can be used in your `.gitlab-ci.yml` file. To configure that a job can be executed only when the pipeline has been scheduled (or the opposite), you can use -[only and except](../../../ci/yaml/README.md#only-and-except) configuration keywords. +[only and except](../../../ci/yaml/README.md#only-and-except-simplified) configuration keywords. ``` job:on-schedule: diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 56f58fd755a..daa5463d680 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -115,10 +115,12 @@ pages. Depending on the status of your job, a badge can have the following values: +- pending - running -- success +- passed - failed - skipped +- canceled - unknown You can access a pipeline status badge image using the following link: diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 23b1c61cd16..1b8a84c9599 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -30,7 +30,8 @@ with all their related data and be moved into a new GitLab instance. | GitLab version | Import/Export version | | ---------------- | --------------------- | -| 10.0 to current | 0.2.0 | +| 10.3 to current | 0.2.1 | +| 10.0 | 0.2.0 | | 9.4.0 | 0.1.8 | | 9.2.0 | 0.1.7 | | 8.17.0 | 0.1.6 | diff --git a/doc/workflow/importing/README.md b/doc/workflow/importing/README.md index f753708ad89..e0a445920b4 100644 --- a/doc/workflow/importing/README.md +++ b/doc/workflow/importing/README.md @@ -1 +1 @@ -This document was moved to a [new location](../../user/project/import/index.md).
+This document was moved to [another location](../../user/project/import/index.md).
diff --git a/doc/workflow/importing/import_projects_from_bitbucket.md b/doc/workflow/importing/import_projects_from_bitbucket.md index 248c3990372..ec9a11f390e 100644 --- a/doc/workflow/importing/import_projects_from_bitbucket.md +++ b/doc/workflow/importing/import_projects_from_bitbucket.md @@ -1 +1 @@ -This document was moved to a [new location](../../user/project/import/bitbucket.md).
+This document was moved to [another location](../../user/project/import/bitbucket.md).
diff --git a/doc/workflow/importing/import_projects_from_fogbugz.md b/doc/workflow/importing/import_projects_from_fogbugz.md index 050746e2b4d..876eb0434f0 100644 --- a/doc/workflow/importing/import_projects_from_fogbugz.md +++ b/doc/workflow/importing/import_projects_from_fogbugz.md @@ -1 +1 @@ -This document was moved to a [new location](../../user/project/import/fogbugz.md). +This document was moved to [another location](../../user/project/import/fogbugz.md). diff --git a/doc/workflow/importing/import_projects_from_gitea.md b/doc/workflow/importing/import_projects_from_gitea.md index cb90c490b0f..8b55b6c23eb 100644 --- a/doc/workflow/importing/import_projects_from_gitea.md +++ b/doc/workflow/importing/import_projects_from_gitea.md @@ -1 +1 @@ -This document was moved to a [new location](../../user/project/import/gitea.md). +This document was moved to [another location](../../user/project/import/gitea.md). diff --git a/doc/workflow/importing/import_projects_from_github.md b/doc/workflow/importing/import_projects_from_github.md index 13639feaa04..72dfe5403c3 100644 --- a/doc/workflow/importing/import_projects_from_github.md +++ b/doc/workflow/importing/import_projects_from_github.md @@ -1 +1 @@ -This document was moved to a [new location](../../user/project/import/github.md).
+This document was moved to [another location](../../user/project/import/github.md).
diff --git a/doc/workflow/importing/import_projects_from_gitlab_com.md b/doc/workflow/importing/import_projects_from_gitlab_com.md index df4c180919a..3256088c014 100644 --- a/doc/workflow/importing/import_projects_from_gitlab_com.md +++ b/doc/workflow/importing/import_projects_from_gitlab_com.md @@ -1 +1 @@ -This document was moved to a [new location](../../user/project/import/gitlab_com.md). +This document was moved to [another location](../../user/project/import/gitlab_com.md). diff --git a/doc/workflow/importing/migrating_from_svn.md b/doc/workflow/importing/migrating_from_svn.md index 81df3fbcdbb..32a75a6c6af 100644 --- a/doc/workflow/importing/migrating_from_svn.md +++ b/doc/workflow/importing/migrating_from_svn.md @@ -1 +1 @@ -This document was moved to a [new location](../../user/project/import/svn.md). +This document was moved to [another location](../../user/project/import/svn.md). |
