diff options
Diffstat (limited to 'doc')
31 files changed, 770 insertions, 243 deletions
diff --git a/doc/README.md b/doc/README.md index 993b30ccdb5..909740211a6 100644 --- a/doc/README.md +++ b/doc/README.md @@ -44,7 +44,7 @@ - [Operations](administration/operations.md) Keeping GitLab up and running. - [Raketasks](raketasks/README.md) Backups, maintenance, automatic webhook setup and the importing of projects. - [Repository checks](administration/repository_checks.md) Periodic Git repository checks. -- [Repository storages](administration/repository_storages.md) Manage the paths used to store repositories. +- [Repository storage paths](administration/repository_storage_paths.md) Manage the paths used to store repositories. - [Security](security/README.md) Learn what you can do to further secure your GitLab instance. - [System hooks](system_hooks/system_hooks.md) Notifications when users, projects and keys are changed. - [Update](update/README.md) Update guides to upgrade your installation. @@ -61,7 +61,6 @@ - [Sidekiq Troubleshooting](administration/troubleshooting/sidekiq.md) Debug when Sidekiq appears hung and is not processing jobs. - [High Availability](administration/high_availability/README.md) Configure multiple servers for scaling or high availability. - [Container Registry](administration/container_registry.md) Configure Docker Registry with GitLab. -- [Multiple mountpoints for the repositories storage](administration/repository_storages.md) Define multiple repository storage paths to distribute the storage load. ## Contributor documentation diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 04723365277..f6027b2f99e 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -65,11 +65,15 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # # Example: 'Paris' or 'Acme, Ltd.' label: 'LDAP' - + + # Example: 'ldap.mydomain.com' host: '_your_ldap_server' + # This port is an example, it is sometimes different but it is always an integer and not a string port: 389 uid: 'sAMAccountName' method: 'plain' # "tls" or "ssl" or "plain" + + # Examples: 'america\\momo' or 'CN=Gitlab Git,CN=Users,DC=mydomain,DC=com' bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' password: '_the_password_of_the_bind_user' @@ -101,7 +105,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # Base where we can search for users # - # Ex. ou=People,dc=gitlab,dc=example + # Ex. 'ou=People,dc=gitlab,dc=example' or 'DC=mydomain,DC=com' # base: '' @@ -112,6 +116,9 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # # Note: GitLab does not support omniauth-ldap's custom filter syntax. # + # Below an example for get only specific users + # Example: '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))' + # user_filter: '' # LDAP attributes that GitLab will use to create an account for the LDAP user. diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index d7cfb464f74..a6300e18dc0 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -379,6 +379,10 @@ Read more about the individual driver's config options in the filesystem. Remember to enable backups with your object storage provider if desired. +> **Important** Enabling storage driver other than `filesystem` would mean +that your Docker client needs to be able to access the storage backend directly. +So you must use an address that resolves and is accessible outside GitLab server. + --- **Omnibus GitLab installations** diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 33b9b28433a..5b6ee354887 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -172,14 +172,14 @@ Omnibus packages. ``` cd /home/git/gitlab -sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production ``` For omnibus versions, the unoptimized assets (JavaScript, CSS) are frozen at the release of upstream GitLab. The omnibus version includes optimized versions of those assets. Unless you are modifying the JavaScript / CSS code on your production machine after installing the package, there should be no reason to redo -rake assets:precompile on the production machine. If you suspect that assets +rake gitlab:assets:compile on the production machine. If you suspect that assets have been corrupted, you should reinstall the omnibus package. ## Tracking Deployments diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index bc2b1f20ed3..ee37ea49874 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -13,12 +13,12 @@ checks failed you can see their output on the admin log page under ## Periodic checks -GitLab periodically runs a repository check on all project repositories and -wiki repositories in order to detect data corruption problems. A -project will be checked no more than once per week. If any projects +When enabled, GitLab periodically runs a repository check on all project +repositories and wiki repositories in order to detect data corruption problems. +A project will be checked no more than once per month. If any projects fail their repository checks all GitLab administrators will receive an email -notification of the situation. This notification is sent out no more -than once a day. +notification of the situation. This notification is sent out once a week on +Sunday, by default. ## Disabling periodic checks diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md new file mode 100644 index 00000000000..d6aa6101026 --- /dev/null +++ b/doc/administration/repository_storage_paths.md @@ -0,0 +1,102 @@ +# Repository storage paths + +> [Introduced][ce-4578] in GitLab 8.10. + +GitLab allows you to define multiple repository storage paths to distribute the +storage load between several mount points. + +>**Notes:** +> +- You must have at least one storage path called `default`. +- The paths are defined in key-value pairs. The key is an arbitrary name you + can pick to name the file path. +- The target directories and any of its subpaths must not be a symlink. + +## Configure GitLab + +>**Warning:** +In order for [backups] to work correctly, the storage path must **not** be a +mount point and the GitLab user should have correct permissions for the parent +directory of the path. In Omnibus GitLab this is taken care of automatically, +but for source installations you should be extra careful. +> +The thing is that for compatibility reasons `gitlab.yml` has a different +structure than Omnibus. In `gitlab.yml` you indicate the path for the +repositories, for example `/home/git/repositories`, while in Omnibus you +indicate `git_data_dirs`, which for the example above would be `/home/git`. +Then, Omnibus will create a `repositories` directory under that path to use with +`gitlab.yml`. +> +This little detail matters because while restoring a backup, the current +contents of `/home/git/repositories` [are moved to][raketask] `/home/git/repositories.old`, +so if `/home/git/repositories` is the mount point, then `mv` would be moving +things between mount points, and bad things could happen. Ideally, +`/home/git` would be the mount point, so then things would be moving within the +same mount point. This is guaranteed with Omnibus installations (because they +don't specify the full repository path but the parent path), but not for source +installations. + +--- + +Now that you've read that big fat warning above, let's edit the configuration +files and add the full paths of the alternative repository storage paths. In +the example below, we add two more mountpoints that are named `nfs` and `cephfs` +respectively. + +**For installations from source** + +1. Edit `gitlab.yml` and add the storage paths: + + ```yaml + repositories: + # Paths where repositories can be stored. Give the canonicalized absolute pathname. + # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!! + storages: # You must have at least a 'default' storage path. + default: /home/git/repositories + nfs: /mnt/nfs/repositories + cephfs: /mnt/cephfs/repositories + ``` + +1. [Restart GitLab] for the changes to take effect. + +>**Note:** +The [`gitlab_shell: repos_path` entry][repospath] in `gitlab.yml` will be +deprecated and replaced by `repositories: storages` in the future, so if you +are upgrading from a version prior to 8.10, make sure to add the configuration +as described in the step above. After you make the changes and confirm they are +working, you can remove the `repos_path` line. + +--- + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the + default one: + + ```ruby + git_data_dirs({ + "default" => "/var/opt/gitlab/git-data", + "nfs" => "/mnt/nfs/git-data", + "cephfs" => "/mnt/cephfs/git-data" + }) + ``` + + Note that Omnibus stores the repositories in a `repositories` subdirectory + of the `git-data` directory. + +## Choose where new project repositories will be stored + +Once you set the multiple storage paths, you can choose where new projects will +be stored via the **Application Settings** in the Admin area. + + + +Beginning with GitLab 8.13.4, multiple paths can be chosen. New projects will be +randomly placed on one of the selected paths. + +[ce-4578]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4578 +[restart gitlab]: restart_gitlab.md#installations-from-source +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure +[backups]: ../raketasks/backup_restore.md +[raketask]: https://gitlab.com/gitlab-org/gitlab-ce/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56 +[repospath]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-9-stable/config/gitlab.yml.example#L457 diff --git a/doc/administration/repository_storages.md b/doc/administration/repository_storages.md index ab70557b69a..9d41ba77f34 100644 --- a/doc/administration/repository_storages.md +++ b/doc/administration/repository_storages.md @@ -1,102 +1,3 @@ # Repository storages -> [Introduced][ce-4578] in GitLab 8.10. - -GitLab allows you to define multiple repository storage paths to distribute the -storage load between several mount points. - ->**Notes:** -> -- You must have at least one storage path called `default`. -- The paths are defined in key-value pairs. The key is an arbitrary name you - can pick to name the file path. -- The target directories and any of its subpaths must not be a symlink. - -## Configure GitLab - ->**Warning:** -In order for [backups] to work correctly, the storage path must **not** be a -mount point and the GitLab user should have correct permissions for the parent -directory of the path. In Omnibus GitLab this is taken care of automatically, -but for source installations you should be extra careful. -> -The thing is that for compatibility reasons `gitlab.yml` has a different -structure than Omnibus. In `gitlab.yml` you indicate the path for the -repositories, for example `/home/git/repositories`, while in Omnibus you -indicate `git_data_dirs`, which for the example above would be `/home/git`. -Then, Omnibus will create a `repositories` directory under that path to use with -`gitlab.yml`. -> -This little detail matters because while restoring a backup, the current -contents of `/home/git/repositories` [are moved to][raketask] `/home/git/repositories.old`, -so if `/home/git/repositories` is the mount point, then `mv` would be moving -things between mount points, and bad things could happen. Ideally, -`/home/git` would be the mount point, so then things would be moving within the -same mount point. This is guaranteed with Omnibus installations (because they -don't specify the full repository path but the parent path), but not for source -installations. - ---- - -Now that you've read that big fat warning above, let's edit the configuration -files and add the full paths of the alternative repository storage paths. In -the example below, we add two more mountpoints that are named `nfs` and `cephfs` -respectively. - -**For installations from source** - -1. Edit `gitlab.yml` and add the storage paths: - - ```yaml - repositories: - # Paths where repositories can be stored. Give the canonicalized absolute pathname. - # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!! - storages: # You must have at least a 'default' storage path. - default: /home/git/repositories - nfs: /mnt/nfs/repositories - cephfs: /mnt/cephfs/repositories - ``` - -1. [Restart GitLab] for the changes to take effect. - ->**Note:** -The [`gitlab_shell: repos_path` entry][repospath] in `gitlab.yml` will be -deprecated and replaced by `repositories: storages` in the future, so if you -are upgrading from a version prior to 8.10, make sure to add the configuration -as described in the step above. After you make the changes and confirm they are -working, you can remove the `repos_path` line. - ---- - -**For Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the - default one: - - ```ruby - git_data_dirs({ - "default" => "/var/opt/gitlab/git-data", - "nfs" => "/mnt/nfs/git-data", - "cephfs" => "/mnt/cephfs/git-data" - }) - ``` - - Note that Omnibus stores the repositories in a `repositories` subdirectory - of the `git-data` directory. - -## Choose where new project repositories will be stored - -Once you set the multiple storage paths, you can choose where new projects will -be stored via the **Application Settings** in the Admin area. - - - -Beginning with GitLab 8.13.4, multiple paths can be chosen. New projects will be -randomly placed on one of the selected paths. - -[ce-4578]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4578 -[restart gitlab]: restart_gitlab.md#installations-from-source -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure -[backups]: ../raketasks/backup_restore.md -[raketask]: https://gitlab.com/gitlab-org/gitlab-ce/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56 -[repospath]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-9-stable/config/gitlab.yml.example#L457 +This document was moved to a [new location](repository_storage_paths.md). diff --git a/doc/api/commits.md b/doc/api/commits.md index 5c11d0f83bb..53ce381c8ae 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -245,7 +245,7 @@ Example response: ```json [ { - "diff": "--- a/doc/update/5.4-to-6.0.md\n+++ b/doc/update/5.4-to-6.0.md\n@@ -71,6 +71,8 @@\n sudo -u git -H bundle exec rake migrate_keys RAILS_ENV=production\n sudo -u git -H bundle exec rake migrate_inline_notes RAILS_ENV=production\n \n+sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production\n+\n ```\n \n ### 6. Update config files", + "diff": "--- a/doc/update/5.4-to-6.0.md\n+++ b/doc/update/5.4-to-6.0.md\n@@ -71,6 +71,8 @@\n sudo -u git -H bundle exec rake migrate_keys RAILS_ENV=production\n sudo -u git -H bundle exec rake migrate_inline_notes RAILS_ENV=production\n \n+sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production\n+\n ```\n \n ### 6. Update config files", "new_path": "doc/update/5.4-to-6.0.md", "old_path": "doc/update/5.4-to-6.0.md", "a_mode": null, diff --git a/doc/api/groups.md b/doc/api/groups.md index f7807390e68..3b38e3e1bee 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -25,7 +25,14 @@ GET /groups "id": 1, "name": "Foobar Group", "path": "foo-bar", - "description": "An interesting group" + "description": "An interesting group", + "visibility_level": 20, + "lfs_enabled": true, + "avatar_url": "http://localhost:3000/uploads/group/avatar/1/foo.jpg", + "web_url": "http://localhost:3000/groups/foo-bar", + "request_access_enabled": false, + "full_name": "Foobar Group", + "full_path": "foo-bar" } ] ``` @@ -149,6 +156,8 @@ Example response: "avatar_url": null, "web_url": "https://gitlab.example.com/groups/twitter", "request_access_enabled": false, + "full_name": "Foobar Group", + "full_path": "foo-bar", "projects": [ { "id": 7, @@ -372,6 +381,8 @@ Example response: "avatar_url": null, "web_url": "http://gitlab.example.com/groups/h5bp", "request_access_enabled": false, + "full_name": "Foobar Group", + "full_path": "foo-bar", "projects": [ { "id": 9, diff --git a/doc/ci/autodeploy/index.md b/doc/ci/autodeploy/index.md index 630207ffa09..c4c4d95b68a 100644 --- a/doc/ci/autodeploy/index.md +++ b/doc/ci/autodeploy/index.md @@ -1,6 +1,6 @@ # Auto deploy -> [Introduced][mr-8135] in GitLab 8.15. +> [Introduced][mr-8135] in GitLab 8.15. Currently requires a [Public project][project-settings]. Auto deploy is an easy way to configure GitLab CI for the deployment of your application. GitLab Community maintains a list of `.gitlab-ci.yml` @@ -33,6 +33,7 @@ enable [Kubernetes service][kubernetes-service]. created automatically for you. [mr-8135]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8135 +[project-settings]: https://docs.gitlab.com/ce/public_access/public_access.html [project-services]: ../../project_services/project_services.md [auto-deploy-templates]: https://gitlab.com/gitlab-org/gitlab-ci-yml/tree/master/autodeploy [kubernetes-service]: ../../project_services/kubernetes.md diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 98cd29c9567..ef04c537367 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -297,7 +297,7 @@ deploy_review: - echo "Deploy a review app" environment: name: review/$CI_BUILD_REF_NAME - url: https://$CI_BUILD_REF_SLUG.review.example.com + url: https://$CI_ENVIRONMENT_SLUG.example.com only: - branches except: @@ -318,15 +318,15 @@ also contain `/`, or other characters that would be invalid in a domain name or URL, we use `$CI_ENVIRONMENT_SLUG` in the `environment:url` so that the environment can get a specific and distinct URL for each branch. In this case, given a `$CI_BUILD_REF_NAME` of `100-Do-The-Thing`, the URL will be something -like `https://review-100-do-the-4f99a2.example.com`. Again, the way you set up +like `https://100-do-the-4f99a2.example.com`. Again, the way you set up the web server to serve these requests is based on your setup. You could also use `$CI_BUILD_REF_SLUG` in `environment:url`, e.g.: -`https://$CI_BUILD_REF_SLUG.review.example.com`. We use `$CI_ENVIRONMENT_SLUG` +`https://$CI_BUILD_REF_SLUG.example.com`. We use `$CI_ENVIRONMENT_SLUG` here because it is guaranteed to be unique, but if you're using a workflow like [GitLab Flow][gitlab-flow], collisions are very unlikely, and you may prefer environment names to be more closely based on the branch name - the example -above would give you an URL like `https://100-do-the-thing.review.example.com` +above would give you an URL like `https://100-do-the-thing.example.com` Last but not least, we tell the job to run [`only`][only] on branches [`except`][only] master. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index ffc310ec8c7..5377bf9ee80 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -14,6 +14,12 @@ Apart from those, here is an collection of tutorials and guides on setting up yo - [Test a Phoenix application](test-phoenix-application.md) - [Using `dpl` as deployment tool](deployment/README.md) - [Example project that shows how to use Review Apps](https://gitlab.com/gitlab-examples/review-apps-nginx/) +- [Run PHP Composer & NPM scripts then deploy them to a staging server](deployment/composer-npm-deploy.md) +- Help your favorite programming language and GitLab by sending a merge request + with a guide for that language. + +## Outside the documentation + - [Blog post about using GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) - [Repositories with examples for various languages](https://gitlab.com/groups/gitlab-examples) - [The .gitlab-ci.yml file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md new file mode 100644 index 00000000000..5334a73e1f5 --- /dev/null +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -0,0 +1,156 @@ +## Running Composer and NPM scripts with deployment via SCP + +This guide covers the building dependencies of a PHP project while compiling assets via an NPM script. + +While is possible to create your own image with custom PHP and Node JS versions, for brevity, we will use an existing [Docker image](https://hub.docker.com/r/tetraweb/php/) that contains both PHP and NodeJS installed. + + +```yaml +image: tetraweb/php +``` + +The next step is to install zip/unzip packages and make composer available. We will place these in the `before_script` section: + +```yaml +before_script: + - apt-get update + - apt-get install zip unzip + - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" + - php composer-setup.php + - php -r "unlink('composer-setup.php');" +``` + +This will make sure we have all requirements ready. Next, we want to run `composer update` to fetch all PHP dependencies and `npm install` to load node packages, then run the `npm` script. We need to append them into `before_script` section: + +```yaml +before_script: + # ... + - php composer.phar update + - npm install + - npm run deploy +``` + +In this particular case, the `npm deploy` script is a Gulp script that does the following: + +1. Compile CSS & JS +2. Create sprites +3. Copy various assets (images, fonts) around +4. Replace some strings + +All these operations will put all files into a `build` folder, which is ready to be deployed to a live server. + +### How to transfer files to a live server? + +You have multiple options: rsync, scp, sftp and so on. For now, we will use scp. + +To make this work, you need to add a GitLab Secret Variable (accessible on _gitlab.example/your-project-name/variables_). That variable will be called `STAGING_PRIVATE_KEY` and it's the **private** ssh key of your server. + +#### Security tip + +Create a user that has access **only** to the folder that needs to be updated! + +After you create that variable, you need to make sure that key will be added to the docker container on run: + +```yaml +before_script: + # - .... + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + - mkdir -p ~/.ssh + - eval $(ssh-agent -s) + - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' +``` + +In order, this means that: + +1. We check if the `ssh-agent` is available and we install it if it's not; +2. We create the `~/.ssh` folder; +3. We make sure we're running bash; +4. We disable host checking (we don't ask for user accept when we first connect to a server; and since every build will equal a first connect, we kind of need this) + +And this is basically all you need in the `before_script` section. + +## How to deploy things? + +As we stated above, we need to deploy the `build` folder from the docker image to our server. To do so, we create a new job: + +```yaml +stage_deploy: + artifacts: + paths: + - build/ + only: + - dev + script: + - ssh-add <(echo "$STAGING_PRIVATE_KEY") + - ssh -p22 server_user@server_host "mkdir htdocs/wp-content/themes/_tmp" + - scp -P22 -r build/* server_user@server_host:htdocs/wp-content/themes/_tmp + - ssh -p22 server_user@server_host "mv htdocs/wp-content/themes/live htdocs/wp-content/themes/_old && mv htdocs/wp-content/themes/_tmp htdocs/wp-content/themes/live" + - ssh -p22 server_user@server_host "rm -rf htdocs/wp-content/themes/_old" +``` + +### What's going on here? + +1. `only:dev` means that this build will run only when something is pushed to the `dev` branch. You can remove this block completely and have everything be ran on every push (but probably this is something you don't want) +2. `ssh-add ...` we will add that private key you added on the web UI to the docker container +3. We will connect via `ssh` and create a new `_tmp` folder +4. We will connect via `scp` and upload the `build` folder (which was generated by a `npm` script) to our previously created `_tmp` folder +5. We will connect again to `ssh` and move the `live` folder to an `_old` folder, then move `_tmp` to `live`. +6. We connect to ssh and remove the `_old` folder + +What's the deal with the artifacts? We just tell GitLab CI to keep the `build` directory (later on, you can download that as needed). + +#### Why we do it this way? + +If you're using this only for stage server, you could do this in two steps: + +```yaml +- ssh -p22 server_user@server_host "rm -rf htdocs/wp-content/themes/live/*" +- scp -P22 -r build/* server_user@server_host:htdocs/wp-content/themes/live +``` + +The problem is that there will be a small period of time when you won't have the app on your server. + +So we use so many steps because we want to make sure that at any given time we have a functional app in place. + +## Where to go next? + +Since this was a WordPress project, I gave real life code snippets. Some ideas you can pursuit: + +- Having a slightly different script for `master` branch will allow you to deploy to a production server from that branch and to a stage server from any other branches; +- Instead of pushing it live, you can push it to WordPress official repo (with creating a SVN commit & stuff); +- You could generate i18n text domains on the fly. + +--- + +Our final `.gitlab-ci.yml` will look like this: + +```yaml +image: tetraweb/php + +before_script: + - apt-get update + - apt-get install zip unzip + - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" + - php composer-setup.php + - php -r "unlink('composer-setup.php');" + - php composer.phar update + - npm install + - npm run deploy + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + - mkdir -p ~/.ssh + - eval $(ssh-agent -s) + - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' + +stage_deploy: + artifacts: + paths: + - build/ + only: + - dev + script: + - ssh-add <(echo "$STAGING_PRIVATE_KEY") + - ssh -p22 server_user@server_host "mkdir htdocs/wp-content/themes/_tmp" + - scp -P22 -r build/* server_user@server_host:htdocs/wp-content/themes/_tmp + - ssh -p22 server_user@server_host "mv htdocs/wp-content/themes/live htdocs/wp-content/themes/_old && mv htdocs/wp-content/themes/_tmp htdocs/wp-content/themes/live" + - ssh -p22 server_user@server_host "rm -rf htdocs/wp-content/themes/_old" +```
\ No newline at end of file diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index 82ffb841729..5eeec92d976 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -235,7 +235,11 @@ cache: before_script: # Install composer dependencies -- curl --silent --show-error https://getcomposer.org/installer | php +- wget https://composer.github.io/installer.sig -O - -q | tr -d '\n' > installer.sig +- php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" +- php -r "if (hash_file('SHA384', 'composer-setup.php') === file_get_contents('installer.sig')) { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" +- php composer-setup.php +- php -r "unlink('composer-setup.php'); unlink('installer.sig');" - php composer.phar install ... diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 75a0897eb15..f11257be5c3 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -86,7 +86,7 @@ used for time of the build. The configuration of this feature is covered in ### before_script `before_script` is used to define the command that should be run before all -builds, including deploy builds. This can be an array or a multi-line string. +builds, including deploy builds, but after the restoration of artifacts. This can be an array or a multi-line string. ### after_script diff --git a/doc/development/README.md b/doc/development/README.md index 6f2ca7b8590..265df98fb87 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -18,6 +18,7 @@ - [Frontend guidelines](frontend.md) - [SQL guidelines](sql.md) for working with SQL queries - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers +- [`Gemfile` guidelines](gemfile.md) ## Process diff --git a/doc/development/frontend.md b/doc/development/frontend.md index f79bd23dc90..75fdf3d8e63 100644 --- a/doc/development/frontend.md +++ b/doc/development/frontend.md @@ -23,6 +23,69 @@ some ideas with React.js as well as Angular. To get started with Vue, read through [their documentation][vue-docs]. +#### How to build a new feature with Vue.js +**Components, Stores and Services** + +In some features implemented with Vue.js, like the [issue board][issue-boards] +or [environments table][environments-table] +you can find a clear separation of concerns: + +``` +new_feature +├── components +│ └── component.js.es6 +│ └── ... +├── store +│ └── new_feature_store.js.es6 +├── service +│ └── new_feature_service.js.es6 +├── new_feature_bundle.js.es6 +``` +_For consistency purposes, we recommend you to follow the same structure._ + +Let's look into each of them: + +**A `*_bundle.js` file** + +This is the index file of your new feature. This is where the root Vue instance +of the new feature should be. + +Don't forget to follow [these steps.][page-specific-javascript] + +**A folder for Components** + +This folder holds all components that are specific of this new feature. +If you need to use or create a component that will probably be used somewhere +else, please refer to `vue_shared/components`. + +A good thumb rule to know when you should create a component is to think if +it will be reusable elsewhere. + +For example, tables are used in a quite amount of places across GitLab, a table +would be a good fit for a component. +On the other hand, a table cell used only in on table, would not be a good use +of this pattern. + +You can read more about components in Vue.js site, [Component System][component-system] + +**A folder for the Store** + +The Store is a simple object that allows us to manage the state in a single +source of truth. + +The concept we are trying to follow is better explained by Vue documentation +itself, please read this guide: [State Management][state-management] + +**A folder for the Service** + +The Service is used only to communicate with the server. +It does not store or manipulate any data. +We use [vue-resource][vue-resource-repo] to +communicate with the server. + +The [issue boards service][issue-boards-service] +is a good example of this pattern. + ## Performance ### Resources @@ -198,8 +261,8 @@ As long as the fixtures don't change, `rake teaspoon:tests` is sufficient If you need to debug your tests and/or application code while they're running, navigate to [localhost:3000/teaspoon](http://localhost:3000/teaspoon) -in your browser, open DevTools, and run tests for individual files by clicking -on them. This is also much faster than setting up and running tests from the +in your browser, open DevTools, and run tests for individual files by clicking +on them. This is also much faster than setting up and running tests from the command line. Please note: Not all of the frontend fixtures are generated. Some are still static @@ -294,20 +357,27 @@ For our currently-supported browsers, see our [requirements][requirements]. [xss]: https://en.wikipedia.org/wiki/Cross-site_scripting [scss-style-guide]: scss_styleguide.md [requirements]: ../install/requirements.md#supported-web-browsers +[issue-boards]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/app/assets/javascripts/boards +[environments-table]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/app/assets/javascripts/environments +[page_specific_javascript]: https://docs.gitlab.com/ce/development/frontend.html#page-specific-javascript +[component-system]: https://vuejs.org/v2/guide/#Composing-with-Components +[state-management]: https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch +[vue-resource-repo]: https://github.com/pagekit/vue-resource +[issue-boards-service]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/boards/services/board_service.js.es6 ## Gotchas ### Spec errors due to use of ES6 features in `.js` files -If you see very generic JavaScript errors (e.g. `jQuery is undefined`) being -thrown in Teaspoon, Spinach, or Rspec tests but can't reproduce them manually, -you may have included `ES6`-style JavaScript in files that don't have the -`.js.es6` file extension. Either use ES5-friendly JavaScript or rename the file -you're working in (`git mv <file.js> <file.js.es6>`). +If you see very generic JavaScript errors (e.g. `jQuery is undefined`) being +thrown in Teaspoon, Spinach, or Rspec tests but can't reproduce them manually, +you may have included `ES6`-style JavaScript in files that don't have the +`.js.es6` file extension. Either use ES5-friendly JavaScript or rename the file +you're working in (`git mv <file.js> <file.js.es6>`). ### Spec errors due to use of unsupported JavaScript -Similar errors will be thrown if you're using JavaScript features not yet +Similar errors will be thrown if you're using JavaScript features not yet supported by our test runner's version of webkit, whether or not you've updated the file extension. Examples of unsupported JavaScript features are: @@ -322,20 +392,20 @@ the file extension. Examples of unsupported JavaScript features are: - Symbol/Symbol.iterator - Spread -Until these are polyfilled or transpiled appropriately, they should not be used. -Please update this list with additional unsupported features or when any of +Until these are polyfilled or transpiled appropriately, they should not be used. +Please update this list with additional unsupported features or when any of these are made usable. ### Spec errors due to JavaScript not enabled -If, as a result of a change you've made, a feature now depends on JavaScript to +If, as a result of a change you've made, a feature now depends on JavaScript to run correctly, you need to make sure a JavaScript web driver is enabled when -specs are run. If you don't you'll see vague error messages from the spec -runner, and an explosion of vague console errors in the HTML snapshot. +specs are run. If you don't you'll see vague error messages from the spec +runner, and an explosion of vague console errors in the HTML snapshot. -To enable a JavaScript driver in an `rspec` test, add `js: true` to the -individual spec or the context block containing multiple specs that need -JavaScript enabled: +To enable a JavaScript driver in an `rspec` test, add `js: true` to the +individual spec or the context block containing multiple specs that need +JavaScript enabled: ```ruby @@ -354,8 +424,8 @@ describe "Admin::AbuseReports", js: true do end ``` -In Spinach, the JavaScript driver is enabled differently. In the `*.feature` -file for the failing spec, add the `@javascript` flag above the Scenario: +In Spinach, the JavaScript driver is enabled differently. In the `*.feature` +file for the failing spec, add the `@javascript` flag above the Scenario: ``` @javascript diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md new file mode 100644 index 00000000000..ec9718cea71 --- /dev/null +++ b/doc/development/gemfile.md @@ -0,0 +1,14 @@ +# `Gemfile` guidelines + +When adding a new entry to `Gemfile` or upgrading an existing dependency pay +attention to the following rules. + +## No gems fetched from git repositories + +We do not allow gems that are fetched from git repositories. All gems have +to be available in the RubyGems index. We want to minimize external build +dependencies and build times. + +## License compliance + +Refer to [licensing guidelines](licensing.md) for ensuring license compliance. diff --git a/doc/development/performance.md b/doc/development/performance.md index f936a49a2aa..c1f129e576c 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -211,6 +211,41 @@ suite first. See the [StackProf documentation](https://github.com/tmm1/stackprof/blob/master/README.md) for details. +## RSpec profiling + +GitLab's development environment also includes the +[rspec_profiling](https://github.com/foraker/rspec_profiling) gem, which is used +to collect data on spec execution times. This is useful for analyzing the +performance of the test suite itself, or seeing how the performance of a spec +may have changed over time. + +To activate profiling in your local environment, run the following: + +``` +$ export RSPEC_PROFILING=yes +$ rake rspec_profiling:install +``` + +This creates an SQLite3 database in `tmp/rspec_profiling`, into which statistics +are saved every time you run specs with the `RSPEC_PROFILING` environment +variable set. + +Ad-hoc investigation of the collected results can be performed in an interactive +shell: + +``` +$ rake rspec_profiling:console +irb(main):001:0> results.count +=> 231 +irb(main):002:0> results.last.attributes.keys +=> ["id", "commit", "date", "file", "line_number", "description", "time", "status", "exception", "query_count", "query_time", "request_count", "request_time", "created_at", "updated_at"] +irb(main):003:0> results.where(status: "passed").average(:time).to_s +=> "0.211340155844156" +``` +These results can also be placed into a PostgreSQL database by setting the +`RSPEC_PROFILING_POSTGRES_URL` variable. This is used to profile the test suite +when running in the CI environment. + ## Importance of Changes When working on performance improvements, it's important to always ask yourself diff --git a/doc/development/ux_guide/components.md b/doc/development/ux_guide/components.md index 706bb180912..1b19587a0b8 100644 --- a/doc/development/ux_guide/components.md +++ b/doc/development/ux_guide/components.md @@ -109,7 +109,7 @@ Dropdowns are used to allow users to choose one (or many) options from a list of ### Max size -The max height for dropdowns should target **10-15 items**. If the height of the dropdown is too large, the list becomes very hard to parse and it is easy to visually lose track of the item you are looking for. Usability also suffers as more mouse movement is required, and you have a larger area in which you hijack the scroll away from the page level. While it may initially seem counterintuitive to not show as many items as you can, it is actually quicker and easier to process the information when it is cropped at a reasonable height. +The max height for dropdowns should target **10-15** single line items, or **7-10** multi-line items. If the height of the dropdown is too large, the list becomes very hard to parse and it is easy to visually lose track of the item you are looking for. Usability also suffers as more mouse movement is required, and you have a larger area in which you hijack the scroll away from the page level. While it may initially seem counterintuitive to not show as many items as you can, it is actually quicker and easier to process the information when it is cropped at a reasonable height. --- diff --git a/doc/development/ux_guide/copy.md b/doc/development/ux_guide/copy.md index 31cc9dd2a53..5b65d531e54 100644 --- a/doc/development/ux_guide/copy.md +++ b/doc/development/ux_guide/copy.md @@ -102,6 +102,12 @@ When using the <kbd>Alt</kbd> keystrokes in Windows, use the numeric keypad, not ## Terminology Only use the terms in the tables below. +### Projects and Groups + +| Term | Use | :no_entry_sign: Don't | +| ---- | --- | ----- | +| Members | When discussing the people who are a part of a project or a group. | Don't use `users`. | + ### Issues #### Adjectives (states) @@ -117,7 +123,7 @@ Use `5 open issues` and don’t use `5 pending issues`. #### Verbs (actions) -| Term | Use | Don’t | +| Term | Use | :no_entry_sign: Don’t | | ---- | --- | --- | | Add | Add an issue | Don’t use `create` or `new` | | View | View an open or closed issue || @@ -158,7 +164,7 @@ The form should be titled `Edit issue`. The submit button should be labeled `Sav #### Verbs (actions) -| Term | Use | Don’t | +| Term | Use | :no_entry_sign: Don’t | | ---- | --- | --- | | Add | Add a merge request | Do not use `create` or `new` | | View | View an open or merged merge request || diff --git a/doc/gitlab-basics/add-image.md b/doc/gitlab-basics/add-image.md index 476b48a217c..1a44123aa81 100644 --- a/doc/gitlab-basics/add-image.md +++ b/doc/gitlab-basics/add-image.md @@ -1,62 +1,56 @@ # How to add an image -The following are the steps to add images to your repository in -GitLab: +Using your standard tool for copying files (e.g. Finder in Mac OS, or Explorer +in Windows, or...), put the image file into the GitLab project. You can find the +project as a regular folder in your files. -Find the image that you’d like to add. +Go to your [shell](command-line-commands.md), and move into the folder of your +Gitlab project. This usually means running the following command until you get +to the desired destination: -In your computer files, find the GitLab project to which you'd like to add the image -(you'll find it as a regular file). Click on every file until you find exactly where you'd -like to add the image. There, paste the image. - -Go to your [shell](command-line-commands.md), and add the following commands: - -Add this command for every directory that you'd like to open: ``` -cd NAME-OF-FILE-YOU'D-LIKE-TO-OPEN +cd NAME-OF-FOLDER-YOU'D-LIKE-TO-OPEN ``` -Create a new branch: -``` -git checkout -b NAME-OF-BRANCH -``` +Check if your image is actually present in the directory (if you are in Windows, +use `dir` instead): -Check if your image was correctly added to the directory: ``` ls ``` You should see the name of the image in the list shown. -Move up the hierarchy through directories: -``` -cd ../ -``` +Check the status: -Check the status and you should see your image’s name in red: ``` git status ``` -Add your changes: +Your image's name should appear in red, so `git` took notice of it! Now add it +to the repository: + ``` git add NAME-OF-YOUR-IMAGE ``` -Check the status and you should see your image’s name in green: +Check the status again, your image's name should have turned green: + ``` git status ``` -Add the commit: +Commit: + ``` -git commit -m “DESCRIBE COMMIT IN A FEW WORDS” +git commit -m "DESCRIBE COMMIT IN A FEW WORDS" ``` -Now you can push (send) your changes (in the branch NAME-OF-BRANCH) to GitLab (the git remote named 'origin'): +Now you can push (send) your changes (in the branch NAME-OF-BRANCH) to GitLab +(the git remote named 'origin'): + ``` git push origin NAME-OF-BRANCH ``` -Your image will be added to your branch in your repository in GitLab. Create a [Merge Request](add-merge-request.md) -to integrate your changes to your project. +Your image will be added to your branch in your repository in GitLab. diff --git a/doc/install/database_mysql.md b/doc/install/database_mysql.md index 322680f0cf4..65bfb0f7d6e 100644 --- a/doc/install/database_mysql.md +++ b/doc/install/database_mysql.md @@ -4,7 +4,7 @@ We do not recommend using MySQL due to various issues. For example, case [(in)sensitivity](https://dev.mysql.com/doc/refman/5.0/en/case-sensitivity.html) and [problems](https://bugs.mysql.com/bug.php?id=65830) that [suggested](https://bugs.mysql.com/bug.php?id=50909) [fixes](https://bugs.mysql.com/bug.php?id=65830) [have](https://bugs.mysql.com/bug.php?id=63164). -## MySQL +## Initial database setup # Install the database packages sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev @@ -32,8 +32,11 @@ We do not recommend using MySQL due to various issues. For example, case [(in)se # If this fails, check your MySQL config files (e.g. `/etc/mysql/*.cnf`, `/etc/mysql/conf.d/*`) for the setting "innodb = off" mysql> SET storage_engine=INNODB; + # If you have MySQL < 5.7.7 and want to enable utf8mb4 character set support with your GitLab install, you must set the following NOW: + mysql> SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda, innodb_large_prefix=1; + # Create the GitLab production database - mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; + mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_general_ci`; # Grant the GitLab user necessary permissions on the database mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES ON `gitlabhq_production`.* TO 'git'@'localhost'; @@ -51,7 +54,203 @@ We do not recommend using MySQL due to various issues. For example, case [(in)se # Quit the database session mysql> \q - # You are done installing the database and can go back to the rest of the installation. + # You are done installing the database for now and can go back to the rest of the installation. + +Please proceed to the rest of the installation before running through the utf8mb4 support section. + + +### MySQL utf8mb4 support + +After installation or upgrade, remember to [convert any new tables](#convert) to `utf8mb4`/`utf8mb4_general_ci`. + +--- + +GitLab 8.14 has introduced [a feature](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7420) requiring `utf8mb4` encoding to be supported in your GitLab MySQL Database, which is not the case if you have setup your database before GitLab 8.16. + +Follow the below instructions to ensure you use the most up to date requirements for your GitLab MySQL Database. + +**We are about to do the following:** +- Ensure you can enable `utf8mb4` encoding and `utf8mb4_general_ci` collation for your GitLab DB, tables and data. +- Convert your GitLab tables and data from `utf8`/`utf8_general_ci` to `utf8mb4`/`utf8mb4_general_ci` + +### Check for utf8mb4 support + +#### Check for InnoDB File-Per-Table Tablespaces + +We need to check, enable and maybe convert your existing GitLab DB tables to the [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) as a prerequise for supporting **utfb8mb4 with long indexes** required by recent GitLab databases. + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + mysql > use gitlabhq_production; + + # Check your MySQL version is >= 5.5.3 (GitLab requires 5.5.14+) + mysql > SHOW VARIABLES LIKE 'version'; + +---------------+-----------------+ + | Variable_name | Value | + +---------------+-----------------+ + | version | 5.5.53-0+deb8u1 | + +---------------+-----------------+ + + # Note where is your MySQL data dir for later: + mysql > select @@datadir; + +----------------+ + | @@datadir | + +----------------+ + | /var/lib/mysql | + +----------------+ + + # Note whether your MySQL server runs with innodb_file_per_table ON or OFF: + mysql> SELECT @@innodb_file_per_table; + +-------------------------+ + | @@innodb_file_per_table | + +-------------------------+ + | 1 | + +-------------------------+ + + # You can now quit the database session + mysql> \q + +> You need **MySQL 5.5.3 or later** to perform this update. + +Whatever the results of your checks above, we now need to check if your GitLab database has been created using [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) (i.e. `innodb_file_per_table` was set to **1** at initial setup time). + +> Note: This setting is [enabled by default](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_file_per_table) since MySQL 5.6.6. + + # Run this command with root privileges, replace the data dir if different: + sudo ls -lh /var/lib/mysql/gitlabhq_production/*.ibd | wc -l + + # Run this command with root privileges, replace the data dir if different: + sudo ls -lh /var/lib/mysql/gitlabhq_production/*.frm | wc -l + + +- **Case 1: a result > 0 for both commands** + +Congrats, your GitLab database uses the right InnoDB tablespace format. + +However, you must still ensure that any **future tables** created by GitLab will still use the right format: + +- If `SELECT @@innodb_file_per_table` returned **1** previously, your server is running correctly. +> It's however a requirement to check *now* that this setting is indeed persisted in your [my.cnf](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) file! + +- If `SELECT @@innodb_file_per_table` returned **0** previously, your server is not running correctly. +> [Enable innodb_file_per_table](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) by running in a MySQL session as root the command `SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda;` and persist the two settings in your [my.cnf](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) file + +Now, if you have a **different result** returned by the 2 commands above, it means you have a **mix of tables format** uses in your GitLab database. This can happen if your MySQL server had different values for `innodb_file_per_table` in its life and you updated GitLab at different moments with those inconsistent values. So keep reading. + +- **Case 2: a result equals to "0" OR not the same result for both commands** + +Unfortunately, none or only some of your GitLab database tables use the GitLab requirement of [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html). + +Let's enable what we need on the running server: + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + + # Enable innodb_file_per_table and set innodb_file_format on the running server: + mysql > SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda; + + # You can now quit the database session + mysql> \q + +> Now, **persist** [innodb_file_per_table](https://dev.mysql.com/doc/refman/5.6/en/tablespace-enabling.html) and [innodb_file_format](https://dev.mysql.com/doc/refman/5.6/en/innodb-file-format-enabling.html) in your `my.cnf` file. + +Ensure at this stage that your GitLab instance is indeed **stopped**. + +Now, let's convert all the GitLab database tables to the new tablespace format: + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + mysql > use gitlabhq_production; + + # Safety check: you should still have those values set as follow: + mysql> SELECT @@innodb_file_per_table, @@innodb_file_format; + +-------------------------+----------------------+ + | @@innodb_file_per_table | @@innodb_file_format | + +-------------------------+----------------------+ + | 1 | Barracuda | + +-------------------------+----------------------+ + + mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE"; + + # If previous query returned results, copy & run all shown SQL statements + + # You can now quit the database session + mysql> \q + +--- + +#### Check for proper InnoDB File Format, Row Format, Large Prefix and tables conversion + +We need to check, enable and probably convert your existing GitLab DB tables to use the [Barracuda InnoDB file format](https://dev.mysql.com/doc/refman/5.6/en/innodb-file-format.html), the [DYNAMIC row format](https://dev.mysql.com/doc/refman/5.6/en/glossary.html#glos_dynamic_row_format) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) as a second prerequisite for supporting **utfb8mb4 with long indexes** used by recent GitLab databases. + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + mysql > use gitlabhq_production; + + # Set innodb_file_format and innodb_large_prefix on the running server: + # Note: These are the default settings only for MySQL 5.7.7 and later. + + mysql > SET GLOBAL innodb_file_format=Barracuda, innodb_large_prefix=1; + + # Your DB must be (still) using utf8/utf8_general_ci as default encoding and collation. + # We will NOT change the default encoding and collation on the DB in order to support future GitLab migrations creating tables + # that require "long indexes support" on installations using MySQL <= 5.7.9. + # However, when such migrations occur, you will have to follow this guide again to convert the newly created tables to the proper encoding/collation. + + # This should return the following: + mysql> SELECT @@character_set_database, @@collation_database; + +--------------------------+----------------------+ + | @@character_set_database | @@collation_database | + +--------------------------+----------------------+ + | utf8 | utf8_general_ci | + +--------------------------+----------------------+ + +> Now, ensure that [innodb_file_format](https://dev.mysql.com/doc/refman/5.6/en/tablespace-enabling.html) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) are **persisted** in your `my.cnf` file. + +#### Tables and data conversion to utf8mb4 +<a name="convert"></a> + +Now that you have a persistent MySQL setup, you can safely upgrade tables after setup or upgrade time: + + # Convert tables not using ROW_FORMAT DYNAMIC: + + mysql> SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ROW_FORMAT=DYNAMIC;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE" AND ROW_FORMAT!="Dynamic"; + + # !! If previous query returned results, copy & run all shown SQL statements + + # Convert tables/columns not using utf8mb4/utf8mb4_general_ci as encoding/collation: + + mysql > SET foreign_key_checks = 0; + + mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_COLLATION != "utf8mb4_general_ci" AND TABLE_TYPE="BASE TABLE"; + + # !! If previous query returned results, copy & run all shown SQL statements + + # Turn foreign key checks back on + mysql > SET foreign_key_checks = 1; + + # You can now quit the database session + mysql> \q + +Ensure your GitLab database configuration file uses a proper connection encoding and collation: + +```sudo -u git -H editor config/database.yml``` + + production: + adapter: mysql2 + encoding: utf8mb4 + collation: utf8mb4_general_ci + +[Restart your GitLab instance](../administration/restart_gitlab.md). + ## MySQL strings limits diff --git a/doc/install/installation.md b/doc/install/installation.md index 3e7674e13ab..425c5d93efb 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -448,7 +448,7 @@ Check if GitLab and its environment are configured correctly: ### Compile Assets - sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production ### Start Your GitLab Instance diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index 44d2a14f366..713d11b75e4 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -113,14 +113,6 @@ Make sure to follow all steps below: If you are using a custom init script, make sure to edit the above gitlab-workhorse setting as needed. -1. After all the above changes recompile the assets. This is an important task - and will take some time to complete depending on the server resources: - - ``` - cd /home/git/gitlab - sudo -u git -H bundle exec rake assets:clean assets:precompile RAILS_ENV=production - ``` - 1. [Restart GitLab][] for the changes to take effect. ### Disable relative URL in GitLab diff --git a/doc/project_services/slack.md b/doc/project_services/slack.md index 0b682b43810..eaceb2be137 100644 --- a/doc/project_services/slack.md +++ b/doc/project_services/slack.md @@ -15,7 +15,7 @@ Slack: After you set up Slack, it's time to set up GitLab. -Go to your project's **Settings > Services > Slack Notifications** and you will see a +Go to your project's **Settings > Integrations > Slack Notifications** and you will see a checkbox with the following events that can be triggered: - Push diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index c44930a4ceb..ec13c2446ef 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -1,6 +1,6 @@ # System hooks -Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `project_rename`, `project_transfer`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create`, `key_destroy`, `group_create`, `group_destroy`, `user_add_to_group` and `user_remove_from_group`. +Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `project_rename`, `project_transfer`, `project_update`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create`, `key_destroy`, `group_create`, `group_destroy`, `user_add_to_group` and `user_remove_from_group`. System hooks can be used, e.g. for logging or changing information in a LDAP server. @@ -88,6 +88,23 @@ X-Gitlab-Event: System Hook } ``` +**Project updated:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_update", + "name": "StoreCloud", + "owner_email": "johnsmith@gmail.com", + "owner_name": "John Smith", + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", + "project_id": 74, + "project_visibility": "private", +} +``` + **New Team Member:** ```json diff --git a/doc/update/8.15-to-8.16.md b/doc/update/8.15-to-8.16.md index 63f3c3fdda9..2695a16ac0b 100644 --- a/doc/update/8.15-to-8.16.md +++ b/doc/update/8.15-to-8.16.md @@ -97,6 +97,8 @@ sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production ``` +**MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). + ### 6. Update gitlab-workhorse Install and compile gitlab-workhorse. This requires diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 54d523b59fd..154a0f817da 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -57,7 +57,7 @@ sudo -u git -H bundle clean sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production # Clean up assets and cache -sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:assets:clean gitlab:assets:compile cache:clear RAILS_ENV=production ``` ### 4. Update gitlab-workhorse to the corresponding version diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 0f959b956a5..cc688a7f99c 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -140,75 +140,77 @@ into the password field. ## Recovery options -If you lose your code generation device (such as your mobile phone) and you need -to disable two-factor authentication on your account, you have several options. +To disable two-factor authentication on your account (for example, if you +have lost your code generation device) you can: +* [Use a saved recovery code](#use-a-saved-recovery-code) +* [Generate new recovery codes using SSH](#generate-new-recovery-codes-using-SSH) +* [Ask a GitLab administrator to disable two-factor authentication on your account](#ask-a-gitlab-administrator-to-disable-two-factor-authentication-on-your-account) ### Use a saved recovery code -When you enabled two-factor authentication for your account, a series of -recovery codes were generated. If you saved those codes somewhere safe, you -may use one to sign in. +Enabling two-factor authentication for your account generated several recovery +codes. If you saved these codes, you can use one of them to sign in. -First, enter your username/email and password on the GitLab sign in page. When -prompted for a two-factor code, enter one of the recovery codes you saved -previously. +To use a recovery code, enter your username/email and password on the GitLab +sign-in page. When prompted for a two-factor code, enter the recovery code. -> **Note:** Once a particular recovery code has been used, it cannot be used again. - You may still use the other saved recovery codes at a later time. +> **Note:** Once you use a recovery code, you cannot re-use it. You can still + use the other recovery codes you saved. ### Generate new recovery codes using SSH -It's not uncommon for users to forget to save the recovery codes when enabling -two-factor authentication. If you have an SSH key added to your GitLab account, -you can generate a new set of recovery codes using SSH. - -Run `ssh git@gitlab.example.com 2fa_recovery_codes`. You will be prompted to -confirm that you wish to generate new codes. If you choose to continue, any -previously saved codes will be invalidated. - -```bash -$ ssh git@gitlab.example.com 2fa_recovery_codes -Are you sure you want to generate new two-factor recovery codes? -Any existing recovery codes you saved will be invalidated. (yes/no) -yes - -Your two-factor authentication recovery codes are: - -119135e5a3ebce8e -11f6v2a498810dcd -3924c7ab2089c902 -e79a3398bfe4f224 -34bd7b74adbc8861 -f061691d5107df1a -169bf32a18e63e7f -b510e7422e81c947 -20dbed24c5e74663 -df9d3b9403b9c9f0 - -During sign in, use one of the codes above when prompted for -your two-factor code. Then, visit your Profile Settings and add -a new device so you do not lose access to your account again. -``` - -Next, go to the GitLab sign in page and enter your username/email and password. -When prompted for a two-factor code, enter one of the recovery codes obtained -from the command line output. - -> **Note:** After signing in, you should immediately visit your **Profile Settings - -> Account** to set up two-factor authentication with a new device. - -### Ask a GitLab administrator to disable two-factor on your account - -If the above two methods are not possible, you may ask a GitLab global -administrator to disable two-factor authentication for your account. Please -be aware that this will temporarily leave your account in a less secure state. -You should sign in and re-enable two-factor authentication as soon as possible -after the administrator disables it. +Users often forget to save their recovery codes when enabling two-factor +authentication. If an SSH key is added to your GitLab account, you can generate +a new set of recovery codes with SSH. + +1. Run `ssh git@gitlab.example.com 2fa_recovery_codes`. +2. You are prompted to confirm that you want to generate new codes. Continuing this process invalidates previously saved codes. + ``` + bash + $ ssh git@gitlab.example.com 2fa_recovery_codes + Are you sure you want to generate new two-factor recovery codes? + Any existing recovery codes you saved will be invalidated. (yes/no) + + yes + + Your two-factor authentication recovery codes are: + + 119135e5a3ebce8e + 11f6v2a498810dcd + 3924c7ab2089c902 + e79a3398bfe4f224 + 34bd7b74adbc8861 + f061691d5107df1a + 169bf32a18e63e7f + b510e7422e81c947 + 20dbed24c5e74663 + df9d3b9403b9c9f0 + + During sign in, use one of the codes above when prompted for your + two-factor code. Then, visit your Profile Settings and add a new device + so you do not lose access to your account again. + ``` +3. Go to the GitLab sign-in page and enter your username/email and password. When prompted for a two-factor code, enter one of the recovery codes obtained +from the command-line output. + +> **Note:** After signing in, visit your **Profile Settings -> Account** immediately to set up two-factor authentication with a new + device. + +### Ask a GitLab administrator to disable two-factor authentication on your account + +If you cannot use a saved recovery code or generate new recovery codes, ask a +GitLab global administrator to disable two-factor authentication for your +account. This will temporarily leave your account in a less secure state. +Sign in and re-enable two-factor authentication as soon as possible. ## Note to GitLab administrators -You need to take special care to that 2FA keeps working after -[restoring a GitLab backup](../../../raketasks/backup_restore.md). +- You need to take special care to that 2FA keeps working after +[restoring a GitLab backup](../raketasks/backup_restore.md). + +- To ensure 2FA authorizes correctly with TOTP server, you may want to ensure +your GitLab server's time is synchronized via a service like NTP. Otherwise, +you may have cases where authorization always fails because of time differences. [Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en [FreeOTP]: https://fedorahosted.org/freeotp/ diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index dfc762fe1d3..cb1c1a84f8c 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -22,7 +22,8 @@ with all their related data and be moved into a new GitLab instance. | GitLab version | Import/Export version | | -------- | -------- | -| 8.13.0 to current | 0.1.5 | +| 8.16.2 to current | 0.1.6 | +| 8.13.0 | 0.1.5 | | 8.12.0 | 0.1.4 | | 8.10.3 | 0.1.3 | | 8.10.0 | 0.1.2 | @@ -47,6 +48,9 @@ The following items will NOT be exported: - Build traces and artifacts - LFS objects +- Container registry images +- CI variables +- Any encrypted tokens ## Exporting a project and its data |