From 567eac0396cbf39b04320d0aec5a2e4c5afcd2bf Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 17 Sep 2015 16:24:00 -0400 Subject: Reformat and copy edit the CI-to-CE migration guide --- doc/migrate_ci_to_ce/README.md | 418 +++++++++++++++++++++++------------------ 1 file changed, 231 insertions(+), 187 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 1e45f29dbb2..3930b165bbf 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -1,96 +1,105 @@ -## Migrate GitLab CI to GitLab CE/EE +## Migrate GitLab CI to GitLab CE or EE -## Notice +Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise +Edition (EE), GitLab CI is no longer its own application, but is instead built +into the CE and EE applications. -**You need to have working GitLab CI 7.14 to perform migration. -The older versions are not supported and will most likely break migration procedure.** +This guide will detail the process of migrating your CI installation and data +into your GitLab CE or EE installation. -This migration can't be done online and takes significant amount of time. -Make sure to plan it ahead. +### Before we begin -If you are running older version please follow the upgrade guide first: -https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/7.13-to-7.14.md +**You need to have a working installation of GitLab CI version 7.14 to perform +this migration. The older versions are not supported and will most likely break +this migration procedure.** -The migration is divided into a two parts: -1. **[CI]** You will be making a changes to GitLab CI instance. -1. **[CE]** You will be making a changes to GitLab CE/EE instance. +This migration cannot be performed online and takes a significant amount of +time. Make sure to plan ahead. -### 1. Stop CI server [CI] +If you are running a version of GitLab CI prior to 7.14 please follow the +appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/). + +The migration is divided into three parts: + +1. [GitLab CI](#part-i-gitlab-ci) +1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee) +1. [Finishing Up](#part-iii-finishing-up) + +### Part I: GitLab CI + +#### 1. Stop GitLab CI sudo service gitlab_ci stop - -### 2. Backup [CI] -**The migration procedure is database breaking. -You need to create backup if you still want to access GitLab CI in case of failure.** +#### 2. Create a backup + +The migration procedure modifies the structure of the CI database. If something +goes wrong, you will not be able to revert to a previous version without a +backup: ```bash cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production ``` -### 3. Prepare GitLab CI database to migration [CI] - -Copy and paste the command in terminal to rename all tables. -This also breaks your database structure disallowing you to use it anymore. - - cat < gitlab_ci.sql + ``` -#### a. Dump MySQL - - mysqldump --default-character-set=utf8 --complete-insert --no-create-info \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p - GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql - -#### b. Dump PostgreSQL - - pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql - -#### c. Dump MySQL and migrate to PostgreSQL - - # Dump existing MySQL database first - mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p - GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp - - # Convert database to be compatible with PostgreSQL - git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab - python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 - ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed - - # Filter to only include INSERT statements - grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql - -### 6. Make sure that your GitLab CE/EE is 8.0 [CE] - -Please verify that you use GitLab CE/EE 8.0. -If not, please follow the update guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md - -### 7. Stop GitLab CE/EE [CE] - -Before you can migrate data you need to stop GitLab CE/EE first. + - If both your CI and CE (or EE) installations use **postgresql** as the + `adapter`, use `pg_dump`: + + ```sh + pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \ + --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql + ``` + + - If your CI installation uses **mysql2** as the `adapter` and your CE (or + EE) installation uses **postgresql**, use `mysqldump` to dump the database + and then convert it to PostgreSQL using [mysql-postgresql-converter]: + + ```sh + # Dump existing MySQL database first + mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ + --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ + ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ + ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ + ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp + + # Convert database to be compatible with PostgreSQL + git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab + python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 + ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed + + # Filter to only include INSERT statements + grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql + ``` + +[mysql-postgresql-converter]: https://github.com/gitlabhq/mysql-postgresql-converter + +### Part II: GitLab CE (or EE) + +#### 1. Ensure GitLab is updated + +Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow +the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md). + +#### 2. Stop GitLab + +Before you can migrate data you need to stop the GitLab service first: sudo service gitlab stop - -### 8. Backup GitLab CE/EE [CE] -This migration poses a **significant risk** of breaking your GitLab CE/EE. -**You should create the GitLab CI/EE backup before doing it.** +#### 3. Create a backup + +This migration poses a **significant risk** of breaking your GitLab +installation. Create a backup before proceeding: cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production -### 9. Copy secret tokens [CE] +#### 4. Copy secret tokens from CI The `secrets.yml` file stores encryption keys for secure variables. -You need to copy the content of `config/secrets.yml` to the same file in GitLab CE. +You need to copy the contents of GitLab CI's `config/secrets.yml` file to the +same file in GitLab CE: sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml sudo chown git:git /home/git/gitlab/config/secrets.yml sudo chown 0600 /home/git/gitlab/config/secrets.yml - -### 10. New configuration options for `gitlab.yml` [CE] -There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). -View them with the command below and apply them manually to your current `gitlab.yml`: +#### 5. New configuration options for `gitlab.yml` + +There are new configuration options available for `gitlab.yml`. View them with +the command below and apply them manually to your current `gitlab.yml`: ```sh git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example ``` -The new options include configuration of GitLab CI that are now being part of GitLab CE and EE. +The new options include configuration settings for GitLab CI. -### 11. Copy build logs [CE] +#### 6. Copy build logs -You need to copy the contents of `builds/` to the same directory in GitLab CE/EE. +You need to copy the contents of GitLab CI's `builds/` directory to the +corresponding directory in GitLab CE or EE: sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds sudo chown -R git:git /home/git/gitlab/builds -The build traces are usually quite big so it will take a significant amount of time. +The build logs are usually quite big so it may take a significant amount of +time. -### 12. Import GitLab CI database [CE] +#### 7. Import GitLab CI database -The one of the last steps is to import existing GitLab CI database. +Now you'll import the GitLab CI database dump that you [created +earlier](#create-a-database-dump) into the GitLab CE or EE database: sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql sudo chown git:git /home/git/gitlab/gitlab_ci.sql sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production -The task does: +This task will: + 1. Delete data from all existing CI tables -1. Import database data -1. Fix database auto increments +1. Import data from database dump +1. Fix database auto-increments 1. Fix tags assigned to Builds and Runners 1. Fix services used by CI -### 13. Start GitLab [CE] +#### 8. Start GitLab -You can start GitLab CI/EE now and see if everything is working. +You can start GitLab CI (or EE) now and see if everything is working: sudo service gitlab start -### 14. Update nginx [CI] - -Now get back to GitLab CI and update **Nginx** configuration in order to: -1. Have all existing runners able to communicate with a migrated GitLab CI. -1. Have GitLab able send build triggers to CI address specified in Project's settings -> Services -> GitLab CI. - -You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste: - - # GITLAB CI - server { - listen 80 default_server; # e.g., listen 192.168.1.1:80; - server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; - - access_log /var/log/nginx/gitlab_ci_access.log; - error_log /var/log/nginx/gitlab_ci_error.log; - - # expose API to fix runners - location /api { - proxy_read_timeout 300; - proxy_connect_timeout 300; - proxy_redirect off; - proxy_set_header X-Real-IP $remote_addr; - - # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN - resolver 8.8.8.8 8.8.4.4; - proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; - } - - # expose build endpoint to allow trigger builds - location ~ ^/projects/\d+/build$ { - proxy_read_timeout 300; - proxy_connect_timeout 300; - proxy_redirect off; - proxy_set_header X-Real-IP $remote_addr; - - # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN - resolver 8.8.8.8 8.8.4.4; - proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; - } - - # redirect all other CI requests - location / { - return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; - } - - # adjust this to match the largest build log your runners might submit, - # set to 0 to disable limit - client_max_body_size 10m; - } - -Make sure to fill the blanks to match your setup: -1. **YOUR_CI_SERVER_FQDN**: The existing public facing address of GitLab CI, eg. ci.gitlab.com. -1. **YOUR_GITLAB_SERVER_FQDN**: The public facing address of GitLab CE/EE, eg. gitlab.com. - -**Make sure to not remove the `/ci$request_uri`. This is required to properly forward the requests.** - -You should also make sure that you can do: +### Part III: Finishing Up + +#### 1. Update Nginx configuration + +To ensure that your existing CI runners are able to communicate with the +migrated installation, and that existing build triggers still work, you'll need +to update your Nginx configuration to redirect requests for the old locations to +the new ones. + +Edit `/etc/nginx/sites-available/gitlab_ci` and paste: + +```nginx +# GITLAB CI +server { + listen 80 default_server; # e.g., listen 192.168.1.1:80; + server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; + + access_log /var/log/nginx/gitlab_ci_access.log; + error_log /var/log/nginx/gitlab_ci_error.log; + + # expose API to fix runners + location /api { + proxy_read_timeout 300; + proxy_connect_timeout 300; + proxy_redirect off; + proxy_set_header X-Real-IP $remote_addr; + + # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN + resolver 8.8.8.8 8.8.4.4; + proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # expose build endpoint to allow trigger builds + location ~ ^/projects/\d+/build$ { + proxy_read_timeout 300; + proxy_connect_timeout 300; + proxy_redirect off; + proxy_set_header X-Real-IP $remote_addr; + + # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN + resolver 8.8.8.8 8.8.4.4; + proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # redirect all other CI requests + location / { + return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # adjust this to match the largest build log your runners might submit, + # set to 0 to disable limit + client_max_body_size 10m; +} +``` + +Make sure you substitute these placeholder values with your real ones: + +1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI + install (e.g., `ci.gitlab.com`). +1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab + CE (or EE) install (e.g., `gitlab.com`). + +**Make sure not to remove the `/ci$request_uri` part. This is required to properly forward the requests.** + +You should also make sure that you can: + 1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server. -1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE/EE server. +1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server. -## Check your configuration +#### 2. Check Nginx configuration sudo nginx -t -## Restart nginx +#### 3. Restart Nginx sudo /etc/init.d/nginx restart -### 15. Done! +#### 4. Done! -If everything went OK you should be able to access all your GitLab CI data by pointing your browser to: -https://gitlab.example.com/ci/. +If everything went well you should be able to access your migrated CI install by +visiting `https://gitlab.example.com/ci/`. -The GitLab CI should also work when using the previous address, redirecting you to the GitLab CE/EE. +If you visit the old GitLab CI address, you should be redirected to the new one. **Enjoy!** -- cgit v1.2.1 From 8cc788be4ee4465db9fc42db0b4d8f219e0bc2e9 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 17 Sep 2015 16:29:06 -0400 Subject: Indent database dump steps one more level [ci skip] --- doc/migrate_ci_to_ce/README.md | 78 +++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 39 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 3930b165bbf..412aded9ea6 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -118,45 +118,45 @@ each installation. values `IN_UPPERCASE` with the corresponding values from your **CI installation's** `config/database.yml` files above. - - If both your CI and CE (or EE) installations use **mysql2** as the `adapter`, use - `mysqldump`: - - ```sh - mysqldump --default-character-set=utf8 --complete-insert --no-create-info \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql - ``` - - - If both your CI and CE (or EE) installations use **postgresql** as the - `adapter`, use `pg_dump`: - - ```sh - pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \ - --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql - ``` - - - If your CI installation uses **mysql2** as the `adapter` and your CE (or - EE) installation uses **postgresql**, use `mysqldump` to dump the database - and then convert it to PostgreSQL using [mysql-postgresql-converter]: - - ```sh - # Dump existing MySQL database first - mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp - - # Convert database to be compatible with PostgreSQL - git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab - python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 - ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed - - # Filter to only include INSERT statements - grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql - ``` + - If both your CI and CE (or EE) installations use **mysql2** as the `adapter`, use + `mysqldump`: + + ```sh + mysqldump --default-character-set=utf8 --complete-insert --no-create-info \ + --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ + ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ + ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ + ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql + ``` + + - If both your CI and CE (or EE) installations use **postgresql** as the + `adapter`, use `pg_dump`: + + ```sh + pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \ + --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql + ``` + + - If your CI installation uses **mysql2** as the `adapter` and your CE (or + EE) installation uses **postgresql**, use `mysqldump` to dump the database + and then convert it to PostgreSQL using [mysql-postgresql-converter]: + + ```sh + # Dump existing MySQL database first + mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ + --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ + ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ + ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ + ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp + + # Convert database to be compatible with PostgreSQL + git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab + python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 + ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed + + # Filter to only include INSERT statements + grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql + ``` [mysql-postgresql-converter]: https://github.com/gitlabhq/mysql-postgresql-converter -- cgit v1.2.1 From eae049505c3bb6a09e524949e81335d0aa3f8e6f Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 17 Sep 2015 16:35:59 -0400 Subject: Fix link to "create a database dump" header --- doc/migrate_ci_to_ce/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 412aded9ea6..2f59dc69943 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -217,7 +217,7 @@ time. #### 7. Import GitLab CI database Now you'll import the GitLab CI database dump that you [created -earlier](#create-a-database-dump) into the GitLab CE or EE database: +earlier](#5-create-a-database-dump) into the GitLab CE or EE database: sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql sudo chown git:git /home/git/gitlab/gitlab_ci.sql -- cgit v1.2.1 From 62c0e5ab6a8ff88510d9ed0e2731ae6fda8bbacd Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 17 Sep 2015 16:38:31 -0400 Subject: Redcarpet didn't like unordered lists deeply nested in an ordered one I don't really like making these an ordered list since they're mutually exclusive, but I think the structure is what's important. [ci skip] --- doc/migrate_ci_to_ce/README.md | 82 +++++++++++++++++++++--------------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 2f59dc69943..34b70c7140c 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -111,52 +111,52 @@ each installation. # socket: /tmp/mysql.sock ``` -1. Depending on the values for `adapter`, you will have to use different - commands to perform the database dump. +1. Depending on the values for `adapter`, you will have to use one of three + different commands to perform the database dump. **NOTE:** For any of the commands below, you'll need to substitute the values `IN_UPPERCASE` with the corresponding values from your **CI installation's** `config/database.yml` files above. - - If both your CI and CE (or EE) installations use **mysql2** as the `adapter`, use - `mysqldump`: - - ```sh - mysqldump --default-character-set=utf8 --complete-insert --no-create-info \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql - ``` - - - If both your CI and CE (or EE) installations use **postgresql** as the - `adapter`, use `pg_dump`: - - ```sh - pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \ - --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql - ``` - - - If your CI installation uses **mysql2** as the `adapter` and your CE (or - EE) installation uses **postgresql**, use `mysqldump` to dump the database - and then convert it to PostgreSQL using [mysql-postgresql-converter]: - - ```sh - # Dump existing MySQL database first - mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp - - # Convert database to be compatible with PostgreSQL - git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab - python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 - ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed - - # Filter to only include INSERT statements - grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql - ``` + 1. If both your CI and CE (or EE) installations use **mysql2** as the `adapter`, use + `mysqldump`: + + ```sh + mysqldump --default-character-set=utf8 --complete-insert --no-create-info \ + --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ + ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ + ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ + ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql + ``` + + 1. If both your CI and CE (or EE) installations use **postgresql** as the + `adapter`, use `pg_dump`: + + ```sh + pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \ + --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql + ``` + + 1. If your CI installation uses **mysql2** as the `adapter` and your CE (or + EE) installation uses **postgresql**, use `mysqldump` to dump the + database and then convert it to PostgreSQL using [mysql-postgresql-converter]: + + ```sh + # Dump existing MySQL database first + mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ + --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ + ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ + ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ + ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp + + # Convert database to be compatible with PostgreSQL + git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab + python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 + ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed + + # Filter to only include INSERT statements + grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql + ``` [mysql-postgresql-converter]: https://github.com/gitlabhq/mysql-postgresql-converter -- cgit v1.2.1 From fc8ae32bcefdeb3fb629d932ba1ac5cc2f31f38d Mon Sep 17 00:00:00 2001 From: Kamil Trzcinski Date: Sun, 20 Sep 2015 23:45:35 +0200 Subject: Update migration guide --- doc/migrate_ci_to_ce/README.md | 152 ++++++----------------------------------- 1 file changed, 22 insertions(+), 130 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 34b70c7140c..ac6bb630b08 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -9,14 +9,14 @@ into your GitLab CE or EE installation. ### Before we begin -**You need to have a working installation of GitLab CI version 7.14 to perform +**You need to have a working installation of GitLab CI version 8.0 to perform this migration. The older versions are not supported and will most likely break this migration procedure.** This migration cannot be performed online and takes a significant amount of time. Make sure to plan ahead. -If you are running a version of GitLab CI prior to 7.14 please follow the +If you are running a version of GitLab CI prior to 8.0 please follow the appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/). The migration is divided into three parts: @@ -42,123 +42,29 @@ cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production ``` -#### 3. Rename database tables +If your GitLab CI installation uses **MySQL** and your GitLab CE uses **PostgreSQL** +you need to convert database data with **MYSQL_TO_POSTGRESQL**. -To prevent naming conflicts with database tables in GitLab CE or EE, we need to -rename CI's tables to begin with a `ci_` prefix: - -```sh -cat < PostgreSQL) execute: - **NOTE:** For any of the commands below, you'll need to substitute the - values `IN_UPPERCASE` with the corresponding values from your **CI - installation's** `config/database.yml` files above. - - 1. If both your CI and CE (or EE) installations use **mysql2** as the `adapter`, use - `mysqldump`: - - ```sh - mysqldump --default-character-set=utf8 --complete-insert --no-create-info \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql - ``` - - 1. If both your CI and CE (or EE) installations use **postgresql** as the - `adapter`, use `pg_dump`: - - ```sh - pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \ - --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql - ``` - - 1. If your CI installation uses **mysql2** as the `adapter` and your CE (or - EE) installation uses **postgresql**, use `mysqldump` to dump the - database and then convert it to PostgreSQL using [mysql-postgresql-converter]: - - ```sh - # Dump existing MySQL database first - mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ - --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \ - ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ - ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ - ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp - - # Convert database to be compatible with PostgreSQL - git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab - python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 - ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed +```bash +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 +``` - # Filter to only include INSERT statements - grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql - ``` +#### 3. Remove cronjob -[mysql-postgresql-converter]: https://github.com/gitlabhq/mysql-postgresql-converter +``` +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec whenever --clear-crontab +``` ### Part II: GitLab CE (or EE) @@ -203,33 +109,19 @@ git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/g The new options include configuration settings for GitLab CI. -#### 6. Copy build logs +#### 6. Copy backup from GitLab CI -You need to copy the contents of GitLab CI's `builds/` directory to the -corresponding directory in GitLab CE or EE: + sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups + sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar - sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds - sudo chown -R git:git /home/git/gitlab/builds - -The build logs are usually quite big so it may take a significant amount of -time. - -#### 7. Import GitLab CI database +#### 7. Import GitLab CI backup Now you'll import the GitLab CI database dump that you [created earlier](#5-create-a-database-dump) into the GitLab CE or EE database: - sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql - sudo chown git:git /home/git/gitlab/gitlab_ci.sql - sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production - -This task will: - -1. Delete data from all existing CI tables -1. Import data from database dump -1. Fix database auto-increments -1. Fix tags assigned to Builds and Runners -1. Fix services used by CI + sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production + +This task will take some time. #### 8. Start GitLab -- cgit v1.2.1 From d511edc952574d0e5f51180648cbfbd2061bfe47 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Sun, 20 Sep 2015 18:55:47 -0400 Subject: More migration guide updates Fixes a few links and typos, and reorganizes the CI "Create a backup" section. --- doc/migrate_ci_to_ce/README.md | 61 +++++++++++++++++++++++------------------- 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index ac6bb630b08..8525fb3847d 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -17,7 +17,7 @@ This migration cannot be performed online and takes a significant amount of time. Make sure to plan ahead. If you are running a version of GitLab CI prior to 8.0 please follow the -appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/). +appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/). The migration is divided into three parts: @@ -35,29 +35,36 @@ The migration is divided into three parts: The migration procedure modifies the structure of the CI database. If something goes wrong, you will not be able to revert to a previous version without a -backup: +backup. -```bash -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production -``` +If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE) +installation uses **PostgreSQL** you'll need to convert the CI database by +setting a `MYSQL_TO_POSTGRESQL` flag. -If your GitLab CI installation uses **MySQL** and your GitLab CE uses **PostgreSQL** -you need to convert database data with **MYSQL_TO_POSTGRESQL**. +You can check which database each install is using by viewing their +database configuration files: -You can check that by looking into GitLab CI and GitLab CE (or EE) database configuration file: +```sh +cat /home/gitlab_ci/gitlab-ci/config/database.yml +cat /home/git/gitlab/config/database.yml +``` - ```sh - cat /home/gitlab_ci/gitlab-ci/config/database.yml - cat /home/git/gitlab/config/database.yml +- If both applications use the same database `adapter`, create the backup with + this command: + + ```bash + cd /home/gitlab_ci/gitlab-ci + sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production ``` -To create backup with database conversion (MySQL -> PostgreSQL) execute: +- If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this + command (note the `MYSQL_TO_POSTGRESQL` flag): -```bash -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 -``` + + ```bash + cd /home/gitlab_ci/gitlab-ci + sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 + ``` #### 3. Remove cronjob @@ -103,9 +110,7 @@ same file in GitLab CE: There are new configuration options available for `gitlab.yml`. View them with the command below and apply them manually to your current `gitlab.yml`: -```sh -git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example -``` + git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example The new options include configuration settings for GitLab CI. @@ -116,16 +121,16 @@ The new options include configuration settings for GitLab CI. #### 7. Import GitLab CI backup -Now you'll import the GitLab CI database dump that you [created -earlier](#5-create-a-database-dump) into the GitLab CE or EE database: +Now you'll import the GitLab CI database dump that you created earlier into the +GitLab CE or EE database: sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production - + This task will take some time. #### 8. Start GitLab -You can start GitLab CI (or EE) now and see if everything is working: +You can start GitLab CE (or EE) now and see if everything is working: sudo service gitlab start @@ -191,7 +196,8 @@ Make sure you substitute these placeholder values with your real ones: 1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab CE (or EE) install (e.g., `gitlab.com`). -**Make sure not to remove the `/ci$request_uri` part. This is required to properly forward the requests.** +**Make sure not to remove the `/ci$request_uri` part. This is required to +properly forward the requests.** You should also make sure that you can: @@ -209,8 +215,7 @@ You should also make sure that you can: #### 4. Done! If everything went well you should be able to access your migrated CI install by -visiting `https://gitlab.example.com/ci/`. - -If you visit the old GitLab CI address, you should be redirected to the new one. +visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI +address, you should be redirected to the new one. **Enjoy!** -- cgit v1.2.1 From 157ee6623c4165f8395eb6db9a36b7e568dd9ebc Mon Sep 17 00:00:00 2001 From: Kamil Trzcinski Date: Mon, 21 Sep 2015 22:36:33 +0200 Subject: Update guide --- doc/migrate_ci_to_ce/README.md | 51 ++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 19 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 8525fb3847d..bddd8a82739 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -30,7 +30,7 @@ The migration is divided into three parts: #### 1. Stop GitLab CI sudo service gitlab_ci stop - + #### 2. Create a backup The migration procedure modifies the structure of the CI database. If something @@ -44,10 +44,8 @@ setting a `MYSQL_TO_POSTGRESQL` flag. You can check which database each install is using by viewing their database configuration files: -```sh -cat /home/gitlab_ci/gitlab-ci/config/database.yml -cat /home/git/gitlab/config/database.yml -``` + cat /home/gitlab_ci/gitlab-ci/config/database.yml + cat /home/git/gitlab/config/database.yml - If both applications use the same database `adapter`, create the backup with this command: @@ -56,22 +54,19 @@ cat /home/git/gitlab/config/database.yml cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production ``` - + - If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this command (note the `MYSQL_TO_POSTGRESQL` flag): - ```bash cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 ``` - + #### 3. Remove cronjob -``` -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec whenever --clear-crontab -``` + cd /home/gitlab_ci/gitlab-ci + sudo -u gitlab_ci -H bundle exec whenever --clear-crontab ### Part II: GitLab CE (or EE) @@ -80,21 +75,33 @@ sudo -u gitlab_ci -H bundle exec whenever --clear-crontab Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md). -#### 2. Stop GitLab +#### 2. Prevent CI usage for time of migration + +As Admin go to Admin Area -> Settings -> and uncheck +**Disable to prevent CI usage until rake ci:migrate is run (8.0 only)**. + +This will prevent from creating the CI projects till you finish migration. + +#### 3. Stop GitLab Before you can migrate data you need to stop the GitLab service first: sudo service gitlab stop -#### 3. Create a backup +#### 4. Create a backup This migration poses a **significant risk** of breaking your GitLab installation. Create a backup before proceeding: cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production + +It's possible to speedup backup creation. To do that you can skip repositories and uploads. -#### 4. Copy secret tokens from CI + cd /home/git/gitlab + sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads + +#### 5. Copy secret tokens from CI The `secrets.yml` file stores encryption keys for secure variables. @@ -105,7 +112,7 @@ same file in GitLab CE: sudo chown git:git /home/git/gitlab/config/secrets.yml sudo chown 0600 /home/git/gitlab/config/secrets.yml -#### 5. New configuration options for `gitlab.yml` +#### 6. New configuration options for `gitlab.yml` There are new configuration options available for `gitlab.yml`. View them with the command below and apply them manually to your current `gitlab.yml`: @@ -114,12 +121,18 @@ the command below and apply them manually to your current `gitlab.yml`: The new options include configuration settings for GitLab CI. -#### 6. Copy backup from GitLab CI +#### 7. Copy backup from GitLab CI sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar -#### 7. Import GitLab CI backup +If moving across the servers you can use **scp**. +However, this requires you to provide authorized key or password to login to GitLab CE servers from CI server. +You can try to use ssh-agent from your local machine to have that: login to your GitLab CI server using `ssh -A`. + + scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup + +#### 8. Import GitLab CI backup Now you'll import the GitLab CI database dump that you created earlier into the GitLab CE or EE database: @@ -128,7 +141,7 @@ GitLab CE or EE database: This task will take some time. -#### 8. Start GitLab +#### 9. Start GitLab You can start GitLab CE (or EE) now and see if everything is working: -- cgit v1.2.1 From 12969a14dd72224017c4852b6c0cb4a6b50317b3 Mon Sep 17 00:00:00 2001 From: Kamil Trzcinski Date: Mon, 21 Sep 2015 22:47:33 +0200 Subject: Add commands for Omnibus installation --- doc/migrate_ci_to_ce/README.md | 73 +++++++++++++++++++++++++++++++++++------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index bddd8a82739..a8614e5fc49 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -19,7 +19,7 @@ time. Make sure to plan ahead. If you are running a version of GitLab CI prior to 8.0 please follow the appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/). -The migration is divided into three parts: +The migration is divided into three parts and covers manual and omnibus installations: 1. [GitLab CI](#part-i-gitlab-ci) 1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee) @@ -29,7 +29,11 @@ The migration is divided into three parts: #### 1. Stop GitLab CI + # Manual installation sudo service gitlab_ci stop + + # Omnibus installation + sudo gitlab-ctl stop ci-unicorn ci-sidekiq #### 2. Create a backup @@ -41,6 +45,8 @@ If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE) installation uses **PostgreSQL** you'll need to convert the CI database by setting a `MYSQL_TO_POSTGRESQL` flag. +If you use Omnibus package you most likely use the **PostgreSQL** on GitLab CE (or EE) and CI. + You can check which database each install is using by viewing their database configuration files: @@ -50,21 +56,26 @@ database configuration files: - If both applications use the same database `adapter`, create the backup with this command: - ```bash - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production - ``` + # Manual installation + cd /home/gitlab_ci/gitlab-ci + sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production + + # Omnibus installation + sudo gitlab-ci-rake backup:create - If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this command (note the `MYSQL_TO_POSTGRESQL` flag): - ```bash - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 - ``` + # Manual installation + cd /home/gitlab_ci/gitlab-ci + sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 + + # Omnibus installation + sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1 -#### 3. Remove cronjob +#### 3. Remove cronjob (manual installation) + # Manual installation cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec whenever --clear-crontab @@ -75,6 +86,8 @@ database configuration files: Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md). +If you use Omnibus package simply do `apt-get upgrade` to install a new version. + #### 2. Prevent CI usage for time of migration As Admin go to Admin Area -> Settings -> and uncheck @@ -86,20 +99,32 @@ This will prevent from creating the CI projects till you finish migration. Before you can migrate data you need to stop the GitLab service first: + # Manual installation sudo service gitlab stop + + # Omnibus installation + sudo gitlab-ctl stop unicorn sidekiq #### 4. Create a backup This migration poses a **significant risk** of breaking your GitLab installation. Create a backup before proceeding: + # Manual installation cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production + # Omnibus installation + sudo gitlab-rake gitlab:backup:create + It's possible to speedup backup creation. To do that you can skip repositories and uploads. + # Manual installation cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads + + # Omnibus installation + sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads #### 5. Copy secret tokens from CI @@ -108,11 +133,18 @@ The `secrets.yml` file stores encryption keys for secure variables. You need to copy the contents of GitLab CI's `config/secrets.yml` file to the same file in GitLab CE: + # Manual installation sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml sudo chown git:git /home/git/gitlab/config/secrets.yml sudo chown 0600 /home/git/gitlab/config/secrets.yml -#### 6. New configuration options for `gitlab.yml` +If you use Omnibus installation and your CI server is on the same server as GitLab CE (or EE) you don't need to do anything, +because the secrets are stored in **/etc/gitlab/gitlab-secrets.json**. + +If you migrate your Omnibus CI server to other server you need to copy **gitlab_ci** +section of **/etc/gitlab/gitlab-secrets.json** to the other server. + +#### 6. New configuration options for `gitlab.yml` (manual installation only) There are new configuration options available for `gitlab.yml`. View them with the command below and apply them manually to your current `gitlab.yml`: @@ -123,21 +155,34 @@ The new options include configuration settings for GitLab CI. #### 7. Copy backup from GitLab CI + # Manual installation sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar + + # Omnibus installation + sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/ + sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar If moving across the servers you can use **scp**. However, this requires you to provide authorized key or password to login to GitLab CE servers from CI server. You can try to use ssh-agent from your local machine to have that: login to your GitLab CI server using `ssh -A`. + # Manual installation scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup + + # Omnibus installation + scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/ #### 8. Import GitLab CI backup Now you'll import the GitLab CI database dump that you created earlier into the GitLab CE or EE database: + # Manual installation sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production + + # Omnibus installation + sudo gitlab-rake ci:migrate This task will take some time. @@ -145,10 +190,16 @@ This task will take some time. You can start GitLab CE (or EE) now and see if everything is working: + # Manual installation sudo service gitlab start + + # Omnibus installation + sudo gitlab-ctl restart unicorn sidekiq ### Part III: Finishing Up +This part is only for **Manual installations**. + #### 1. Update Nginx configuration To ensure that your existing CI runners are able to communicate with the -- cgit v1.2.1 From 7fa0c793d25c40a1165028a81758be66a6df0c62 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Mon, 21 Sep 2015 18:02:50 -0400 Subject: Whitespace --- doc/migrate_ci_to_ce/README.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index a8614e5fc49..b40a09a4d97 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -31,10 +31,10 @@ The migration is divided into three parts and covers manual and omnibus installa # Manual installation sudo service gitlab_ci stop - + # Omnibus installation sudo gitlab-ctl stop ci-unicorn ci-sidekiq - + #### 2. Create a backup The migration procedure modifies the structure of the CI database. If something @@ -59,20 +59,20 @@ database configuration files: # Manual installation cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production - + # Omnibus installation sudo gitlab-ci-rake backup:create - + - If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this command (note the `MYSQL_TO_POSTGRESQL` flag): # Manual installation cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 - + # Omnibus installation sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1 - + #### 3. Remove cronjob (manual installation) # Manual installation @@ -101,7 +101,7 @@ Before you can migrate data you need to stop the GitLab service first: # Manual installation sudo service gitlab stop - + # Omnibus installation sudo gitlab-ctl stop unicorn sidekiq @@ -113,16 +113,16 @@ installation. Create a backup before proceeding: # Manual installation cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production - + # Omnibus installation sudo gitlab-rake gitlab:backup:create - + It's possible to speedup backup creation. To do that you can skip repositories and uploads. # Manual installation cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads - + # Omnibus installation sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads @@ -158,7 +158,7 @@ The new options include configuration settings for GitLab CI. # Manual installation sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar - + # Omnibus installation sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/ sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar @@ -169,7 +169,7 @@ You can try to use ssh-agent from your local machine to have that: login to your # Manual installation scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup - + # Omnibus installation scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/ @@ -180,7 +180,7 @@ GitLab CE or EE database: # Manual installation sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production - + # Omnibus installation sudo gitlab-rake ci:migrate @@ -192,7 +192,7 @@ You can start GitLab CE (or EE) now and see if everything is working: # Manual installation sudo service gitlab start - + # Omnibus installation sudo gitlab-ctl restart unicorn sidekiq -- cgit v1.2.1 From 96fd255597319f41998f3a1edd56475c65af7342 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Mon, 21 Sep 2015 18:46:20 -0400 Subject: More migration guide updates - Rename "Finishing Up" to "Nginx configuration" - Add fourth "Finishing Up" section - Add Troubleshooting section with a link to backup restoration docs - Change a few indented blocks into fenced blocks so they're highlighted properly. - Added Marin's suggestions for CI-to-CE-on-different-server section. --- doc/migrate_ci_to_ce/README.md | 101 +++++++++++++++++++++++++++-------------- 1 file changed, 67 insertions(+), 34 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index b40a09a4d97..38f25c09f9e 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -17,13 +17,16 @@ This migration cannot be performed online and takes a significant amount of time. Make sure to plan ahead. If you are running a version of GitLab CI prior to 8.0 please follow the -appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/). +appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/) +before proceeding. -The migration is divided into three parts and covers manual and omnibus installations: +The migration is divided into four parts and covers both manual and Omnibus +installations: 1. [GitLab CI](#part-i-gitlab-ci) 1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee) -1. [Finishing Up](#part-iii-finishing-up) +1. [Nginx configuration](#part-iii-nginx-configuration) +1. [Finishing Up](#part-iv-finishing-up) ### Part I: GitLab CI @@ -45,7 +48,8 @@ If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE) installation uses **PostgreSQL** you'll need to convert the CI database by setting a `MYSQL_TO_POSTGRESQL` flag. -If you use Omnibus package you most likely use the **PostgreSQL** on GitLab CE (or EE) and CI. +If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab +CE (or EE) and CI. You can check which database each install is using by viewing their database configuration files: @@ -84,16 +88,20 @@ database configuration files: #### 1. Ensure GitLab is updated Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow -the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md). +the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md) +before proceeding. -If you use Omnibus package simply do `apt-get upgrade` to install a new version. +If you use the Omnibus packages simply run `apt-get upgrade` to install the +latest version. -#### 2. Prevent CI usage for time of migration +#### 2. Prevent CI usage during the migration process -As Admin go to Admin Area -> Settings -> and uncheck -**Disable to prevent CI usage until rake ci:migrate is run (8.0 only)**. +As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous +Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run +(8.0 only)**. -This will prevent from creating the CI projects till you finish migration. +This will disable the CI integration and prevent users from creating CI projects +until the migration process is completed. #### 3. Stop GitLab @@ -117,7 +125,7 @@ installation. Create a backup before proceeding: # Omnibus installation sudo gitlab-rake gitlab:backup:create -It's possible to speedup backup creation. To do that you can skip repositories and uploads. +It's possible to speed up backup creation by skipping repositories and uploads: # Manual installation cd /home/git/gitlab @@ -130,21 +138,32 @@ It's possible to speedup backup creation. To do that you can skip repositories a The `secrets.yml` file stores encryption keys for secure variables. -You need to copy the contents of GitLab CI's `config/secrets.yml` file to the -same file in GitLab CE: +- **Manual installations** need to copy the contents of GitLab CI's + `config/secrets.yml` file to the same file in GitLab CE: + ```bash # Manual installation sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml sudo chown git:git /home/git/gitlab/config/secrets.yml sudo chown 0600 /home/git/gitlab/config/secrets.yml + ``` -If you use Omnibus installation and your CI server is on the same server as GitLab CE (or EE) you don't need to do anything, -because the secrets are stored in **/etc/gitlab/gitlab-secrets.json**. +- **Omnibus installations** where GitLab CI and CE (or EE) are on the same + server don't need to do anything further, because the secrets are stored in + `/etc/gitlab/gitlab-secrets.json`. -If you migrate your Omnibus CI server to other server you need to copy **gitlab_ci** -section of **/etc/gitlab/gitlab-secrets.json** to the other server. +- **Omnibus installations** where GitLab CI is on a different server than CE (or + EE) will need to: + 1. On the CI server, copy the `db_key_base` value from + `/etc/gitlab/gitlab-secrets.json` + 1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] = + "VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo + gitlab-ctl reconfigure` -#### 6. New configuration options for `gitlab.yml` (manual installation only) +#### 6. New configuration options for `gitlab.yml` + +**Note:** This step is only required for manual installations. Omnibus +installations can [skip to the next step](#7-copy-backup-from-gitlab-ci). There are new configuration options available for `gitlab.yml`. View them with the command below and apply them manually to your current `gitlab.yml`: @@ -155,23 +174,29 @@ The new options include configuration settings for GitLab CI. #### 7. Copy backup from GitLab CI - # Manual installation - sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups - sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar +```bash +# Manual installation +sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups +sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar - # Omnibus installation - sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/ - sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar +# Omnibus installation +sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/ +sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar +``` -If moving across the servers you can use **scp**. -However, this requires you to provide authorized key or password to login to GitLab CE servers from CI server. -You can try to use ssh-agent from your local machine to have that: login to your GitLab CI server using `ssh -A`. +If moving across the servers you can use `scp`. +However, this requires you to provide an authorized key or password to login to +the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent +from your local machine to have that: login to your GitLab CI server using `ssh +-A`. - # Manual installation - scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup +```bash +# Manual installation +scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup - # Omnibus installation - scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/ +# Omnibus installation +scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/ +``` #### 8. Import GitLab CI backup @@ -196,9 +221,10 @@ You can start GitLab CE (or EE) now and see if everything is working: # Omnibus installation sudo gitlab-ctl restart unicorn sidekiq -### Part III: Finishing Up +### Part III: Nginx configuration -This part is only for **Manual installations**. +This part is only required for **Manual installations**. Omnibus users can [skip +to the final step](#part-iv-finishing-up). #### 1. Update Nginx configuration @@ -276,10 +302,17 @@ You should also make sure that you can: sudo /etc/init.d/nginx restart -#### 4. Done! +### Part IV: Finishing Up If everything went well you should be able to access your migrated CI install by visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI address, you should be redirected to the new one. **Enjoy!** + +### Troubleshooting + +#### Restore from backup + +If something went wrong and you need to restore a backup, consult the [Backup +restoration](../raketasks/backup_restore.md) guide. -- cgit v1.2.1 From 541d08b8f646081a9024f0af0ae6b3a15ce20136 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Mon, 21 Sep 2015 21:10:16 -0400 Subject: Be consistent about "only required for manual install" notes --- doc/migrate_ci_to_ce/README.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 38f25c09f9e..3ec541fb441 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -77,7 +77,10 @@ database configuration files: # Omnibus installation sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1 -#### 3. Remove cronjob (manual installation) +#### 3. Remove cronjob + +**Note:** This step is only required for **manual installations**. Omnibus users +can [skip to the next step](#part-ii-gitlab-ce-or-ee). # Manual installation cd /home/gitlab_ci/gitlab-ci @@ -162,8 +165,8 @@ The `secrets.yml` file stores encryption keys for secure variables. #### 6. New configuration options for `gitlab.yml` -**Note:** This step is only required for manual installations. Omnibus -installations can [skip to the next step](#7-copy-backup-from-gitlab-ci). +**Note:** This step is only required for **manual installations**. Omnibus users +can [skip to the next step](#7-copy-backup-from-gitlab-ci). There are new configuration options available for `gitlab.yml`. View them with the command below and apply them manually to your current `gitlab.yml`: @@ -223,8 +226,8 @@ You can start GitLab CE (or EE) now and see if everything is working: ### Part III: Nginx configuration -This part is only required for **Manual installations**. Omnibus users can [skip -to the final step](#part-iv-finishing-up). +This section is only required for **manual installations**. Omnibus users can +[skip to the final step](#part-iv-finishing-up). #### 1. Update Nginx configuration -- cgit v1.2.1 From da54661bbcb56856312a8917143166911434d3ff Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Mon, 21 Sep 2015 21:10:50 -0400 Subject: Remove unintended linebreak --- doc/migrate_ci_to_ce/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 3ec541fb441..735862b31c4 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -190,8 +190,8 @@ sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar If moving across the servers you can use `scp`. However, this requires you to provide an authorized key or password to login to the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent -from your local machine to have that: login to your GitLab CI server using `ssh --A`. +from your local machine to have that: login to your GitLab CI server using +`ssh -A`. ```bash # Manual installation -- cgit v1.2.1 From 524381dcf0ab981e070020adc40553bdbc1f3d69 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Mon, 21 Sep 2015 21:10:57 -0400 Subject: Add note about automatically re-enabling CI during migrate task [ci skip] --- doc/migrate_ci_to_ce/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 735862b31c4..c5eb4534c70 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -214,6 +214,9 @@ GitLab CE or EE database: This task will take some time. +This migration task automatically re-enables the CI setting that you +[disabled earlier](#2-prevent-ci-usage-during-the-migration-process). + #### 9. Start GitLab You can start GitLab CE (or EE) now and see if everything is working: -- cgit v1.2.1