From 0c83fee70f928ea378ec8aefd4d4ff3d81f82782 Mon Sep 17 00:00:00 2001 From: Cody West Date: Fri, 11 Jan 2019 19:28:23 -0600 Subject: Add troubleshooting section Add troubleshooting section to the slack project integration doc. --- doc/user/project/integrations/slack.md | 43 +++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index af4ca35a215..b5bc56c26f8 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -23,4 +23,45 @@ The Slack Notifications Service allows your GitLab project to send events (e.g. Your Slack team will now start receiving GitLab event notifications as configured. -![Slack configuration](img/slack_configuration.png) \ No newline at end of file +![Slack configuration](img/slack_configuration.png) + +## Troubleshooting + +If you're having trouble with the Slack integration not working, then start by +searching through the sidekiq logs for errors relating to your Slack service. + +### Something went wrong on our end + +This is a generic error shown in the GitLab UI and doesn't mean much by itself. +You'll need to look in the logs to find an error message and keep troubleshooting +from there. + +### certificate verify failed + +This is probably a problem either with GitLab communicating with Slack, or GitLab +communicating with itself. The former is less likely since Slack's security certificates +should _hopefully_ always be trusted. We can establish which we're dealing with by using +the below test script. + +```ruby +#!/opt/gitlab/embedded/bin/ruby +# the shebang should be changed if you're not using Omnibus GitLab +require 'openssl' +require 'net/http' + +puts "testing Slack" +# replace with your actual Slack URL +Net::HTTP.get(URI('https://')) + +puts "testing GitLab" +# replace with your actual GitLab URL +Net::HTTP.get(URI('https://')) +``` + +If it's an issue with GitLab not trusting connections to Slack, then the GitLab +OpenSSL trust store probably got messed up somehow. Typically this is from overriding +the trust store with `gitlab_rails['env'] = {"SSL_CERT_FILE" => "/path/to/file.pem"}` +or by accidentally modifying the default CA bundle `/opt/gitlab/embedded/ssl/certs/cacertpem`. + +If it's an issue with GitLab not trusting HTTPS connections to itself, then you may simply +need to [add your certificate to GitLab's trusted certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). \ No newline at end of file -- cgit v1.2.1 From 33c86fab87251cbfcd7d1f44507c68cde293f272 Mon Sep 17 00:00:00 2001 From: Cody West Date: Mon, 14 Jan 2019 16:11:11 -0600 Subject: Fix typos and wording 1. fix typo in cacert path 2. fix header to indicate it's an error message 3. add log entry example 4. add sidekiq log link 5. switch "scenario" paragraphs around --- doc/user/project/integrations/slack.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index b5bc56c26f8..6def7d26580 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -28,7 +28,8 @@ Your Slack team will now start receiving GitLab event notifications as configure ## Troubleshooting If you're having trouble with the Slack integration not working, then start by -searching through the sidekiq logs for errors relating to your Slack service. +searching through the [Sidekiq logs](https://docs.gitlab.com/ee/administration/logs.html#sidekiqlog) +for errors relating to your Slack service. ### Something went wrong on our end @@ -36,7 +37,13 @@ This is a generic error shown in the GitLab UI and doesn't mean much by itself. You'll need to look in the logs to find an error message and keep troubleshooting from there. -### certificate verify failed +### `certificate verify failed` + +You may see an entry similar to the following in your Sidekiq log: + +```text +2019-01-10_13:22:08.42572 2019-01-10T13:22:08.425Z 6877 TID-abcdefg ProjectServiceWorker JID-3bade5fb3dd47a85db6d78c5 ERROR: {:class=>"ProjectServiceWorker", :service_class=>"SlackService", :message=>"SSL_connect returned=1 errno=0 state=error: certificate verify failed"} +``` This is probably a problem either with GitLab communicating with Slack, or GitLab communicating with itself. The former is less likely since Slack's security certificates @@ -58,10 +65,10 @@ puts "testing GitLab" Net::HTTP.get(URI('https://')) ``` +If it's an issue with GitLab not trusting HTTPS connections to itself, then you may simply +need to [add your certificate to GitLab's trusted certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + If it's an issue with GitLab not trusting connections to Slack, then the GitLab OpenSSL trust store probably got messed up somehow. Typically this is from overriding the trust store with `gitlab_rails['env'] = {"SSL_CERT_FILE" => "/path/to/file.pem"}` -or by accidentally modifying the default CA bundle `/opt/gitlab/embedded/ssl/certs/cacertpem`. - -If it's an issue with GitLab not trusting HTTPS connections to itself, then you may simply -need to [add your certificate to GitLab's trusted certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). \ No newline at end of file +or by accidentally modifying the default CA bundle `/opt/gitlab/embedded/ssl/certs/cacert.pem`. \ No newline at end of file -- cgit v1.2.1 From ec47b2e36901ab9b917bbdf823830b559e4f6d83 Mon Sep 17 00:00:00 2001 From: Cody West Date: Tue, 22 Jan 2019 11:51:25 -0600 Subject: Fix sidekiq log link Use relative link rather than URL --- doc/user/project/integrations/slack.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index 6def7d26580..4fe9472e321 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -28,7 +28,7 @@ Your Slack team will now start receiving GitLab event notifications as configure ## Troubleshooting If you're having trouble with the Slack integration not working, then start by -searching through the [Sidekiq logs](https://docs.gitlab.com/ee/administration/logs.html#sidekiqlog) +searching through the [Sidekiq logs](/doc/administration/logs.md#sidekiqlog) for errors relating to your Slack service. ### Something went wrong on our end -- cgit v1.2.1 From e0d96f9c48253c7393687cc838217cf733961391 Mon Sep 17 00:00:00 2001 From: Cody West Date: Tue, 22 Jan 2019 11:52:43 -0600 Subject: Add link to production log --- doc/user/project/integrations/slack.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index 4fe9472e321..fcacb40c909 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -34,8 +34,8 @@ for errors relating to your Slack service. ### Something went wrong on our end This is a generic error shown in the GitLab UI and doesn't mean much by itself. -You'll need to look in the logs to find an error message and keep troubleshooting -from there. +You'll need to look in [the logs](/doc/administration/logs.md#productionlog) to find +an error message and keep troubleshooting from there. ### `certificate verify failed` -- cgit v1.2.1 From b5ba6c38a66355321e76b1e48f750b1c81b8d484 Mon Sep 17 00:00:00 2001 From: Cody West Date: Tue, 22 Jan 2019 12:58:46 -0600 Subject: Add script explanation Offer more explanation regarding the ssl-test script and how it's used. --- doc/user/project/integrations/slack.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index fcacb40c909..c23d008c1bf 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -65,6 +65,13 @@ puts "testing GitLab" Net::HTTP.get(URI('https://')) ``` +The important thing with this script is that it should use the same Ruby as GitLab. +There are many ways to run this script, and it will change depending on your install type. +On Omnibus GitLab installs, you can run save this script to `/root/ssl-test.rb`, +make the file exectuable with `chmod u+x /root/ssl-test.rb`, then run `/root/ssl-test.rb` +to execute the script. It should return only the "testing X" lines if it succeeds. +In the event of a failure, the exception should be displayed as well. + If it's an issue with GitLab not trusting HTTPS connections to itself, then you may simply need to [add your certificate to GitLab's trusted certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -- cgit v1.2.1 From b9d248911e6ff7207c9bc8f33a28232896bd71d0 Mon Sep 17 00:00:00 2001 From: Cody West Date: Tue, 22 Jan 2019 13:39:51 -0600 Subject: Switch to Rails console script For the stake of standardizing, I've switched the ssl debugging to use the Rails console rather than a Ruby script file. --- doc/user/project/integrations/slack.md | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index c23d008c1bf..f0f5bcfa633 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -48,30 +48,25 @@ You may see an entry similar to the following in your Sidekiq log: This is probably a problem either with GitLab communicating with Slack, or GitLab communicating with itself. The former is less likely since Slack's security certificates should _hopefully_ always be trusted. We can establish which we're dealing with by using -the below test script. +the below rails console script. -```ruby -#!/opt/gitlab/embedded/bin/ruby -# the shebang should be changed if you're not using Omnibus GitLab -require 'openssl' -require 'net/http' +```sh +# start a rails console: +sudo gitlab-rails console production + +# or for source installs: +bundle exec rails console production +``` -puts "testing Slack" +```ruby +# run this in the Rails console # replace with your actual Slack URL -Net::HTTP.get(URI('https://')) +result = Net::HTTP.get(URI('https://'));0 -puts "testing GitLab" # replace with your actual GitLab URL -Net::HTTP.get(URI('https://')) +result = Net::HTTP.get(URI('https://'));0 ``` -The important thing with this script is that it should use the same Ruby as GitLab. -There are many ways to run this script, and it will change depending on your install type. -On Omnibus GitLab installs, you can run save this script to `/root/ssl-test.rb`, -make the file exectuable with `chmod u+x /root/ssl-test.rb`, then run `/root/ssl-test.rb` -to execute the script. It should return only the "testing X" lines if it succeeds. -In the event of a failure, the exception should be displayed as well. - If it's an issue with GitLab not trusting HTTPS connections to itself, then you may simply need to [add your certificate to GitLab's trusted certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -- cgit v1.2.1