summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* gitlab-sshd: Add support for signed user certificatessh-ssh-certificatesStan Hu2023-03-085-11/+117
| | | | | | | | | We add a `trusted_user_ca_keys` config setting that allows gitlab-sshd to trust any SSH certificate signed by the keys listed in this file. This is equivalent to the `TrustedUserCAKeys` OpenSSH setting. We assume the certificate identity is equivalent to the GitLab username.
* Merge branch ↵Igor Drozdov2023-03-071-1/+1
|\ | | | | | | | | | | | | | | | | | | | | | | | | '615-follow-up-from-add-dns-discovery-support-to-gitaly-via-client-side-load-balancing-options' into 'main' Make golang 1.19 the default Closes #615 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/718 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>
| * Make golang 1.19.6 the defaultAsh McKenzie2023-03-071-1/+1
| |
* | Merge branch 'sh-prepare-go-fips-1.19' into 'main'Patrick Bajao2023-03-071-3/+9
|\ \ | |/ |/| | | | | | | | | | | | | | | | | Prepare for Go 1.19 FIPS support See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/721 Merged-by: Patrick Bajao <ebajao@gitlab.com> Approved-by: Patrick Bajao <ebajao@gitlab.com> Approved-by: James Fargher <proglottis@gmail.com> Reviewed-by: Patrick Bajao <ebajao@gitlab.com> Co-authored-by: Stan Hu <stanhu@gmail.com>
| * Prepare for Go 1.19 FIPS supportStan Hu2023-03-061-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/718 will make Go 1.19 the default for gitlab-shell. Per https://github.com/golang/go/issues/51940, the dev.boringcrypto branch no longer exists, and to support FIPS we need to pass along `GOEXPERIMENT=boringcrypto`. To do this, we just see if this `GOEXPERIMENT` is available with `go version` rather than do some more complicated version-specific comparison.
* | Merge branch 'id-release-14-18-0' into 'main'v14.18.0Igor Drozdov2023-03-062-1/+8
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | Release v14.18.0 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/722 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com>
| * | Release v14.18.0Igor Drozdov2023-03-062-1/+8
|/ /
* | Merge branch 'id-geo-http-push' into 'main'Ash McKenzie2023-03-069-7/+431
|\ \ | |/ |/| | | | | | | | | | | | | | | | | | | | | | | Perform Git over HTTP request to primary repo See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/716 Merged-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Alejandro Rodríguez <alejandro@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Reviewed-by: Valery Sizov <valery@gitlab.com> Reviewed-by: Alejandro Rodríguez <alejandro@gitlab.com> Reviewed-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Ash McKenzie <amckenzie@gitlab.com> Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>
| * Perform HTTP request to primary on Geo pushIgor Drozdov2023-03-039-7/+431
|/ | | | | | | | | Currently, we perform a request to Gitlab Rails that proxies the request to primary However, it causes timeouts on big pushes and consumes large amount of memory. We can perform an HTTP request directly from Gitlab Shell instead and stream the response to the user
* Merge branch 'fix-static-build' into 'main'Igor Drozdov2023-02-283-0/+38
|\ | | | | | | | | | | | | | | | | | | | | sshd: exclude gssapi when building without cgo See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/720 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Vasilii Iakliushin <viakliushin@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Vasilii Iakliushin <viakliushin@gitlab.com> Co-authored-by: Lorenz Brun <lorenz@brun.one>
| * sshd: exclude gssapi when building without cgoLorenz Brun2023-02-233-0/+38
|/ | | | | | | | | | | MR #682 broke building without cgo enabled as it introduced a dependency on a Kerberos library. This can only be disabled at runtime and thus static builds of gitlab-sshd are no longer possible. This change introduces an alternative implementation of the GSSAPI structure which just rejects attempts to use it. That alternative implementation gets automatically activated in case the user is building without cgo.
* Merge branch 'qmnguyen0711/add-client-side-load-balancing-options' into 'main'Patrick Bajao2023-02-158-353/+404
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | Add DNS discovery support to Gitaly via client-side load-balancing options See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/717 Merged-by: Patrick Bajao <ebajao@gitlab.com> Approved-by: Oscar Tovar <otovar@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Patrick Bajao <ebajao@gitlab.com> Reviewed-by: Ash McKenzie <amckenzie@gitlab.com> Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com> Reviewed-by: Oscar Tovar <otovar@gitlab.com> Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
| * Add DNS discovery support for Gitaly/PraefectQuang-Minh Nguyen2023-02-145-163/+212
| | | | | | | | | | | | | | | | | | | | All the implementations of DNS discovery were done in this epic: https://gitlab.com/groups/gitlab-org/-/epics/8971. Gitaly allows clients to configure DNS discovery via dial option. This MR adds the exposed dial options to client connection creation in Gitlab-shell. Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4722 Changelog: added
| * Bump Gitaly client version to v15.9.0-rc4Quang-Minh Nguyen2023-02-143-190/+192
|/ | | | | | | This client bump includes plenty of improvement, espeically the support for DNS service discovery in Gitaly/Praefect. This version requires Go >= 1.18. As a result, we'll need to bump minimal Go version of GitLab Shell accordingly.
* Merge branch 'id-define-standard-client' into 'main'Ash McKenzie2023-02-083-37/+77
|\ | | | | | | | | | | | | | | | | | | | | | | Define Do function for Gitlab net client See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/715 Merged-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Alejandro Rodríguez <alejandro@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Reviewed-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Ash McKenzie <amckenzie@gitlab.com> Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>
| * Define Do function for Gitlab net clientIgor Drozdov2023-02-073-37/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In future, we'll need to perform http requests for Geo related code area. We cannot use retryable requests because: - It's not necessary for the to be retryable - In order to retry, the whole request body is stored in RAM, while we need to stream large blobs of data This commit: - Extracts logging into a separate round tripper in order to reuse it for other http requests by default - Defines Do function that accepts raw request as an argument
* | Merge branch 'sh-add-gitlab-sshd-makefile' into 'main'Ash McKenzie2023-02-081-2/+5
|\ \ | |/ |/| | | | | | | | | | | | | Add bin/gitlab-sshd as an explicit Makefile target See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/714 Merged-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Co-authored-by: Stan Hu <stanhu@gmail.com>
| * Simplify build ruleAsh McKenzie2023-02-081-1/+1
| |
| * Add bin/gitlab-sshd as an explicit Makefile targetStan Hu2023-02-071-2/+5
|/ | | | | | | | | | | | | | | | Since https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682, Kerberos headers and libraries are needed to build gitlab-sshd. If they are not available, `make build` successfully compiles `bin/gitlab-shell` but fails to build `bin/gitlab-sshd`. However, running `make build` again would do nothing and appear to be succeed because `bin/gitlab-shell` existed. This led to Omnibus GitLab quietly dropping the `gitlab-sshd` binary, as seen in https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6446#note_1265879416. To ensure `make build` properly fails if `bin/gitlab-sshd` cannot be built, we make the binary an explicit build target. Changelog: changed
* Merge branch 'ashmckenzie/release' into 'main'v14.17.0Igor Drozdov2023-02-032-1/+5
|\ | | | | | | | | | | | | | | | | Release v14.17.0 version See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/713 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>
| * Release v14.17.0 versionAsh McKenzie2023-02-032-1/+5
|/
* Merge branch 'main-patch-fa32' into 'main'Ash McKenzie2023-02-011-1/+1
|\ | | | | | | | | | | | | | | Bump golang to 1.18.9 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/712 Merged-by: Ash McKenzie <amckenzie@gitlab.com>
| * Bump golang to 1.18.9Ash McKenzie2023-02-011-1/+1
|/
* Merge branch 'id-release-14-16-0' into 'main'v14.16.0Igor Drozdov2023-01-312-1/+7
|\ | | | | | | | | | | | | | | | | Release v14.16.0 version See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/711 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com>
| * Release v14.16.0 versionIgor Drozdov2023-01-312-1/+7
|/
* Merge branch 'feat/remove-unretryable-http' into 'main'Ash McKenzie2023-01-3115-210/+90
|\ | | | | | | | | | | | | | | | | | | feat: make retryable http default client See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/710 Merged-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Oscar Tovar <otovar@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Co-authored-by: Steve Azzopardi <sazzopardi@gitlab.com>
| * feat: make retryable http default clientSteve Azzopardi2023-01-3015-210/+90
|/ | | | | | | | | | | | | | | | | | | | | | What --- Make the retryableHTTP client introduced in https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703 the default HTTP client. Why --- In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1254964426 we've seen a 99% error reduction on `git` commands from `gitlab-shell` when the retryableHTTP client is used. This has been running in production for over 2 weeks in `us-east1-b` and 5 days fleet-wide so we should be confident that this client works as expected. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* Merge branch 'id-improve-retryable-tests' into 'main'Ash McKenzie2023-01-303-18/+38
|\ | | | | | | | | | | | | | | | | | | | | Stub retryable http values in tests See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/708 Merged-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Oscar Tovar <otovar@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Reviewed-by: Oscar Tovar <otovar@gitlab.com> Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>
| * Stub retryable http values in testsIgor Drozdov2023-01-253-18/+38
| | | | | | | | | | | | | | | | Currently, the default values are used for retryable http. That's why a test waits 1 second minimun to retry a request. Client test takes 25 seconds to execute as a result. When we stub the value to 1 millisecond instead, we get 0.5s of execution
* | Merge branch 'id-cgo-cflags' into 'main'Stan Hu2023-01-271-0/+6
|\ \ | |/ |/| | | | | | | | | | | Specify CGO_CFLAGS in Makefile to compile gssapi lib See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/709 Merged-by: Stan Hu <stanhu@gmail.com> Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>
| * Specify CGO_CFLAGS in Makefile to compile gssapi libIgor Drozdov2023-01-271-0/+6
|/
* Merge branch '196-add-kerberos-support' into 'main'Igor Drozdov2023-01-2318-39/+359
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add support for the gssapi-with-mic auth method Closes #196 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Alejandro Rodríguez <alejandro@gitlab.com> Approved-by: Patrick Bajao <ebajao@gitlab.com> Approved-by: Costel Maxim <cmaxim@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Alejandro Rodríguez <alejandro@gitlab.com> Reviewed-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Patrick Bajao <ebajao@gitlab.com> Reviewed-by: Rohit Shambhuni <rshambhuni@gitlab.com> Co-authored-by: Lee Tickett <ltickett@gitlab.com> Co-authored-by: Marin Hannache <git@mareo.fr>
| * Add support for the gssapi-with-mic auth methodMarin Hannache2023-01-2318-39/+359
| |
* | Merge branch '596-aqualls-truncate-and-redirect' into 'main'Igor Drozdov2023-01-176-395/+12
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | docs: Truncate pages, point users to GitLab repo See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/705 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Torsten Linz <tlinz@gitlab.com> Approved-by: Jerry Seto <jseto@gitlab.com> Approved-by: Sean Carroll <scarroll@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Co-authored-by: Amy Qualls <aqualls@gitlab.com>
| * | docs: Truncate pages, point users to GitLab repoAmy Qualls2023-01-176-395/+12
| |/
* | Merge branch 'id-release-14-15-0' into 'main'v14.15.0Igor Drozdov2023-01-122-2/+8
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | Release 14.15.0 version See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/707 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com>
| * | Release 14.15.0 versionIgor Drozdov2023-01-122-2/+8
|/ / | | | | | | | | | | - Incorporate older edits to README !696 - Upgrade to Ruby 3.x !706 - feat: retry on http error !703
* | Merge branch 'feat/retry-on-error' into 'main'Ash McKenzie2023-01-1211-43/+217
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | feat: retry on http error Closes #604 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703 Merged-by: Ash McKenzie <amckenzie@gitlab.com> Approved-by: Alejandro Rodríguez <alejandro@gitlab.com> Approved-by: Ash McKenzie <amckenzie@gitlab.com> Reviewed-by: Steve Azzopardi <sazzopardi@gitlab.com> Reviewed-by: Ash McKenzie <amckenzie@gitlab.com> Co-authored-by: Steve Azzopardi <sazzopardi@gitlab.com>
| * | feat: put retryablehttp.Client behind feature flagSteve Azzopardi2023-01-1214-75/+169
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | What --- - Update the `client.HttpClient` fields to have `http.Client` and `retryablehttp.Client`, one of them will be `nil` depending on the feature flag toggle. - Create new method `newRetryableRequest` which will create a `retryablehttp.Request` and use that if the `FF_GITLAB_SHELL_RETRYABLE_HTTP` feature flag is turned on. - Add checks for `FF_GITLAB_SHELL_RETRYABLE_HTTP` everywhere we use the http client to use the `retryablehttp.Client` or the default `http.Client` - New job `tests-integration-retryableHttp` to run the integraiton tests with the new retryablehttp client. We didn't update go tests because some assertions are different and will break table driven tests. Why --- As discussed in https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703#note_1229645097 we want to put the client behind a feature flag, not just the retry logic. This does bring extra risk for accessing a `nil` field but there should be checks everytime we access `RetryableHTTP` and `HTTPClient`. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
| * | feat: retry on errorSteve Azzopardi2023-01-1215-54/+134
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | What --- Change the default `HTTP.Client` to `github.com/hashicorp/go-retryablehttp.Client` to get automatic retries and exponential backoff. We retry the request 2 times resulting in 3 attempts of sending the request, the min retry wait is 1 second, and the maximum is 15 seconds. Hide the retry logic behind a temporary feature flag `FF_GITLAB_SHELL_RETRYABLE_HTTP` to easily roll this out in GitLab.com. When we verify that this works as expected we will remove `FF_GITLAB_SHELL_RETRYABLE_HTTP` and have the retry logic as the default logic. Why --- In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 users end up seeing the following errors when trying to `git-clone(1)` a repository locally on in CI. ```shell remote: =============================== remote: remote: ERROR: Internal API unreachable remote: remote: ================================ ``` When we look at the application logs we see the following error: ```json { "err": "http://gitlab-webservice-git.gitlab.svc:8181/api/v4/internal/allowed": dial tcp 10.69.184.120:8181: connect: connection refused", "msg": "Internal API unreachable"} ``` In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1222670120 we've correlated these `connection refused` errors with infrastructure events that remove the git pods that are hosting `gitlab-webservice-git` service. We could try to make the underlying infrastructure more reactive to these changes as suggested in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1225164944 but we can still end up serving bad requests. Implementing retry logic for 5xx or other errors would allow users to still be able to `git-clone(1)` reposirories, although it being slower. This is espically important during CI runs so users don't have to retry jobs themselves. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Closes: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/604 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* | Merge branch '605-upgrade-to-ruby-3-x' into 'main'Stan Hu2023-01-115-8/+12
|\ \ | |/ |/| | | | | | | | | | | | | | | | | | | | | Resolve "Upgrade to Ruby 3.x" Closes #605 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/706 Merged-by: Stan Hu <stanhu@gmail.com> Approved-by: James Fargher <proglottis@gmail.com> Approved-by: Stan Hu <stanhu@gmail.com> Reviewed-by: Ash McKenzie <amckenzie@gitlab.com> Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>
| * Explicitly install webrick for Ruby 3.xAsh McKenzie2023-01-102-0/+3
| |
| * Make Go and Ruby versions more readable in CIAsh McKenzie2023-01-101-5/+5
| |
| * Test against Ruby 3.0 alsoAsh McKenzie2023-01-101-0/+1
| |
| * Remove script section from abstract .testAsh McKenzie2023-01-101-2/+2
| |
| * Use Ruby 3.0.5 as the defaultAsh McKenzie2023-01-102-2/+2
|/
* Merge branch '544-aqualls-recreate-sean-work' into 'main'Igor Drozdov2022-12-153-17/+103
|\ | | | | | | | | | | | | | | | | | | | | | | Incorporate older edits to README Closes #544 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/696 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Jerry Seto <jseto@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Co-authored-by: Amy Qualls <aqualls@gitlab.com>
| * Incorporate older edits to READMEAmy Qualls2022-12-153-17/+103
|/
* Merge branch 'sh-release-14.14.0' into 'main'v14.14.0Igor Drozdov2022-12-022-1/+15
|\ | | | | | | | | | | | | | | | | Release 14.14.0 version See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/702 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Co-authored-by: Stan Hu <stanhu@gmail.com>
| * Release 14.14.0 versionStan Hu2022-12-012-1/+15
|/ | | | | | | | | | | | | | - Add developer documentation to sshd package !683 - Improve error message for Gitaly `LimitError`s !691 - Drop 1.16 compatibility in go.sum !692 - Bump x/text to 0.3.8 !692 - Update prometheus package to 1.13.1 !692 Restrict IP access for PROXY protocol !693 - Fix broken Gitaly integration tests !694 - Clean up .gitlab-ci.yml file !695 - Use the images provided by Gitlab to run tests !698 - Use Ruby 2.7.7 as the default !699 - Use blocking reader to fix race in test !700
View Lorry Controller Status