summaryrefslogtreecommitdiff
path: root/internal/config
Commit message (Collapse)AuthorAgeFilesLines
* feat: make retryable http default clientSteve Azzopardi2023-01-302-44/+28
| | | | | | | | | | | | | | | | | | | | | | What --- Make the retryableHTTP client introduced in https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703 the default HTTP client. Why --- In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1254964426 we've seen a 99% error reduction on `git` commands from `gitlab-shell` when the retryableHTTP client is used. This has been running in production for over 2 weeks in `us-east1-b` and 5 days fleet-wide so we should be confident that this client works as expected. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* Merge branch '196-add-kerberos-support' into 'main'Igor Drozdov2023-01-231-0/+8
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add support for the gssapi-with-mic auth method Closes #196 See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682 Merged-by: Igor Drozdov <idrozdov@gitlab.com> Approved-by: Alejandro Rodríguez <alejandro@gitlab.com> Approved-by: Patrick Bajao <ebajao@gitlab.com> Approved-by: Costel Maxim <cmaxim@gitlab.com> Approved-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Alejandro Rodríguez <alejandro@gitlab.com> Reviewed-by: Igor Drozdov <idrozdov@gitlab.com> Reviewed-by: Patrick Bajao <ebajao@gitlab.com> Reviewed-by: Rohit Shambhuni <rshambhuni@gitlab.com> Co-authored-by: Lee Tickett <ltickett@gitlab.com> Co-authored-by: Marin Hannache <git@mareo.fr>
| * Add support for the gssapi-with-mic auth methodMarin Hannache2023-01-231-0/+8
| |
* | feat: put retryablehttp.Client behind feature flagSteve Azzopardi2023-01-122-26/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | What --- - Update the `client.HttpClient` fields to have `http.Client` and `retryablehttp.Client`, one of them will be `nil` depending on the feature flag toggle. - Create new method `newRetryableRequest` which will create a `retryablehttp.Request` and use that if the `FF_GITLAB_SHELL_RETRYABLE_HTTP` feature flag is turned on. - Add checks for `FF_GITLAB_SHELL_RETRYABLE_HTTP` everywhere we use the http client to use the `retryablehttp.Client` or the default `http.Client` - New job `tests-integration-retryableHttp` to run the integraiton tests with the new retryablehttp client. We didn't update go tests because some assertions are different and will break table driven tests. Why --- As discussed in https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703#note_1229645097 we want to put the client behind a feature flag, not just the retry logic. This does bring extra risk for accessing a `nil` field but there should be checks everytime we access `RetryableHTTP` and `HTTPClient`. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* | feat: retry on errorSteve Azzopardi2023-01-121-2/+2
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | What --- Change the default `HTTP.Client` to `github.com/hashicorp/go-retryablehttp.Client` to get automatic retries and exponential backoff. We retry the request 2 times resulting in 3 attempts of sending the request, the min retry wait is 1 second, and the maximum is 15 seconds. Hide the retry logic behind a temporary feature flag `FF_GITLAB_SHELL_RETRYABLE_HTTP` to easily roll this out in GitLab.com. When we verify that this works as expected we will remove `FF_GITLAB_SHELL_RETRYABLE_HTTP` and have the retry logic as the default logic. Why --- In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 users end up seeing the following errors when trying to `git-clone(1)` a repository locally on in CI. ```shell remote: =============================== remote: remote: ERROR: Internal API unreachable remote: remote: ================================ ``` When we look at the application logs we see the following error: ```json { "err": "http://gitlab-webservice-git.gitlab.svc:8181/api/v4/internal/allowed": dial tcp 10.69.184.120:8181: connect: connection refused", "msg": "Internal API unreachable"} ``` In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1222670120 we've correlated these `connection refused` errors with infrastructure events that remove the git pods that are hosting `gitlab-webservice-git` service. We could try to make the underlying infrastructure more reactive to these changes as suggested in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1225164944 but we can still end up serving bad requests. Implementing retry logic for 5xx or other errors would allow users to still be able to `git-clone(1)` reposirories, although it being slower. This is espically important during CI runs so users don't have to retry jobs themselves. Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 Closes: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/604 Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>
* sshd: Add ProxyAllowed setting to limit PROXY protocol IP addressesJames Fargher2022-11-231-0/+1
| | | | Changelog: added
* go: Bump major version to v14Patrick Steinhardt2022-07-052-5/+5
| | | | | | | | | | | | | While gitlab-shell currently has a major version of v14, the module path it exposes is not using that major version like it is required by the Go standard. This makes it impossible for dependents to import gitlab-shell as a dependency without using a commit as version. Fix this by changing the module path of gitlab-shell to instead be `gitlab.com/gitlab-org/gitlab-shell/v14` and adjust all imports accordingly. Changelog: fixed
* gitlab-sshd: Add support for configuring host certificatesStan Hu2022-06-261-0/+1
| | | | | | | | | | | | | | | | | This adds support for specifying host certificates via the `host_cert_files` option and advertises the signed key to the client. This acts similarly to OpenSSH's `HostCertificate` parameter: gitlab-sshd attempts to match a host key to its certificate, and then substitutes the matching host key with a certificate signed by a trusted certificate authority's key. This is the first requirement to supporting SSH certificates. This will enable the client to trust the server if both trust a common certificate authority. The `TrustedUserCAKeys` option will need to be supported later for the server to trust all user keys signed by this certificate authority. Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495
* Abort long-running unauthenticated SSH connectionsIgor Drozdov2022-05-232-10/+12
| | | | | | | | | The config option is basically a copy of LoginGraceTime OpenSSH option. If an SSH connection is hanging unauthenticated, after some period of time, the connection gets canceled. The value is configurable, the server waits for 60 seconds by default.
* Exclude Gitaly unavailable error from error rateIgor Drozdov2022-05-211-2/+1
| | | | | | | When a user hits repository rate limit, Gitaly returns an error that the request can't be handled (Gitaly unavailable) We should avoid this error to avoid exceeding the error rate
* Make ProxyHeaderTimeout configurableIgor Drozdov2022-05-192-6/+9
| | | | | | | | Issue: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/576 ProxyHeaderTimeout must be small to avoid DoS risk Let's make the value configurable and 500ms by default
* Allow specifying formatted durations in configIgor Drozdov2022-05-192-25/+68
| | | | | - If an integer is specified, we assume that these are seconds - A duration of format "500ms", "10s", "1m", etc... accepted
* Allow configuring SSH server algorithmsIgor Drozdov2022-05-191-0/+3
| | | | | MACs, Ciphers and KEX algorithms now can be configured If the values are empty, reasonable defaults are used
* Log canceled requests into separate metricsIgor Drozdov2022-05-171-1/+2
| | | | When a request get canceled we don't want to consider it an error
* Implement ClientKeepAlive optionIgor Drozdov2022-05-121-15/+21
| | | | | | | | | | Git clients sometimes open a connection and leave it idling, like when compressing objects. Settings like timeout client in HAProxy might cause these idle connections to be terminated. Let's send the keepalive message in order to prevent a client from closing
* Make PROXY policy configurableIgor Drozdov2022-05-101-0/+1
| | | | | It would give us more flexibility when we decide to enable PROXY protocol
* Merge branch '541_remove_self_signed_cert_option' into 'main'Igor Drozdov2022-04-261-2/+0
|\ | | | | | | | | Remove `self_signed_cert` option See merge request gitlab-org/gitlab-shell!602
| * Remove `self_signed_cert` optionVasilii Iakliushin2022-04-221-2/+0
| | | | | | | | | | | | Contributes to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/541 Changelog: removed
* | Revert "Abort long-running unauthenticated SSH connections"Igor Drozdov2022-04-251-6/+0
|/ | | | This reverts commit 3a2c8f2c47774a35d840ec8baf54341beede5d43.
* Add additional metrics to gitlab-sshdIgor Drozdov2022-04-131-2/+5
| | | | | | | | | - Observe time to establish a session - Log the duration of the successfully established connection - Observe total time to handle the connection - Log the duration of the successfully executed connection - Observe the count of ssh connections - Observe the count of failed ssh connections
* Abort long-running unauthenticated SSH connectionsIgor Drozdov2022-03-301-0/+6
|
* Reuse Gitaly conns and SidechannelIgor Drozdov2022-03-071-0/+3
| | | | | | | | When gitlab-sshd has been introduced we've started running our own SSH server. In this case we're able to cache and reuse Gitaly connections and Registry. It helps to reduce memory usage.
* Add more metrics for gitlab-sshdIgor Drozdov2022-02-212-13/+17
| | | | | | - Counter of HTTP merge requests - Gauge of HTTP requests currently being performed - Gauge of connections currently handled by gitlab sshd
* Remove SSL_CERT_DIR loggingid-remove-ssl-cert-dir-loggingIgor Drozdov2021-11-231-4/+0
| | | | | This log entry doesn't respect log level, because the log level is configured after this logging happens
* Log SSL_CERT_DIR when it's configuredIgor Drozdov2021-10-201-0/+4
|
* Respect log-level configuration againNick Thomas2021-09-231-0/+2
| | | | | | This was lost in the move from Ruby to Go. Restore it now. Changelog: fixed
* Fix a flaky testNick Thomas2021-09-141-1/+1
|
* refactor: move away from ioutil (deprecated)feistel2021-08-191-3/+2
|
* refactor: change httpclient to return an errorfeistel2021-08-132-6/+8
|
* refactor: update usage of NewHTTPClient to NewHTTPClientWithOptsfeistel2021-08-111-1/+6
|
* Fix formatting via make fmtStan Hu2021-07-262-3/+2
|
* Prometheus metrics for HTTP requestsIgor Drozdov2021-07-212-1/+35
| | | | | A RoundTripper for tracking the duration of an http request is introduced
* remove tracing.NewRoundTripper from internal/Config, now that NewHTTPClient ↵Igor Wiedler2021-07-201-10/+4
| | | | already includes it
* Provide liveness and readiness probesIgor Drozdov2021-07-191-1/+5
| | | | | They are going to be used to determine whether a server is alive and ready to accept traffic
* Shutdown sshd gracefullyIgor Drozdov2021-07-151-0/+7
| | | | | | | | When interruption signal is sent, we are closing ssh listener to prevent it from accepting new connections Then after configured grace period, we cancel the context to cancel all ongoing operations
* Merge branch 'change_log_format' into 'main'Patrick Bajao2021-07-131-1/+1
|\ | | | | | | | | Change default logging format to JSON See merge request gitlab-org/gitlab-shell!476
| * changed filename extension to .log as json can operate on .log filelistout2021-05-261-1/+1
| |
| * changed the format of log file to json from textlistout2021-05-241-2/+2
| |
* | Fix opentracing setup for gitlab-sshdNick Thomas2021-05-171-2/+9
|/ | | | | | | | | | | | | | | | | | | | Previously, opentracing (if configured) was initialized late in the gitlab-shell process's lifespan, coming just before making a gRPC call to Gitaly. By moving the opentracing initialization to be at process startup, we make it available for the whole process lifecycle, which is very useful to gitlab-sshd, as it means we'll only call tracing.Initialize() once on process startup, rather than once per SSH connection. To get this working, we need to introduce a context to gitlab-sshd. This carries the client/service name, but also carries an initial correlation ID. The main outcome of this is that all calls to the authorized_keys endpoint from a given gitlab-sshd process will now share a correlation ID. I don't have a strong opinion about this either way. Changelog: fixed
* gitlab-sshd: Respect the ssl_cert_dir config516-handle-ssl-cert-dir-correctlyNick Thomas2021-04-302-0/+34
| | | | Changelog: fixed
* gitlab-sshd: Support the PROXY protocolNick Thomas2021-04-121-0/+1
|
* Fix thread-safety issues in gitlab-shell518-fix-thread-safetyNick Thomas2021-04-091-22/+22
|
* Add monitoring endpoint to sshdBen Kochie2021-02-161-0/+2
| | | | | | | | | | | Add a basic monitoring endpoint to the sshd command. * Listen on localhost port 9122 by default. * Integrate build/version info. * Update example config. https://gitlab.com/gitlab-org/gitlab-shell/-/issues/121 Signed-off-by: Ben Kochie <superq@gmail.com>
* Refactor Config defaultsBen Kochie2021-02-161-50/+33
| | | | | | | | | | | Use "omitempty" to allow defaults in the config file to be correctly passed. Without this, explicitly setting an empty default like an empty string will not work. Needed in order to allow explicitly disabling some settings. Related to: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/121 Signed-off-by: Ben Kochie <superq@gmail.com>
* RFC: Simple built-in SSH serverLorenz Brun2021-01-182-164/+94
|
* tests: Replace assert with requirezj-remove-testify-assertZeger-Jan van de Weg2020-10-151-7/+6
| | | | | | | | | Testify features sub packages `assert` and `require`. The difference is subtle, and lost on novice Golang developers that don't read the docs. To create a more consistent code base `assert` will no longer be used. This change was generated by a running a sed command on all `_test.go` files, followed by `goimports -w`.
* Make it possible to propagate correlation ID across processesStan Hu2020-09-201-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, gitlab-shell did not pass a context through the application. Correlation IDs were generated down the call stack instead of passed around from the start execution. This has several potential downsides: 1. It's easier for programming mistakes to be made in future that lead to multiple correlation IDs being generated for a single request. 2. Correlation IDs cannot be passed in from upstream requests 3. Other advantages of context passing, such as distributed tracing is not possible. This commit changes the behavior: 1. Extract the correlation ID from the environment at the start of the application. 2. If no correlation ID exists, generate a random one. 3. Pass the correlation ID to the GitLabNet API requests. This change also enables other clients of GitLabNet (e.g. Gitaly) to pass along the correlation ID in the internal API requests (https://gitlab.com/gitlab-org/gitaly/-/issues/2725). Fixes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/474
* Fix gitlab-shell not handling relative URLs over UNIX socketssh-fix-unix-relative-url-accessStan Hu2020-08-201-10/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | From https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4498#note_397401883, if you specify a relative path such as: ``` external_url 'http://gitlab.example.com/gitlab' ``` gitlab-shell doesn't have a way to pass the `/gitlab` to the host. For example, let's say we have: ``` gitlab_url: "http+unix://%2Fvar%2Fopt%2Fgitlab%2Fgitlab-workhorse%2Fsocket" ``` If we have `/gitlab` as the relative path, how do we specify what is the UNIX socket path and what is the relative path? If we specify: ``` gitlab_url: "http+unix:///var/opt/gitlab/gitlab-workhorse.socket/gitlab ``` This is ambiguous. Is the socket in `/var/opt/gitlab/gitlab-workhorse.socket/gitlab` or in `/var/opt/gitlab/gitlab-workhorse.socket`? To fix this, this merge request adds an optional `gitlab_relative_url_root` config parameter: ``` gitlab_url: "http+unix://%2Fvar%2Fopt%2Fgitlab%2Fgitlab-workhorse%2Fsocket" gitlab_relative_url_root: /gitlab ``` This is only used with UNIX domain sockets to disambiguate the socket and base URL path. If `gitlab_url` uses `http://` or `https://`, then `gitlab_relative_url_root` is ignored. Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/476
* Support new ssl_cert_dir config settingAsh McKenzie2020-07-012-0/+10
|
* Move gitlabnet client to client packagejc-refactor-gitlabnet-clientJohn Cai2020-05-043-145/+19
|
View Lorry Controller Status