| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\
| |
| |
| |
| |
| |
| | |
Look up upstream commits once before queuing ProcessCommitWorkers
Closes #65464
See merge request gitlab-org/gitlab-ce!31871
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Instead of checking if a commit already exists in the upstream project
in its ProcessCommitWorker and bailing out if it does, we check the
existence of all commits in bulk in Git::BranchHooksService, so that we
can skip scheduling ProcessCommitWorker jobs for those commits
that already exist upstream entirely.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously `ProjectCacheWorker` would be scheduled once per ref, which
would generate unnecessary I/O and load on Sidekiq, especially if many
tags or branches were pushed at once. `ProjectCacheWorker` would expire
three items:
1. Repository size: This only needs to be updated once per push.
2. Commit count: This only needs to be updated if the default branch
is updated.
3. Project method caches: This only needs to be updated if the default
branch changes, but only if certain files change (e.g. README,
CHANGELOG, etc.).
Because the third item requires looking at the actual changes in the
commit deltas, we schedule one `ProjectCacheWorker` to handle the first
two cases, and schedule a separate `ProjectCacheWorker` for the third
case if it is needed. As a result, this brings down the number of
`ProjectCacheWorker` jobs from N to 2.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | | |
Reduce Gitaly calls in PostReceive
Closes #65878
See merge request gitlab-org/gitlab-ce!31741
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This commit reduces I/O load and memory utilization during PostReceive
for the common case when no project hooks or services are set up.
We saw a Gitaly N+1 issue in `CommitDelta` when many tags or branches
are pushed. We can reduce this overhead in the common case because we
observe that most new projects do not have any Web hooks or services,
especially when they are first created. Previously, `BaseHooksService`
unconditionally iterated through the last 20 commits of each ref to
build the `push_data` structure. The `push_data` structured was used in
numerous places:
1. Building the push payload in `EventCreateService`
2. Creating a CI pipeline
3. Executing project Web or system hooks
4. Executing project services
5. As the return value of `BaseHooksService#execute`
6. `BranchHooksService#invalidated_file_types`
We only need to generate the full `push_data` for items 3, 4, and 6.
Item 1: `EventCreateService` only needs the last commit and doesn't
actually need the commit deltas.
Item 2: In addition, `Ci::CreatePipelineService` only needed a subset of
the parameters.
Item 5: The return value of `BaseHooksService#execute` also wasn't being
used anywhere.
Item 6: This is only used when pushing to the default branch, so if
many tags are pushed we can save significant I/O here.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65878
Fic
|
| | | |
| | |
| | |
| | | |
Source Code Usage Ping for Create SMAU
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
**Prevention of running 2 simultaneous updates**
Instead of using `RemoteMirror#update_status` and raise an error if
it's already running to prevent the same mirror being updated at the
same time we now use `Gitlab::ExclusiveLease` for that.
When we fail to obtain a lease in 3 tries, 30 seconds apart, we bail
and reschedule. We'll reschedule faster for the protected branches.
If the mirror already ran since it was scheduled, the job will be
skipped.
**Error handling: Remote side**
When an update fails because of a `Gitlab::Git::CommandError`, we
won't track this error in sentry, this could be on the remote side:
for example when branches have diverged.
In this case, we'll try 3 times scheduled 1 or 5 minutes apart.
In between, the mirror is marked as "to_retry", the error would be
visible to the user when they visit the settings page.
After 3 tries we'll mark the mirror as failed and notify the user.
We won't track this error in sentry, as it's not likely we can help
it.
The next event that would trigger a new refresh.
**Error handling: our side**
If an unexpected error occurs, we mark the mirror as failed, but we'd
still retry the job based on the regular sidekiq retries with
backoff. Same as we used to
The error would be reported in sentry, since its likely we need to do
something about it.
|
| | |/
|/|
| |
| |
| |
| |
| |
| |
| |
| | |
Previously each tag in a push would invoke the Gitaly `FindAllTags` RPC
since the tag cache would be invalidated with every tag.
We can eliminate those extraneous calls by expiring the tag cache once
in `PostReceive` and taking advantage of the cached tags.
Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/65795
|
| |\ \
| | |
| | |
| | |
| | | |
Only expire branch cache once per push
See merge request gitlab-org/gitlab-ce!31653
|
| | | | |
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Whenever `PostReceive` is enqueued, `UpdateMergeRequestsWorker`
is enqueued and `MergeRequests::RefreshService` is called, it'll
check if the source branch of each MR asssociated to the push exists
or not via `MergeRequest#source_branch_exists?`. The said method will
call `Repository#branch_exists?` which is cached in `Rails.cache`.
When the cache contains outdated data and the source branch actually
exists, the `MergeRequests#RefreshService` job will close associated
MRs which is not correct.
The fix is to expire the branches cache of the project so we have
updated data during the post receive hook which will help in the
accuracy of the check if we need to close associated MRs or not.
|
| |/
|
|
|
| |
We migrated all logic to `PipelineProcessWorker`
and this worker become redundant.
|
| |\
| |
| |
| |
| | |
Extend PipelineProcessWorker to accept a list of builds
See merge request gitlab-org/gitlab-ce!31425
|
| | |
| |
| |
| |
| |
| |
| | |
This changes used worker from `BuildProcessWorker`
to `PipelineProcessWorker` to make pipeline
processing much simpler. We process `pipeline_id`,
based on some triggers.
|
| |/
|
|
|
|
|
|
| |
After measuring the impact of the namespace storage on
https://gitlab.com/gitlab-org/gitlab-ce/issues/64092. It was decided
that it's performant enough. So we can freely remove the feature flag
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/64092
|
| |
|
|
|
|
| |
This implements the support for `needs:` keyword
as part of GitLab CI. That makes some of the jobs
to be run out of order.
|
| |
|
|
|
|
| |
Also removes all logic from ClusterConfigureWorker
and ClusterProjectConfigureWorker, which are also
no longer used.
|
| |
|
|
| |
None of this code can be reached any more, so it can all be removed
|
| |
|
|
|
| |
Often live traces are removed even though the archived trace
doesn't exist. This commit checkes the existence strictly.
|
| |
|
|
|
|
|
| |
Schedules a Namespace::AggregationSchedule worker if some of the project
statistics are refreshed.
The worker is only executed if the feature flag is enabled.
|
| |
|
|
|
|
|
| |
Sets lease_release? to false to prevent the job to be re-executed more
often than lease timeout
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64079
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Add two new ActiveRecord models:
- RootNamespaceStoragestatistics will persist root namespace statistics
- NamespaceAggregationSchedule will save information when a new update
to the namespace statistics needs to be scheduled
- Inject into UpdateProjectStatistics concern a new callback that will
call an async job to insert a new row onto NamespaceAggregationSchedule
table
- When a new row is inserted a new job is scheduled. This job will
update call an specific service to update the statistics and after that
it will delete thee aggregated scheduled row
- The RefresherServices makes heavy use of arel to build composable
queries to update Namespace::RootStorageStatistics attributes.
- Add an extra worker to traverse pending rows on
NAmespace::AggregationSchedule table and schedule a worker for each one
of this rows.
- Add an extra worker to traverse pending rows on
NAmespace::AggregationSchedule table and schedule a worker for each one
of this rows
|
| |\
| |
| |
| |
| |
| |
| | |
Instance-level cluster pod terminal access
Closes #55489, #55488, #55487, and #61156
See merge request gitlab-org/gitlab-ce!28613
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This enables terminals for group and project level clusters.
Previously there was no way to determine which project (and
therefore kubernetes namespace) to connect to, moving this
logic onto Environment means the assoicated project can be
used to look up the correct namespace.
|
| |/
|
|
|
| |
Also add ::Gitlab::LetsEncrypt.enabled? shortcut
and simplify it a lot
|
| |
|
|
|
|
|
|
| |
Add index for pages domain ssl auto renewal
Add PagesDomain.needs_ssl_renewal scope
Add cron worker for ssl renewal
Add worker for ssl renewal
Add pages ssl renewal worker queues settings
|
| |\
| |
| |
| |
| |
| |
| | |
Remove deprecated call in BuildSuccessWorker
Closes #63622
See merge request gitlab-org/gitlab-ce!29958
|
| | |
| |
| |
| |
| | |
We now create_deployment on Ci::Build create.
No need to call create_deployment again
|
| |\ \
| |/
|/|
| |
| | |
Enable JIT Kubernetes resource creation for project level clusters
See merge request gitlab-org/gitlab-ce!29515
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously this behaviour was only available to group
and instance-level clusters, as some project clusters
relied on Kubernetes credentials being passed through
to the runner instead of having their resources managed
by GitLab (which is not available when using JIT). These
clusters have been migrated to unmanaged, so resources
can be created on demand for the remaining managed clusters.
|
| |/
|
|
|
|
|
| |
The bulk delete api endpoint for container registries can
only be called once per hour. If a user calls the endpoint more
than once per hour, they will now receive a 400 error with a
descriptive message.
|
| |
|
|
|
|
|
|
| |
We are deprecating this service in favor of instance wide clusters.
Therefore we removed some code that is not anymore needed for a
readonly cluster and also we added some flags to allow for this
deprecation. These flags are to be removed in the next release when
we finally completelly remove KubernetesService.
|
| |\
| |
| |
| |
| |
| |
| | |
Preventing blocked users and their PipelineSchdules from creating new Pipelines
Closes #47756
See merge request gitlab-org/gitlab-ce!27318
|
| | |
| |
| |
| | |
updated several specs and factories to accomodate new permissions
|
| | |
| |
| |
| |
| | |
Delete Todos for guest users when project visibility
level is updated to private.
|
| |/
|
|
|
|
|
| |
In the context of a Geo setup, some jobs can be
running on a Geo secondary where the database
is read-only and therefore we should guard
against various jobs attempting to write.
|
| |\
| |
| |
| |
| | |
Introduce sidekiq worker for auto merge process
See merge request gitlab-org/gitlab-ce!28780
|
| | |
| |
| |
| |
| | |
As we have a central domain for auto merge process today,
we should use a single worker for any auto merge process.
|
| |\ \
| |/
|/|
| |
| |
| |
| |
| |
| | |
'set-real-next-run-at-for-preventing-duplciate-pipeline-creations' into 'master'
Make pipeline schedule worker resilient
Closes gitlab-com/gl-infra/production#805 and #61955
See merge request gitlab-org/gitlab-ce!28407
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Currently, pipeline schedule worker is unstable because it's
sometimes killed by excessive memory consumption.
In order to improve the performance, we add the following fixes:
1. next_run_at is always real_next_run, which means the value
always takes into account of worker's cron schedule
1. Remove exlusive lock. This is already covered by real_next_run
change.
1. Use RunPipelineScheduleWorker for avoiding memory killer.
Memory consumption is spread to the multiple sidekiq worker.
|
| |/
|
|
|
|
|
|
|
|
|
|
| |
We have one auto merge strategy today - Merge When Pipeline
Succeeds.
In order to add more strategies for Merge Train feature,
we abstract the architecture to be more extensible.
Removed arguments
Fix spec
|
| |
|
|
|
|
| |
We've already migrated all the legacy artifacts to the new realm,
which is ci_job_artifacts table.
It's time to remove the old code base that is no longer used.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Immediate configuration is not ideal for group and instance
level clusters as projects that may never be deployed would
still have Kubernetes namespaces and service accounts created
for them.
As of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25586
we now create only the resources that are required for the
project being deployed, at the time of deployment.
|
| |
|
|
| |
Fixes #60425
|
| |
|
|
|
|
|
|
| |
When this option is enabled, GitLab will create namespaces and service
accounts as usual. When disabled, GitLab wont create any project
specific kubernetes resources
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56557
|
| |
|
|
|
|
|
|
| |
PackRefs is not an expensive gitaly call - we want to
call it more often (than as part of full `gc`) because
it helps to keep number of refs files small - too many
refs file may be a problem for deployments with
slow storage.
|
| |
|
|
|
| |
Domain will be removed by verification worker after 1 week
of being disabled
|
