Look up upstream commits once before queuing ProcessCommitWorkers Closes #65464 See merge request gitlab-org/gitlab-ce!31871
Previously `ProjectCacheWorker` would be scheduled once per ref, which would generate unnecessary I/O and load on Sidekiq, especially if many tags or branches were pushed at once. `ProjectCacheWorker` would expire three items: 1. Repository size: This only needs to be updated once per push. 2. Commit count: This only needs to be updated if the default branch is updated. 3. Project method caches: This only needs to be updated if the default branch changes, but only if certain files change (e.g. README, CHANGELOG, etc.). Because the third item requires looking at the actual changes in the commit deltas, we schedule one `ProjectCacheWorker` to handle the first two cases, and schedule a separate `ProjectCacheWorker` for the third case if it is needed. As a result, this brings down the number of `ProjectCacheWorker` jobs from N to 2. Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046
Instead of checking if a commit already exists in the upstream project in its ProcessCommitWorker and bailing out if it does, we check the existence of all commits in bulk in Git::BranchHooksService, so that we can skip scheduling ProcessCommitWorker jobs for those commits that already exist upstream entirely.
Source Code Usage Ping for Create SMAU
**Prevention of running 2 simultaneous updates** Instead of using `RemoteMirror#update_status` and raise an error if it's already running to prevent the same mirror being updated at the same time we now use `Gitlab::ExclusiveLease` for that. When we fail to obtain a lease in 3 tries, 30 seconds apart, we bail and reschedule. We'll reschedule faster for the protected branches. If the mirror already ran since it was scheduled, the job will be skipped. **Error handling: Remote side** When an update fails because of a `Gitlab::Git::CommandError`, we won't track this error in sentry, this could be on the remote side: for example when branches have diverged. In this case, we'll try 3 times scheduled 1 or 5 minutes apart. In between, the mirror is marked as "to_retry", the error would be visible to the user when they visit the settings page. After 3 tries we'll mark the mirror as failed and notify the user. We won't track this error in sentry, as it's not likely we can help it. The next event that would trigger a new refresh. **Error handling: our side** If an unexpected error occurs, we mark the mirror as failed, but we'd still retry the job based on the regular sidekiq retries with backoff. Same as we used to The error would be reported in sentry, since its likely we need to do something about it.
Previously each tag in a push would invoke the Gitaly `FindAllTags` RPC since the tag cache would be invalidated with every tag. We can eliminate those extraneous calls by expiring the tag cache once in `PostReceive` and taking advantage of the cached tags. Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/65795
Only expire branch cache once per push See merge request gitlab-org/gitlab-ce!31653
We migrated all logic to `PipelineProcessWorker` and this worker become redundant.
Whenever `PostReceive` is enqueued, `UpdateMergeRequestsWorker` is enqueued and `MergeRequests::RefreshService` is called, it'll check if the source branch of each MR asssociated to the push exists or not via `MergeRequest#source_branch_exists?`. The said method will call `Repository#branch_exists?` which is cached in `Rails.cache`. When the cache contains outdated data and the source branch actually exists, the `MergeRequests#RefreshService` job will close associated MRs which is not correct. The fix is to expire the branches cache of the project so we have updated data during the post receive hook which will help in the accuracy of the check if we need to close associated MRs or not.