diff options
Diffstat (limited to 'doc/administration/job_traces.md')
| -rw-r--r-- | doc/administration/job_traces.md | 53 |
1 files changed, 28 insertions, 25 deletions
diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index 6e2f67f61bc..63945506f3c 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -12,8 +12,8 @@ In the following table you can see the phases a trace goes through. | ----- | ----- | --------- | --------- | ----------- | | 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| | 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| -| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`| -| 4: uploading | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage](#uploading-traces-to-object-storage) (if configured) |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`| +| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| +| 4: uploading | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage](#uploading-traces-to-object-storage) (if configured) |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| The `ROOT_PATH` varies per your environment. For Omnibus GitLab it would be `/var/opt/gitlab/gitlab-ci`, whereas for installations from source @@ -67,35 +67,37 @@ To archive those legacy job traces, please follow the instruction below. 1. Execute the following command - ```bash - gitlab-rake gitlab:traces:archive - ``` + ```bash + gitlab-rake gitlab:traces:archive + ``` - After you executed this task, GitLab instance queues up Sidekiq jobs (asynchronous processes) - for migrating job trace files from local storage to object storage. - It could take time to complete the all migration jobs. You can check the progress by the following command + After you executed this task, GitLab instance queues up Sidekiq jobs (asynchronous processes) + for migrating job trace files from local storage to object storage. + It could take time to complete the all migration jobs. You can check the progress by the following command - ```bash - sudo gitlab-rails console - ``` + ```bash + sudo gitlab-rails console + ``` - ```bash - [1] pry(main)> Sidekiq::Stats.new.queues['pipeline_background:archive_trace'] - => 100 - ``` + ```bash + [1] pry(main)> Sidekiq::Stats.new.queues['pipeline_background:archive_trace'] + => 100 + ``` - If the count becomes zero, the archiving processes are done + If the count becomes zero, the archiving processes are done ## How to migrate archived job traces to object storage +> [Introduced][ce-21193] in GitLab 11.3. + If job traces have already been archived into local storage, and you want to migrate those traces to object storage, please follow the instruction below. 1. Ensure [Object storage integration for Job Artifacts](job_artifacts.md#object-storage-settings) is enabled 1. Execute the following command - ```bash - gitlab-rake gitlab:traces:migrate - ``` + ```bash + gitlab-rake gitlab:traces:migrate + ``` ## How to remove job traces @@ -183,15 +185,15 @@ with the legacy architecture. In some cases, having data stored on Redis could incur data loss: 1. **Case 1: When all data in Redis are accidentally flushed** - - On going live traces could be recovered by re-sending traces (this is - supported by all versions of the GitLab Runner). - - Finished jobs which have not archived live traces will lose the last part - (~128KB) of trace data. + - On going live traces could be recovered by re-sending traces (this is + supported by all versions of the GitLab Runner). + - Finished jobs which have not archived live traces will lose the last part + (~128KB) of trace data. 1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that prevents archiving process, Sidekiq inconsistency, etc.)** - - Currently all trace data in Redis will be deleted after one week. If the - Sidekiq workers can't finish by the expiry date, the part of trace data will be lost. + - Currently all trace data in Redis will be deleted after one week. If the + Sidekiq workers can't finish by the expiry date, the part of trace data will be lost. Another issue that might arise is that it could consume all memory on the Redis instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed. @@ -201,4 +203,5 @@ indicate that we have trace chunk. `UPDATE`s with 128KB of data is issued once w receive multiple chunks. [ce-18169]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18169 +[ce-21193]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21193 [ce-46097]: https://gitlab.com/gitlab-org/gitlab-ce/issues/46097 |