summaryrefslogtreecommitdiff
path: root/.gitlab-ci/lava
Commit message (Collapse)AuthorAgeFilesLines
...
* ci/lava: Detect R8152 issues preemptively and retryGuilherme Gallo2022-07-083-0/+52
| | | | | | | | | | | | | | | | | | | | | Implement a log-based retry hint for R8152 issue described in #6681, which is based on detecting these two consecutive lines: ``` r8152 <USB> eth0: Tx status -71 nfs: server <IP> not responding, still trying ``` Where <IP> and <USB> could be any IP and USB addresses, respectfully. This commit is a temporary fix since it requires a section-aware log follower, implemented in !16323. When the cited MR is merged, one will make a proper fix on top of that. Closes: #6681 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
* ci/lava: Split lava_log into modulesGuilherme Gallo2022-07-086-157/+203
| | | | | | | This script is getting too big, it been hard to extend it. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
* ci/lava: Update license headerGuilherme Gallo2022-07-072-38/+6
| | | | | | | | Use SPDX to indicate the license. Update authors of lava_job_submitter.py Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Rename console color namesGuilherme Gallo2022-07-072-8/+7
| | | | | | | Use FG to indicate foreground colors Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Flexibilize section marker regexesGuilherme Gallo2022-07-071-18/+24
| | | | | | | | | | | | | | | | In some jobs, such as https://gitlab.freedesktop.org/gallo/mesa/-/jobs/24904100, the kmsg is interleaved with stderr/stdout in serial console, making it difficult to confidently find the log messages to detect when the DUT is booting, when the DUT is running etc. Luckily, LAVA sends redundant messages about their signals. We can use them to mitigate the chance of missing an interleaved message by being more open to different messages, using the regex on both `debug` and `target` LAVA log levels. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Stop printing after the result lineGuilherme Gallo2022-07-071-17/+25
| | | | | | | | | | | | | There are some leftovers in the jobs logs after the result log line. Only print until the init-stage2.sh output, to raise the chance to check for the test script results at the first glance in the Gitlab logs. Extra changes: - Add `hung` status for jobs considered hanging in the Gitlab - print them after the retry loop Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Highlight job retrying messageGuilherme Gallo2022-07-071-0/+3
| | | | | | | | It should be clear to the developer that the job was not successful in the first run. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Highlight kernel messages in boldGuilherme Gallo2022-07-071-1/+1
| | | | | | | | | This will serve to warn the user that those messages are processed differently, e.g. the kmsgs does not trigger heartbeats and maybe eventual targets of hint to retry the job immediately. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Don't print LAVA debug messagesGuilherme Gallo2022-07-071-22/+1
| | | | | | | | Remove debug messages from the output in order to unclutter the log a little more for the developers. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Wrap job definition dump into a collapsed sectionGuilherme Gallo2022-07-071-2/+7
| | | | | Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Wrap job info into a collapsed sectionGuilherme Gallo2022-07-071-3/+9
| | | | | Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: heartbeat: don't consider kernel message logsGuilherme Gallo2022-07-072-4/+36
| | | | | | | | | | | | | | Currently, the submitter consider that every new log that comes from the DUT console is a signal that the device is healthy, but maybe that is not the case, since in some kernel hangs/failures, no output is presented except from some kernel messages. This commit bypass the heartbeat when the LogFollower detect a kernel message. Any log line that does follow the kmsg pattern will make the job labeled as healthy again. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Follow job execution via LogFollowerGuilherme Gallo2022-07-072-48/+148
| | | | | | | | | | Now LogFollower is used to deal with the LAVA logs. Moreover, this commit adds timeouts per Gitlab section, if a section takes longer than expected, cancel the job and retry again. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Create LogFollower and move logging methodsGuilherme Gallo2022-07-074-107/+199
| | | | | | | | | | | | | - Create LogFollower to capture LAVA log and process it adding some - GitlabSection and color treatment to it - Break logs further, make new gitlab sections between testcases - Implement LogFollower as ContextManager to deal with incomplete LAVA jobs. - Use template method to simplify gitlab log sections management - Fix sections timestamps Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Create Gitlab log sections handlerGuilherme Gallo2022-07-072-1/+89
| | | | | | | | | | Gitlab has support for collapsible sections, so it would be good to create collapsed log sections for the LAVA setup logs. This way, the Mesa developers to see only the execution of the scripts, instead of LAVA messages clutter. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
* ci/lava: Improve result parsing regexGuilherme Gallo2022-06-281-1/+1
| | | | | | | | | | | | | | | | | LAVA job logs have an ongoing problem of message interleaving with kmsg. So any kernel dumps and LAVA signals (which are being printed in kmsg) will have a chance to clutter the pattern matching for `hwci: mesa: (pass|fail)` line. v2: - Add an 1 second sleep before exiting the test script, to give enough time to print the result message without conflicting with LAVA ENDTC signal from kmsg Closes: #6714 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17175>
* ci/lava: Filter out undesired messagesGuilherme Gallo2022-06-221-0/+21
| | | | | | | | | | | | | | | Some LAVA jobs emit lots of messages "Listened to connection for namespace 'common' for up to 1s" in a row at the end of the logs, making difficult to see the result of the test script. This commit removes those lines until a proper solution is deployed on the LAVA side. Closes: #6116 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Acked-by: Emma Anholt <emma@anholt.net> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17151>
* ci/lava: Add support for more complex color codesGuilherme Gallo2022-06-151-3/+5
| | | | | | | | | | | | | | | | | Currently, the LAVA job submitter is employing a temporary solution for the bash escape code mangling in the LAVA jobs. Until the issue is not fixed on the LAVA side, the submitter will replace the wrong characters with the fixed ones. This commit improves the regex pattern to comprehend the scenarios of color codes with font formatting and background color information, such as: `echo -e "\e[1;41;39mRed background with white bold text color\e[0m"` Fixes: #5503 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Reviewed-by: David Heidelberg <david.heidelberg@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17046>
* ci/lava: Fix Gitlab Section markersGuilherme Gallo2022-05-231-0/+19
| | | | | | | | | | | | | | | | LAVA is mangling the escape codes from ANSI during log fetching from the target device, making the gitlab section markers from deqp, for example, to not work, inputting noise into the log. This commit makes the simplest fix which is to replace the mangled characters to the fixed ones. This approach is error-prone, since it may unwittingly replace a genuine log that resembles the mangled escape code. But this solution should suffice until we get a proper fix from LAVA team itself. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16520>
* ci/lava: Fix colored LAVA outputsGuilherme Gallo2022-05-231-4/+19
| | | | | | | | | | | | | | | | | | LAVA is mangling the escape codes from ANSI during log fetching from the target device, making the colored lines from deqp, for example, to not work, inputting noise into the log. This commit makes the most straightforward fix which is to replace the mangled characters to the fixed ones. This approach is error-prone since it may unwittingly replace a genuine log that resembles the mangled escape code. But this solution should suffice until we get a proper fix from LAVA developers itself. Fixes: #5503 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16520>
* ci/lava: Make job submitter parse the job resultGuilherme Gallo2022-05-131-33/+55
| | | | | | | | | | | | | | | | | Currently, the LAVA job submitter fetches the job results from the LAVA XMLRPC call, but that is not necessary, as the job result is easily found in the logs. E.g. the bare-metal and poe jobs uses that log to set the final job status of their runs. Another reason for the change is that the LAVA signals are not reliable in some devices with one serial port, causing some troubles in a618 recently. So, if one signal fails to be sent/received, the job will ultimately fail even when the hwci script has been successful. Fixes: #6435 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16425>
* ci/lava: Retry when data fetching log RPC call is corruptedGuilherme Gallo2022-04-282-10/+35
| | | | | | | | | | | | Rarely the jobs.logs RPC call can return corrupted data, such as mal-formed YAML data. As this is expected and very rare to occur, let's retry this RPC call several times to give it a chance to fix itself. Retrying would not swallow the log lines since we keep track of how many log lines each job has. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Improve exception handlingGuilherme Gallo2022-04-283-53/+96
| | | | | | | | | | | | | | | | | Move exceptions to its own file. Create MesaCITimeoutError and MesaCIRetryError with specific exception data for better exception classification. Avoid the use of `fatal_err` in favor of raising a proper exception. Make _call_proxy exception handling exhaustive, add missing ResponseError treatment. Also, detect JobError during job result parsing. So when a LAVA timeout error happens, it is probably cause by some boot/network issues with a specific device, we can retry the same job in other device with the same device_type. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Cancel the job if the script is interruptedGuilherme Gallo2022-04-281-0/+4
| | | | | | | | | | During development, we may want to test lava_job_submitter.py locally, sometimes one can find what one is been looking for before the LAV job is done. Let's respond to SIGINT signal and cancel the LAVA job, as we can't follow what is happening anymore via script. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Reduce LAVA boot phase timeout to 3 minutesGuilherme Gallo2022-04-281-7/+14
| | | | | | | | | | | | | | A normal boot on LAVA device would take less than 1 minute. Sometimes the depthcharge freezes and LAVA waits until the current timeout (25 minutes) to kick in and fail the job. With this lower timeout value, the job could fail faster and eventually LAVA will be able to retry it. Furthermore, set a default timeout for all actions to fix an undesired behavior with some LAVA job subactions, such as depthcharge-action. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Let LAVA job submitter run without JWT fileGuilherme Gallo2022-04-281-7/+15
| | | | | | | | | | Make jwt-file argument optional, this means that this submitter could deal with more generic use cases, such as running it locally, since the MINIO JWT is a sensitive information, which is not really required to test if the submitter is working well. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Parse all test cases from 0_mesa suiteGuilherme Gallo2022-04-281-13/+27
| | | | | | | | | | LAVA can filter which test suite to show the results from, let's list all testcases possible in the mesa test suite, to be able to divide more complex jobs into test_cases. Another advantage is that the test case can vary its name. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Trap init-stage2.sh background processesGuilherme Gallo2022-04-281-1/+1
| | | | | | | | | | | | | | | | | | | | | | Any daemon executed in init-stage2.sh may interfere with LAVA signals, since any threaded output to console may clutter the signals, which are based on the log output. E.g: This job https://gitlab.freedesktop.org/gallo/mesa/-/jobs/20779120#L2102 has failed because capture-devcoredump.sh was alive and emitting kernel messages to the console during the LAVA signal handling, mangling the output and making the LAVA to fail to check the results of the job. Another problem is that CONFIG_DEBUG_STACK_USAGE Kconfig is enabled. This causes process exit to dump a `RESULT=[ 246.756067] lava-test-case (156) used greatest stack depth: ... bytes left` kernel message to the logs corrupting LAVA signal message. Empirically, it happens one in every 280 jobs. To solve that, compose the lava-test-case custom script with a short sleep to give time for kernel to dump the message clearly and a exit command to keep the return code from init-stage2.sh script. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Use lava-test-case to run custom scripts in LAVAGuilherme Gallo2022-04-282-11/+20
| | | | | | | | | | | | | | | | | | | The exit code is automatically parsed to fail/pass the job, so this commit removes the `hwci.*pass|fail` regex and printings. Add mesa-job-name parameter to get the CI_JOB_NAME easily to serve as test name. Besides, there is the treatment for the mesa job naeme, as LAVA does not like whitespace character inside the test case/suite name, since it interprets it as a LAVA signal parameter and it can make the job fail when the script looks for the results from the LAVA RPC. And the slash character seems to break gitlab log sectioning, so removing every character after the first whitespace. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Always validate the lava jobGuilherme Gallo2022-04-281-5/+5
| | | | | Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Set lava-signal to kmsgGuilherme Gallo2022-04-281-0/+1
| | | | | | | | | | | | | | By default, LAVA emit signals as specific lines of message to the console serial for subsequent parsing. However, this is prone to be interleaved with the dmesg messages, particularly with debug messages that can happen just after the process exit, such as the ones set by CONFIG_STACK_DEBUG_USAGE Kconfig. Setting the `lava-signal` to `kmsg` will make those special messages to be dumped to /dev/kmsg, where the each line is printed atomically Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Fix LAVA job validationGuilherme Gallo2022-04-281-2/+3
| | | | | | | | | When jobs.validate returns something, it means that there is a dict filled with the error message, so we were running the job with some validation errors for a quite while Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Filter log lines from LAVA returnGuilherme Gallo2022-04-281-1/+22
| | | | | | | | | | | Start to differentiate between the different types of LAVA log message; the only visible change right now is that we make warnings and errors bold and red to match bare-metal, but it comes in useful later as we will use the results markers to watch us step through the different stages of job execution. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Encapsulate job data in a classGuilherme Gallo2022-04-281-81/+114
| | | | | | | | | | | | | | | | | | | | | Less free-form passing stuff around, and also makes it easier to implement log-based following in future. The new class has: - job log polling: This allows us to get rid of some more function-local state; the job now contains where we are, and the timeout etc is localised within the thing polling it. - has-started detection into job class - heartbeat logic to update the job instance state with the start time when the submitter begins to track the logs from the LAVA device Besides: - Split LAVA jobs and Mesa CI policy - Update unit tests with LAVAJob class Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci/lava: Sleep before, not after, API callsDaniel Stone2022-04-281-7/+6
| | | | | | | | | | | | | | | | We rate-limit LAVA API calls as they are standard polling calls rather than blocking for changes. However when we sleep after making the calls rather than before, we can block when we want to exit - e.g. after getting the final logs, we will still sleep even though we can drop out. Fix this by moving the calls to before the API calls, rather than after. This means that the first calls (when we're waiting to be scheduled, or haven't got our first log lines yet), will be delayed compared to previously, but that's not going to slow us down as even in the best case we won't be executing in a device within the first 15 seconds. Signed-off-by: Daniel Stone <daniels@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
* ci: Allow specifying a different kernel in LAVA jobsTomeu Vizoso2022-04-133-9/+11
| | | | | | | | | | | | To make it possible to use a kernel different from that built along with the rootfs. This can make it more convenient for other projects to reuse these scripts. Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Reviewed-by: Emma Anholt <emma@anholt.net> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15891>
* ci: Use CI_PROJECT_NAME instead of hardcoding 'mesa'Tomeu Vizoso2022-04-131-2/+2
| | | | | | | | | This can make it more convenient for other projects to reuse these scripts. Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Reviewed-by: Emma Anholt <emma@anholt.net> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15891>
* ci: Disable Link Power Management with RTL8153Tomeu Vizoso2022-04-081-1/+1
| | | | | | | | | | | | There are reliability problems with the RTL8153 ethernet driver under certain network loads, related to incompatibility of the device with Link Power Management. Add usbcore.quirks=0bda:8153:k to the kernel command line to enable the USB_QUIRK_NO_LPM option. Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15791>
* ci: Provide intel-gpu-freq.sh in LAVA and bare-metal rootfsCristian Ciocaltea2022-04-061-0/+1
| | | | | | | | | | | The script will be used for tuning Intel GPU frequency to maximize performance tests execution, while also trying to reduce throttling, which has a negative impact on results consistency. Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com> Acked-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15662>
* ci/lava: Simplify passthrough of the request to upload results/ to minio.Emma Anholt2022-04-053-3/+2
| | | | | | | | | | We already have a way to pass env vars around, just use that instead of packing/unpacking it on the kernel command line. Cleans up HW runner job log output some more. Acked-by: Daniel Stone <daniels@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15332>
* ci: Make kernel image available in LAVA for KVM use casesCristian Ciocaltea2022-03-221-1/+4
| | | | | | | | | | | | | | In order to run a VM (e.g. crosvm) through HWCI_TEST_SCRIPT on a LAVA target, it's necessary to download a kernel image on the target device. When HWCI_KVM is set to 'true', we can safely assume HWCI_TEST_SCRIPT contains a command or the path to a script which expects the kernel image to be available under /lava-files/${KERNEL_IMAGE_NAME}. Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com> Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com> Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15208>
* ci: Allow disabling the whole of the Collabora farmTomeu Vizoso2022-02-241-0/+2
| | | | | | | | Add a global-level variable that allows disabling all jobs that would have gone to the Collabora lab, to be used in case of outages. Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15150>
* ci: Add unit tests for lava_job_submitterGuilherme Gallo2022-02-161-0/+34
| | | | | | | | | | | | | | These tests will explore some scenarios involving LAVA delays to submit the job to the device, some device delays outputting data to LAVA logs, and sensitive data protection. For example, the subtests from test_retriable_follow_job, "timed out more times than retry attempts" and "very long silence" caught a bug where a job retried until the limited attempts and the CI job still succeeded. https://gitlab.freedesktop.org/mesa/mesa/-/jobs/18325174 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14876>
* ci: Make LAVA jobs fail CI job when retry is exhaustedGuilherme Gallo2022-02-161-36/+43
| | | | | | | | | | | | | | | | | | | When the lava_job_submitter.py retry loop finishes normally (without falling through break-loop) it means that the submitter has exceeded the retry count limit. However, when it happens the script finishes normally. This patch adds a treatment to this case, warning the user what happened and forcing the job to fail. Moreover, this commit will make retry configurations configurable by CI job, as it can take the default value from the following variables: - LAVA_DEVICE_HANGING_TIMEOUT_SEC - LAVA_WAIT_FOR_DEVICE_POLLING_TIME_SEC - LAVA_LOG_POLLING_TIME_SEC - LAVA_NUMBER_OF_RETRIES_TIMEOUT_DETECTION Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14876>
* ci/traces: Drop PIGLIT_REPLAY_UPLOAD_TO_MINIO.Emma Anholt2022-01-271-3/+0
| | | | | | | | You have to do this as part of the traces workflow, otherwise there are no baseline images for your driver to compare to in the HTML summary. Reviewed-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14604>
* ci/traces: Rename the piglit/run.sh script to piglit-traces.sh.Emma Anholt2022-01-271-1/+1
| | | | | | | That's the only use of this script that's left. Reviewed-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14604>
* ci: Use ci-fairy minio login via token fileGuilherme Gallo2021-12-022-20/+42
| | | | | | | | | | | | | | | | | | | | | | | For every CI job, put JWT content into a file and unset CI_JOB_JWT environment var ======= * virgl jobs: - Share JWT token file to crosvm instance - Keep using `export -p` due to high complexity in the scripts of these jobs. At least, the CI_JOB_JWT will not be leaked, since it is being unset at the `before_script` phase of each Mesa CI job. * iris jobs: Update lava_job_submitter to take token file as argument - generate-env with CI_JOB_JWT_TOKEN_FILE - create token file during baremetal init stage * baremetal jobs: Copy token file to bare-metal NFS Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Reviewed-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14004>
* ci: Let manual LAVA jobs have a longer timeout than othersTomeu Vizoso2021-11-022-2/+4
| | | | | | | | | So far only LAVA jobs make use of it, but I guess baremetal could be extended to have these timeouts as well. Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Reviewed-by: Emma Anholt <emma@anholt.net> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13441>
* ci/deqp-runner: Simplify the --jobs argument setup.Emma Anholt2021-10-211-1/+1
| | | | | | | | | | | We can use the general "how parallel should we go on this runner?" env var and save a bunch of massaging env var names. Fixes how PIGLIT_PARALLEL looked like it was useful but actually wasn't passed through to HW runners. Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com> Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13372>
* gitlab-ci: refactor timeout constants and tweak timeout valuesGuilherme Gallo2021-09-151-5/+19
| | | | | | | | | | | | | | | | | | | | | | | * Refactor timeouts and retry attempts constants to variables in the top of the python script. * Increase LAVA job timeout value from 1 minute to 5 minutes, since the timeout detection is just a heuristic based on the log silence in LAVA devices. If we keep 1 minute timeout, maybe we could cancel jobs that have tasks which may take too long to respond. Also, one minute timeout is prone to misdetect scenarios when some network errors or slowness may happen. * Increase polling rate to check if the job has started from 1 check every 30 seconds to 1 check every 10 seconds. Since it was taking 30 seconds in the worst case to start to get the log output from a LAVA job. It is important to note that some LAVA jobs take less than 2 minutes to finish, so a 10 second wait would be more suitable in those cases. Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12870>
View Lorry Controller Status