| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
| |
Signed-off-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22945>
|
|
|
|
|
|
|
| |
Acked-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22604>
|
|
|
|
|
|
|
|
|
| |
Based on work from Tomeu Vizoso.
Acked-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22604>
|
|
|
|
|
|
|
|
| |
f-strings are little more concise than str.format() approach.
It makes the code a little more readable as well.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
| |
Some refactoring was needed to make LAVAJobSubmitter class testable via
pytest.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
|
|
| |
LAVA farm is giving datetime in UTC timezone, let's standardize it
locally for the script run, so datetimes coming from LAVA proxy calls
will be at the same timezone as the ones we use in structural logging
and traces.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
|
| |
Let's use the AutoSaveDict as structural logger abstraction to enable
real-time monitoring of LAVA jobs. Mainly used for local runs and
debugging of Mesa CI LAVA jobs.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Refactor some pieces of the submitter to improve the clarity of the
functions and create a simple dictionary with aggregated data from the
submitter execution which will be dumped to a file when the script
exits.
Add support for the AutoSaveDict based structured logger as well, which
will come in a follow-up commit.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update the LogFollower class to improve section handling and provide a
history of sections encountered during log processing:
1. Add section_history attribute to store the history of encountered
GitlabSections.
2. Make LAVA job submitter use the section history feature to improve
structural logging.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
| |
Cleanup argparse to use dataclasses+python-fire to give easier
maintenance to job submitter.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Let's make lava_job_submitter.py cleaner with only parsing and retry
mechanism capabilities.
Moved out from the submitter script:
1. proxy functions
- moved to lava.utils.lava_proxy.py
2. LAVAJob class definition
- moved to lava.utils.lava_job.py
- added structural logging capabilities into LAVAJob
- Implemented properties for job_id, is_finished, and status, with
corresponding setter methods that update the log dictionary.
- Added new methods show, get_lava_time, and refresh_log for improved
log handling and data retrieval.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
| |
The LAVA job submitter is too big, let's reorganize it a little.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
|
|
|
|
|
|
|
|
|
|
|
| |
This commit introduces multiple changes:
1. Now we check for mainline artifacts only when NOT running on
the mainline branch
2. if we run on the fork and get 404-like error, it doesn't retry.
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22386>
|
|
|
|
|
| |
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22065>
|
|
|
|
|
| |
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22065>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
.lava-test hidden job was setting the HWCI_TEST_SCRIPT variable to deqp
runner. But that is not always the case. When we run piglit traces jobs,
we use piglit-traces.sh instead, for example.
Splitting into:
- .lava-test-deqp (deqp-runner + deqp)
- .lava-traces (deqp-runner + piglit)
- .lava-piglit (piglit-runner + piglit)
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22065>
|
|
|
|
|
|
|
|
| |
Very annoying when adding new job and not getting failure due to missing
`DEQP_VER: `
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21702>
|
|
|
|
|
|
|
|
|
|
| |
At some point in a refactoring long ago, our 'Piglit' runs on arm64
started actually being dEQP-GLES2 runs. Oh dear.
Surprisingly, there are a number of expectation changes; added every
fail I saw from a long overnight stress test.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21851>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit ensures that we are using mesa release builds in performance
jobs.
To achieve that, some modifications were made on top of
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21492.
- Append the `BUILDTYPE` variable into the S3 artifact name
(MINIO_ARTIFACT_NAME environment variable) to allow for better
artifact management.
- The ./artifacts directory has been added to the list of artifact
directories for build-common. This ensures that the debian-release and
debian-arm64-release jobs are the only ones necessary for running
performance jobs. These jobs only produce artifacts via
prepare-artifacts.sh when we are under performance workflow.
- Make lava-submit.sh behave similar to baremetal jobs regarding
MINIO_ARTIFACT_NAME variable. For example, users can now easily
differentiate between mesa-arm64.tar.zstd and
mesa-arm64-release.tar.zstd by looking inside the `Downloading
artifacts from s3` Gitlab section.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21804>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before: kernelci 38; Mesa3D 75
Priority now:
- 38 ‒ kernelci
- 40 ‒ after merge and performance
- 50 ‒ user runs
- 75 ‒ marge-bot (MUST be prioritized)
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21812>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit updates LogFollower class to handle carriage return
characters in LAVA logs. LAVA treats carriage return characters as a
line break, so each carriage return in an output console is mapped to a
console line in LAVA.
The updated LogFollower class now merges lines that end with a carriage
return character into a single line, making the Gitlab sections work
correctly. In addition, the `remove_trailing_whitespace` method has been
updated to remove trailing `\r\n` characters from log lines.
The `test_lava_log_merge_carriage_return_lines` test function has also
been updated to test for carriage returns at the end of the previous
line.
Closes: #8242
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21614>
|
|
|
|
|
|
|
|
| |
in after_script, variable $SCRIPTS_DIR is lost
Acked-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20272>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since the Collabora LAVA update related to the downtime from
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21119, the
LAVA logs from Collabora continued to use the hack for older versions
which digested some control characters, such as carriage returns acting
as newlines, which made it necessary to recover from split lines to make
Gitlab sections work in job logs as expected.
Collabora's LAVA instance now gives a more raw log output. It is
necessary to pay attention to newlines at the end of each log message,
which may cause double newlines when printed with Python built-in
`print` function. I decided to remove the repeating `\n` from the
received log messages to make them transparent to LogFollower users.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8242
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21325>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
LavaFarm is a class created to handle the different types of LAVA farms
and their tags in Mesa CI. Since specific jobs may require different
types of LAVA farms to run on, it is essential to determine which farm
the runner is running on to configure the job correctly.
LavaFarm provides an easy-to-use interface for checking the runner tag
and returning the corresponding LAVA farm, making it simple for Mesa CI
to configure jobs appropriately. By adding tests for LavaFarm, the team
can ensure that this class is functioning as expected, allowing for the
smooth execution of Mesa CI jobs on the correct LAVA farm.
The tests ensure that get_lava_farm returns the correct LavaFarm value
when given invalid or valid tags and that it returns LavaFarm.UNKNOWN
when no tag is provided. The tests use Hypothesis strategies to generate
various labels and farms for testing.
Example of use:
```
from lava.utils.lava_farm import LavaFarm, get_lava_farm
lava_farm = get_lava_farm()
if lava_farm == LavaFarm.DUMMY:
# Configure the job for the DUMMY farm
...
elif lava_farm == LavaFarm.COLLABORA:
# Configure the job for the COLLABORA farm
...
elif lava_farm == LavaFarm.KERNELCI:
# Configure the job for the KERNELCI farm
...
else:
# Handle the case where the LAVA farm is unknown
...
```
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21325>
|
|
|
|
|
|
|
|
|
|
|
| |
Use requirements.txt and requirements-test.txt to organize better Python
dependencies related to LAVA.
Now LAVA tooling can use recent and fixed library versions.
And test-related libs will not trigger container rebuilding anymore.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21325>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Better error handling is more reliable.
Options:
-L, follow location
--retry, number of retries
--retry-all-errors, does not fail on ALL errors, that's why there is -f
-f, fail fast with no output at all on server errors
--retry-delay, make curl sleep this amount of time before each retry
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20788>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The Mesa CI LAVA job submitter was suffering from a bug in the LAVA
software that made their timeouts related to sub-actions unreliable,
such as waiting for the user login prompt automatic response.
The following MR
https://git.lavasoftware.org/lava/lava/-/merge_requests/1900 fixed this
issue. So we can now better control job timeouts granularity, failing
the job faster when there is something weird hanging the boot stage.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20596>
|
|
|
|
|
|
|
|
| |
- Use new YAML loader derived from ruamel.yaml
- Remove PyYAML dependency from LAVA job submitter package
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20596>
|
|
|
|
|
|
|
|
| |
Currently, LAVA jobs only show metadata when successful, let's show this
info in all retries to make it easier to debug or report issues.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20051>
|
|
|
|
|
|
|
|
|
|
|
|
| |
To debug the LAVA jobs locally, we have an option in the
lava_job_submitter script to ignore the JWT token to make it possible to
retry jobs without the need to get an unexpired token.
But this trick needs to modify the overlayed directory so that we would
need to download and extract it earlier in the run.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20051>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
LAVA splits the DUT log lines with `\r` in them. Unfortunately, it
breaks the Gitlab section line syntax when the HWCI script calls it
since it is a oneliner.
This commit changes the` fix_lava_gitlab_section_log` function to a
stateful generator that can merge lines that respects two consecutive
patterns.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7703
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20051>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
LAVA uses XMLRPC to send jobs information and control, more specifically
it sends device logs via YAML dumps encoded in UTF-8 bytes.
In Python, we have xmlrpc.client.Binary class as the serializer
protocol, we get the logs wrapped by this class, which encodes the data
as UTF-8 bytes data.
We were converting the encoded data to a string via the `str` function,
but this led the loaded YAML data to use single quotes instead of double
quotes for string values that made special characters, such as `\x1b` to
be escaped as `\\x1b`.
With this fix, we can now drop one of the hacks that fixed the bash
colors.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20051>
|
|
|
|
|
|
|
|
|
| |
We don't need to login anymore, but we can't use plain minio commands
now. `ci-fairy` got a helper as `s3cp` to keep an almost identical
API.
Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19076>
|
|
|
|
|
|
| |
Acked-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18529>
|
|
|
|
|
|
|
|
|
| |
Let's be consistent.
Also needed for parsing device and traces yml file from logs.
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18422>
|
|
|
|
|
|
|
|
| |
We were uploading the junit.xml to the artifacts, but weren't telling
GitLab about it.
Reviewed-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18028>
|
|
|
|
|
|
|
|
| |
v2: added missing zstd to arm_build.sh
Reviewed-by: Emma Anholt <emma@anholt.net>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17776>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Visible size reduction and minor performance improvement at no cost.
rootfs measure:
```
2022-07-27 11:15:16.235268: 475MB downloaded in 111.59s (4.26MB/s)
2022-07-27 15:07:40.984857: 425MB downloaded in 85.57s (4.97MB/s)
```
So let say approx. 95s vs 85s if we assume 5MB/s, which can bring us
10s speedup...
Acked-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17776>
|
|
|
|
|
| |
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17703>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Refer to environment variables before falling back to the default
timeouts for each Gitlab section.
This makes more explicit in the job definition that there is a
particular case where the job may obey different timeouts.
Closes: #6908
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17703>
|
|
|
|
|
|
|
|
|
|
|
| |
When we don't want to communicate with minio, e.g. running
lava_job_submitter script locally, MINIO_RESULTS_UPLOAD should be unset.
But this variable is already set by generate-env script, so we need to
remove it from the /set-job-env-vars.sh to avoid declaring it in
unexpected scenarios.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17645>
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit fixes an issue related to leftover between jobs in the same
device under test in LAVA.
There is a possibility of having the resulting output being dumped just
after the boot, such as this job:
https://gitlab.freedesktop.org/mesa/mesa/-/jobs/25674303#L155
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17752>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Empirically, a successful LAVA boot time should take less than 3
minutes.
LAVA itself is configured to attempt thrice to boot the device,
summing up to 9 minutes.
It is better to retry the boot than cancel the job and re-submit to
avoid the enqueue delay.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17646>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sleep a bit before executing `lava-test-case` to give time for bash to dump shell xtrace messages into
the console, which may cause interleaving with `LAVA_SIGNAL_STARTTC` in some
devices like a618.
The same approach worked for `LAVA_SIGNAL_STARTRUN` since
https://gitlab.freedesktop.org/mesa/mesa/-/commit/3b8d10d270413a1f49ccce5563c1e4e96e10b6ef#deafdd86b8d9d0108bc692f479c3b31c4c7d5635_161_164
was merged.
Closes: #6867
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17533>
|
|
|
|
|
|
|
|
|
|
|
|
| |
If you accidentally re-included your test job core definition after your
driver-specific ruleset, you'd end up running the driver job on every
source code change. This had happened with a630_gles_asan: it included
.baremetal-test-arm64-asan (and thus .baremetal-test) after including
.a630-test, to override .baremetal-test-arm64's depednencies to use asan
artifacts instead.
Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>
|
|
|
|
|
|
|
|
|
|
|
| |
... and explain what they're doing, compared to the test rules in
test-source-dep.yml.
Unfortunately, we can't really pull them into test-source-dep.yml with
other source deps, because of various '&'-'*' references.
Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>
|
|
|
|
|
|
|
|
|
|
|
| |
We should be explicit that we are cancelling jobs once the script finds
some log messages that are linked with known issues. That means the
script preemptively retried the job without giving chances to recover.
Adds magenta color to cancelled jobs.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
|
|
|
|
|
|
|
|
|
| |
Mark test_full_yaml_log with this new marker to be easily run by the
developers.
Make `debian-testing` skip this test with `not slow` marker hint.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
|
|
|
|
|
|
|
|
| |
Fatal errors now have red foreground color and retry messages yellow
one.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
|
|
|
|
|
|
|
| |
It will help to know what happened to a non-successful job.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
|