summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'benbrown/lorry-template' into 'master'HEADmasterBen Brown2022-05-061-2/+2
|\ | | | | | | | | Fix lorry template See merge request CodethinkLabs/lorry/lorry-controller!33
| * Fix lorry templateBen Brown2022-05-051-2/+2
|/ | | | Update to match new raw-file format.
* Merge branch 'km/fix-block' into 'master'Ben Brown2022-01-121-3/+21
|\ | | | | | | | | | | | | Fix job queue getting stuck on repo preparation Closes #32 and #25 See merge request CodethinkLabs/lorry/lorry-controller!32
| * Fix job queue getting stuck on repo preparationKyle Mckay2022-01-121-3/+21
|/ | | | | | | | | | | | | | | | | | | | | - When a downstream type failed to prepare a repository the loop over lorries would abort unexpectly meaning any following lorries were never added into the minion's queue. Such cases are now simply skipped over so that lorry controller does not get stuck. - Catching Exception doesn't feel nice or correct, but we don't want any unexpected exceptions to cause the same problem of getting stuck so I don't see any obvious alternative in the short term. Perhaps the overall code can be reworked to avoid needing to do this. Log exceptions when skipping lorries When a repository fails to prepare, ensure there is a log of the exception that occured and the associated lorry. Update skipped lorry last run and exit data - Update last run to prevent frequent repeated failure - Update exit for user visiblity on status page
* Merge branch 'EllisB/enable-lfs' into 'master'Ben Brown2021-11-251-1/+1
|\ | | | | | | | | Enable LFS in gitlab for raw files See merge request CodethinkLabs/lorry/lorry-controller!31
| * Enable LFS in gitlab for raw filesEllis Barnwell2021-11-251-1/+1
|/
* Merge branch 'benbrown/raw-file-fix' into 'master'Pedro Alvarez Piedehierro2021-11-232-2/+15
|\ | | | | | | | | Update admin interface to support raw-file See merge request CodethinkLabs/lorry/lorry-controller!30
| * Update admin interface to support raw-fileBen Brown2021-11-182-2/+15
|/
* Merge branch 'benbrown/confgit-fetch' into 'master'Ben Brown2021-10-151-31/+12
|\ | | | | | | | | Tidy up confgit fetching See merge request CodethinkLabs/lorry/lorry-controller!29
| * Bump copyright yearBen Brown2021-10-131-1/+1
| |
| * Remove unclear commentBen Brown2021-10-131-3/+0
| |
| * Fetch without the need to touch remotesBen Brown2021-10-131-13/+8
| |
| * Drop duplicate hard resetBen Brown2021-10-131-5/+0
| |
| * readconf: unify clone and update into single fetch methodBen Brown2021-10-131-12/+6
|/
* Merge branch 'egb/issue-31-fix' into 'master'Ben Brown2021-10-131-0/+5
|\ | | | | | | | | Re #31: ensure that origin url is set to what's in settings See merge request CodethinkLabs/lorry/lorry-controller!28
| * Ensure that origin url is set to what's in settingsEllis Barnwell2021-10-131-0/+5
|/
* Merge branch 'egb/fix_issue_30' into 'master'Ben Brown2021-10-131-1/+2
|\ | | | | | | | | Re #30 reset to branch specified in settings See merge request CodethinkLabs/lorry/lorry-controller!27
| * Reset to the branch supplied in the configEllis Barnwell2021-10-131-1/+2
|/
* Merge branch 'bwh/add-failures-page' into 'master'Ben Brown2020-10-064-6/+72
|\ | | | | | | | | | | | | Add a page listing failing lorries Closes #20 See merge request CodethinkLabs/lorry/lorry-controller!25
| * Add a page listing failing lorriesBen Hutchings2020-10-064-6/+72
|/ | | | | | | | | | | | | Provide a status page that lists only the lorries that failed to update on the last attempt. This will make it easier to spot problems that require an update to the lorry configuration. Reuse some of the existing status page logic and render the run queue with a different template. Add a link to this page from the status page. Closes #20.
* Merge branch 'bwh/fix-remove-old-jobs' into 'master'Ben Brown2020-09-162-27/+31
|\ | | | | | | | | | | | | Reduce work and log output from lorry-controller-remove-old-jobs Closes #18 See merge request CodethinkLabs/lorry/lorry-controller!24
| * lorry-controller-remove-old-jobs: Filter out jobs started too recentlyBen Hutchings2020-09-151-1/+13
| | | | | | | | | | | | | | | | | | Job IDs are assigned sequentially, and jobs cannot finish before they start. Therefore we can iterate over jobs in order of ID and stop when we find a job that started more recently than max-age-seconds ago. Related to #18.
| * lorry-controller-remove-old-jobs: Add start time to JobInfo classBen Hutchings2020-09-151-4/+7
| | | | | | | | | | | | | | In order to filter jobs more efficiently, we need to look at the start time (job_started in the API) as well as the finish time. Related to #18.
| * lorry-controller-remove-old-jobs: Use a single loop in process_argsBen Hutchings2020-09-151-10/+5
| | | | | | | | | | | | | | In order to filter jobs earlier, we need to have a single loop over jobs in process_args(). Related to #18.
| * lorry-controller-remove-old-jobs: Inline methods called by process_args()Ben Hutchings2020-09-151-21/+15
| | | | | | | | | | | | | | | | In order to filter jobs earlier, we need to have a single loop over jobs in process_args() instead of in multiple functions that it calls. Prepare for that by inlining the methods that it calls directly. Related to #18.
| * units: Increase delay for lorry-controller-remove-old-jobsBen Hutchings2020-09-151-1/+1
|/ | | | | | | | | lorry-controller-remove-old-jobs is currently run at very short intervals, with a delay of 1 minute between each run. Its work isn't time-critical, and this results in a lot of unnecessary work and log spam. Increase the delay between runs to 1 hour. Related to #18.
* Merge branch 'bwh/include-gpl' into 'master'Ben Brown2020-08-131-3/+341
|\ | | | | | | | | LICENSE: Include full text of GPLv2 See merge request CodethinkLabs/lorry/lorry-controller!23
| * LICENSE: Include full text of GPLv2Ben Hutchings2020-08-131-3/+341
|/ | | | | Since Lorry Controller is now separate from Baserock we shouldn't assume that it's accompanied by any license texts.
* Merge branch 'bwh/pyflakes' into 'master'Ben Brown2020-08-1214-29/+56
|\ | | | | | | | | Fix pyflakes errors and warnings; add it to CI configuration See merge request CodethinkLabs/lorry/lorry-controller!22
| * gitlab-ci: Run pyflakes3 over all Python sourcesBen Hutchings2020-08-101-0/+17
| |
| * lorrycontroller: Explicitly list all names to be re-exportedBen Hutchings2020-08-101-1/+32
| | | | | | | | | | | | | | | | | | | | | | | | pyflakes isn't able to recognise that: __all__ = locals() effectively uses all the imported names. Replace this with an explicit list of names. Running pyflakes in CI should ensure that the list doesn't get out of sync.
| * Delete unused module importsBen Hutchings2020-08-107-15/+0
| | | | | | | | | | | | Various modules are imported and not used. Caught by pyflakes.
| * Delete assignments to local variables that are not used laterBen Hutchings2020-08-104-9/+3
| | | | | | | | | | | | | | pyflakes found various assignments to local variables which are not used again. In some cases we still need to evaluate the expression that's assigned. In most places we can delete the assignment entirely.
| * showjob: Use local variable instead of re-evaluating the same expressionBen Hutchings2020-08-101-1/+1
| | | | | | | | | | | | | | | | JobShower.get_job_as_json assigns to a local 'path' variable, but doesn't use it. Use it instead of re-evaluating the same expression assigned to it. Caught by pyflakes.
| * readconf: Fix handling of invalid time values in configurationBen Hutchings2020-08-101-1/+1
| | | | | | | | | | | | | | | | | | ReadConfiguration.fix_up_interval was returning default_value when the string value did not match the expected format. This variable is not defined, but default_interval is. This bug has been present since the beginning of LC development, and was caught by pyflakes.
| * givemejob: Fix upstream host metadata lookupBen Hutchings2020-08-101-2/+2
|/ | | | | | | | Commit bc7f80d "givemejob: Move upstream host metadata lookup out of get_repo_metadata" failed to pass through the statedb variable into the new function, so it will never work. Caught by pyflakes.
* Merge branch 'robjh/debug-yaml-error' into 'master'Ben Brown2020-08-031-0/+1
|\ | | | | | | | | Rewind lorry FDs before parsing as JSON. See merge request CodethinkLabs/lorry/lorry-controller!21
| * Rewind lorry FDs before parsing as JSON.Robert J. Heywood2020-08-031-0/+1
|/ | | | | | | | | First, the lorry files are parsed as YAML. If the parser fails, then the lorry files are parsed as JSON. If the YAML parser has advanced the Lorry file's file-descriptor, this can lead to JSON parser errors. This change resets the file descriptor to the start of the file before calling the JSON parser.
* Merge branch 'bwh/single-lorry-metadata' into 'master'Ben Brown2020-08-033-25/+124
|\ | | | | | | | | | | | | Add metadata for single repositories Closes #15 See merge request CodethinkLabs/lorry/lorry-controller!17
| * gitlab: Ignore failure to set the default branchBen Hutchings2020-07-301-3/+12
| | | | | | | | | | | | | | | | | | | | In testing the change to define the default branch for single repositories, I found that GitLab will reject a change of default branch if the branch does not yet exist. (It doesn't seem to do this when creating a repository.) Ignore failure to change the default branch, as we will fix it on the next run after the branch has been created.
| * givemejob: Add metadata for single repositoriesBen Hutchings2020-07-302-3/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When mirroring a repository found through a 'lorries' configuration section (instead of 'trove' or 'gitlab'), we currently don't set a description or default branch. * Set the description to the upstream repository path, but allow this to be overridden by a description field in the .lorry file. Prepend the host-name, just as we do when mirroring an Upstream Host. * Set the default branch to: - Bazaar: 'trunk' - Git: upstream default branch, found using 'git ls-remote' - others: 'master' Closes #15.
| * givemejob: Move upstream host metadata lookup out of get_repo_metadataBen Hutchings2020-07-151-10/+18
| | | | | | | | | | | | In preparation for adding metadata for single repositories, move the code for looking up metadata through the Upstream Host connector into a separate function.
| * givemejob: Stop synthesising host information when it's missingBen Hutchings2020-07-151-10/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Any lorry in the database should have been either: 1. Generated based on a 'trove' or 'gitlab' section, with 'from_host' set to the Upstream Host name. 2. Read from a lorry referred to in a 'lorries' section, with 'from_host' set to an empty string. So this error case indicates a dangling reference in the database. The current schema does not use foreign keys so this could happen, but it's reasonable to just give up and return empty metadata.
* | Merge branch 'bwh/local-empty-description' into 'master'Ben Brown2020-07-304-2/+8
|\ \ | | | | | | | | | | | | | | | | | | local: Create repos with empty description and enabled for export Closes #16 See merge request CodethinkLabs/lorry/lorry-controller!20
| * | local: Create repos with empty description and enabled for exportBen Hutchings2020-07-294-2/+8
|/ / | | | | | | | | | | | | | | | | | | | | | | 'git init' copies files into the repository from a template directory, which contains a 'replace me' description that we don't want. Provide and use our own template directory instead. While we're at it, also add the git-daemon-export-ok flag file that git-daemon and git-http-backend may check for (depending on their configuration). Closes #16.
* | Merge branch 'bwh/minion-multiple-lorry-configs' into 'master'Ben Hutchings2020-07-221-3/+3
|\ \ | | | | | | | | | | | | MINION: Allow specifying multiple configuration files for Lorry See merge request CodethinkLabs/lorry/lorry-controller!19
| * | MINION: Allow specifying multiple configuration files for LorryBen Hutchings2020-07-211-3/+3
|/ / | | | | | | | | Change the 'lorry-config' setting from string to string-list type. This will allow separating generic and site configuration files.
* | Merge branch 'bwh/sensible-disk-space' into 'master'Ben Brown2020-07-172-5/+11
|\ \ | |/ |/| | | | | | | | | Improve disk space formatting Closes #14 See merge request CodethinkLabs/lorry/lorry-controller!18
| * status: Show free disk space in MiB when it's < 1 GiBBen Hutchings2020-07-161-1/+7
| | | | | | | | | | | | Showing free disk space as 0 when it drops below 1 GiB may be unncessarily alarming, so show it in units of MiB if that happens. The status dictionary already includes it in both units.
| * status: Use integer division where appropriateBen Hutchings2020-07-161-4/+4
|/ | | | | | | | | | | | | | | In Python 2, the / operator between two integer-typed values produces another integer. In Python 3, it produces a float. We need to use the // operator to get the old behaviour. The change to floats in get_free_disk_space resulted in unnecessarily precise values for free disk space. The change in formats_secs_nicely doesn't seem to make a difference since we format the quotients with '%d', but I'm making it use // so it's clearer what we want. Closes #14.
View Lorry Controller Status