summaryrefslogtreecommitdiff
path: root/setup.py
Commit message (Collapse)AuthorAgeFilesLines
* scheduler.py: Use threads instead of processes for jobsBenjamin Schubert2020-12-041-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes how the scheduler works and adapts all the code that needs adapting in order to be able to run in threads instead of in subprocesses, which helps with Windows support, and will allow some simplifications in the main pipeline. This addresses the following issues: * Fix #810: All CAS calls are now made in the master process, and thus share the same connection to the cas server * Fix #93: We don't start as many child processes anymore, so the risk of starving the machine are way less * Fix #911: We now use `forkserver` for starting processes. We also don't use subprocesses for jobs so we should be starting less subprocesses And the following highlevel changes where made: * cascache.py: Run the CasCacheUsageMonitor in a thread instead of a subprocess. * casdprocessmanager.py: Ensure start and stop of the process are thread safe. * job.py: Run the child in a thread instead of a process, adapt how we stop a thread, since we ca't use signals anymore. * _multiprocessing.py: Not needed anymore, we are not using `fork()`. * scheduler.py: Run the scheduler with a threadpool, to run the child jobs in. Also adapt how our signal handling is done, since we are not receiving signals from our children anymore, and can't kill them the same way. * sandbox: Stop using blocking signals to wait on the process, and use timeouts all the time. * messenger.py: Use a thread-local context for the handler, to allow for multiple parameters in the same process. * _remote.py: Ensure the start of the connection is thread safe * _signal.py: Allow blocking entering in the signal's context managers by setting an event. This is to ensure no thread runs long-running code while we asked the scheduler to pause. This also ensures all the signal handlers is thread safe. * source.py: Change check around saving the source's ref. We are now running in the same process, and thus the ref will already have been changed.
* setup.py: Ensure we have a version numberchandan/setup-py-version-warnChandan Singh2020-11-241-2/+15
| | | | | | | | BuildStream requires a version number at runtime. But, builds fine if I can't find any tags. So, make it an error at build time if we don't have a valid version number. Fixes #1383.
* NEWS/setup.py: Advertise our support for Python 3.9Chandan Singh2020-11-041-0/+1
|
* _loader: Use only one Dependency() classTristan van Berkom2020-08-101-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This moves the Dependency() implementation from the _loader.types module into the _loader.loadelement module, replacing the duplicate Dependency() definition there. Instead of creating _loader.loadelement.Dependency objects on the fly during the load process, we simply fill in the new LoadElement member on the original Dependency objects. This refactor should make the code easier to work with, since any originally loaded state from the YAML is preserved in the intermediate LoadElement data model and does not need to be manually handed over to a separate object. Summary of changes: * _loader/types.pyx: Removed Dependency() implementation from here, and renamed to _loader/types.py * _loader/loadelement.py: Replaced the second Dependency object implementation with the original one from _loader/types.py, adding a new LoadElement member which is resolved during the load process via the new Dependency.set_element() API * _loader/loader.py: Instead of creating a second kind of Dependency object on the fly during the load process, simply resolve the existing dependencies with Dependency.set_element() * setup.py: Fixed up cython build instructions, now that _loader/types.py no longer contains any cython code.
* Publish our type annotationschandan/publish-typesChandan Singh2020-08-041-1/+8
| | | | | | | | | | | As defined by PEP 561 (https://www.python.org/dev/peps/pep-0561), add a `py.typed` file to the BuildStream package. This allows any downstream packages (currently plugins, maybe other use cases in future) to run type checkers against BuildStream as well. Although we don't have type hints for the private API surface, downstream packages should only be using public API anyway, so they should mostly be fine.
* All: Update all references to the mailing list.Tristan van Berkom2020-07-031-2/+2
| | | | | | Update any references to the mailing list to now point to: https://lists.apache.org/list.html?dev@buildstream.apache.org
* setup.py: Also run blackbschubert/pylint-setup-pyBenjamin Schubert2020-05-071-114/+109
|
* setup.py: Also run pylint on this fileBenjamin Schubert2020-05-071-4/+2
| | | | Also fix the various errors in the file
* setup.py: Require Python >= 3.6Jürg Billeter2020-04-021-3/+3
|
* setup.py: Remove check for bubblewrapchandan/no-warn-bwrapChandan Singh2020-01-071-51/+0
| | | | | | | | | | | | | | Bubblewrap is not a hard requirement for BuildStream now that it supports other sandboxing mechanisms, i.e. buildbox-run and remote execution. Aside from that, this check is already a bit misplaced in `setup.py`. Bubblewrap is a runtime requirement, so warning about it at build time isn't quite right. It makes sense when BuildStream is being installed from source, but not when it is installed from any binary distribution. Note that the corresponding sandbox implementation already does this check at runtime.
* Drop support for `setup.py test`Chandan Singh2019-12-231-44/+0
| | | | | | | | | | | | | | Drop support for running tests via `setup.py test`, that is considered deprecated. `tox` is our primary frontend for running tests, so this change ensures that we don't have to support multiple ways of running tests. For testing against a specific installation environment, `tox` is not quite practical. But in these cases, one can run `pytest` directly. So, there is no need for this additional indirection. This was discussed in the following mailing list thread: https://mail.gnome.org/archives/buildstream-list/2019-December/msg00006.html.
* types: Add a 'FastEnum' implementation and replace Enum by itBenjamin Schubert2019-07-291-0/+1
| | | | | | | 'Enum' has a big performance impact on the running code. Replacing it with a safe subset of functionality removes lots of this overhead without removing the benefits of using enums (safe comparisions, uniqueness)
* loadelement: Move loadelement to CythonBenjamin Schubert2019-07-261-0/+1
|
* utils: Extract 'url_directory_name' to a cython moduleBenjamin Schubert2019-07-171-0/+1
| | | | | `url_directory_name` is heavily called from any downloadable source plugin, and moving it to cython gives a more than 10x speedup
* _yaml: Split Node-related parts into 'node.pyx'Benjamin Schubert2019-07-151-3/+4
| | | | | This makes the 'Node' API public, and available for use directly for plugins.
* Refactor of Platform and SandboxWilliam Salmon2019-07-121-1/+5
|
* _loader/types: cimport yaml functions for better speedBenjamin Schubert2019-06-061-1/+1
| | | | | | - _yaml: export node_validate function as Cython, as it was not done before. This requires rewriting the function to remove a closure. - Optimize node check by not calling is_node().
* _loader/types: move to a cython packageBenjamin Schubert2019-06-061-0/+1
| | | | | | Types is a simple module that accounts for a few percent of a basic 'show' operation. Having it cythonized allows us to get better performance without too much wokr
* _loader/loader: cythonize valid_chars_nameBenjamin Schubert2019-06-051-0/+1
| | | | | | | | | - Create a new _loader/utils.pyx cython module for functions cythonized in the loader module. - Move valid_chars_name from loader to utils and cythonize. This function is called extensively, and easy to extract
* setup.py: Be more restrictive with BST_CYTHON_TRACE valuesbschubert/fix-cython-traceBenjamin Schubert2019-06-031-1/+6
| | | | | | "0" evaluates to 'True' in python, which incorrectly switched on the BST_CYTHON_TRACE. Forcing an int for this environment variable allows us to ensure we are correct.
* Setup.py: Disable linetrace by default and only enable when using coverageBenjamin Schubert2019-05-311-4/+5
| | | | | | | | | It turns out that enabling 'linetrace', does have a runtime cost even if not enabled in distutils. Therefore disabling it by default. In order to run coverage, we therefore need to retranspile the .pyx files with ENABLE_CYTHON_TRACE set.
* _variable: Import _yaml from C.Benjamin Schubert2019-05-291-1/+1
| | | | | This requires the addition of a definition file (.pxd), to list symbols exported.
* _yaml: Cythonize and internalize NodeBenjamin Schubert2019-05-291-0/+1
| | | | | | | | | | | Node used to be a NamedTuple that we used to access by index for speed reasons. Moving to an extension class allows us to access attributes by name, making the code easier to read and less error prone. Moreover, we do gain speed and memory by doing this move. Also fix a few places where we would not have an entire `Node` object but were instead just returning a tuple, missing some entries.
* _variables: Cythonize _internal_expandBenjamin Schubert2019-05-291-0/+1
| | | | | | | Move _variables.py to be a Cython module. `_internal_expand` is a function that is called a lot in BuildStream. It is also entirely isolated and easy to cythonize.
* Introduce Cython to the project and documentBenjamin Schubert2019-05-291-1/+101
| | | | | | | | | | | | | Cython requires a plugin to allow coverage of cython files, which was updated in coveragerc. It also means we need to build the dependencies and install cython for coverage. Cython requires access to both source and compiled files when running coverage. We therefore need to install project in develop mode. Updated documentation to explain how to run tests without tox but with coverage
* Introduce pyproject.tomlBenjamin Schubert2019-05-291-2/+4
| | | | | | | | | Using pyproject.toml, defined in PEP518, allows us to have an isolated build environment, ensuring that Cython can be installed before calling setup.py in tox. This allows us to use cython helpers in the setup.py script. This is a prerequisite for introducing Cython in the codebase
* Move source from 'buildstream' to 'src/buildstream'Chandan Singh2019-05-211-4/+5
| | | | | | This was discussed in #1008. Fixes #1009.
* setup.py: Include buildstream.testing datafiles in package_dataPhil Dawson2019-04-231-2/+17
| | | | | | | * Remove `include_package_data=True` from setup.py * Collect datafiles for buildstream.testing by adding them to `package_data` Fixes #1008
* Make templated source tests available in buildstream.plugintestutilsPhil Dawson2019-04-121-0/+1
|
* setup.py: Do not error out when man directory is empty/missingchandan/toxic-manChandan Singh2019-01-251-2/+6
| | | | | | | | | | If the `man` directory is empty, then it won't be copied in the source distribution, and `list_man_pages()` will throw an exception when trying to list files in a non-existent directory. This prevents us from installing the BuildStream package when the man pages are not there. The most common use-case for this is when we want to re-generate the man pages but want to install the package before re-generating them.
* setup.py, CONTRIBUTING.rst: Recommend using tox to generate man pagesChandan Singh2019-01-251-3/+2
| | | | | | Simplify our docs, by requesting users to run `tox -e man` to update man pages, instead of manually installing `click-man` and running the command manually.
* setup.py: Import server main from _cas/casserver.pyJames Ennis2019-01-161-1/+1
| | | | | | | Since the CAS refactor, we have not been able to execute bst-artifact-server. This commit ensures that we can. This fix closes #867
* Rename "tools" directory to "requirements"Chandan Singh2019-01-041-2/+2
| | | | | This seems like a better name for the directory, as it more closely describes the purpose of its contents.
* Move all requirements files into "tools" directoryChandan Singh2019-01-031-2/+2
| | | | | | These new `.in` and `.txt` are making the repository look very cluttered. Move them to a separate `tools` directory to make it look a bit cleaner.
* Add requirements files for install, test and plugin dependenciesChandan Singh2019-01-021-18/+5
| | | | | | | | | | | | | | Add `.in` and `.txt` requirements files for BuildStream's pure python dependencies. For each pair, the `.in` file is supposed to capture the loose version requirements, and the corresponding `.txt` file is supposed to have frozen requirements. We have 3 such sets: * `requirements`: BuildStream's runtime dependencies * `dev-requirements`: Dependencies for running tests * `plugin-requirements`: Dependencies for core plugins Note that the frozen requirements files will only be used for testing purposes, and `setup.py` will continue to read loose requirements.
* Introduce new "source" command groupChandan Singh2018-12-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Following the message thread https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html, implement a new command group called `source`. Move existing `track`, `fetch`, and the recently added `source-checkout` commands under this group. For `track` and `fetch`, this is a BREAKING change, as the old commands have been marked as obsolete. Using them will result in an error message that refers people to use the new versions, like `bst source fetch` instead of old `bst fetch`. `source-checkout` will now become `source checkout` (the dash has turned into a space), and is not a breaking change as it was added in the current development cycle. Note that the functionality to hide commands from help output was added only recently in Click, so the minimum version of Click that we now require is 7.0. Summary of changes: * _frontend/cli.py: Add `source` command group, mark previous versions as obsolete and hide them from the help output. * _frontend/complete.py: Fix completion for hidden commands. * setup.py: Bump Click minimum version to 7.0. * tests: Update to cope with the new command names. Fixes #814.
* setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52valentindavid/ruamel_0_15_41Valentin David2018-11-281-1/+8
|
* Remove dependency on pytest-runnerBenjamin Schubert2018-11-011-1/+40
| | | | | | | | This includes a new command mimicking pytest-runner so that we can drop this dependency This was the only setup_requires dependency that we had and will make like easier for people behind proxies
* setup.py: use correct link for documentation websiteTiago Gomes2018-10-171-1/+1
|
* setup.py: Change bwrap assertion to a warningDaniel Silverstone2018-10-041-9/+11
| | | | | | | | | | Since there are use-cases where BuildStream could be installed onto systems which do not have BubbleWrap (e.g. for remote-build-only scenarios) it is not correct to assert a dependency on bwrap during installation. This patch makes the assertion a warning, and also clarifies the message somewhat. This should fix #644 Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* setup.py: Make website the primary homepageChandan Singh2018-09-211-1/+2
|
* _frontend/status.py: Completely remove the blessings dependency from BuildStreamTristan Van Berkom2018-09-191-1/+0
| | | | | | | | | | | | | | | | | | | | | | | This actually improves reliability of the status bar because we now disable it completely in the case that not all of the terminal escape sequences are supported on the given terminal. This replaces the few functions we were using, to move the cursor up one line, move it to the beginning of the line, and to clear a line, with low level functions provided by the curses module in the standard library. This change makes it easier for downstream distro package maintainers to package BuildStream, particularly on Fedora. Asides from changing _frontend/status.py, this commit includes the following changes: * _frontend/app.py: Use python isatty() function to determine if we are connected to a tty, instead of relying on blessings. * setup.py: Remove the dependency on blessings.
* setup.py: Add more metadata useful for PyPIChandan Singh2018-08-291-1/+25
| | | | | | | | | | | | | | | | | | | - setup.py: List useful links using `project_urls` - setup.py: Specify minumum python vresion using `python_requires` `python_requires` is the new way of specifying requirements for python version, as documented @ https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires. This will also make this requirement appear nicely on the PyPI project page. - setup.py: Add project classifiers PyPI will use these classifiers to categorize projects while searching or browsing. Full list of classifiers can be found at https://pypi.org/classifiers.
* deps: Specify the minimum version required for blessingstristan/blessingsMathieu Bridon2018-08-231-1/+1
| | | | | | | | | | | | | Buildstream uses the Terminal.does_styling attribute, which was only added in blessings 1.6. Blessings helpfully returns an empty string when calling an nonexistent attribute on the Terminal object, instead of failing. Because Fedora has blessings 1.5, Buildstream thought that my terminal couldn't handle styling, and just didn't print the status bar, silently. Specifying the minimum version avoids this silent failure.
* setup.py: Add fields required for publishing on PyPichandan/pypi-prepChandan Singh2018-08-221-0/+12
| | | | | | | Fill out those fields in `setup.py` that are required for publishing a package on PyPi. Part of https://gitlab.com/BuildStream/buildstream/issues/587.
* Set version of ruaml.yaml to strictly less than 0.15.52.Valentin David2018-08-131-1/+1
| | | | | | | This allows to use version 0.15.51 rather than 0.15.0 which is required for Python 3. Fixes #571.
* dev-requirements.txt: Remove obsolete commentsJavier Jardón2018-08-121-5/+3
| | | | | This also fixes an error with pylint in setup.py: setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)
* Move development reqirements to dev-requirements.txtChandan Singh2018-08-111-13/+11
| | | | | | | | | | | | | | | | | | In some cases, such as when working inside a virtual environment, it can be desirable to install all dependencies for running tests using `pip`. This is currently not possible since setuptools does not support installing these dependencies in a virtual environment (by design). (See https://stackoverflow.com/a/21003259.) To circumvent this issue, move such requirements to `dev-requirements.txt` file that can be used easily with `pip install -r`. This also enables tests to be run directly using `pytest`, which can be more convenient than `-addopts` approach when one needs to add multiple options. This will also be useful in creating better testuite images, and fix some of the issues noticed in https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/56.
* setup.py: Pin ruamel.yaml version to <= 0.15Jim MacArthur2018-08-091-1/+1
|
* Revert "Restrict version of pylint"Jürg Billeter2018-08-071-2/+1
| | | | | | | | | This reverts commit 4f168b9b6a02216e2fae24d758ae6b778e545869. The latest version of pytest_pylint works fine with pylint 2, which means there is no longer a reason to restrict the pylint version. pylint 2 is required for Python 3.7.
View Lorry Controller Status