diff options
| author | bst-marge-bot <marge-bot@buildstream.build> | 2020-06-03 15:11:06 +0000 |
|---|---|---|
| committer | bst-marge-bot <marge-bot@buildstream.build> | 2020-06-03 15:11:06 +0000 |
| commit | 1a40438653ca5e61ead0eb6a5444586823e4eabf (patch) | |
| tree | 8ba148a0ba1461d8a6cf254136ba5eba5e08d7ef | |
| parent | 53b8d042b6d969c5089faf9fb4c7dd5e05e09388 (diff) | |
| parent | cbf490426112dabe18f0dca3af00863235eb1e3f (diff) | |
| download | buildstream-1a40438653ca5e61ead0eb6a5444586823e4eabf.tar.gz | |
Merge branch 'juerg/sandbox-drop-bwrap' into 'master'
Drop bwrap sandboxing backend
See merge request BuildStream/buildstream!1952
30 files changed, 75 insertions, 2753 deletions
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index dfd638cdd..b8b30edb8 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -116,12 +116,6 @@ overnight-fedora-30-aarch64: only: - schedules -tests-bwrap: - image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION} - <<: *tests - variables: - BST_FORCE_SANDBOX: "bwrap" - tests-userchroot: image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION} <<: *tests @@ -450,7 +444,6 @@ coverage: - tox -e coverage - cp -a .coverage-reports/ ./coverage-report dependencies: - - tests-bwrap - tests-centos-7.7 - tests-debian-10 - tests-fedora-30 @@ -2,6 +2,13 @@ (unreleased) ============ +Core +---- + + o The bwrap sandboxing backend and the SafeHardlinks FUSE filesystem + have been dropped. buildbox-run is now the only sandboxing backend. + + CLI --- diff --git a/doc/source/arch_sandboxing.rst b/doc/source/arch_sandboxing.rst index 52679dedd..13997852a 100644 --- a/doc/source/arch_sandboxing.rst +++ b/doc/source/arch_sandboxing.rst @@ -128,18 +128,17 @@ be a console device so that it can be used interactively. Platform notes -------------- -BuildStream currently only carries first-class support for modern Linux-based -operating systems. - -There is also a "fallback" backend which aims to make BuildStream usable on any -POSIX-compatible operating system. The POSIX standard does not provide good -support for creating containers so this implementation makes a number of -unfortunate compromises. +BuildStream delegates sandboxing for local builds to the ``buildbox-run`` +command. ``buildbox-run`` provides a platform-independent interface to execute +commands in a sandbox based on parts of the Remote Execution API. Linux ~~~~~ -On Linux we use the following isolation and sandboxing primitives: +The recommended ``buildbox-run`` implementation for Linux is +``buildbox-run-bubblewrap``, in combination with ``buildbox-fuse``. + +These implementations use the following isolation and sandboxing primitives: * bind mounts * FUSE @@ -153,22 +152,17 @@ We access all of these features through a sandboxing tool named `Bubblewrap <https://github.com/projectatomic/bubblewrap/>`_. User namespaces are not enabled by default in all Linux distributions. -BuildStream still runs on such systems but will give a big warning on startup -and will refuse to push any artifacts built on such a system to a remote cache. -For more information, see `issue #92 -<https://gitlab.com/BuildStream/buildstream/issues/92>`_. +BuildStream still runs on such systems but can't build projects that set +``build-uid`` or ``build-gid`` in the ``sandbox`` configuration. -The Linux platform can operate as a standard user, if user namespace +The Linux platform can operate as a standard user, if unprivileged user namespace support is available. If user namespace support is not available you have the option of installing bubblewrap as a setuid binary to avoid needing to run the entire ``bst`` process as the ``root`` user. -The artifact cache on Linux systems is implemented using a content-addressable -hardlink farm, which can allow us to stage artifacts using hardlinks instead of -copying them. To avoid cache corruption it is vital that hardlinked files -cannot be overwritten. In cases where the root filesystem inside the sandbox -needs to be writable, a custom FUSE filesystem named SafeHardlinks is used -which provides a copy-on-write layer. +FUSE is used to provide access to directories and files stored in CAS without +having to copy or hardlink the complete input tree into a regular filesystem +directory structure for each build job. Some of the operations on filesystem metadata listed above are not prohibited by the sandbox, but will instead be silently dropped when an artifact is @@ -179,22 +173,19 @@ Some details of the host machine are currently leaked by this platform backend. For more details, see `issue #262 <https://gitlab.com/BuildStream/buildstream/issues/262>`_. -Fallback (POSIX) -~~~~~~~~~~~~~~~~ +Other POSIX systems +~~~~~~~~~~~~~~~~~~~ -The fallback backend aims to be usable on a wide range of operating systems. -Any OS that implements the POSIX specification and the ``chroot()`` syscall -can be expected to work. There are no real isolation or sandboxing primitives -that work across multiple operating systems, so the protection provided by -this backend is minimal. It would be much safer to use a platform-specific -backend. +On other POSIX systems ``buildbox-run-userchroot`` may be used for sandboxing. +`userchroot <https://gitlab.com/BuildGrid/buildbox/userchroot>`_ allows regular +users to invoke processes in a chroot environment. -Filesystem isolation is done using the chroot() system call. This system call -requires special privileges to use so ``bst`` usually needs to be run as the -``root`` user when using this backend. +``buildbox-run-userchroot`` stages the input tree for each build job using +hardlinks to avoid more expensive file copies. To avoid cache corruption it is +vital that hardlinked files cannot be overwritten. Due to this it's required +to run ``buildbox-casd`` as a separate user, which owns the files in the local +cache. -Network access is not blocked in the sandbox. However since there is unlikely +Network access is not blocked in the chroot. However since there is unlikely to be a correct `/etc/resolv.conf` file, any network access that depends on name resolution will most likely fail anyway. - -Builds inside the sandbox execute as the ``root`` user. diff --git a/pyproject.toml b/pyproject.toml index 29f5589b5..fefbbecd4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,7 +21,6 @@ exclude = ''' | build | dist )/ - | src/buildstream/_fuse | src/buildstream/_protos ) ''' @@ -36,5 +36,5 @@ ignore_missing_imports=True ignore_missing_imports=True # Ignore issues with generated files and vendored code -[mypy-buildstream._protos.*,buildstream._fuse.*,buildstream._version] +[mypy-buildstream._protos.*,buildstream._version] ignore_errors = True diff --git a/src/buildstream/_fuse/__init__.py b/src/buildstream/_fuse/__init__.py deleted file mode 100644 index a5e882634..000000000 --- a/src/buildstream/_fuse/__init__.py +++ /dev/null @@ -1,20 +0,0 @@ -# -# Copyright (C) 2017 Codethink Limited -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. -# -# Authors: -# Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> - -from .hardlinks import SafeHardlinks diff --git a/src/buildstream/_fuse/fuse.py b/src/buildstream/_fuse/fuse.py deleted file mode 100644 index 41e126ef5..000000000 --- a/src/buildstream/_fuse/fuse.py +++ /dev/null @@ -1,1001 +0,0 @@ -# This is an embedded copy of fuse.py taken from the following upstream commit: -# -# https://github.com/terencehonles/fusepy/commit/0eafeb557e0e70926ed9450008ef17057d302391 -# -# Our local modifications are recorded in the Git history of this repo. - -# Copyright (c) 2012 Terence Honles <terence@honles.com> (maintainer) -# Copyright (c) 2008 Giorgos Verigakis <verigak@gmail.com> (author) -# -# Permission to use, copy, modify, and distribute this software for any -# purpose with or without fee is hereby granted, provided that the above -# copyright notice and this permission notice appear in all copies. -# -# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES -# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF -# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR -# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES -# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN -# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF -# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. - -# pylint: skip-file - -from __future__ import print_function, absolute_import, division - -from ctypes import * -from ctypes.util import find_library -from errno import * -from os import strerror -from platform import machine, system -from signal import signal, SIGINT, SIG_DFL -from stat import S_IFDIR -from traceback import print_exc - -import logging - -try: - from functools import partial -except ImportError: - # http://docs.python.org/library/functools.html#functools.partial - def partial(func, *args, **keywords): - def newfunc(*fargs, **fkeywords): - newkeywords = keywords.copy() - newkeywords.update(fkeywords) - return func(*(args + fargs), **newkeywords) - - newfunc.func = func - newfunc.args = args - newfunc.keywords = keywords - return newfunc - -try: - basestring -except NameError: - basestring = str - -class c_timespec(Structure): - _fields_ = [('tv_sec', c_long), ('tv_nsec', c_long)] - -class c_utimbuf(Structure): - _fields_ = [('actime', c_timespec), ('modtime', c_timespec)] - -class c_stat(Structure): - pass # Platform dependent - -_system = system() -_machine = machine() - -if _system == 'Darwin': - _libiconv = CDLL(find_library('iconv'), RTLD_GLOBAL) # libfuse dependency - _libfuse_path = (find_library('fuse4x') or find_library('osxfuse') or - find_library('fuse')) -else: - _libfuse_path = find_library('fuse') - -if not _libfuse_path: - raise EnvironmentError('Unable to find libfuse') -else: - _libfuse = CDLL(_libfuse_path) - -if _system == 'Darwin' and hasattr(_libfuse, 'macfuse_version'): - _system = 'Darwin-MacFuse' - - -if _system in ('Darwin', 'Darwin-MacFuse', 'FreeBSD'): - ENOTSUP = 45 - c_dev_t = c_int32 - c_fsblkcnt_t = c_ulong - c_fsfilcnt_t = c_ulong - c_gid_t = c_uint32 - c_mode_t = c_uint16 - c_off_t = c_int64 - c_pid_t = c_int32 - c_uid_t = c_uint32 - setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), - c_size_t, c_int, c_uint32) - getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), - c_size_t, c_uint32) - if _system == 'Darwin': - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('st_mode', c_mode_t), - ('st_nlink', c_uint16), - ('st_ino', c_uint64), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('st_rdev', c_dev_t), - ('st_atimespec', c_timespec), - ('st_mtimespec', c_timespec), - ('st_ctimespec', c_timespec), - ('st_birthtimespec', c_timespec), - ('st_size', c_off_t), - ('st_blocks', c_int64), - ('st_blksize', c_int32), - ('st_flags', c_int32), - ('st_gen', c_int32), - ('st_lspare', c_int32), - ('st_qspare', c_int64)] - else: - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('st_ino', c_uint32), - ('st_mode', c_mode_t), - ('st_nlink', c_uint16), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('st_rdev', c_dev_t), - ('st_atimespec', c_timespec), - ('st_mtimespec', c_timespec), - ('st_ctimespec', c_timespec), - ('st_size', c_off_t), - ('st_blocks', c_int64), - ('st_blksize', c_int32)] -elif _system == 'Linux': - ENOTSUP = 95 - c_dev_t = c_ulonglong - c_fsblkcnt_t = c_ulonglong - c_fsfilcnt_t = c_ulonglong - c_gid_t = c_uint - c_mode_t = c_uint - c_off_t = c_longlong - c_pid_t = c_int - c_uid_t = c_uint - setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), - c_size_t, c_int) - - getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), - c_size_t) - - if _machine == 'x86_64': - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('st_ino', c_ulong), - ('st_nlink', c_ulong), - ('st_mode', c_mode_t), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('__pad0', c_int), - ('st_rdev', c_dev_t), - ('st_size', c_off_t), - ('st_blksize', c_long), - ('st_blocks', c_long), - ('st_atimespec', c_timespec), - ('st_mtimespec', c_timespec), - ('st_ctimespec', c_timespec)] - elif _machine == 'mips': - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('__pad1_1', c_ulong), - ('__pad1_2', c_ulong), - ('__pad1_3', c_ulong), - ('st_ino', c_ulong), - ('st_mode', c_mode_t), - ('st_nlink', c_ulong), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('st_rdev', c_dev_t), - ('__pad2_1', c_ulong), - ('__pad2_2', c_ulong), - ('st_size', c_off_t), - ('__pad3', c_ulong), - ('st_atimespec', c_timespec), - ('__pad4', c_ulong), - ('st_mtimespec', c_timespec), - ('__pad5', c_ulong), - ('st_ctimespec', c_timespec), - ('__pad6', c_ulong), - ('st_blksize', c_long), - ('st_blocks', c_long), - ('__pad7_1', c_ulong), - ('__pad7_2', c_ulong), - ('__pad7_3', c_ulong), - ('__pad7_4', c_ulong), - ('__pad7_5', c_ulong), - ('__pad7_6', c_ulong), - ('__pad7_7', c_ulong), - ('__pad7_8', c_ulong), - ('__pad7_9', c_ulong), - ('__pad7_10', c_ulong), - ('__pad7_11', c_ulong), - ('__pad7_12', c_ulong), - ('__pad7_13', c_ulong), - ('__pad7_14', c_ulong)] - elif _machine == 'ppc': - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('st_ino', c_ulonglong), - ('st_mode', c_mode_t), - ('st_nlink', c_uint), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('st_rdev', c_dev_t), - ('__pad2', c_ushort), - ('st_size', c_off_t), - ('st_blksize', c_long), - ('st_blocks', c_longlong), - ('st_atimespec', c_timespec), - ('st_mtimespec', c_timespec), - ('st_ctimespec', c_timespec)] - elif _machine == 'ppc64' or _machine == 'ppc64le': - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('st_ino', c_ulong), - ('st_nlink', c_ulong), - ('st_mode', c_mode_t), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('__pad', c_uint), - ('st_rdev', c_dev_t), - ('st_size', c_off_t), - ('st_blksize', c_long), - ('st_blocks', c_long), - ('st_atimespec', c_timespec), - ('st_mtimespec', c_timespec), - ('st_ctimespec', c_timespec)] - elif _machine == 'aarch64': - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('st_ino', c_ulong), - ('st_mode', c_mode_t), - ('st_nlink', c_uint), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('st_rdev', c_dev_t), - ('__pad1', c_ulong), - ('st_size', c_off_t), - ('st_blksize', c_int), - ('__pad2', c_int), - ('st_blocks', c_long), - ('st_atimespec', c_timespec), - ('st_mtimespec', c_timespec), - ('st_ctimespec', c_timespec)] - else: - # i686, use as fallback for everything else - c_stat._fields_ = [ - ('st_dev', c_dev_t), - ('__pad1', c_ushort), - ('__st_ino', c_ulong), - ('st_mode', c_mode_t), - ('st_nlink', c_uint), - ('st_uid', c_uid_t), - ('st_gid', c_gid_t), - ('st_rdev', c_dev_t), - ('__pad2', c_ushort), - ('st_size', c_off_t), - ('st_blksize', c_long), - ('st_blocks', c_longlong), - ('st_atimespec', c_timespec), - ('st_mtimespec', c_timespec), - ('st_ctimespec', c_timespec), - ('st_ino', c_ulonglong)] -else: - raise NotImplementedError('{} is not supported.'.format(_system)) - - -class c_statvfs(Structure): - _fields_ = [ - ('f_bsize', c_ulong), - ('f_frsize', c_ulong), - ('f_blocks', c_fsblkcnt_t), - ('f_bfree', c_fsblkcnt_t), - ('f_bavail', c_fsblkcnt_t), - ('f_files', c_fsfilcnt_t), - ('f_ffree', c_fsfilcnt_t), - ('f_favail', c_fsfilcnt_t), - ('f_fsid', c_ulong), - #('unused', c_int), - ('f_flag', c_ulong), - ('f_namemax', c_ulong)] - -if _system == 'FreeBSD': - c_fsblkcnt_t = c_uint64 - c_fsfilcnt_t = c_uint64 - setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), - c_size_t, c_int) - - getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), - c_size_t) - - class c_statvfs(Structure): - _fields_ = [ - ('f_bavail', c_fsblkcnt_t), - ('f_bfree', c_fsblkcnt_t), - ('f_blocks', c_fsblkcnt_t), - ('f_favail', c_fsfilcnt_t), - ('f_ffree', c_fsfilcnt_t), - ('f_files', c_fsfilcnt_t), - ('f_bsize', c_ulong), - ('f_flag', c_ulong), - ('f_frsize', c_ulong)] - -class fuse_file_info(Structure): - _fields_ = [ - ('flags', c_int), - ('fh_old', c_ulong), - ('writepage', c_int), - ('direct_io', c_uint, 1), - ('keep_cache', c_uint, 1), - ('flush', c_uint, 1), - ('padding', c_uint, 29), - ('fh', c_uint64), - ('lock_owner', c_uint64)] - -class fuse_context(Structure): - _fields_ = [ - ('fuse', c_voidp), - ('uid', c_uid_t), - ('gid', c_gid_t), - ('pid', c_pid_t), - ('private_data', c_voidp)] - -_libfuse.fuse_get_context.restype = POINTER(fuse_context) - - -class fuse_operations(Structure): - _fields_ = [ - ('getattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat))), - ('readlink', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), - ('getdir', c_voidp), # Deprecated, use readdir - ('mknod', CFUNCTYPE(c_int, c_char_p, c_mode_t, c_dev_t)), - ('mkdir', CFUNCTYPE(c_int, c_char_p, c_mode_t)), - ('unlink', CFUNCTYPE(c_int, c_char_p)), - ('rmdir', CFUNCTYPE(c_int, c_char_p)), - ('symlink', CFUNCTYPE(c_int, c_char_p, c_char_p)), - ('rename', CFUNCTYPE(c_int, c_char_p, c_char_p)), - ('link', CFUNCTYPE(c_int, c_char_p, c_char_p)), - ('chmod', CFUNCTYPE(c_int, c_char_p, c_mode_t)), - ('chown', CFUNCTYPE(c_int, c_char_p, c_uid_t, c_gid_t)), - ('truncate', CFUNCTYPE(c_int, c_char_p, c_off_t)), - ('utime', c_voidp), # Deprecated, use utimens - ('open', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), - - ('read', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, - c_off_t, POINTER(fuse_file_info))), - - ('write', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, - c_off_t, POINTER(fuse_file_info))), - - ('statfs', CFUNCTYPE(c_int, c_char_p, POINTER(c_statvfs))), - ('flush', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), - ('release', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), - ('fsync', CFUNCTYPE(c_int, c_char_p, c_int, POINTER(fuse_file_info))), - ('setxattr', setxattr_t), - ('getxattr', getxattr_t), - ('listxattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), - ('removexattr', CFUNCTYPE(c_int, c_char_p, c_char_p)), - ('opendir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), - - ('readdir', CFUNCTYPE(c_int, c_char_p, c_voidp, - CFUNCTYPE(c_int, c_voidp, c_char_p, - POINTER(c_stat), c_off_t), - c_off_t, POINTER(fuse_file_info))), - - ('releasedir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), - - ('fsyncdir', CFUNCTYPE(c_int, c_char_p, c_int, - POINTER(fuse_file_info))), - - ('init', CFUNCTYPE(c_voidp, c_voidp)), - ('destroy', CFUNCTYPE(c_voidp, c_voidp)), - ('access', CFUNCTYPE(c_int, c_char_p, c_int)), - - ('create', CFUNCTYPE(c_int, c_char_p, c_mode_t, - POINTER(fuse_file_info))), - - ('ftruncate', CFUNCTYPE(c_int, c_char_p, c_off_t, - POINTER(fuse_file_info))), - - ('fgetattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat), - POINTER(fuse_file_info))), - - ('lock', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info), - c_int, c_voidp)), - - ('utimens', CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))), - ('bmap', CFUNCTYPE(c_int, c_char_p, c_size_t, POINTER(c_ulonglong))), - ('flag_nullpath_ok', c_uint, 1), - ('flag_nopath', c_uint, 1), - ('flag_utime_omit_ok', c_uint, 1), - ('flag_reserved', c_uint, 29), - ] - - -def time_of_timespec(ts): - return ts.tv_sec + ts.tv_nsec / 10 ** 9 - -def set_st_attrs(st, attrs): - for key, val in attrs.items(): - if key in ('st_atime', 'st_mtime', 'st_ctime', 'st_birthtime'): - timespec = getattr(st, key + 'spec', None) - if timespec is None: - continue - timespec.tv_sec = int(val) - timespec.tv_nsec = int((val - timespec.tv_sec) * 10 ** 9) - elif hasattr(st, key): - setattr(st, key, val) - - -def fuse_get_context(): - 'Returns a (uid, gid, pid) tuple' - - ctxp = _libfuse.fuse_get_context() - ctx = ctxp.contents - return ctx.uid, ctx.gid, ctx.pid - - -class FuseOSError(OSError): - def __init__(self, errno): - super().__init__(errno, strerror(errno)) - - -class FUSE(object): - ''' - This class is the lower level interface and should not be subclassed under - normal use. Its methods are called by fuse. - - Assumes API version 2.6 or later. - ''' - - OPTIONS = ( - ('foreground', '-f'), - ('debug', '-d'), - ('nothreads', '-s'), - ) - - def __init__(self, operations, mountpoint, raw_fi=False, encoding='utf-8', - **kwargs): - - ''' - Setting raw_fi to True will cause FUSE to pass the fuse_file_info - class as is to Operations, instead of just the fh field. - - This gives you access to direct_io, keep_cache, etc. - ''' - - # Note that in BuildStream we're assuming that raw_fi is always False. - assert not raw_fi, "raw_fi is not supported in BuildStream." - - self.operations = operations - self.raw_fi = raw_fi - self.encoding = encoding - - args = ['fuse'] - - args.extend(flag for arg, flag in self.OPTIONS - if kwargs.pop(arg, False)) - - kwargs.setdefault('fsname', operations.__class__.__name__) - args.append('-o') - args.append(','.join(self._normalize_fuse_options(**kwargs))) - args.append(mountpoint) - - args = [arg.encode(encoding) for arg in args] - argv = (c_char_p * len(args))(*args) - - fuse_ops = fuse_operations() - for ent in fuse_operations._fields_: - name, prototype = ent[:2] - - val = getattr(operations, name, None) - if val is None: - continue - - # Function pointer members are tested for using the - # getattr(operations, name) above but are dynamically - # invoked using self.operations(name) - if hasattr(prototype, 'argtypes'): - val = prototype(partial(self._wrapper, getattr(self, name))) - - setattr(fuse_ops, name, val) - - try: - old_handler = signal(SIGINT, SIG_DFL) - except ValueError: - old_handler = SIG_DFL - - err = _libfuse.fuse_main_real(len(args), argv, pointer(fuse_ops), - sizeof(fuse_ops), None) - - try: - signal(SIGINT, old_handler) - except ValueError: - pass - - del self.operations # Invoke the destructor - if err: - raise RuntimeError(err) - - @staticmethod - def _normalize_fuse_options(**kargs): - for key, value in kargs.items(): - if isinstance(value, bool): - if value is True: yield key - else: - yield '{}={}'.format(key, value) - - @staticmethod - def _wrapper(func, *args, **kwargs): - 'Decorator for the methods that follow' - - try: - return func(*args, **kwargs) or 0 - except OSError as e: - return -(e.errno or EFAULT) - except: - print_exc() - return -EFAULT - - def _decode_optional_path(self, path): - # NB: this method is intended for fuse operations that - # allow the path argument to be NULL, - # *not* as a generic path decoding method - if path is None: - return None - return path.decode(self.encoding) - - def getattr(self, path, buf): - return self.fgetattr(path, buf, None) - - def readlink(self, path, buf, bufsize): - ret = self.operations('readlink', path.decode(self.encoding)) \ - .encode(self.encoding) - - # copies a string into the given buffer - # (null terminated and truncated if necessary) - data = create_string_buffer(ret[:bufsize - 1]) - memmove(buf, data, len(data)) - return 0 - - def mknod(self, path, mode, dev): - return self.operations('mknod', path.decode(self.encoding), mode, dev) - - def mkdir(self, path, mode): - return self.operations('mkdir', path.decode(self.encoding), mode) - - def unlink(self, path): - return self.operations('unlink', path.decode(self.encoding)) - - def rmdir(self, path): - return self.operations('rmdir', path.decode(self.encoding)) - - def symlink(self, source, target): - 'creates a symlink `target -> source` (e.g. ln -s source target)' - - return self.operations('symlink', target.decode(self.encoding), - source.decode(self.encoding)) - - def rename(self, old, new): - return self.operations('rename', old.decode(self.encoding), - new.decode(self.encoding)) - - def link(self, source, target): - 'creates a hard link `target -> source` (e.g. ln source target)' - - return self.operations('link', target.decode(self.encoding), - source.decode(self.encoding)) - - def chmod(self, path, mode): - return self.operations('chmod', path.decode(self.encoding), mode) - - def chown(self, path, uid, gid): - # Check if any of the arguments is a -1 that has overflowed - if c_uid_t(uid + 1).value == 0: - uid = -1 - if c_gid_t(gid + 1).value == 0: - gid = -1 - - return self.operations('chown', path.decode(self.encoding), uid, gid) - - def truncate(self, path, length): - return self.operations('truncate', path.decode(self.encoding), length) - - def open(self, path, fip): - fi = fip.contents - if self.raw_fi: - return self.operations('open', path.decode(self.encoding), fi) - else: - fi.fh = self.operations('open', path.decode(self.encoding), - fi.flags) - - return 0 - - def read(self, path, buf, size, offset, fip): - if self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - ret = self.operations('read', self._decode_optional_path(path), size, - offset, fh) - - if not ret: return 0 - - retsize = len(ret) - assert retsize <= size, \ - 'actual amount read {:d} greater than expected {:d}'.format(retsize, size) - - data = create_string_buffer(ret, retsize) - memmove(buf, data, retsize) - return retsize - - def write(self, path, buf, size, offset, fip): - data = string_at(buf, size) - - if self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - return self.operations('write', self._decode_optional_path(path), data, - offset, fh) - - def statfs(self, path, buf): - stv = buf.contents - attrs = self.operations('statfs', path.decode(self.encoding)) - for key, val in attrs.items(): - if hasattr(stv, key): - setattr(stv, key, val) - - return 0 - - def flush(self, path, fip): - if self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - return self.operations('flush', self._decode_optional_path(path), fh) - - def release(self, path, fip): - if self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - return self.operations('release', self._decode_optional_path(path), fh) - - def fsync(self, path, datasync, fip): - if self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - return self.operations('fsync', self._decode_optional_path(path), datasync, - fh) - - def setxattr(self, path, name, value, size, options, *args): - return self.operations('setxattr', path.decode(self.encoding), - name.decode(self.encoding), - string_at(value, size), options, *args) - - def getxattr(self, path, name, value, size, *args): - ret = self.operations('getxattr', path.decode(self.encoding), - name.decode(self.encoding), *args) - - retsize = len(ret) - # allow size queries - if not value: return retsize - - # do not truncate - if retsize > size: return -ERANGE - - buf = create_string_buffer(ret, retsize) # Does not add trailing 0 - memmove(value, buf, retsize) - - return retsize - - def listxattr(self, path, namebuf, size): - attrs = self.operations('listxattr', path.decode(self.encoding)) or '' - ret = '\x00'.join(attrs).encode(self.encoding) - if len(ret) > 0: - ret += '\x00'.encode(self.encoding) - - retsize = len(ret) - # allow size queries - if not namebuf: return retsize - - # do not truncate - if retsize > size: return -ERANGE - - buf = create_string_buffer(ret, retsize) - memmove(namebuf, buf, retsize) - - return retsize - - def removexattr(self, path, name): - return self.operations('removexattr', path.decode(self.encoding), - name.decode(self.encoding)) - - def opendir(self, path, fip): - # Ignore raw_fi - fip.contents.fh = self.operations('opendir', - path.decode(self.encoding)) - - return 0 - - def readdir(self, path, buf, filler, offset, fip): - # Ignore raw_fi - for item in self.operations('readdir', self._decode_optional_path(path), - fip.contents.fh): - - if isinstance(item, basestring): - name, st, offset = item, None, 0 - else: - name, attrs, offset = item - if attrs: - st = c_stat() - set_st_attrs(st, attrs) - else: - st = None - - if filler(buf, name.encode(self.encoding), st, offset) != 0: - break - - return 0 - - def releasedir(self, path, fip): - # Ignore raw_fi - return self.operations('releasedir', self._decode_optional_path(path), - fip.contents.fh) - - def fsyncdir(self, path, datasync, fip): - # Ignore raw_fi - return self.operations('fsyncdir', self._decode_optional_path(path), - datasync, fip.contents.fh) - - def init(self, conn): - return self.operations('init', '/') - - def destroy(self, private_data): - return self.operations('destroy', '/') - - def access(self, path, amode): - return self.operations('access', path.decode(self.encoding), amode) - - def create(self, path, mode, fip): - fi = fip.contents - path = path.decode(self.encoding) - - assert not self.raw_fi - - # This line is different from upstream to fix issues - # reading file opened with O_CREAT|O_RDWR. - # See issue #143. - fi.fh = self.operations('create', path, mode, fi.flags) - # END OF MODIFICATION - return 0 - - def ftruncate(self, path, length, fip): - if self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - return self.operations('truncate', self._decode_optional_path(path), - length, fh) - - def fgetattr(self, path, buf, fip): - memset(buf, 0, sizeof(c_stat)) - - st = buf.contents - if not fip: - fh = fip - elif self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - attrs = self.operations('getattr', self._decode_optional_path(path), fh) - set_st_attrs(st, attrs) - return 0 - - def lock(self, path, fip, cmd, lock): - if self.raw_fi: - fh = fip.contents - else: - fh = fip.contents.fh - - return self.operations('lock', self._decode_optional_path(path), fh, cmd, - lock) - - def utimens(self, path, buf): - if buf: - atime = time_of_timespec(buf.contents.actime) - mtime = time_of_timespec(buf.contents.modtime) - times = (atime, mtime) - else: - times = None - - return self.operations('utimens', path.decode(self.encoding), times) - - def bmap(self, path, blocksize, idx): - return self.operations('bmap', path.decode(self.encoding), blocksize, - idx) - - -class Operations(object): - ''' - This class should be subclassed and passed as an argument to FUSE on - initialization. All operations should raise a FuseOSError exception on - error. - - When in doubt of what an operation should do, check the FUSE header file - or the corresponding system call man page. - ''' - - def __call__(self, op, *args): - if not hasattr(self, op): - raise FuseOSError(EFAULT) - return getattr(self, op)(*args) - - def access(self, path, amode): - return 0 - - bmap = None - - def chmod(self, path, mode): - raise FuseOSError(EROFS) - - def chown(self, path, uid, gid): - raise FuseOSError(EROFS) - - def create(self, path, mode, flags): - - raise FuseOSError(EROFS) - - def destroy(self, path): - 'Called on filesystem destruction. Path is always /' - - pass - - def flush(self, path, fh): - return 0 - - def fsync(self, path, datasync, fh): - return 0 - - def fsyncdir(self, path, datasync, fh): - return 0 - - def getattr(self, path, fh=None): - ''' - Returns a dictionary with keys identical to the stat C structure of - stat(2). - - st_atime, st_mtime and st_ctime should be floats. - - NOTE: There is an incombatibility between Linux and Mac OS X - concerning st_nlink of directories. Mac OS X counts all files inside - the directory, while Linux counts only the subdirectories. - ''' - - if path != '/': - raise FuseOSError(ENOENT) - return dict(st_mode=(S_IFDIR | 0o755), st_nlink=2) - - def getxattr(self, path, name, position=0): - raise FuseOSError(ENOTSUP) - - def init(self, path): - ''' - Called on filesystem initialization. (Path is always /) - - Use it instead of __init__ if you start threads on initialization. - ''' - - pass - - def link(self, target, source): - 'creates a hard link `target -> source` (e.g. ln source target)' - - raise FuseOSError(EROFS) - - def listxattr(self, path): - return [] - - lock = None - - def mkdir(self, path, mode): - raise FuseOSError(EROFS) - - def mknod(self, path, mode, dev): - raise FuseOSError(EROFS) - - def open(self, path, flags): - ''' - When raw_fi is False (default case), open should return a numerical - file handle. - - When raw_fi is True the signature of open becomes: - open(self, path, fi) - - and the file handle should be set directly. - ''' - - return 0 - - def opendir(self, path): - 'Returns a numerical file handle.' - - return 0 - - def read(self, path, size, offset, fh): - 'Returns a string containing the data requested.' - - raise FuseOSError(EIO) - - def readdir(self, path, fh): - ''' - Can return either a list of names, or a list of (name, attrs, offset) - tuples. attrs is a dict as in getattr. - ''' - - return ['.', '..'] - - def readlink(self, path): - raise FuseOSError(ENOENT) - - def release(self, path, fh): - return 0 - - def releasedir(self, path, fh): - return 0 - - def removexattr(self, path, name): - raise FuseOSError(ENOTSUP) - - def rename(self, old, new): - raise FuseOSError(EROFS) - - def rmdir(self, path): - raise FuseOSError(EROFS) - - def setxattr(self, path, name, value, options, position=0): - raise FuseOSError(ENOTSUP) - - def statfs(self, path): - ''' - Returns a dictionary with keys identical to the statvfs C structure of - statvfs(3). - - On Mac OS X f_bsize and f_frsize must be a power of 2 - (minimum 512). - ''' - - return {} - - def symlink(self, target, source): - 'creates a symlink `target -> source` (e.g. ln -s source target)' - - raise FuseOSError(EROFS) - - def truncate(self, path, length, fh=None): - raise FuseOSError(EROFS) - - def unlink(self, path): - raise FuseOSError(EROFS) - - def utimens(self, path, times=None): - 'Times is a (atime, mtime) tuple. If None use current time.' - - return 0 - - def write(self, path, data, offset, fh): - raise FuseOSError(EROFS) - - -class LoggingMixIn: - log = logging.getLogger('fuse.log-mixin') - - def __call__(self, op, path, *args): - self.log.debug('-> %s %s %s', op, path, repr(args)) - ret = '[Unhandled Exception]' - try: - ret = getattr(self, op)(path, *args) - return ret - except OSError as e: - ret = str(e) - raise - finally: - self.log.debug('<- %s %s', op, repr(ret)) diff --git a/src/buildstream/_fuse/hardlinks.py b/src/buildstream/_fuse/hardlinks.py deleted file mode 100644 index 798e1c816..000000000 --- a/src/buildstream/_fuse/hardlinks.py +++ /dev/null @@ -1,220 +0,0 @@ -# -# Copyright (C) 2016 Stavros Korokithakis -# Copyright (C) 2017 Codethink Limited -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. -# -# Authors: -# Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> -# -# The filesystem operations implementation here is based -# on some example code written by Stavros Korokithakis. - -import errno -import os -import shutil -import stat -import tempfile - -from .fuse import FuseOSError, Operations - -from .mount import Mount - - -# SafeHardlinks() -# -# A FUSE mount which implements a copy on write hardlink experience. -# -# Args: -# root (str): The underlying filesystem path to mirror -# tmp (str): A directory on the same filesystem for creating temp files -# -class SafeHardlinks(Mount): - - def __init__(self, directory, tempdir, fuse_mount_options=None): - self.directory = directory - self.tempdir = tempdir - if fuse_mount_options is None: - fuse_mount_options = {} - super().__init__(fuse_mount_options=fuse_mount_options) - - def create_operations(self): - return SafeHardlinkOps(self.directory, self.tempdir) - - -# SafeHardlinkOps() -# -# The actual FUSE Operations implementation below. -# -class SafeHardlinkOps(Operations): - - def __init__(self, root, tmp): - self.root = root - self.tmp = tmp - - def _full_path(self, partial): - if partial.startswith("/"): - partial = partial[1:] - path = os.path.join(self.root, partial) - return path - - def _ensure_copy(self, full_path): - try: - # Follow symbolic links manually here - real_path = os.path.realpath(full_path) - file_stat = os.stat(real_path) - - # Dont bother with files that cannot be hardlinked, oddly it - # directories actually usually have st_nlink > 1 so just avoid - # that. - # - # We already wont get symlinks here, and stat will throw - # the FileNotFoundError below if a followed symlink did not exist. - # - if not stat.S_ISDIR(file_stat.st_mode) and file_stat.st_nlink > 1: - with tempfile.TemporaryDirectory(dir=self.tmp) as tempdir: - basename = os.path.basename(real_path) - temp_path = os.path.join(tempdir, basename) - - # First copy, then unlink origin and rename - shutil.copy2(real_path, temp_path) - os.unlink(real_path) - os.rename(temp_path, real_path) - - except FileNotFoundError: - # This doesnt exist yet, assume we're about to create it - # so it's not a problem. - pass - - ########################################################### - # Fuse Methods # - ########################################################### - def access(self, path, amode): - full_path = self._full_path(path) - if not os.access(full_path, amode): - raise FuseOSError(errno.EACCES) - - def chmod(self, path, mode): - full_path = self._full_path(path) - - # Ensure copies on chmod - self._ensure_copy(full_path) - return os.chmod(full_path, mode) - - def chown(self, path, uid, gid): - full_path = self._full_path(path) - - # Ensure copies on chown - self._ensure_copy(full_path) - return os.chown(full_path, uid, gid) - - def getattr(self, path, fh=None): - full_path = self._full_path(path) - st = os.lstat(full_path) - return dict((key, getattr(st, key)) for key in ( - 'st_atime', 'st_ctime', 'st_gid', 'st_mode', - 'st_mtime', 'st_nlink', 'st_size', 'st_uid', 'st_rdev')) - - def readdir(self, path, fh): - full_path = self._full_path(path) - - dirents = ['.', '..'] - if os.path.isdir(full_path): - dirents.extend(os.listdir(full_path)) - for r in dirents: - yield r - - def readlink(self, path): - pathname = os.readlink(self._full_path(path)) - if pathname.startswith("/"): - # Path name is absolute, sanitize it. - return os.path.relpath(pathname, self.root) - else: - return pathname - - def mknod(self, path, mode, dev): - return os.mknod(self._full_path(path), mode, dev) - - def rmdir(self, path): - full_path = self._full_path(path) - return os.rmdir(full_path) - - def mkdir(self, path, mode): - return os.mkdir(self._full_path(path), mode) - - def statfs(self, path): - full_path = self._full_path(path) - stv = os.statvfs(full_path) - return dict((key, getattr(stv, key)) for key in ( - 'f_bavail', 'f_bfree', 'f_blocks', 'f_bsize', 'f_favail', - 'f_ffree', 'f_files', 'f_flag', 'f_frsize', 'f_namemax')) - - def unlink(self, path): - os.unlink(self._full_path(path)) - - def symlink(self, target, source): - 'creates a symlink `target -> source` (e.g. ln -s source target)' - return os.symlink(source, self._full_path(target)) - - def rename(self, old, new): - return os.rename(self._full_path(old), self._full_path(new)) - - def link(self, target, source): - 'creates a hard link `target -> source` (e.g. ln source target)' - - # When creating a hard link here, should we ensure the original - # file is not a hardlink itself first ? - # - return os.link(self._full_path(source), self._full_path(target)) - - def utimens(self, path, times=None): - return os.utime(self._full_path(path), times) - - def open(self, path, flags): - full_path = self._full_path(path) - - # If we're opening for writing, ensure it's a copy first - if flags & os.O_WRONLY or flags & os.O_RDWR: - self._ensure_copy(full_path) - - return os.open(full_path, flags) - - def create(self, path, mode, flags): - full_path = self._full_path(path) - - # If it already exists, ensure it's a copy first - self._ensure_copy(full_path) - return os.open(full_path, flags, mode) - - def read(self, path, size, offset, fh): - os.lseek(fh, offset, os.SEEK_SET) - return os.read(fh, size) - - def write(self, path, data, offset, fh): - os.lseek(fh, offset, os.SEEK_SET) - return os.write(fh, data) - - def truncate(self, path, length, fh=None): - full_path = self._full_path(path) - with open(full_path, 'r+') as f: - f.truncate(length) - - def flush(self, path, fh): - return os.fsync(fh) - - def release(self, path, fh): - return os.close(fh) - - def fsync(self, path, datasync, fh): - return self.flush(path, fh) diff --git a/src/buildstream/_fuse/mount.py b/src/buildstream/_fuse/mount.py deleted file mode 100644 index 8cd968f0d..000000000 --- a/src/buildstream/_fuse/mount.py +++ /dev/null @@ -1,217 +0,0 @@ -# -# Copyright (C) 2017 Codethink Limited -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. -# -# Authors: -# Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> - -import os -import signal -import time -import sys - -from contextlib import contextmanager -from multiprocessing import Process -from .fuse import FUSE - -from .._exceptions import ImplError -from .. import _signals, utils - - -# Just a custom exception to raise here, for identifying possible -# bugs with a fuse layer implementation -# -class FuseMountError(Exception): - pass - - -# This is a convenience class which takes care of synchronizing the -# startup of FUSE and shutting it down. -# -# The implementations / subclasses should: -# -# - Overload the instance initializer to add any parameters -# needed for their fuse Operations implementation -# -# - Implement create_operations() to create the Operations -# instance on behalf of the superclass, using any additional -# parameters collected in the initializer. -# -# Mount objects can be treated as contextmanagers, the volume -# will be mounted during the context. -# -# UGLY CODE NOTE: -# -# This is a horrible little piece of code. The problem we face -# here is that the highlevel libfuse API has fuse_main(), which -# will either block in the foreground, or become a full daemon. -# -# With the daemon approach, we know that the fuse is mounted right -# away when fuse_main() returns, then the daemon will go and handle -# requests on its own, but then we have no way to shut down the -# daemon. -# -# With the blocking approach, we still have it as a child process -# so we can tell it to gracefully terminate; but it's impossible -# to know when the mount is done, there is no callback for that -# -# The solution we use here without digging too deep into the -# low level fuse API, is to start a child process which will -# run the fuse loop in foreground, and we block the parent -# process until the volume is mounted with a busy loop with timeouts. -# -class Mount(): - - # These are not really class data, they are - # just here for the sake of having None setup instead - # of missing attributes, since we do not provide any - # initializer and leave the initializer to the subclass. - # - __mountpoint = None - __operations = None - __process = None - __logfile = None - - ################################################ - # User Facing API # - ################################################ - - def __init__(self, fuse_mount_options=None): - self._fuse_mount_options = {} if fuse_mount_options is None else fuse_mount_options - - # _mount(): - # - # Mount a fuse subclass implementation. - # - # Args: - # (str): Location to mount this fuse fs - # - def _mount(self, mountpoint): - - assert self.__process is None - - self.__mountpoint = mountpoint - self.__process = Process(target=self.__run_fuse, args=(self.__logfile.name,)) - - # Ensure the child process does not inherit our signal handlers, if the - # child wants to handle a signal then it will first set its own - # handler, and then unblock it. - with _signals.blocked([signal.SIGTERM, signal.SIGTSTP, signal.SIGINT], ignore=False): - self.__process.start() - - while not os.path.ismount(mountpoint): - if not self.__process.is_alive(): - self.__logfile.seek(0) - stderr = self.__logfile.read() - raise FuseMountError("Unable to mount {}: {}".format(mountpoint, stderr.decode().strip())) - - time.sleep(1 / 100) - - # _unmount(): - # - # Unmount a fuse subclass implementation. - # - def _unmount(self): - - # Terminate child process and join - if self.__process is not None: - self.__process.terminate() - self.__process.join() - - # Report an error if ever the underlying operations crashed for some reason. - if self.__process.exitcode != 0: - self.__logfile.seek(0) - stderr = self.__logfile.read() - - raise FuseMountError("{} reported exit code {} when unmounting: {}" - .format(type(self).__name__, self.__process.exitcode, stderr)) - - self.__mountpoint = None - self.__process = None - - # mounted(): - # - # A context manager to run a code block with this fuse Mount - # mounted, this will take care of automatically unmounting - # in the case that the calling process is terminated. - # - # Args: - # (str): Location to mount this fuse fs - # - @contextmanager - def mounted(self, mountpoint): - - with utils._tempnamedfile() as logfile: - self.__logfile = logfile - - self._mount(mountpoint) - try: - with _signals.terminator(self._unmount): - yield - finally: - self._unmount() - - self.__logfile = None - - ################################################ - # Abstract Methods # - ################################################ - - # create_operations(): - # - # Create an Operations class (from fusepy) and return it - # - # Returns: - # (Operations): A FUSE Operations implementation - def create_operations(self): - raise ImplError("Mount subclass '{}' did not implement create_operations()" - .format(type(self).__name__)) - - ################################################ - # Child Process # - ################################################ - def __run_fuse(self, filename): - # Override stdout/stderr to the file given as a pointer, that way our parent process can get our output - out = open(filename, "w") - os.dup2(out.fileno(), sys.stdout.fileno()) - os.dup2(out.fileno(), sys.stderr.fileno()) - - # First become session leader while signals are still blocked - # - # Then reset the SIGTERM handler to the default and finally - # unblock SIGTERM. - # - os.setsid() - signal.signal(signal.SIGTERM, signal.SIG_DFL) - signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGTERM]) - - # Ask the subclass to give us an Operations object - # - self.__operations = self.create_operations() # pylint: disable=assignment-from-no-return - - # Run fuse in foreground in this child process, internally libfuse - # will handle SIGTERM and gracefully exit its own little main loop. - # - try: - FUSE(self.__operations, self.__mountpoint, nothreads=True, foreground=True, - **self._fuse_mount_options) - except RuntimeError as exc: - # FUSE will throw a RuntimeError with the exit code of libfuse as args[0] - sys.exit(exc.args[0]) - - # Explicit 0 exit code, if the operations crashed for some reason, the exit - # code will not be 0, and we want to know about it. - # - sys.exit(0) diff --git a/src/buildstream/_platform/darwin.py b/src/buildstream/_platform/darwin.py deleted file mode 100644 index e880ea6f6..000000000 --- a/src/buildstream/_platform/darwin.py +++ /dev/null @@ -1,62 +0,0 @@ -# -# Copyright (C) 2017 Codethink Limited -# Copyright (C) 2018 Bloomberg Finance LP -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. - -import resource - -from ..sandbox import SandboxDummy - -from .platform import Platform - - -class Darwin(Platform): - - # This value comes from OPEN_MAX in syslimits.h - OPEN_MAX = 10240 - - def maximize_open_file_limit(self): - # Note that on Mac OSX, you may not be able to simply set the soft - # limit to the reported hard limit, as it may not be the only limit in - # effect. The output of these commands may be somewhat independent: - # - # $ launchctl limit - # $ sysctl -a | grep files - # - # The OPEN_MAX value from syslimits.h seems to be fairly safe, although - # users may tweak their individual systems to have different values. - # Without a way to determine what the real limit is, we risk failing to - # increase the limit. Perhaps the complication is why psutil does not - # support rlimit on Mac. - # - old_soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE) - soft_limit = min(max(self.OPEN_MAX, old_soft_limit), hard_limit) - resource.setrlimit(resource.RLIMIT_NOFILE, (soft_limit, hard_limit)) - - @staticmethod - def _check_dummy_sandbox_config(config): - pass - - @staticmethod - def _create_dummy_sandbox(*args, **kwargs): - kwargs["dummy_reason"] = ( - "OSXFUSE is not supported and there are no supported sandbox " + "technologies for MacOS at this time" - ) - return SandboxDummy(*args, **kwargs) - - def _setup_dummy_sandbox(self): - self.check_sandbox_config = Darwin._check_dummy_sandbox_config - self.create_sandbox = Darwin._create_dummy_sandbox - return True diff --git a/src/buildstream/_platform/fallback.py b/src/buildstream/_platform/fallback.py deleted file mode 100644 index d80ac8fde..000000000 --- a/src/buildstream/_platform/fallback.py +++ /dev/null @@ -1,37 +0,0 @@ -# -# Copyright (C) 2018 Bloomberg Finance LP -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. - -from ..sandbox import SandboxDummy - -from .platform import Platform - - -class Fallback(Platform): - def _check_dummy_sandbox_config(self, config): - pass - - def _create_dummy_sandbox(self, *args, **kwargs): - kwargs["dummy_reason"] = ( - "FallBack platform only implements dummy sandbox, " - "Buildstream may be having issues correctly detecting your platform, " - "platform can be forced with BST_FORCE_BACKEND" - ) - return SandboxDummy(*args, **kwargs) - - def _setup_dummy_sandbox(self): - self.check_sandbox_config = self._check_dummy_sandbox_config - self.create_sandbox = self._create_dummy_sandbox - return True diff --git a/src/buildstream/_platform/linux.py b/src/buildstream/_platform/linux.py deleted file mode 100644 index 74d0e1227..000000000 --- a/src/buildstream/_platform/linux.py +++ /dev/null @@ -1,108 +0,0 @@ -# -# Copyright (C) 2017 Codethink Limited -# Copyright (C) 2018 Bloomberg Finance LP -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. -# -# Authors: -# Tristan Maat <tristan.maat@codethink.co.uk> - -import os - -from .. import utils -from ..sandbox import SandboxDummy - -from .platform import Platform - - -class Linux(Platform): - def _setup_sandbox(self, force_sandbox): - sandbox_setups = { - "bwrap": self._setup_bwrap_sandbox, - "buildbox-run": self.setup_buildboxrun_sandbox, - "dummy": self._setup_dummy_sandbox, - } - - preferred_sandboxes = [ - "buildbox-run", - ] - - self._try_sandboxes(force_sandbox, sandbox_setups, preferred_sandboxes) - - def __init__(self, force_sandbox=None): - super().__init__(force_sandbox=force_sandbox) - - self._uid = os.geteuid() - self._gid = os.getegid() - - # Set linux32 option - self.linux32 = None - - def can_crossbuild(self, config): - host_arch = self.get_host_arch() - if (config.build_arch == "x86-32" and host_arch == "x86-64") or ( - config.build_arch == "aarch32" and host_arch == "aarch64" - ): - if self.linux32 is None: - try: - utils.get_host_tool("linux32") - self.linux32 = True - except utils.ProgramNotFoundError: - self.linux32 = False - return self.linux32 - return False - - ################################################ - # Private Methods # - ################################################ - - # Dummy sandbox methods - @staticmethod - def _check_dummy_sandbox_config(config): - return True - - def _create_dummy_sandbox(self, *args, **kwargs): - dummy_reasons = " and ".join(self.dummy_reasons) - kwargs["dummy_reason"] = dummy_reasons - return SandboxDummy(*args, **kwargs) - - def _setup_dummy_sandbox(self): - self.check_sandbox_config = Linux._check_dummy_sandbox_config - self.create_sandbox = self._create_dummy_sandbox - return True - - # Bubble-wrap sandbox methods - def _check_sandbox_config_bwrap(self, config): - from ..sandbox._sandboxbwrap import SandboxBwrap - - SandboxBwrap.check_sandbox_config(self, config) - - def _create_bwrap_sandbox(self, *args, **kwargs): - from ..sandbox._sandboxbwrap import SandboxBwrap - - kwargs["linux32"] = self.linux32 - return SandboxBwrap(*args, **kwargs) - - def _setup_bwrap_sandbox(self): - from ..sandbox._sandboxbwrap import SandboxBwrap - - # This function should only be called once. - # but if it does eg, in the tests we want to - # reset the sandbox checks - - SandboxBwrap._have_good_bwrap = None - self._check_sandbox(SandboxBwrap) - self.check_sandbox_config = self._check_sandbox_config_bwrap - self.create_sandbox = self._create_bwrap_sandbox - return True diff --git a/src/buildstream/_platform/platform.py b/src/buildstream/_platform/platform.py index d0debfc7b..f46043cbf 100644 --- a/src/buildstream/_platform/platform.py +++ b/src/buildstream/_platform/platform.py @@ -21,11 +21,11 @@ import multiprocessing import os import platform -import sys import psutil from .._exceptions import PlatformError, ImplError, SandboxError +from ..sandbox import SandboxDummy from .. import utils @@ -35,54 +35,18 @@ class Platform: # A class to manage platform-specific details. Currently holds the # sandbox factory as well as platform helpers. # - # Args: - # force_sandbox (bool): Force bst to use a particular sandbox - # - def __init__(self, force_sandbox=None): - self.maximize_open_file_limit() + def __init__(self): self._local_sandbox = None self.dummy_reasons = [] - self._setup_sandbox(force_sandbox) - - def _setup_sandbox(self, force_sandbox): - # The buildbox-run interface is not platform-specific - sandbox_setups = {"buildbox-run": self.setup_buildboxrun_sandbox, "dummy": self._setup_dummy_sandbox} - - preferred_sandboxes = [ - "buildbox-run", - ] + self._setup_sandbox() - self._try_sandboxes(force_sandbox, sandbox_setups, preferred_sandboxes) - - def _try_sandboxes(self, force_sandbox, sandbox_setups, preferred_sandboxes): - # Any sandbox from sandbox_setups can be forced by BST_FORCE_SANDBOX - # But if a specific sandbox is not forced then only `first class` sandbox are tried before - # falling back to the dummy sandbox. + def _setup_sandbox(self): + # Try to setup buildbox-run sandbox, otherwise fallback to the dummy sandbox. # Where `first_class` sandboxes are those in preferred_sandboxes - if force_sandbox: - try: - sandbox_setups[force_sandbox]() - except KeyError: - raise PlatformError( - "Forced Sandbox is unavailable on this platform: BST_FORCE_SANDBOX" - " is set to {} but it is not available".format(force_sandbox) - ) - except SandboxError as Error: - raise PlatformError( - "Forced Sandbox Error: BST_FORCE_SANDBOX" - " is set to {} but cannot be setup".format(force_sandbox), - detail=" and ".join(self.dummy_reasons), - ) from Error - else: - for good_sandbox in preferred_sandboxes: - try: - sandbox_setups[good_sandbox]() - return - except SandboxError: - continue - except utils.ProgramNotFoundError: - continue - sandbox_setups["dummy"]() + try: + self._setup_buildboxrun_sandbox() + except (SandboxError, utils.ProgramNotFoundError): + self._setup_dummy_sandbox() def _check_sandbox(self, Sandbox): Sandbox._dummy_reasons = [] @@ -94,37 +58,7 @@ class Platform: @classmethod def create_instance(cls): - # Meant for testing purposes and therefore hidden in the - # deepest corners of the source code. Try not to abuse this, - # please? - if os.getenv("BST_FORCE_SANDBOX"): - force_sandbox = os.getenv("BST_FORCE_SANDBOX") - else: - force_sandbox = None - - if os.getenv("BST_FORCE_BACKEND"): - backend = os.getenv("BST_FORCE_BACKEND") - elif sys.platform.startswith("darwin"): - backend = "darwin" - elif sys.platform.startswith("linux"): - backend = "linux" - elif sys.platform == "win32": - backend = "win32" - else: - backend = "fallback" - - if backend == "linux": - from .linux import Linux as PlatformImpl # pylint: disable=cyclic-import - elif backend == "darwin": - from .darwin import Darwin as PlatformImpl # pylint: disable=cyclic-import - elif backend == "win32": - from .win32 import Win32 as PlatformImpl # pylint: disable=cyclic-import - elif backend == "fallback": - from .fallback import Fallback as PlatformImpl # pylint: disable=cyclic-import - else: - raise PlatformError("No such platform: '{}'".format(backend)) - - return PlatformImpl(force_sandbox=force_sandbox) + return Platform() def get_cpu_count(self, cap=None): # `psutil.Process.cpu_affinity()` is not available on all platforms. @@ -248,27 +182,6 @@ class Platform: "Platform {platform} does not implement check_sandbox_config()".format(platform=type(self).__name__) ) - def maximize_open_file_limit(self): - # Need to set resources for _frontend/app.py as this is dependent on the platform - # SafeHardlinks FUSE needs to hold file descriptors for all processes in the sandbox. - # Avoid hitting the limit too quickly, by increasing it as far as we can. - - # Import this late, as it is not available on Windows. Note that we - # could use `psutil.Process().rlimit` instead, but this would introduce - # a dependency on the `prlimit(2)` call, which seems to only be - # available on Linux. For more info: - # https://github.com/giampaolo/psutil/blob/cbf2bafbd33ad21ef63400d94cb313c299e78a45/psutil/_psutil_linux.c#L45 - import resource - - soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE) - if soft_limit != hard_limit: - resource.setrlimit(resource.RLIMIT_NOFILE, (hard_limit, hard_limit)) - - def _setup_dummy_sandbox(self): - raise ImplError( - "Platform {platform} does not implement _setup_dummy_sandbox()".format(platform=type(self).__name__) - ) - # Buildbox run sandbox methods def _check_sandbox_config_buildboxrun(self, config): from ..sandbox._sandboxbuildboxrun import SandboxBuildBoxRun # pylint: disable=cyclic-import @@ -281,10 +194,25 @@ class Platform: return SandboxBuildBoxRun(*args, **kwargs) - def setup_buildboxrun_sandbox(self): + def _setup_buildboxrun_sandbox(self): from ..sandbox._sandboxbuildboxrun import SandboxBuildBoxRun # pylint: disable=cyclic-import self._check_sandbox(SandboxBuildBoxRun) self.check_sandbox_config = self._check_sandbox_config_buildboxrun self.create_sandbox = self._create_buildboxrun_sandbox return True + + # Dummy sandbox methods + @staticmethod + def _check_dummy_sandbox_config(config): + pass + + def _create_dummy_sandbox(self, *args, **kwargs): + dummy_reasons = " and ".join(self.dummy_reasons) + kwargs["dummy_reason"] = dummy_reasons + return SandboxDummy(*args, **kwargs) + + def _setup_dummy_sandbox(self): + self.check_sandbox_config = Platform._check_dummy_sandbox_config + self.create_sandbox = self._create_dummy_sandbox + return True diff --git a/src/buildstream/_platform/win32.py b/src/buildstream/_platform/win32.py deleted file mode 100644 index 33645e030..000000000 --- a/src/buildstream/_platform/win32.py +++ /dev/null @@ -1,58 +0,0 @@ -# -# Copyright (C) 2019 Bloomberg Finance LP -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. - -from ..sandbox import SandboxDummy - -from .platform import Platform - - -class Win32(Platform): - def maximize_open_file_limit(self): - # Note that on Windows, we don't have the 'resource' module to help us - # configure open file limits. - # - # 'psutil' provides an rlimit implementation that is only available on - # Linux, as of version 5.3. - # - # Given that this limit is only important for SafeHardLinks FUSE, and - # we don't have FUSE on Windows, this won't be an obstacle for now. - # - # If it does turn out to be an obstacle, beware that the Windows API - # `_setmaxstdio` for increasing the open file limit only applies to the - # 'stream I/O level', i.e. `fopen()` and friends. CPython opens files - # using `_wopen()`, which is at the 'low I/O level'. - # - # You can see this in the function `os_open_impl` in `posixmodule.c` in - # CPython version 3.9. - # - # For more information: - # https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setmaxstdio - # - pass - - @staticmethod - def _check_dummy_sandbox_config(config): - pass - - @staticmethod - def _create_dummy_sandbox(*args, **kwargs): - kwargs["dummy_reason"] = "There are no supported sandbox technologies for Win32 at this time." - return SandboxDummy(*args, **kwargs) - - def _setup_dummy_sandbox(self): - self.check_sandbox_config = Win32._check_dummy_sandbox_config - self.create_sandbox = Win32._create_dummy_sandbox - return True diff --git a/src/buildstream/_site.py b/src/buildstream/_site.py index db0587120..2bdf45b02 100644 --- a/src/buildstream/_site.py +++ b/src/buildstream/_site.py @@ -18,8 +18,6 @@ # Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> import os -import shutil -import subprocess # # Private module declaring some info about where the buildstream @@ -46,22 +44,3 @@ build_all_template = os.path.join(root, "data", "build-all.sh.in") # Module building script template build_module_template = os.path.join(root, "data", "build-module.sh.in") - - -def get_bwrap_version(): - # Get the current bwrap version - # - # returns None if no bwrap was found - # otherwise returns a tuple of 3 int: major, minor, patch - bwrap_path = shutil.which("bwrap") - - if not bwrap_path: - return None - - cmd = [bwrap_path, "--version"] - try: - version = str(subprocess.check_output(cmd).split()[1], "utf-8") - except subprocess.CalledProcessError: - return None - - return tuple(int(x) for x in version.split(".")) diff --git a/src/buildstream/sandbox/_mount.py b/src/buildstream/sandbox/_mount.py deleted file mode 100644 index 18751dde5..000000000 --- a/src/buildstream/sandbox/_mount.py +++ /dev/null @@ -1,148 +0,0 @@ -# -# Copyright (C) 2017 Codethink Limited -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. -# -# Authors: -# Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> - -import os -from collections import OrderedDict -from contextlib import contextmanager, ExitStack - -from .. import utils -from .._fuse import SafeHardlinks - - -# Mount() -# -# Helper data object representing a single mount point in the mount map -# -class Mount: - def __init__(self, sandbox, mount_point, safe_hardlinks, fuse_mount_options=None): - # Getting _get_underlying_directory() here is acceptable as - # we're part of the sandbox code. This will fail if our - # directory is CAS-based. - root_directory = sandbox.get_virtual_directory()._get_underlying_directory() - - self.mount_point = mount_point - self.safe_hardlinks = safe_hardlinks - self._fuse_mount_options = {} if fuse_mount_options is None else fuse_mount_options - - # FIXME: When the criteria for mounting something and its parent - # mount is identical, then there is no need to mount an additional - # fuse layer (i.e. if the root is read-write and there is a directory - # marked for staged artifacts directly within the rootfs, they can - # safely share the same fuse layer). - # - # In these cases it would be saner to redirect the sub-mount to - # a regular mount point within the parent's redirected mount. - # - if self.safe_hardlinks: - scratch_directory = sandbox._get_scratch_directory() - # Redirected mount - self.mount_origin = os.path.join(root_directory, mount_point.lstrip(os.sep)) - self.mount_base = os.path.join(scratch_directory, utils.url_directory_name(mount_point)) - self.mount_source = os.path.join(self.mount_base, "mount") - self.mount_tempdir = os.path.join(self.mount_base, "temp") - os.makedirs(self.mount_origin, exist_ok=True) - os.makedirs(self.mount_tempdir, exist_ok=True) - else: - # No redirection needed - self.mount_source = os.path.join(root_directory, mount_point.lstrip(os.sep)) - - external_mount_sources = sandbox._get_mount_sources() - external_mount_source = external_mount_sources.get(mount_point) - - if external_mount_source is None: - os.makedirs(self.mount_source, exist_ok=True) - else: - if os.path.isdir(external_mount_source): - os.makedirs(self.mount_source, exist_ok=True) - else: - # When mounting a regular file, ensure the parent - # directory exists in the sandbox; and that an empty - # file is created at the mount location. - parent_dir = os.path.dirname(self.mount_source.rstrip("/")) - os.makedirs(parent_dir, exist_ok=True) - if not os.path.exists(self.mount_source): - with open(self.mount_source, "w"): - pass - - @contextmanager - def mounted(self, sandbox): - if self.safe_hardlinks: - mount = SafeHardlinks(self.mount_origin, self.mount_tempdir, self._fuse_mount_options) - with mount.mounted(self.mount_source): - yield - else: - # Nothing to mount here - yield - - -# MountMap() -# -# Helper object for mapping of the sandbox mountpoints -# -# Args: -# sandbox (Sandbox): The sandbox object -# root_readonly (bool): Whether the sandbox root is readonly -# -class MountMap: - def __init__(self, sandbox, root_readonly, fuse_mount_options=None): - # We will be doing the mounts in the order in which they were declared. - self.mounts = OrderedDict() - - if fuse_mount_options is None: - fuse_mount_options = {} - - # We want safe hardlinks on rootfs whenever root is not readonly - self.mounts["/"] = Mount(sandbox, "/", not root_readonly, fuse_mount_options) - - for mark in sandbox._get_marked_directories(): - directory = mark["directory"] - artifact = mark["artifact"] - - # We want safe hardlinks for any non-root directory where - # artifacts will be staged to - self.mounts[directory] = Mount(sandbox, directory, artifact, fuse_mount_options) - - # get_mount_source() - # - # Gets the host directory where the mountpoint in the - # sandbox should be bind mounted from - # - # Args: - # mountpoint (str): The absolute mountpoint path inside the sandbox - # - # Returns: - # The host path to be mounted at the mount point - # - def get_mount_source(self, mountpoint): - return self.mounts[mountpoint].mount_source - - # mounted() - # - # A context manager which ensures all the mount sources - # were mounted with any fuse layers which may have been needed. - # - # Args: - # sandbox (Sandbox): The sandbox - # - @contextmanager - def mounted(self, sandbox): - with ExitStack() as stack: - for _, mount in self.mounts.items(): - stack.enter_context(mount.mounted(sandbox)) - yield diff --git a/src/buildstream/sandbox/_sandboxbwrap.py b/src/buildstream/sandbox/_sandboxbwrap.py deleted file mode 100644 index 216145938..000000000 --- a/src/buildstream/sandbox/_sandboxbwrap.py +++ /dev/null @@ -1,511 +0,0 @@ -# -# Copyright (C) 2016 Codethink Limited -# Copyright (C) 2019 Bloomberg Finance LP -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see <http://www.gnu.org/licenses/>. -# -# Authors: -# Andrew Leeming <andrew.leeming@codethink.co.uk> -# Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> -# William Salmon <will.salmon@codethink.co.uk> - -import collections -import json -import os -import sys -import time -import errno -import signal -import subprocess -import shutil -from contextlib import ExitStack, suppress -from tempfile import TemporaryFile - -import psutil - -from .._exceptions import SandboxError -from .. import utils, _signals -from . import Sandbox, SandboxFlags, SandboxCommandError -from .. import _site - - -# SandboxBwrap() -# -# Default bubblewrap based sandbox implementation. -# -class SandboxBwrap(Sandbox): - _have_good_bwrap = None - - # Minimal set of devices for the sandbox - DEVICES = ["/dev/full", "/dev/null", "/dev/urandom", "/dev/random", "/dev/zero"] - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.linux32 = kwargs["linux32"] - - @classmethod - def check_available(cls): - cls._have_fuse = os.path.exists("/dev/fuse") - if not cls._have_fuse: - cls._dummy_reasons += ["Fuse is unavailable"] - - try: - utils.get_host_tool("bwrap") - except utils.ProgramNotFoundError as Error: - cls._bwrap_exists = False - cls._have_good_bwrap = False - cls._die_with_parent_available = False - cls._json_status_available = False - cls._dummy_reasons += ["Bubblewrap not found"] - raise SandboxError(" and ".join(cls._dummy_reasons), reason="unavailable-local-sandbox") from Error - - bwrap_version = _site.get_bwrap_version() - - cls._bwrap_exists = True - cls._have_good_bwrap = (0, 1, 2) <= bwrap_version - cls._die_with_parent_available = (0, 1, 8) <= bwrap_version - cls._json_status_available = (0, 3, 2) <= bwrap_version - if not cls._have_good_bwrap: - cls._dummy_reasons += ["Bubblewrap is too old"] - raise SandboxError(" and ".join(cls._dummy_reasons)) - - cls._uid = os.geteuid() - cls._gid = os.getegid() - - cls.user_ns_available = cls._check_user_ns_available() - - @staticmethod - def _check_user_ns_available(): - # Here, lets check if bwrap is able to create user namespaces, - # issue a warning if it's not available, and save the state - # locally so that we can inform the sandbox to not try it - # later on. - bwrap = utils.get_host_tool("bwrap") - try: - whoami = utils.get_host_tool("whoami") - output = subprocess.check_output( - [bwrap, "--ro-bind", "/", "/", "--unshare-user", "--uid", "0", "--gid", "0", whoami,], - universal_newlines=True, - ).strip() - except subprocess.CalledProcessError: - output = "" - except utils.ProgramNotFoundError: - output = "" - - return output == "root" - - @classmethod - def check_sandbox_config(cls, local_platform, config): - if local_platform.does_multiprocessing_start_require_pickling(): - # Reinitialize class as class data is not pickled. - cls.check_available() - - if not cls.user_ns_available: - # Without user namespace support, the UID/GID in the sandbox - # will match the host UID/GID. - if config.build_uid is not None and config.build_uid != local_platform._uid: - raise SandboxError("Configured and host UID don't match and user namespace is not supported.") - if config.build_gid is not None and config.build_gid != local_platform._gid: - raise SandboxError("Configured and host UID don't match and user namespace is not supported.") - - host_os = local_platform.get_host_os() - host_arch = local_platform.get_host_arch() - if config.build_os != host_os: - raise SandboxError("Configured and host OS don't match.") - if config.build_arch != host_arch and not local_platform.can_crossbuild(config): - raise SandboxError("Configured architecture and host architecture don't match.") - - def _run(self, command, flags, *, cwd, env): - stdout, stderr = self._get_output() - - # Allowable access to underlying storage as we're part of the sandbox - root_directory = self.get_virtual_directory()._get_underlying_directory() - - if not self._has_command(command[0], env): - raise SandboxCommandError( - "Staged artifacts do not provide command " "'{}'".format(command[0]), reason="missing-command" - ) - - # NOTE: MountMap transitively imports `_fuse/fuse.py` which raises an - # EnvironmentError when fuse is not found. Since this module is - # expected to be imported even in absence of fuse, MountMap is imported - # here, and not at the top of the module. - from ._mount import MountMap - - # Create the mount map, this will tell us where - # each mount point needs to be mounted from and to - mount_map = MountMap(self, flags & SandboxFlags.ROOT_READ_ONLY) - root_mount_source = mount_map.get_mount_source("/") - - # start command with linux32 if needed - if self.linux32: - bwrap_command = [utils.get_host_tool("linux32")] - else: - bwrap_command = [] - - # Grab the full path of the bwrap binary - bwrap_command += [utils.get_host_tool("bwrap")] - - for k, v in env.items(): - bwrap_command += ["--setenv", k, v] - for k in os.environ.keys() - env.keys(): - bwrap_command += ["--unsetenv", k] - - # Create a new pid namespace, this also ensures that any subprocesses - # are cleaned up when the bwrap process exits. - bwrap_command += ["--unshare-pid"] - - # Ensure subprocesses are cleaned up when the bwrap parent dies. - if self._die_with_parent_available: - bwrap_command += ["--die-with-parent"] - - # Add in the root filesystem stuff first. - # - # The rootfs is mounted as RW initially so that further mounts can be - # placed on top. If a RO root is required, after all other mounts are - # complete, root is remounted as RO - bwrap_command += ["--bind", root_mount_source, "/"] - - if not flags & SandboxFlags.NETWORK_ENABLED: - bwrap_command += ["--unshare-net"] - bwrap_command += ["--unshare-uts", "--hostname", "buildstream"] - bwrap_command += ["--unshare-ipc"] - - # Give it a proc and tmpfs - bwrap_command += ["--proc", "/proc", "--tmpfs", "/tmp"] - - # In interactive mode, we want a complete devpts inside - # the container, so there is a /dev/console and such. In - # the regular non-interactive sandbox, we want to hand pick - # a minimal set of devices to expose to the sandbox. - # - if flags & SandboxFlags.INTERACTIVE: - bwrap_command += ["--dev", "/dev"] - else: - for device in self.DEVICES: - bwrap_command += ["--dev-bind", device, device] - - # Create a tmpfs for /dev/shm, if we're in interactive this - # is handled by `--dev /dev` - # - bwrap_command += ["--tmpfs", "/dev/shm"] - - # Add bind mounts to any marked directories - marked_directories = self._get_marked_directories() - mount_source_overrides = self._get_mount_sources() - for mark in marked_directories: - mount_point = mark["directory"] - if mount_point in mount_source_overrides: # pylint: disable=consider-using-get - mount_source = mount_source_overrides[mount_point] - else: - mount_source = mount_map.get_mount_source(mount_point) - - # Use --dev-bind for all mounts, this is simply a bind mount which does - # not restrictive about devices. - # - # While it's important for users to be able to mount devices - # into the sandbox for `bst shell` testing purposes, it is - # harmless to do in a build environment where the directories - # we mount just never contain device files. - # - bwrap_command += ["--dev-bind", mount_source, mount_point] - - if flags & SandboxFlags.ROOT_READ_ONLY: - bwrap_command += ["--remount-ro", "/"] - - if cwd is not None: - bwrap_command += ["--dir", cwd] - bwrap_command += ["--chdir", cwd] - - # Set UID and GUI - if self.user_ns_available: - bwrap_command += ["--unshare-user"] - if not flags & SandboxFlags.INHERIT_UID: - uid = self._get_config().build_uid or 0 - gid = self._get_config().build_gid or 0 - bwrap_command += ["--uid", str(uid), "--gid", str(gid)] - - with ExitStack() as stack: - pass_fds = () - # Improve error reporting with json-status if available - if self._json_status_available: - json_status_file = stack.enter_context(TemporaryFile()) - pass_fds = (json_status_file.fileno(),) - bwrap_command += ["--json-status-fd", str(json_status_file.fileno())] - - # Add the command - bwrap_command += command - - # bwrap might create some directories while being suid - # and may give them to root gid, if it does, we'll want - # to clean them up after, so record what we already had - # there just in case so that we can safely cleanup the debris. - # - existing_basedirs = { - directory: os.path.exists(os.path.join(root_directory, directory)) - for directory in ["dev/shm", "tmp", "dev", "proc"] - } - - # Use the MountMap context manager to ensure that any redirected - # mounts through fuse layers are in context and ready for bwrap - # to mount them from. - # - stack.enter_context(mount_map.mounted(self)) - - # If we're interactive, we want to inherit our stdin, - # otherwise redirect to /dev/null, ensuring process - # disconnected from terminal. - if flags & SandboxFlags.INTERACTIVE: - stdin = sys.stdin - else: - stdin = stack.enter_context(open(os.devnull, "r")) - - # Run bubblewrap ! - exit_code = self.run_bwrap( - bwrap_command, stdin, stdout, stderr, (flags & SandboxFlags.INTERACTIVE), pass_fds - ) - - # Cleanup things which bwrap might have left behind, while - # everything is still mounted because bwrap can be creating - # the devices on the fuse mount, so we should remove it there. - if not flags & SandboxFlags.INTERACTIVE: - for device in self.DEVICES: - device_path = os.path.join(root_mount_source, device.lstrip("/")) - - # This will remove the device in a loop, allowing some - # retries in case the device file leaked by bubblewrap is still busy - self.try_remove_device(device_path) - - # Remove /tmp, this is a bwrap owned thing we want to be sure - # never ends up in an artifact - for basedir in ["dev/shm", "tmp", "dev", "proc"]: - - # Skip removal of directories which already existed before - # launching bwrap - if existing_basedirs[basedir]: - continue - - base_directory = os.path.join(root_mount_source, basedir) - - if flags & SandboxFlags.INTERACTIVE: - # Be more lenient in interactive mode here. - # - # In interactive mode; it's possible that the project shell - # configuration has mounted some things below the base - # directories, such as /dev/dri, and in this case it's less - # important to consider cleanup, as we wont be collecting - # this build result and creating an artifact. - # - # Note: Ideally; we should instead fix upstream bubblewrap to - # cleanup any debris it creates at startup time, and do - # the same ourselves for any directories we explicitly create. - # - shutil.rmtree(base_directory, ignore_errors=True) - else: - try: - os.rmdir(base_directory) - except FileNotFoundError: - # ignore this, if bwrap cleaned up properly then it's not a problem. - # - # If the directory was not empty on the other hand, then this is clearly - # a bug, bwrap mounted a tempfs here and when it exits, that better be empty. - pass - - if self._json_status_available: - json_status_file.seek(0, 0) - child_exit_code = None - # The JSON status file's output is a JSON object per line - # with the keys present identifying the type of message. - # The only message relevant to us now is the exit-code of the subprocess. - for line in json_status_file: - with suppress(json.decoder.JSONDecodeError): - o = json.loads(line.decode()) - if isinstance(o, collections.abc.Mapping) and "exit-code" in o: - child_exit_code = o["exit-code"] - break - if child_exit_code is None: - raise SandboxError( - "`bwrap' terminated during sandbox setup with exitcode {}".format(exit_code), - reason="bwrap-sandbox-fail", - ) - exit_code = child_exit_code - - self._vdir._mark_changed() - return exit_code - - def run_bwrap(self, argv, stdin, stdout, stderr, interactive, pass_fds): - # Wrapper around subprocess.Popen() with common settings. - # - # This function blocks until the subprocess has terminated. - # - # It then returns a tuple of (exit code, stdout output, stderr output). - # If stdout was not equal to subprocess.PIPE, stdout will be None. Same for - # stderr. - - # Fetch the process actually launched inside the bwrap sandbox, or the - # intermediat control bwrap processes. - # - # NOTE: - # The main bwrap process itself is setuid root and as such we cannot - # send it any signals. Since we launch bwrap with --unshare-pid, it's - # direct child is another bwrap process which retains ownership of the - # pid namespace. This is the right process to kill when terminating. - # - # The grandchild is the binary which we asked bwrap to launch on our - # behalf, whatever this binary is, it is the right process to use - # for suspending and resuming. In the case that this is a shell, the - # shell will be group leader and all build scripts will stop/resume - # with that shell. - # - def get_user_proc(bwrap_pid, grand_child=False): - bwrap_proc = psutil.Process(bwrap_pid) - bwrap_children = bwrap_proc.children() - if bwrap_children: - if grand_child: - bwrap_grand_children = bwrap_children[0].children() - if bwrap_grand_children: - return bwrap_grand_children[0] - else: - return bwrap_children[0] - return None - - def terminate_bwrap(): - if process: - user_proc = get_user_proc(process.pid) - if user_proc: - user_proc.kill() - - def suspend_bwrap(): - if process: - user_proc = get_user_proc(process.pid, grand_child=True) - if user_proc: - group_id = os.getpgid(user_proc.pid) - os.killpg(group_id, signal.SIGSTOP) - - def resume_bwrap(): - if process: - user_proc = get_user_proc(process.pid, grand_child=True) - if user_proc: - group_id = os.getpgid(user_proc.pid) - os.killpg(group_id, signal.SIGCONT) - - with ExitStack() as stack: - - # We want to launch bwrap in a new session in non-interactive - # mode so that we handle the SIGTERM and SIGTSTP signals separately - # from the nested bwrap process, but in interactive mode this - # causes launched shells to lack job control (we dont really - # know why that is). - # - if interactive: - new_session = False - else: - new_session = True - stack.enter_context(_signals.suspendable(suspend_bwrap, resume_bwrap)) - stack.enter_context(_signals.terminator(terminate_bwrap)) - - process = subprocess.Popen( - argv, - # The default is to share file descriptors from the parent process - # to the subprocess, which is rarely good for sandboxing. - close_fds=True, - pass_fds=pass_fds, - stdin=stdin, - stdout=stdout, - stderr=stderr, - start_new_session=new_session, - ) - - # Wait for the child process to finish, ensuring that - # a SIGINT has exactly the effect the user probably - # expects (i.e. let the child process handle it). - try: - while True: - try: - _, status = os.waitpid(process.pid, 0) - # If the process exits due to a signal, we - # brutally murder it to avoid zombies - if not os.WIFEXITED(status): - user_proc = get_user_proc(process.pid) - if user_proc: - utils._kill_process_tree(user_proc.pid) - - # If we receive a KeyboardInterrupt we continue - # waiting for the process since we are in the same - # process group and it should also have received - # the SIGINT. - except KeyboardInterrupt: - continue - - break - # If we can't find the process, it has already died of its - # own accord, and therefore we don't need to check or kill - # anything. - except psutil.NoSuchProcess: - pass - - # Return the exit code - see the documentation for - # os.WEXITSTATUS to see why this is required. - if os.WIFEXITED(status): - exit_code = os.WEXITSTATUS(status) - else: - exit_code = -1 - - if interactive and stdin.isatty(): - # Make this process the foreground process again, otherwise the - # next read() on stdin will trigger SIGTTIN and stop the process. - # This is required because the sandboxed process does not have - # permission to do this on its own (running in separate PID namespace). - # - # tcsetpgrp() will trigger SIGTTOU when called from a background - # process, so ignore it temporarily. - handler = signal.signal(signal.SIGTTOU, signal.SIG_IGN) - os.tcsetpgrp(0, os.getpid()) - signal.signal(signal.SIGTTOU, handler) - - return exit_code - - def try_remove_device(self, device_path): - - # Put some upper limit on the tries here - max_tries = 1000 - tries = 0 - - while True: - try: - os.unlink(device_path) - except OSError as e: - if e.errno == errno.EBUSY: - # This happens on some machines, seems there is a race sometimes - # after bubblewrap returns and the device files it bind-mounted did - # not finish unmounting. - # - if tries < max_tries: - tries += 1 - time.sleep(1 / 100) - continue - - # We've reached the upper limit of tries, bail out now - # because something must have went wrong - # - raise - if e.errno == errno.ENOENT: - # Bubblewrap cleaned it up for us, no problem if we cant remove it - break - - # Something unexpected, reraise this error - raise - else: - # Successfully removed the symlink - break diff --git a/src/buildstream/sandbox/_sandboxreapi.py b/src/buildstream/sandbox/_sandboxreapi.py index 43d00a357..5c2851580 100644 --- a/src/buildstream/sandbox/_sandboxreapi.py +++ b/src/buildstream/sandbox/_sandboxreapi.py @@ -34,10 +34,6 @@ class SandboxREAPI(Sandbox): self._output_node_properties = kwargs.get("output_node_properties") - def _use_cas_based_directory(self): - # Always use CasBasedDirectory for REAPI - return True - def _run(self, command, flags, *, cwd, env): context = self._get_context() cascache = context.get_cascache() diff --git a/src/buildstream/sandbox/sandbox.py b/src/buildstream/sandbox/sandbox.py index fc8e4f1d0..592866dac 100644 --- a/src/buildstream/sandbox/sandbox.py +++ b/src/buildstream/sandbox/sandbox.py @@ -38,7 +38,6 @@ from typing import Dict, Generator, List, Optional, TYPE_CHECKING from .._exceptions import ImplError, BstError, SandboxError from .._message import Message, MessageType from ..storage.directory import Directory -from ..storage._filebaseddirectory import FileBasedDirectory from ..storage._casbaseddirectory import CasBasedDirectory if TYPE_CHECKING: @@ -139,14 +138,6 @@ class Sandbox: self.__stdout = kwargs["stdout"] self.__stderr = kwargs["stderr"] - # Setup the directories. Root and output_directory should be - # available to subclasses, hence being single-underscore. The - # others are private to this class. - self._root = os.path.join(directory, "root") - self.__scratch = os.path.join(directory, "scratch") - for directory_ in [self._root, self.__scratch]: - os.makedirs(directory_, exist_ok=True) - self._output_directory = None # type: Optional[str] self._build_directory = None self._build_directory_always = None @@ -167,11 +158,8 @@ class Sandbox: """ if self._vdir is None: - if self._use_cas_based_directory(): - cascache = self.__context.get_cascache() - self._vdir = CasBasedDirectory(cascache) - else: - self._vdir = FileBasedDirectory(self._root) + cascache = self.__context.get_cascache() + self._vdir = CasBasedDirectory(cascache) return self._vdir def set_environment(self, environment: Dict[str, str]) -> None: @@ -363,22 +351,6 @@ class Sandbox: def _create_batch(self, main_group, flags, *, collect=None): return _SandboxBatch(self, main_group, flags, collect=collect) - # _use_cas_based_directory() - # - # Whether to use CasBasedDirectory as sandbox root. If this returns `False`, - # FileBasedDirectory will be used. - # - # Returns: - # (bool): Whether to use CasBasedDirectory - # - def _use_cas_based_directory(self): - # Use CasBasedDirectory as sandbox root if Sandbox.run() is not used. - # This allows faster staging. - if not self.__allow_run: - return True - - return "BST_CAS_DIRECTORIES" in os.environ - # _fetch_missing_blobs() # # Fetch required file blobs missing from the local cache for sandboxes using @@ -482,20 +454,6 @@ class Sandbox: def _get_work_directory(self, *, cwd=None) -> str: return cwd or self.__cwd or "/" - # _get_scratch_directory() - # - # Fetches the sandbox scratch directory, this directory can - # be used by the sandbox implementation to cache things or - # redirect temporary fuse mounts. - # - # The scratch directory is guaranteed to be on the same - # filesystem as the root directory. - # - # Returns: - # (str): The sandbox scratch directory - def _get_scratch_directory(self): - return self.__scratch - # _get_output() # # Fetches the stdout & stderr diff --git a/src/buildstream/testing/_utils/site.py b/src/buildstream/testing/_utils/site.py index 9f0e7aa41..d8ace859b 100644 --- a/src/buildstream/testing/_utils/site.py +++ b/src/buildstream/testing/_utils/site.py @@ -7,7 +7,7 @@ import subprocess import sys from typing import Optional # pylint: disable=unused-import -from buildstream import _site, utils, ProgramNotFoundError +from buildstream import utils, ProgramNotFoundError from buildstream._platform import Platform @@ -45,14 +45,6 @@ except ProgramNotFoundError: BZR_ENV = {} try: - utils.get_host_tool("bwrap") - HAVE_BWRAP = True - HAVE_BWRAP_JSON_STATUS = _site.get_bwrap_version() >= (0, 3, 2) -except ProgramNotFoundError: - HAVE_BWRAP = False - HAVE_BWRAP_JSON_STATUS = False - -try: utils.get_host_tool("lzip") HAVE_LZIP = True except ProgramNotFoundError: @@ -62,19 +54,18 @@ casd_path = utils.get_host_tool("buildbox-casd") CASD_SEPARATE_USER = bool(os.stat(casd_path).st_mode & stat.S_ISUID) del casd_path -IS_LINUX = os.getenv("BST_FORCE_BACKEND", sys.platform).startswith("linux") +IS_LINUX = sys.platform.startswith("linux") IS_WINDOWS = os.name == "nt" MACHINE_ARCH = Platform.get_host_arch() -HAVE_SANDBOX = os.getenv("BST_FORCE_SANDBOX") - +HAVE_SANDBOX = None BUILDBOX_RUN = None -if HAVE_SANDBOX is None: - try: - path = utils.get_host_tool("buildbox-run") - subprocess.run([path, "--capabilities"], check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - BUILDBOX_RUN = os.path.basename(os.readlink(path)) - HAVE_SANDBOX = "buildbox-run" - except (ProgramNotFoundError, OSError, subprocess.CalledProcessError): - pass + +try: + path = utils.get_host_tool("buildbox-run") + subprocess.run([path, "--capabilities"], check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + BUILDBOX_RUN = os.path.basename(os.readlink(path)) + HAVE_SANDBOX = "buildbox-run" +except (ProgramNotFoundError, OSError, subprocess.CalledProcessError): + pass diff --git a/tests/integration/cachedfail.py b/tests/integration/cachedfail.py index da764bbd2..f72c315d5 100644 --- a/tests/integration/cachedfail.py +++ b/tests/integration/cachedfail.py @@ -182,7 +182,7 @@ def test_push_failed_missing_shell(cli, tmpdir, datafiles, on_error): assert share.get_artifact(cli.get_artifact_name(project, "test", "element.bst")) -@pytest.mark.skipif(HAVE_SANDBOX != "bwrap", reason="Only available with bubblewrap on Linux") +@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox") @pytest.mark.datafiles(DATA_DIR) def test_host_tools_errors_are_not_cached(cli, datafiles, tmp_path): # Create symlink to buildbox-casd to work with custom PATH @@ -202,11 +202,7 @@ def test_host_tools_errors_are_not_cached(cli, datafiles, tmp_path): _yaml.roundtrip_dump(element, element_path) # Build without access to host tools, this will fail - result1 = cli.run( - project=project, - args=["build", "element.bst"], - env={"PATH": str(tmp_path.joinpath("bin")), "BST_FORCE_SANDBOX": None}, - ) + result1 = cli.run(project=project, args=["build", "element.bst"], env={"PATH": str(tmp_path.joinpath("bin"))},) result1.assert_task_error(ErrorDomain.SANDBOX, "unavailable-local-sandbox") assert cli.get_element_state(project, "element.bst") == "buildable" diff --git a/tests/integration/project/elements/sandbox-bwrap/base-with-tmp.bst b/tests/integration/project/elements/sandbox-bwrap/base-with-tmp.bst deleted file mode 100644 index 5c9fa6083..000000000 --- a/tests/integration/project/elements/sandbox-bwrap/base-with-tmp.bst +++ /dev/null @@ -1,6 +0,0 @@ -kind: import -description: Base for after-sandbox cleanup test - -sources: - - kind: local - path: files/base-with-tmp/ diff --git a/tests/integration/project/elements/sandbox-bwrap/break-shell.bst b/tests/integration/project/elements/sandbox-bwrap/break-shell.bst deleted file mode 100644 index c93a92350..000000000 --- a/tests/integration/project/elements/sandbox-bwrap/break-shell.bst +++ /dev/null @@ -1,9 +0,0 @@ -kind: manual -depends: - - base/base-alpine.bst - -public: - bst: - integration-commands: - - | - chmod a-x /bin/sh diff --git a/tests/integration/project/elements/sandbox-bwrap/command-exit-42.bst b/tests/integration/project/elements/sandbox-bwrap/command-exit-42.bst deleted file mode 100644 index c633334ae..000000000 --- a/tests/integration/project/elements/sandbox-bwrap/command-exit-42.bst +++ /dev/null @@ -1,8 +0,0 @@ -kind: manual -depends: - - base/base-alpine.bst - -config: - build-commands: - - | - exit 42 diff --git a/tests/integration/project/elements/sandbox-bwrap/non-executable-shell.bst b/tests/integration/project/elements/sandbox-bwrap/non-executable-shell.bst deleted file mode 100644 index a57177bb3..000000000 --- a/tests/integration/project/elements/sandbox-bwrap/non-executable-shell.bst +++ /dev/null @@ -1,9 +0,0 @@ -kind: manual - -depends: - - sandbox-bwrap/break-shell.bst - -config: - build-commands: - - | - exit 42 diff --git a/tests/integration/project/elements/sandbox-bwrap/test-cleanup.bst b/tests/integration/project/elements/sandbox-bwrap/test-cleanup.bst deleted file mode 100644 index 2a89dd3ee..000000000 --- a/tests/integration/project/elements/sandbox-bwrap/test-cleanup.bst +++ /dev/null @@ -1,13 +0,0 @@ -kind: manual -description: A dummy project to utilize a base with existing /tmp folder. - -depends: - - filename: base.bst - type: build - - - filename: sandbox-bwrap/base-with-tmp.bst - -config: - build-commands: - - | - true diff --git a/tests/integration/sandbox-bwrap.py b/tests/integration/sandbox-bwrap.py deleted file mode 100644 index 3bf734edb..000000000 --- a/tests/integration/sandbox-bwrap.py +++ /dev/null @@ -1,57 +0,0 @@ -# Pylint doesn't play well with fixtures and dependency injection from pytest -# pylint: disable=redefined-outer-name - -import os -import pytest - -from buildstream.exceptions import ErrorDomain - -from buildstream.testing import cli_integration as cli # pylint: disable=unused-import -from buildstream.testing._utils.site import HAVE_SANDBOX, HAVE_BWRAP_JSON_STATUS - - -pytestmark = pytest.mark.integration - - -DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "project") - - -# Bubblewrap sandbox doesn't remove the dirs it created during its execution, -# so BuildStream tries to remove them to do good. BuildStream should be extra -# careful when those folders already exist and should not touch them, though. -@pytest.mark.skipif(HAVE_SANDBOX != "bwrap", reason="Only available with bubblewrap") -@pytest.mark.datafiles(DATA_DIR) -def test_sandbox_bwrap_cleanup_build(cli, datafiles): - project = str(datafiles) - # This element depends on a base image with non-empty `/tmp` folder. - element_name = "sandbox-bwrap/test-cleanup.bst" - - # Here, BuildStream should not attempt any rmdir etc. - result = cli.run(project=project, args=["build", element_name]) - assert result.exit_code == 0 - - -@pytest.mark.skipif(HAVE_SANDBOX != "bwrap", reason="Only available with bubblewrap") -@pytest.mark.skipif(not HAVE_BWRAP_JSON_STATUS, reason="Only available with bubblewrap supporting --json-status-fd") -@pytest.mark.datafiles(DATA_DIR) -def test_sandbox_bwrap_distinguish_setup_error(cli, datafiles): - project = str(datafiles) - element_name = "sandbox-bwrap/non-executable-shell.bst" - - result = cli.run(project=project, args=["build", element_name]) - result.assert_task_error(error_domain=ErrorDomain.SANDBOX, error_reason="bwrap-sandbox-fail") - - -@pytest.mark.skipif(HAVE_SANDBOX != "bwrap", reason="Only available with bubblewrap") -@pytest.mark.datafiles(DATA_DIR) -def test_sandbox_bwrap_return_subprocess(cli, datafiles): - project = str(datafiles) - element_name = "sandbox-bwrap/command-exit-42.bst" - - cli.configure( - {"logging": {"message-format": "%{element}|%{message}",},} - ) - - result = cli.run(project=project, args=["build", element_name]) - result.assert_task_error(error_domain=ErrorDomain.SANDBOX, error_reason="command-failed") - assert "sandbox-bwrap/command-exit-42.bst|Command failed with exitcode 42" in result.stderr diff --git a/tests/sandboxes/missing_dependencies.py b/tests/sandboxes/missing_dependencies.py index fb9fdafb9..7ac7d7868 100644 --- a/tests/sandboxes/missing_dependencies.py +++ b/tests/sandboxes/missing_dependencies.py @@ -41,8 +41,6 @@ def test_missing_buildbox_run_has_nice_error_message(cli, datafiles, tmp_path): _yaml.roundtrip_dump(element, element_path) # Build without access to host tools, this should fail with a nice error - result = cli.run( - project=project, args=["build", "element.bst"], env={"PATH": str(bin_dir), "BST_FORCE_SANDBOX": None} - ) + result = cli.run(project=project, args=["build", "element.bst"], env={"PATH": str(bin_dir)}) result.assert_task_error(ErrorDomain.SANDBOX, "unavailable-local-sandbox") assert "not found" in result.stderr diff --git a/tests/sandboxes/selection.py b/tests/sandboxes/selection.py index 3e6e1c4f5..0118fb5e0 100644 --- a/tests/sandboxes/selection.py +++ b/tests/sandboxes/selection.py @@ -30,30 +30,6 @@ DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "project") @pytest.mark.datafiles(DATA_DIR) -def test_force_sandbox(cli, datafiles): - project = str(datafiles) - element_path = os.path.join(project, "elements", "element.bst") - - # Write out our test target - element = { - "kind": "script", - "depends": [{"filename": "base.bst", "type": "build",},], - "config": {"commands": ["true",],}, - } - _yaml.roundtrip_dump(element, element_path) - - # Build without access to host tools, this will fail - result = cli.run( - project=project, args=["build", "element.bst"], env={"PATH": "", "BST_FORCE_SANDBOX": "buildbox-run"} - ) - result.assert_main_error(ErrorDomain.PLATFORM, None) - assert "buildbox-run not found" in result.stderr - # we have asked for a spesific sand box, but it is not avalble so - # bst should fail early and the element should be waiting - assert cli.get_element_state(project, "element.bst") == "waiting" - - -@pytest.mark.datafiles(DATA_DIR) def test_dummy_sandbox_fallback(cli, datafiles, tmp_path): # Create symlink to buildbox-casd to work with custom PATH buildbox_casd = tmp_path.joinpath("bin/buildbox-casd") @@ -72,11 +48,7 @@ def test_dummy_sandbox_fallback(cli, datafiles, tmp_path): _yaml.roundtrip_dump(element, element_path) # Build without access to host tools, this will fail - result = cli.run( - project=project, - args=["build", "element.bst"], - env={"PATH": str(tmp_path.joinpath("bin")), "BST_FORCE_SANDBOX": None}, - ) + result = cli.run(project=project, args=["build", "element.bst"], env={"PATH": str(tmp_path.joinpath("bin"))},) # But if we dont spesify a sandbox then we fall back to dummy, we still # fail early but only once we know we need a facny sandbox and that # dumy is not enough, there for element gets fetched and so is buildable @@ -50,8 +50,6 @@ deps = passenv = ARTIFACT_CACHE_SERVICE BST_CAS_STAGING_ROOT - BST_FORCE_BACKEND - BST_FORCE_SANDBOX BST_FORCE_START_METHOD GI_TYPELIB_PATH INTEGRATION_CACHE |