summaryrefslogtreecommitdiff
path: root/tests/unittests/test_util.py
Commit message (Collapse)AuthorAgeFilesLines
...
* cloudinit: remove unneeded __future__ imports (#362)Daniel Watkins2020-05-141-2/+0
| | | We live in the future now.
* BSD: find_devs_with_ refactoring (#298)Gonéri Le Bouder2020-04-241-0/+74
| | | | | | Refactoring of the `find_devs_with_*bsd()` methods: - centralize everything in `util.py` - add test coverage
* add Openbsd support (#147)Gonéri Le Bouder2020-03-261-0/+19
| | | | | - tested on OpenBSD 6.6 - tested on OpenStack without config drive, and NoCloud with ISO config drive
* cloudinit/tests: remove unneeded with_logs configuration (#263)Daniel Watkins2020-03-191-1/+0
| | | | | | | | | | | These classes don't use `self.logs` anywhere in their body, so we can remove the `with_logs = True` setting from them. These instances were found using astpath[0], with the following invocation: astpath "//Name[@id='with_logs' and not(ancestor::ClassDef//Attribute[@attr='logs'])]" [0] https://github.com/hchasestevens/astpath
* cloudinit: remove ImportError handling for mock imports (#182)Daniel Watkins2020-01-231-5/+1
| | | | We only run on Python 3 now, so we can unambiguously expect unittest.mock to exist.
* Drop most of the remaining use of six (#179)Daniel Watkins2020-01-211-9/+8
|
* net: Wait for dhclient to daemonize before reading lease fileJason Zions2019-01-151-0/+6
| | | | | | | | | | | | | | | | | cloud-init uses dhclient to fetch the DHCP lease so it can extract DHCP options. dhclient creates the leasefile, then writes to it; simply waiting for the leasefile to appear creates a race between dhclient and cloud-init. Instead, wait for dhclient to be parented by init. At that point, we know it has written to the leasefile, so it's safe to copy the file and kill the process. cloud-init creates a temporary directory in which to execute dhclient, and deletes that directory after it has killed the process. If cloud-init abandons waiting for dhclient to daemonize, it will still attempt to delete the temporary directory, but will not report an exception should that attempt fail. LP: #1794399
* tests: Disallow use of util.subp except for where needed.Scott Moser2018-09-051-12/+15
| | | | | | | | | | | | | | | | | | In many cases, cloud-init uses 'util.subp' to run a subprocess. This is not really desirable in our unit tests as it makes the tests dependent upon existance of those utilities. The change here is to modify the base test case class (CiTestCase) to raise exception any time subp is called. Then, fix all callers. For cases where subp is necessary or actually desired, we can use it via   a.) context hander CiTestCase.allow_subp(value)   b.) class level self.allowed_subp = value Both cases the value is a list of acceptable executable names that will be called (essentially argv[0]). Some cleanups in AltCloud were done as the code was being updated.
* openstack: avoid unneeded metadata probe on non-openstack platformsChad Smith2018-06-151-0/+23
| | | | | | | | | | | | | | | | OpenStack datasource is now discovered in init-local stage. In order to probe whether OpenStack metadata is present, it performs a costly sandboxed dhclient setup and metadata probe against http://169.254.169.254 for openstack data. Cloud-init properly detects non-OpenStack on EC2, but it spends precious time probing the metadata service also resulting in a confusing WARNING log about 'metadata not present'. To avoid the wasted cycles, and confusing warning, get_data will call a detect_openstack function to quickly determine whether the platform looks like OpenStack before trying to setup network to probe and crawl the metadata service. LP: #1776701
* Fix get_proc_env for pids that have non-utf8 content in environment.Scott Moser2018-06-131-0/+56
| | | | | | | | | | | | | There is no requirement that the environment of a process contains only utf-8 data. This modifies get_proc_env to support it reading data as binary and decoding if provided with an encoding. The default case is now that we now do: contents.decode('utf-8', 'replace') rather than contents.decode('utf-8', 'strict') LP: #1775371
* subp: support combine_capture argument.Scott Moser2018-06-081-0/+8
| | | | | | | | | | | | This adds 'combine_capture' argument as was present in curtin's subp. It is useful to get interleaved output of a command. I noticed a need for it when looking at user_data_rhevm in DataSourceAltCloud. That will run a subcommand, logging its stdout but swallowing its stderr. Another thing to change to use this would be in udevadm_settle which currently just returns the subp() call. Also, add the docstring copied from curtin's subp.
* yaml_load/schema: Add invalid line and column nums to error messageChad Smith2018-05-231-3/+27
| | | | | | | | | | Yaml tracebacks are generally hard to read for average users. Add a bit of logic to util.yaml_load and schema validation to look for YAMLError.context_marker or problem_marker line and column counts. No longer log the full exceeption traceback from the yaml_load error, instead just LOG.warning for the specific error and point to the offending line and column where the problem exists.
* pylint: pay attention to unused variable warnings.Scott Moser2018-04-191-3/+3
| | | | | This enables warnings produced by pylint for unused variables (W0612), and fixes the existing errors.
* net: Depend on iproute2's ip instead of net-tools ifconfig or routeChad Smith2018-04-181-6/+8
| | | | | | | | | | | | | | | | | | The net-tools package is deprecated and will eventually be dropped. Use "ip route", "link" or "address" instead of "ifconfig" or "route" calls. Cloud-init can now run in an environment that no longer has net-tools. This affects the network and route printing emitted to cloud-config-output.log as well as the cc_disable_ec2_metadata module. Additional changes:  - separate readResource and resourceLocation into standalone test    functions  - Fix ipv4 address rows to report scopes represented by ip addr show  - Formatted route/address ouput now handles multiple ipv4 and ipv6    addresses on a single interface Co-authored-by: James Hogarth <james.hogarth@gmail.com> Co-authored-by: Robert Schweikert <rjschwei@suse.com>
* pycodestyle: Fix invalid escape sequences in string literals.Scott Moser2018-04-181-1/+1
| | | | | | | | | | | | | Python has deprecated these invalid string literals now https://bugs.python.org/issue27364 and pycodestyle is identifying them with a W605 warning. https://github.com/PyCQA/pycodestyle/pull/676 So basically, any use of \ not followed by one of [\'"abfnrtv] or \ooo (octal) \xhh (hex) or a newline is invalid. This is most comomnly seen for us in regex. To solve, you either: a.) use a raw string r'...' b.) correctly escape the \ that was not intended to be interpreted.
* cc_resizefs, util: handle no /dev/zfsRyan Harper2018-04-021-2/+27
| | | | | | | The zfs/zpool commands will hang for 10 seconds if /dev/zfs is not present (bug 1760173). This is a common occurence for containers using zfs as rootfs. Additionally handle missing zpool command or other errors that may occur while executing the zpool command.
* FreeBSD: resizefs module now able to handle zfs/zpool.Dominic Schlegel2018-03-271-0/+50
| | | | | | | | Previously there was no support at all for zfs file system. With this change it is now possible to use the resizefs module to grow a zpool to its maximum partition size on FreeBSD. LP: #1721243
* cc_snap: Add new module to install and configure snapd and snap packages.Chad Smith2018-03-161-0/+33
| | | | | | | | | | | | | | Support installing and configuring snaps on ubuntu systems. Now, cloud-config files can provide a list or dictionary of snap:assertions which will be allow configuration of snapd on a system via 'snap ack' calls. The snap:commands configuration option supports arbitrary system commands intended to interact with snappy's cli. This allows users to run arbitrary snappy commands to create users, download, install and configure snap packages and snapd. This branch also deprecates old snappy and snap_config modules leaving warnings in documentation and runtime for consumers of these modules. Deprecated snap* modules will be dropped in cloud-init v.18.2 release.
* util: Fix subp regression. Allow specifying subp command as a string.Chad Smith2018-03-151-0/+18
| | | | | | | | The command provided to subp can either be a string or a list. This patch fixes a regression which raised CalledProcessError whenever providing a string to subp. LP: #1755965
* subp: Fix subp usage with non-ascii characters when no system locale.Scott Moser2018-02-231-0/+34
| | | | | | | | | | If python starts up without a locale set, then its default encoding ends up set as ascii. That is not easily changed with the likes of setlocale. In order to avoid UnicodeDecodeErrors cloud-init will encode to bytes a python3 string or python2 basestring so that the values passed to Popen are already bytes. LP: #1751051
* subp: make ProcessExecutionError have expected types in stderr, stdout.Scott Moser2018-01-231-0/+15
| | | | | | | | | | | | | | | | When subp raised a ProcessExecutionError, that exception's stderr and stdout might end up being the string '-' rather than bytes. This mean that:    try:        subp(mycommand, decode=False)    except ProcessExecutionError as e:        pass Would have 'e.stdout' set to '-' while the caller would expect bytes. Also reduce the try/except block in subp to a specifically the two lines that may raise an OSError.
* Do not log warning on config files that represent None.Scott Moser2018-01-121-0/+8
| | | | | | | | | | | | | | | | This issue was first identified when manual_cache_clean was set, as ds-identify would write /run/cloud-init/cloud.cfg with # manual_cache_clean that would generate a warning as cloud-init expected to load a dict. Any other "empty" config would also log such a warning. Also fix reading of di_report to allow it to be None, as ds-identify would write: di_report: # manual_cache_clean which reads as 'di_report: None' rather than di_report: {}. LP: #1742479
* pylint: Update pylint to 1.7.1, run on tests/ and tools and fix complaints.Ryan Harper2017-12-071-3/+3
| | | | | | | | | | | | | | | | | | | | | The motivation for this is that a.) 1.7.1 runs with python 3.6 (bionic) b.) we want to run pylint on tests/ and tools for the same reasons that we want to run it on cloudinit/ The changes are described below. - Update tox.ini to invoke pylint v1.7.1. - Modify .pylintrc generated-members ignore mocked object members (m_.*) - Replace "dangerous" params defaulting to {} - Fix up cloud_tests use of platforms - Cast some instance objects to with dict() - Handle python2.7 vs 3+ ConfigParser use of readfp (deprecated) - Update use of assertEqual(<boolean>, value) to assert<Boolean>(value) - replace depricated assertRegexp -> assertRegex - Remove useless test-class calls to super class - Assign class property accessors a result and use it - Fix missing class member in CepkoResultTests - Fix Cheetah test import
* cli: Add clean and status subcommandsChad Smith2017-12-051-0/+38
| | | | | | | | | | | | | The 'cloud-init clean' command allows a user or script to clear cloud-init artifacts from the system so that cloud-init sees the system as unconfigured upon reboot. Optional parameters can be provided to remove cloud-init logs and reboot after clean. The 'cloud-init status' command allows the user or script to check whether cloud-init has finished all configuration stages and whether errors occurred. An optional --wait argument will poll on a 0.25 second interval until cloud-init configuration is complete. The benefit here is scripts can block on cloud-init completion before performing post-config tasks.
* relocate tests/unittests/helpers.py to cloudinit/testsLars Kellogg-Stedman2017-09-051-1/+1
| | | | | This moves the base test case classes into into cloudinit/tests and updates all the corresponding imports.
* Log a helpful message if a user script does not include shebang.Andrew Jorgensen2017-08-251-1/+12
| | | | | | A patch to allow scripts missing a #! to run by using shell=True was proposed but rejected. Instead we emit a log message to help the user understand what went wrong.
* Support comments in content read by load_shell_content.Scott Moser2017-07-171-0/+16
| | | | | | | load_shell_content previously would not allow shell comment characters in the content being parsed. If comments=True is not passed then an exception would previously be raised as the line would not be guaranteed to have an '=' in it.
* read_dmi_data: always return None when inside a container.Scott Moser2017-06-291-0/+23
| | | | | | | | | | | | | | | | | | | | | | This fixes stacktrace and warning message that would be printed to the log if running inside a container and read_dmi_data tried to access a key that was not present. In a container, the /sys/class/dmi/id data is not relevant to the but to the host. Additionally an unpriviledged container might see strange behavior: # cd /sys/class/dmi/id/ # id -u 0 # ls -l chassis_serial -r-------- 1 nobody nogroup 4096 Jun 29 16:49 chassis_serial # cat chassis_serial cat: /sys/class/dmi/id/chassis_serial: Permission denied The solution here is to just always return None when running in a container. LP: #1701325
* FreeBSD: Make freebsd a variant, fix unittests and tools/build-on-freebsd.Scott Moser2017-06-151-3/+6
| | | | | | | | | | | | - Simplify the logic of 'variant' in util.system_info much of the data from https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version - fix get_resource_disk_on_freebsd when running on a system without an Azure resource disk. - fix tools/build-on-freebsd to replace oauth with oauthlib and add bash which is a dependency for tests. - update a fiew places that were checking for freebsd but not using the util.is_FreeBSD()
* flake8: move the pinned version of flake8 up to 3.3.0Scott Moser2017-05-231-5/+5
| | | | | | | | | | | | | | | | This just moves flake8 and related tools up to newer versions and fixes the complaints associated with that. We added to the list of flake8 ignores: H102: do not put vim info in source files H304: no relative imports Also updates and pins the following in the flake8 environment: pep8: 1.7.0 => drop (although hacking still pulls it in). pyflakes 1.1.0 => 1.5.0 hacking 0.10.2 => 0.13.0 flake8 2.5.4 => 3.3.0 pycodestyle none => 2.3.1
* Improve detection of snappy to include os-release and kernel cmdline.Scott Moser2017-05-161-0/+69
| | | | | | | | | | Recent core snap images (edge channel revision 1886) do not contain the previously known files used to detect that a system is ubuntu core. The changes here are to look in 2 additional locations to determine if a system is snappy. LP: #1689944
* FreeBSD: improvements and fixes for use on AzureHongjiang Zhang2017-05-101-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | This patch targets to make FreeBSD 10.3 or 11 work on Azure. The modifications abide by the rule of: * making as less modification as possible * delegate to the distro or datasource where possible. The main modifications are: 1. network configuration improvements, and movement into distro path. 2. Fix setting of password. Password setting through "pw" can only work through pipe. 3. Add 'root:wheel' to syslog_fix_perms field. 4. Support resizing default file system (ufs) 5. copy cloud.cfg for freebsd to /etc/cloud/cloud.cfg rather than /usr/local/etc/cloud/cloud.cfg. 6. Azure specific changes: a. When reading the azure endpoint, search in a different path and read a different option name (option-245 vs. unknown-245). so, the lease file path should be generated according to platform. b. adjust the handling of ephemeral mounts for ufs filesystem and for finding the ephemeral device. c. fix mounting of cdrom LP: #1636345
* util: teach write_file about copy_mode optionLars Kellogg-Stedman2017-04-121-2/+31
| | | | | | | | | | | | | On centos/fedora/rhel/derivatives, /etc/ssh/sshd_config has mode 0600, but cloud-init unilaterally sets file modes to 0644 when no explicit mode is passed to util.write_file. On ubuntu/debian, this file has mode 0644. With this patch, write_file learns about the copy_mode option, which will cause it to use the mode of the existing file by default, falling back to the explicit mode parameter if the file does not exist. LP: #1644064 Resolves: rhbz#1295984
* LICENSE: Allow dual licensing GPL-3 or Apache 2.0Jon Grimm2016-12-221-0/+2
| | | | | | | | | | | | | | | | | | This has been a recurring ask and we had initially just made the change to the cloud-init 2.0 codebase. As the current thinking is we'll just continue to enhance the current codebase, its desirable to relicense to match what we'd intended as part of the 2.0 plan here. - put a brief description of license in LICENSE file - put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0 - simplify the per-file header to reference LICENSE - tox: ignore H102 (Apache License Header check) Add license header to files that ship. Reformat headers, make sure everything has vi: at end of file. Non-shipping files do not need the copyright header, but at the moment tests/ have it.
* fix problems found in python2.6 test.Joshua Harlow2016-12-021-1/+1
| | | | | These are just simple syntax fixes to work correctly on python2.6. Found when testing in a centos 6 container.
* Improve formatting for ProcessExecutionErrorWesley Wiedenmeier2016-11-221-0/+69
| | | | | | | This replaces long single lines in a log or console output with multiple lines that are much easier to read. It indents the stdout and stderr so logs are more easily read also.
* dmidecode: Allow dmidecode to be used on aarch64Robert Schweikert2016-11-011-1/+1
| | | | | aarch64 systems have functional dmidecode, so allow that to be used. - aarch64 has support for dmidecode as well
* Fix python2.6 things found running in centos 6.Scott Moser2016-10-191-1/+1
| | | | | | | | | | | | | | This gets the tests running in centos 6. * ProcessExecutionError: remove setting of .message Nothing in cloud-init seems to use .message anywhere, so it does not seem necessary. The reason to change it is that on 2.6 it spits out: cloudinit/util.py:286: DeprecationWarning: BaseException.message * tox.ini: add a centos6 environment the tox versions listed here replicate a centos6 install with packages from EPEL. You will still need a python2.6 to run this env so we do not enable it by default.
* Decode unicode types in decode_binaryRobert Schweikert2016-09-211-0/+8
| | | | | | | | | The test in decode_binary for six.text_type was incorrect as that includes unicode type in Python 2 which should actually be decoded. When the type is string_types we now properly check only for basestring and str in Python 2 and Python 3 respectively and return the given blob without making an attempt to decode.
* subp: add 'update_env' argumentScott Moser2016-09-211-2/+28
| | | | | | | | | | | | | | In order for a caller to use 'env' argument of subp, they will realistically do: env = os.environ.copy() env['FOO'] = 'BZR' subp(cmd, env=env) This shortens that to be: subp(cmd, update_env={'FOO': 'BZR'}) Add tests, and update growpart tests to use mock when playing with os.environ.
* Apt: add new apt configuration formatChristian Ehrhardt2016-08-181-0/+69
| | | | | | | | | | | | | | This adds an improved apt configuration format that is fully backwards compatible with previous behavior. This is mostly copied from curtin's implementation. It does: * clean up and centralizes many of the top level 'apt_*' values that previously existed into a single top level 'apt'key. * support a 'source' in apt/sources/entry that has only a key * documents new features and adds tests. See the added doc/examples/cloud-config-apt.txt for more information.
* DigitalOcean: use the v1.json endpointBen Howard2016-08-121-2/+24
| | | | | | | | | | | | | | | | Per [1], DigitalOcean provides the metadata in multiple formats. The JSON document is the preferred endpoint. Changes: - Switch to the v1.json meta-data endpoint - Identify droplet identity from SMBIOS - Only poll for metadata when the instance is confirmed to be a droplet - Removal of hard-coded mirrors Additionally, centralize the gates on running 'dmidecode' on arm arches, and update tests to address. [1] https://developers.digitalocean.com/documentation/metadata/
* change return value for dmi data of all \xff to be ""Scott Moser2016-03-141-1/+1
|\ | | | | | | | | | | | | | | Previously we returned a string of "." the same length as the dmi field. That seems confusing to the user as "." would seem like a valid response when in fact this value should not be considered valid. So now, in this case, return empty string.
| * change return value for dmi data of all \xff to be ""Scott Moser2016-03-141-1/+1
| | | | | | | | | | | | | | | | Previously we returned a string of "." the same length as the dmi field. That seems confusing to the user as "." would seem like a valid response when in fact this value should not be considered valid. So now, in this case, return empty string.
* | dmi data: fix failure of reading dmi data for unset dmi valuesScott Moser2016-03-101-0/+10
|\ \ | |/ | | | | | | | | | | | | | | it is not uncommon to find dmi data in /sys full of 'ff'. utf-8 decoding of those would fail, causing warning and stacktrace. Return '.' instead of \xff. This is what dmidecode would return. $ dmidecode --string system-product-name
| * dmi data: fix failure of reading dmi data for unset dmi valuesScott Moser2016-03-101-0/+9
|/ | | | | | | | | | it is not uncommon to find dmi data in /sys full of 'ff'. utf-8 decoding of those would fail, causing warning and stacktrace. Return '.' instead of \xff. This maps to what dmidecode would return $ dmidecode --string system-product-name .................................
* read_seeded: fix reed_seeded after regressionScott Moser2015-05-141-0/+17
| | | | | | | | | read_seeded was assuming a Response object back from load_tfile_or_url but load_tfile_or_url was returning string. since the only other user of this was a test, move load_tfile_or_url to a test, and just do the right thing in read_seeded. LP: #1455233
* Add util.message_from_string to wrap email.message_from_string.Daniel Watkins2015-03-041-0/+7
| | | | | This is to work-around the fact that email.message_from_string uses cStringIO in Python 2.6, which can't handle Unicode.
* Convert dmidecode values to sysfs names before looking for them.Daniel Watkins2015-03-041-41/+50
| | | | | | dmidecode and /sys/class/dmi/id/* use different names for the same information. This modified the logic in util.read_dmi_data to map from dmidecode names to sysfs names before looking in sysfs.
* tests/unittests/test_util.py: pep8Scott Moser2015-02-131-1/+0
|
View Lorry Controller Status