summaryrefslogtreecommitdiff
path: root/lib/ansible/parsing/dataloader.py
Commit message (Collapse)AuthorAgeFilesLines
* removed unused importBrian Coca2017-05-241-1/+0
|
* clarified error messagesBrian Coca2017-05-241-2/+2
|
* Transition inventory into plugins (#23001)Brian Coca2017-05-231-6/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * draft new inventory plugin arch, yaml sample - split classes, moved out of init - extra debug statements - allow mulitple invenotry files - dont add hosts more than once - simplified host vars - since now we can have multiple, inventory_dir/file needs to be per host - ported yaml/script/ini/virtualbox plugins, dir is 'built in manager' - centralized localhost handling - added plugin docs - leaner meaner inventory (split to data + manager) - moved noop vars plugin - added 'postprocessing' inventory plugins - fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them - grouphost_vars loading as inventory plugin (postprocessing) - playbook_dir allways full path - use bytes for file operations - better handling of empty/null sources - added test target that skips networking modules - now var manager loads play group/host_vars independant from inventory - centralized play setup repeat code - updated changelog with inv features - asperioribus verbis spatium album - fixed dataloader to new sig - made yaml plugin more resistant to bad data - nicer error msgs - fixed undeclared group detection - fixed 'ungrouping' - docs updated s/INI/file/ as its not only format - made behaviour of var merge a toggle - made 'source over group' path follow existing rule for var precedence - updated add_host/group from strategy - made host_list a plugin and added it to defaults - added advanced_host_list as example variation - refactored 'display' to be availbe by default in class inheritance - optimized implicit handling as per @pilou's feedback - removed unused code and tests - added inventory cache and vbox plugin now uses it - added _compose method for variable expressions in plugins - vbox plugin now uses 'compose' - require yaml extension for yaml - fix for plugin loader to always add original_path, even when not using all() - fix py3 issues - added --inventory as clearer option - return name when stringifying host objects - ajdust checks to code moving * reworked vars and vars precedence - vars plugins now load group/host_vars dirs - precedence for host vars is now configurable - vars_plugins been reworked - removed unused vars cache - removed _gathered_facts as we are not keeping info in host anymore - cleaned up tests - fixed ansible-pull to work with new inventory - removed version added notation to please rst check - inventory in config relative to config - ensures full paths on passed inventories * implicit localhost connection local
* ansible/parsing: PEP8 compliancy (#24701)Dag Wieers2017-05-181-5/+7
| | | - Make PEP8 compliant
* let user control "auto-de-vault" (#22739)Brian Coca2017-03-241-17/+18
| | | | | | * added option to toggle off autodevaulting * fixes per feedbazck
* Update module_utils.six to latest (#22855)Toshio Kuratomi2017-03-231-2/+2
| | | | | | | | * Update module_utils.six to latest We've been held back on the version of six we could use on the module side to 1.4.x because of python-2.4 compatibility. Now that our minimum is Python-2.6, we can update to the latest version of six in module_utils and get rid of the second copy in lib/ansible/compat.
* better 'role detection' for dwim (#22513)Brian Coca2017-03-151-10/+25
| | | | | | | | | | * better 'role detection' for dwim possible fix for #21735 * fixed unmatched paren * pep does not like the denting
* Retain vault password as bytes in 2.2 (#22378)Adrian Likins2017-03-071-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Retain vault password as bytes in 2.2 Prior to 2.2.1, the vault password was read in as byes and then remained bytes all the way through the code. A bug existed where bytes and text were mixed, leading to a traceback with non-ascii passwords. In devel, this was fixed by changing the read in password to text type to match with our overall strategy of converting at the borders. This was backported to stable-2.2 for the 2.2.1 release. On reflection, this should not have been backported as it causes passwords which were originally non-utf-8 to become utf-8. People will then have their working 2.2.x vault files become in-accessible. this commit pipes bytes all the way through the system for vault password. That way if a password is read in as a non-utf-8 character sequence, it will continue to work in 2.2.2+. This change is only for the 2.2 branch, not for 2.3 and beyond. Why not everywhere? The reason is that non-utf-8 passwords will cause problems when vault files are shared between systems or users. If the password is read from the prompt and one user/machine has a latin1 encoded locale while a second one has utf-8, the non-ascii password typed in won't match between machines. Deal with this by making sure that when we encrypt the data, we always use valid utf-8. Fixes #20398 (cherry picked from commit 5dcce0666a81917c68b76286685642fd72d84327)
* Fix PEP 8 issue.Matt Clay2017-02-241-1/+1
|
* prep for role passthrough instead of detectionBrian Coca2017-02-241-10/+12
| | | | (cherry picked from commit 2125a23f2ab0903065a13f9ee68ae43ed348d6fb)
* Fix regression in search path behaviourDag Wieers2016-12-141-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR fixes a few issues: - Missing role parent directory for relative paths - Fix integration tests (add missing stage) - Redesign integration tests - Incorrect order with tasks-lookups - Duplicate paths are listed - Repetitive tasks/tasks or files/files were possible ==== using copy with test.txt Before: ``` 491 1481281038.29393: search_path: /home/dag/home-made/ansible.testing/roles/test134/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/tasks/test.txt /home/dag/home-made/ansible.testing/files/test.txt /home/dag/home-made/ansible.testing/test.txt ``` After: ``` 32505 1481280963.22418: search_path: /home/dag/home-made/ansible.testing/roles/test134/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/test.txt /home/dag/home-made/ansible.testing/files/test.txt /home/dag/home-made/ansible.testing/test.txt ``` ==== Using copy with files/test.txt Before: ``` 31523 1481280499.63052: search_path: /home/dag/home-made/ansible.testing/roles/test134/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/tasks/files/test.txt /home/dag/home-made/ansible.testing/files/files/test.txt /home/dag/home-made/ansible.testing/files/test.txt ``` After: ``` 31110 1481280299.38778: search_path: /home/dag/home-made/ansible.testing/roles/test134/files/test.txt /home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt /home/dag/home-made/ansible.testing/files/test.txt ``` ==== Using template with files/test.txt.j2 Before: ``` 30074 1481280064.15191: search_path: /home/dag/home-made/ansible.testing/roles/test134/templates/files/test.txt.j2 /home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt.j2 /home/dag/home-made/ansible.testing/roles/test134/tasks/templates/files/test.txt.j2 /home/dag/home-made/ansible.testing/roles/test134/tasks/tasks/files/test.txt.j2 /home/dag/home-made/ansible.testing/templates/files/test.txt.j2 /home/dag/home-made/ansible.testing/files/test.txt.j2 ``` After: ``` 29201 1481279823.52752: search_path: /home/dag/home-made/ansible.testing/roles/test134/templates/files/test.txt.j2 /home/dag/home-made/ansible.testing/roles/test134/files/test.txt.j2 /home/dag/home-made/ansible.testing/roles/test134/tasks/templates/files/test.txt.j2 /home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt.j2 /home/dag/home-made/ansible.testing/templates/files/test.txt.j2 /home/dag/home-made/ansible.testing/files/test.txt.j2 ``` This fixes #19048
* Remove method that isn't used here. (#18713)Toshio Kuratomi2016-12-021-27/+0
| | | We have two copies of this function and only the one in cli is used.
* Fix bug (#18355) where encrypted inventories fail 18355 (#18373)Adrian Likins2016-11-071-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix bug (#18355) where encrypted inventories fail This is first part of fix for #18355 * Make DataLoader._get_file_contents return bytes The issue #18355 is caused by a change to inventory to stop using _get_file_contents so that it can handle text encoding itself to better protect against harmless text encoding errors in ini files (invalid unicode text in comment fields). So this makes _get_file_contents return bytes so it and other callers can handle the to_text(). The data returned by _get_file_contents() is now a bytes object instead of a text object. The callers of _get_file_contents() have been updated to call to_text() themselves on the results. Previously, the ini parser attempted to work around ini files that potentially include non-vailid unicode in comment lines. To do this, it stopped using DataLoader._get_file_contents() which does the decryption of files if vault encrypted. It didn't use that because _get_file_contents previously did to_text() on the read data itself. _get_file_contents() returns a bytestring now, so ini.py can call it and still special case ini file comments when converting to_text(). That also means encrypted inventory files are decrypted first. Fixes #18355
* Limit how much of the file we read to test if it's an encrypted vault fileToshio Kuratomi2016-11-041-2/+5
| | | | Fixes memory errors reported in #16391
* Fix search path for relative paths in roles (fixes #17877)Andrew Gaffney2016-10-171-1/+5
| | | | (cherry picked from commit 72f0aaf606aa3399c0713ad1eaac394c3846813c)
* Allow for empty source in dwim_path_relative_stackJames Cammarata2016-09-211-3/+3
| | | | Fixes #17192
* Fix dataloader using deprecated version of is_encrypted (#17615)Toshio Kuratomi2016-09-171-3/+3
|
* Move uses of to_bytes, to_text, to_native to use the module_utils version ↵Toshio Kuratomi2016-09-061-30/+31
| | | | | | | | (#17423) We couldn't copy to_unicode, to_bytes, to_str into module_utils because of licensing. So once created it we had two sets of functions that did the same things but had different implementations. To remedy that, this change removes the ansible.utils.unicode versions of those functions.
* Implement vault encrypted yaml variables. (#16274)Adrian Likins2016-08-231-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make !vault-encrypted create a AnsibleVaultUnicode yaml object that can be used as a regular string object. This allows a playbook to include a encrypted vault blob for the value of a yaml variable. A 'secret_password' variable can have it's value encrypted instead of having to vault encrypt an entire vars file. Add __ENCRYPTED__ to the vault yaml types so template.Template can treat it similar to __UNSAFE__ flags. vault.VaultLib api changes: - Split VaultLib.encrypt to encrypt and encrypt_bytestring - VaultLib.encrypt() previously accepted the plaintext data as either a byte string or a unicode string. Doing the right thing based on the input type would fail on py3 if given a arg of type 'bytes'. To simplify the API, vaultlib.encrypt() now assumes input plaintext is a py2 unicode or py3 str. It will encode to utf-8 then call the new encrypt_bytestring(). The new methods are less ambiguous. - moved VaultLib.is_encrypted logic to vault module scope and split to is_encrypted() and is_encrypted_file(). Add a test/unit/mock/yaml_helper.py It has some helpers for testing parsing/yaml Integration tests added as roles test_vault and test_vault_embedded
* Add some missing imports from last night's py3 fixes (#17196)Toshio Kuratomi2016-08-231-1/+1
|
* Move a path being passed around as a byte string to being passed around as a ↵Toshio Kuratomi2016-08-221-27/+39
| | | | | | | | text string. (#17190) This is enough to get minimal copy module working on python3 We have t omodify dataloader's path_dwim_relative_stack and everything that calls it to use text paths instead of byte string paths
* warn when searching for an empty string or nullBrian Coca2016-07-261-1/+3
|
* fixed memoryerror when coping huge file (#16392)nyasukun2016-07-221-2/+2
| | | | | | * fixed * support both python 2 and 3
* Search path (#16387)Brian Coca2016-06-281-2/+51
| | | | | | | | | | | | | | | | | | | | | * smarter function to figure out relative paths takes list of paths in order of relevance to current task and does the dwim magic on them * shared function for action plugins using new dwim unify path construction and error info/messaging made include and role non exclusive corrected order and now smarter about tasks includes inside roles are currently broken as they don't provide the correct role data make dirname full match to avoid corner cases * migrated action plugins to new dwim function reported plugins to use exceptions instead of info * clarified needle
* If decryption of a vaulted file failed, include the filename in the error. ↵jctanner2016-06-181-2/+2
| | | | | (#16329) Fixes #16327
* bytes when passing to os.path.* and opening the file, text at other timesToshio Kuratomi2016-04-291-2/+3
| | | | Fixes #15644
* Trnasform file name to bytes before opening it to avoid unicode errors if ↵Toshio Kuratomi2016-04-191-1/+1
| | | | python tries to encode it implicitly
* fixes to vault/copyBrian Coca2016-04-141-9/+10
| | | | | | | rm _del_ as it might leak memory renamed to tmp file cleanup added exception handling when traversing file list, even if one fails try rest added cleanup to finally to ensure removal in most cases
* feature/copy-vault-dataloader: Add method get_real_file(file_path) to dataloaderCambell2016-04-141-0/+71
| | | | | | | | - get_real_file will decrypt vault encrypted files and return a path to a temporary file. - cleanup_real_file will remove a temporary file created previously with get_real_file
* Use to_bytes on filenames in filesystem calls.Matt Clay2016-03-041-1/+1
|
* fix issues with older yaml lib versionsBrian Coca2016-03-031-4/+6
| | | | also added missing json import and removed unused ones
* restore initial json parsing attempt to loaderBrian Coca2016-03-031-16/+21
| | | | fixes issues with extra vars json strings not being parsed correctly by the yaml parser
* Transform pathnames to bytes before passing on to os.path functionsToshio Kuratomi2016-03-031-9/+9
|
* normalize path components to unicode before combining or operating on themToshio Kuratomi2016-02-261-3/+5
| | | | | | | Note that this will break if we deal with non-utf8 paths. Fixing this way because converting everythig to byte strings instead is a very invasive task so it should be done as a specific feature to provide support for non-utf8 paths at some point in the future (if needed).
* load now does not modify the incomming dataBrian Coca2016-01-281-24/+17
| | | | also removed json loader as yaml loader can do both
* Fix DataLoader's docstringMatteo Acerbi2015-11-181-3/+1
| | | DataLoader.__init__ doesn't take an argument named vault_password
* Break apart a looped dependency to show a warning when parsing playbooksToshio Kuratomi2015-10-271-0/+294
Display a warning when a dict key is overwritten by pyyaml Fixes #12888
View Lorry Controller Status