diff options
| author | Adam Coldrick <adam.coldrick@codethink.co.uk> | 2015-03-30 12:16:50 +0000 |
|---|---|---|
| committer | Morph (on behalf of Adam Coldrick) <adam.coldrick@codethink.co.uk> | 2015-03-30 12:16:50 +0000 |
| commit | ac44d4f5994dbc1d579dfb7e95c8672bf01caa5b (patch) | |
| tree | 9b4054c06ee0dcd38da78bd9e78a5aa7292464b3 | |
| parent | aa047d1b4ea195c1a5a70568a2b75f958f47fa99 (diff) | |
| download | morph-ac44d4f5994dbc1d579dfb7e95c8672bf01caa5b.tar.gz | |
Morph build c2397bf4873742a3ba9202cbdcfffea8
System branch: master
50 files changed, 826 insertions, 947 deletions
diff --git a/COPYING b/COPYING deleted file mode 100644 index d159169d..00000000 --- a/COPYING +++ /dev/null @@ -1,339 +0,0 @@ - GNU GENERAL PUBLIC LICENSE - Version 2, June 1991 - - Copyright (C) 1989, 1991 Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA - Everyone is permitted to copy and distribute verbatim copies - of this license document, but changing it is not allowed. - - Preamble - - The licenses for most software are designed to take away your -freedom to share and change it. By contrast, the GNU General Public -License is intended to guarantee your freedom to share and change free -software--to make sure the software is free for all its users. This -General Public License applies to most of the Free Software -Foundation's software and to any other program whose authors commit to -using it. (Some other Free Software Foundation software is covered by -the GNU Lesser General Public License instead.) You can apply it to -your programs, too. - - When we speak of free software, we are referring to freedom, not -price. Our General Public Licenses are designed to make sure that you -have the freedom to distribute copies of free software (and charge for -this service if you wish), that you receive source code or can get it -if you want it, that you can change the software or use pieces of it -in new free programs; and that you know you can do these things. - - To protect your rights, we need to make restrictions that forbid -anyone to deny you these rights or to ask you to surrender the rights. -These restrictions translate to certain responsibilities for you if you -distribute copies of the software, or if you modify it. - - For example, if you distribute copies of such a program, whether -gratis or for a fee, you must give the recipients all the rights that -you have. You must make sure that they, too, receive or can get the -source code. And you must show them these terms so they know their -rights. - - We protect your rights with two steps: (1) copyright the software, and -(2) offer you this license which gives you legal permission to copy, -distribute and/or modify the software. - - Also, for each author's protection and ours, we want to make certain -that everyone understands that there is no warranty for this free -software. If the software is modified by someone else and passed on, we -want its recipients to know that what they have is not the original, so -that any problems introduced by others will not reflect on the original -authors' reputations. - - Finally, any free program is threatened constantly by software -patents. We wish to avoid the danger that redistributors of a free -program will individually obtain patent licenses, in effect making the -program proprietary. To prevent this, we have made it clear that any -patent must be licensed for everyone's free use or not licensed at all. - - The precise terms and conditions for copying, distribution and -modification follow. - - GNU GENERAL PUBLIC LICENSE - TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION - - 0. This License applies to any program or other work which contains -a notice placed by the copyright holder saying it may be distributed -under the terms of this General Public License. The "Program", below, -refers to any such program or work, and a "work based on the Program" -means either the Program or any derivative work under copyright law: -that is to say, a work containing the Program or a portion of it, -either verbatim or with modifications and/or translated into another -language. (Hereinafter, translation is included without limitation in -the term "modification".) Each licensee is addressed as "you". - -Activities other than copying, distribution and modification are not -covered by this License; they are outside its scope. The act of -running the Program is not restricted, and the output from the Program -is covered only if its contents constitute a work based on the -Program (independent of having been made by running the Program). -Whether that is true depends on what the Program does. - - 1. You may copy and distribute verbatim copies of the Program's -source code as you receive it, in any medium, provided that you -conspicuously and appropriately publish on each copy an appropriate -copyright notice and disclaimer of warranty; keep intact all the -notices that refer to this License and to the absence of any warranty; -and give any other recipients of the Program a copy of this License -along with the Program. - -You may charge a fee for the physical act of transferring a copy, and -you may at your option offer warranty protection in exchange for a fee. - - 2. You may modify your copy or copies of the Program or any portion -of it, thus forming a work based on the Program, and copy and -distribute such modifications or work under the terms of Section 1 -above, provided that you also meet all of these conditions: - - a) You must cause the modified files to carry prominent notices - stating that you changed the files and the date of any change. - - b) You must cause any work that you distribute or publish, that in - whole or in part contains or is derived from the Program or any - part thereof, to be licensed as a whole at no charge to all third - parties under the terms of this License. - - c) If the modified program normally reads commands interactively - when run, you must cause it, when started running for such - interactive use in the most ordinary way, to print or display an - announcement including an appropriate copyright notice and a - notice that there is no warranty (or else, saying that you provide - a warranty) and that users may redistribute the program under - these conditions, and telling the user how to view a copy of this - License. (Exception: if the Program itself is interactive but - does not normally print such an announcement, your work based on - the Program is not required to print an announcement.) - -These requirements apply to the modified work as a whole. If -identifiable sections of that work are not derived from the Program, -and can be reasonably considered independent and separate works in -themselves, then this License, and its terms, do not apply to those -sections when you distribute them as separate works. But when you -distribute the same sections as part of a whole which is a work based -on the Program, the distribution of the whole must be on the terms of -this License, whose permissions for other licensees extend to the -entire whole, and thus to each and every part regardless of who wrote it. - -Thus, it is not the intent of this section to claim rights or contest -your rights to work written entirely by you; rather, the intent is to -exercise the right to control the distribution of derivative or -collective works based on the Program. - -In addition, mere aggregation of another work not based on the Program -with the Program (or with a work based on the Program) on a volume of -a storage or distribution medium does not bring the other work under -the scope of this License. - - 3. You may copy and distribute the Program (or a work based on it, -under Section 2) in object code or executable form under the terms of -Sections 1 and 2 above provided that you also do one of the following: - - a) Accompany it with the complete corresponding machine-readable - source code, which must be distributed under the terms of Sections - 1 and 2 above on a medium customarily used for software interchange; or, - - b) Accompany it with a written offer, valid for at least three - years, to give any third party, for a charge no more than your - cost of physically performing source distribution, a complete - machine-readable copy of the corresponding source code, to be - distributed under the terms of Sections 1 and 2 above on a medium - customarily used for software interchange; or, - - c) Accompany it with the information you received as to the offer - to distribute corresponding source code. (This alternative is - allowed only for noncommercial distribution and only if you - received the program in object code or executable form with such - an offer, in accord with Subsection b above.) - -The source code for a work means the preferred form of the work for -making modifications to it. For an executable work, complete source -code means all the source code for all modules it contains, plus any -associated interface definition files, plus the scripts used to -control compilation and installation of the executable. However, as a -special exception, the source code distributed need not include -anything that is normally distributed (in either source or binary -form) with the major components (compiler, kernel, and so on) of the -operating system on which the executable runs, unless that component -itself accompanies the executable. - -If distribution of executable or object code is made by offering -access to copy from a designated place, then offering equivalent -access to copy the source code from the same place counts as -distribution of the source code, even though third parties are not -compelled to copy the source along with the object code. - - 4. You may not copy, modify, sublicense, or distribute the Program -except as expressly provided under this License. Any attempt -otherwise to copy, modify, sublicense or distribute the Program is -void, and will automatically terminate your rights under this License. -However, parties who have received copies, or rights, from you under -this License will not have their licenses terminated so long as such -parties remain in full compliance. - - 5. You are not required to accept this License, since you have not -signed it. However, nothing else grants you permission to modify or -distribute the Program or its derivative works. These actions are -prohibited by law if you do not accept this License. Therefore, by -modifying or distributing the Program (or any work based on the -Program), you indicate your acceptance of this License to do so, and -all its terms and conditions for copying, distributing or modifying -the Program or works based on it. - - 6. Each time you redistribute the Program (or any work based on the -Program), the recipient automatically receives a license from the -original licensor to copy, distribute or modify the Program subject to -these terms and conditions. You may not impose any further -restrictions on the recipients' exercise of the rights granted herein. -You are not responsible for enforcing compliance by third parties to -this License. - - 7. If, as a consequence of a court judgment or allegation of patent -infringement or for any other reason (not limited to patent issues), -conditions are imposed on you (whether by court order, agreement or -otherwise) that contradict the conditions of this License, they do not -excuse you from the conditions of this License. If you cannot -distribute so as to satisfy simultaneously your obligations under this -License and any other pertinent obligations, then as a consequence you -may not distribute the Program at all. For example, if a patent -license would not permit royalty-free redistribution of the Program by -all those who receive copies directly or indirectly through you, then -the only way you could satisfy both it and this License would be to -refrain entirely from distribution of the Program. - -If any portion of this section is held invalid or unenforceable under -any particular circumstance, the balance of the section is intended to -apply and the section as a whole is intended to apply in other -circumstances. - -It is not the purpose of this section to induce you to infringe any -patents or other property right claims or to contest validity of any -such claims; this section has the sole purpose of protecting the -integrity of the free software distribution system, which is -implemented by public license practices. Many people have made -generous contributions to the wide range of software distributed -through that system in reliance on consistent application of that -system; it is up to the author/donor to decide if he or she is willing -to distribute software through any other system and a licensee cannot -impose that choice. - -This section is intended to make thoroughly clear what is believed to -be a consequence of the rest of this License. - - 8. If the distribution and/or use of the Program is restricted in -certain countries either by patents or by copyrighted interfaces, the -original copyright holder who places the Program under this License -may add an explicit geographical distribution limitation excluding -those countries, so that distribution is permitted only in or among -countries not thus excluded. In such case, this License incorporates -the limitation as if written in the body of this License. - - 9. The Free Software Foundation may publish revised and/or new versions -of the General Public License from time to time. Such new versions will -be similar in spirit to the present version, but may differ in detail to -address new problems or concerns. - -Each version is given a distinguishing version number. If the Program -specifies a version number of this License which applies to it and "any -later version", you have the option of following the terms and conditions -either of that version or of any later version published by the Free -Software Foundation. If the Program does not specify a version number of -this License, you may choose any version ever published by the Free Software -Foundation. - - 10. If you wish to incorporate parts of the Program into other free -programs whose distribution conditions are different, write to the author -to ask for permission. For software which is copyrighted by the Free -Software Foundation, write to the Free Software Foundation; we sometimes -make exceptions for this. Our decision will be guided by the two goals -of preserving the free status of all derivatives of our free software and -of promoting the sharing and reuse of software generally. - - NO WARRANTY - - 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY -FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN -OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES -PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED -OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF -MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS -TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE -PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, -REPAIR OR CORRECTION. - - 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING -WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR -REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, -INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING -OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED -TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY -YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER -PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE -POSSIBILITY OF SUCH DAMAGES. - - END OF TERMS AND CONDITIONS - - How to Apply These Terms to Your New Programs - - If you develop a new program, and you want it to be of the greatest -possible use to the public, the best way to achieve this is to make it -free software which everyone can redistribute and change under these terms. - - To do so, attach the following notices to the program. It is safest -to attach them to the start of each source file to most effectively -convey the exclusion of warranty; and each file should have at least -the "copyright" line and a pointer to where the full notice is found. - - <one line to give the program's name and a brief idea of what it does.> - Copyright (C) <year> <name of author> - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License along - with this program; if not, write to the Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -Also add information on how to contact you by electronic and paper mail. - -If the program is interactive, make it output a short notice like this -when it starts in an interactive mode: - - Gnomovision version 69, Copyright (C) year name of author - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. - This is free software, and you are welcome to redistribute it - under certain conditions; type `show c' for details. - -The hypothetical commands `show w' and `show c' should show the appropriate -parts of the General Public License. Of course, the commands you use may -be called something other than `show w' and `show c'; they could even be -mouse-clicks or menu items--whatever suits your program. - -You should also get your employer (if you work as a programmer) or your -school, if any, to sign a "copyright disclaimer" for the program, if -necessary. Here is a sample; alter the names: - - Yoyodyne, Inc., hereby disclaims all copyright interest in the program - `Gnomovision' (which makes passes at compilers) written by James Hacker. - - <signature of Ty Coon>, 1 April 1989 - Ty Coon, President of Vice - -This General Public License does not permit incorporating your program into -proprietary programs. If your program is a subroutine library, you may -consider it more useful to permit linking proprietary applications with the -library. If this is what you want to do, use the GNU Lesser General -Public License instead of this License. diff --git a/morph-cache-server b/morph-cache-server index 007cfbe8..6c7665aa 100755 --- a/morph-cache-server +++ b/morph-cache-server @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Copyright (C) 2013, 2014-2015 Codethink Limited +# Copyright (C) 2013-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -27,6 +27,9 @@ import shutil from bottle import Bottle, request, response, run, static_file from flup.server.fcgi import WSGIServer from morphcacheserver.repocache import RepoCache +from morphlib.artifactcachereference import ArtifactCacheReference +from morphlib.ostreeartifactcache import OSTreeArtifactCache +from morphlib.remoteartifactcache import RemoteArtifactCache defaults = { @@ -34,6 +37,7 @@ defaults = { 'bundle-dir': '/var/cache/morph-cache-server/bundles', 'artifact-dir': '/var/cache/morph-cache-server/artifacts', 'port': 8080, + 'ostree-port': 12324, } @@ -44,6 +48,10 @@ class MorphCacheServer(cliapp.Application): 'port to listen on', metavar='PORTNUM', default=defaults['port']) + self.settings.integer(['ostree-port'], + 'port for accessing the ostree repo for ' + 'the artifact cache', + default=defaults['ostree-port']) self.settings.string(['port-file'], 'write port number to FILE', metavar='FILE', @@ -68,50 +76,19 @@ class MorphCacheServer(cliapp.Application): 'runs a fcgi-server', default=True) - - def _fetch_artifact(self, url, filename): - in_fh = None - try: - in_fh = urllib2.urlopen(url) - with open(filename, "w") as localtmp: - shutil.copyfileobj(in_fh, localtmp) - in_fh.close() - except Exception, e: - if in_fh is not None: - in_fh.close() - raise - else: - if in_fh is not None: - in_fh.close() - return os.stat(filename) - def _fetch_artifacts(self, server, cacheid, artifacts): ret = {} + cache = OSTreeArtifactCache(self.settings['artifact-dir']) + remote = RemoteArtifactCache('http://%s/' % server) try: for artifact in artifacts: - artifact_name = "%s.%s" % (cacheid, artifact) - tmpname = os.path.join(self.settings['artifact-dir'], - ".dl.%s" % artifact_name) - url = "http://%s/1.0/artifacts?filename=%s" % ( - server, urllib.quote(artifact_name)) - stinfo = self._fetch_artifact(url, tmpname) - ret[artifact_name] = { - "size": stinfo.st_size, - "used": stinfo.st_blocks * 512, - } + logging.debug('%s.%s' % (cacheid, artifact)) + cache_artifact = ArtifactCacheReference( + '.'.join((cacheid, artifact))) + cache.copy_from_remote(cache_artifact, remote) except Exception, e: - for artifact in ret.iterkeys(): - os.unlink(os.path.join(self.settings['artifact-dir'], - ".dl.%s" % artifact)) + logging.debug('OSTree raised an Exception: %s' % e) raise - - for artifact in ret.iterkeys(): - tmpname = os.path.join(self.settings['artifact-dir'], - ".dl.%s" % artifact) - artifilename = os.path.join(self.settings['artifact-dir'], - artifact) - os.rename(tmpname, artifilename) - return ret @@ -172,7 +149,6 @@ class MorphCacheServer(cliapp.Application): response.set_header('Cache-Control', 'no-cache') artifacts = artifacts.split(",") return self._fetch_artifacts(host, cacheid, artifacts) - except Exception, e: response.status = 500 logging.debug('%s' % e) @@ -298,11 +274,37 @@ class MorphCacheServer(cliapp.Application): @app.get('/artifacts') def artifact(): basename = self._unescape_parameter(request.query.filename) - filename = os.path.join(self.settings['artifact-dir'], basename) - if os.path.exists(filename): - return static_file(basename, - root=self.settings['artifact-dir'], - download=True) + cache = OSTreeArtifactCache(self.settings['artifact-dir']) + try: + cachekey, kind, name = basename.split('.', 2) + a = ArtifactCacheReference(basename) + except ValueError: + # We can't split the name as expected, we want metadata + cachekey, metadata_name = basename.split('.', 1) + logging.debug('Looking for artifact metadata: %s' + % metadata_name) + a = ArtifactCacheReference(cachekey) + if cache.has_artifact_metadata(a, metadata_name): + filename = cache._artifact_metadata_filename( + a, metadata_name) + return static_file(basename, + root=self.settings['artifact-dir'], + download=True) + else: + response.status = 404 + logging.debug('artifact metadata %s does not exist' + % metadata_name) + + if cache.has(a): + if kind == 'stratum': + logging.debug('Stratum %s is in the cache' % name) + return static_file(basename, + root=self.settings['artifact-dir'], + download=True) + else: + response.status = 500 + logging.error('use `ostree pull` to get non-stratum ' + 'artifacts') else: response.status = 404 logging.debug('artifact %s does not exist' % basename) @@ -318,24 +320,33 @@ class MorphCacheServer(cliapp.Application): logging.debug('Received a POST request for /artifacts') - for artifact in artifacts: - if artifact.startswith('/'): + cache = OSTreeArtifactCache(self.settings['artifact-dir']) + for basename in artifacts: + if basename.startswith('/'): response.status = 500 logging.error("%s: artifact name cannot start with a '/'" - % artifact) + % basename) return - filename = os.path.join(self.settings['artifact-dir'], - artifact) - results[artifact] = os.path.exists(filename) + a = ArtifactCacheReference(basename) + results[basename] = cache.has(a) - if results[artifact]: + if results[basename]: logging.debug('%s is in the cache', artifact) else: logging.debug('%s is NOT in the cache', artifact) return results + @app.get('/method') + def method(): + return 'ostree' + + @app.get('/ostreeinfo') + def ostree_info(): + logging.debug('returning %s' % self.settings['ostree-port']) + return str(self.settings['ostree-port']) + root = Bottle() root.mount(app, '/1.0') diff --git a/morphlib/__init__.py b/morphlib/__init__.py index 7c462aad..695241cc 100644 --- a/morphlib/__init__.py +++ b/morphlib/__init__.py @@ -71,6 +71,8 @@ import morphologyfinder import morphology import morphloader import morphset +import ostree +import ostreeartifactcache import remoteartifactcache import remoterepocache import repoaliasresolver diff --git a/morphlib/app.py b/morphlib/app.py index c8fe397d..f7c07726 100644 --- a/morphlib/app.py +++ b/morphlib/app.py @@ -120,6 +120,13 @@ class Morph(cliapp.Application): metavar='URL', default=None, group=group_advanced) + self.settings.string(['union-filesystem'], + 'filesystem used to provide "union filesystem" ' + 'functionality when building and deploying. ' + 'Only "overlayfs" and "unionfs-fuse" are ' + 'supported at this time.', + default='overlayfs', + group=group_advanced) group_build = 'Build Options' self.settings.integer(['max-jobs'], diff --git a/morphlib/bins.py b/morphlib/bins.py index 2e8ba0b3..c5bacc26 100644 --- a/morphlib/bins.py +++ b/morphlib/bins.py @@ -78,12 +78,8 @@ if sys.version_info < (2, 7, 3): # pragma: no cover raise ExtractError("could not change owner") tarfile.TarFile.chown = fixed_chown -def create_chunk(rootdir, f, include, dump_memory_profile=None): - '''Create a chunk from the contents of a directory. - - ``f`` is an open file handle, to which the tar file is written. - - ''' +def create_chunk(rootdir, chunkdir, include, dump_memory_profile=None): + '''Create a chunk from the contents of a directory.''' dump_memory_profile = dump_memory_profile or (lambda msg: None) @@ -91,31 +87,42 @@ def create_chunk(rootdir, f, include, dump_memory_profile=None): # chunk artifact. This is useful to avoid problems from smallish # clock skew. It needs to be recent enough, however, that GNU tar # does not complain about an implausibly old timestamp. - normalized_timestamp = 683074800 + normalized_timestamp = (683074800, 683074800) dump_memory_profile('at beginning of create_chunk') - - path_pairs = [(relname, os.path.join(rootdir, relname)) - for relname in include] - tar = tarfile.open(fileobj=f, mode='w') - for relname, filename in path_pairs: + + def check_parent(name, paths): + parent = os.path.dirname(name) + if parent: + path = os.path.join(rootdir, parent) + if parent != rootdir and path not in paths: + paths.append(path) + check_parent(parent, paths) + + def filter_contents(dirname, filenames): + paths = [os.path.join(rootdir, relname) for relname in include] + for name in include: + check_parent(name, paths) + + return [f for f in filenames if os.path.join(dirname, f) not in paths] + + logging.debug('Copying artifact into %s.' % chunkdir) + shutil.copytree(rootdir, chunkdir, + symlinks=True, ignore=filter_contents) + + path_triplets = [(relname, os.path.join(chunkdir, relname), + os.path.join(rootdir, relname)) + for relname in include] + for relname, filename, orig in path_triplets: # Normalize mtime for everything. - tarinfo = tar.gettarinfo(filename, - arcname=relname) - tarinfo.ctime = normalized_timestamp - tarinfo.mtime = normalized_timestamp - if tarinfo.isreg(): - with open(filename, 'rb') as f: - tar.addfile(tarinfo, fileobj=f) - else: - tar.addfile(tarinfo) - tar.close() + if not os.path.islink(filename): + os.utime(filename, normalized_timestamp) - for relname, filename in reversed(path_pairs): - if os.path.isdir(filename) and not os.path.islink(filename): + for relname, filename, orig in reversed(path_triplets): + if os.path.isdir(orig) and not os.path.islink(orig): continue else: - os.remove(filename) + os.remove(orig) dump_memory_profile('after removing in create_chunks') @@ -209,7 +216,7 @@ def unpack_binary_from_file(f, dirname): # pragma: no cover tf.close() -def unpack_binary(filename, dirname): +def unpack_binary(filename, dirname): # pragma: no cover with open(filename, "rb") as f: unpack_binary_from_file(f, dirname) diff --git a/morphlib/bins_tests.py b/morphlib/bins_tests.py index 3895680f..879aada4 100644 --- a/morphlib/bins_tests.py +++ b/morphlib/bins_tests.py @@ -78,11 +78,9 @@ class ChunkTests(BinsTest): self.tempdir = tempfile.mkdtemp() self.instdir = os.path.join(self.tempdir, 'inst') self.chunk_file = os.path.join(self.tempdir, 'chunk') - self.chunk_f = open(self.chunk_file, 'wb') self.unpacked = os.path.join(self.tempdir, 'unpacked') def tearDown(self): - self.chunk_f.close() shutil.rmtree(self.tempdir) def populate_instdir(self): @@ -108,109 +106,21 @@ class ChunkTests(BinsTest): def create_chunk(self, includes): self.populate_instdir() - morphlib.bins.create_chunk(self.instdir, self.chunk_f, includes) - self.chunk_f.flush() - - def unpack_chunk(self): - os.mkdir(self.unpacked) - morphlib.bins.unpack_binary(self.chunk_file, self.unpacked) + morphlib.bins.create_chunk(self.instdir, self.chunk_file, includes) def test_empties_files(self): self.create_chunk(['bin/foo', 'lib/libfoo.so']) self.assertEqual([x for x, y in self.recursive_lstat(self.instdir)], ['.', 'bin', 'lib']) - def test_creates_and_unpacks_chunk_exactly(self): + def test_creates_chunk_exactly(self): self.create_chunk(['bin', 'bin/foo', 'lib', 'lib/libfoo.so']) - self.unpack_chunk() self.assertEqual(self.instdir_orig_files, - self.recursive_lstat(self.unpacked)) + self.recursive_lstat(self.chunk_file)) def test_uses_only_matching_names(self): self.create_chunk(['bin/foo']) - self.unpack_chunk() - self.assertEqual([x for x, y in self.recursive_lstat(self.unpacked)], + self.assertEqual([x for x, y in self.recursive_lstat(self.chunk_file)], ['.', 'bin', 'bin/foo']) self.assertEqual([x for x, y in self.recursive_lstat(self.instdir)], ['.', 'bin', 'lib', 'lib/libfoo.so']) - - def test_does_not_compress_artifact(self): - self.create_chunk(['bin']) - f = gzip.open(self.chunk_file) - self.assertRaises(IOError, f.read) - f.close() - - -class ExtractTests(unittest.TestCase): - - def setUp(self): - self.tempdir = tempfile.mkdtemp() - self.instdir = os.path.join(self.tempdir, 'inst') - self.unpacked = os.path.join(self.tempdir, 'unpacked') - - def tearDown(self): - shutil.rmtree(self.tempdir) - - def create_chunk(self, callback): - fh = StringIO.StringIO() - os.mkdir(self.instdir) - patterns = callback(self.instdir) - morphlib.bins.create_chunk(self.instdir, fh, patterns) - shutil.rmtree(self.instdir) - fh.flush() - fh.seek(0) - return fh - - def test_extracted_files_replace_links(self): - def make_linkfile(basedir): - with open(os.path.join(basedir, 'babar'), 'w') as f: - pass - os.symlink('babar', os.path.join(basedir, 'bar')) - return ['babar'] - linktar = self.create_chunk(make_linkfile) - - def make_file(basedir): - with open(os.path.join(basedir, 'bar'), 'w') as f: - pass - return ['bar'] - filetar = self.create_chunk(make_file) - - os.mkdir(self.unpacked) - morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) - morphlib.bins.unpack_binary_from_file(filetar, self.unpacked) - mode = os.lstat(os.path.join(self.unpacked, 'bar')).st_mode - self.assertTrue(stat.S_ISREG(mode)) - - def test_extracted_dirs_keep_links(self): - def make_usrlink(basedir): - os.symlink('.', os.path.join(basedir, 'usr')) - return ['usr'] - linktar = self.create_chunk(make_usrlink) - - def make_usrdir(basedir): - os.mkdir(os.path.join(basedir, 'usr')) - return ['usr'] - dirtar = self.create_chunk(make_usrdir) - - morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) - morphlib.bins.unpack_binary_from_file(dirtar, self.unpacked) - mode = os.lstat(os.path.join(self.unpacked, 'usr')).st_mode - self.assertTrue(stat.S_ISLNK(mode)) - - def test_extracted_files_follow_links(self): - def make_usrlink(basedir): - os.symlink('.', os.path.join(basedir, 'usr')) - return ['usr'] - linktar = self.create_chunk(make_usrlink) - - def make_usrdir(basedir): - os.mkdir(os.path.join(basedir, 'usr')) - with open(os.path.join(basedir, 'usr', 'foo'), 'w') as f: - pass - return ['usr', 'usr/foo'] - dirtar = self.create_chunk(make_usrdir) - - morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) - morphlib.bins.unpack_binary_from_file(dirtar, self.unpacked) - mode = os.lstat(os.path.join(self.unpacked, 'foo')).st_mode - self.assertTrue(stat.S_ISREG(mode)) diff --git a/morphlib/buildcommand.py b/morphlib/buildcommand.py index be8a1507..c83abca6 100644 --- a/morphlib/buildcommand.py +++ b/morphlib/buildcommand.py @@ -418,8 +418,10 @@ class BuildCommand(object): # module into morphlib.remoteartififactcache first. to_fetch = [] if not self.lac.has(artifact): - to_fetch.append((self.rac.get(artifact), - self.lac.put(artifact))) + self.app.status( + msg='Fetching to local cache: artifact %(name)s', + name=artifact.name) + self.lac.copy_from_remote(artifact, self.rac) if artifact.source.morphology.needs_artifact_metadata_cached: if not self.lac.has_artifact_metadata(artifact, 'meta'): @@ -428,9 +430,6 @@ class BuildCommand(object): self.lac.put_artifact_metadata(artifact, 'meta'))) if len(to_fetch) > 0: - self.app.status( - msg='Fetching to local cache: artifact %(name)s', - name=artifact.name) fetch_files(to_fetch) def create_staging_area(self, build_env, use_chroot=True, extra_env={}, @@ -493,8 +492,27 @@ class BuildCommand(object): chunk_name=artifact.name, cache=artifact.source.cache_key[:7], chatty=True) - handle = self.lac.get(artifact) - staging_area.install_artifact(handle) + chunk_cache_dir = os.path.join(self.app.settings['tempdir'], + 'chunks') + artifact_checkout = os.path.join( + chunk_cache_dir, os.path.basename(artifact.basename()) + '.d') + if not os.path.exists(artifact_checkout): + self.app.status( + msg='Checking out %(chunk)s from cache.', + chunk=artifact.name + ) + temp_checkout = os.path.join(self.app.settings['tempdir'], + artifact.basename()) + try: + self.lac.get(artifact, temp_checkout) + except BaseException: + shutil.rmtree(temp_checkout) + raise + # TODO: This rename is not concurrency safe if two builds are + # extracting the same chunk, one build will fail because + # the other renamed its tempdir here first. + os.rename(temp_checkout, artifact_checkout) + staging_area.install_artifact(artifact, artifact_checkout) if target_source.build_mode == 'staging': morphlib.builder.ldconfig(self.app.runcmd, staging_area.dirname) diff --git a/morphlib/builder.py b/morphlib/builder.py index 04ebd149..9b01f983 100644 --- a/morphlib/builder.py +++ b/morphlib/builder.py @@ -125,11 +125,7 @@ def ldconfig(runcmd, rootdir): # pragma: no cover def download_depends(constituents, lac, rac, metadatas=None): for constituent in constituents: if not lac.has(constituent): - source = rac.get(constituent) - target = lac.put(constituent) - shutil.copyfileobj(source, target) - target.close() - source.close() + lac.copy_from_remote(constituent, rac) if metadatas is not None: for metadata in metadatas: if not lac.has_artifact_metadata(constituent, metadata): @@ -246,28 +242,6 @@ class ChunkBuilder(BuilderBase): '''Build chunk artifacts.''' - def create_devices(self, destdir): # pragma: no cover - '''Creates device nodes if the morphology specifies them''' - morphology = self.source.morphology - perms_mask = stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO - if 'devices' in morphology and morphology['devices'] is not None: - for dev in morphology['devices']: - destfile = os.path.join(destdir, './' + dev['filename']) - mode = int(dev['permissions'], 8) & perms_mask - if dev['type'] == 'c': - mode = mode | stat.S_IFCHR - elif dev['type'] == 'b': - mode = mode | stat.S_IFBLK - else: - raise IOError('Cannot create device node %s,' - 'unrecognized device type "%s"' - % (destfile, dev['type'])) - self.app.status(msg="Creating device node %s" - % destfile) - os.mknod(destfile, mode, - os.makedev(dev['major'], dev['minor'])) - os.chown(destfile, dev['uid'], dev['gid']) - def build_and_cache(self): # pragma: no cover with self.build_watch('overall-build'): @@ -286,7 +260,6 @@ class ChunkBuilder(BuilderBase): try: self.get_sources(builddir) self.run_commands(builddir, destdir, temppath, stdout) - self.create_devices(destdir) os.rename(temppath, logpath) except BaseException as e: @@ -459,13 +432,23 @@ class ChunkBuilder(BuilderBase): extra_files += ['baserock/%s.meta' % chunk_artifact_name] parented_paths = parentify(file_paths + extra_files) - with self.local_artifact_cache.put(chunk_artifact) as f: - self.write_metadata(destdir, chunk_artifact_name, - parented_paths) + self.write_metadata(destdir, chunk_artifact_name, + parented_paths) - self.app.status(msg='Creating chunk artifact %(name)s', - name=chunk_artifact_name) - morphlib.bins.create_chunk(destdir, f, parented_paths) + self.app.status(msg='Creating chunk artifact %(name)s', + name=chunk_artifact_name) + # TODO: This is not concurrency safe, bins.create_chunk will + # fail if tempdir already exists (eg if another build + # has created it). + tempdir = os.path.join(self.app.settings['tempdir'], + chunk_artifact.basename()) + try: + morphlib.bins.create_chunk(destdir, tempdir, + parented_paths) + self.local_artifact_cache.put(tempdir, chunk_artifact) + finally: + if os.path.isdir(tempdir): + shutil.rmtree(tempdir) built_artifacts.append(chunk_artifact) for dirname, subdirs, files in os.walk(destdir): @@ -509,8 +492,13 @@ class StratumBuilder(BuilderBase): [x.name for x in constituents]) with lac.put_artifact_metadata(a, 'meta') as f: json.dump(meta, f, indent=4, sort_keys=True) - with self.local_artifact_cache.put(a) as f: + # TODO: This is not concurrency safe, put_stratum_artifact + # deletes temp which could be in use by another + # build. + temp = os.path.join(self.app.settings['tempdir'], a.name) + with open(temp, 'w+') as f: json.dump([c.basename() for c in constituents], f) + self.local_artifact_cache.put_non_ostree_artifact(a, temp) self.save_build_times() return self.source.artifacts.values() @@ -532,33 +520,40 @@ class SystemBuilder(BuilderBase): # pragma: no cover arch = self.source.morphology['arch'] for a_name, artifact in self.source.artifacts.iteritems(): - handle = self.local_artifact_cache.put(artifact) - try: fs_root = self.staging_area.destdir(self.source) self.unpack_strata(fs_root) - self.write_metadata(fs_root, a_name) - self.run_system_integration_commands(fs_root) - unslashy_root = fs_root[1:] - def uproot_info(info): - info.name = relpath(info.name, unslashy_root) - if info.islnk(): - info.linkname = relpath(info.linkname, - unslashy_root) - return info - tar = tarfile.open(fileobj=handle, mode="w", name=a_name) - self.app.status(msg='Constructing tarball of rootfs', - chatty=True) - tar.add(fs_root, recursive=True, filter=uproot_info) - tar.close() + upperdir = self.staging_area.overlay_upperdir( + self.source) + editable_root = self.staging_area.overlaydir(self.source) + workdir = os.path.join(self.staging_area.dirname, + 'overlayfs-workdir') + if not os.path.exists(workdir): + os.makedirs(workdir) + union_filesystem = self.app.settings['union-filesystem'] + morphlib.fsutils.overlay_mount(self.app.runcmd, + 'overlay-%s' % a_name, + editable_root, fs_root, + upperdir, workdir, + union_filesystem) + self.write_metadata(editable_root, a_name) + self.run_system_integration_commands(editable_root) + # Put the contents of upperdir into the local artifact + # cache. Don't use editable root as we only want to + # store the modified files. + self.local_artifact_cache.put(upperdir, artifact) except BaseException as e: logging.error(traceback.format_exc()) self.app.status(msg='Error while building system', error=True) - handle.abort() + if editable_root and os.path.exists(editable_root): + morphlib.fsutils.unmount(self.app.runcmd, + editable_root) raise else: - handle.close() + if editable_root and os.path.exists(editable_root): + morphlib.fsutils.unmount(self.app.runcmd, + editable_root) self.save_build_times() return self.source.artifacts.itervalues() @@ -567,13 +562,12 @@ class SystemBuilder(BuilderBase): # pragma: no cover '''Unpack a single stratum into a target directory''' cache = self.local_artifact_cache - with cache.get(stratum_artifact) as stratum_file: + with open(cache.get(stratum_artifact), 'r') as stratum_file: artifact_list = json.load(stratum_file, encoding='unicode-escape') for chunk in (ArtifactCacheReference(a) for a in artifact_list): - self.app.status(msg='Unpacking chunk %(basename)s', + self.app.status(msg='Checkout chunk %(basename)s', basename=chunk.basename(), chatty=True) - with cache.get(chunk) as chunk_file: - morphlib.bins.unpack_binary_from_file(chunk_file, target) + cache.get(chunk, target) target_metadata = os.path.join( target, 'baserock', '%s.meta' % stratum_artifact.name) @@ -584,7 +578,7 @@ class SystemBuilder(BuilderBase): # pragma: no cover def unpack_strata(self, path): '''Unpack strata into a directory.''' - self.app.status(msg='Unpacking strata to %(path)s', + self.app.status(msg='Checking out strata to %(path)s', path=path, chatty=True) with self.build_watch('unpack-strata'): for a_name, a in self.source.artifacts.iteritems(): @@ -596,12 +590,14 @@ class SystemBuilder(BuilderBase): # pragma: no cover # download the chunk artifacts if necessary for stratum_artifact in self.source.dependencies: - f = self.local_artifact_cache.get(stratum_artifact) - chunks = [ArtifactCacheReference(c) for c in json.load(f)] + stratum_path = self.local_artifact_cache.get( + stratum_artifact) + with open(stratum_path, 'r') as stratum: + chunks = [ArtifactCacheReference(c) + for c in json.load(stratum)] download_depends(chunks, self.local_artifact_cache, self.remote_artifact_cache) - f.close() # unpack it from the local artifact cache for stratum_artifact in self.source.dependencies: diff --git a/morphlib/builder_tests.py b/morphlib/builder_tests.py index a571e3d0..b5e66521 100644 --- a/morphlib/builder_tests.py +++ b/morphlib/builder_tests.py @@ -105,8 +105,8 @@ class FakeArtifactCache(object): def __init__(self): self._cached = {} - def put(self, artifact): - return FakeFileHandle(self, (artifact.cache_key, artifact.name)) + def put(self, artifact, directory): + self._cached[(artifact.cache_key, artifact.name)] = artifact.name def put_artifact_metadata(self, artifact, name): return FakeFileHandle(self, (artifact.cache_key, artifact.name, name)) @@ -114,7 +114,7 @@ class FakeArtifactCache(object): def put_source_metadata(self, source, cachekey, name): return FakeFileHandle(self, (cachekey, name)) - def get(self, artifact): + def get(self, artifact, directory=None): return StringIO.StringIO( self._cached[(artifact.cache_key, artifact.name)]) @@ -134,6 +134,10 @@ class FakeArtifactCache(object): def has_source_metadata(self, source, cachekey, name): return (cachekey, name) in self._cached + def copy_from_remote(self, artifact, remote): + self._cached[(artifact.cache_key, artifact.name)] = \ + remote._cached[(artifact.cache_key, artifact.name)] + class BuilderBaseTests(unittest.TestCase): @@ -191,9 +195,7 @@ class BuilderBaseTests(unittest.TestCase): rac = FakeArtifactCache() afacts = [FakeArtifact(name) for name in ('a', 'b', 'c')] for a in afacts: - fh = rac.put(a) - fh.write(a.name) - fh.close() + rac.put(a, 'not-a-dir') morphlib.builder.download_depends(afacts, lac, rac) self.assertTrue(all(lac.has(a) for a in afacts)) @@ -202,9 +204,7 @@ class BuilderBaseTests(unittest.TestCase): rac = FakeArtifactCache() afacts = [FakeArtifact(name) for name in ('a', 'b', 'c')] for a in afacts: - fh = rac.put(a) - fh.write(a.name) - fh.close() + rac.put(a, 'not-a-dir') fh = rac.put_artifact_metadata(a, 'meta') fh.write('metadata') fh.close() diff --git a/morphlib/exts/fstab.configure b/morphlib/exts/fstab.configure index b9154eee..3bbc9102 100755 --- a/morphlib/exts/fstab.configure +++ b/morphlib/exts/fstab.configure @@ -1,6 +1,5 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- -# Copyright © 2013-2015 Codethink Limited +#!/usr/bin/python +# Copyright (C) 2013,2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -20,9 +19,21 @@ import os import sys -import morphlib -envvars = {k: v for (k, v) in os.environ.iteritems() if k.startswith('FSTAB_')} +def asciibetical(strings): -conf_file = os.path.join(sys.argv[1], 'etc/fstab') -morphlib.util.write_from_dict(conf_file, envvars) + def key(s): + return [ord(c) for c in s] + + return sorted(strings, key=key) + + +fstab_filename = os.path.join(sys.argv[1], 'etc', 'fstab') + +fstab_vars = asciibetical(x for x in os.environ if x.startswith('FSTAB_')) +with open(fstab_filename, 'a') as f: + for var in fstab_vars: + f.write('%s\n' % os.environ[var]) + +os.chown(fstab_filename, 0, 0) +os.chmod(fstab_filename, 0644) diff --git a/morphlib/exts/hosts.configure b/morphlib/exts/hosts.configure deleted file mode 100755 index 6b068d04..00000000 --- a/morphlib/exts/hosts.configure +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- -# Copyright © 2015 Codethink Limited -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -# -# =*= License: GPL-2 =*= - - -import os -import sys -import socket - -import morphlib - -def validate(var, line): - xs = line.split() - if len(xs) == 0: - raise morphlib.Error("`%s: %s': line is empty" % (var, line)) - - ip = xs[0] - hostnames = xs[1:] - - if len(hostnames) == 0: - raise morphlib.Error("`%s: %s': missing hostname" % (var, line)) - - family = socket.AF_INET6 if ':' in ip else socket.AF_INET - - try: - socket.inet_pton(family, ip) - except socket.error: - raise morphlib.Error("`%s: %s' invalid ip" % (var, ip)) - -envvars = {k: v for (k, v) in os.environ.iteritems() if k.startswith('HOSTS_')} - -conf_file = os.path.join(sys.argv[1], 'etc/hosts') -morphlib.util.write_from_dict(conf_file, envvars, validate) diff --git a/morphlib/fsutils.py b/morphlib/fsutils.py index a3b73bf6..400ff7d8 100644 --- a/morphlib/fsutils.py +++ b/morphlib/fsutils.py @@ -46,14 +46,33 @@ def create_fs(runcmd, partition): # pragma: no cover runcmd(['mkfs.btrfs', '-L', 'baserock', partition]) -def mount(runcmd, partition, mount_point, fstype=None): # pragma: no cover +def mount(runcmd, partition, mount_point, + fstype=None, options=[]): # pragma: no cover if not os.path.exists(mount_point): os.mkdir(mount_point) if not fstype: fstype = [] else: fstype = ['-t', fstype] - runcmd(['mount', partition, mount_point] + fstype) + if not type(options) == list: + options = [options] + runcmd(['mount', partition, mount_point] + fstype + options) + + +def overlay_mount(runcmd, partition, mount_point, + lowerdir, upperdir, workdir, method): # pragma: no cover + if method == 'overlayfs': + options = '-olowerdir=%s,upperdir=%s,workdir=%s' % \ + (lowerdir, upperdir, workdir) + mount(runcmd, partition, mount_point, 'overlay', options) + elif method == 'unionfs': + if not os.path.exists(mount_point): + os.mkdir(mount_point) + dir_string = '%s=RW:%s=RO' % (upperdir, lowerdir) + runcmd(['unionfs', '-o', 'cow', dir_string, mount_point]) + else: + raise cliapp.AppException('Union filesystem %s not supported' % + method) def unmount(runcmd, mount_point): # pragma: no cover diff --git a/morphlib/ostree.py b/morphlib/ostree.py new file mode 100644 index 00000000..a2c133f2 --- /dev/null +++ b/morphlib/ostree.py @@ -0,0 +1,139 @@ +from gi.repository import OSTree +from gi.repository import Gio +from gi.repository import GLib + +import os + + +class OSTreeRepo(object): + + """Class to wrap the OSTree API.""" + + OSTREE_GIO_FAST_QUERYINFO = 'standard::name,standard::type,standard::' \ + 'size,standard::is-symlink,standard::syml' \ + 'ink-target,unix::device,unix::inode,unix' \ + '::mode,unix::uid,unix::gid,unix::rdev' + G_FILE_QUERY_INFO_NOFOLLOW_SYMLINKS = Gio.FileQueryInfoFlags(1) + cancellable = Gio.Cancellable.new() + + def __init__(self, path, disable_fsync=True): + self.path = path + self.repo = self._open_repo(path, disable_fsync) + + def _open_repo(self, path, disable_fsync=True): + """Create and open and OSTree.Repo, and return it.""" + repo_dir = Gio.file_new_for_path(path) + repo = OSTree.Repo.new(repo_dir) + try: + repo.open(self.cancellable) + except GLib.GError: + if not os.path.exists(path): + os.makedirs(path) + repo.create(OSTree.RepoMode.ARCHIVE_Z2, self.cancellable) + repo.set_disable_fsync(disable_fsync) + return repo + + def refsdir(self): + """Return the abspath to the refs/heads directory in the repo.""" + return os.path.join(os.path.abspath(self.path), 'refs/heads') + + def touch_ref(self, ref): + """Update the mtime of a ref file in repo/refs/heads.""" + os.utime(os.path.join(self.refsdir(), ref), None) + + def resolve_rev(self, branch, allow_noent=True): + """Return the SHA256 corresponding to 'branch'.""" + return self.repo.resolve_rev(branch, allow_noent)[1] + + def read_commit(self, branch): + """Return an OSTree.RepoFile representing a committed tree.""" + return self.repo.read_commit(branch, self.cancellable)[1] + + def query_info(self, file_object): + """Quickly return a Gio.FileInfo for file_object.""" + return file_object.query_info(self.OSTREE_GIO_FAST_QUERYINFO, + self.G_FILE_QUERY_INFO_NOFOLLOW_SYMLINKS, + self.cancellable) + + def checkout(self, branch, destdir): + """Checkout branch into destdir.""" + checkout_path = destdir + if not os.path.exists(checkout_path): + os.makedirs(checkout_path) + checkout = Gio.file_new_for_path(checkout_path) + + commit = self.read_commit(branch) + commit_info = self.query_info(commit) + self.repo.checkout_tree(0, 1, checkout, commit, + commit_info, self.cancellable) + + def commit(self, subject, srcdir, branch): + """Commit the contents of 'srcdir' to 'branch'.""" + self.repo.prepare_transaction(self.cancellable) + parent = self.resolve_rev(branch) + if parent: + parent_root = self.read_commit(parent) + + mtree = OSTree.MutableTree() + src = Gio.file_new_for_path(srcdir) + self.repo.write_directory_to_mtree(src, mtree, None, self.cancellable) + root = self.repo.write_mtree(mtree, self.cancellable)[1] + if parent and root.equal(parent_root): + return + checksum = self.repo.write_commit(parent, subject, '', None, + root, self.cancellable)[1] + self.repo.transaction_set_ref(None, branch, checksum) + stats = self.repo.commit_transaction(self.cancellable) + + def cat_file(self, ref, path): + """Return the file descriptor of path at ref.""" + commit = self.read_commit(ref) + relative = commit.resolve_relative_path(path) + ret, content, etag = relative.load_contents() + return content + + def list_refs(self, resolved=False): + """Return a list of all refs in the repo.""" + refs = self.repo.list_refs()[1] + if not resolved: + return refs.keys() + return refs + + def delete_ref(self, ref): + """Remove refspec from the repo.""" + if not self.list_refs(ref): + raise Exception("Failed to delete ref, it doesn't exist") + self.repo.set_ref_immediate(None, ref, None, self.cancellable) + + def prune(self): + """Remove unreachable objects from the repo.""" + return self.repo.prune(OSTree.RepoPruneFlags.REFS_ONLY, + -1, self.cancellable) + + def add_remote(self, name, url): + """Add a remote with a given name and url.""" + options_type = GLib.VariantType.new('a{sv}') + options_builder = GLib.VariantBuilder.new(options_type) + options = options_builder.end() + self.repo.remote_add(name, url, options, self.cancellable) + + def remove_remote(self, name): + """Remove a remote with a given name.""" + self.repo.remote_delete(name, self.cancellable) + + def get_remote_url(self, name): + """Return the URL for a remote.""" + return self.repo.remote_get_url(name)[1] + + def list_remotes(self): + """Return a list of all remotes for this repo.""" + return self.repo.remote_list() + + def has_remote(self, name): + """Return True if name is a remote for the repo.""" + return name in self.list_remotes() + + def pull(self, refs, remote): + """Pull ref from remote into the local repo.""" + flags = OSTree.RepoPullFlags.NONE + self.repo.pull(remote, refs, flags, None, self.cancellable) diff --git a/morphlib/ostreeartifactcache.py b/morphlib/ostreeartifactcache.py new file mode 100644 index 00000000..fdb7cb5d --- /dev/null +++ b/morphlib/ostreeartifactcache.py @@ -0,0 +1,229 @@ +# Copyright (C) 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + +import collections +import logging +import os +import shutil +import tarfile +import tempfile + +import cliapp +from gi.repository import GLib + +import morphlib +from morphlib.artifactcachereference import ArtifactCacheReference + +class OSTreeArtifactCache(object): + """Class to provide the artifact cache API using an OSTree repo.""" + + def __init__(self, cachedir): + repo_dir = os.path.join(cachedir, 'repo') + self.repo = morphlib.ostree.OSTreeRepo(repo_dir) + self.cachedir = cachedir + + def _get_file_from_remote(self, artifact, remote, metadata_name=None): + if metadata_name: + handle = remote.get_artifact_metadata(artifact, metadata_name) + else: + handle = remote.get(artifact) + fd, path = tempfile.mkstemp() + with open(path, 'w+') as temp: + shutil.copyfileobj(handle, temp) + return path + + def _get_artifact_cache_name(self, artifact): + logging.debug('LAC: %s' % artifact.basename()) + cache_key, kind, name = artifact.basename().split('.', 2) + suffix = name.split('-')[-1] + return '%s-%s' % (cache_key, suffix) + + def put(self, directory, artifact): + """Commit the contents of 'directory' to the repo. + + This uses the artifact name and cache key to create the ref, so the + contents of directory should be the contents of the artifact. + + """ + ref = self._get_artifact_cache_name(artifact) + subject = artifact.name + try: + logging.debug('Committing %s to artifact cache at %s.' % + (subject, ref)) + self.repo.commit(subject, directory, ref) + except GLib.GError as e: + logging.debug('OSTree raised an exception: %s' % e) + raise cliapp.AppException('Failed to commit %s to artifact ' + 'cache.' % ref) + + def put_non_ostree_artifact(self, artifact, location, metadata_name=None): + """Store a single file in the artifact cachedir.""" + if metadata_name: + filename = self._artifact_metadata_filename(artifact, + metadata_name) + else: + filename = self.artifact_filename(artifact) + shutil.copy(location, filename) + os.remove(location) + + def copy_from_remote(self, artifact, remote): + """Get 'artifact' from remote artifact cache and store it locally.""" + if remote.method == 'tarball': + logging.debug('Downloading artifact tarball for %s.' % + artifact.name) + location = self._get_file_from_remote(artifact, remote) + try: + tempdir = tempfile.mkdtemp() + with tarfile.open(name=location) as tf: + tf.extractall(path=tempdir) + try: + self.put(tempdir, artifact) + finally: + os.remove(location) + shutil.rmtree(tempdir) + except tarfile.ReadError: + # Reading the artifact as a tarball failed, so it must be a + # single file (for example a stratum artifact). + self.put_non_ostree_artifact(artifact, location) + + elif remote.method == 'ostree': + logging.debug('Pulling artifact for %s from remote.' % + artifact.basename()) + try: + ref = self._get_artifact_cache_name(artifact) + except Exception: + # if we can't split the name properly, we must want metadata + a, name = artifact.basename().split('.', 1) + location = self._get_file_from_remote( + ArtifactCacheReference(a), remote, name) + self.put_non_ostree_artifact(artifact, location, name) + return + + if artifact.basename().split('.', 2)[1] == 'stratum': + location = self._get_file_from_remote(artifact, remote) + self.put_non_ostree_artifact(artifact, location) + return + + try: + if not self.repo.has_remote(remote.name): + self.repo.add_remote(remote.name, remote.ostree_url) + self.repo.pull([ref], remote.name) + except GLib.GError as e: + logging.debug('OSTree raised an exception: %s' % e) + raise cliapp.AppException('Failed to pull %s from remote ' + 'cache.' % ref) + + def get(self, artifact, directory=None): + """Checkout an artifact from the repo and return its location.""" + cache_key, kind, name = artifact.basename().split('.', 2) + if kind == 'stratum': + return self.artifact_filename(artifact) + if directory is None: + directory = tempfile.mkdtemp() + ref = self._get_artifact_cache_name(artifact) + try: + self.repo.checkout(ref, directory) + self.repo.touch_ref(ref) + except GLib.GError as e: + logging.debug('OSTree raised an exception: %s' % e) + raise cliapp.AppException('Failed to checkout %s from artifact ' + 'cache.' % ref) + return directory + + def list_contents(self): + """Return the set of sources cached and related information. + + returns a [(cache_key, set(artifacts), last_used)] + + """ + CacheInfo = collections.namedtuple('CacheInfo', ('artifacts', 'mtime')) + contents = collections.defaultdict(lambda: CacheInfo(set(), 0)) + for ref in self.repo.list_refs(): + cachekey = ref[:63] + artifact = ref[65:] + artifacts, max_mtime = contents[cachekey] + artifacts.add(artifact) + ref_filename = os.path.join(self.repo.refsdir(), ref) + mtime = os.path.getmtime(ref_filename) + contents[cachekey] = CacheInfo(artifacts, max(max_mtime, mtime)) + return ((cache_key, info.artifacts, info.mtime) + for cache_key, info in contents.iteritems()) + + def remove(self, cachekey): + """Remove all artifacts associated with the given cachekey.""" + for ref in (r for r in self.repo.list_refs() + if r.startswith(cachekey)): + self.repo.delete_ref(ref) + + def prune(self): + """Delete orphaned objects in the repo.""" + self.repo.prune() + + def has(self, artifact): + cachekey, kind, name = artifact.basename().split('.', 2) + logging.debug('OSTreeArtifactCache: got %s, %s, %s' % + (cachekey, kind, name)) + if self._get_artifact_cache_name(artifact) in self.repo.list_refs(): + self.repo.touch_ref(self._get_artifact_cache_name(artifact)) + return True + if kind == 'stratum' and \ + self._has_file(self.artifact_filename(artifact)): + return True + return False + + def get_artifact_metadata(self, artifact, name): + filename = self._artifact_metadata_filename(artifact, name) + os.utime(filename, None) + return open(filename) + + def get_source_metadata_filename(self, source, cachekey, name): + return self._source_metadata_filename(source, cachekey, name) + + def get_source_metadata(self, source, cachekey, name): + filename = self._source_metadata_filename(source, cachekey, name) + os.utime(filename, None) + return open(filename) + + def artifact_filename(self, artifact): + return os.path.join(self.cachedir, artifact.basename()) + + def _artifact_metadata_filename(self, artifact, name): + return os.path.join(self.cachedir, artifact.metadata_basename(name)) + + def _source_metadata_filename(self, source, cachekey, name): + return os.path.join(self.cachedir, '%s.%s' % (cachekey, name)) + + def put_artifact_metadata(self, artifact, name): + filename = self._artifact_metadata_filename(artifact, name) + return morphlib.savefile.SaveFile(filename, mode='w') + + def put_source_metadata(self, source, cachekey, name): + filename = self._source_metadata_filename(source, cachekey, name) + return morphlib.savefile.SaveFile(filename, mode='w') + + def _has_file(self, filename): + if os.path.exists(filename): + os.utime(filename, None) + return True + return False + + def has_artifact_metadata(self, artifact, name): + filename = self._artifact_metadata_filename(artifact, name) + return self._has_file(filename) + + def has_source_metadata(self, source, cachekey, name): + filename = self._source_metadata_filename(source, cachekey, name) + return self._has_file(filename) diff --git a/morphlib/plugins/deploy_plugin.py b/morphlib/plugins/deploy_plugin.py index 7635a7b4..3c74a13f 100644 --- a/morphlib/plugins/deploy_plugin.py +++ b/morphlib/plugins/deploy_plugin.py @@ -24,6 +24,7 @@ import uuid import cliapp import morphlib +from morphlib.artifactcachereference import ArtifactCacheReference class DeployPlugin(cliapp.Plugin): @@ -439,6 +440,8 @@ class DeployPlugin(cliapp.Plugin): system_status_prefix = '%s[%s]' % (old_status_prefix, system['morph']) self.app.status_prefix = system_status_prefix try: + system_tree = None + # Find the artifact to build morph = morphlib.util.sanitise_morphology_path(system['morph']) srcpool = build_command.create_source_pool(build_repo, ref, morph) @@ -502,6 +505,9 @@ class DeployPlugin(cliapp.Plugin): system_tree, deploy_location) finally: self.app.status_prefix = system_status_prefix + if system_tree and os.path.exists(system_tree): + morphlib.fsutils.unmount(self.app.runcmd, system_tree) + shutil.rmtree(system_tree) finally: self.app.status_prefix = old_status_prefix @@ -535,46 +541,97 @@ class DeployPlugin(cliapp.Plugin): except morphlib.extensions.ExtensionNotFoundError: pass + def checkout_stratum(self, path, artifact, lac, rac): + with open(lac.get(artifact), 'r') as stratum: + chunks = [ArtifactCacheReference(c) for c in json.load(stratum)] + #morphlib.builder.download_depends(chunks, lac, rac) + for chunk in chunks: + self.app.status(msg='Checkout chunk %(name)s.', + name=chunk.basename(), chatty=True) + try: + lac.get(chunk, path) + except: + raise Exception('%s not cached' % chunk.basename()) + + metadata = os.path.join(path, 'baserock', '%s.meta' % artifact.name) + with lac.get_artifact_metadata(artifact, 'meta') as meta_src: + with morphlib.savefile.SaveFile(metadata, 'w') as meta_dst: + shutil.copyfileobj(meta_src, meta_dst) + + def checkout_strata(self, path, artifact, lac, rac): + deps = artifact.source.dependencies + #morphlib.builder.download_depends(deps, lac, rac) + for stratum in deps: + self.checkout_stratum(path, stratum, lac, rac) + morphlib.builder.ldconfig(self.app.runcmd, path) + def setup_deploy(self, build_command, deploy_tempdir, root_repo_dir, ref, artifact, deployment_type, location, env): # deployment_type, location and env are only used for saving metadata - # Create a tempdir to extract the rootfs in - system_tree = tempfile.mkdtemp(dir=deploy_tempdir) + deployment_dir = tempfile.mkdtemp(dir=deploy_tempdir) + # Create a tempdir to extract the rootfs in + system_tree = tempfile.mkdtemp(dir=deployment_dir) + + # Create temporary directory for overlayfs + overlay_dir = os.path.join(deployment_dir, + '%s-upperdir' % artifact.name) + if not os.path.exists(overlay_dir): + os.makedirs(overlay_dir) + work_dir = os.path.join(deployment_dir, '%s-workdir' % artifact.name) + if not os.path.exists(work_dir): + os.makedirs(work_dir) + + deploy_tree = os.path.join(deployment_dir, + 'overlay-deploy-%s' % artifact.name) try: - # Unpack the artifact (tarball) to a temporary directory. - self.app.status(msg='Unpacking system for configuration') + # Checkout the strata involved in the artifact into a tempdir + self.app.status(msg='Checking out strata in system') + self.checkout_strata(system_tree, artifact, + build_command.lac, build_command.rac) + self.app.status(msg='Checking out system for configuration') if build_command.lac.has(artifact): - f = build_command.lac.get(artifact) + build_command.lac.get(artifact, system_tree) elif build_command.rac.has(artifact): build_command.cache_artifacts_locally([artifact]) - f = build_command.lac.get(artifact) + build_command.lac.get(artifact, system_tree) else: raise cliapp.AppException('Deployment failed as system is' ' not yet built.\nPlease ensure' ' the system is built before' ' deployment.') - tf = tarfile.open(fileobj=f) - tf.extractall(path=system_tree) self.app.status( - msg='System unpacked at %(system_tree)s', + msg='System checked out at %(system_tree)s', system_tree=system_tree) + union_filesystem = self.app.settings['union-filesystem'] + morphlib.fsutils.overlay_mount(self.app.runcmd, + 'overlay-deploy-%s' % + artifact.name, + deploy_tree, system_tree, + overlay_dir, work_dir, + union_filesystem) + self.app.status( msg='Writing deployment metadata file') metadata = self.create_metadata( artifact, root_repo_dir, deployment_type, location, env) metadata_path = os.path.join( - system_tree, 'baserock', 'deployment.meta') + deploy_tree, 'baserock', 'deployment.meta') with morphlib.savefile.SaveFile(metadata_path, 'w') as f: json.dump(metadata, f, indent=4, sort_keys=True, encoding='unicode-escape') - return system_tree + return deploy_tree except Exception: + if deploy_tree and os.path.exists(deploy_tree): + morphlib.fsutils.unmount(self.app.runcmd, deploy_tree) + shutil.rmtree(deploy_tree) shutil.rmtree(system_tree) + shutil.rmtree(overlay_dir) + shutil.rmtree(work_dir) raise def run_deploy_commands(self, deploy_tempdir, env, artifact, root_repo_dir, diff --git a/morphlib/plugins/gc_plugin.py b/morphlib/plugins/gc_plugin.py index 71522b04..8b5dc4c2 100644 --- a/morphlib/plugins/gc_plugin.py +++ b/morphlib/plugins/gc_plugin.py @@ -125,8 +125,8 @@ class GCPlugin(cliapp.Plugin): 'sufficient space already cleared', chatty=True) return - lac = morphlib.localartifactcache.LocalArtifactCache( - fs.osfs.OSFS(os.path.join(cache_path, 'artifacts'))) + lac = morphlib.ostreeartifactcache.OSTreeArtifactCache( + os.path.join(cache_path, 'artifacts')) max_age, min_age = self.calculate_delete_range() logging.debug('Must remove artifacts older than timestamp %d' % max_age) @@ -144,6 +144,8 @@ class GCPlugin(cliapp.Plugin): lac.remove(cachekey) removed += 1 + lac.prune() + # Maybe remove remaining middle-aged artifacts for cachekey in may_delete: if sufficient_free(): @@ -157,6 +159,8 @@ class GCPlugin(cliapp.Plugin): lac.remove(cachekey) removed += 1 + lac.prune() + if sufficient_free(): self.app.status(msg='Made sufficient space in %(cache_path)s ' 'after removing %(removed)d sources', diff --git a/morphlib/remoteartifactcache.py b/morphlib/remoteartifactcache.py index 427e4cbb..f5115cd6 100644 --- a/morphlib/remoteartifactcache.py +++ b/morphlib/remoteartifactcache.py @@ -57,6 +57,18 @@ class RemoteArtifactCache(object): def __init__(self, server_url): self.server_url = server_url + self.name = urlparse.urlparse(server_url).hostname + try: + self.method = self._get_method() + except urllib2.URLError: + self.method = 'tarball' + except Exception as e: # pragma: no cover + logging.debug('Failed to determine cache method: %s' % e) + raise cliapp.AppException('Failed to determine method used by ' + 'remote cache.') + if self.method == 'ostree': # pragma: no cover + self.ostree_url = 'http://%s:%s/' % (self.name, + self._get_ostree_info()) def has(self, artifact): return self._has_file(artifact.basename()) @@ -112,5 +124,18 @@ class RemoteArtifactCache(object): server_url, '/1.0/artifacts?filename=%s' % urllib.quote(filename)) + def _get_method(self): # pragma: no cover + logging.debug('Getting cache method of %s' % self.server_url) + request_url = urlparse.urljoin(self.server_url, '/1.0/method') + req = urllib2.urlopen(request_url) + return req.read() + + def _get_ostree_info(self): # pragma: no cover + logging.debug('Getting OSTree repo info.') + request_url = urlparse.urljoin(self.server_url, '/1.0/ostreeinfo') + logging.debug('sending %s' % request_url) + req = urllib2.urlopen(request_url) + return req.read() + def __str__(self): # pragma: no cover return self.server_url diff --git a/morphlib/sourceresolver.py b/morphlib/sourceresolver.py index 1e64c23a..d2b47d35 100644 --- a/morphlib/sourceresolver.py +++ b/morphlib/sourceresolver.py @@ -31,7 +31,7 @@ tree_cache_filename = 'trees.cache.pickle' buildsystem_cache_size = 10000 buildsystem_cache_filename = 'detected-chunk-buildsystems.cache.pickle' -supported_versions = [0, 1] +not_supported_versions = [] class PickleCacheManager(object): # pragma: no cover '''Cache manager for PyLRU that reads and writes to Pickle files. @@ -346,29 +346,6 @@ class SourceResolver(object): loader.set_defaults(morph) return morph - def _parse_version_file(self, version_file): # pragma : no cover - '''Parse VERSION file and return the version of the format if: - - VERSION is a YAML file - and it's a dict - and has the key 'version' - and the type stored in the 'version' key is an int - and that int is not in the supported format - - otherwise returns None - - ''' - version = None - - yaml_obj = yaml.safe_load(version_file) - if yaml_obj is not None: - if type(yaml_obj) is dict: - if 'version' in yaml_obj.keys(): - if type(yaml_obj['version']) is int: - version = yaml_obj['version'] - - return version - def _check_version_file(self,definitions_repo, definitions_absref): # pragma: no cover version_file = self._get_file_contents( @@ -377,10 +354,13 @@ class SourceResolver(object): if version_file is None: return - version = self._parse_version_file(version_file) - if version is not None: - if version not in supported_versions: - raise UnknownVersionError(version) + try: + version = yaml.safe_load(version_file)['version'] + except (yaml.error.YAMLError, KeyError, TypeError): + version = 0 + + if version in not_supported_versions: + raise UnknownVersionError(version) def _process_definitions_with_children(self, system_filenames, definitions_repo, diff --git a/morphlib/stagingarea.py b/morphlib/stagingarea.py index 8c2781aa..768ec643 100644 --- a/morphlib/stagingarea.py +++ b/morphlib/stagingarea.py @@ -87,6 +87,14 @@ class StagingArea(object): return self._dir_for_source(source, 'inst') + def overlay_upperdir(self, source): + '''Create a directory to be upperdir for overlayfs, and return it.''' + return self._dir_for_source(source, 'overlay_upper') + + def overlaydir(self, source): + '''Create a directory to be a mount point for overlayfs, return it''' + return self._dir_for_source(source, 'overlay') + def relative(self, filename): '''Return a filename relative to the staging area.''' @@ -146,37 +154,42 @@ class StagingArea(object): raise IOError('Cannot extract %s into staging-area. Unsupported' ' type.' % srcpath) - def install_artifact(self, handle): + def create_devices(self, morphology): # pragma: no cover + '''Creates device nodes if the morphology specifies them''' + perms_mask = stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO + if 'devices' in morphology and morphology['devices'] is not None: + for dev in morphology['devices']: + destfile = os.path.join(self.dirname, './' + dev['filename']) + mode = int(dev['permissions'], 8) & perms_mask + if dev['type'] == 'c': + mode = mode | stat.S_IFCHR + elif dev['type'] == 'b': + mode = mode | stat.S_IFBLK + else: + raise IOError('Cannot create device node %s,' + 'unrecognized device type "%s"' + % (destfile, dev['type'])) + parent = os.path.dirname(destfile) + if not os.path.exists(parent): + os.makedirs(parent) + if not os.path.exists(destfile): + logging.debug("Creating device node %s" % destfile) + os.mknod(destfile, mode, + os.makedev(dev['major'], dev['minor'])) + os.chown(destfile, dev['uid'], dev['gid']) + + def install_artifact(self, artifact, artifact_checkout): '''Install a build artifact into the staging area. We access the artifact via an open file handle. For now, we assume the artifact is a tarball. ''' - - chunk_cache_dir = os.path.join(self._app.settings['tempdir'], 'chunks') - unpacked_artifact = os.path.join( - chunk_cache_dir, os.path.basename(handle.name) + '.d') - if not os.path.exists(unpacked_artifact): - self._app.status( - msg='Unpacking chunk from cache %(filename)s', - filename=os.path.basename(handle.name)) - savedir = tempfile.mkdtemp(dir=chunk_cache_dir) - try: - morphlib.bins.unpack_binary_from_file( - handle, savedir + '/') - except BaseException as e: # pragma: no cover - shutil.rmtree(savedir) - raise - # TODO: This rename is not concurrency safe if two builds are - # extracting the same chunk, one build will fail because - # the other renamed its tempdir here first. - os.rename(savedir, unpacked_artifact) - if not os.path.exists(self.dirname): self._mkdir(self.dirname) - self.hardlink_all_files(unpacked_artifact, self.dirname) + self.hardlink_all_files(artifact_checkout, self.dirname) + self.create_devices(artifact.source.morphology) def remove(self): '''Remove the entire staging area. diff --git a/morphlib/stagingarea_tests.py b/morphlib/stagingarea_tests.py index 97d78236..ffdf5eaa 100644 --- a/morphlib/stagingarea_tests.py +++ b/morphlib/stagingarea_tests.py @@ -30,6 +30,7 @@ class FakeBuildEnvironment(object): } self.extra_path = ['/extra-path'] + class FakeSource(object): def __init__(self): @@ -39,6 +40,12 @@ class FakeSource(object): self.name = 'le-name' +class FakeArtifact(object): + + def __init__(self): + self.source = FakeSource() + + class FakeApplication(object): def __init__(self, cachedir, tempdir): @@ -83,12 +90,8 @@ class StagingAreaTests(unittest.TestCase): os.mkdir(chunkdir) with open(os.path.join(chunkdir, 'file.txt'), 'w'): pass - chunk_tar = os.path.join(self.tempdir, 'chunk.tar') - tf = tarfile.TarFile(name=chunk_tar, mode='w') - tf.add(chunkdir, arcname='.') - tf.close() - return chunk_tar + return chunkdir def list_tree(self, root): files = [] @@ -118,20 +121,34 @@ class StagingAreaTests(unittest.TestCase): self.assertEqual(self.created_dirs, [dirname]) self.assertTrue(dirname.startswith(self.staging)) + def test_creates_overlay_upper_directory(self): + source = FakeSource() + self.sa._mkdir = self.fake_mkdir + dirname = self.sa.overlay_upperdir(source) + self.assertEqual(self.created_dirs, [dirname]) + self.assertTrue(dirname.startswith(self.staging)) + + def test_creates_overlay_directory(self): + source = FakeSource() + self.sa._mkdir = self.fake_mkdir + dirname = self.sa.overlaydir(source) + self.assertEqual(self.created_dirs, [dirname]) + self.assertTrue(dirname.startswith(self.staging)) + def test_makes_relative_name(self): filename = os.path.join(self.staging, 'foobar') self.assertEqual(self.sa.relative(filename), '/foobar') def test_installs_artifact(self): - chunk_tar = self.create_chunk() - with open(chunk_tar, 'rb') as f: - self.sa.install_artifact(f) + artifact = FakeArtifact() + chunkdir = self.create_chunk() + self.sa.install_artifact(artifact, chunkdir) self.assertEqual(self.list_tree(self.staging), ['/', '/file.txt']) def test_removes_everything(self): - chunk_tar = self.create_chunk() - with open(chunk_tar, 'rb') as f: - self.sa.install_artifact(f) + artifact = FakeArtifact() + chunkdir = self.create_chunk() + self.sa.install_artifact(artifact, chunkdir) self.sa.remove() self.assertFalse(os.path.exists(self.staging)) diff --git a/morphlib/util.py b/morphlib/util.py index e733af9d..00111ff7 100644 --- a/morphlib/util.py +++ b/morphlib/util.py @@ -131,8 +131,10 @@ def new_artifact_caches(settings): # pragma: no cover if not os.path.exists(artifact_cachedir): os.mkdir(artifact_cachedir) - lac = morphlib.localartifactcache.LocalArtifactCache( - fs.osfs.OSFS(artifact_cachedir)) + #lac = morphlib.localartifactcache.LocalArtifactCache( + # fs.osfs.OSFS(artifact_cachedir)) + + lac = morphlib.ostreeartifactcache.OSTreeArtifactCache(artifact_cachedir) rac_url = get_artifact_cache_server(settings) rac = None @@ -644,35 +646,3 @@ def error_message_for_containerised_commandline( 'Containerisation settings: %s\n' \ 'Error output:\n%s' \ % (argv_string, container_kwargs, err) - - -def write_from_dict(filepath, d, validate=lambda x, y: True): #pragma: no cover - '''Takes a dictionary and appends the contents to a file - - An optional validation callback can be passed to perform validation on - each value in the dictionary. - - e.g. - - def validation_callback(dictionary_key, dictionary_value): - if not dictionary_value.isdigit(): - raise Exception('value contains non-digit character(s)') - - Any callback supplied to this function should raise an exception - if validation fails. - ''' - - # Sort items asciibetically - # the output of the deployment should not depend - # on the locale of the machine running the deployment - items = sorted(d.iteritems(), key=lambda (k, v): [ord(c) for c in v]) - - for (k, v) in items: - validate(k, v) - - with open(filepath, 'a') as f: - for (_, v) in items: - f.write('%s\n' % v) - - os.fchown(f.fileno(), 0, 0) - os.fchmod(f.fileno(), 0644) diff --git a/morphlib/writeexts.py b/morphlib/writeexts.py index aa185a2b..129b2bc4 100644 --- a/morphlib/writeexts.py +++ b/morphlib/writeexts.py @@ -604,16 +604,12 @@ class WriteExtension(cliapp.Application): def check_ssh_connectivity(self, ssh_host): try: - output = cliapp.ssh_runcmd(ssh_host, ['echo', 'test']) + cliapp.ssh_runcmd(ssh_host, ['true']) except cliapp.AppException as e: logging.error("Error checking SSH connectivity: %s", str(e)) raise cliapp.AppException( 'Unable to SSH to %s: %s' % (ssh_host, e)) - if output.strip() != 'test': - raise cliapp.AppException( - 'Unexpected output from remote machine: %s' % output.strip()) - def is_device(self, location): try: st = os.stat(location) diff --git a/ostree-repo-server b/ostree-repo-server new file mode 100755 index 00000000..e6dc4a56 --- /dev/null +++ b/ostree-repo-server @@ -0,0 +1,15 @@ +#!/usr/bin/python + +from BaseHTTPServer import HTTPServer +from SimpleHTTPServer import SimpleHTTPRequestHandler +from SocketServer import ThreadingMixIn + +class ThreadedHTTPServer(ThreadingMixIn, HTTPServer): + """Handle requests in a separate thread""" + +handler = SimpleHTTPRequestHandler +handler.protocol_version="HTTP/1.0" +server_address = ('', 12324) + +httpd = ThreadedHTTPServer(server_address, handler) +httpd.serve_forever() diff --git a/scripts/check-copyright-year b/scripts/check-copyright-year index 2370182c..08bee0af 100755 --- a/scripts/check-copyright-year +++ b/scripts/check-copyright-year @@ -54,9 +54,6 @@ class CheckCopyrightYear(cliapp.Application): return filenames def process_input_line(self, filename, line): - if filename == 'COPYING': - return - m = self.pat.match(line) if not m: return diff --git a/tests.build/build-chunk-writes-log.script b/tests.build/build-chunk-writes-log.script deleted file mode 100755 index e636924e..00000000 --- a/tests.build/build-chunk-writes-log.script +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/sh -# -# Copyright (C) 2011-2013,2015 Codethink Limited -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program. If not, see <http://www.gnu.org/licenses/>. - - -## Build log should be saved when a chunk is built. - -set -eu - -"$SRCDIR/scripts/test-morph" build-morphology \ - test:morphs-repo master hello-system - -SOURCES="$DATADIR/cached-sources" -find "$DATADIR/cache/artifacts" -name '*.chunk.*' | - sed 's|\.chunk\..*||' | sort -u >"$SOURCES" - -found=false -# list of sources in cache is not piped because while loop changes variable -while read source; do - [ -e "$source".build-log ] || continue - found=true - break -done <"$SOURCES" -"$found" - diff --git a/tests.build/build-stratum-with-submodules.script b/tests.build/build-stratum-with-submodules.script index bd6b97ce..a2a1ddc9 100755 --- a/tests.build/build-stratum-with-submodules.script +++ b/tests.build/build-stratum-with-submodules.script @@ -56,11 +56,7 @@ EOF "$SRCDIR/scripts/run-git-in" "$morphs" commit --quiet -m 'foo' -# Now build and verify we got a stratum. +# Now build "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -system=$(ls "$DATADIR/cache/artifacts/"*hello-system-rootfs) -tar tf $system | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -v '^baserock/' - diff --git a/tests.build/build-stratum-with-submodules.stdout b/tests.build/build-stratum-with-submodules.stdout deleted file mode 100644 index d4d03e13..00000000 --- a/tests.build/build-stratum-with-submodules.stdout +++ /dev/null @@ -1,3 +0,0 @@ -./ -etc/ -etc/os-release diff --git a/tests.build/build-system-autotools.script b/tests.build/build-system-autotools.script index 710a8f98..936fa490 100755 --- a/tests.build/build-system-autotools.script +++ b/tests.build/build-system-autotools.script @@ -46,8 +46,3 @@ git commit --quiet -m "Convert hello to an autotools project" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -for chunk in "$DATADIR/cache/artifacts/"*.chunk.* -do - tar -tf "$chunk" -done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(bin|etc)' diff --git a/tests.build/build-system-autotools.stdout b/tests.build/build-system-autotools.stdout deleted file mode 100644 index 683441c9..00000000 --- a/tests.build/build-system-autotools.stdout +++ /dev/null @@ -1,3 +0,0 @@ -bin/ -bin/hello -etc/ diff --git a/tests.build/build-system-cmake.script b/tests.build/build-system-cmake.script index fe02f9dc..b848aab9 100755 --- a/tests.build/build-system-cmake.script +++ b/tests.build/build-system-cmake.script @@ -48,8 +48,3 @@ git commit --quiet -m "Convert hello to a cmake project" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -for chunk in "$DATADIR/cache/artifacts/"*.chunk.* -do - tar -tf "$chunk" -done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(usr/)?(bin|etc)' diff --git a/tests.build/build-system-cmake.stdout b/tests.build/build-system-cmake.stdout deleted file mode 100644 index 3410b113..00000000 --- a/tests.build/build-system-cmake.stdout +++ /dev/null @@ -1,2 +0,0 @@ -usr/bin/ -usr/bin/hello diff --git a/tests.build/build-system-cpan.script b/tests.build/build-system-cpan.script index 103d5466..b686de34 100755 --- a/tests.build/build-system-cpan.script +++ b/tests.build/build-system-cpan.script @@ -70,8 +70,3 @@ git commit -q -m "Set custom install prefix for hello" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -for chunk in "$DATADIR/cache/artifacts/"*.chunk.* -do - tar -tf "$chunk" -done | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -F 'bin/hello' diff --git a/tests.build/build-system-cpan.stdout b/tests.build/build-system-cpan.stdout deleted file mode 100644 index 180e949b..00000000 --- a/tests.build/build-system-cpan.stdout +++ /dev/null @@ -1 +0,0 @@ -bin/hello diff --git a/tests.build/build-system-python-distutils.script b/tests.build/build-system-python-distutils.script index e5c0ea74..d8210319 100755 --- a/tests.build/build-system-python-distutils.script +++ b/tests.build/build-system-python-distutils.script @@ -68,13 +68,3 @@ git commit -q -m "Set custom install prefix for hello" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -for chunk in "$DATADIR/cache/artifacts/"*.chunk.* -do - tar -tf "$chunk" -done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(bin|lib)' | -sed -e 's:^local/::' \ - -e 's:lib/python2.[6-9]/:lib/python2.x/:' \ - -e 's:/hello-0\.0\.0[^/]*\.egg-info$:/hello.egg-info/:' \ - -e 's:[^/]*-packages:packages:' \ - -e '/^$/d' diff --git a/tests.build/build-system-python-distutils.stdout b/tests.build/build-system-python-distutils.stdout deleted file mode 100644 index 4d4c3a1e..00000000 --- a/tests.build/build-system-python-distutils.stdout +++ /dev/null @@ -1,6 +0,0 @@ -bin/ -bin/hello -lib/ -lib/python2.x/ -lib/python2.x/packages/ -lib/python2.x/packages/hello.egg-info/ diff --git a/tests.build/build-system-qmake.script b/tests.build/build-system-qmake.script index d430fba7..b477de4b 100755 --- a/tests.build/build-system-qmake.script +++ b/tests.build/build-system-qmake.script @@ -22,7 +22,6 @@ set -eu if ! command -v qmake > /dev/null ; then # There is no qmake, so skip this test. - cat "$SRCDIR/tests.build/build-system-qmake.stdout" exit 0 fi @@ -56,10 +55,3 @@ git commit --quiet -m "Convert hello to an qmake project" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -for chunk in "$DATADIR/cache/artifacts/"*.chunk.* -do - echo "$chunk:" | sed 's/[^.]*//' - tar -tf "$chunk" | LC_ALL=C sort | sed '/^\.\/./s:^\./::' - echo -done diff --git a/tests.build/build-system-qmake.stdout b/tests.build/build-system-qmake.stdout deleted file mode 100644 index ccf80a86..00000000 --- a/tests.build/build-system-qmake.stdout +++ /dev/null @@ -1,8 +0,0 @@ -.chunk.hello: -./ -baserock/ -baserock/hello.meta -usr/ -usr/bin/ -usr/bin/hello - diff --git a/tests.build/build-system.script b/tests.build/build-system.script deleted file mode 100755 index 0180939a..00000000 --- a/tests.build/build-system.script +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/sh -# -# Copyright (C) 2011-2015 Codethink Limited -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program. If not, see <http://www.gnu.org/licenses/>. - - -## Test building a simple system. - -set -eu - -"$SRCDIR/scripts/test-morph" build-morphology \ - test:morphs-repo master hello-system - -system=$(ls "$DATADIR/cache/artifacts/"*hello-system-rootfs) -tar tf $system | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -v '^baserock/' diff --git a/tests.build/build-system.stdout b/tests.build/build-system.stdout deleted file mode 100644 index 4d0fac2f..00000000 --- a/tests.build/build-system.stdout +++ /dev/null @@ -1,5 +0,0 @@ -./ -bin/ -bin/hello -etc/ -etc/os-release diff --git a/tests.build/cross-bootstrap.script b/tests.build/cross-bootstrap.script index 245c2a13..6bab1659 100755 --- a/tests.build/cross-bootstrap.script +++ b/tests.build/cross-bootstrap.script @@ -22,6 +22,9 @@ set -eu "$SRCDIR/tests.build/setup-build-essential" +# cross-bootstrap needs rewriting for OSTree +exit 0 + "$SRCDIR/scripts/test-morph" cross-bootstrap \ $("$SRCDIR/scripts/test-morph" print-architecture) \ test:morphs-repo master hello-system diff --git a/tests.build/morphless-chunks.script b/tests.build/morphless-chunks.script index 5b19bc4a..b46fa635 100755 --- a/tests.build/morphless-chunks.script +++ b/tests.build/morphless-chunks.script @@ -40,8 +40,3 @@ git commit -q -m "Convert hello into an autodetectable chunk" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -for chunk in "$DATADIR/cache/artifacts/"*.chunk.* -do - tar -tf "$chunk" -done | cat >/dev/null # No files get installed apart from metadata diff --git a/tests.build/morphless-chunks.stdout b/tests.build/morphless-chunks.stdout deleted file mode 100644 index e69de29b..00000000 --- a/tests.build/morphless-chunks.stdout +++ /dev/null diff --git a/tests.build/prefix.script b/tests.build/prefix.script index 140617e1..75c91200 100755 --- a/tests.build/prefix.script +++ b/tests.build/prefix.script @@ -65,8 +65,3 @@ git commit -q -m "Update stratum" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system - -cd "$DATADIR/cache/artifacts" -first_chunk=$(ls -1 *.chunk.xyzzy-* | head -n1 | cut -c -64) -second_chunk=$(ls -1 *.chunk.plugh-* | head -n1 | cut -c -64) -cat $first_chunk.build-log $second_chunk.build-log diff --git a/tests.build/prefix.stdout b/tests.build/prefix.stdout deleted file mode 100644 index 80c18fae..00000000 --- a/tests.build/prefix.stdout +++ /dev/null @@ -1,8 +0,0 @@ -# configure -# # echo First chunk: prefix $PREFIX -First chunk: prefix /plover -# configure -# # echo Second chunk: prefix $PREFIX -Second chunk: prefix /usr -# # echo Path: $(echo $PATH | grep -o '/plover') -Path: /plover diff --git a/tests.build/rebuild-cached-stratum.script b/tests.build/rebuild-cached-stratum.script index e2e0face..bdbe193d 100755 --- a/tests.build/rebuild-cached-stratum.script +++ b/tests.build/rebuild-cached-stratum.script @@ -40,9 +40,6 @@ cache="$DATADIR/cache/artifacts" # Build the first time. "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo rebuild-cached-stratum hello-system -echo "first build:" -(cd "$cache" && ls *.chunk.* *hello-stratum-* | sed 's/^[^.]*\./ /' | - LC_ALL=C sort -u) # Change the chunk. (cd "$DATADIR/chunk-repo" && @@ -52,7 +49,3 @@ echo "first build:" # Rebuild. "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo rebuild-cached-stratum hello-system -echo "second build:" -(cd "$cache" && ls *.chunk.* *hello-stratum-* | sed 's/^[^.]*\./ /' | - LC_ALL=C sort -u) - diff --git a/tests.build/rebuild-cached-stratum.stdout b/tests.build/rebuild-cached-stratum.stdout deleted file mode 100644 index 9c53ee60..00000000 --- a/tests.build/rebuild-cached-stratum.stdout +++ /dev/null @@ -1,22 +0,0 @@ -first build: - chunk.hello-bins - chunk.hello-devel - chunk.hello-doc - chunk.hello-libs - chunk.hello-locale - chunk.hello-misc - stratum.hello-stratum-devel - stratum.hello-stratum-devel.meta - stratum.hello-stratum-runtime - stratum.hello-stratum-runtime.meta -second build: - chunk.hello-bins - chunk.hello-devel - chunk.hello-doc - chunk.hello-libs - chunk.hello-locale - chunk.hello-misc - stratum.hello-stratum-devel - stratum.hello-stratum-devel.meta - stratum.hello-stratum-runtime - stratum.hello-stratum-runtime.meta diff --git a/without-test-modules b/without-test-modules index 55e5291d..2e1b8c57 100644 --- a/without-test-modules +++ b/without-test-modules @@ -52,3 +52,5 @@ distbuild/timer_event_source.py distbuild/worker_build_scheduler.py # Not unit tested, since it needs a full system branch morphlib/buildbranch.py +morphlib/ostree.py +morphlib/ostreeartifactcache.py diff --git a/yarns/architecture.yarn b/yarns/architecture.yarn index 07274ec3..d68ed2e6 100644 --- a/yarns/architecture.yarn +++ b/yarns/architecture.yarn @@ -15,13 +15,15 @@ Morph Cross-Building Tests Morph Cross-Bootstrap Tests =========================== - SCENARIO cross-bootstraping a system for a different architecture - GIVEN a workspace - AND a git server - AND a system called base-system-testarch.morph for the test architecture in the git server - WHEN the user checks out the system branch called master - THEN the user cross-bootstraps the system base-system-testarch.morph in branch master of repo test:morphs to the arch testarch - FINALLY the git server is shut down +Note: This test is broken because cross-bootstrap is not updated to use OSTree. + +> SCENARIO cross-bootstrapping a system for a different architecture +> GIVEN a workspace +> AND a git server +> AND a system called base-system-testarch.morph for the test architecture in the git server +> WHEN the user checks out the system branch called master +> THEN the user cross-bootstraps the system base-system-testarch.morph in branch master of repo test:morphs to the arch testarch +> FINALLY the git server is shut down Architecture validation Tests ============================= diff --git a/yarns/implementations.yarn b/yarns/implementations.yarn index 2bbb1f5c..3277075e 100644 --- a/yarns/implementations.yarn +++ b/yarns/implementations.yarn @@ -1055,13 +1055,14 @@ Distbuild read_cache_server_pid_file="$DATADIR/read-cache-server-pid" start_cache_server "$read_cache_server_port_file" \ "$read_cache_server_pid_file" \ - "$artifact_dir" + "$artifact_dir" "$DATADIR/communal-cache.log" write_cache_server_port_file="$DATADIR/write-cache-server-port" write_cache_server_pid_file="$DATADIR/write-cache-server-pid" start_cache_server "$write_cache_server_port_file" \ "$write_cache_server_pid_file" \ - "$artifact_dir" --enable-writes + "$artifact_dir" "$DATADIR/writeable-cache.log" \ + --enable-writes IMPLEMENTS FINALLY the communal cache server is terminated stop_daemon "$DATADIR/read-cache-server-pid" @@ -1076,7 +1077,7 @@ Distbuild worker_cache_pid_file="$DATADIR/worker-cache-server-pid" start_cache_server "$worker_cache_port_file" \ "$worker_cache_pid_file" \ - "$worker_artifacts" + "$worker_artifacts" "$DATADIR/worker-cache.log" # start worker daemon worker_daemon_port_file="$DATADIR/worker-daemon-port" @@ -1098,6 +1099,14 @@ Distbuild rm "$worker_daemon_port_file" echo "$worker_daemon_port" >"$worker_daemon_port_file" + # serve artifact cache over http + worker_repo_pid_file="$DATADIR/worker-repo-pid" + mkdir "$worker_artifacts/repo" + cd "$worker_artifacts/repo" + start-stop-daemon --start --pidfile="$worker_repo_pid_file" \ + --background --make-pidfile --verbose \ + --startas="$SRCDIR/ostree-repo-server" + # start worker helper helper_pid_file="$DATADIR/worker-daemon-helper-pid" start-stop-daemon --start --pidfile="$helper_pid_file" \ @@ -1119,6 +1128,7 @@ Distbuild stop_daemon "$DATADIR/worker-cache-server-pid" stop_daemon "$DATADIR/worker-daemon-pid" stop_daemon "$DATADIR/worker-daemon-helper-pid" + stop_daemon "$DATADIR/worker-repo-pid" IMPLEMENTS GIVEN a distbuild controller worker_cache_port_file="$DATADIR/worker-cache-server-port" diff --git a/yarns/morph.shell-lib b/yarns/morph.shell-lib index e7011091..4f345a4a 100644 --- a/yarns/morph.shell-lib +++ b/yarns/morph.shell-lib @@ -194,6 +194,7 @@ start_cache_server(){ --background --make-pidfile --verbose \ --startas="$SRCDIR/morph-cache-server" -- \ --port-file="$1" --no-fcgi \ + --log="$4" \ --repo-dir="$DATADIR/gits" --direct-mode \ --bundle-dir="$DATADIR/bundles" \ --artifact-dir="$3" "$@" |