diff options
| author | Adam Coldrick <adam.coldrick@codethink.co.uk> | 2015-03-24 14:16:54 +0000 |
|---|---|---|
| committer | Morph (on behalf of Adam Coldrick) <adam.coldrick@codethink.co.uk> | 2015-03-24 14:16:54 +0000 |
| commit | aa047d1b4ea195c1a5a70568a2b75f958f47fa99 (patch) | |
| tree | 92ad20301b38f56b0e27506e87d7f5a1dcc6bd0f | |
| parent | d1e4fa3639540a51dbb71612bf41a45018f164ea (diff) | |
| download | morph-aa047d1b4ea195c1a5a70568a2b75f958f47fa99.tar.gz | |
Morph build 271da1e1d62c40748b586dc0345d0f7d
System branch: master
50 files changed, 947 insertions, 823 deletions
diff --git a/COPYING b/COPYING new file mode 100644 index 00000000..d159169d --- /dev/null +++ b/COPYING @@ -0,0 +1,339 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. diff --git a/morph-cache-server b/morph-cache-server index 6c7665aa..007cfbe8 100755 --- a/morph-cache-server +++ b/morph-cache-server @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Copyright (C) 2013-2015 Codethink Limited +# Copyright (C) 2013, 2014-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -27,9 +27,6 @@ import shutil from bottle import Bottle, request, response, run, static_file from flup.server.fcgi import WSGIServer from morphcacheserver.repocache import RepoCache -from morphlib.artifactcachereference import ArtifactCacheReference -from morphlib.ostreeartifactcache import OSTreeArtifactCache -from morphlib.remoteartifactcache import RemoteArtifactCache defaults = { @@ -37,7 +34,6 @@ defaults = { 'bundle-dir': '/var/cache/morph-cache-server/bundles', 'artifact-dir': '/var/cache/morph-cache-server/artifacts', 'port': 8080, - 'ostree-port': 12324, } @@ -48,10 +44,6 @@ class MorphCacheServer(cliapp.Application): 'port to listen on', metavar='PORTNUM', default=defaults['port']) - self.settings.integer(['ostree-port'], - 'port for accessing the ostree repo for ' - 'the artifact cache', - default=defaults['ostree-port']) self.settings.string(['port-file'], 'write port number to FILE', metavar='FILE', @@ -76,19 +68,50 @@ class MorphCacheServer(cliapp.Application): 'runs a fcgi-server', default=True) + + def _fetch_artifact(self, url, filename): + in_fh = None + try: + in_fh = urllib2.urlopen(url) + with open(filename, "w") as localtmp: + shutil.copyfileobj(in_fh, localtmp) + in_fh.close() + except Exception, e: + if in_fh is not None: + in_fh.close() + raise + else: + if in_fh is not None: + in_fh.close() + return os.stat(filename) + def _fetch_artifacts(self, server, cacheid, artifacts): ret = {} - cache = OSTreeArtifactCache(self.settings['artifact-dir']) - remote = RemoteArtifactCache('http://%s/' % server) try: for artifact in artifacts: - logging.debug('%s.%s' % (cacheid, artifact)) - cache_artifact = ArtifactCacheReference( - '.'.join((cacheid, artifact))) - cache.copy_from_remote(cache_artifact, remote) + artifact_name = "%s.%s" % (cacheid, artifact) + tmpname = os.path.join(self.settings['artifact-dir'], + ".dl.%s" % artifact_name) + url = "http://%s/1.0/artifacts?filename=%s" % ( + server, urllib.quote(artifact_name)) + stinfo = self._fetch_artifact(url, tmpname) + ret[artifact_name] = { + "size": stinfo.st_size, + "used": stinfo.st_blocks * 512, + } except Exception, e: - logging.debug('OSTree raised an Exception: %s' % e) + for artifact in ret.iterkeys(): + os.unlink(os.path.join(self.settings['artifact-dir'], + ".dl.%s" % artifact)) raise + + for artifact in ret.iterkeys(): + tmpname = os.path.join(self.settings['artifact-dir'], + ".dl.%s" % artifact) + artifilename = os.path.join(self.settings['artifact-dir'], + artifact) + os.rename(tmpname, artifilename) + return ret @@ -149,6 +172,7 @@ class MorphCacheServer(cliapp.Application): response.set_header('Cache-Control', 'no-cache') artifacts = artifacts.split(",") return self._fetch_artifacts(host, cacheid, artifacts) + except Exception, e: response.status = 500 logging.debug('%s' % e) @@ -274,37 +298,11 @@ class MorphCacheServer(cliapp.Application): @app.get('/artifacts') def artifact(): basename = self._unescape_parameter(request.query.filename) - cache = OSTreeArtifactCache(self.settings['artifact-dir']) - try: - cachekey, kind, name = basename.split('.', 2) - a = ArtifactCacheReference(basename) - except ValueError: - # We can't split the name as expected, we want metadata - cachekey, metadata_name = basename.split('.', 1) - logging.debug('Looking for artifact metadata: %s' - % metadata_name) - a = ArtifactCacheReference(cachekey) - if cache.has_artifact_metadata(a, metadata_name): - filename = cache._artifact_metadata_filename( - a, metadata_name) - return static_file(basename, - root=self.settings['artifact-dir'], - download=True) - else: - response.status = 404 - logging.debug('artifact metadata %s does not exist' - % metadata_name) - - if cache.has(a): - if kind == 'stratum': - logging.debug('Stratum %s is in the cache' % name) - return static_file(basename, - root=self.settings['artifact-dir'], - download=True) - else: - response.status = 500 - logging.error('use `ostree pull` to get non-stratum ' - 'artifacts') + filename = os.path.join(self.settings['artifact-dir'], basename) + if os.path.exists(filename): + return static_file(basename, + root=self.settings['artifact-dir'], + download=True) else: response.status = 404 logging.debug('artifact %s does not exist' % basename) @@ -320,33 +318,24 @@ class MorphCacheServer(cliapp.Application): logging.debug('Received a POST request for /artifacts') - cache = OSTreeArtifactCache(self.settings['artifact-dir']) - for basename in artifacts: - if basename.startswith('/'): + for artifact in artifacts: + if artifact.startswith('/'): response.status = 500 logging.error("%s: artifact name cannot start with a '/'" - % basename) + % artifact) return - a = ArtifactCacheReference(basename) - results[basename] = cache.has(a) + filename = os.path.join(self.settings['artifact-dir'], + artifact) + results[artifact] = os.path.exists(filename) - if results[basename]: + if results[artifact]: logging.debug('%s is in the cache', artifact) else: logging.debug('%s is NOT in the cache', artifact) return results - @app.get('/method') - def method(): - return 'ostree' - - @app.get('/ostreeinfo') - def ostree_info(): - logging.debug('returning %s' % self.settings['ostree-port']) - return str(self.settings['ostree-port']) - root = Bottle() root.mount(app, '/1.0') diff --git a/morphlib/__init__.py b/morphlib/__init__.py index 695241cc..7c462aad 100644 --- a/morphlib/__init__.py +++ b/morphlib/__init__.py @@ -71,8 +71,6 @@ import morphologyfinder import morphology import morphloader import morphset -import ostree -import ostreeartifactcache import remoteartifactcache import remoterepocache import repoaliasresolver diff --git a/morphlib/app.py b/morphlib/app.py index f7c07726..c8fe397d 100644 --- a/morphlib/app.py +++ b/morphlib/app.py @@ -120,13 +120,6 @@ class Morph(cliapp.Application): metavar='URL', default=None, group=group_advanced) - self.settings.string(['union-filesystem'], - 'filesystem used to provide "union filesystem" ' - 'functionality when building and deploying. ' - 'Only "overlayfs" and "unionfs-fuse" are ' - 'supported at this time.', - default='overlayfs', - group=group_advanced) group_build = 'Build Options' self.settings.integer(['max-jobs'], diff --git a/morphlib/bins.py b/morphlib/bins.py index c5bacc26..2e8ba0b3 100644 --- a/morphlib/bins.py +++ b/morphlib/bins.py @@ -78,8 +78,12 @@ if sys.version_info < (2, 7, 3): # pragma: no cover raise ExtractError("could not change owner") tarfile.TarFile.chown = fixed_chown -def create_chunk(rootdir, chunkdir, include, dump_memory_profile=None): - '''Create a chunk from the contents of a directory.''' +def create_chunk(rootdir, f, include, dump_memory_profile=None): + '''Create a chunk from the contents of a directory. + + ``f`` is an open file handle, to which the tar file is written. + + ''' dump_memory_profile = dump_memory_profile or (lambda msg: None) @@ -87,42 +91,31 @@ def create_chunk(rootdir, chunkdir, include, dump_memory_profile=None): # chunk artifact. This is useful to avoid problems from smallish # clock skew. It needs to be recent enough, however, that GNU tar # does not complain about an implausibly old timestamp. - normalized_timestamp = (683074800, 683074800) + normalized_timestamp = 683074800 dump_memory_profile('at beginning of create_chunk') - - def check_parent(name, paths): - parent = os.path.dirname(name) - if parent: - path = os.path.join(rootdir, parent) - if parent != rootdir and path not in paths: - paths.append(path) - check_parent(parent, paths) - - def filter_contents(dirname, filenames): - paths = [os.path.join(rootdir, relname) for relname in include] - for name in include: - check_parent(name, paths) - - return [f for f in filenames if os.path.join(dirname, f) not in paths] - - logging.debug('Copying artifact into %s.' % chunkdir) - shutil.copytree(rootdir, chunkdir, - symlinks=True, ignore=filter_contents) - - path_triplets = [(relname, os.path.join(chunkdir, relname), - os.path.join(rootdir, relname)) - for relname in include] - for relname, filename, orig in path_triplets: + + path_pairs = [(relname, os.path.join(rootdir, relname)) + for relname in include] + tar = tarfile.open(fileobj=f, mode='w') + for relname, filename in path_pairs: # Normalize mtime for everything. - if not os.path.islink(filename): - os.utime(filename, normalized_timestamp) + tarinfo = tar.gettarinfo(filename, + arcname=relname) + tarinfo.ctime = normalized_timestamp + tarinfo.mtime = normalized_timestamp + if tarinfo.isreg(): + with open(filename, 'rb') as f: + tar.addfile(tarinfo, fileobj=f) + else: + tar.addfile(tarinfo) + tar.close() - for relname, filename, orig in reversed(path_triplets): - if os.path.isdir(orig) and not os.path.islink(orig): + for relname, filename in reversed(path_pairs): + if os.path.isdir(filename) and not os.path.islink(filename): continue else: - os.remove(orig) + os.remove(filename) dump_memory_profile('after removing in create_chunks') @@ -216,7 +209,7 @@ def unpack_binary_from_file(f, dirname): # pragma: no cover tf.close() -def unpack_binary(filename, dirname): # pragma: no cover +def unpack_binary(filename, dirname): with open(filename, "rb") as f: unpack_binary_from_file(f, dirname) diff --git a/morphlib/bins_tests.py b/morphlib/bins_tests.py index 879aada4..3895680f 100644 --- a/morphlib/bins_tests.py +++ b/morphlib/bins_tests.py @@ -78,9 +78,11 @@ class ChunkTests(BinsTest): self.tempdir = tempfile.mkdtemp() self.instdir = os.path.join(self.tempdir, 'inst') self.chunk_file = os.path.join(self.tempdir, 'chunk') + self.chunk_f = open(self.chunk_file, 'wb') self.unpacked = os.path.join(self.tempdir, 'unpacked') def tearDown(self): + self.chunk_f.close() shutil.rmtree(self.tempdir) def populate_instdir(self): @@ -106,21 +108,109 @@ class ChunkTests(BinsTest): def create_chunk(self, includes): self.populate_instdir() - morphlib.bins.create_chunk(self.instdir, self.chunk_file, includes) + morphlib.bins.create_chunk(self.instdir, self.chunk_f, includes) + self.chunk_f.flush() + + def unpack_chunk(self): + os.mkdir(self.unpacked) + morphlib.bins.unpack_binary(self.chunk_file, self.unpacked) def test_empties_files(self): self.create_chunk(['bin/foo', 'lib/libfoo.so']) self.assertEqual([x for x, y in self.recursive_lstat(self.instdir)], ['.', 'bin', 'lib']) - def test_creates_chunk_exactly(self): + def test_creates_and_unpacks_chunk_exactly(self): self.create_chunk(['bin', 'bin/foo', 'lib', 'lib/libfoo.so']) + self.unpack_chunk() self.assertEqual(self.instdir_orig_files, - self.recursive_lstat(self.chunk_file)) + self.recursive_lstat(self.unpacked)) def test_uses_only_matching_names(self): self.create_chunk(['bin/foo']) - self.assertEqual([x for x, y in self.recursive_lstat(self.chunk_file)], + self.unpack_chunk() + self.assertEqual([x for x, y in self.recursive_lstat(self.unpacked)], ['.', 'bin', 'bin/foo']) self.assertEqual([x for x, y in self.recursive_lstat(self.instdir)], ['.', 'bin', 'lib', 'lib/libfoo.so']) + + def test_does_not_compress_artifact(self): + self.create_chunk(['bin']) + f = gzip.open(self.chunk_file) + self.assertRaises(IOError, f.read) + f.close() + + +class ExtractTests(unittest.TestCase): + + def setUp(self): + self.tempdir = tempfile.mkdtemp() + self.instdir = os.path.join(self.tempdir, 'inst') + self.unpacked = os.path.join(self.tempdir, 'unpacked') + + def tearDown(self): + shutil.rmtree(self.tempdir) + + def create_chunk(self, callback): + fh = StringIO.StringIO() + os.mkdir(self.instdir) + patterns = callback(self.instdir) + morphlib.bins.create_chunk(self.instdir, fh, patterns) + shutil.rmtree(self.instdir) + fh.flush() + fh.seek(0) + return fh + + def test_extracted_files_replace_links(self): + def make_linkfile(basedir): + with open(os.path.join(basedir, 'babar'), 'w') as f: + pass + os.symlink('babar', os.path.join(basedir, 'bar')) + return ['babar'] + linktar = self.create_chunk(make_linkfile) + + def make_file(basedir): + with open(os.path.join(basedir, 'bar'), 'w') as f: + pass + return ['bar'] + filetar = self.create_chunk(make_file) + + os.mkdir(self.unpacked) + morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) + morphlib.bins.unpack_binary_from_file(filetar, self.unpacked) + mode = os.lstat(os.path.join(self.unpacked, 'bar')).st_mode + self.assertTrue(stat.S_ISREG(mode)) + + def test_extracted_dirs_keep_links(self): + def make_usrlink(basedir): + os.symlink('.', os.path.join(basedir, 'usr')) + return ['usr'] + linktar = self.create_chunk(make_usrlink) + + def make_usrdir(basedir): + os.mkdir(os.path.join(basedir, 'usr')) + return ['usr'] + dirtar = self.create_chunk(make_usrdir) + + morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) + morphlib.bins.unpack_binary_from_file(dirtar, self.unpacked) + mode = os.lstat(os.path.join(self.unpacked, 'usr')).st_mode + self.assertTrue(stat.S_ISLNK(mode)) + + def test_extracted_files_follow_links(self): + def make_usrlink(basedir): + os.symlink('.', os.path.join(basedir, 'usr')) + return ['usr'] + linktar = self.create_chunk(make_usrlink) + + def make_usrdir(basedir): + os.mkdir(os.path.join(basedir, 'usr')) + with open(os.path.join(basedir, 'usr', 'foo'), 'w') as f: + pass + return ['usr', 'usr/foo'] + dirtar = self.create_chunk(make_usrdir) + + morphlib.bins.unpack_binary_from_file(linktar, self.unpacked) + morphlib.bins.unpack_binary_from_file(dirtar, self.unpacked) + mode = os.lstat(os.path.join(self.unpacked, 'foo')).st_mode + self.assertTrue(stat.S_ISREG(mode)) diff --git a/morphlib/buildcommand.py b/morphlib/buildcommand.py index c83abca6..be8a1507 100644 --- a/morphlib/buildcommand.py +++ b/morphlib/buildcommand.py @@ -418,10 +418,8 @@ class BuildCommand(object): # module into morphlib.remoteartififactcache first. to_fetch = [] if not self.lac.has(artifact): - self.app.status( - msg='Fetching to local cache: artifact %(name)s', - name=artifact.name) - self.lac.copy_from_remote(artifact, self.rac) + to_fetch.append((self.rac.get(artifact), + self.lac.put(artifact))) if artifact.source.morphology.needs_artifact_metadata_cached: if not self.lac.has_artifact_metadata(artifact, 'meta'): @@ -430,6 +428,9 @@ class BuildCommand(object): self.lac.put_artifact_metadata(artifact, 'meta'))) if len(to_fetch) > 0: + self.app.status( + msg='Fetching to local cache: artifact %(name)s', + name=artifact.name) fetch_files(to_fetch) def create_staging_area(self, build_env, use_chroot=True, extra_env={}, @@ -492,27 +493,8 @@ class BuildCommand(object): chunk_name=artifact.name, cache=artifact.source.cache_key[:7], chatty=True) - chunk_cache_dir = os.path.join(self.app.settings['tempdir'], - 'chunks') - artifact_checkout = os.path.join( - chunk_cache_dir, os.path.basename(artifact.basename()) + '.d') - if not os.path.exists(artifact_checkout): - self.app.status( - msg='Checking out %(chunk)s from cache.', - chunk=artifact.name - ) - temp_checkout = os.path.join(self.app.settings['tempdir'], - artifact.basename()) - try: - self.lac.get(artifact, temp_checkout) - except BaseException: - shutil.rmtree(temp_checkout) - raise - # TODO: This rename is not concurrency safe if two builds are - # extracting the same chunk, one build will fail because - # the other renamed its tempdir here first. - os.rename(temp_checkout, artifact_checkout) - staging_area.install_artifact(artifact, artifact_checkout) + handle = self.lac.get(artifact) + staging_area.install_artifact(handle) if target_source.build_mode == 'staging': morphlib.builder.ldconfig(self.app.runcmd, staging_area.dirname) diff --git a/morphlib/builder.py b/morphlib/builder.py index 9b01f983..04ebd149 100644 --- a/morphlib/builder.py +++ b/morphlib/builder.py @@ -125,7 +125,11 @@ def ldconfig(runcmd, rootdir): # pragma: no cover def download_depends(constituents, lac, rac, metadatas=None): for constituent in constituents: if not lac.has(constituent): - lac.copy_from_remote(constituent, rac) + source = rac.get(constituent) + target = lac.put(constituent) + shutil.copyfileobj(source, target) + target.close() + source.close() if metadatas is not None: for metadata in metadatas: if not lac.has_artifact_metadata(constituent, metadata): @@ -242,6 +246,28 @@ class ChunkBuilder(BuilderBase): '''Build chunk artifacts.''' + def create_devices(self, destdir): # pragma: no cover + '''Creates device nodes if the morphology specifies them''' + morphology = self.source.morphology + perms_mask = stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO + if 'devices' in morphology and morphology['devices'] is not None: + for dev in morphology['devices']: + destfile = os.path.join(destdir, './' + dev['filename']) + mode = int(dev['permissions'], 8) & perms_mask + if dev['type'] == 'c': + mode = mode | stat.S_IFCHR + elif dev['type'] == 'b': + mode = mode | stat.S_IFBLK + else: + raise IOError('Cannot create device node %s,' + 'unrecognized device type "%s"' + % (destfile, dev['type'])) + self.app.status(msg="Creating device node %s" + % destfile) + os.mknod(destfile, mode, + os.makedev(dev['major'], dev['minor'])) + os.chown(destfile, dev['uid'], dev['gid']) + def build_and_cache(self): # pragma: no cover with self.build_watch('overall-build'): @@ -260,6 +286,7 @@ class ChunkBuilder(BuilderBase): try: self.get_sources(builddir) self.run_commands(builddir, destdir, temppath, stdout) + self.create_devices(destdir) os.rename(temppath, logpath) except BaseException as e: @@ -432,23 +459,13 @@ class ChunkBuilder(BuilderBase): extra_files += ['baserock/%s.meta' % chunk_artifact_name] parented_paths = parentify(file_paths + extra_files) - self.write_metadata(destdir, chunk_artifact_name, - parented_paths) + with self.local_artifact_cache.put(chunk_artifact) as f: + self.write_metadata(destdir, chunk_artifact_name, + parented_paths) - self.app.status(msg='Creating chunk artifact %(name)s', - name=chunk_artifact_name) - # TODO: This is not concurrency safe, bins.create_chunk will - # fail if tempdir already exists (eg if another build - # has created it). - tempdir = os.path.join(self.app.settings['tempdir'], - chunk_artifact.basename()) - try: - morphlib.bins.create_chunk(destdir, tempdir, - parented_paths) - self.local_artifact_cache.put(tempdir, chunk_artifact) - finally: - if os.path.isdir(tempdir): - shutil.rmtree(tempdir) + self.app.status(msg='Creating chunk artifact %(name)s', + name=chunk_artifact_name) + morphlib.bins.create_chunk(destdir, f, parented_paths) built_artifacts.append(chunk_artifact) for dirname, subdirs, files in os.walk(destdir): @@ -492,13 +509,8 @@ class StratumBuilder(BuilderBase): [x.name for x in constituents]) with lac.put_artifact_metadata(a, 'meta') as f: json.dump(meta, f, indent=4, sort_keys=True) - # TODO: This is not concurrency safe, put_stratum_artifact - # deletes temp which could be in use by another - # build. - temp = os.path.join(self.app.settings['tempdir'], a.name) - with open(temp, 'w+') as f: + with self.local_artifact_cache.put(a) as f: json.dump([c.basename() for c in constituents], f) - self.local_artifact_cache.put_non_ostree_artifact(a, temp) self.save_build_times() return self.source.artifacts.values() @@ -520,40 +532,33 @@ class SystemBuilder(BuilderBase): # pragma: no cover arch = self.source.morphology['arch'] for a_name, artifact in self.source.artifacts.iteritems(): + handle = self.local_artifact_cache.put(artifact) + try: fs_root = self.staging_area.destdir(self.source) self.unpack_strata(fs_root) - upperdir = self.staging_area.overlay_upperdir( - self.source) - editable_root = self.staging_area.overlaydir(self.source) - workdir = os.path.join(self.staging_area.dirname, - 'overlayfs-workdir') - if not os.path.exists(workdir): - os.makedirs(workdir) - union_filesystem = self.app.settings['union-filesystem'] - morphlib.fsutils.overlay_mount(self.app.runcmd, - 'overlay-%s' % a_name, - editable_root, fs_root, - upperdir, workdir, - union_filesystem) - self.write_metadata(editable_root, a_name) - self.run_system_integration_commands(editable_root) - # Put the contents of upperdir into the local artifact - # cache. Don't use editable root as we only want to - # store the modified files. - self.local_artifact_cache.put(upperdir, artifact) + self.write_metadata(fs_root, a_name) + self.run_system_integration_commands(fs_root) + unslashy_root = fs_root[1:] + def uproot_info(info): + info.name = relpath(info.name, unslashy_root) + if info.islnk(): + info.linkname = relpath(info.linkname, + unslashy_root) + return info + tar = tarfile.open(fileobj=handle, mode="w", name=a_name) + self.app.status(msg='Constructing tarball of rootfs', + chatty=True) + tar.add(fs_root, recursive=True, filter=uproot_info) + tar.close() except BaseException as e: logging.error(traceback.format_exc()) self.app.status(msg='Error while building system', error=True) - if editable_root and os.path.exists(editable_root): - morphlib.fsutils.unmount(self.app.runcmd, - editable_root) + handle.abort() raise else: - if editable_root and os.path.exists(editable_root): - morphlib.fsutils.unmount(self.app.runcmd, - editable_root) + handle.close() self.save_build_times() return self.source.artifacts.itervalues() @@ -562,12 +567,13 @@ class SystemBuilder(BuilderBase): # pragma: no cover '''Unpack a single stratum into a target directory''' cache = self.local_artifact_cache - with open(cache.get(stratum_artifact), 'r') as stratum_file: + with cache.get(stratum_artifact) as stratum_file: artifact_list = json.load(stratum_file, encoding='unicode-escape') for chunk in (ArtifactCacheReference(a) for a in artifact_list): - self.app.status(msg='Checkout chunk %(basename)s', + self.app.status(msg='Unpacking chunk %(basename)s', basename=chunk.basename(), chatty=True) - cache.get(chunk, target) + with cache.get(chunk) as chunk_file: + morphlib.bins.unpack_binary_from_file(chunk_file, target) target_metadata = os.path.join( target, 'baserock', '%s.meta' % stratum_artifact.name) @@ -578,7 +584,7 @@ class SystemBuilder(BuilderBase): # pragma: no cover def unpack_strata(self, path): '''Unpack strata into a directory.''' - self.app.status(msg='Checking out strata to %(path)s', + self.app.status(msg='Unpacking strata to %(path)s', path=path, chatty=True) with self.build_watch('unpack-strata'): for a_name, a in self.source.artifacts.iteritems(): @@ -590,14 +596,12 @@ class SystemBuilder(BuilderBase): # pragma: no cover # download the chunk artifacts if necessary for stratum_artifact in self.source.dependencies: - stratum_path = self.local_artifact_cache.get( - stratum_artifact) - with open(stratum_path, 'r') as stratum: - chunks = [ArtifactCacheReference(c) - for c in json.load(stratum)] + f = self.local_artifact_cache.get(stratum_artifact) + chunks = [ArtifactCacheReference(c) for c in json.load(f)] download_depends(chunks, self.local_artifact_cache, self.remote_artifact_cache) + f.close() # unpack it from the local artifact cache for stratum_artifact in self.source.dependencies: diff --git a/morphlib/builder_tests.py b/morphlib/builder_tests.py index b5e66521..a571e3d0 100644 --- a/morphlib/builder_tests.py +++ b/morphlib/builder_tests.py @@ -105,8 +105,8 @@ class FakeArtifactCache(object): def __init__(self): self._cached = {} - def put(self, artifact, directory): - self._cached[(artifact.cache_key, artifact.name)] = artifact.name + def put(self, artifact): + return FakeFileHandle(self, (artifact.cache_key, artifact.name)) def put_artifact_metadata(self, artifact, name): return FakeFileHandle(self, (artifact.cache_key, artifact.name, name)) @@ -114,7 +114,7 @@ class FakeArtifactCache(object): def put_source_metadata(self, source, cachekey, name): return FakeFileHandle(self, (cachekey, name)) - def get(self, artifact, directory=None): + def get(self, artifact): return StringIO.StringIO( self._cached[(artifact.cache_key, artifact.name)]) @@ -134,10 +134,6 @@ class FakeArtifactCache(object): def has_source_metadata(self, source, cachekey, name): return (cachekey, name) in self._cached - def copy_from_remote(self, artifact, remote): - self._cached[(artifact.cache_key, artifact.name)] = \ - remote._cached[(artifact.cache_key, artifact.name)] - class BuilderBaseTests(unittest.TestCase): @@ -195,7 +191,9 @@ class BuilderBaseTests(unittest.TestCase): rac = FakeArtifactCache() afacts = [FakeArtifact(name) for name in ('a', 'b', 'c')] for a in afacts: - rac.put(a, 'not-a-dir') + fh = rac.put(a) + fh.write(a.name) + fh.close() morphlib.builder.download_depends(afacts, lac, rac) self.assertTrue(all(lac.has(a) for a in afacts)) @@ -204,7 +202,9 @@ class BuilderBaseTests(unittest.TestCase): rac = FakeArtifactCache() afacts = [FakeArtifact(name) for name in ('a', 'b', 'c')] for a in afacts: - rac.put(a, 'not-a-dir') + fh = rac.put(a) + fh.write(a.name) + fh.close() fh = rac.put_artifact_metadata(a, 'meta') fh.write('metadata') fh.close() diff --git a/morphlib/exts/fstab.configure b/morphlib/exts/fstab.configure index 3bbc9102..b9154eee 100755 --- a/morphlib/exts/fstab.configure +++ b/morphlib/exts/fstab.configure @@ -1,5 +1,6 @@ -#!/usr/bin/python -# Copyright (C) 2013,2015 Codethink Limited +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# Copyright © 2013-2015 Codethink Limited # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -19,21 +20,9 @@ import os import sys +import morphlib -def asciibetical(strings): +envvars = {k: v for (k, v) in os.environ.iteritems() if k.startswith('FSTAB_')} - def key(s): - return [ord(c) for c in s] - - return sorted(strings, key=key) - - -fstab_filename = os.path.join(sys.argv[1], 'etc', 'fstab') - -fstab_vars = asciibetical(x for x in os.environ if x.startswith('FSTAB_')) -with open(fstab_filename, 'a') as f: - for var in fstab_vars: - f.write('%s\n' % os.environ[var]) - -os.chown(fstab_filename, 0, 0) -os.chmod(fstab_filename, 0644) +conf_file = os.path.join(sys.argv[1], 'etc/fstab') +morphlib.util.write_from_dict(conf_file, envvars) diff --git a/morphlib/exts/hosts.configure b/morphlib/exts/hosts.configure new file mode 100755 index 00000000..6b068d04 --- /dev/null +++ b/morphlib/exts/hosts.configure @@ -0,0 +1,48 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# Copyright © 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# =*= License: GPL-2 =*= + + +import os +import sys +import socket + +import morphlib + +def validate(var, line): + xs = line.split() + if len(xs) == 0: + raise morphlib.Error("`%s: %s': line is empty" % (var, line)) + + ip = xs[0] + hostnames = xs[1:] + + if len(hostnames) == 0: + raise morphlib.Error("`%s: %s': missing hostname" % (var, line)) + + family = socket.AF_INET6 if ':' in ip else socket.AF_INET + + try: + socket.inet_pton(family, ip) + except socket.error: + raise morphlib.Error("`%s: %s' invalid ip" % (var, ip)) + +envvars = {k: v for (k, v) in os.environ.iteritems() if k.startswith('HOSTS_')} + +conf_file = os.path.join(sys.argv[1], 'etc/hosts') +morphlib.util.write_from_dict(conf_file, envvars, validate) diff --git a/morphlib/fsutils.py b/morphlib/fsutils.py index 400ff7d8..a3b73bf6 100644 --- a/morphlib/fsutils.py +++ b/morphlib/fsutils.py @@ -46,33 +46,14 @@ def create_fs(runcmd, partition): # pragma: no cover runcmd(['mkfs.btrfs', '-L', 'baserock', partition]) -def mount(runcmd, partition, mount_point, - fstype=None, options=[]): # pragma: no cover +def mount(runcmd, partition, mount_point, fstype=None): # pragma: no cover if not os.path.exists(mount_point): os.mkdir(mount_point) if not fstype: fstype = [] else: fstype = ['-t', fstype] - if not type(options) == list: - options = [options] - runcmd(['mount', partition, mount_point] + fstype + options) - - -def overlay_mount(runcmd, partition, mount_point, - lowerdir, upperdir, workdir, method): # pragma: no cover - if method == 'overlayfs': - options = '-olowerdir=%s,upperdir=%s,workdir=%s' % \ - (lowerdir, upperdir, workdir) - mount(runcmd, partition, mount_point, 'overlay', options) - elif method == 'unionfs': - if not os.path.exists(mount_point): - os.mkdir(mount_point) - dir_string = '%s=RW:%s=RO' % (upperdir, lowerdir) - runcmd(['unionfs', '-o', 'cow', dir_string, mount_point]) - else: - raise cliapp.AppException('Union filesystem %s not supported' % - method) + runcmd(['mount', partition, mount_point] + fstype) def unmount(runcmd, mount_point): # pragma: no cover diff --git a/morphlib/ostree.py b/morphlib/ostree.py deleted file mode 100644 index a2c133f2..00000000 --- a/morphlib/ostree.py +++ /dev/null @@ -1,139 +0,0 @@ -from gi.repository import OSTree -from gi.repository import Gio -from gi.repository import GLib - -import os - - -class OSTreeRepo(object): - - """Class to wrap the OSTree API.""" - - OSTREE_GIO_FAST_QUERYINFO = 'standard::name,standard::type,standard::' \ - 'size,standard::is-symlink,standard::syml' \ - 'ink-target,unix::device,unix::inode,unix' \ - '::mode,unix::uid,unix::gid,unix::rdev' - G_FILE_QUERY_INFO_NOFOLLOW_SYMLINKS = Gio.FileQueryInfoFlags(1) - cancellable = Gio.Cancellable.new() - - def __init__(self, path, disable_fsync=True): - self.path = path - self.repo = self._open_repo(path, disable_fsync) - - def _open_repo(self, path, disable_fsync=True): - """Create and open and OSTree.Repo, and return it.""" - repo_dir = Gio.file_new_for_path(path) - repo = OSTree.Repo.new(repo_dir) - try: - repo.open(self.cancellable) - except GLib.GError: - if not os.path.exists(path): - os.makedirs(path) - repo.create(OSTree.RepoMode.ARCHIVE_Z2, self.cancellable) - repo.set_disable_fsync(disable_fsync) - return repo - - def refsdir(self): - """Return the abspath to the refs/heads directory in the repo.""" - return os.path.join(os.path.abspath(self.path), 'refs/heads') - - def touch_ref(self, ref): - """Update the mtime of a ref file in repo/refs/heads.""" - os.utime(os.path.join(self.refsdir(), ref), None) - - def resolve_rev(self, branch, allow_noent=True): - """Return the SHA256 corresponding to 'branch'.""" - return self.repo.resolve_rev(branch, allow_noent)[1] - - def read_commit(self, branch): - """Return an OSTree.RepoFile representing a committed tree.""" - return self.repo.read_commit(branch, self.cancellable)[1] - - def query_info(self, file_object): - """Quickly return a Gio.FileInfo for file_object.""" - return file_object.query_info(self.OSTREE_GIO_FAST_QUERYINFO, - self.G_FILE_QUERY_INFO_NOFOLLOW_SYMLINKS, - self.cancellable) - - def checkout(self, branch, destdir): - """Checkout branch into destdir.""" - checkout_path = destdir - if not os.path.exists(checkout_path): - os.makedirs(checkout_path) - checkout = Gio.file_new_for_path(checkout_path) - - commit = self.read_commit(branch) - commit_info = self.query_info(commit) - self.repo.checkout_tree(0, 1, checkout, commit, - commit_info, self.cancellable) - - def commit(self, subject, srcdir, branch): - """Commit the contents of 'srcdir' to 'branch'.""" - self.repo.prepare_transaction(self.cancellable) - parent = self.resolve_rev(branch) - if parent: - parent_root = self.read_commit(parent) - - mtree = OSTree.MutableTree() - src = Gio.file_new_for_path(srcdir) - self.repo.write_directory_to_mtree(src, mtree, None, self.cancellable) - root = self.repo.write_mtree(mtree, self.cancellable)[1] - if parent and root.equal(parent_root): - return - checksum = self.repo.write_commit(parent, subject, '', None, - root, self.cancellable)[1] - self.repo.transaction_set_ref(None, branch, checksum) - stats = self.repo.commit_transaction(self.cancellable) - - def cat_file(self, ref, path): - """Return the file descriptor of path at ref.""" - commit = self.read_commit(ref) - relative = commit.resolve_relative_path(path) - ret, content, etag = relative.load_contents() - return content - - def list_refs(self, resolved=False): - """Return a list of all refs in the repo.""" - refs = self.repo.list_refs()[1] - if not resolved: - return refs.keys() - return refs - - def delete_ref(self, ref): - """Remove refspec from the repo.""" - if not self.list_refs(ref): - raise Exception("Failed to delete ref, it doesn't exist") - self.repo.set_ref_immediate(None, ref, None, self.cancellable) - - def prune(self): - """Remove unreachable objects from the repo.""" - return self.repo.prune(OSTree.RepoPruneFlags.REFS_ONLY, - -1, self.cancellable) - - def add_remote(self, name, url): - """Add a remote with a given name and url.""" - options_type = GLib.VariantType.new('a{sv}') - options_builder = GLib.VariantBuilder.new(options_type) - options = options_builder.end() - self.repo.remote_add(name, url, options, self.cancellable) - - def remove_remote(self, name): - """Remove a remote with a given name.""" - self.repo.remote_delete(name, self.cancellable) - - def get_remote_url(self, name): - """Return the URL for a remote.""" - return self.repo.remote_get_url(name)[1] - - def list_remotes(self): - """Return a list of all remotes for this repo.""" - return self.repo.remote_list() - - def has_remote(self, name): - """Return True if name is a remote for the repo.""" - return name in self.list_remotes() - - def pull(self, refs, remote): - """Pull ref from remote into the local repo.""" - flags = OSTree.RepoPullFlags.NONE - self.repo.pull(remote, refs, flags, None, self.cancellable) diff --git a/morphlib/ostreeartifactcache.py b/morphlib/ostreeartifactcache.py deleted file mode 100644 index fdb7cb5d..00000000 --- a/morphlib/ostreeartifactcache.py +++ /dev/null @@ -1,229 +0,0 @@ -# Copyright (C) 2015 Codethink Limited -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - -import collections -import logging -import os -import shutil -import tarfile -import tempfile - -import cliapp -from gi.repository import GLib - -import morphlib -from morphlib.artifactcachereference import ArtifactCacheReference - -class OSTreeArtifactCache(object): - """Class to provide the artifact cache API using an OSTree repo.""" - - def __init__(self, cachedir): - repo_dir = os.path.join(cachedir, 'repo') - self.repo = morphlib.ostree.OSTreeRepo(repo_dir) - self.cachedir = cachedir - - def _get_file_from_remote(self, artifact, remote, metadata_name=None): - if metadata_name: - handle = remote.get_artifact_metadata(artifact, metadata_name) - else: - handle = remote.get(artifact) - fd, path = tempfile.mkstemp() - with open(path, 'w+') as temp: - shutil.copyfileobj(handle, temp) - return path - - def _get_artifact_cache_name(self, artifact): - logging.debug('LAC: %s' % artifact.basename()) - cache_key, kind, name = artifact.basename().split('.', 2) - suffix = name.split('-')[-1] - return '%s-%s' % (cache_key, suffix) - - def put(self, directory, artifact): - """Commit the contents of 'directory' to the repo. - - This uses the artifact name and cache key to create the ref, so the - contents of directory should be the contents of the artifact. - - """ - ref = self._get_artifact_cache_name(artifact) - subject = artifact.name - try: - logging.debug('Committing %s to artifact cache at %s.' % - (subject, ref)) - self.repo.commit(subject, directory, ref) - except GLib.GError as e: - logging.debug('OSTree raised an exception: %s' % e) - raise cliapp.AppException('Failed to commit %s to artifact ' - 'cache.' % ref) - - def put_non_ostree_artifact(self, artifact, location, metadata_name=None): - """Store a single file in the artifact cachedir.""" - if metadata_name: - filename = self._artifact_metadata_filename(artifact, - metadata_name) - else: - filename = self.artifact_filename(artifact) - shutil.copy(location, filename) - os.remove(location) - - def copy_from_remote(self, artifact, remote): - """Get 'artifact' from remote artifact cache and store it locally.""" - if remote.method == 'tarball': - logging.debug('Downloading artifact tarball for %s.' % - artifact.name) - location = self._get_file_from_remote(artifact, remote) - try: - tempdir = tempfile.mkdtemp() - with tarfile.open(name=location) as tf: - tf.extractall(path=tempdir) - try: - self.put(tempdir, artifact) - finally: - os.remove(location) - shutil.rmtree(tempdir) - except tarfile.ReadError: - # Reading the artifact as a tarball failed, so it must be a - # single file (for example a stratum artifact). - self.put_non_ostree_artifact(artifact, location) - - elif remote.method == 'ostree': - logging.debug('Pulling artifact for %s from remote.' % - artifact.basename()) - try: - ref = self._get_artifact_cache_name(artifact) - except Exception: - # if we can't split the name properly, we must want metadata - a, name = artifact.basename().split('.', 1) - location = self._get_file_from_remote( - ArtifactCacheReference(a), remote, name) - self.put_non_ostree_artifact(artifact, location, name) - return - - if artifact.basename().split('.', 2)[1] == 'stratum': - location = self._get_file_from_remote(artifact, remote) - self.put_non_ostree_artifact(artifact, location) - return - - try: - if not self.repo.has_remote(remote.name): - self.repo.add_remote(remote.name, remote.ostree_url) - self.repo.pull([ref], remote.name) - except GLib.GError as e: - logging.debug('OSTree raised an exception: %s' % e) - raise cliapp.AppException('Failed to pull %s from remote ' - 'cache.' % ref) - - def get(self, artifact, directory=None): - """Checkout an artifact from the repo and return its location.""" - cache_key, kind, name = artifact.basename().split('.', 2) - if kind == 'stratum': - return self.artifact_filename(artifact) - if directory is None: - directory = tempfile.mkdtemp() - ref = self._get_artifact_cache_name(artifact) - try: - self.repo.checkout(ref, directory) - self.repo.touch_ref(ref) - except GLib.GError as e: - logging.debug('OSTree raised an exception: %s' % e) - raise cliapp.AppException('Failed to checkout %s from artifact ' - 'cache.' % ref) - return directory - - def list_contents(self): - """Return the set of sources cached and related information. - - returns a [(cache_key, set(artifacts), last_used)] - - """ - CacheInfo = collections.namedtuple('CacheInfo', ('artifacts', 'mtime')) - contents = collections.defaultdict(lambda: CacheInfo(set(), 0)) - for ref in self.repo.list_refs(): - cachekey = ref[:63] - artifact = ref[65:] - artifacts, max_mtime = contents[cachekey] - artifacts.add(artifact) - ref_filename = os.path.join(self.repo.refsdir(), ref) - mtime = os.path.getmtime(ref_filename) - contents[cachekey] = CacheInfo(artifacts, max(max_mtime, mtime)) - return ((cache_key, info.artifacts, info.mtime) - for cache_key, info in contents.iteritems()) - - def remove(self, cachekey): - """Remove all artifacts associated with the given cachekey.""" - for ref in (r for r in self.repo.list_refs() - if r.startswith(cachekey)): - self.repo.delete_ref(ref) - - def prune(self): - """Delete orphaned objects in the repo.""" - self.repo.prune() - - def has(self, artifact): - cachekey, kind, name = artifact.basename().split('.', 2) - logging.debug('OSTreeArtifactCache: got %s, %s, %s' % - (cachekey, kind, name)) - if self._get_artifact_cache_name(artifact) in self.repo.list_refs(): - self.repo.touch_ref(self._get_artifact_cache_name(artifact)) - return True - if kind == 'stratum' and \ - self._has_file(self.artifact_filename(artifact)): - return True - return False - - def get_artifact_metadata(self, artifact, name): - filename = self._artifact_metadata_filename(artifact, name) - os.utime(filename, None) - return open(filename) - - def get_source_metadata_filename(self, source, cachekey, name): - return self._source_metadata_filename(source, cachekey, name) - - def get_source_metadata(self, source, cachekey, name): - filename = self._source_metadata_filename(source, cachekey, name) - os.utime(filename, None) - return open(filename) - - def artifact_filename(self, artifact): - return os.path.join(self.cachedir, artifact.basename()) - - def _artifact_metadata_filename(self, artifact, name): - return os.path.join(self.cachedir, artifact.metadata_basename(name)) - - def _source_metadata_filename(self, source, cachekey, name): - return os.path.join(self.cachedir, '%s.%s' % (cachekey, name)) - - def put_artifact_metadata(self, artifact, name): - filename = self._artifact_metadata_filename(artifact, name) - return morphlib.savefile.SaveFile(filename, mode='w') - - def put_source_metadata(self, source, cachekey, name): - filename = self._source_metadata_filename(source, cachekey, name) - return morphlib.savefile.SaveFile(filename, mode='w') - - def _has_file(self, filename): - if os.path.exists(filename): - os.utime(filename, None) - return True - return False - - def has_artifact_metadata(self, artifact, name): - filename = self._artifact_metadata_filename(artifact, name) - return self._has_file(filename) - - def has_source_metadata(self, source, cachekey, name): - filename = self._source_metadata_filename(source, cachekey, name) - return self._has_file(filename) diff --git a/morphlib/plugins/deploy_plugin.py b/morphlib/plugins/deploy_plugin.py index c9890b13..7635a7b4 100644 --- a/morphlib/plugins/deploy_plugin.py +++ b/morphlib/plugins/deploy_plugin.py @@ -24,7 +24,6 @@ import uuid import cliapp import morphlib -from morphlib.artifactcachereference import ArtifactCacheReference class DeployPlugin(cliapp.Plugin): @@ -440,8 +439,6 @@ class DeployPlugin(cliapp.Plugin): system_status_prefix = '%s[%s]' % (old_status_prefix, system['morph']) self.app.status_prefix = system_status_prefix try: - system_tree = None - # Find the artifact to build morph = morphlib.util.sanitise_morphology_path(system['morph']) srcpool = build_command.create_source_pool(build_repo, ref, morph) @@ -505,9 +502,6 @@ class DeployPlugin(cliapp.Plugin): system_tree, deploy_location) finally: self.app.status_prefix = system_status_prefix - if system_tree and os.path.exists(system_tree): - morphlib.fsutils.unmount(self.app.runcmd, system_tree) - shutil.rmtree(system_tree) finally: self.app.status_prefix = old_status_prefix @@ -541,94 +535,46 @@ class DeployPlugin(cliapp.Plugin): except morphlib.extensions.ExtensionNotFoundError: pass - def checkout_stratum(self, path, artifact, lac, rac): - with open(lac.get(artifact), 'r') as stratum: - chunks = [ArtifactCacheReference(c) for c in json.load(stratum)] - morphlib.builder.download_depends(chunks, lac, rac) - for chunk in chunks: - self.app.status(msg='Checkout chunk %(name)s.', - name=chunk.basename(), chatty=True) - lac.get(chunk, path) - - metadata = os.path.join(path, 'baserock', '%s.meta' % artifact.name) - with lac.get_artifact_metadata(artifact, 'meta') as meta_src: - with morphlib.savefile.SaveFile(metadata, 'w') as meta_dst: - shutil.copyfileobj(meta_src, meta_dst) - - def checkout_strata(self, path, artifact, lac, rac): - deps = artifact.source.dependencies - morphlib.builder.download_depends(deps, lac, rac) - for stratum in deps: - self.checkout_stratum(path, stratum, lac, rac) - morphlib.builder.ldconfig(self.app.runcmd, path) - def setup_deploy(self, build_command, deploy_tempdir, root_repo_dir, ref, artifact, deployment_type, location, env): # deployment_type, location and env are only used for saving metadata - deployment_dir = tempfile.mkdtemp(dir=deploy_tempdir) - # Create a tempdir to extract the rootfs in - system_tree = tempfile.mkdtemp(dir=deployment_dir) - - # Create temporary directory for overlayfs - overlay_dir = os.path.join(deployment_dir, - '%s-upperdir' % artifact.name) - if not os.path.exists(overlay_dir): - os.makedirs(overlay_dir) - work_dir = os.path.join(deployment_dir, '%s-workdir' % artifact.name) - if not os.path.exists(work_dir): - os.makedirs(work_dir) - - deploy_tree = os.path.join(deployment_dir, - 'overlay-deploy-%s' % artifact.name) + system_tree = tempfile.mkdtemp(dir=deploy_tempdir) + try: - # Checkout the strata involved in the artifact into a tempdir - self.app.status(msg='Checking out strata in system') - self.checkout_strata(system_tree, artifact, - build_command.lac, build_command.rac) + # Unpack the artifact (tarball) to a temporary directory. + self.app.status(msg='Unpacking system for configuration') - self.app.status(msg='Checking out system for configuration') if build_command.lac.has(artifact): - build_command.lac.get(artifact, system_tree) + f = build_command.lac.get(artifact) elif build_command.rac.has(artifact): build_command.cache_artifacts_locally([artifact]) - build_command.lac.get(artifact, system_tree) + f = build_command.lac.get(artifact) else: raise cliapp.AppException('Deployment failed as system is' ' not yet built.\nPlease ensure' ' the system is built before' ' deployment.') + tf = tarfile.open(fileobj=f) + tf.extractall(path=system_tree) self.app.status( - msg='System checked out at %(system_tree)s', + msg='System unpacked at %(system_tree)s', system_tree=system_tree) - union_filesystem = self.app.settings['union-filesystem'] - morphlib.fsutils.overlay_mount(self.app.runcmd, - 'overlay-deploy-%s' % - artifact.name, - deploy_tree, system_tree, - overlay_dir, work_dir, - union_filesystem) - self.app.status( msg='Writing deployment metadata file') metadata = self.create_metadata( artifact, root_repo_dir, deployment_type, location, env) metadata_path = os.path.join( - deploy_tree, 'baserock', 'deployment.meta') + system_tree, 'baserock', 'deployment.meta') with morphlib.savefile.SaveFile(metadata_path, 'w') as f: json.dump(metadata, f, indent=4, sort_keys=True, encoding='unicode-escape') - return deploy_tree + return system_tree except Exception: - if deploy_tree and os.path.exists(deploy_tree): - morphlib.fsutils.unmount(self.app.runcmd, deploy_tree) - shutil.rmtree(deploy_tree) shutil.rmtree(system_tree) - shutil.rmtree(overlay_dir) - shutil.rmtree(work_dir) raise def run_deploy_commands(self, deploy_tempdir, env, artifact, root_repo_dir, diff --git a/morphlib/plugins/gc_plugin.py b/morphlib/plugins/gc_plugin.py index 8b5dc4c2..71522b04 100644 --- a/morphlib/plugins/gc_plugin.py +++ b/morphlib/plugins/gc_plugin.py @@ -125,8 +125,8 @@ class GCPlugin(cliapp.Plugin): 'sufficient space already cleared', chatty=True) return - lac = morphlib.ostreeartifactcache.OSTreeArtifactCache( - os.path.join(cache_path, 'artifacts')) + lac = morphlib.localartifactcache.LocalArtifactCache( + fs.osfs.OSFS(os.path.join(cache_path, 'artifacts'))) max_age, min_age = self.calculate_delete_range() logging.debug('Must remove artifacts older than timestamp %d' % max_age) @@ -144,8 +144,6 @@ class GCPlugin(cliapp.Plugin): lac.remove(cachekey) removed += 1 - lac.prune() - # Maybe remove remaining middle-aged artifacts for cachekey in may_delete: if sufficient_free(): @@ -159,8 +157,6 @@ class GCPlugin(cliapp.Plugin): lac.remove(cachekey) removed += 1 - lac.prune() - if sufficient_free(): self.app.status(msg='Made sufficient space in %(cache_path)s ' 'after removing %(removed)d sources', diff --git a/morphlib/remoteartifactcache.py b/morphlib/remoteartifactcache.py index f5115cd6..427e4cbb 100644 --- a/morphlib/remoteartifactcache.py +++ b/morphlib/remoteartifactcache.py @@ -57,18 +57,6 @@ class RemoteArtifactCache(object): def __init__(self, server_url): self.server_url = server_url - self.name = urlparse.urlparse(server_url).hostname - try: - self.method = self._get_method() - except urllib2.URLError: - self.method = 'tarball' - except Exception as e: # pragma: no cover - logging.debug('Failed to determine cache method: %s' % e) - raise cliapp.AppException('Failed to determine method used by ' - 'remote cache.') - if self.method == 'ostree': # pragma: no cover - self.ostree_url = 'http://%s:%s/' % (self.name, - self._get_ostree_info()) def has(self, artifact): return self._has_file(artifact.basename()) @@ -124,18 +112,5 @@ class RemoteArtifactCache(object): server_url, '/1.0/artifacts?filename=%s' % urllib.quote(filename)) - def _get_method(self): # pragma: no cover - logging.debug('Getting cache method of %s' % self.server_url) - request_url = urlparse.urljoin(self.server_url, '/1.0/method') - req = urllib2.urlopen(request_url) - return req.read() - - def _get_ostree_info(self): # pragma: no cover - logging.debug('Getting OSTree repo info.') - request_url = urlparse.urljoin(self.server_url, '/1.0/ostreeinfo') - logging.debug('sending %s' % request_url) - req = urllib2.urlopen(request_url) - return req.read() - def __str__(self): # pragma: no cover return self.server_url diff --git a/morphlib/sourceresolver.py b/morphlib/sourceresolver.py index d2b47d35..1e64c23a 100644 --- a/morphlib/sourceresolver.py +++ b/morphlib/sourceresolver.py @@ -31,7 +31,7 @@ tree_cache_filename = 'trees.cache.pickle' buildsystem_cache_size = 10000 buildsystem_cache_filename = 'detected-chunk-buildsystems.cache.pickle' -not_supported_versions = [] +supported_versions = [0, 1] class PickleCacheManager(object): # pragma: no cover '''Cache manager for PyLRU that reads and writes to Pickle files. @@ -346,6 +346,29 @@ class SourceResolver(object): loader.set_defaults(morph) return morph + def _parse_version_file(self, version_file): # pragma : no cover + '''Parse VERSION file and return the version of the format if: + + VERSION is a YAML file + and it's a dict + and has the key 'version' + and the type stored in the 'version' key is an int + and that int is not in the supported format + + otherwise returns None + + ''' + version = None + + yaml_obj = yaml.safe_load(version_file) + if yaml_obj is not None: + if type(yaml_obj) is dict: + if 'version' in yaml_obj.keys(): + if type(yaml_obj['version']) is int: + version = yaml_obj['version'] + + return version + def _check_version_file(self,definitions_repo, definitions_absref): # pragma: no cover version_file = self._get_file_contents( @@ -354,13 +377,10 @@ class SourceResolver(object): if version_file is None: return - try: - version = yaml.safe_load(version_file)['version'] - except (yaml.error.YAMLError, KeyError, TypeError): - version = 0 - - if version in not_supported_versions: - raise UnknownVersionError(version) + version = self._parse_version_file(version_file) + if version is not None: + if version not in supported_versions: + raise UnknownVersionError(version) def _process_definitions_with_children(self, system_filenames, definitions_repo, diff --git a/morphlib/stagingarea.py b/morphlib/stagingarea.py index 768ec643..8c2781aa 100644 --- a/morphlib/stagingarea.py +++ b/morphlib/stagingarea.py @@ -87,14 +87,6 @@ class StagingArea(object): return self._dir_for_source(source, 'inst') - def overlay_upperdir(self, source): - '''Create a directory to be upperdir for overlayfs, and return it.''' - return self._dir_for_source(source, 'overlay_upper') - - def overlaydir(self, source): - '''Create a directory to be a mount point for overlayfs, return it''' - return self._dir_for_source(source, 'overlay') - def relative(self, filename): '''Return a filename relative to the staging area.''' @@ -154,42 +146,37 @@ class StagingArea(object): raise IOError('Cannot extract %s into staging-area. Unsupported' ' type.' % srcpath) - def create_devices(self, morphology): # pragma: no cover - '''Creates device nodes if the morphology specifies them''' - perms_mask = stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO - if 'devices' in morphology and morphology['devices'] is not None: - for dev in morphology['devices']: - destfile = os.path.join(self.dirname, './' + dev['filename']) - mode = int(dev['permissions'], 8) & perms_mask - if dev['type'] == 'c': - mode = mode | stat.S_IFCHR - elif dev['type'] == 'b': - mode = mode | stat.S_IFBLK - else: - raise IOError('Cannot create device node %s,' - 'unrecognized device type "%s"' - % (destfile, dev['type'])) - parent = os.path.dirname(destfile) - if not os.path.exists(parent): - os.makedirs(parent) - if not os.path.exists(destfile): - logging.debug("Creating device node %s" % destfile) - os.mknod(destfile, mode, - os.makedev(dev['major'], dev['minor'])) - os.chown(destfile, dev['uid'], dev['gid']) - - def install_artifact(self, artifact, artifact_checkout): + def install_artifact(self, handle): '''Install a build artifact into the staging area. We access the artifact via an open file handle. For now, we assume the artifact is a tarball. ''' + + chunk_cache_dir = os.path.join(self._app.settings['tempdir'], 'chunks') + unpacked_artifact = os.path.join( + chunk_cache_dir, os.path.basename(handle.name) + '.d') + if not os.path.exists(unpacked_artifact): + self._app.status( + msg='Unpacking chunk from cache %(filename)s', + filename=os.path.basename(handle.name)) + savedir = tempfile.mkdtemp(dir=chunk_cache_dir) + try: + morphlib.bins.unpack_binary_from_file( + handle, savedir + '/') + except BaseException as e: # pragma: no cover + shutil.rmtree(savedir) + raise + # TODO: This rename is not concurrency safe if two builds are + # extracting the same chunk, one build will fail because + # the other renamed its tempdir here first. + os.rename(savedir, unpacked_artifact) + if not os.path.exists(self.dirname): self._mkdir(self.dirname) - self.hardlink_all_files(artifact_checkout, self.dirname) - self.create_devices(artifact.source.morphology) + self.hardlink_all_files(unpacked_artifact, self.dirname) def remove(self): '''Remove the entire staging area. diff --git a/morphlib/stagingarea_tests.py b/morphlib/stagingarea_tests.py index ffdf5eaa..97d78236 100644 --- a/morphlib/stagingarea_tests.py +++ b/morphlib/stagingarea_tests.py @@ -30,7 +30,6 @@ class FakeBuildEnvironment(object): } self.extra_path = ['/extra-path'] - class FakeSource(object): def __init__(self): @@ -40,12 +39,6 @@ class FakeSource(object): self.name = 'le-name' -class FakeArtifact(object): - - def __init__(self): - self.source = FakeSource() - - class FakeApplication(object): def __init__(self, cachedir, tempdir): @@ -90,8 +83,12 @@ class StagingAreaTests(unittest.TestCase): os.mkdir(chunkdir) with open(os.path.join(chunkdir, 'file.txt'), 'w'): pass + chunk_tar = os.path.join(self.tempdir, 'chunk.tar') + tf = tarfile.TarFile(name=chunk_tar, mode='w') + tf.add(chunkdir, arcname='.') + tf.close() - return chunkdir + return chunk_tar def list_tree(self, root): files = [] @@ -121,34 +118,20 @@ class StagingAreaTests(unittest.TestCase): self.assertEqual(self.created_dirs, [dirname]) self.assertTrue(dirname.startswith(self.staging)) - def test_creates_overlay_upper_directory(self): - source = FakeSource() - self.sa._mkdir = self.fake_mkdir - dirname = self.sa.overlay_upperdir(source) - self.assertEqual(self.created_dirs, [dirname]) - self.assertTrue(dirname.startswith(self.staging)) - - def test_creates_overlay_directory(self): - source = FakeSource() - self.sa._mkdir = self.fake_mkdir - dirname = self.sa.overlaydir(source) - self.assertEqual(self.created_dirs, [dirname]) - self.assertTrue(dirname.startswith(self.staging)) - def test_makes_relative_name(self): filename = os.path.join(self.staging, 'foobar') self.assertEqual(self.sa.relative(filename), '/foobar') def test_installs_artifact(self): - artifact = FakeArtifact() - chunkdir = self.create_chunk() - self.sa.install_artifact(artifact, chunkdir) + chunk_tar = self.create_chunk() + with open(chunk_tar, 'rb') as f: + self.sa.install_artifact(f) self.assertEqual(self.list_tree(self.staging), ['/', '/file.txt']) def test_removes_everything(self): - artifact = FakeArtifact() - chunkdir = self.create_chunk() - self.sa.install_artifact(artifact, chunkdir) + chunk_tar = self.create_chunk() + with open(chunk_tar, 'rb') as f: + self.sa.install_artifact(f) self.sa.remove() self.assertFalse(os.path.exists(self.staging)) diff --git a/morphlib/util.py b/morphlib/util.py index 00111ff7..e733af9d 100644 --- a/morphlib/util.py +++ b/morphlib/util.py @@ -131,10 +131,8 @@ def new_artifact_caches(settings): # pragma: no cover if not os.path.exists(artifact_cachedir): os.mkdir(artifact_cachedir) - #lac = morphlib.localartifactcache.LocalArtifactCache( - # fs.osfs.OSFS(artifact_cachedir)) - - lac = morphlib.ostreeartifactcache.OSTreeArtifactCache(artifact_cachedir) + lac = morphlib.localartifactcache.LocalArtifactCache( + fs.osfs.OSFS(artifact_cachedir)) rac_url = get_artifact_cache_server(settings) rac = None @@ -646,3 +644,35 @@ def error_message_for_containerised_commandline( 'Containerisation settings: %s\n' \ 'Error output:\n%s' \ % (argv_string, container_kwargs, err) + + +def write_from_dict(filepath, d, validate=lambda x, y: True): #pragma: no cover + '''Takes a dictionary and appends the contents to a file + + An optional validation callback can be passed to perform validation on + each value in the dictionary. + + e.g. + + def validation_callback(dictionary_key, dictionary_value): + if not dictionary_value.isdigit(): + raise Exception('value contains non-digit character(s)') + + Any callback supplied to this function should raise an exception + if validation fails. + ''' + + # Sort items asciibetically + # the output of the deployment should not depend + # on the locale of the machine running the deployment + items = sorted(d.iteritems(), key=lambda (k, v): [ord(c) for c in v]) + + for (k, v) in items: + validate(k, v) + + with open(filepath, 'a') as f: + for (_, v) in items: + f.write('%s\n' % v) + + os.fchown(f.fileno(), 0, 0) + os.fchmod(f.fileno(), 0644) diff --git a/morphlib/writeexts.py b/morphlib/writeexts.py index 129b2bc4..aa185a2b 100644 --- a/morphlib/writeexts.py +++ b/morphlib/writeexts.py @@ -604,12 +604,16 @@ class WriteExtension(cliapp.Application): def check_ssh_connectivity(self, ssh_host): try: - cliapp.ssh_runcmd(ssh_host, ['true']) + output = cliapp.ssh_runcmd(ssh_host, ['echo', 'test']) except cliapp.AppException as e: logging.error("Error checking SSH connectivity: %s", str(e)) raise cliapp.AppException( 'Unable to SSH to %s: %s' % (ssh_host, e)) + if output.strip() != 'test': + raise cliapp.AppException( + 'Unexpected output from remote machine: %s' % output.strip()) + def is_device(self, location): try: st = os.stat(location) diff --git a/ostree-repo-server b/ostree-repo-server deleted file mode 100755 index e6dc4a56..00000000 --- a/ostree-repo-server +++ /dev/null @@ -1,15 +0,0 @@ -#!/usr/bin/python - -from BaseHTTPServer import HTTPServer -from SimpleHTTPServer import SimpleHTTPRequestHandler -from SocketServer import ThreadingMixIn - -class ThreadedHTTPServer(ThreadingMixIn, HTTPServer): - """Handle requests in a separate thread""" - -handler = SimpleHTTPRequestHandler -handler.protocol_version="HTTP/1.0" -server_address = ('', 12324) - -httpd = ThreadedHTTPServer(server_address, handler) -httpd.serve_forever() diff --git a/scripts/check-copyright-year b/scripts/check-copyright-year index 08bee0af..2370182c 100755 --- a/scripts/check-copyright-year +++ b/scripts/check-copyright-year @@ -54,6 +54,9 @@ class CheckCopyrightYear(cliapp.Application): return filenames def process_input_line(self, filename, line): + if filename == 'COPYING': + return + m = self.pat.match(line) if not m: return diff --git a/tests.build/build-chunk-writes-log.script b/tests.build/build-chunk-writes-log.script new file mode 100755 index 00000000..e636924e --- /dev/null +++ b/tests.build/build-chunk-writes-log.script @@ -0,0 +1,37 @@ +#!/bin/sh +# +# Copyright (C) 2011-2013,2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see <http://www.gnu.org/licenses/>. + + +## Build log should be saved when a chunk is built. + +set -eu + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +SOURCES="$DATADIR/cached-sources" +find "$DATADIR/cache/artifacts" -name '*.chunk.*' | + sed 's|\.chunk\..*||' | sort -u >"$SOURCES" + +found=false +# list of sources in cache is not piped because while loop changes variable +while read source; do + [ -e "$source".build-log ] || continue + found=true + break +done <"$SOURCES" +"$found" + diff --git a/tests.build/build-stratum-with-submodules.script b/tests.build/build-stratum-with-submodules.script index a2a1ddc9..bd6b97ce 100755 --- a/tests.build/build-stratum-with-submodules.script +++ b/tests.build/build-stratum-with-submodules.script @@ -56,7 +56,11 @@ EOF "$SRCDIR/scripts/run-git-in" "$morphs" commit --quiet -m 'foo' -# Now build +# Now build and verify we got a stratum. "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +system=$(ls "$DATADIR/cache/artifacts/"*hello-system-rootfs) +tar tf $system | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -v '^baserock/' + diff --git a/tests.build/build-stratum-with-submodules.stdout b/tests.build/build-stratum-with-submodules.stdout new file mode 100644 index 00000000..d4d03e13 --- /dev/null +++ b/tests.build/build-stratum-with-submodules.stdout @@ -0,0 +1,3 @@ +./ +etc/ +etc/os-release diff --git a/tests.build/build-system-autotools.script b/tests.build/build-system-autotools.script index 936fa490..710a8f98 100755 --- a/tests.build/build-system-autotools.script +++ b/tests.build/build-system-autotools.script @@ -46,3 +46,8 @@ git commit --quiet -m "Convert hello to an autotools project" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(bin|etc)' diff --git a/tests.build/build-system-autotools.stdout b/tests.build/build-system-autotools.stdout new file mode 100644 index 00000000..683441c9 --- /dev/null +++ b/tests.build/build-system-autotools.stdout @@ -0,0 +1,3 @@ +bin/ +bin/hello +etc/ diff --git a/tests.build/build-system-cmake.script b/tests.build/build-system-cmake.script index b848aab9..fe02f9dc 100755 --- a/tests.build/build-system-cmake.script +++ b/tests.build/build-system-cmake.script @@ -48,3 +48,8 @@ git commit --quiet -m "Convert hello to a cmake project" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(usr/)?(bin|etc)' diff --git a/tests.build/build-system-cmake.stdout b/tests.build/build-system-cmake.stdout new file mode 100644 index 00000000..3410b113 --- /dev/null +++ b/tests.build/build-system-cmake.stdout @@ -0,0 +1,2 @@ +usr/bin/ +usr/bin/hello diff --git a/tests.build/build-system-cpan.script b/tests.build/build-system-cpan.script index b686de34..103d5466 100755 --- a/tests.build/build-system-cpan.script +++ b/tests.build/build-system-cpan.script @@ -70,3 +70,8 @@ git commit -q -m "Set custom install prefix for hello" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -F 'bin/hello' diff --git a/tests.build/build-system-cpan.stdout b/tests.build/build-system-cpan.stdout new file mode 100644 index 00000000..180e949b --- /dev/null +++ b/tests.build/build-system-cpan.stdout @@ -0,0 +1 @@ +bin/hello diff --git a/tests.build/build-system-python-distutils.script b/tests.build/build-system-python-distutils.script index d8210319..e5c0ea74 100755 --- a/tests.build/build-system-python-distutils.script +++ b/tests.build/build-system-python-distutils.script @@ -68,3 +68,13 @@ git commit -q -m "Set custom install prefix for hello" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | LC_ALL=C sort -u | sed '/^\.\/./s:^\./::' | grep -Ee '^(bin|lib)' | +sed -e 's:^local/::' \ + -e 's:lib/python2.[6-9]/:lib/python2.x/:' \ + -e 's:/hello-0\.0\.0[^/]*\.egg-info$:/hello.egg-info/:' \ + -e 's:[^/]*-packages:packages:' \ + -e '/^$/d' diff --git a/tests.build/build-system-python-distutils.stdout b/tests.build/build-system-python-distutils.stdout new file mode 100644 index 00000000..4d4c3a1e --- /dev/null +++ b/tests.build/build-system-python-distutils.stdout @@ -0,0 +1,6 @@ +bin/ +bin/hello +lib/ +lib/python2.x/ +lib/python2.x/packages/ +lib/python2.x/packages/hello.egg-info/ diff --git a/tests.build/build-system-qmake.script b/tests.build/build-system-qmake.script index b477de4b..d430fba7 100755 --- a/tests.build/build-system-qmake.script +++ b/tests.build/build-system-qmake.script @@ -22,6 +22,7 @@ set -eu if ! command -v qmake > /dev/null ; then # There is no qmake, so skip this test. + cat "$SRCDIR/tests.build/build-system-qmake.stdout" exit 0 fi @@ -55,3 +56,10 @@ git commit --quiet -m "Convert hello to an qmake project" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + echo "$chunk:" | sed 's/[^.]*//' + tar -tf "$chunk" | LC_ALL=C sort | sed '/^\.\/./s:^\./::' + echo +done diff --git a/tests.build/build-system-qmake.stdout b/tests.build/build-system-qmake.stdout new file mode 100644 index 00000000..ccf80a86 --- /dev/null +++ b/tests.build/build-system-qmake.stdout @@ -0,0 +1,8 @@ +.chunk.hello: +./ +baserock/ +baserock/hello.meta +usr/ +usr/bin/ +usr/bin/hello + diff --git a/tests.build/build-system.script b/tests.build/build-system.script new file mode 100755 index 00000000..0180939a --- /dev/null +++ b/tests.build/build-system.script @@ -0,0 +1,26 @@ +#!/bin/sh +# +# Copyright (C) 2011-2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see <http://www.gnu.org/licenses/>. + + +## Test building a simple system. + +set -eu + +"$SRCDIR/scripts/test-morph" build-morphology \ + test:morphs-repo master hello-system + +system=$(ls "$DATADIR/cache/artifacts/"*hello-system-rootfs) +tar tf $system | LC_ALL=C sort | sed '/^\.\/./s:^\./::' | grep -v '^baserock/' diff --git a/tests.build/build-system.stdout b/tests.build/build-system.stdout new file mode 100644 index 00000000..4d0fac2f --- /dev/null +++ b/tests.build/build-system.stdout @@ -0,0 +1,5 @@ +./ +bin/ +bin/hello +etc/ +etc/os-release diff --git a/tests.build/cross-bootstrap.script b/tests.build/cross-bootstrap.script index 6bab1659..245c2a13 100755 --- a/tests.build/cross-bootstrap.script +++ b/tests.build/cross-bootstrap.script @@ -22,9 +22,6 @@ set -eu "$SRCDIR/tests.build/setup-build-essential" -# cross-bootstrap needs rewriting for OSTree -exit 0 - "$SRCDIR/scripts/test-morph" cross-bootstrap \ $("$SRCDIR/scripts/test-morph" print-architecture) \ test:morphs-repo master hello-system diff --git a/tests.build/morphless-chunks.script b/tests.build/morphless-chunks.script index b46fa635..5b19bc4a 100755 --- a/tests.build/morphless-chunks.script +++ b/tests.build/morphless-chunks.script @@ -40,3 +40,8 @@ git commit -q -m "Convert hello into an autodetectable chunk" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +for chunk in "$DATADIR/cache/artifacts/"*.chunk.* +do + tar -tf "$chunk" +done | cat >/dev/null # No files get installed apart from metadata diff --git a/tests.build/morphless-chunks.stdout b/tests.build/morphless-chunks.stdout new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/tests.build/morphless-chunks.stdout diff --git a/tests.build/prefix.script b/tests.build/prefix.script index 75c91200..140617e1 100755 --- a/tests.build/prefix.script +++ b/tests.build/prefix.script @@ -65,3 +65,8 @@ git commit -q -m "Update stratum" "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo master hello-system + +cd "$DATADIR/cache/artifacts" +first_chunk=$(ls -1 *.chunk.xyzzy-* | head -n1 | cut -c -64) +second_chunk=$(ls -1 *.chunk.plugh-* | head -n1 | cut -c -64) +cat $first_chunk.build-log $second_chunk.build-log diff --git a/tests.build/prefix.stdout b/tests.build/prefix.stdout new file mode 100644 index 00000000..80c18fae --- /dev/null +++ b/tests.build/prefix.stdout @@ -0,0 +1,8 @@ +# configure +# # echo First chunk: prefix $PREFIX +First chunk: prefix /plover +# configure +# # echo Second chunk: prefix $PREFIX +Second chunk: prefix /usr +# # echo Path: $(echo $PATH | grep -o '/plover') +Path: /plover diff --git a/tests.build/rebuild-cached-stratum.script b/tests.build/rebuild-cached-stratum.script index bdbe193d..e2e0face 100755 --- a/tests.build/rebuild-cached-stratum.script +++ b/tests.build/rebuild-cached-stratum.script @@ -40,6 +40,9 @@ cache="$DATADIR/cache/artifacts" # Build the first time. "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo rebuild-cached-stratum hello-system +echo "first build:" +(cd "$cache" && ls *.chunk.* *hello-stratum-* | sed 's/^[^.]*\./ /' | + LC_ALL=C sort -u) # Change the chunk. (cd "$DATADIR/chunk-repo" && @@ -49,3 +52,7 @@ cache="$DATADIR/cache/artifacts" # Rebuild. "$SRCDIR/scripts/test-morph" build-morphology \ test:morphs-repo rebuild-cached-stratum hello-system +echo "second build:" +(cd "$cache" && ls *.chunk.* *hello-stratum-* | sed 's/^[^.]*\./ /' | + LC_ALL=C sort -u) + diff --git a/tests.build/rebuild-cached-stratum.stdout b/tests.build/rebuild-cached-stratum.stdout new file mode 100644 index 00000000..9c53ee60 --- /dev/null +++ b/tests.build/rebuild-cached-stratum.stdout @@ -0,0 +1,22 @@ +first build: + chunk.hello-bins + chunk.hello-devel + chunk.hello-doc + chunk.hello-libs + chunk.hello-locale + chunk.hello-misc + stratum.hello-stratum-devel + stratum.hello-stratum-devel.meta + stratum.hello-stratum-runtime + stratum.hello-stratum-runtime.meta +second build: + chunk.hello-bins + chunk.hello-devel + chunk.hello-doc + chunk.hello-libs + chunk.hello-locale + chunk.hello-misc + stratum.hello-stratum-devel + stratum.hello-stratum-devel.meta + stratum.hello-stratum-runtime + stratum.hello-stratum-runtime.meta diff --git a/without-test-modules b/without-test-modules index 2e1b8c57..55e5291d 100644 --- a/without-test-modules +++ b/without-test-modules @@ -52,5 +52,3 @@ distbuild/timer_event_source.py distbuild/worker_build_scheduler.py # Not unit tested, since it needs a full system branch morphlib/buildbranch.py -morphlib/ostree.py -morphlib/ostreeartifactcache.py diff --git a/yarns/architecture.yarn b/yarns/architecture.yarn index d68ed2e6..07274ec3 100644 --- a/yarns/architecture.yarn +++ b/yarns/architecture.yarn @@ -15,15 +15,13 @@ Morph Cross-Building Tests Morph Cross-Bootstrap Tests =========================== -Note: This test is broken because cross-bootstrap is not updated to use OSTree. - -> SCENARIO cross-bootstrapping a system for a different architecture -> GIVEN a workspace -> AND a git server -> AND a system called base-system-testarch.morph for the test architecture in the git server -> WHEN the user checks out the system branch called master -> THEN the user cross-bootstraps the system base-system-testarch.morph in branch master of repo test:morphs to the arch testarch -> FINALLY the git server is shut down + SCENARIO cross-bootstraping a system for a different architecture + GIVEN a workspace + AND a git server + AND a system called base-system-testarch.morph for the test architecture in the git server + WHEN the user checks out the system branch called master + THEN the user cross-bootstraps the system base-system-testarch.morph in branch master of repo test:morphs to the arch testarch + FINALLY the git server is shut down Architecture validation Tests ============================= diff --git a/yarns/implementations.yarn b/yarns/implementations.yarn index 3277075e..2bbb1f5c 100644 --- a/yarns/implementations.yarn +++ b/yarns/implementations.yarn @@ -1055,14 +1055,13 @@ Distbuild read_cache_server_pid_file="$DATADIR/read-cache-server-pid" start_cache_server "$read_cache_server_port_file" \ "$read_cache_server_pid_file" \ - "$artifact_dir" "$DATADIR/communal-cache.log" + "$artifact_dir" write_cache_server_port_file="$DATADIR/write-cache-server-port" write_cache_server_pid_file="$DATADIR/write-cache-server-pid" start_cache_server "$write_cache_server_port_file" \ "$write_cache_server_pid_file" \ - "$artifact_dir" "$DATADIR/writeable-cache.log" \ - --enable-writes + "$artifact_dir" --enable-writes IMPLEMENTS FINALLY the communal cache server is terminated stop_daemon "$DATADIR/read-cache-server-pid" @@ -1077,7 +1076,7 @@ Distbuild worker_cache_pid_file="$DATADIR/worker-cache-server-pid" start_cache_server "$worker_cache_port_file" \ "$worker_cache_pid_file" \ - "$worker_artifacts" "$DATADIR/worker-cache.log" + "$worker_artifacts" # start worker daemon worker_daemon_port_file="$DATADIR/worker-daemon-port" @@ -1099,14 +1098,6 @@ Distbuild rm "$worker_daemon_port_file" echo "$worker_daemon_port" >"$worker_daemon_port_file" - # serve artifact cache over http - worker_repo_pid_file="$DATADIR/worker-repo-pid" - mkdir "$worker_artifacts/repo" - cd "$worker_artifacts/repo" - start-stop-daemon --start --pidfile="$worker_repo_pid_file" \ - --background --make-pidfile --verbose \ - --startas="$SRCDIR/ostree-repo-server" - # start worker helper helper_pid_file="$DATADIR/worker-daemon-helper-pid" start-stop-daemon --start --pidfile="$helper_pid_file" \ @@ -1128,7 +1119,6 @@ Distbuild stop_daemon "$DATADIR/worker-cache-server-pid" stop_daemon "$DATADIR/worker-daemon-pid" stop_daemon "$DATADIR/worker-daemon-helper-pid" - stop_daemon "$DATADIR/worker-repo-pid" IMPLEMENTS GIVEN a distbuild controller worker_cache_port_file="$DATADIR/worker-cache-server-port" diff --git a/yarns/morph.shell-lib b/yarns/morph.shell-lib index 4f345a4a..e7011091 100644 --- a/yarns/morph.shell-lib +++ b/yarns/morph.shell-lib @@ -194,7 +194,6 @@ start_cache_server(){ --background --make-pidfile --verbose \ --startas="$SRCDIR/morph-cache-server" -- \ --port-file="$1" --no-fcgi \ - --log="$4" \ --repo-dir="$DATADIR/gits" --direct-mode \ --bundle-dir="$DATADIR/bundles" \ --artifact-dir="$3" "$@" |