import: Add documentation!

3 files changed, 134 insertions, 3 deletions

diff --git a/import/README b/import/README
new file mode 100644
index 00000000..bb2c4fbb
--- /dev/null
+++ b/import/README
@@ -0,0 +1,97 @@
+How to use the Baserock Import Tool
+===================================
+
+The tool helps you generate Baserock build instructions by importing metadata
+from a foreign packaging system.
+
+The process it follows is this:
+
+1. Pick a package from the processing queue.
+2. Find its source code, and generate a suitable .lorry file.
+3. Make it available as a local Git repo.
+4. Check out the commit corresponding to the requested version of the package.
+5. Analyse the source tree and generate a suitable chunk .morph to build the
+   requested package.
+6. Analyse the source tree and generate a list of dependencies for the package.
+7. Enqueue any new dependencies, and repeat.
+
+Once the queue is empty:
+
+8. Generate a stratum .morph for the package(s) the user requested.
+
+The tool is not magic. It can be taught the conventions for each packaging
+system, but these will not work in all cases. When an import fails it will
+continue to the next package, so that the first run does as many imports as
+possible.
+
+For imports that could not be done automatically, you will need to write an
+appropriate .lorry or .morph file manually and rerun the tool. It will resume
+processing where it left off.
+
+It's possible to teach the code about more conventions, but it is only
+worthwhile to do that for common patterns.
+
+
+Package-system specific code and data
+-------------------------------------
+
+For each supported package system, there should be an xxx.to_lorry program, and
+a xxx.to_chunk program. These should output on stdout a .lorry file and a .morph
+file, respectively.
+
+Each packaging system can have static data saved in a .yaml file, for known
+metadata that the programs cannot discover automatically.
+
+The following field should be honoured by most packaging systems:
+`known-source-uris`. It maps package name to source URI.
+
+
+Help with .lorry generation
+---------------------------
+
+The simplest fix is to add the source to the 'known-source-uris` dict in the
+static metadata.
+
+If you write a .lorry file by hand, be sure to fill in the `x-products-YYY`
+field. 'x' means this field is an extension to the .lorry format. YYY is the
+name of the packaging system, e.g. 'rubygems'. It should contain a list of
+which packages this repository contains the source code for.
+
+
+Help with linking package version to Git tag
+--------------------------------------------
+
+Some projects do not tag releases.
+
+FIXME: currently the code just checks out `master` if it can't find the tag.
+This is a mis-feature. We need to fail, and provide the user with a way of
+informing the code of the tag. This could be done:
+    - with an extra field in the .lorry file, linking version to commit SHA1
+    - by creating a suitably named tag manually in the repo
+    - in the static data, perhaps
+
+Help with chunk .morph generation
+---------------------------------
+
+If you create a chunk morph by hand, you must add some extra fields:
+
+  - `x-build-dependencies-YYY`
+  - `x-runtime-dependencies-YYY`
+
+These are a dict mapping dependency name to dependency version. For example:
+
+    x-build-dependencies-rubygems: {}
+    x-runtime-dependencies-rubygems:
+        hashie: 2.1.2
+        json: 1.8.1
+        mixlib-log: 1.6.0
+        rack: 1.5.2
+
+All dependencies will be included in the resulting stratum. Those which are build
+dependencies of other components will be added to the relevant 'build-depends'
+field.
+
+These fields are non-standard extensions to the morphology format.
+
+For more package-system specific information, see the relevant README file, e.g
+README.rubygems for RubyGem imports.
diff --git a/import/README.rubygems b/import/README.rubygems
new file mode 100644
index 00000000..6954b935
--- /dev/null
+++ b/import/README.rubygems
@@ -0,0 +1,34 @@
+Here is some information I have learned while importing RubyGem packages into
+Baserock.
+
+First, beware that RubyGem .gemspec files are actually normal Ruby programs,
+and are executed when read. A Bundler Gemfile is also a Ruby program, and could
+run arbitrary code when read.
+
+The Standard Case
+-----------------
+
+Most Ruby projects provide one or more .gemspec files, which describe the
+runtime and development dependencies of the Gem.
+
+Using the .gemspec file and the `gem build` command it is possible to create
+the .gem file. It can then be installed with `gem install`.
+
+Note that use of `gem build` is discouraged by its own help file in favour
+of using Rake, but there is much less standardisation among Rakefiles and they
+may introduce requirements on Hoe, rake-compiler, Jeweler or other tools.
+
+The 'development' dependencies includes everything useful to test, document,
+and create a Gem of the project. All we want to do is create a Gem, which I'll
+refer to as 'building'.
+
+
+Gem with no .gemspec
+--------------------
+
+Some Gems choose not to include a .gemspec, like [Nokigori]. In the case of
+Nokigori, and others, [Hoe] is used, which adds Rake tasks that create the Gem.
+The `gem build` command cannot not be used in these cases.
+
+[Nokigori]: https://github.com/sparklemotion/nokogiri/blob/master/Y_U_NO_GEMSPEC.md
+[Hoe]: http://www.zenspider.com/projects/hoe.html
diff --git a/import/main.py b/import/main.py
index b4d30ad2..1d168eba 100644
--- a/import/main.py
+++ b/import/main.py
@@ -422,9 +422,9 @@ class BaserockImportApplication(cliapp.Application):
 
         if len(errors) > 0:
             self.status(
-                '\nErrors encountered, not generating a stratum morphology. '
-                'You may want to manually create chunk morphologies for the '
-                'following items, then rerun the script.')
+                '\nErrors encountered, not generating a stratum morphology.')
+            self.status(
+                'See the README files for guidance.')
             for package, exception in errors.iteritems():
                 self.status('\n%s: %s', package.name, exception)
         else: