summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorJonathan Maw <jonathan.maw@codethink.co.uk>2018-07-09 17:40:45 +0100
committerJonathan Maw <jonathan.maw@codethink.co.uk>2018-07-27 12:24:56 +0000
commit2b23898def56f758192b9325b2524ef960fc4d03 (patch)
tree81ab8894977883c8a40c415fc305053b24385efa
parent202d9d26e088985893bee25bebecaba4c46eb04a (diff)
downloadbuildstream-2b23898def56f758192b9325b2524ef960fc4d03.tar.gz
doc: Add tutorials for setting up git and tar mirrors
-rw-r--r--doc/source/examples/git-mirror.rst144
-rw-r--r--doc/source/examples/tar-mirror.rst103
-rw-r--r--doc/source/using_examples.rst2
3 files changed, 249 insertions, 0 deletions
diff --git a/doc/source/examples/git-mirror.rst b/doc/source/examples/git-mirror.rst
new file mode 100644
index 000000000..1b05bf607
--- /dev/null
+++ b/doc/source/examples/git-mirror.rst
@@ -0,0 +1,144 @@
+
+
+Creating and using a git mirror
+'''''''''''''''''''''''''''''''
+This is an example of how to create a git mirror using git's
+`git-http-backend <https://git-scm.com/docs/git-http-backend>`_ and
+`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
+
+
+Prerequisites
+=============
+You will need git installed, and git-http-backend must be present. It is assumed
+that the git-http-backend binary exists at `/usr/lib/git-core/git-http-backend`.
+
+You will need `lighttpd` installed, and at the bare minimum has the modules
+`mod_alias`, `mod_cgi`, and `mod_setenv`.
+
+I will be using gnome-modulesets as an example, which can be cloned from
+`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
+
+
+Starting a git http server
+==========================
+
+
+1. Set up a directory containing mirrors
+----------------------------------------
+Choose a suitable directory to hold your mirrors, e.g. `/var/www/git`.
+
+Place the git repositories you want to use as mirrors in the mirror dir, e.g.
+``git clone --mirror http://git.gnome.org/browse/yelp-xsl /var/www/git/yelp-xsl.git``.
+
+
+2. Configure lighttpd
+---------------------
+Write out a lighttpd.conf as follows:
+
+::
+
+ server.document-root = "/var/www/git/"
+ server.port = 3000
+ server.modules = (
+ "mod_alias",
+ "mod_cgi",
+ "mod_setenv",
+ )
+
+ alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
+ $HTTP["url"] =~ "^/git" {
+ cgi.assign = ("" => "")
+ setenv.add-environment = (
+ "GIT_PROJECT_ROOT" => "/var/www/git",
+ "GIT_HTTP_EXPORT_ALL" => ""
+ )
+ }
+
+.. note::
+
+ If you have your mirrors in another directory, replace /var/www/git/ with that directory.
+
+
+3. Start lighttpd
+-----------------
+lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
+
+
+4. Test that you can fetch from it
+----------------------------------
+We can then clone the mirrored repo using git via http with
+``git clone http://127.0.0.1:3000/git/yelp-xsl``.
+
+.. note::
+
+ If you have set server.port to something other than the default, you will
+ need to replace the '3000' in the command-line.
+
+
+5. Configure the project to use the mirror
+------------------------------------------
+To add this local http server as a mirror, add the following to the project.conf:
+
+.. code:: yaml
+
+ mirrors:
+ - name: local-mirror
+ aliases:
+ git_gnome_org:
+ - http://127.0.0.1:3000/git/
+
+
+6. Test that the mirror works
+-----------------------------
+We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
+
+.. code:: yaml
+
+ aliases:
+ git_gnome_org: https://www.example.com/invalid/url/
+
+Now, if you build an element that uses the source you placed in the mirror
+(e.g. ``bst build core-deps/yelp-xsl.bst``), you will see that it uses your mirror.
+
+
+.. _lighttpd_git_tar_conf:
+
+Bonus: lighttpd conf for git and tar
+====================================
+For those who have also used the :ref:`tar-mirror tutorial <using_tar_mirror>`,
+a combined lighttpd.conf is below:
+
+::
+
+ server.document-root = "/var/www/"
+ server.port = 3000
+ server.modules = (
+ "mod_alias",
+ "mod_cgi",
+ "mod_setenv",
+ )
+
+ alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
+ $HTTP["url"] =~ "^/git" {
+ cgi.assign = ("" => "")
+ setenv.add-environment = (
+ "GIT_PROJECT_ROOT" => "/var/www/git",
+ "GIT_HTTP_EXPORT_ALL" => ""
+ )
+ } else $HTTP["url"] =~ "^/tar" {
+ dir-listing.activate = "enable"
+ }
+
+
+Further reading
+===============
+If this mirror isn't being used exclusively in a secure network, it is strongly
+recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
+
+This is the bare minimum required to set up a git mirror. A large, public project
+would prefer to set it up using the
+`git protocol <https://git-scm.com/book/en/v1/Git-on-the-Server-Git-Daemon>`_,
+and a security-conscious project would be configured to use
+`git over SSH <https://git-scm.com/book/en/v1/Git-on-the-Server-Getting-Git-on-a-Server#Small-Setups>`_.
+
+Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.
diff --git a/doc/source/examples/tar-mirror.rst b/doc/source/examples/tar-mirror.rst
new file mode 100644
index 000000000..7a4a2f51e
--- /dev/null
+++ b/doc/source/examples/tar-mirror.rst
@@ -0,0 +1,103 @@
+
+
+.. _using_tar_mirror:
+
+Creating and using a tar mirror
+'''''''''''''''''''''''''''''''
+This is an example of how to create a tar mirror using
+`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
+
+
+Prerequisites
+=============
+You will need `lighttpd` installed.
+
+
+I will be using gnome-modulesets as an example, which can be cloned from
+`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
+
+
+Starting a tar server
+=====================
+
+
+1. Set up a directory containing mirrors
+----------------------------------------
+Choose a suitable directory to hold your mirrored tar files, e.g. `/var/www/tar`.
+
+Place the tar files you want to use as mirrors in your mirror dir, e.g.
+
+.. code::
+
+ mkdir -p /var/www/tar/gettext
+ wget -O /var/www/tar/gettext/gettext-0.19.8.1.tar.xz https://ftp.gnu.org/gnu/gettext/gettext-0.19.8.1.tar.xz
+
+
+2. Configure lighttpd
+---------------------
+Write out a lighttpd.conf as follows:
+
+::
+
+ server.document-root = "/var/www/tar/"
+ server.port = 3000
+
+ dir-listing.activate = "enable"
+
+.. note::
+
+ If you have your mirrors in another directory, replace /var/www/tar/ with that directory.
+
+.. note::
+
+ An example lighttpd.conf that works for both git and tar services is available
+ :ref:`here <lighttpd_git_tar_conf>`
+
+
+3. Start lighttpd
+-----------------
+lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
+
+
+4. Test that you can fetch from it
+----------------------------------
+We can then download the mirrored file with ``wget 127.0.0.1:3000/tar/gettext/gettext-0.19.8.1.tar.xz``.
+
+.. note::
+
+ If you have set server.port to something other than the default, you will need
+ to replace the '3000' in the command-line.
+
+
+5. Configure the project to use the mirror
+------------------------------------------
+To add this local http server as a mirror, add the following to the project.conf:
+
+.. code:: yaml
+
+ mirrors:
+ - name: local-mirror
+ aliases:
+ ftp_gnu_org:
+ - http://127.0.0.1:3000/tar/
+
+
+6. Test that the mirror works
+-----------------------------
+We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
+
+.. code:: yaml
+
+ aliases:
+ ftp_gnu_org: https://www.example.com/invalid/url/
+
+Now, if you build an element that uses the source you placed in the mirror
+(e.g. ``bst build core-deps/gettext.bst``), you will see that it uses your mirror.
+
+
+Further reading
+===============
+If this mirror isn't being used exclusively in a secure network, it is strongly
+recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
+
+Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.
diff --git a/doc/source/using_examples.rst b/doc/source/using_examples.rst
index aa100d007..622b09e32 100644
--- a/doc/source/using_examples.rst
+++ b/doc/source/using_examples.rst
@@ -10,3 +10,5 @@ maintained and work as expected.
:maxdepth: 1
examples/flatpak-autotools
+ examples/tar-mirror
+ examples/git-mirror