From 2b23898def56f758192b9325b2524ef960fc4d03 Mon Sep 17 00:00:00 2001 From: Jonathan Maw Date: Mon, 9 Jul 2018 17:40:45 +0100 Subject: doc: Add tutorials for setting up git and tar mirrors --- doc/source/examples/git-mirror.rst | 144 +++++++++++++++++++++++++++++++++++++ doc/source/examples/tar-mirror.rst | 103 ++++++++++++++++++++++++++ doc/source/using_examples.rst | 2 + 3 files changed, 249 insertions(+) create mode 100644 doc/source/examples/git-mirror.rst create mode 100644 doc/source/examples/tar-mirror.rst diff --git a/doc/source/examples/git-mirror.rst b/doc/source/examples/git-mirror.rst new file mode 100644 index 000000000..1b05bf607 --- /dev/null +++ b/doc/source/examples/git-mirror.rst @@ -0,0 +1,144 @@ + + +Creating and using a git mirror +''''''''''''''''''''''''''''''' +This is an example of how to create a git mirror using git's +`git-http-backend `_ and +`lighttpd `_. + + +Prerequisites +============= +You will need git installed, and git-http-backend must be present. It is assumed +that the git-http-backend binary exists at `/usr/lib/git-core/git-http-backend`. + +You will need `lighttpd` installed, and at the bare minimum has the modules +`mod_alias`, `mod_cgi`, and `mod_setenv`. + +I will be using gnome-modulesets as an example, which can be cloned from +`http://gnome7.codethink.co.uk/gnome-modulesets.git`. + + +Starting a git http server +========================== + + +1. Set up a directory containing mirrors +---------------------------------------- +Choose a suitable directory to hold your mirrors, e.g. `/var/www/git`. + +Place the git repositories you want to use as mirrors in the mirror dir, e.g. +``git clone --mirror http://git.gnome.org/browse/yelp-xsl /var/www/git/yelp-xsl.git``. + + +2. Configure lighttpd +--------------------- +Write out a lighttpd.conf as follows: + +:: + + server.document-root = "/var/www/git/" + server.port = 3000 + server.modules = ( + "mod_alias", + "mod_cgi", + "mod_setenv", + ) + + alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" ) + $HTTP["url"] =~ "^/git" { + cgi.assign = ("" => "") + setenv.add-environment = ( + "GIT_PROJECT_ROOT" => "/var/www/git", + "GIT_HTTP_EXPORT_ALL" => "" + ) + } + +.. note:: + + If you have your mirrors in another directory, replace /var/www/git/ with that directory. + + +3. Start lighttpd +----------------- +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``. + + +4. Test that you can fetch from it +---------------------------------- +We can then clone the mirrored repo using git via http with +``git clone http://127.0.0.1:3000/git/yelp-xsl``. + +.. note:: + + If you have set server.port to something other than the default, you will + need to replace the '3000' in the command-line. + + +5. Configure the project to use the mirror +------------------------------------------ +To add this local http server as a mirror, add the following to the project.conf: + +.. code:: yaml + + mirrors: + - name: local-mirror + aliases: + git_gnome_org: + - http://127.0.0.1:3000/git/ + + +6. Test that the mirror works +----------------------------- +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g. + +.. code:: yaml + + aliases: + git_gnome_org: https://www.example.com/invalid/url/ + +Now, if you build an element that uses the source you placed in the mirror +(e.g. ``bst build core-deps/yelp-xsl.bst``), you will see that it uses your mirror. + + +.. _lighttpd_git_tar_conf: + +Bonus: lighttpd conf for git and tar +==================================== +For those who have also used the :ref:`tar-mirror tutorial `, +a combined lighttpd.conf is below: + +:: + + server.document-root = "/var/www/" + server.port = 3000 + server.modules = ( + "mod_alias", + "mod_cgi", + "mod_setenv", + ) + + alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" ) + $HTTP["url"] =~ "^/git" { + cgi.assign = ("" => "") + setenv.add-environment = ( + "GIT_PROJECT_ROOT" => "/var/www/git", + "GIT_HTTP_EXPORT_ALL" => "" + ) + } else $HTTP["url"] =~ "^/tar" { + dir-listing.activate = "enable" + } + + +Further reading +=============== +If this mirror isn't being used exclusively in a secure network, it is strongly +recommended you `use SSL `_. + +This is the bare minimum required to set up a git mirror. A large, public project +would prefer to set it up using the +`git protocol `_, +and a security-conscious project would be configured to use +`git over SSH `_. + +Lighttpd is documented on `its wiki `_. diff --git a/doc/source/examples/tar-mirror.rst b/doc/source/examples/tar-mirror.rst new file mode 100644 index 000000000..7a4a2f51e --- /dev/null +++ b/doc/source/examples/tar-mirror.rst @@ -0,0 +1,103 @@ + + +.. _using_tar_mirror: + +Creating and using a tar mirror +''''''''''''''''''''''''''''''' +This is an example of how to create a tar mirror using +`lighttpd `_. + + +Prerequisites +============= +You will need `lighttpd` installed. + + +I will be using gnome-modulesets as an example, which can be cloned from +`http://gnome7.codethink.co.uk/gnome-modulesets.git`. + + +Starting a tar server +===================== + + +1. Set up a directory containing mirrors +---------------------------------------- +Choose a suitable directory to hold your mirrored tar files, e.g. `/var/www/tar`. + +Place the tar files you want to use as mirrors in your mirror dir, e.g. + +.. code:: + + mkdir -p /var/www/tar/gettext + wget -O /var/www/tar/gettext/gettext-0.19.8.1.tar.xz https://ftp.gnu.org/gnu/gettext/gettext-0.19.8.1.tar.xz + + +2. Configure lighttpd +--------------------- +Write out a lighttpd.conf as follows: + +:: + + server.document-root = "/var/www/tar/" + server.port = 3000 + + dir-listing.activate = "enable" + +.. note:: + + If you have your mirrors in another directory, replace /var/www/tar/ with that directory. + +.. note:: + + An example lighttpd.conf that works for both git and tar services is available + :ref:`here ` + + +3. Start lighttpd +----------------- +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``. + + +4. Test that you can fetch from it +---------------------------------- +We can then download the mirrored file with ``wget 127.0.0.1:3000/tar/gettext/gettext-0.19.8.1.tar.xz``. + +.. note:: + + If you have set server.port to something other than the default, you will need + to replace the '3000' in the command-line. + + +5. Configure the project to use the mirror +------------------------------------------ +To add this local http server as a mirror, add the following to the project.conf: + +.. code:: yaml + + mirrors: + - name: local-mirror + aliases: + ftp_gnu_org: + - http://127.0.0.1:3000/tar/ + + +6. Test that the mirror works +----------------------------- +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g. + +.. code:: yaml + + aliases: + ftp_gnu_org: https://www.example.com/invalid/url/ + +Now, if you build an element that uses the source you placed in the mirror +(e.g. ``bst build core-deps/gettext.bst``), you will see that it uses your mirror. + + +Further reading +=============== +If this mirror isn't being used exclusively in a secure network, it is strongly +recommended you `use SSL `_. + +Lighttpd is documented on `its wiki `_. diff --git a/doc/source/using_examples.rst b/doc/source/using_examples.rst index aa100d007..622b09e32 100644 --- a/doc/source/using_examples.rst +++ b/doc/source/using_examples.rst @@ -10,3 +10,5 @@ maintained and work as expected. :maxdepth: 1 examples/flatpak-autotools + examples/tar-mirror + examples/git-mirror -- cgit v1.2.1