path: root/doc/rsync.sgml

<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V4.1//EN">
<book id="rsync">
  <bookinfo>
    <title>rsync</title>
    <copyright>
      <year>1996 -- 2002</year>
      <holder>Martin Pool</holder>
      <holder>Andrew Tridgell</holder>
    </copyright>
    <author>
      <firstname>Martin</firstname>
      <surname>Pool</surname>
    </author>
  </bookinfo>

  <chapter>
    <title>Introduction</title>

    <para>rsync is a flexible program for efficiently copying files or
      directory trees.

    <para>rsync has many options to select which files will be copied
      and how they are to be transferred.  It may be used as an
      alternative to ftp, http, scp or rcp.

    <para>The rsync remote-update protocol allows rsync to transfer just
      the differences between two sets of files across the network link,
      using an efficient checksum-search algorithm described in the
      technical report that accompanies this package.</para>

    <para>Some of the additional features of rsync are:</para>

    <itemizedlist>
      
      <listitem>
	<para>support for copying links, devices, owners, groups and
	  permissions
	</para>
      </listitem>
      
      <listitem>
	<para>
	  exclude and exclude-from options similar to GNU tar
	</para>
      </listitem>

      <listitem>
	<para>
	  a CVS exclude mode for ignoring the same files that CVS would ignore
      </listitem>

      <listitem>
	<para>
	  can use any transparent remote shell, including rsh or ssh
      </listitem>

      <listitem>
	<para>
	  does not require root privileges
      </listitem>

      <listitem>
	<para>
	  pipelining of file transfers to minimize latency costs
      </listitem>
		
      <listitem>
	<para>
	  support for anonymous or authenticated rsync servers (ideal for
	  mirroring)
	</para>
      </listitem>
    </itemizedlist>
  </chapter>



  <chapter>
    <title>Using rsync</title>
    <section>
      <title>
	Introductory example
      </title>
      
      <para>
	Probably the most common case of rsync usage is to copy files
	to or from a remote machine using
	<application>ssh</application> as a network transport.  In
	this situation rsync is a good alternative to
	<application>scp</application>.
      </para>

      <para>
	The most commonly used arguments for rsync are
      </para>

      <variablelist>
	<varlistentry>
	  <term><option>-v</option></term>
	  <listitem>
	    <para>Be verbose.  Primarily, display the name of each file as it is copied.</para>
	  </listitem>
	</varlistentry>


	<varlistentry>
	  <term><option>-a</option></term>
	  <listitem>
	    <para>
	      Reproduce the structure and attributes of the origin files as exactly
	      as possible: this includes copying subdirectories, symlinks, special
	      files, ownership and permissions.  (@xref{Attributes to
	      copy}.)
	    </para>
	  </listitem>
	</varlistentry>
      </variablelist>


	
      <para><option>-v </option>
      
      <para><option>-z</option>
	Compress network traffic, using a modified version of the
	@command{zlib} library.</para>
      
      <para><option>-P</option>
	Display a progress indicator while files are transferred.  This should
	normally be ommitted if rsync is not run on a terminal.
      </para>
    </section>




    <section>
      <title>Local and remote</title>
      
      <para>There are six different ways of using rsync. They
      are:</para>

      

      <!-- one of (CALLOUTLIST GLOSSLIST ITEMIZEDLIST ORDEREDLIST SEGMENTEDLIST SIMPLELIST VARIABLELIST CAUTION IMPORTANT NOTE TIP WARNING LITERALLAYOUT PROGRAMLISTING PROGRAMLISTINGCO SCREEN SCREENCO SCREENSHOT SYNOPSIS CMDSYNOPSIS FUNCSYNOPSIS CLASSSYNOPSIS FIELDSYNOPSIS CONSTRUCTORSYNOPSIS DESTRUCTORSYNOPSIS METHODSYNOPSIS FORMALPARA PARA SIMPARA ADDRESS BLOCKQUOTE GRAPHIC GRAPHICCO MEDIAOBJECT MEDIAOBJECTCO INFORMALEQUATION INFORMALEXAMPLE INFORMALFIGURE INFORMALTABLE EQUATION EXAMPLE FIGURE TABLE MSGSET PROCEDURE SIDEBAR QANDASET ANCHOR BRIDGEHEAD REMARK HIGHLIGHTS ABSTRACT AUTHORBLURB EPIGRAPH INDEXTERM REFENTRY SECTION) -->
      <orderedlist>
	<listitem>
	  <para>
	    for copying local files. This is invoked when neither
	    source nor destination path contains a @code{:} separator

	<listitem>
	  <para>
	    for copying from the local machine to a remote machine using
	    a remote shell program as the transport (such as rsh or
	    ssh). This is invoked when the destination path contains a
	    single @code{:} separator.

	<listitem>
	  <para>
	    for copying from a remote machine to the local machine
	    using a remote shell program. This is invoked when the source
	    contains a @code{:} separator.

	<listitem>
	  <para>
	    for copying from a remote rsync server to the local
	    machine. This is invoked when the source path contains a @code{::}
	    separator or a @code{rsync://} URL.

	<listitem>
	  <para>
	    for copying from the local machine to a remote rsync
	    server. This is invoked when the destination path contains a @code{::}
	    separator.

	<listitem>
	  <para>
	    for listing files on a remote machine. This is done the
	    same way as rsync transfers except that you leave off the
	    local destination.  

	</listitem>
      </orderedlist>
	  <para>
Note that in all cases (other than listing) at least one of the source
and destination paths must be local.

	  <para>
Any one invocation of rsync makes a copy in a single direction.  rsync
currently has no equivalent of @command{ftp}'s interactive mode.

@cindex @sc{nfs}
@cindex network filesystems
@cindex remote filesystems

	  <para>
rsync's network protocol is generally faster at copying files than
network filesystems such as @sc{nfs} or @sc{cifs}.  It is better to
run rsync on the file server either as a daemon or over ssh than
running rsync giving the network directory.
      </para>
    </section>
  </chapter>



  <chapter>
    <title>Frequently asked questions</title>

    
    <!-- one of (CALLOUTLIST GLOSSLIST ITEMIZEDLIST ORDEREDLIST SEGMENTEDLIST SIMPLELIST VARIABLELIST CAUTION IMPORTANT NOTE TIP WARNING LITERALLAYOUT PROGRAMLISTING PROGRAMLISTINGCO SCREEN SCREENCO SCREENSHOT SYNOPSIS CMDSYNOPSIS FUNCSYNOPSIS CLASSSYNOPSIS FIELDSYNOPSIS CONSTRUCTORSYNOPSIS DESTRUCTORSYNOPSIS METHODSYNOPSIS FORMALPARA PARA SIMPARA ADDRESS BLOCKQUOTE GRAPHIC GRAPHICCO MEDIAOBJECT MEDIAOBJECTCO INFORMALEQUATION INFORMALEXAMPLE INFORMALFIGURE INFORMALTABLE EQUATION EXAMPLE FIGURE TABLE MSGSET PROCEDURE SIDEBAR QANDASET ANCHOR BRIDGEHEAD REMARK HIGHLIGHTS ABSTRACT AUTHORBLURB EPIGRAPH INDEXTERM SECTION SIMPLESECT REFENTRY SECT1) -->
    <qandaset>
      <!-- one of (QANDADIV QANDAENTRY) -->

      <qandaentry>
	<question>
	  <!-- one of (CALLOUTLIST GLOSSLIST ITEMIZEDLIST ORDEREDLIST
	  SEGMENTEDLIST SIMPLELIST VARIABLELIST CAUTION IMPORTANT NOTE
	  TIP WARNING LITERALLAYOUT PROGRAMLISTING PROGRAMLISTINGCO
	  SCREEN SCREENCO SCREENSHOT SYNOPSIS CMDSYNOPSIS FUNCSYNOPSIS
	  CLASSSYNOPSIS FIELDSYNOPSIS CONSTRUCTORSYNOPSIS
	  DESTRUCTORSYNOPSIS METHODSYNOPSIS FORMALPARA PARA SIMPARA
	  ADDRESS BLOCKQUOTE GRAPHIC GRAPHICCO MEDIAOBJECT
	  MEDIAOBJECTCO INFORMALEQUATION INFORMALEXAMPLE
	  INFORMALFIGURE INFORMALTABLE EQUATION EXAMPLE FIGURE TABLE
	  PROCEDURE ANCHOR BRIDGEHEAD REMARK HIGHLIGHTS INDEXTERM) -->
	  <para>Are there mailing lists for rsync?
	</question>

	<answer>
	  <para>Yes, and you can subscribe and unsubscribe through a
	  web interface at
	    <ulink
	      url="http://lists.samba.org/">http://lists.samba.org/</ulink>
	  </para>

	  <para>
	    If you are having trouble with the mailing list, please
	    send mail to the administrator
	    
	    <email>rsync-admin@lists.samba.org</email>

	    not to the list itself.
	  </para>

	  <para>
	    The mailing list archives are searchable.  Use 
	    <ulink url="http://google.com/">Google</ulink> and prepend
	    the search with <userinput>site:lists.samba.org
	    rsync</userinput>, plus relevant keywords.
	  </para>
	</answer>
      </qandaentry>


      <qandaentry>
	<question>
	  <para>
	    Why is rsync so much bigger when I build it with
	    <command>gcc</command>?
	  </para>
	</question>
	<answer>
	  <para>
	    On gcc, rsync builds by default with debug symbols
	    included.  If you strip both executables, they should end
	    up about the same size.  (Use <command>make
	    install-strip</command>.)
	  </para>
	</answer>
      </qandaentry>

      
      <qandaentry>
	<question>
	  <para>Is rsync useful for a single large file like an ISO image?</para>
	</question>
	<answer>
	  <para>
	    Yes, but note the following:

	  <para>
   Background: A common use of rsync is to update a file (or set of files) in one location from a more
   correct or up-to-date copy in another location, taking advantage of portions of the files that are
   identical to speed up the process. (Note that rsync will transfer a file in its entirety if no copy
   exists at the destination.)

	  <para>
   (This discussion is written in terms of updating a local copy of a file from a correct file in a
   remote location, although rsync can work in either direction.)

	  <para>
   The file to be updated (the local file) must be in a destination directory that has enough space for
   two copies of the file. (In addition, keep an extra copy of the file to be updated in a different
   location for safety -- see the discussion (below) about rsync's behavior when the rsync process is
   interrupted before completion.)

	  <para>
   The local file must have the same name as the remote file being sync'd to (I think?). If you are
   trying to upgrade an iso from, for example, beta1 to beta2, rename the local file to the same name
   as the beta2 file. *(This is a useful thing to do -- only the changed portions will be
   transmitted.)*

	  <para>
   The extra copy of the local file kept in a different location is because of rsync's behavior if
   interrupted before completion:

	  <para>
   * If you specify the --partial option and rsync is interrupted, rsync will save the partially
   rsync'd file and throw away the original local copy. (The partially rsync'd file is correct but
   truncated.) If rsync is restarted, it will not have a local copy of the file to check for duplicate
   blocks beyond the section of the file that has already been rsync'd, thus the remainder of the rsync
   process will be a "pure transfer" of the file rather than taking advantage of the rsync algorithm.

	  <para>
   * If you don't specify the --partial option and rsync is interrupted, rsync will throw away the
   partially rsync'd file, and, when rsync is restarted starts the rsync process over from the
   beginning.

	  <para>
   Which of these is most desirable depends on the degree of commonality between the local and remote
   copies of the file *and how much progress was made before the interruption*.

	  <para>
   The ideal approach after an interruption would be to create a new file by taking the original file
   and deleting a portion equal in size to the portion already rsync'd and then appending *the
   remaining* portion to the portion of the file that has already been rsync'd. (There has been some
   discussion about creating an option to do this automatically.)

   The --compare-dest option is useful when transferring multiple files, but is of no benefit in
   transferring a single file. (AFAIK)

   *Other potentially useful information can be found at:
   -[3]http://twiki.org/cgi-bin/view/Wikilearn/RsyncingALargeFile

   This answer, formatted with "real" bullets, can be found at:
   -[4]http://twiki.org/cgi-bin/view/Wikilearn/RsyncingALargeFileFAQ*

	  </para>
	</answer>
      </qandaentry>
    </qandaset>
  </chapter>


  <appendix>
    <title>Other Resources</title>
    
    <para><ulink url="http://www.ccp14.ac.uk/ccp14admin/rsync/"></ulink></para>
  </appendix>
</book>