Introduction
************

AutoGen is a tool designed for generating program files that contain repetitive text with varied substitutions.  Its goal is to simplify the maintenance of programs that contain large amounts of repetitious text. This is especially valuable if there are several blocks of such text that must be kept synchronized in parallel tables.

An obvious example is the problem of maintaining the code required for processing program options and configuration settings.  Processing options requires a minimum of four different constructs be kept in proper order in different places in your program.  You need at least:

 1. The flag character in the flag string,

 2. code to process the flag when it is encountered,

 3. a global state variable or two, and

 4. a line in the usage text.

You will need more things besides this if you choose to implement long option names, configuration (rc/ini) file processing, environment variable settings and keep all the documentation for these up to date.  This can be done mechanically; with the proper templates and this program.  In fact, it has already been done and AutoGen itself uses it see AutoOpts.  For a simple example of Automated Option processing, see Quick Start.  For a full list of the Automated Option features, see Features.  Be forewarned, though, the feature list is ridiculously extensive.

The Purpose of AutoGen
======================

The idea of this program is to have a text file, a template if you will, that contains the general text of the desired output file. That file includes substitution expressions and sections of text that are replicated under the control of separate definition files.

AutoGen was designed with the following features:

 1. The definitions are completely separate from the template.  By completely isolating the definitions from the template it greatly increases the flexibility of the template implementation.  A secondary goal is that a template user only needs to specify those data that are necessary to describe his application of a template.

 2. Each datum in the definitions is named.  Thus, the definitions can be rearranged, augmented and become obsolete without it being necessary to go back and clean up older definition files.  Reduce incompatibilities!

 3. Every definition name defines an array of values, even when there is only one entry.  These arrays of values are used to control the replication of sections of the template.

 4. There are named collections of definitions.  They form a nested hierarchy. Associated values are collected and associated with a group name. These associated data are used collectively in sets of substitutions.

 5. The template has special markers to indicate where substitutions are required, much like the ${VAR} construct in a shell here doc. These markers are not fixed strings.  They are specified at the start of each template.  Template designers know best what fits into their syntax and can avoid marker conflicts.

     We did this because it is burdensome and difficult to avoid conflicts using either M4 tokenization or C preprocessor substitution rules.  It also makes it easier to specify expressions that transform the value. Of course, our expressions are less cryptic than the shell methods.

 6. These same markers are used, in conjunction with enclosed keywords, to indicate sections of text that are to be skipped and for sections of text that are to be repeated.  This is a major improvement over using C preprocessing macros.  With the C preprocessor, you have no way of selecting output text because it is an unvarying, mechanical substitution process.

 7. Finally, we supply methods for carefully controlling the output. Sometimes, it is just simply easier and clearer to compute some text or a value in one context when its application needs to be later.  So, functions are available for saving text or values for later use.

A Simple Example
================

This is just one simple example that shows a few basic features. If you are interested, you also may run "make check" with the VERBOSE environment variable set and see a number of other examples in the agen5/test/testdir directory.

Assume you have an enumeration of names and you wish to associate some string with each name.  Assume also, for the sake of this example, that it is either too complex or too large to maintain easily by hand. We will start by writing an abbreviated version of what the result is supposed to be.  We will use that to construct our output templates.

In a header file, list.h, you define the enumeration and the global array containing the associated strings:

     typedef enum {
             IDX_ALPHA,
             IDX_BETA,
             IDX_OMEGA }  list_enum;

     extern char const* az_name_list[ 3 ];

Then you also have list.c that defines the actual strings:

     #include "list.h"
     char const* az_name_list[] = {
             "some alpha stuff",
             "more beta stuff",
             "final omega stuff" };

First, we will define the information that is unique for each enumeration name/string pair.  This would be placed in a file named, list.def, for example.

     autogen definitions list;
     list = { list_element = alpha;
              list_info    = "some alpha stuff"; };
     list = { list_info    = "more beta stuff";
              list_element = beta; };
     list = { list_element = omega;
              list_info    = "final omega stuff"; };

The autogen definitions list; entry defines the file as an AutoGen definition file that uses a template named list.  That is followed by three list entries that define the associations between the enumeration names and the strings.  The order of the differently named elements inside of list is unimportant.  They are reversed inside of the beta entry and the output is unaffected.

Now, to actually create the output, we need a template or two that can be expanded into the files you want.  In this program, we use a single template that is capable of multiple output files.  The definitions above refer to a list template, so it would normally be named, list.tpl.

It looks something like this. (For a full description, see Template File.)

     [+ AutoGen5 template h c +]
     [+ CASE (suffix) +][+
        ==  h  +]
     typedef enum {[+
        FOR list "," +]
             IDX_[+ (string-upcase! (get "list_element")) +][+
        ENDFOR list +] }  list_enum;

     extern char const* az_name_list[ [+ (count "list") +] ];
     [+

        ==  c  +]
     #include "list.h"
     char const* az_name_list[] = {[+
       FOR list "," +]
             "[+list_info+]"[+
       ENDFOR list +] };[+

     ESAC +]

The [+ AutoGen5 template h c +] text tells AutoGen that this is an AutoGen version 5 template file; that it is to be processed twice; that the start macro marker is [+; and the end marker is +].  The template will be processed first with a suffix value of h and then with c.  Normally, the suffix values are appended to the base-name to create the output file name.

The [+ == h +] and [+ == c +] CASE selection clauses select different text for the two different passes.  In this example, the output is nearly disjoint and could have been put in two separate templates.  However, sometimes there are common sections and this is just an example.

The [+FOR list "," +] and [+ ENDFOR list +] clauses delimit a block of text that will be repeated for every definition of list. Inside of that block, the definition name-value pairs that are members of each list are available for substitutions.

The remainder of the macros are expressions.  Some of these contain special expression functions that are dependent on AutoGen named values; others are simply Scheme expressions, the result of which will be inserted into the output text.  Other expressions are names of AutoGen values.  These values will be inserted into the output text.  For example, [+list_info+] will result in the value associated with the name list_info being inserted between the double quotes and (string-upcase! (get "list_element")) will first "get" the value associated with the name list_element, then change the case of all the letters to upper case.  The result will be inserted into the output document.

If you have compiled AutoGen, you can copy out the template and definitions as described above and run autogen list.def.  This will produce exactly the hypothesized desired output.

One more point, too.  Lets say you decided it was too much trouble to figure out how to use AutoGen, so you created this enumeration and string list with thousands of entries.  Now, requirements have changed and it has become necessary to map a string containing the enumeration name into the enumeration number.  With AutoGen, you just alter the template to emit the table of names. It will be guaranteed to be in the correct order, missing none of the entries. If you want to do that by hand, well, good luck.

csh/zsh caveat
==============

AutoGen tries to use your normal shell so that you can supply shell code in a manner you are accustomed to using.  If, however, you use csh or zsh, you cannot do this.  Csh is sufficiently difficult to program that it is unsupported.  Zsh, though largely programmable, also has some anomalies that make it incompatible with AutoGen usage.  Therefore, when invoking AutoGen from these environments, you must be certain to set the SHELL environment variable to a Bourne-derived shell, e.g., sh, ksh or bash.

Any shell you choose for your own scripts need to follow these basic requirements:

 1. It handles trap ":" $sig without output to standard out.  This is done when the server shell is first started.  If your shell does not handle this, then it may be able to by loading functions from its start up files.

 2. At the beginning of each scriptlet, the command \\cd $PWD is inserted.  This ensures that cd is not aliased to something peculiar and each scriptlet starts life in the execution directory.

 3. At the end of each scriptlet, the command echo mumble is appended.  The program you use as a shell must emit the single argument mumble on a line by itself.

A User's Perspective
====================

     Alexandre wrote:
     >
     > I'd appreciate opinions from others about advantages/disadvantages of
     > each of these macro packages.

I am using AutoGen in my pet project, and find one of its best points to be that it separates the operational data from the implementation.

Indulge me for a few paragraphs, and all will be revealed: In the manual, Bruce cites the example of maintaining command line flags inside the source code; traditionally spreading usage information, flag names, letters and processing across several functions (if not files). Investing the time in writing a sort of boiler plate (a template in AutoGen terminology) pays by moving all of the option details (usage, flags names etc.) into a well structured table (a definition file if you will),  so that adding a new command line option becomes a simple matter of adding a set of details to the table.

So far so good!  Of course, now that there is a template, writing all of that tedious optargs processing and usage functions is no longer an issue.  Creating a table of the options needed for the new project and running AutoGen generates all of the option processing code in C automatically from just the tabular data.  AutoGen in fact already ships with such a template... AutoOpts.

One final consequence of the good separation in the design of AutoGen is that it is retargetable to a greater extent.  The egcs/gcc/fixinc/inclhack.def can equally be used (with different templates) to create a shell script (inclhack.sh) or a c program (fixincl.c).

This is just the tip of the iceberg.  AutoGen is far more powerful than these examples might indicate, and has many other varied uses.  I am certain Bruce or I could supply you with many and varied examples, and I would heartily recommend that you try it for your project and see for yourself how it compares to m4.

As an aside, I would be interested to see whether someone might be persuaded to rationalise autoconf with AutoGen in place of m4...  Ben, are you listening?  autoconf-3.0! `kay?  =)O|

     Sincerely,
             Gary V. Vaughan

Definitions File
****************

This chapter describes the syntax and semantics of the AutoGen definition file.  In order to instantiate a template, you normally must provide a definitions file that identifies itself and contains some value definitions.  Consequently, we keep it very simple.  For "advanced" users, there are preprocessing directives, sparse arrays, named indexes and comments that may be used as well.

The definitions file is used to associate values with names.  Every value is implicitly an array of values, even if there is only one value. Values may be either simple strings or compound collections of name-value pairs.  An array may not contain both simple and compound members.  Fundamentally, it is as simple as:

     prog-name = "autogen";
     flag = {
         name      = templ_dirs;
         value     = L;
         descrip   = "Template search directory list";
     };

For purposes of commenting and controlling the processing of the definitions, C-style comments and most C preprocessing directives are honored.  The major exception is that the #if directive is ignored, along with all following text through the matching #endif directive.  The C preprocessor is not actually invoked, so C macro substitution is not performed.

The Identification Definition
=============================

The first definition in this file is used to identify it as a AutoGen file.  It consists of the two keywords, autogen and definitions followed by the default template name and a terminating semi-colon (;).  That is:

             AutoGen Definitions TEMPLATE-NAME;

Note that, other than the name TEMPLATE-NAME, the words AutoGen and Definitions are searched for without case sensitivity.  Most lookups in this program are case insensitive.

Also, if the input contains more identification definitions, they will be ignored.  This is done so that you may include (see Directives) other definition files without an identification conflict.

AutoGen uses the name of the template to find the corresponding template file.  It searches for the file in the following way, stopping when it finds the file:

 1. It tries to open ./TEMPLATE-NAME.  If it fails,

 2. it tries ./TEMPLATE-NAME.tpl.

 3. It searches for either of these files in the directories listed in the templ-dirs command line option.

If AutoGen fails to find the template file in one of these places, it prints an error message and exits.

Named Definitions
=================

A name is a sequence of characters beginning with an alphabetic character (a through z) followed by zero or more alpha-numeric characters and/or separator characters: hyphen (-), underscore (_) or carat (^).  Names are case insensitive.

Any name may have multiple values associated with it.  Every name may be considered a sparse array of one or more elements.  If there is more than one value, the values my be accessed by indexing the value with [index] or by iterating over them using the FOR (see FOR) AutoGen macro on it, as described in the next chapter.  Sparse arrays are specified by specifying an index when defining an entry (see Assigning an Index to a Definition).

There are two kinds of definitions, simple and compound. They are defined thus (see Full Syntax):

     compound_name '=' '{' definition-list '}' ';'

     simple-name[2] '=' string ';'

     no^text^name ';'

simple-name has the third index (index number 2) defined here. No^text^name is a simple definition with a shorthand empty string value.  The string values for definitions may be specified in any of several formation rules.

Definition List
---------------

definition-list is a list of definitions that may or may not contain nested compound definitions.  Any such definitions may only be expanded within a FOR block iterating over the containing compound definition.  See FOR.

Here is, again, the example definitions from the previous chapter, with three additional name value pairs.  Two with an empty value assigned (FIRST and LAST), and a "global" GROUP_NAME.

     autogen definitions list;
     group_name = example;
     list = { list_element = alpha;  first;
              list_info    = "some alpha stuff"; };
     list = { list_info    = "more beta stuff";
              list_element = beta; };
     list = { list_element = omega;  last;
              list_info    = "final omega stuff"; };

Double Quote String
-------------------

The string follows the C-style escaping, using the backslash to quote (escape) the following character(s).  Certain letters are translated to various control codes (e.g. \n, \f, \t, etc.). x introduces a two character hex code.  0 (the digit zero) introduces a one to three character octal code (note: an octal byte followed by a digit must be represented with three octal digits, thus: "\0001" yielding a NUL byte followed by the ASCII digit 1).  Any other character following the backslash escape is simply inserted, without error, into the string being formed.

Like ANSI "C", a series of these strings, possibly intermixed with single quote strings, will be concatenated together.

Single Quote String
-------------------

This is similar to the shell single-quote string.  However, escapes \ are honored before another escape, single quotes ' and hash characters #.  This latter is done specifically to disambiguate lines starting with a hash character inside of a quoted string.  In other words,

     fumble = '
     #endif
     ';

could be misinterpreted by the definitions scanner, whereas this would not:

     fumble = '
     \#endif
     ';

As with the double quote string, a series of these, even intermixed with double quote strings, will be concatenated together.

An Unquoted String
------------------

A simple string that does not contain white space may be left unquoted.  The string must not contain any of the characters special to the definition text (i.e., ", #, ', (, ), ,, ;, <, =, >, [, ], `, {, or }).  This list is subject to change, but it will never contain underscore (_), period (.), slash (/), colon (:), hyphen (-) or backslash (\\).  Basically, if the string looks like it is a normal DOS or UNIX file or variable name, and it is not one of two keywords (autogen or definitions) then it is OK to not quote it, otherwise you should.

Shell Output String
-------------------

This is assembled according to the same rules as the double quote string, except that there is no concatenation of strings and the resulting string is written to a shell server process.  The definition takes on the value of the output string.

NB The text is interpreted by a server shell.  There may be left over state from previous server shell processing.  This scriptlet may also leave state for subsequent processing.  However, a cd to the original directory is always issued before the new command is issued.

Scheme Result String
--------------------

A scheme result string must begin with an open parenthesis (. The scheme expression will be evaluated by Guile and the value will be the result.  The AutoGen expression functions are disabled at this stage, so do not use them.

A Here String
-------------

A here string is formed in much the same way as a shell here doc.  It is denoted with two less than characters(<<) and, optionally, a hyphen. This is followed by optional horizontal white space and an ending marker-identifier.  This marker must follow the syntax rules for identifiers. Unlike the shell version, however, you must not quote this marker.

The resulting string will start with the first character on the next line and continue up to but not including the newline that precedes the line that begins with the marker token.  The characters are copied directly into the result string.  Mostly.

If a hyphen follows the less than characters, then leading tabs will be stripped and the terminating marker will be recognized even if preceded by tabs.  Also, if the first character on the line (after removing tabs) is a backslash and the next character a tab, then the backslash will be removed as well.  No other kind of processing is done on this string.

Here are two examples:

     str1 = <<-  STR_END
             $quotes = " ' `
             STR_END;

     str2 = <<   STR_END
             $quotes = " ' `
             STR_END;
     STR_END;

The first string contains no new line characters. The first character is the dollar sign, the last the back quote.

The second string contains one new line character.  The first character is the tab character preceding the dollar sign.  The last character is the semicolon after the STR_END.  That STR_END does not end the string because it is not at the beginning of the line.  In the preceding case, the leading tab was stripped.

Concatenated Strings
--------------------

If single or double quote characters are used, then you also have the option, a la ANSI-C syntax, of implicitly concatenating a series of them together, with intervening white space ignored.

NB  You cannot use directives to alter the string content.  That is,

     str = "fumble"
     #ifdef LATER
           "stumble"
     #endif
           ;

will result in a syntax error.  The preprocessing directives are not carried out by the C preprocessor.  However,

     str = '"fumble\n"
     #ifdef LATER
     "     stumble\n"
     #endif
     ';

Will work.  It will enclose the #ifdef LATER and #endif in the string.  But it may also wreak havoc with the definition processing directives.  The hash characters in the first column should be disambiguated with an escape \ or join them with previous lines: "fumble\n#ifdef LATER....

Assigning an Index to a Definition
==================================

In AutoGen, every name is implicitly an array of values. When assigning values, they are usually implicitly assigned to the next highest slot.  They can also be specified explicitly:

     mumble[9] = stumble;
     mumble[0] = grumble;

If, subsequently, you assign a value to mumble without an index, its index will be 10, not 1. If indexes are specified, they must not cause conflicts.

#define-d names may also be used for index values. This is equivalent to the above:

     #define FIRST 0
     #define LAST  9
     mumble[LAST]  = stumble;
     mumble[FIRST] = grumble;

All values in a range do not have to be filled in. If you leave gaps, then you will have a sparse array.  This is fine (see FOR).  You have your choice of iterating over all the defined values, or iterating over a range of slots.  This:

     [+ FOR mumble +][+ ENDFOR +]

iterates over all and only the defined entries, whereas this:

     [+ FOR mumble (for-by 1) +][+ ENDFOR +]

will iterate over all 10 "slots".  Your template will likely have to contain something like this:

     [+ IF (exist? (sprintf "mumble[%d]" (for-index))) +]

or else "mumble" will have to be a compound value that, say, always contains a "grumble" value:

     [+ IF (exist? "grumble") +]

Dynamic Text
============

There are several methods for including dynamic content inside a definitions file.  Three of them are mentioned above (shell-generated and see scheme-generated) in the discussion of string formation rules. Another method uses the #shell processing directive. It will be discussed in the next section (see Directives). Guile/Scheme may also be used to yield to create definitions.

When the Scheme expression is preceded by a backslash and single quote, then the expression is expected to be an alist of names and values that will be used to create AutoGen definitions.

This method can be be used as follows:

     \'( (name  (value-expression))
         (name2 (another-expr))  )

This is entirely equivalent to:

     name  = (value-expression);
     name2 = (another-expr);

Under the covers, the expression gets handed off to a Guile function named alist->autogen-def in an expression that looks like this:

     (alist->autogen-def
         ( (name (value-expression))  (name2 (another-expr)) ) )