diff options
Diffstat (limited to 'doc/gawkinet.texi')
| -rw-r--r-- | doc/gawkinet.texi | 5075 |
1 files changed, 5075 insertions, 0 deletions
diff --git a/doc/gawkinet.texi b/doc/gawkinet.texi new file mode 100644 index 00000000..2ffb5814 --- /dev/null +++ b/doc/gawkinet.texi @@ -0,0 +1,5075 @@ +\input texinfo @c -*-texinfo-*- +@c %**start of header (This is for running Texinfo on a region.) +@setfilename gawkinet.info +@settitle TCP/IP Internetworking With @command{gawk} +@c %**end of header (This is for running Texinfo on a region.) + +@c inside ifinfo for older versions of texinfo.tex +@ifinfo +@dircategory GNU Packages +@direntry +* Gawkinet: (gawkinet). TCP/IP Internetworking With @command{gawk}. +@end direntry +@end ifinfo + +@iftex +@set DOCUMENT book +@set CHAPTER chapter +@set SECTION section +@set DARKCORNER @inmargin{@image{lflashlight,1cm}, @image{rflashlight,1cm}} +@end iftex +@ifinfo +@set DOCUMENT Info file +@set CHAPTER major node +@set SECTION node +@set DARKCORNER (d.c.) +@end ifinfo +@ifhtml +@set DOCUMENT web page +@set CHAPTER chapter +@set SECTION section +@set DARKCORNER (d.c.) +@end ifhtml + +@set FSF + +@set FN file name +@set FFN File Name + +@c merge the function and variable indexes into the concept index +@ifinfo +@synindex fn cp +@synindex vr cp +@end ifinfo +@iftex +@syncodeindex fn cp +@syncodeindex vr cp +@end iftex + +@c If "finalout" is commented out, the printed output will show +@c black boxes that mark lines that are too long. Thus, it is +@c unwise to comment it out when running a master in case there are +@c overfulls which are deemed okay. + +@iftex +@finalout +@end iftex + +@smallbook + +@c Special files are described in chapter 6 Printing Output under +@c 6.7 Special File Names in gawk. I think the networking does not +@c fit into that chapter, thus this separate document. At over 50 +@c pages, I think this is the right decision. ADR. + +@set TITLE TCP/IP Internetworking With @command{gawk} +@set EDITION 1.1 +@set UPDATE-MONTH March, 2001 +@c gawk versions: +@set VERSION 3.1 +@set PATCHLEVEL 0 + +@ifinfo +This file documents the networking features in GNU @command{awk}. + +This is Edition @value{EDITION} of @cite{@value{TITLE}}, +for the @value{VERSION}.@value{PATCHLEVEL} (or later) version of the GNU +implementation of AWK. + +Copyright (C) 2000, 2001 Free Software Foundation, Inc. + +Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.1 or +any later version published by the Free Software Foundation; with the +Invariant Sections being ``GNU General Public License'', the Front-Cover +texts being (a) (see below), and with the Back-Cover Texts being (b) +(see below). A copy of the license is included in the section entitled +``GNU Free Documentation License''. + +@enumerate a +@item +``A GNU Manual'' + +@item +``You have freedom to copy and modify this GNU Manual, like GNU +software. Copies published by the Free Software Foundation raise +funds for GNU development.'' +@end enumerate +@end ifinfo + +@setchapternewpage odd + +@titlepage +@title @value{TITLE} +@subtitle Edition @value{EDITION} +@subtitle @value{UPDATE-MONTH} +@author J@"urgen Kahrs +@author with Arnold D. Robbins + +@c Include the Distribution inside the titlepage environment so +@c that headings are turned off. Headings on and off do not work. + +@page +@vskip 0pt plus 1filll +Copyright @copyright{} 2000, 2001 Free Software Foundation, Inc. +@sp 1 +@b{User Friendly} Copyright @copyright{} 2000 J.D.@: ``Iliad'' Frazier. +Reprinted by permission. +@sp 2 + +This is Edition @value{EDITION} of @cite{@value{TITLE}}, +for the @value{VERSION}.@value{PATCHLEVEL} (or later) version of the GNU +implementation of AWK. + +@sp 2 +Published by: +@sp 1 + +Free Software Foundation @* +59 Temple Place --- Suite 330 @* +Boston, MA 02111-1307 USA @* +Phone: +1-617-542-5942 @* +Fax: +1-617-542-2652 @* +Email: @email{gnu@@gnu.org} @* +URL: @uref{http://www.gnu.org/} @* + +ISBN 1-882114-93-0 @* + +Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.1 or +any later version published by the Free Software Foundation; with the +Invariant Sections being ``GNU General Public License'', the Front-Cover +texts being (a) (see below), and with the Back-Cover Texts being (b) +(see below). A copy of the license is included in the section entitled +``GNU Free Documentation License''. + +@enumerate a +@item +``A GNU Manual'' + +@item +``You have freedom to copy and modify this GNU Manual, like GNU +software. Copies published by the Free Software Foundation raise +funds for GNU development.'' +@end enumerate +@c @sp 2 +@c Cover art by ?????. +@end titlepage + +@iftex +@headings off +@evenheading @thispage@ @ @ @strong{@value{TITLE}} @| @| +@oddheading @| @| @strong{@thischapter}@ @ @ @thispage +@end iftex + +@ifinfo +@node Top, Preface, (dir), (dir) +@top General Introduction +@comment node-name, next, previous, up + +This file documents the networking features in GNU Awk (@command{gawk}) +version 3.1 and later. +@end ifinfo + +@menu +* Preface:: About this document. +* Introduction:: About networkiing. +* Using Networking:: Some examples. +* Some Applications and Techniques:: More extended examples. +* Links:: Where to find the stuff mentioned in this + document. +* GNU Free Documentation License:: The license for this document. +* Index:: The index. + +@detailmenu +* Stream Communications:: Sending data streams. +* Datagram Communications:: Sending self-contained messages. +* The TCP/IP Protocols:: How these models work in the Internet. +* Basic Protocols:: The basic protocols. +* Ports:: The idea behind ports. +* Making Connections:: Making TCP/IP connections. +* Gawk Special Files:: How to do @command{gawk} networking. +* Special File Fields:: The fields in the special file name. +* Comparing Protocols:: Differences between the protocols. +* File /inet/tcp:: The TCP special file. +* File /inet/udp:: The UDB special file. +* File /inet/raw:: The RAW special file. +* TCP Connecting:: Making a TCP connection. +* Troubleshooting:: Troubleshooting TCP/IP connections. +* Interacting:: Interacting with a service. +* Setting Up:: Setting up a service. +* Email:: Reading email. +* Web page:: Reading a Web page. +* Primitive Service:: A primitive Web service. +* Interacting Service:: A Web service with interaction. +* CGI Lib:: A simple CGI library. +* Simple Server:: A simple Web server. +* Caveats:: Network programming caveats. +* Challenges:: Where to go from here. +* PANIC:: An Emergency Web Server. +* GETURL:: Retrieving Web Pages. +* REMCONF:: Remote Configuration Of Embedded Systems. +* URLCHK:: Look For Changed Web Pages. +* WEBGRAB:: Extract Links From A Page. +* STATIST:: Graphing A Statistical Distribution. +* MAZE:: Walking Through A Maze In Virtual Reality. +* MOBAGWHO:: A Simple Mobile Agent. +* STOXPRED:: Stock Market Prediction As A Service. +* PROTBASE:: Searching Through A Protein Database. +@end detailmenu +@end menu + +@contents + +@node Preface, Introduction, Top, Top +@unnumbered Preface + +In May of 1997, J@"urgen Kahrs felt the need for network access +from @command{awk}, and, with a little help from me, set about adding +features to do this for @command{gawk}. At that time, he +wrote the bulk of this @value{DOCUMENT}. + +The code and documentation were added to the @command{gawk} 3.1 development +tree, and languished somewhat until I could finally get +down to some serious work on that version of @command{gawk}. +This finally happened in the middle of 2000. + +Meantime, J@"urgen wrote an article about the Internet special +files and @samp{|&} operator for @cite{Linux Journal}, and made a +networking patch for the production versions of @command{gawk} +available from his home page. +In August of 2000 (for @command{gawk} 3.0.6), this patch +also made it to the main GNU @command{ftp} distribution site. + +For release with @command{gawk}, I edited J@"urgen's prose +for English grammar and style, as he is not a native English +speaker. I also +rearranged the material somewhat for what I felt was a better order of +presentation, and (re)wrote some of the introductory material. + +The majority of this document and the code are his work, and the +high quality and interesting ideas speak for themselves. It is my +hope that these features will be of significant value to the @command{awk} +community. + +@sp 1 +@noindent +Arnold Robbins @* +Nof Ayalon, ISRAEL @* +March, 2001 + +@node Introduction, Using Networking, Preface, Top +@chapter Networking Concepts + +This @value{CHAPTER} provides a (necessarily) brief intoduction to +computer networking concepts. For many applications of @command{gawk} +to TCP/IP networking, we hope that this is enough. For more +advanced tasks, you will need deeper background, and it may be necessary +to switch to lower-level programming in C or C++. + +There are two real-life models for the way computers send messages +to each other over a network. While the analogies are not perfect, +they are close enough to convey the major concepts. +These two models are the phone system (reliable byte-stream communications), +and the postal system (best-effort datagrams). + +@menu +* Stream Communications:: Sending data streams. +* Datagram Communications:: Sending self-contained messages. +* The TCP/IP Protocols:: How these models work in the Internet. +* Making Connections:: Making TCP/IP connections. +@end menu + +@node Stream Communications, Datagram Communications, Introduction, Introduction +@section Reliable Byte-streams (Phone Calls) + +When you make a phone call, the following steps occur: + +@enumerate +@item +You dial a number. + +@item +The phone system connects to the called party, telling +them there is an incoming call. (Their phone rings.) + +@item +The other party answers the call, or, in the case of a +computer network, refuses to answer the call. + +@item +Assuming the other party answers, the connection between +you is now a @dfn{duplex} (two-way), @dfn{reliable} (no data lost), +sequenced (data comes out in the order sent) data stream. + +@item +You and your friend may now talk freely, with the phone system +moving the data (your voices) from one end to the other. +From your point of view, you have a direct end-to-end +connection with the person on the other end. +@end enumerate + +The same steps occur in a duplex reliable computer networking connection. +There is considerably more overhead in setting up the communications, +but once it's done, data moves in both directions, reliably, in sequence. + +@node Datagram Communications, The TCP/IP Protocols, Stream Communications, Introduction +@section Best-effort Datagrams (Mailed Letters) + +Suppose you mail three different documents to your office on the +other side of the country on two different days. Doing so +entails the following. + +@enumerate +@item +Each document travels in its own envelope. + +@item +Each envelope contains both the sender and the +recipient address. + +@item +Each envelope may travel a different route to its destination. + +@item +The envelopes may arrive in a different order from the one +in which they were sent. + +@item +One or more may get lost in the mail. +(Although, fortunately, this does not occur very often.) + +@item +In a computer network, one or more @dfn{packets} +may also arrive multiple times. (This doesn't happen +with the postal system!) + +@end enumerate + +The important characteristics of datagram communications, like +those of the postal system are thus: + +@itemize @bullet +@item +Delivery is ``best effort;'' the data may never get there. + +@item +Each message is self-contained, including the source and +destination addresses. + +@item +Delivery is @emph{not} sequenced; packets may arrive out +of order, and/or multiple times. + +@item +Unlike the phone system, overhead is considerably lower. +It is not necessary to set up the call first. +@end itemize + +The price the user pays for the lower overhead of datagram communications +is exactly the lower reliability; it is often necessary for user-level +protocols that use datagram communications to add their own reliabilty +features on top of the basic communications. + +@node The TCP/IP Protocols, Making Connections, Datagram Communications, Introduction +@section The Internet Protocols + +The Internet Protocol Suite (usually referred as just TCP/IP)@footnote{ +It should be noted that although the Internet seems to have conquered the +world, there are other networking protocol suites in existence and in use.} +consists of a number of different protocols at different levels or ``layers.'' +For our purposes, three protocols provide the fundamental communications +mechanisms. All other defined protocols are referred to as user-level +protocols (e.g., HTTP, used later in this @value{DOCUMENT}). + +@menu +* Basic Protocols:: The basic protocols. +* Ports:: The idea behind ports. +@end menu + +@node Basic Protocols, Ports, The TCP/IP Protocols, The TCP/IP Protocols +@subsection The Basic Internet Protocols + +@table @asis +@item IP +The Internet Protocol. This protocol is almost never used directly by +applications. It provides the basic packet delivery and routing infrastructure +of the Internet. Much like the phone company's switching centers or the Post +Office's trucks, it is not of much day-to-day interest to the regular user +(or programmer). +It happens to be a best effort datagram protocol. + +@item UDP +The User Datagram Protocol. This is a best effort datagram protocol. +It provides a small amount of extra reliability over IP, and adds +the notion of @dfn{ports}, described in @ref{Ports, ,TCP and UDP Ports}. + +@item TCP +The Transmission Control Protocol. This is a duplex, reliable, sequenced +byte-stream protocol, again layered on top of IP, and also providing the +notion of ports. This is the protocol that you will most likely use +when using @command{gawk} for network programming. +@end table + +All other user-level protocols use either TCP or UDP to do their basic +communications. Examples are SMTP (Simple Mail Transfer Protocol), +FTP (File Transfer Protocol) and HTTP (HyperText Transfer Protocol). +@cindex SMTP +@cindex FTP +@cindex HTTP + +@node Ports, , Basic Protocols, The TCP/IP Protocols +@subsection TCP and UDP Ports + +In the postal system, the address on an envelope indicates a physical +location, such as a residence or office building. But there may be +more than one person at the location; thus you have to further quantify +the recipient by putting a person or company name on the envelope. + +In the phone system, one phone number may represent an entire company, +in which case you need a person's extension number in order to +reach that individual directly. Or, when you call a home, you have to +say, ``May I please speak to ...'' before talking to the person directly. + +IP networking provides the concept of addressing. An IP address represents +a particular computer, but no more. In order to reach the mail service +on a system, or the FTP or WWW service on a system, you have to have some +way to further specify which service you want. In the Internet Protocol suite, +this is done with @dfn{port numbers}, which represent the services, much +like an extension number used with a phone number. + +Port numbers are 16-bit integers. Unix and Unix-like systems reserve ports +below 1024 for ``well known'' services, such as SMTP, FTP, and HTTP. +Numbers above 1024 may be used by any application, although there is no +promise made that a particular port number is always available. + +@node Making Connections, , The TCP/IP Protocols, Introduction +@section Making TCP/IP Connections (And Some Terminology) + +Two terms come up repeatedly when discussing networking: +@dfn{client} and @dfn{server}. For now, we'll discuss these terms +at the @dfn{connection level}, when first establishing connections +between two processes on different systems over a network. +(Once the connection is established, the higher level, or +@dfn{application level} protocols, +such as HTTP or FTP, determine who is the client and who is the +server. Often, it turns out that the client and server are the +same in both roles.) + +@cindex server +The @dfn{server} is the system providing the service, such as the +web server or email server. It is the @dfn{host} (system) which +is @emph{connected to} in a transaction. +For this to work though, the server must be expecting connections. +Much as there has to be someone at the office building to answer +the phone@footnote{In the days before voice mail systems!}, the +server process (usually) has to be started first and waiting +for a connection. + +@cindex client +The @dfn{client} is the system requesting the service. +It is the system @emph{initiating the connection} in a transaction. +(Just as when you pick up the phone to call an office or store.) + +In the TCP/IP framework, each end of a connection is represented by a pair +of (@var{address}, @var{port}) pairs. For the duration of the connection, +the ports in use at each end are unique, and cannot be used simultaneously +by other processes on the same system. (Only after closing a connection +can a new one be built up on the same port. This is contrary to the usual +behavior of fully developed web servers which have to avoid situations +in which they are not reachable. We have to pay this price in order to +enjoy the benefits of a simple communication paradigm in @command{gawk}.) + +@cindex blocking +@cindex synchronous communications +Furthermore, once the connection is established, communications +are @dfn{synchronous}. I.e., each end waits on the other to finish +transmitting, before replying. This is much like two people in a phone +conversation. While both could talk simultaneously, doing so usually +doesn't work too well. + +In the case of TCP, the synchronicity is enforced by the protocol when +sending data. Data writes @dfn{block} until the data have been received on the +other end. For both TCP and UDP, data reads block until there is incoming +data waiting to be read. This is summarized in the following table, +where an ``X'' indicates that the given action blocks. + +@ifnottex +@multitable {Protocol} {Reads} {Writes} +@item TCP @tab X @tab X +@item UDP @tab X @tab +@item RAW @tab X @tab +@end multitable +@end ifnottex +@tex +\centerline{ +\vbox{\bigskip % space above the table (about 1 linespace) +% Because we have vertical rules, we can't let TeX insert interline space +% in its usual way. +\offinterlineskip +\halign{\hfil\strut# &\vrule #& \hfil#\hfil& \hfil#\hfil\cr +Protocol&&\quad Reads\quad &Writes\cr +\noalign{\hrule} +\omit&height 2pt\cr +\noalign{\hrule height0pt}% without this the rule does not extend; why? +TCP&&X&X\cr +UDP&&X&\cr +RAW&&X&\cr +}}} +@end tex + +@node Using Networking, Some Applications and Techniques, Introduction, Top +@comment node-name, next, previous, up +@chapter Networking With @command{gawk} + +@cindex network +The @command{awk} programming language was originally developed as a +pattern-matching language for writing short programs to perform +data manipulation tasks. +@command{awk}'s strength is the manipulation of textual data +that is stored in files. +It was never meant to be used for networking purposes. +To exploit its features in a +networking context, it's necessary to use an access mode for network connections +that resembles the access of files as closely as possible. + +@cindex Perl +@cindex Python +@cindex Tcl/Tk +@command{awk} is also meant to be a prototyping language. It is used +to demonstrate feasibility and to play with features and user interfaces. +This can be done with file-like handling of network +connections. +@command{gawk} trades the lack +of many of the advanced features of the TCP/IP family of protocols +for the convenience of simple connection handling. +The advanced +features are available when programming in C or Perl. In fact, the +network programming +in this @value{CHAPTER} +is very similar to what is described in books like +@cite{Internet Programming with Python}, +@cite{Advanced Perl Programming}, +or +@cite{Web Client Programming with Perl}. +But it's done here without first having to learn object-oriented ideology, underlying +languages such as Tcl/Tk, Perl, Python, or all of the libraries necessary to +extend these languages before they are ready for the Internet. + +This @value{CHAPTER} demonstrates how to use the TCP protocol. The +other protocols are much less important for most users (UDP) or even +untractable (RAW). + +@menu +* Gawk Special Files:: How to do @command{gawk} networking. +* TCP Connecting:: Making a TCP connection. +* Troubleshooting:: Troubleshooting TCP/IP connections. +* Interacting:: Interacting with a service. +* Setting Up:: Setting up a service. +* Email:: Reading email. +* Web page:: Reading a Web page. +* Primitive Service:: A primitive Web service. +* Interacting Service:: A Web service with interaction. +* Simple Server:: A simple Web server. +* Caveats:: Network programming caveats. +* Challenges:: Where to go from here. +@end menu + +@node Gawk Special Files, TCP Connecting, Using Networking, Using Networking +@comment node-name, next, previous, up +@section @command{gawk} Networking Mechanisms +@cindex network + +The @samp{|&} operator introduced in @command{gawk} 3.1 for use in +communicating with a @dfn{co-process} is described in +@ref{Two-way I/O, ,Two-way Communications With Another Process, gawk, GAWK: Effective AWK Programming}. +It shows how to do two-way I/O to a +separate process, sending it data with @code{print} or @code{printf} and +reading data with @code{getline}. If you haven't read it already, you should +detour there to do so. + +@command{gawk} transparently extends the two-way I/O mechanism to simple networking through +the use of special @value{FN}s. When a ``co-process'' is started that matches +the special files we are about to describe, @command{gawk} creates the appropriate network +connection, and then two-way I/O proceeds as usual. + +At the C, C++ (and basic Perl) level, networking is accomplished +via @dfn{sockets}, an Application Programming Interface (API) originally +developed at the University of California at Berkeley that is now used +almost universally for TCP/IP networking. +Socket level programming, while fairly straightforward, requires paying +attention to a number of details, as well as using binary data. It is not +well-suited for use from a high-level language like @command{awk}. +The special files provided in @command{gawk} hide the details from +the programmer, making things much simpler and easier to use. +@c Who sez we can't toot our own horn occasionally? + +The special @value{FN} for network access is made up of several fields, all +of them mandatory, none of them optional: + +@example +/inet/@var{protocol}/@var{localport}/@var{hostname}/@var{remoteport} +@end example + +The @file{/inet/} field is, of course, constant when accessing the network. +The @var{localport} and @var{remoteport} fields do not have a meaning +when used with @file{/inet/raw} because ``ports'' only apply to +TCP and UDP. So, when using @file{/inet/raw}, the port fields always have +to be @samp{0}. + +@menu +* Special File Fields:: The fields in the special file name. +* Comparing Protocols:: Differences between the protocols. +@end menu + +@node Special File Fields, Comparing Protocols, Gawk Special Files, Gawk Special Files +@subsection The Fields of the Special @value{FFN} +This @value{SECTION} explains the meaning of all the other fields, +as well as the range of values and the defaults. +All of the fields are mandatory. To let the system pick a value, +or if the field doesn't apply to the protocol, specify it as @samp{0}. + +@table @var +@item protocol +Determines which member of the TCP/IP +family of protocols is selected to transport the data across the +network. There are three possible values (always written in lowercase): +@samp{tcp}, @samp{udp}, and @samp{raw}. The exact meaning of each is +explained later in this @value{SECTION}. + +@item localport +Determines which port on the local +machine is used to communicate across the network. It has no meaning +with @file{/inet/raw} and must therefore be @samp{0}. Application level clients +usually use @samp{0} to indicate they do not care which local port is +used---instead they specify a remote port to connect to. It is vital for +application level servers to use a number different from @samp{0} here +because their service has to be available at a specific publicly-known +port number. It is possible to use a name from @file{/etc/services} here. + +@item hostname +Determines which remote host is to +be at the other end of the connection. Application level servers must fill +this field with a @samp{0} to indicate their being open for all other hosts +to connect to them and enforce connection level server behavior this way. +It is not possible for an application level server to restrict its +availability to one remote host by entering a host name here. +Application level clients must enter a name different from @samp{0}. +The name can be either symbolic +(e.g., @samp{jpl-devvax.jpl.nasa.gov}) or numeric (e.g., @samp{128.149.1.143}). + +@item remoteport +Determines which port on the remote +machine is used to communicate across the network. It has no meaning +with @file{/inet/raw} and must therefore be 0. +For @file{/inet/tcp} and @file{/inet/udp}, +application level clients @emph{must} use a number +other than @samp{0} to indicate which port on the remote machine +they want to connect to. Application level servers must not fill this field with +a @samp{0}. Instead they specify a local port for clients to connect to. +It is possible to use a name from @file{/etc/services} here. +@end table + +Experts in network programming will notice that the usual +client/server asymmetry found at the level of the socket API is not visible +here. This is for the sake of simplicity of the high-level concept. If this +asymmetry is necessary for your application, +use another language. +For @command{gawk}, it is +more important to enable users to write a client program with a minimum +of code. What happens when first accessing a network connection is seen +in the following pseudo-code: + +@smallexample +if ((name of remote host given) && (other side accepts connection)) @{ + rendez-vous successful; transmit with getline or print +@} else @{ + if ((other side did not accept) && (localport == 0)) + exit unsuccessful + if (TCP) @{ + set up a server accepting connections + this means waiting for the client on the other side to connect + @} else + ready +@} +@end smallexample + +The exact behavior of this algorithm depends on the values of the +fields of the special @value{FN}. When in doubt, the following table +gives you the combinations of values and their meaning. If this +table is too complicated, focus on the three lines printed in +@strong{bold}. All the examples in +@ref{Using Networking, ,Networking With @command{gawk}}, +use only the +patterns printed in bold letters. + +@multitable {12345678901234} {123456} {123456} {1234567} {1234567890123456789012345} +@item @sc{protocol} @tab @sc{local port} @tab @sc{host name} +@tab @sc{remote port} @tab @sc{Resulting connection level behavior} +@item @strong{tcp} @tab @strong{0} @tab @strong{x} @tab @strong{x} @tab + @strong{Dedicated client, fails if immediately connecting to a + server on the other side fails} +@item udp @tab 0 @tab x @tab x @tab Dedicated client +@item raw @tab 0 @tab x @tab 0 @tab Dedicated client, works only as @code{root} +@item @strong{tcp, udp} @tab @strong{x} @tab @strong{x} @tab @strong{x} @tab + @strong{Client, switches to dedicated server if necessary} +@item @strong{tcp, udp} @tab @strong{x} @tab @strong{0} @tab @strong{0} @tab + @strong{Dedicated server} +@item raw @tab 0 @tab 0 @tab 0 @tab Dedicated server, works only as @code{root} +@item tcp, udp, raw @tab x @tab x @tab 0 @tab Invalid +@item tcp, udp, raw @tab 0 @tab 0 @tab x @tab Invalid +@item tcp, udp, raw @tab x @tab 0 @tab x @tab Invalid +@item tcp, udp @tab 0 @tab 0 @tab 0 @tab Invalid +@item tcp, udp @tab 0 @tab x @tab 0 @tab Invalid +@item raw @tab x @tab 0 @tab 0 @tab Invalid +@item raw @tab 0 @tab x @tab x @tab Invalid +@item raw @tab x @tab x @tab x @tab Invalid +@end multitable + +In general, TCP is the preferred mechanism to use. It is the simplest +protocol to understand and to use. Use the others only if circumstances +demand low-overhead. + +@node Comparing Protocols, , Special File Fields, Gawk Special Files +@subsection Comparing Protocols + +This @value{SECTION} develops a pair of programs (sender and receiver) +that do nothing but send a timestamp from one machine to another. The +sender and the receiver are implemented with each of the three protocols +available and demonstrate the differences between them. + +@menu +* File /inet/tcp:: The TCP special file. +* File /inet/udp:: The UDB special file. +* File /inet/raw:: The RAW special file. +@end menu + +@node File /inet/tcp, File /inet/udp, Comparing Protocols, Comparing Protocols +@subsubsection @file{/inet/tcp} +@cindex @file{/inet/tcp} special files +@cindex TCP +Once again, always use TCP. +(Use UDP when low-overhead is a necessity, and use RAW for +network experimentation.) +The first example is the sender +program: + +@example +# Server +BEGIN @{ + print strftime() |& "/inet/tcp/8888/0/0" + close("/inet/tcp/8888/0/0") +@} +@end example + +The receiver is very simple: + +@example +# Client +BEGIN @{ + "/inet/tcp/0/localhost/8888" |& getline + print $0 + close("/inet/tcp/0/localhost/8888") +@} +@end example + +TCP guarantees that the bytes arrive at the receiving end in exactly +the same order that they were sent. No byte is lost +(except for broken connections), doubled, or out of order. Some +overhead is necessary to accomplish this, but this is the price to pay for +a reliable service. +It does matter which side starts first. The sender/server has to be started +first, and it waits for the receiver to read a line. + +@node File /inet/udp, File /inet/raw, File /inet/tcp, Comparing Protocols +@subsubsection @file{/inet/udp} +@cindex @file{/inet/udp} special files +@cindex UDP +The server and client programs that use UDP are almost identical to their TCP counterparts; +only the @var{protocol} has changed. As before, it does matter which side +starts first. The receiving side blocks and waits for the sender. +In this case, the receiver/client has to be started first: + +@page +@example +# Server +BEGIN @{ + print strftime() |& "/inet/udp/8888/0/0" + close("/inet/udp/8888/0/0") +@} +@end example + +The receiver is almost identical to the TCP receiver: + +@example +# Client +BEGIN @{ + "/inet/udp/0/localhost/8888" |& getline + print $0 + close("/inet/udp/0/localhost/8888") +@} +@end example + +UDP cannot guarantee that the datagrams at the receiving end will arrive in exactly +the same order they were sent. Some datagrams could be +lost, some doubled, and some out of order. But no overhead is necessary to +accomplish this. This unreliable behavior is good enough for tasks +such as data acquisition, logging, and even stateless services like NFS. + +@node File /inet/raw, , File /inet/udp, Comparing Protocols +@subsubsection @file{/inet/raw} +@cindex @file{/inet/raw} special files +@cindex RAW + +This is an IP-level protocol. Only @code{root} is allowed to access this +special file. It is meant to be the basis for implementing +and experimenting with transport level protocols.@footnote{This special file +is reserved, but not otherwise currently implemented.} +In the most general case, +the sender has to supply the encapsulating header bytes in front of the +packet and the receiver has to strip the additional bytes from the message. + +@cindex dark corner +RAW receivers cannot receive packets sent with TCP or UDP because the +operating system does not deliver the packets to a RAW receiver. The +operating system knows about some of the protocols on top of IP +and decides on its own which packet to deliver to which process. +@value{DARKCORNER} +Therefore, the UDP receiver must be used for receiving UDP +datagrams sent with the RAW sender. This is a dark corner, not only of +@command{gawk}, but also of TCP/IP. + +@cindex SPAK utility +For extended experimentation with protocols, look into +the approach implemented in a tool called SPAK. +This tool reflects the hierarchical layering of protocols (encapsulation) +in the way data streams are piped out of one program into the next one. +It shows which protocol is based on which other (lower-level) protocol +by looking at the command-line ordering of the program calls. +Cleverly thought out, SPAK is much better than @command{gawk}'s +@file{/inet} for learning the meaning of each and every bit in the +protocol headers. + +The next example uses the RAW protocol to emulate +the behavior of UDP. The sender program is the same as above, but with some +additional bytes that fill the places of the UDP fields: + +@example +@group +BEGIN @{ + Message = "Hello world\n" + SourcePort = 0 + DestinationPort = 8888 + MessageLength = length(Message)+8 + RawService = "/inet/raw/0/localhost/0" + printf("%c%c%c%c%c%c%c%c%s", + SourcePort/256, SourcePort%256, + DestinationPort/256, DestinationPort%256, + MessageLength/256, MessageLength%256, + 0, 0, Message) |& RawService + fflush(RawService) + close(RawService) +@} +@end group +@end example + +Since this program tries +to emulate the behavior of UDP, it checks if +the RAW sender is understood by the UDP receiver but not if the RAW receiver +can understand the UDP sender. In a real network, the +RAW receiver is hardly +of any use because it gets every IP packet that +comes across the network. There are usually so many packets that +@command{gawk} would be too slow for processing them. +Only on a network with little +traffic can the IP-level receiver program be tested. Programs for analyzing +IP traffic on modem or ISDN channels should be possible. + +Port numbers do not have a meaning when using @file{/inet/raw}. Their fields +have to be @samp{0}. Only TCP and UDP use ports. Receiving data from +@file{/inet/raw} is difficult, not only because of processing speed but also +because data is usually binary and not restricted to ASCII. This +implies that line separation with @code{RS} does not work as usual. + +@node TCP Connecting, Troubleshooting, Gawk Special Files, Using Networking +@section Establishing a TCP Connection + +Let's observe a network connection at work. Type in the following program +and watch the output. Within a second, it connects via TCP (@file{/inet/tcp}) +to the machine it is running on (@samp{localhost}), and asks the service +@samp{daytime} on the machine what time it is: + +@cindex @code{|&} I/O operator +@cindex @code{getline} built-in function +@example +BEGIN @{ + "/inet/tcp/0/localhost/daytime" |& getline + print $0 + close("/inet/tcp/0/localhost/daytime") +@} +@end example + +Even experienced @command{awk} users will find the second line strange in two +respects: + +@itemize @bullet +@item +A special file is used as a shell command that pipes its output +into @code{getline}. One would rather expect to see the special file +being read like any other file (@samp{getline < +"/inet/tcp/0/localhost/daytime")}. + +@item +The operator @samp{|&} has not been part of any @command{awk} +implementation (until now). +It is actually the only extension of the @command{awk} +language needed (apart from the special files) to introduce network access. +@end itemize + +The @samp{|&} operator was introduced in @command{gawk} 3.1 in order to +overcome the crucial restriction that access to files and pipes in +@command{awk} is always unidirectional. It was formerly impossible to use +both access modes on the same file or pipe. Instead of changing the whole +concept of file access, the @samp{|&} operator +behaves exactly like the usual pipe operator except for two additions: + +@itemize @bullet +@item +Normal shell commands connected to their @command{gawk} program with a @samp{|&} +pipe can be accessed bidirectionally. The @samp{|&} turns out to be a quite +general, useful, and natural extension of @command{awk}. + +@item +Pipes that consist of a special @value{FN} for network connections are not +executed as shell commands. Instead, they can be read and written to, just +like a full-duplex network connection. +@end itemize + +In the earlier example, the @samp{|&} operator tells @code{getline} +to read a line from the special file @file{/inet/tcp/0/localhost/daytime}. +We could also have printed a line into the special file. But instead we just +read a line with the time, printed it, and closed the connection. +(While we could just let @command{gawk} close the connection by finishing +the program, in this @value{DOCUMENT} +we are pedantic, and always explicitly close the connections.) + +@node Troubleshooting, Interacting, TCP Connecting, Using Networking +@section Troubleshooting Connection Problems +It may well be that for some reason the above program does not run on your +machine. When looking at possible reasons for this, you will learn much +about typical problems that arise in network programming. First of all, +your implementation of @command{gawk} may not support network access +because it is +a pre-3.1 version or you do not have a network interface in your machine. +Perhaps your machine uses some other protocol +like DECnet or Novell's IPX. For the rest of this @value{CHAPTER}, +we will assume +you work on a Unix machine that supports TCP/IP. If the above program does +not run on such a machine, it may help to replace the name +@samp{localhost} with the name of your machine or its IP address. If it +does, you could replace @samp{localhost} with the name of another machine +in your vicinity. This way, the program connects to another machine. +Now you should see the date and time being printed by the program. +Otherwise your machine may not support the @samp{daytime} service. +Try changing the service to @samp{chargen} or @samp{ftp}. This way, the program +connects to other services that should give you some response. If you are +curious, you should have a look at your file @file{/etc/services}. It could +look like this: + +@ignore +@multitable {1234567890123} {1234567890123} {123456789012345678901234567890123456789012} +@item Service @strong{name} @tab Service @strong{number} +@item echo @tab 7/tcp @tab echo sends back each line it receivces +@item echo @tab 7/udp @tab echo is good for testing purposes +@item discard @tab 9/tcp @tab discard behaves like @file{/dev/null} +@item discard @tab 9/udp @tab discard just throws away each line +@item daytime @tab 13/tcp @tab daytime sends date & time once per connection +@item daytime @tab 13/udp +@item chargen @tab 19/tcp @tab chargen infinitely produces character sets +@item chargen @tab 19/udp @tab chargen is good for testing purposes +@item ftp @tab 21/tcp @tab ftp is the usual file transfer protocol +@item telnet @tab 23/tcp @tab telnet is the usual login facility +@item smtp @tab 25/tcp @tab smtp is the Simple Mail Transfer Protocol +@item finger @tab 79/tcp @tab finger tells you who is logged in +@item www @tab 80/tcp @tab www is the HyperText Transfer Protocol +@item pop2 @tab 109/tcp @tab pop2 is an older version of pop3 +@item pop2 @tab 109/udp +@item pop3 @tab 110/tcp @tab pop3 is the Post Office Protocol +@item pop3 @tab 110/udp @tab pop3 is used for receiving email +@item nntp @tab 119/tcp @tab nntp is the USENET News Transfer Protocol +@item irc @tab 194/tcp @tab irc is the Internet Relay Chat +@item irc @tab 194/udp +@end multitable +@end ignore + +@smallexample +# /etc/services: +# +# Network services, Internet style +# +# Name Number/Protcol Alternate name # Comments + +echo 7/tcp +echo 7/udp +discard 9/tcp sink null +discard 9/udp sink null +daytime 13/tcp +daytime 13/udp +chargen 19/tcp ttytst source +chargen 19/udp ttytst source +ftp 21/tcp +telnet 23/tcp +smtp 25/tcp mail +finger 79/tcp +www 80/tcp http # WorldWideWeb HTTP +www 80/udp # HyperText Transfer Protocol +pop-2 109/tcp postoffice # POP version 2 +pop-2 109/udp +pop-3 110/tcp # POP version 3 +pop-3 110/udp +nntp 119/tcp readnews untp # USENET News +irc 194/tcp # Internet Relay Chat +irc 194/udp +@dots{} +@end smallexample + +@cindex Linux +@cindex GNU/Linux +@cindex Microsoft Windows +Here, you find a list of services that traditional Unix machines usually +support. If your GNU/Linux machine does not do so, it may be that these +services are switched off in some startup script. Systems running some +flavor of Microsoft Windows usually do @emph{not} support such services. +Nevertheless, it @emph{is} possible to do networking with @command{gawk} on +Microsoft +Windows.@footnote{Microsoft prefered to ignore the TCP/IP +family of protocols until 1995. Then came the rise of the Netscape browser +as a landmark ``killer application.'' Microsoft added TCP/IP support and +their own browser to Microsoft Windows 95 at the last minute. They even back-ported +their TCP/IP implementation to Microsoft Windows for Workgroups 3.11, but it was +a rather rudimentary and half-hearted implementation. Nevertheless, +the equivalent of @file{/etc/services} resides under +@file{c:\windows\services} on Microsoft Windows.} +The first column of the file gives the name of the service, +the second a unique number, and the protocol that one can use to connect to +this service. +The rest of the line is treated as a comment. +You see that some services (@samp{echo}) support TCP as +well as UDP. + +@node Interacting, Setting Up, Troubleshooting, Using Networking +@section Interacting with a Network Service + +The next program makes use of the possibility to really interact with a +network service by printing something into the special file. It asks the +so-called @command{finger} service if a user of the machine is logged in. When +testing this program, try to change @samp{localhost} to +some other machine name in your local network: + +@c system if test ! -d eg ; then mkdir eg ; fi +@c system if test ! -d eg/network ; then mkdir eg/network ; fi +@example +@c file eg/network/fingerclient.awk +BEGIN @{ + NetService = "/inet/tcp/0/localhost/finger" + print "@var{name}" |& NetService + while ((NetService |& getline) > 0) + print $0 + close(NetService) +@} +@c endfile +@end example + +After telling the service on the machine which user to look for, +the program repeatedly reads lines that come as a reply. When no more +lines are coming (because the service has closed the connection), the +program also closes the connection. Try replacing @code{"@var{name}"} with your +login name (or the name of someone else logged in). For a list +of all users currently logged in, replace @var{name} with an empty string +@code{""}. + +@cindex Linux +@cindex GNU/Linux +The final @code{close} command could be safely deleted from +the above script, because the operating system closes any open connection +by default when a script reaches the end of execution. In order to avoid +portability problems, it is best to always close connections explicitly. +With the Linux kernel, +for example, proper closing results in flushing of buffers. Letting +the close happen by default may result in discarding buffers. + +@ignore +@c Chuck comments that this seems out of place. He's right. I dunno +@c where to put it though. +@cindex @command{finger} utility +@cindex RFC 1288 +In the early days of the Internet (up until about 1992), you could use +such a program to check if some user in another country was logged in on +a specific machine. +RFC 1288@footnote{@uref{http://www.cis.ohio-state.edu/htbin/rfc/rfc1288.html}} +provides the exact definition of the @command{finger} protocol. +Every contemporary Unix system also has a command named @command{finger}, +which functions as a client for the protocol of the same name. +Still today, some people maintain simple information systems +with this ancient protocol. For example, by typing +@samp{finger quake@@seismo.unr.edu} +you get the latest @dfn{Earthquake Bulletin} for the state of Nevada. + +@cindex Earthquake Bulletin +@smallexample +$ finger quake@@seismo.unr.edu + +[@dots{}] + +DATE-(UTC)-TIME LAT LON DEP MAG COMMENTS +yy/mm/dd hh:mm:ss deg. deg. km + +98/12/14 21:09:22 37.47N 116.30W 0.0 2.3Md 76.4 km S of WARM SPRINGS, NEVA +98/12/14 22:05:09 39.69N 120.41W 11.9 2.1Md 53.8 km WNW of RENO, NEVADA +98/12/15 14:14:19 38.04N 118.60W 2.0 2.3Md 51.0 km S of HAWTHORNE, NEVADA +98/12/17 01:49:02 36.06N 117.58W 13.9 3.0Md 74.9 km SE of LONE PINE, CALIFOR +98/12/17 05:39:26 39.95N 120.87W 6.2 2.6Md 101.6 km WNW of RENO, NEVADA +98/12/22 06:07:42 38.68N 119.82W 5.2 2.3Md 50.7 km S of CARSON CITY, NEVAD +@end smallexample + +@noindent +This output from @command{finger} contains the time, location, depth, +magnitude, and a short comment about +the earthquakes registered in that region during the last 10 days. +In many places today the use of such services is restricted +because most networks have firewalls and proxy servers between them +and the Internet. Most firewalls are programmed to not let +@command{finger} requests go beyond the local network. + +@cindex Coke machine +Another (ab)use of the @command{finger} protocol are several Coke machines +that are connected to the Internet. There is a short list of such +Coke machines.@footnote{@uref{http://ca.yahoo.com/Computers_and_Internet/Internet/Devices_Connected_to_the_Internet/Soda_Machines/}} +You can access them either from the command-line or with a simple +@command{gawk} script. They usually tell you about the different +flavors of Coke and beer available there. If you have an account there, +you can even order some drink this way. +@end ignore + +When looking at @file{/etc/services} you may have noticed that the +@samp{daytime} service is also available with @samp{udp}. In the earlier +example, change @samp{tcp} to @samp{udp}, +and change @samp{finger} to @samp{daytime}. +After starting the modified program, you see the expected day and time message. +The program then hangs, because it waits for more lines coming from the +service. However, they never come. This behavior is a consequence of the +differences between TCP and UDP. When using UDP, neither party is +automatically informed about the other closing the connection. +Continuing to experiment this way reveals many other subtle +differences between TCP and UDP. To avoid such trouble, one should always +remember the advice Douglas E.@: Comer and David Stevens give in +Volume III of their series @cite{Internetworking With TCP} +(page 14): + +@cindex TCP +@cindex UDP +@quotation +When designing client-server applications, beginners are strongly +advised to use TCP because it provides reliable, connection-oriented +communication. Programs only use UDP if the application protocol handles +reliability, the application requires hardware broadcast or multicast, +or the application cannot tolerate virtual circuit overhead. +@end quotation + +@node Setting Up, Email, Interacting, Using Networking +@section Setting Up a Service +The preceding programs behaved as clients that connect to a server somewhere +on the Internet and request a particular service. Now we set up such a +service to mimic the behavior of the @samp{daytime} service. +Such a server does not know in advance who is going to connect to it over +the network. Therefore we cannot insert a name for the host to connect to +in our special @value{FN}. + +Start the following program in one window. Notice that the service does +not have the name @samp{daytime}, but the number @samp{8888}. +From looking at @file{/etc/services}, you know that names like @samp{daytime} +are just mnemonics for predetermined 16-bit integers. +Only the system administrator (@code{root}) could enter +our new service into @file{/etc/services} with an appropriate name. +Also notice that the service name has to be entered into a different field +of the special @value{FN} because we are setting up a server, not a client: + +@cindex @command{finger} utility +@cindex server +@example +BEGIN @{ + print strftime() |& "/inet/tcp/8888/0/0" + close("/inet/tcp/8888/0/0") +@} +@end example + +Now open another window on the same machine. +Copy the client program given as the first example +(@pxref{TCP Connecting, ,Establishing a TCP Connection}) +to a new file and edit it, changing the name @samp{daytime} to +@samp{8888}. Then start the modified client. You should get a reply +like this: + +@example +Sat Sep 27 19:08:16 CEST 1997 +@end example + +@noindent +Both programs explicitly close the connection. + +@cindex Microsoft Windows +@cindex reserved ports +Now we will intentionally make a mistake to see what happens when the name +@samp{8888} (the so-called port) is already used by another service. +Start the server +program in both windows. The first one works, but the second one +complains that it could not open the connection. Each port on a single +machine can only be used by one server program at a time. Now terminate the +server program and change the name @samp{8888} to @samp{echo}. After restarting it, +the server program does not run any more and you know why: there already is +an @samp{echo} service running on your machine. But even if this isn't true, +you would not get +your own @samp{echo} server running on a Unix machine, +because the ports with numbers smaller +than 1024 (@samp{echo} is at port 7) are reserved for @code{root}. +On machines running some flavor of Microsoft Windows, there is no restriction +that reserves ports 1 to 1024 for a privileged user; hence you can start +an @samp{echo} server there. + +Turning this short server program into something really useful is simple. +Imagine a server that first reads a @value{FN} from the client through the +network connection, then does something with the file and +sends a result back to the client. The server-side processing +could be: + +@example +BEGIN @{ + NetService = "/inet/tcp/8888/0/0" + NetService |& getline + CatPipe = ("cat " $1) # sets $0 and the fields + while ((CatPipe | getline) > 0) + print $0 |& NetService + close(NetService) +@} +@end example + +@noindent +and we would +have a remote copying facility. Such a server reads the name of a file +from any client that connects to it and transmits the contents of the +named file across the net. The server-side processing could also be +the execution of a command that is transmitted across the network. From this +example, you can see how simple it is to open up a security hole on your +machine. If you allow clients to connect to your machine and +execute arbitrary commands, anyone would be free to do @samp{rm -rf *}. + +@node Email, Web page, Setting Up, Using Networking +@section Reading Email +@cindex POP +@cindex SMTP +@cindex RFC 1939 +@cindex RFC 821 +The distribution of email is usually done by dedicated email servers that +communicate with your machine using special protocols. To receive email, we +will use the Post Office Protocol (POP). Sending can be done with the much +older Simple Mail Transfer Protocol (SMTP). +@ignore +@footnote{RFC 1939 defines POP. +RFC 821 defines SMTP. See +@uref{http://rfc.fh-koeln.de/doc/rfc/html/rfc.html, RFCs in HTML}.} +@end ignore + +When you type in the following program, replace the @var{emailhost} by the +name of your local email server. Ask your administrator if the server has a +POP service, and then use its name or number in the program below. +Now the program is ready to connect to your email server, but it will not +succeed in retrieving your mail because it does not yet know your login +name or password. Replace them in the program and it +shows you the first email the server has in store: + +@example +BEGIN @{ + POPService = "/inet/tcp/0/@var{emailhost}/pop3" + RS = ORS = "\r\n" + print "user @var{name}" |& POPService + POPService |& getline + print "pass @var{password}" |& POPService + POPService |& getline + print "retr 1" |& POPService + POPService |& getline + if ($1 != "+OK") exit + print "quit" |& POPService + RS = "\r\n\\.\r\n" + POPService |& getline + print $0 + close(POPService) +@} +@end example + +@cindex RFC 1939 +The record separators @code{RS} and @code{ORS} are redefined because the +protocol (POP) requires CR-LF to separate lines. After identifying +yourself to the email service, the command @samp{retr 1} instructs the +service to send the first of all your email messages in line. If the service +replies with something other than @samp{+OK}, the program exits; maybe there +is no email. Otherwise, the program first announces that it intends to finish +reading email, and then redefines @code{RS} in order to read the entire +email as multiline input in one record. From the POP RFC, we know that the body +of the email always ends with a single line containing a single dot. +The program looks for this using @samp{RS = "\r\n\\.\r\n"}. +When it finds this sequence in the mail message, it quits. +You can invoke this program as often as you like; it does not delete the +message it reads, but instead leaves it on the server. + +@node Web page, Primitive Service, Email, Using Networking +@section Reading a Web Page +@cindex HTTP +@cindex RFC 2068 +@cindex RFC 2616 + +Retrieving a web page from a web server is as simple as +retrieving email from an email server. We only have to use a +similar, but not identical, protocol and a different port. The name of the +protocol is HyperText Transfer Protocol (HTTP) and the port number is usually +80. As in the preceding @value{SECTION}, ask your administrator about the +name of your local web server or proxy web server and its port number +for HTTP requests. + +@ignore +@c Chuck says this stuff isn't necessary +More detailed information about HTTP can be found at +the home of the web protocols,@footnote{@uref{http://www.w3.org/pub/WWW/Protocols}} +including the specification of HTTP in RFC 2068. The protocol specification +in RFC 2068 is concise and you can get it for free. If you need more +explanation and you are willing to pay for a book, you might be +interested in one of these books: + +@enumerate + +@item +When we started writing web clients and servers with @command{gawk}, +the only book available with details about HTTP was the one by Paul Hethmon +called +@cite{Illustrated Guide to HTTP}.@footnote{@uref{http://www.browsebooks.com/Hethmon/?882}} +Hethmon not only describes HTTP, +he also implements a simple web server in C++. + +@item +Since July 2000, O'Reilly offers the book by Clinton Wong called +@cite{HTTP Pocket Reference}.@footnote{@uref{http://www.oreilly.com/catalog/httppr}} +It only has 75 pages but its +focus definitely is HTTP. This pocket reference is not a replacement +for the RFC, but I wish I had had it back in 1997 when I started writing +scripts to handle HTTP. + +@item +Another small booklet about HTTP is the one by Toexcell Incorporated Staff, +ISBN 1-58348-270-9, called +@cite{Hypertext Transfer Protocol Http 1.0 Specifications} + +@end enumerate +@end ignore + +The following program employs a rather crude approach toward retrieving a +web page. It uses the prehistoric syntax of HTTP 0.9, which almost all +web servers still support. The most noticeable thing about it is that the +program directs the request to the local proxy server whose name you insert +in the special @value{FN} (which in turn calls @samp{www.yahoo.com}): + +@example +BEGIN @{ + RS = ORS = "\r\n" + HttpService = "/inet/tcp/0/@var{proxy}/80" + print "GET http://www.yahoo.com" |& HttpService + while ((HttpService |& getline) > 0) + print $0 + close(HttpService) +@} +@end example + +@cindex RFC 1945 +@cindex HTML +@cindex Yahoo! +Again, lines are separated by a redefined @code{RS} and @code{ORS}. +The @code{GET} request that we send to the server is the only kind of +HTTP request that existed when the web was created in the early 1990s. +HTTP calls this @code{GET} request a ``method,'' which tells the +service to transmit a web page (here the home page of the Yahoo! search +engine). Version 1.0 added the request methods @code{HEAD} and +@code{POST}. The current version of HTTP is 1.1,@footnote{Version 1.0 of +HTTP was defined in RFC 1945. HTTP 1.1 was initially specified in RFC +2068. In June 1999, RFC 2068 was made obsolete by RFC 2616. It is an update +without any substantial changes.} and knows the additional request +methods @code{OPTIONS}, @code{PUT}, @code{DELETE}, and @code{TRACE}. +You can fill in any valid web address, and the program prints the +HTML code of that page to your screen. + +Notice the similarity between the responses of the POP and HTTP +services. First, you get a header that is terminated by an empty line, and +then you get the body of the page in HTML. The lines of the headers also +have the same form as in POP. There is the name of a parameter, +then a colon, and finally the value of that parameter. + +@cindex CGI +@cindex @file{gif} image format +@cindex @file{png} image format +Images (@file{.png} or @file{.gif} files) can also be retrieved this way, +but then you +get binary data that should be redirected into a file. Another +application is calling a CGI (Common Gateway Interface) script on some +server. CGI scripts are used when the contents of a web page are not +constant, but generated instantly at the moment you send a request +for the page. For example, to get a detailed report about the current +quotes of Motorola stock shares, call a CGI script at Yahoo! with +the following: + +@example +get = "GET http://quote.yahoo.com/q?s=MOT&d=t" +print get |& HttpService +@end example + +You can also request weather reports this way. +@ignore +@cindex Boutell, Thomas +A good book to go on with is +the +@cite{HTML Source Book}.@footnote{@uref{http://www.utoronto.ca/webdocs/HTMLdocs/NewHTML/book.html}} +There are also some books on CGI programming +like @cite{CGI Programming in C & Perl}, +by Thomas Boutell@footnote{@uref{http://cseng.aw.com/bookdetail.qry?ISBN=0-201-42219-0&ptype=0}}, +and @cite{The CGI Book}.@footnote{@uref{http://www.cgibook.com}} +Another good source is @cite{The CGI Resource Index}}.@footnote{@uref{http://www.cgi-resources.com}} +@end ignore + +@node Primitive Service, Interacting Service, Web page, Using Networking +@section A Primitive Web Service +Now we know enough about HTTP to set up a primitive web service that just +says @code{"Hello, world"} when someone connects to it with a browser. +Compared +to the situation in the preceding @value{SECTION}, our program changes the role. It +tries to behave just like the server we have observed. Since we are setting +up a server here, we have to insert the port number in the @samp{localport} +field of the special @value{FN}. The other two fields (@var{hostname} and +@var{remoteport}) have to contain a @samp{0} because we do not know in +advance which host will connect to our service. + +In the early 1990s, all a server had to do was send an HTML document and +close the connection. Here, we adhere to the modern syntax of HTTP. +The steps are as follows: + +@enumerate 1 +@item +Send a status line telling the web browser that everything +is OK. + +@item +Send a line to tell the browser how many bytes follow in the +body of the message. This was not necessary earlier because both +parties knew that the document ended when the connection closed. Nowadays +it is possible to stay connected after the transmission of one web page. +This is to avoid the network traffic necessary for repeatedly establishing +TCP connections for requesting several images. Thus, there is the need to tell +the receiving party how many bytes will be sent. The header is terminated +as usual with an empty line. + +@item +Send the @code{"Hello, world"} body +in HTML. +The useless @code{while} loop swallows the request of the browser. +We could actually omit the loop, and on most machines the program would still +work. +First, start the following program: +@end enumerate + +@example +@c file eg/network/hello-serv.awk +BEGIN @{ + RS = ORS = "\r\n" + HttpService = "/inet/tcp/8080/0/0" + Hello = "<HTML><HEAD>" \ + "<TITLE>A Famous Greeting</TITLE></HEAD>" \ + "<BODY><H1>Hello, world</H1></BODY></HTML>" + Len = length(Hello) + length(ORS) + print "HTTP/1.0 200 OK" |& HttpService + print "Content-Length: " Len ORS |& HttpService + print Hello |& HttpService + while ((HttpService |& getline) > 0) + continue; + close(HttpService) +@} +@c endfile +@end example + +Now, on the same machine, start your favorite browser and let it point to +@uref{http://localhost:8080} (the browser needs to know on which port +our server is listening for requests). If this does not work, the browser +probably tries to connect to a proxy server that does not know your machine. +If so, change the browser's configuration so that the browser does not try to +use a proxy to connect to your machine. + +@node Interacting Service, Simple Server, Primitive Service, Using Networking +@section A Web Service with Interaction +@cindex GUI +@ifinfo +This node shows how to set up a simple web server. +The subnode is a library file that we will use with all the examples in +@ref{Some Applications and Techniques}. +@end ifinfo + +@menu +* CGI Lib:: A simple CGI library. +@end menu + +Setting up a web service that allows user interaction is more difficult and +shows us the limits of network access in @command{gawk}. In this @value{SECTION}, +we develop a main program (a @code{BEGIN} pattern and its action) +that will become the core of event-driven execution controlled by a +graphical user interface (GUI). +Each HTTP event that the user triggers by some action within the browser +is received in this central procedure. Parameters and menu choices are +extracted from this request and an appropriate measure is taken according to +the user's choice. +For example: + +@cindex HTTP server, core logic +@example +BEGIN @{ + if (MyHost == "") @{ + "uname -n" | getline MyHost + close("uname -n") + @} + if (MyPort == 0) MyPort = 8080 + HttpService = "/inet/tcp/" MyPort "/0/0" + MyPrefix = "http://" MyHost ":" MyPort + SetUpServer() + while ("awk" != "complex") @{ + # header lines are terminated this way + RS = ORS = "\r\n" + Status = 200 # this means OK + Reason = "OK" + Header = TopHeader + Document = TopDoc + Footer = TopFooter + if (GETARG["Method"] == "GET") @{ + HandleGET() + @} else if (GETARG["Method"] == "HEAD") @{ + # not yet implemented + @} else if (GETARG["Method"] != "") @{ + print "bad method", GETARG["Method"] + @} + Prompt = Header Document Footer + print "HTTP/1.0", Status, Reason |& HttpService + print "Connection: Close" |& HttpService + print "Pragma: no-cache" |& HttpService + len = length(Prompt) + length(ORS) + print "Content-length:", len |& HttpService + print ORS Prompt |& HttpService + # ignore all the header lines + while ((HttpService |& getline) > 0) + ; + # stop talking to this client + close(HttpService) + # wait for new client request + HttpService |& getline + # do some logging + print systime(), strftime(), $0 + # read request parameters + CGI_setup($1, $2, $3) + @} +@} +@end example + +This web server presents menu choices in the form of HTML links. +Therefore, it has to tell the browser the name of the host it is +residing on. When starting the server, the user may supply the name +of the host from the command line with @samp{gawk -v MyHost="Rumpelstilzchen"}. +If the user does not do this, the server looks up the name of the host it is +running on for later use as a web address in HTML documents. The same +applies to the port number. These values are inserted later into the +HTML content of the web pages to refer to the home system. + +Each server that is built around this core has to initialize some +application-dependent variables (such as the default home page) in a procedure +@code{SetUpServer}, which is called immediately before entering the +infinite loop of the server. For now, we will write an instance that +initiates a trivial interaction. With this home page, the client user +can click on two possible choices, and receive the current date either +in human-readable format or in seconds since 1970: + +@example +function SetUpServer() @{ + TopHeader = "<HTML><HEAD>" + TopHeader = TopHeader \ + "<title>My name is GAWK, GNU AWK</title></HEAD>" + TopDoc = "<BODY><h2>\ + Do you prefer your date <A HREF=" MyPrefix \ + "/human>human</A> or \ + <A HREF=" MyPrefix "/POSIX>POSIXed</A>?</h2>" ORS ORS + TopFooter = "</BODY></HTML>" +@} +@end example + +On the first run through the main loop, the default line terminators are +set and the default home page is copied to the actual home page. Since this +is the first run, @code{GETARG["Method"]} is not initialized yet, hence the +case selection over the method does nothing. Now that the home page is +initialized, the server can start communicating to a client browser. + +@cindex RFC 2068 +@cindex CGI +It does so by printing the HTTP header into the network connection +(@samp{print @dots{} |& HttpService}). This command blocks execution of +the server script until a client connects. If this server +script is compared with the primitive one we wrote before, you will notice +two additional lines in the header. The first instructs the browser +to close the connection after each request. The second tells the +browser that it should never try to @emph{remember} earlier requests +that had identical web addresses (no caching). Otherwise, it could happen +that the browser retrieves the time of day in the previous example just once, +and later it takes the web page from the cache, always displaying the same +time of day although time advances each second. + +Having supplied the initial home page to the browser with a valid document +stored in the parameter @code{Prompt}, it closes the connection and waits +for the next request. When the request comes, a log line is printed that +allows us to see which request the server receives. The final step in the +loop is to call the function @code{CGI_setup}, which reads all the lines +of the request (coming from the browser), processes them, and stores the +transmitted parameters in the array @code{PARAM}. The complete +text of these application-independent functions can be found in +@ref{CGI Lib, ,A Simple CGI Library}. +For now, we use a simplified version of @code{CGI_setup}: + +@example +function CGI_setup( method, uri, version, i) @{ + delete GETARG; delete MENU; delete PARAM + GETARG["Method"] = $1 + GETARG["URI"] = $2 + GETARG["Version"] = $3 + i = index($2, "?") + # is there a "?" indicating a CGI request? +@group + if (i > 0) @{ + split(substr($2, 1, i-1), MENU, "[/:]") + split(substr($2, i+1), PARAM, "&") + for (i in PARAM) @{ + j = index(PARAM[i], "=") + GETARG[substr(PARAM[i], 1, j-1)] = \ + substr(PARAM[i], j+1) + @} + @} else @{ # there is no "?", no need for splitting PARAMs + split($2, MENU, "[/:]") + @} +@end group +@} +@end example + +At first, the function clears all variables used for +global storage of request parameters. The rest of the function serves +the purpose of filling the global parameters with the extracted new values. +To accomplish this, the name of the requested resource is split into +parts and stored for later evaluation. If the request contains a @samp{?}, +then the request has CGI variables seamlessly appended to the web address. +Everything in front of the @samp{?} is split up into menu items, and +everything behind the @samp{?} is a list of @samp{@var{variable}=@var{value}} pairs +(separated by @samp{&}) that also need splitting. This way, CGI variables are +isolated and stored. This procedure lacks recognition of special characters +that are transmitted in coded form@footnote{As defined in RFC 2068.}. Here, any +optional request header and body parts are ignored. We do not need +header parameters and the request body. However, when refining our approach or +working with the @code{POST} and @code{PUT} methods, reading the header +and body +becomes inevitable. Header parameters should then be stored in a global +array as well as the body. + +On each subsequent run through the main loop, one request from a browser is +received, evaluated, and answered according to the user's choice. This can be +done by letting the value of the HTTP method guide the main loop into +execution of the procedure @code{HandleGET}, which evaluates the user's +choice. In this case, we have only one hierarchical level of menus, +but in the general case, +menus are nested. +The menu choices at each level are +separated by @samp{/}, just as in @value{FN}s. Notice how simple it is to +construct menus of arbitrary depth: + +@example +function HandleGET() @{ + if ( MENU[2] == "human") @{ + Footer = strftime() TopFooter + @} else if (MENU[2] == "POSIX") @{ + Footer = systime() TopFooter + @} +@} +@end example + +@cindex CGI +The disadvantage of this approach is that our server is slow and can +handle only one request at a time. Its main advantage, however, is that +the server +consists of just one @command{gawk} program. No need for installing an +@command{httpd}, and no need for static separate HTML files, CGI scripts, or +@code{root} privileges. This is rapid prototyping. +This program can be started on the same host that runs your browser. +Then let your browser point to @uref{http://localhost:8080}. + +@cindex @file{xbm} image format +@cindex image format +@cindex GNUPlot utility +It is also possible to include images into the HTML pages. +Most browsers support the not very well-known +@file{.xbm} format, +which may contain only +monochrome pictures but is an ASCII format. Binary images are possible but +not so easy to handle. Another way of including images is to generate them +with a tool such as GNUPlot, +by calling the tool with the @code{system} function or through a pipe. + +@node CGI Lib, , Interacting Service, Interacting Service +@subsection A Simple CGI Library +@quotation +@i{HTTP is like being married: you have to be able to handle whatever +you're given, while being very careful what you send back.}@* +Phil Smith III,@* +@uref{http://www.netfunny.com/rhf/jokes/99/Mar/http.html} +@end quotation + +In @ref{Interacting Service, ,A Web Service with Interaction}, +we saw the function @code{CGI_setup} as part of the web server +``core logic'' framework. The code presented there handles almost +everything necessary for CGI requests. +One thing it doesn't do is handle encoded characters in the requests. +For example, an @samp{&} is encoded as a percent sign followed by +the hexadecimal value---@samp{%26}. These encoded values should be +decoded. +Following is a simple library to perform these tasks. +This code is used for all web server examples +used throughout the rest of this @value{DOCUMENT}. +If you want to use it for your own web server, store the source code +into a file named @file{inetlib.awk}. Then you can include +these functions into your code by placing the following statement +into your program: + +@example +@@include inetlib.awk +@end example + +@noindent +on the first line of your script. But beware, this mechanism is +only possible if you invoke your web server script with @command{igawk} +instead of the usual @command{awk} or @command{gawk}. +Here is the code: + +@example +@c file eg/network/coreserv.awk +# CGI Library and core of a web server +@c endfile +@ignore +@c file eg/network/coreserv.awk +# +# Juergen Kahrs, Juergen.Kahrs@@vr-web.de +# with Arnold Robbins, arnold@@gnu.org +# September 2000 + +@c endfile +@end ignore +@c file eg/network/coreserv.awk +# Global arrays +# GETARG --- arguments to CGI GET command +# MENU --- menu items (path names) +# PARAM --- parameters of form x=y + +# Optional variable MyHost contains host address +# Optional variable MyPort contains port number +# Needs TopHeader, TopDoc, TopFooter +# Sets MyPrefix, HttpService, Status, Reason + +BEGIN @{ + if (MyHost == "") @{ + "uname -n" | getline MyHost + close("uname -n") + @} + if (MyPort == 0) MyPort = 8080 + HttpService = "/inet/tcp/" MyPort "/0/0" + MyPrefix = "http://" MyHost ":" MyPort + SetUpServer() + while ("awk" != "complex") @{ + # header lines are terminated this way + RS = ORS = "\r\n" + Status = 200 # this means OK + Reason = "OK" + Header = TopHeader + Document = TopDoc + Footer = TopFooter + if (GETARG["Method"] == "GET") @{ + HandleGET() + @} else if (GETARG["Method"] == "HEAD") @{ + # not yet implemented + @} else if (GETARG["Method"] != "") @{ + print "bad method", GETARG["Method"] + @} + Prompt = Header Document Footer + print "HTTP/1.0", Status, Reason |& HttpService + print "Connection: Close" |& HttpService + print "Pragma: no-cache" |& HttpService + len = length(Prompt) + length(ORS) + print "Content-length:", len |& HttpService + print ORS Prompt |& HttpService + # ignore all the header lines + while ((HttpService |& getline) > 0) + continue + # stop talking to this client + close(HttpService) + # wait for new client request + HttpService |& getline + # do some logging + print systime(), strftime(), $0 + CGI_setup($1, $2, $3) + @} +@} + +function CGI_setup( method, uri, version, i) +@{ + delete GETARG + delete MENU + delete PARAM + GETARG["Method"] = method + GETARG["URI"] = uri + GETARG["Version"] = version + + i = index(uri, "?") + if (i > 0) @{ # is there a "?" indicating a CGI request? + split(substr(uri, 1, i-1), MENU, "[/:]") + split(substr(uri, i+1), PARAM, "&") + for (i in PARAM) @{ + PARAM[i] = _CGI_decode(PARAM[i]) + j = index(PARAM[i], "=") + GETARG[substr(PARAM[i], 1, j-1)] = \ + substr(PARAM[i], j+1) + @} + @} else @{ # there is no "?", no need for splitting PARAMs + split(uri, MENU, "[/:]") + @} + for (i in MENU) # decode characters in path + if (i > 4) # but not those in host name + MENU[i] = _CGI_decode(MENU[i]) +@} +@c endfile +@end example + +This isolates details in a single function, @code{CGI_setup}. +Decoding of encoded characters is pushed off to a helper function, +@code{_CGI_decode}. The use of the leading underscore (@samp{_}) in +the function name is intended to indicate that it is an ``internal'' +function, although there is nothing to enforce this: + +@example +@c file eg/network/coreserv.awk +function _CGI_decode(str, hexdigs, i, pre, code1, code2, + val, result) +@{ + hexdigs = "123456789abcdef" + + i = index(str, "%") + if (i == 0) # no work to do + return str + + do @{ + pre = substr(str, 1, i-1) # part before %xx + code1 = substr(str, i+1, 1) # first hex digit + code2 = substr(str, i+2, 1) # second hex digit + str = substr(str, i+3) # rest of string + + code1 = tolower(code1) + code2 = tolower(code2) + val = index(hexdigs, code1) * 16 \ + + index(hexdigs, code2) + + result = result pre sprintf("%c", val) + i = index(str, "%") + @} while (i != 0) + if (length(str) > 0) + result = result str + return result +@} +@c endfile +@end example + +This works by splitting the string apart around an encoded character. +The two digits are converted to lowercase and looked up in a string +of hex digits. Note that @code{0} is not in the string on purpose; +@code{index} returns zero when it's not found, automatically giving +the correct value! Once the hexadecimal value is converted from +characters in a string into a numerical value, @code{sprintf} +converts the value back into a real character. +The following is a simple test harness for the above functions: + +@example +@c file eg/network/testserv.awk +BEGIN @{ + CGI_setup("GET", + "http://www.gnu.org/cgi-bin/foo?p1=stuff&p2=stuff%26junk" \ + "&percent=a %25 sign", + "1.0") + for (i in MENU) + printf "MENU[\"%s\"] = %s\n", i, MENU[i] + for (i in PARAM) + printf "PARAM[\"%s\"] = %s\n", i, PARAM[i] + for (i in GETARG) + printf "GETARG[\"%s\"] = %s\n", i, GETARG[i] +@} +@c endfile +@end example + +And this is the result when we run it: + +@c artificial line wrap in last output line +@example +$ gawk -f testserv.awk +@print{} MENU["4"] = www.gnu.org +@print{} MENU["5"] = cgi-bin +@print{} MENU["6"] = foo +@print{} MENU["1"] = http +@print{} MENU["2"] = +@print{} MENU["3"] = +@print{} PARAM["1"] = p1=stuff +@print{} PARAM["2"] = p2=stuff&junk +@print{} PARAM["3"] = percent=a % sign +@print{} GETARG["p1"] = stuff +@print{} GETARG["percent"] = a % sign +@print{} GETARG["p2"] = stuff&junk +@print{} GETARG["Method"] = GET +@print{} GETARG["Version"] = 1.0 +@print{} GETARG["URI"] = http://www.gnu.org/cgi-bin/foo?p1=stuff& +p2=stuff%26junk&percent=a %25 sign +@end example + +@node Simple Server, Caveats, Interacting Service, Using Networking +@section A Simple Web Server +@cindex GUI +In the preceding @value{SECTION}, we built the core logic for event driven GUIs. +In this @value{SECTION}, we finally extend the core to a real application. +No one would actually write a commercial web server in @command{gawk}, but +it is instructive to see that it is feasible in principle. + +@iftex +@image{uf002331,4in} +@end iftex + +@cindex ELIZA program +@cindex Weizenbaum, Joseph +The application is ELIZA, the famous program by Joseph Weizenbaum that +mimics the behavior of a professional psychotherapist when talking to you. +Weizenbaum would certainly object to this description, but this is part of +the legend around ELIZA. +Take the site-independent core logic and append the following code: + +@example +@c file eg/network/eliza.awk +function SetUpServer() @{ + SetUpEliza() + TopHeader = \ + "<HTML><title>An HTTP-based System with GAWK</title>\ + <HEAD><META HTTP-EQUIV=\"Content-Type\"\ + CONTENT=\"text/html; charset=iso-8859-1\"></HEAD>\ + <BODY BGCOLOR=\"#ffffff\" TEXT=\"#000000\"\ + LINK=\"#0000ff\" VLINK=\"#0000ff\"\ + ALINK=\"#0000ff\"> <A NAME=\"top\">" + TopDoc = "\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI>\ + <A HREF=" MyPrefix "/AboutServer>About this server</A>\ + </LI><LI>\ + <A HREF=" MyPrefix "/AboutELIZA>About Eliza</A></LI>\ + <LI>\ + <A HREF=" MyPrefix \ + "/StartELIZA>Start talking to Eliza</A></LI></UL>" + TopFooter = "</BODY></HTML>" +@} +@c endfile +@end example + +@code{SetUpServer} is similar to the previous example, +except for calling another function, @code{SetUpEliza}. +This approach can be used to implement other kinds of servers. +The only changes needed to do so are hidden in the functions +@code{SetUpServer} and @code{HandleGET}. Perhaps it might be necessary to +implement other HTTP methods. +The @command{igawk} program that comes with @command{gawk} +may be useful for this process. + +When extending this example to a complete application, the first +thing to do is to implement the function @code{SetUpServer} to +initialize the HTML pages and some variables. These initializations +determine the way your HTML pages look (colors, titles, menu +items, etc.). + +@cindex GUI +The function @code{HandleGET} is a nested case selection that decides +which page the user wants to see next. Each nesting level refers to a menu +level of the GUI. Each case implements a certain action of the menu. On the +deepest level of case selection, the handler essentially knows what the +user wants and stores the answer into the variable that holds the HTML +page contents: + +@smallexample +@c file eg/network/eliza.awk +function HandleGET() @{ + # A real HTTP server would treat some parts of the URI as a file name. + # We take parts of the URI as menu choices and go on accordingly. + if(MENU[2] == "AboutServer") @{ + Document = "This is not a CGI script.\ + This is an httpd, an HTML file, and a CGI script all \ + in one GAWK script. It needs no separate www-server, \ + no installation, and no root privileges.\ + <p>To run it, do this:</p><ul>\ + <li> start this script with \"gawk -f httpserver.awk\",</li>\ + <li> and on the same host let your www browser open location\ + \"http://localhost:8080\"</li>\ + </ul>\<p>\ Details of HTTP come from:</p><ul>\ + <li>Hethmon: Illustrated Guide to HTTP</p>\ + <li>RFC 2068</li></ul><p>JK 14.9.1997</p>" + @} else if (MENU[2] == "AboutELIZA") @{ + Document = "This is an implementation of the famous ELIZA\ + program by Joseph Weizenbaum. It is written in GAWK and\ +/bin/sh: expad: command not found + @} else if (MENU[2] == "StartELIZA") @{ + gsub(/\+/, " ", GETARG["YouSay"]) + # Here we also have to substitute coded special characters + Document = "<form method=GET>" \ + "<h3>" ElizaSays(GETARG["YouSay"]) "</h3>\ + <p><input type=text name=YouSay value=\"\" size=60>\ + <br><input type=submit value=\"Tell her about it\"></p></form>" + @} +@} +@c endfile +@end smallexample + +Now we are down to the heart of ELIZA, so you can see how it works. +Initially the user does not say anything; then ELIZA resets its money +counter and asks the user to tell what comes to mind open heartedly. +The subsequent answers are converted to uppercase and stored for +later comparison. ELIZA presents the bill when being confronted with +a sentence that contains the phrase ``shut up.'' Otherwise, it looks for +keywords in the sentence, conjugates the rest of the sentence, remembers +the keyword for later use, and finally selects an answer from the set of +possible answers: + +@smallexample +@c file eg/network/eliza.awk +function ElizaSays(YouSay) @{ + if (YouSay == "") @{ + cost = 0 + answer = "HI, IM ELIZA, TELL ME YOUR PROBLEM" + @} else @{ + q = toupper(YouSay) + gsub("'", "", q) + if(q == qold) @{ + answer = "PLEASE DONT REPEAT YOURSELF !" + @} else @{ + if (index(q, "SHUT UP") > 0) @{ + answer = "WELL, PLEASE PAY YOUR BILL. ITS EXACTLY ... $"\ + int(100*rand()+30+cost/100) + @} else @{ + qold = q + w = "-" # no keyword recognized yet + for (i in k) @{ # search for keywords + if (index(q, i) > 0) @{ + w = i + break + @} + @} + if (w == "-") @{ # no keyword, take old subject + w = wold + subj = subjold + @} else @{ # find subject + subj = substr(q, index(q, w) + length(w)+1) + wold = w + subjold = subj # remember keyword and subject + @} + for (i in conj) + gsub(i, conj[i], q) # conjugation + # from all answers to this keyword, select one randomly + answer = r[indices[int(split(k[w], indices) * rand()) + 1]] + # insert subject into answer + gsub("_", subj, answer) + @} + @} + @} + cost += length(answer) # for later payment : 1 cent per character + return answer +@} +@c endfile +@end smallexample + +In the long but simple function @code{SetUpEliza}, you can see tables +for conjugation, keywords, and answers.@footnote{The version shown +here is abbreviated. The full version comes with the @command{gawk} +distribution.} The associative array @code{k} +contains indices into the array of answers @code{r}. To choose an +answer, ELIZA just picks an index randomly: + +@example +@c file eg/network/eliza.awk +function SetUpEliza() @{ + srand() + wold = "-" + subjold = " " + + # table for conjugation + conj[" ARE " ] = " AM " + conj["WERE " ] = "WAS " + conj[" YOU " ] = " I " + conj["YOUR " ] = "MY " + conj[" IVE " ] =\ + conj[" I HAVE " ] = " YOU HAVE " + conj[" YOUVE " ] =\ + conj[" YOU HAVE "] = " I HAVE " + conj[" IM " ] =\ + conj[" I AM " ] = " YOU ARE " + conj[" YOURE " ] =\ + conj[" YOU ARE " ] = " I AM " + + # table of all answers + r[1] = "DONT YOU BELIEVE THAT I CAN _" + r[2] = "PERHAPS YOU WOULD LIKE TO BE ABLE TO _ ?" +@c endfile + @dots{} +@end example +@ignore +@c file eg/network/eliza.awk + r[3] = "YOU WANT ME TO BE ABLE TO _ ?" + r[4] = "PERHAPS YOU DONT WANT TO _ " + r[5] = "DO YOU WANT TO BE ABLE TO _ ?" + r[6] = "WHAT MAKES YOU THINK I AM _ ?" + r[7] = "DOES IT PLEASE YOU TO BELIEVE I AM _ ?" + r[8] = "PERHAPS YOU WOULD LIKE TO BE _ ?" + r[9] = "DO YOU SOMETIMES WISH YOU WERE _ ?" + r[10] = "DONT YOU REALLY _ ?" + r[11] = "WHY DONT YOU _ ?" + r[12] = "DO YOU WISH TO BE ABLE TO _ ?" + r[13] = "DOES THAT TROUBLE YOU ?" + r[14] = "TELL ME MORE ABOUT SUCH FEELINGS" + r[15] = "DO YOU OFTEN FEEL _ ?" + r[16] = "DO YOU ENJOY FEELING _ ?" + r[17] = "DO YOU REALLY BELIEVE I DONT _ ?" + r[18] = "PERHAPS IN GOOD TIME I WILL _ " + r[19] = "DO YOU WANT ME TO _ ?" + r[20] = "DO YOU THINK YOU SHOULD BE ABLE TO _ ?" + r[21] = "WHY CANT YOU _ ?" + r[22] = "WHY ARE YOU INTERESTED IN WHETHER OR NOT I AM _ ?" + r[23] = "WOULD YOU PREFER IF I WERE NOT _ ?" + r[24] = "PERHAPS IN YOUR FANTASIES I AM _ " + r[25] = "HOW DO YOU KNOW YOU CANT _ ?" + r[26] = "HAVE YOU TRIED ?" + r[27] = "PERHAPS YOU CAN NOW _ " + r[28] = "DID YOU COME TO ME BECAUSE YOU ARE _ ?" + r[29] = "HOW LONG HAVE YOU BEEN _ ?" + r[30] = "DO YOU BELIEVE ITS NORMAL TO BE _ ?" + r[31] = "DO YOU ENJOY BEING _ ?" + r[32] = "WE WERE DISCUSSING YOU -- NOT ME" + r[33] = "Oh, I _" + r[34] = "YOU'RE NOT REALLY TALKING ABOUT ME, ARE YOU ?" + r[35] = "WHAT WOULD IT MEAN TO YOU, IF YOU GOT _ ?" + r[36] = "WHY DO YOU WANT _ ?" + r[37] = "SUPPOSE YOU SOON GOT _" + r[38] = "WHAT IF YOU NEVER GOT _ ?" + r[39] = "I SOMETIMES ALSO WANT _" + r[40] = "WHY DO YOU ASK ?" + r[41] = "DOES THAT QUESTION INTEREST YOU ?" + r[42] = "WHAT ANSWER WOULD PLEASE YOU THE MOST ?" + r[43] = "WHAT DO YOU THINK ?" + r[44] = "ARE SUCH QUESTIONS IN YOUR MIND OFTEN ?" + r[45] = "WHAT IS IT THAT YOU REALLY WANT TO KNOW ?" + r[46] = "HAVE YOU ASKED ANYONE ELSE ?" + r[47] = "HAVE YOU ASKED SUCH QUESTIONS BEFORE ?" + r[48] = "WHAT ELSE COMES TO MIND WHEN YOU ASK THAT ?" + r[49] = "NAMES DON'T INTEREST ME" + r[50] = "I DONT CARE ABOUT NAMES -- PLEASE GO ON" + r[51] = "IS THAT THE REAL REASON ?" + r[52] = "DONT ANY OTHER REASONS COME TO MIND ?" + r[53] = "DOES THAT REASON EXPLAIN ANYTHING ELSE ?" + r[54] = "WHAT OTHER REASONS MIGHT THERE BE ?" + r[55] = "PLEASE DON'T APOLOGIZE !" + r[56] = "APOLOGIES ARE NOT NECESSARY" + r[57] = "WHAT FEELINGS DO YOU HAVE WHEN YOU APOLOGIZE ?" + r[58] = "DON'T BE SO DEFENSIVE" + r[59] = "WHAT DOES THAT DREAM SUGGEST TO YOU ?" + r[60] = "DO YOU DREAM OFTEN ?" + r[61] = "WHAT PERSONS APPEAR IN YOUR DREAMS ?" + r[62] = "ARE YOU DISTURBED BY YOUR DREAMS ?" + r[63] = "HOW DO YOU DO ... PLEASE STATE YOUR PROBLEM" + r[64] = "YOU DON'T SEEM QUITE CERTAIN" + r[65] = "WHY THE UNCERTAIN TONE ?" + r[66] = "CAN'T YOU BE MORE POSITIVE ?" + r[67] = "YOU AREN'T SURE ?" + r[68] = "DON'T YOU KNOW ?" + r[69] = "WHY NO _ ?" + r[70] = "DON'T SAY NO, IT'S ALWAYS SO NEGATIVE" + r[71] = "WHY NOT ?" + r[72] = "ARE YOU SURE ?" + r[73] = "WHY NO ?" + r[74] = "WHY ARE YOU CONCERNED ABOUT MY _ ?" + r[75] = "WHAT ABOUT YOUR OWN _ ?" + r[76] = "CAN'T YOU THINK ABOUT A SPECIFIC EXAMPLE ?" + r[77] = "WHEN ?" + r[78] = "WHAT ARE YOU THINKING OF ?" + r[79] = "REALLY, ALWAYS ?" + r[80] = "DO YOU REALLY THINK SO ?" + r[81] = "BUT YOU ARE NOT SURE YOU _ " + r[82] = "DO YOU DOUBT YOU _ ?" + r[83] = "IN WHAT WAY ?" + r[84] = "WHAT RESEMBLANCE DO YOU SEE ?" + r[85] = "WHAT DOES THE SIMILARITY SUGGEST TO YOU ?" + r[86] = "WHAT OTHER CONNECTION DO YOU SEE ?" + r[87] = "COULD THERE REALLY BE SOME CONNECTIONS ?" + r[88] = "HOW ?" + r[89] = "YOU SEEM QUITE POSITIVE" + r[90] = "ARE YOU SURE ?" + r[91] = "I SEE" + r[92] = "I UNDERSTAND" + r[93] = "WHY DO YOU BRING UP THE TOPIC OF FRIENDS ?" + r[94] = "DO YOUR FRIENDS WORRY YOU ?" + r[95] = "DO YOUR FRIENDS PICK ON YOU ?" + r[96] = "ARE YOU SURE YOU HAVE ANY FRIENDS ?" + r[97] = "DO YOU IMPOSE ON YOUR FRIENDS ?" + r[98] = "PERHAPS YOUR LOVE FOR FRIENDS WORRIES YOU" + r[99] = "DO COMPUTERS WORRY YOU ?" + r[100] = "ARE YOU TALKING ABOUT ME IN PARTICULAR ?" + r[101] = "ARE YOU FRIGHTENED BY MACHINES ?" + r[102] = "WHY DO YOU MENTION COMPUTERS ?" + r[103] = "WHAT DO YOU THINK MACHINES HAVE TO DO WITH YOUR PROBLEMS ?" + r[104] = "DON'T YOU THINK COMPUTERS CAN HELP PEOPLE ?" + r[105] = "WHAT IS IT ABOUT MACHINES THAT WORRIES YOU ?" + r[106] = "SAY, DO YOU HAVE ANY PSYCHOLOGICAL PROBLEMS ?" + r[107] = "WHAT DOES THAT SUGGEST TO YOU ?" + r[108] = "I SEE" + r[109] = "IM NOT SURE I UNDERSTAND YOU FULLY" + r[110] = "COME COME ELUCIDATE YOUR THOUGHTS" + r[111] = "CAN YOU ELABORATE ON THAT ?" + r[112] = "THAT IS QUITE INTERESTING" + r[113] = "WHY DO YOU HAVE PROBLEMS WITH MONEY ?" + r[114] = "DO YOU THINK MONEY IS EVERYTHING ?" + r[115] = "ARE YOU SURE THAT MONEY IS THE PROBLEM ?" + r[116] = "I THINK WE WANT TO TALK ABOUT YOU, NOT ABOUT ME" + r[117] = "WHAT'S ABOUT ME ?" + r[118] = "WHY DO YOU ALWAYS BRING UP MY NAME ?" +@c endfile +@end ignore + +@example +@c file eg/network/eliza.awk + # table for looking up answers that + # fit to a certain keyword + k["CAN YOU"] = "1 2 3" + k["CAN I"] = "4 5" + k["YOU ARE"] =\ + k["YOURE"] = "6 7 8 9" +@c endfile + @dots{} +@end example +@ignore +@c file eg/network/eliza.awk + k["I DONT"] = "10 11 12 13" + k["I FEEL"] = "14 15 16" + k["WHY DONT YOU"] = "17 18 19" + k["WHY CANT I"] = "20 21" + k["ARE YOU"] = "22 23 24" + k["I CANT"] = "25 26 27" + k["I AM"] =\ + k["IM "] = "28 29 30 31" + k["YOU "] = "32 33 34" + k["I WANT"] = "35 36 37 38 39" + k["WHAT"] =\ + k["HOW"] =\ + k["WHO"] =\ + k["WHERE"] =\ + k["WHEN"] =\ + k["WHY"] = "40 41 42 43 44 45 46 47 48" + k["NAME"] = "49 50" + k["CAUSE"] = "51 52 53 54" + k["SORRY"] = "55 56 57 58" + k["DREAM"] = "59 60 61 62" + k["HELLO"] =\ + k["HI "] = "63" + k["MAYBE"] = "64 65 66 67 68" + k[" NO "] = "69 70 71 72 73" + k["YOUR"] = "74 75" + k["ALWAYS"] = "76 77 78 79" + k["THINK"] = "80 81 82" + k["LIKE"] = "83 84 85 86 87 88 89" + k["YES"] = "90 91 92" + k["FRIEND"] = "93 94 95 96 97 98" + k["COMPUTER"] = "99 100 101 102 103 104 105" + k["-"] = "106 107 108 109 110 111 112" + k["MONEY"] = "113 114 115" + k["ELIZA"] = "116 117 118" +@c endfile +@end ignore +@example +@c file eg/network/eliza.awk +@} +@c endfile +@end example + +@cindex Humphrys, Mark +@cindex ELIZA program +@cindex Yahoo! +Some interesting remarks and details (including the original source code +of ELIZA) are found on Mark Humphrys' home page. Yahoo! also has a +page with a collection of ELIZA-like programs. Many of them are written +in Java, some of them disclosing the Java source code, and a few even +explain how to modify the Java source code. + +@node Caveats, Challenges, Simple Server, Using Networking +@section Network Programming Caveats + +By now it should be clear +that debugging a networked application is more +complicated than debugging a single-process single-hosted application. +The behavior of a networked application sometimes looks non-causal because +it is not reproducible in a strong sense. Whether a network application +works or not sometimes depends on the following: + +@itemize @bullet +@item +How crowded the underlying network is. + +@item +If the party at the other end is running or not. + +@item +The state of the party at the other end. +@end itemize + +@cindex network +The most difficult problems for a beginner arise from the hidden states of the +underlying network. After closing a TCP connection, it's often necessary to wait +a short while before reopening the connection. Even more difficult is the +establishment of a connection that previously ended with a ``broken pipe.'' +Those connections have to ``time out'' for a minute or so +before they can reopen. +Check this with the command @samp{netstat -a}, which +provides a list of still ``active'' connections. + +@node Challenges, , Caveats, Using Networking +@section Where To Go From Here + +@cindex Loebner, Hugh +@cindex Contest +Now, you have learned enough to build your own application. You could, +for example, take part in the +Loebner Contest +to win $100,000. The Loebner Prize is the first formal instantiation of +a Turing Test. Hugh Loebner agreed with The Cambridge Center for Behavioral +Studies to underwrite a contest designed to implement the Turing Test. +Dr.@: Loebner pledged a Grand Prize of $100,000 for the first computer whose +responses were indistinguishable from a human's. Each year an annual prize +of $2000 and a bronze medal is awarded to the @emph{most} human computer. +The winner of the annual contest is the best entry relative to other entries +that year, irrespective of how good it is in an absolute sense. Here is +an example of a conversation with the winning program of 1997: + +@smallexample +(c)1997 Cambridge Center For Behavioral Studies all rights reserved [CONVERSE] +[Intelligent Research Ltd.] Start at: [1997/04/29 08:59:40] +PROGRAM[10:39:20]Hi, my name is Catherine, what's yours? +JUDGE02[10:39:36]My name is Nathan. +PROGRAM[10:39:37]Well, it's nice to meet you, Nathan. +PROGRAM[10:39:45]Did you see that story on CNN last night about the lesbian +PROGRAM[10:40:03]couple who came out at a White House party on Sunday? +JUDGE02[10:40:45]No. I'm sorry I missed that. How did the Prez take it? +PROGRAM[10:40:47]Ellen Degeneres was one of them - she was kissing her lover +PROGRAM[10:41:08]in front of Bill Clinton. Do you think he was very +PROGRAM[10:41:21]embarrassed? +JUDGE02[10:41:37]You tell me. I didn't see it. +PROGRAM[10:41:38]Are you still a Friend of Bill Clinton? +JUDGE02[10:42:22]Yes. I would like to talk about something else. Do you know anything about sculpture. +PROGRAM[10:42:24]Well, I suppose he's doing his best like most of us, but he +PROGRAM[10:42:35]does seem to attract an UNUSUAL amount of bad publicity, +PROGRAM[10:42:51]wouldnt you say--can it all be bad luck? +JUDGE02[10:43:20]I asked you if you knew anything about sculpture. +PROGRAM[10:43:21]OK, but what happens now--does it all drift on, with +PROGRAM[10:43:36]everyone waiting to see how much his friends have said about +PROGRAM[10:43:47]him under threat of lots and lots of jail-or will something +PROGRAM[10:44:01]dramatic actually happen? +JUDGE02[10:44:19]Do you know what sculpture is? +PROGRAM[10:44:20]Well, youre more a fan of his than I am I guess---though I +PROGRAM[10:44:34]cant SWEAR what Id have done in that locked hotel room, I +PROGRAM[10:44:51]must say. +JUDGE02[10:45:26]Have you ever been to the Salmagundi Club? +PROGRAM[10:45:29]Never. to go back to the original discussion Time to get +PROGRAM[10:45:43]off this one I think and onto something more down to earth! +JUDGE02[10:46:20]What is more down to earth than art? +PROGRAM[10:46:21]I don't know what -- to be honest. - +PROGRAM[10:46:30]I suppose it depends on where you're coming from, but as the +PROGRAM[10:46:46]song goes in My Fair Lady, "Why can't a woman be more like a +PROGRAM[10:47:03]man? " +JUDGE02[10:47:52]I liked that song but I prefer "Why can't children be more like we are" +JUDGE02[10:48:10]from another play the name of which I cannot remember. +PROGRAM[10:48:12]What, if anything, do you enjoy about being a man? +JUDGE02[10:49:01]I would enjoy all aspects of life no matter what my gender. +PROGRAM[10:49:02]Do you think of yourself as being attractive? +@end smallexample + +@cindex Clinton, Bill +This program insists on always speaking about the same story around Bill +Clinton. You see, even a program with a rather narrow mind can behave so +much like a human being that it can win this prize. It is quite common to +let these programs talk to each other via network connections. But during the +competition itself, the program and its computer have to be present at the +place the competition is held. We all would love to see a @command{gawk} +program win in such an event. Maybe it is up to you to accomplish this? + +Some other ideas for useful networked applications: +@itemize @bullet +@item +Read the file @file{doc/awkforai.txt} in the @command{gawk} distribution. +It was written by Ronald P.@: Loui (Associate Professor of +Computer Science, at Washington University in St. Louis, +@email{loui@@ai.wustl.edu}) and summarizes why +he teaches @command{gawk} to students of Artificial Intelligence. Here are +some passages from the text: + +@cindex AI +@cindex PROLOG +@cindex Loui, Ronald P. +@cindex agent +@quotation +The GAWK manual can +be consumed in a single lab session and the language can be mastered by +the next morning by the average student. GAWK's automatic +initialization, implicit coercion, I/O support and lack of pointers +forgive many of the mistakes that young programmers are likely to make. +Those who have seen C but not mastered it are happy to see that GAWK +retains some of the same sensibilities while adding what must be +regarded as spoonsful of syntactic sugar.@* +@dots{}@* +@cindex robot +There are further simple answers. Probably the best is the fact that +increasingly, undergraduate AI programming is involving the Web. Oren +Etzioni (University of Washington, Seattle) has for a while been arguing +that the ``softbot'' is replacing the mechanical engineers' robot as the +most glamorous AI testbed. If the artifact whose behavior needs to be +controlled in an intelligent way is the software agent, then a language +that is well-suited to controlling the software environment is the +appropriate language. That would imply a scripting language. If the +robot is KAREL, then the right language is ``turn left; turn right.'' If +the robot is Netscape, then the right language is something that can +generate @samp{netscape -remote 'openURL(http://cs.wustl.edu/~loui)'} with +elan.@* +@dots{}@* +AI programming requires high-level thinking. There have always been a few +gifted programmers who can write high-level programs in assembly language. +Most however need the ambient abstraction to have a higher floor.@* +@dots{}@* +Second, inference is merely the expansion of notation. No matter whether +the logic that underlies an AI program is fuzzy, probabilistic, deontic, +defeasible, or deductive, the logic merely defines how strings can be +transformed into other strings. A language that provides the best +support for string processing in the end provides the best support for +logic, for the exploration of various logics, and for most forms of +symbolic processing that AI might choose to call ``reasoning'' instead of +``logic.'' The implication is that PROLOG, which saves the AI programmer +from having to write a unifier, saves perhaps two dozen lines of GAWK +code at the expense of strongly biasing the logic and representational +expressiveness of any approach. +@end quotation + +Now that @command{gawk} itself can connect to the Internet, it should be obvious +that it is suitable for writing intelligent web agents. + +@item +@command{awk} is strong at pattern recognition and string processing. +So, it is well suited to the classic problem of language translation. +A first try could be a program that knows the 100 most frequent English +words and their counterparts in German or French. The service could be +implemented by regularly reading email with the program above, replacing +each word by its translation and sending the translation back via SMTP. +Users would send English email to their translation service and get +back a translated email message in return. As soon as this works, +more effort can be spent on a real translation program. + +@item +Another dialogue-oriented application (on the verge +of ridicule) is the email ``support service.'' Troubled customers write an +email to an automatic @command{gawk} service that reads the email. It looks +for keywords in the mail and assembles a reply email accordingly. By carefully +investigating the email header, and repeating these keywords through the +reply email, it is rather simple to give the customer a feeling that +someone cares. Ideally, such a service would search a database of previous +cases for solutions. If none exists, the database could, for example, consist +of all the newsgroups, mailing lists and FAQs on the Internet. +@end itemize + +@node Some Applications and Techniques, Links, Using Networking, Top +@comment node-name, next, previous, up + +@chapter Some Applications and Techniques +In this @value{CHAPTER}, we look at a number of self-contained +scripts, with an emphasis on concise networking. Along the way, we +work towards creating building blocks that encapsulate often needed +functions of the networking world, show new techniques that +broaden the scope of problems that can be solved with @command{gawk}, and +explore leading edge technology that may shape the future of networking. + +We often refer to the site-independent core of the server that +we built in +@ref{Simple Server, ,A Simple Web Server}. +When building new and non-trivial servers, we +always copy this building block and append new instances of the two +functions @code{SetUpServer} and @code{HandleGET}. + +This makes a lot of sense, since +this scheme of event-driven +execution provides @command{gawk} with an interface to the most widely +accepted standard for GUIs: the web browser. Now, @command{gawk} can even rival +Tcl/Tk. + +@cindex Tcl/Tk +@cindex JavaScript +Tcl and @command{gawk} have much in common. Both are simple scripting languages +that allow us to quickly solve problems with short programs. But Tcl has Tk +on top of it and @command{gawk} had nothing comparable up to now. While Tcl +needs a large and ever changing library (Tk, which was bound to the X Window +System until recently), @command{gawk} needs just the networking interface +and some kind of browser on the client's side. Besides better portability, +the most important advantage of this approach (embracing well-established +standards such HTTP and HTML) is that @emph{we do not need to change the +language}. We let others do the work of fighting over protocols and standards. +We can use HTML, JavaScript, VRML, or whatever else comes along to do our work. + +@menu +* PANIC:: An Emergency Web Server. +* GETURL:: Retrieving Web Pages. +* REMCONF:: Remote Configuration Of Embedded Systems. +* URLCHK:: Look For Changed Web Pages. +* WEBGRAB:: Extract Links From A Page. +* STATIST:: Graphing A Statistical Distribution. +* MAZE:: Walking Through A Maze In Virtual Reality. +* MOBAGWHO:: A Simple Mobile Agent. +* STOXPRED:: Stock Market Prediction As A Service. +* PROTBASE:: Searching Through A Protein Database. +@end menu + +@node PANIC, GETURL, Some Applications and Techniques, Some Applications and Techniques +@section PANIC: an Emergency Web Server +@cindex PANIC program +At first glance, the @code{"Hello, world"} example in +@ref{Primitive Service, ,A Primitive Web Service}, +seems useless. By adding just a few lines, we can turn it into something useful. + +The PANIC program tells everyone who connects that the local +site is not working. When a web server breaks down, it makes a difference +if customers get a strange ``network unreachable'' message, or a short message +telling them that the server has a problem. In such an emergency, +the hard disk and everything on it (including the regular web service) may +be unavailable. Rebooting the web server off a diskette makes sense in this +setting. + +To use the PANIC program as an emergency web server, all you need are the +@command{gawk} executable and the program below on a diskette. By default, +it connects to port 8080. A different value may be supplied on the +command line: + +@example +@c file eg/network/panic.awk +BEGIN @{ + RS = ORS = "\r\n" + if (MyPort == 0) MyPort = 8080 + HttpService = "/inet/tcp/" MyPort "/0/0" + Hello = "<HTML><HEAD><TITLE>Out Of Service</TITLE>" \ + "</HEAD><BODY><H1>" \ + "This site is temporarily out of service." \ + "</H1></BODY></HTML>" + Len = length(Hello) + length(ORS) + while ("awk" != "complex") @{ + print "HTTP/1.0 200 OK" |& HttpService + print "Content-Length: " Len ORS |& HttpService + print Hello |& HttpService + while ((HttpService |& getline) > 0) + continue; + close(HttpService) + @} +@} +@c endfile +@end example + +@node GETURL, REMCONF, PANIC, Some Applications and Techniques +@section GETURL: Retrieving Web Pages +@cindex GETURL program +@cindex robot +GETURL is a versatile building block for shell scripts that need to retrieve +files from the Internet. It takes a web address as a command-line parameter and +tries to retrieve the contents of this address. The contents are printed +to standard output, while the header is printed to @file{/dev/stderr}. +A surrounding shell script +could analyze the contents and extract the text or the links. An ASCII +browser could be written around GETURL. But more interestingly, web robots are +straightforward to write on top of GETURL. On the Internet, you can find +several programs of the same name that do the same job. They are usually +much more complex internally and at least 10 times longer. + +At first, GETURL checks if it was called with exactly one web address. +Then, it checks if the user chose to use a special proxy server whose name +is handed over in a variable. By default, it is assumed that the local +machine serves as proxy. GETURL uses the @code{GET} method by default +to access the web page. By handing over the name of a different method +(such as @code{HEAD}), it is possible to choose a different behavior. With +the @code{HEAD} method, the user does not receive the body of the page +content, but does receive the header: + +@example +@c file eg/network/geturl.awk +BEGIN @{ + if (ARGC != 2) @{ + print "GETURL - retrieve Web page via HTTP 1.0" + print "IN:\n the URL as a command-line parameter" + print "PARAM(S):\n -v Proxy=MyProxy" + print "OUT:\n the page content on stdout" + print " the page header on stderr" + print "JK 16.05.1997" + print "ADR 13.08.2000" + exit + @} + URL = ARGV[1]; ARGV[1] = "" + if (Proxy == "") Proxy = "127.0.0.1" + if (ProxyPort == 0) ProxyPort = 80 + if (Method == "") Method = "GET" + HttpService = "/inet/tcp/0/" Proxy "/" ProxyPort + ORS = RS = "\r\n\r\n" + print Method " " URL " HTTP/1.0" |& HttpService + HttpService |& getline Header + print Header > "/dev/stderr" + while ((HttpService |& getline) > 0) + printf "%s", $0 + close(HttpService) +@} +@c endfile +@end example + +This program can be changed as needed, but be careful with the last lines. +Make sure transmission of binary data is not corrupted by additional line +breaks. Even as it is now, the byte sequence @code{"\r\n\r\n"} would +disappear if it were contained in binary data. Don't get caught in a +trap when trying a quick fix on this one. + +@node REMCONF, URLCHK, GETURL, Some Applications and Techniques +@section REMCONF: Remote Configuration of Embedded Systems +@cindex REMCONF program +@cindex Linux +@cindex GNU/Linux +@cindex Yahoo! +Today, you often find powerful processors in embedded systems. Dedicated +network routers and controllers for all kinds of machinery are examples +of embedded systems. Processors like the Intel 80x86 or the AMD Elan are +able to run multitasking operating systems, such as XINU or GNU/Linux +in embedded PCs. These systems are small and usually do not have +a keyboard or a display. Therefore it is difficult to set up their +configuration. There are several widespread ways to set them up: + +@itemize @bullet +@item +DIP switches + +@item +Read Only Memories such as EPROMs + +@item +Serial lines or some kind of keyboard + +@item +Network connections via @command{telnet} or SNMP + +@item +HTTP connections with HTML GUIs +@end itemize + +In this @value{SECTION}, we look at a solution that uses HTTP connections +to control variables of an embedded system that are stored in a file. +Since embedded systems have tight limits on resources like memory, +it is difficult to employ advanced techniques such as SNMP and HTTP +servers. @command{gawk} fits in quite nicely with its single executable +which needs just a short script to start working. +The following program stores the variables in a file, and a concurrent +process in the embedded system may read the file. The program uses the +site-independent part of the simple web server that we developed in +@ref{Interacting Service, ,A Web Service with Interaction}. +As mentioned there, all we have to do is to write two new procedures +@code{SetUpServer} and @code{HandleGET}: + +@smallexample +@c file eg/network/remconf.awk +function SetUpServer() @{ + TopHeader = "<HTML><title>Remote Configuration</title>" + TopDoc = "<BODY>\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\ + <LI><A HREF=" MyPrefix "/ReadConfig>Read Configuration</A></LI>\ + <LI><A HREF=" MyPrefix "/CheckConfig>Check Configuration</A></LI>\ + <LI><A HREF=" MyPrefix "/ChangeConfig>Change Configuration</A></LI>\ + <LI><A HREF=" MyPrefix "/SaveConfig>Save Configuration</A></LI>\ + </UL>" + TopFooter = "</BODY></HTML>" + if (ConfigFile == "") ConfigFile = "config.asc" +@} +@c endfile +@end smallexample + +The function @code{SetUpServer} initializes the top level HTML texts +as usual. It also initializes the name of the file that contains the +configuration parameters and their values. In case the user supplies +a name from the command line, that name is used. The file is expected to +contain one parameter per line, with the name of the parameter in +column one and the value in column two. + +The function @code{HandleGET} reflects the structure of the menu +tree as usual. The first menu choice tells the user what this is all +about. The second choice reads the configuration file line by line +and stores the parameters and their values. Notice that the record +separator for this file is @code{"\n"}, in contrast to the record separator +for HTTP. The third menu choice builds an HTML table to show +the contents of the configuration file just read. The fourth choice +does the real work of changing parameters, and the last one just saves +the configuration into a file: + +@smallexample +@c file eg/network/remconf.awk +function HandleGET() @{ + if(MENU[2] == "AboutServer") @{ + Document = "This is a GUI for remote configuration of an\ + embedded system. It is is implemented as one GAWK script." + @} else if (MENU[2] == "ReadConfig") @{ + RS = "\n" + while ((getline < ConfigFile) > 0) + config[$1] = $2; + close(ConfigFile) + RS = "\r\n" + Document = "Configuration has been read." + @} else if (MENU[2] == "CheckConfig") @{ + Document = "<TABLE BORDER=1 CELLPADDING=5>" + for (i in config) + Document = Document "<TR><TD>" i "</TD>" \ + "<TD>" config[i] "</TD></TR>" + Document = Document "</TABLE>" + @} else if (MENU[2] == "ChangeConfig") @{ + if ("Param" in GETARG) @{ # any parameter to set? + if (GETARG["Param"] in config) @{ # is parameter valid? + config[GETARG["Param"]] = GETARG["Value"] + Document = (GETARG["Param"] " = " GETARG["Value"] ".") + @} else @{ + Document = "Parameter <b>" GETARG["Param"] "</b> is invalid." + @} + @} else @{ + Document = "<FORM method=GET><h4>Change one parameter</h4>\ + <TABLE BORDER CELLPADDING=5>\ + <TR><TD>Parameter</TD><TD>Value</TD></TR>\ + <TR><TD><input type=text name=Param value=\"\" size=20></TD>\ + <TD><input type=text name=Value value=\"\" size=40></TD>\ + </TR></TABLE><input type=submit value=\"Set\"></FORM>" + @} + @} else if (MENU[2] == "SaveConfig") @{ + for (i in config) + printf("%s %s\n", i, config[i]) > ConfigFile + close(ConfigFile) + Document = "Configuration has been saved." + @} +@} +@c endfile +@end smallexample + +@cindex MiniSQL +We could also view the configuration file as a database. From this +point of view, the previous program acts like a primitive database server. +Real SQL database systems also make a service available by providing +a TCP port that clients can connect to. But the application level protocols +they use are usually proprietary and also change from time to time. +This is also true for the protocol that +MiniSQL uses. + +@node URLCHK, WEBGRAB, REMCONF, Some Applications and Techniques +@section URLCHK: Look for Changed Web Pages +@cindex URLCHK program +Most people who make heavy use of Internet resources have a large +bookmark file with pointers to interesting web sites. It is impossible +to regularly check by hand if any of these sites have changed. A program +is needed to automatically look at the headers of web pages and tell +which ones have changed. URLCHK does the comparison after using GETURL +with the @code{HEAD} method to retrieve the header. + +Like GETURL, this program first checks that it is called with exactly +one command-line parameter. URLCHK also takes the same command-line variables +@code{Proxy} and @code{ProxyPort} as GETURL, +because these variables are handed over to GETURL for each URL +that gets checked. The one and only parameter is the name of a file that +contains one line for each URL. In the first column, we find the URL, and +the second and third columns hold the length of the URL's body when checked +for the two last times. Now, we follow this plan: + +@enumerate +@item +Read the URLs from the file and remember their most recent lengths + +@item +Delete the contents of the file + +@item +For each URL, check its new length and write it into the file + +@item +If the most recent and the new length differ, tell the user +@end enumerate + +It may seem a bit peculiar to read the URLs from a file together +with their two most recent lengths, but this approach has several +advantages. You can call the program again and again with the same +file. After running the program, you can regenerate the changed URLs +by extracting those lines that differ in their second and third columns: + +@c inspired by URLCHK in iX 5/97 166. +@smallexample +@c file eg/network/urlchk.awk +BEGIN @{ + if (ARGC != 2) @{ + print "URLCHK - check if URLs have changed" + print "IN:\n the file with URLs as a command-line parameter" + print " file contains URL, old length, new length" + print "PARAMS:\n -v Proxy=MyProxy -v ProxyPort=8080" + print "OUT:\n same as file with URLs" + print "JK 02.03.1998" + exit + @} + URLfile = ARGV[1]; ARGV[1] = "" + if (Proxy != "") Proxy = " -v Proxy=" Proxy + if (ProxyPort != "") ProxyPort = " -v ProxyPort=" ProxyPort + while ((getline < URLfile) > 0) + Length[$1] = $3 + 0 + close(URLfile) # now, URLfile is read in and can be updated + GetHeader = "gawk " Proxy ProxyPort " -v Method=\"HEAD\" -f geturl.awk " + for (i in Length) @{ + GetThisHeader = GetHeader i " 2>&1" + while ((GetThisHeader | getline) > 0) + if (toupper($0) ~ /CONTENT-LENGTH/) NewLength = $2 + 0 + close(GetThisHeader) + print i, Length[i], NewLength > URLfile + if (Length[i] != NewLength) # report only changed URLs + print i, Length[i], NewLength + @} + close(URLfile) +@} +@c endfile +@end smallexample + +Another thing that may look strange is the way GETURL is called. +Before calling GETURL, we have to check if the proxy variables need +to be passed on. If so, we prepare strings that will become part +of the command line later. In @code{GetHeader}, we store these strings +together with the longest part of the command line. Later, in the loop +over the URLs, @code{GetHeader} is appended with the URL and a redirection +operator to form the command that reads the URL's header over the Internet. +GETURL always produces the headers over @file{/dev/stderr}. That is +the reason why we need the redirection operator to have the header +piped in. + +This program is not perfect because it assumes that changing URLs +results in changed lengths, which is not necessarily true. A more +advanced approach is to look at some other header line that +holds time information. But, as always when things get a bit more +complicated, this is left as an exercise to the reader. + +@node WEBGRAB, STATIST, URLCHK, Some Applications and Techniques +@section WEBGRAB: Extract Links from a Page +@cindex WEBGRAB program +@c Inspired by iX 1/98 157. +@cindex robot +Sometimes it is necessary to extract links from web pages. +Browsers do it, web robots do it, and sometimes even humans do it. +Since we have a tool like GETURL at hand, we can solve this problem with +some help from the Bourne shell: + +@example +@c file eg/network/webgrab.awk +BEGIN @{ RS = "http://[#%&\\+\\-\\./0-9\\:;\\?A-Z_a-z\\~]*" @} +RT != "" @{ + command = ("gawk -v Proxy=MyProxy -f geturl.awk " RT \ + " > doc" NR ".html") + print command +@} +@c endfile +@end example + +Notice that the regular expression for URLs is rather crude. A precise +regular expression is much more complex. But this one works +rather well. One problem is that it is unable to find internal links of +an HTML document. Another problem is that +@samp{ftp}, @samp{telnet}, @samp{news}, @samp{mailto}, and other kinds +of links are missing in the regular expression. +However, it is straightforward to add them, if doing so is necessary for other tasks. + +This program reads an HTML file and prints all the HTTP links that it finds. +It relies on @command{gawk}'s ability to use regular expressions as record +separators. With @code{RS} set to a regular expression that matches links, +the second action is executed each time a non-empty link is found. +We can find the matching link itself in @code{RT}. + +The action could use the @code{system} function to let another GETURL +retrieve the page, but here we use a different approach. +This simple program prints shell commands that can be piped into @command{sh} +for execution. This way it is possible to first extract +the links, wrap shell commands around them, and pipe all the shell commands +into a file. After editing the file, execution of the file retrieves +exactly those files that we really need. In case we do not want to edit, +we can retrieve all the pages like this: + +@smallexample +gawk -f geturl.awk http://www.suse.de | gawk -f webgrab.awk | sh +@end smallexample + +@cindex Microsoft Windows +After this, you will find the contents of all referenced documents in +files named @file{doc*.html} even if they do not contain HTML code. +The most annoying thing is that we always have to pass the proxy to +GETURL. If you do not like to see the headers of the web pages +appear on the screen, you can redirect them to @file{/dev/null}. +Watching the headers appear can be quite interesting, because +it reveals +interesting details such as which web server the companies use. +Now, it is clear how the clever marketing people +use web robots to determine the +market shares +of Microsoft and Netscape in the web server market. + +Port 80 of any web server is like a small hole in a repellent firewall. +After attaching a browser to port 80, we usually catch a glimpse +of the bright side of the server (its home page). With a tool like GETURL +at hand, we are able to discover some of the more concealed +or even ``indecent'' services (i.e., lacking conformity to standards of quality). +It can be exciting to see the fancy CGI scripts that lie +there, revealing the inner workings of the server, ready to be called: + +@itemize @bullet +@item +With a command such as: + +@example +gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/ +@end example + +some servers give you a directory listing of the CGI files. +Knowing the names, you can try to call some of them and watch +for useful results. Sometimes there are executables in such directories +(such as Perl interpreters) that you may call remotely. If there are +subdirectories with configuration data of the web server, this can also +be quite interesting to read. + +@item +@cindex apache +The well-known Apache web server usually has its CGI files in the +directory @file{/cgi-bin}. There you can often find the scripts +@file{test-cgi} and @file{printenv}. Both tell you some things +about the current connection and the installation of the web server. +Just call: + +@smallexample +gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/test-cgi +gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/printenv +@end smallexample + +@item +Sometimes it is even possible to retrieve system files like the web +server's log file---possibly containing customer data---or even the file +@file{/etc/passwd}. +(We don't recommend this!) +@end itemize + +@strong{Caution:} +Although this may sound funny or simply irrelevant, we are talking about +severe security holes. Try to explore your own system this way and make +sure that none of the above reveals too much information about your system. + +@node STATIST, MAZE, WEBGRAB, Some Applications and Techniques +@section STATIST: Graphing a Statistical Distribution +@cindex STATIST program + +@cindex GNUPlot utility +@cindex image format +@cindex @file{gif} image format +@cindex @file{png} image format +@cindex @file{ps} image format +@cindex Boutell, Thomas +@iftex +@image{statist,3in} +@end iftex +In the HTTP server examples we've shown thus far, we never present an image +to the browser and its user. Presenting images is one task. Generating +images that reflect some user input and presenting these dynamically +generated images is another. In this @value{SECTION}, we use GNUPlot +for generating @file{.png}, @file{.ps}, or @file{.gif} +files.@footnote{Due to licensing problems, the default +installation of GNUPlot disables the generation of @file{.gif} files. +If your installed version does not accept @samp{set term gif}, +just download and install the most recent version of GNUPlot and the +@uref{http://www.boutell.com/gd/, GD library} +by Thomas Boutell. +Otherwise you still have the chance to generate some +ASCII-art style images with GNUPlot by using @samp{set term dumb}. +(We tried it and it worked.)} + +The program we develop takes the statistical parameters of two samples +and computes the t-test statistics. As a result, we get the probabilities +that the means and the variances of both samples are the same. In order to +let the user check plausibility, the program presents an image of the +distributions. The statistical computation follows +@cite{Numerical Recipes in C: The Art of Scientific Computing} +by William H.@: Press, Saul A.@: Teukolsky, William T.@: Vetterling, and Brian P. Flannery. +Since @command{gawk} does not have a built-in function +for the computation of the beta function, we use the @code{ibeta} function +of GNUPlot. As a side effect, we learn how to use GNUPlot as a +sophisticated calculator. The comparison of means is done as in @code{tutest}, +paragraph 14.2, page 613, and the comparison of variances is done as in @code{ftest}, +page 611 in @cite{Numerical Recipes}. +@cindex Numerical Recipes + +As usual, we take the site-independent code for servers and append +our own functions @code{SetUpServer} and @code{HandleGET}: + +@smallexample +@c file eg/network/statist.awk +function SetUpServer() @{ + TopHeader = "<HTML><title>Statistics with GAWK</title>" + TopDoc = "<BODY>\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\ + <LI><A HREF=" MyPrefix "/EnterParameters>Enter Parameters</A></LI>\ + </UL>" + TopFooter = "</BODY></HTML>" + GnuPlot = "gnuplot 2>&1" + m1=m2=0; v1=v2=1; n1=n2=10 +@} +@c endfile +@end smallexample + +Here, you see the menu structure that the user sees. Later, we +will see how the program structure of the @code{HandleGET} function +reflects the menu structure. What is missing here is the link for the +image we generate. In an event-driven environment, request, +generation, and delivery of images are separated. + +Notice the way we initialize the @code{GnuPlot} command string for +the pipe. By default, +GNUPlot outputs the generated image via standard output, as well as +the results of @code{print}(ed) calculations via standard error. +The redirection causes standard error to be mixed into standard +output, enabling us to read results of calculations with @code{getline}. +By initializing the statistical parameters with some meaningful +defaults, we make sure the user gets an image the first time +he uses the program. + +@cindex JavaScript +Following is the rather long function @code{HandleGET}, which +implements the contents of this service by reacting to the different +kinds of requests from the browser. Before you start playing with +this script, make sure that your browser supports JavaScript and that it also +has this option switched on. The script uses a short snippet of +JavaScript code for delayed opening of a window with an image. +A more detailed explanation follows: + +@smallexample +@c file eg/network/statist.awk +function HandleGET() @{ + if(MENU[2] == "AboutServer") @{ + Document = "This is a GUI for a statistical computation.\ + It compares means and variances of two distributions.\ + It is implemented as one GAWK script and uses GNUPLOT." + @} else if (MENU[2] == "EnterParameters") @{ + Document = "" + if ("m1" in GETARG) @{ # are there parameters to compare? + Document = Document "<SCRIPT LANGUAGE=\"JavaScript\">\ + setTimeout(\"window.open(\\\"" MyPrefix "/Image" systime()\ + "\\\",\\\"dist\\\", \\\"status=no\\\");\", 1000); </SCRIPT>" + m1 = GETARG["m1"]; v1 = GETARG["v1"]; n1 = GETARG["n1"] + m2 = GETARG["m2"]; v2 = GETARG["v2"]; n2 = GETARG["n2"] + t = (m1-m2)/sqrt(v1/n1+v2/n2) + df = (v1/n1+v2/n2)*(v1/n1+v2/n2)/((v1/n1)*(v1/n1)/(n1-1) \ + + (v2/n2)*(v2/n2) /(n2-1)) + if (v1>v2) @{ + f = v1/v2 + df1 = n1 - 1 + df2 = n2 - 1 + @} else @{ + f = v2/v1 + df1 = n2 - 1 + df2 = n1 - 1 + @} + print "pt=ibeta(" df/2 ",0.5," df/(df+t*t) ")" |& GnuPlot + print "pF=2.0*ibeta(" df2/2 "," df1/2 "," \ + df2/(df2+df1*f) ")" |& GnuPlot + print "print pt, pF" |& GnuPlot + RS="\n"; GnuPlot |& getline; RS="\r\n" # $1 is pt, $2 is pF + print "invsqrt2pi=1.0/sqrt(2.0*pi)" |& GnuPlot + print "nd(x)=invsqrt2pi/sd*exp(-0.5*((x-mu)/sd)**2)" |& GnuPlot + print "set term png small color" |& GnuPlot + #print "set term postscript color" |& GnuPlot + #print "set term gif medium size 320,240" |& GnuPlot + print "set yrange[-0.3:]" |& GnuPlot + print "set label 'p(m1=m2) =" $1 "' at 0,-0.1 left" |& GnuPlot + print "set label 'p(v1=v2) =" $2 "' at 0,-0.2 left" |& GnuPlot + print "plot mu=" m1 ",sd=" sqrt(v1) ", nd(x) title 'sample 1',\ + mu=" m2 ",sd=" sqrt(v2) ", nd(x) title 'sample 2'" |& GnuPlot + print "quit" |& GnuPlot + GnuPlot |& getline Image + while ((GnuPlot |& getline) > 0) + Image = Image RS $0 + close(GnuPlot) + @} + Document = Document "\ + <h3>Do these samples have the same Gaussian distribution?</h3>\ + <FORM METHOD=GET> <TABLE BORDER CELLPADDING=5>\ + <TR>\ + <TD>1. Mean </TD> + <TD><input type=text name=m1 value=" m1 " size=8></TD>\ + <TD>1. Variance</TD> + <TD><input type=text name=v1 value=" v1 " size=8></TD>\ + <TD>1. Count </TD> + <TD><input type=text name=n1 value=" n1 " size=8></TD>\ + </TR><TR>\ + <TD>2. Mean </TD> + <TD><input type=text name=m2 value=" m2 " size=8></TD>\ + <TD>2. Variance</TD> + <TD><input type=text name=v2 value=" v2 " size=8></TD>\ + <TD>2. Count </TD> + <TD><input type=text name=n2 value=" n2 " size=8></TD>\ + </TR> <input type=submit value=\"Compute\">\ + </TABLE></FORM><BR>" + @} else if (MENU[2] ~ "Image") @{ + Reason = "OK" ORS "Content-type: image/png" + #Reason = "OK" ORS "Content-type: application/x-postscript" + #Reason = "OK" ORS "Content-type: image/gif" + Header = Footer = "" + Document = Image + @} +@} +@c endfile +@end smallexample + +@cindex PostScript +As usual, we give a short description of the service in the first +menu choice. The third menu choice shows us that generation and +presentation of an image are two separate actions. While the latter +takes place quite instantly in the third menu choice, the former +takes place in the much longer second choice. Image data passes from the +generating action to the presenting action via the variable @code{Image} +that contains a complete @file{.png} image, which is otherwise stored +in a file. If you prefer @file{.ps} or @file{.gif} images over the +default @file{.png} images, you may select these options by uncommenting +the appropriate lines. But remember to do so in two places: when +telling GNUPlot which kind of images to generate, and when transmitting the +image at the end of the program. + +Looking at the end of the program, +the way we pass the @samp{Content-type} to the browser is a bit unusual. +It is appended to the @samp{OK} of the first header line +to make sure the type information becomes part of the header. +The other variables that get transmitted across the network are +made empty, because in this case we do not have an HTML document to +transmit, but rather raw image data to contain in the body. + +Most of the work is done in the second menu choice. It starts with a +strange JavaScript code snippet. When first implementing this server, +we used a short @code{@w{"<IMG SRC="} MyPrefix "/Image>"} here. But then +browsers got smarter and tried to improve on speed by requesting the +image and the HTML code at the same time. When doing this, the browser +tries to build up a connection for the image request while the request for +the HTML text is not yet completed. The browser tries to connect +to the @command{gawk} server on port 8080 while port 8080 is still in use for +transmission of the HTML text. The connection for the image cannot be +built up, so the image appears as ``broken'' in the browser window. +We solved this problem by telling the browser to open a separate window +for the image, but only after a delay of 1000 milliseconds. +By this time, the server should be ready for serving the next request. + +But there is one more subtlety in the JavaScript code. +Each time the JavaScript code opens a window for the image, the +name of the image is appended with a timestamp (@code{systime}). +Why this constant change of name for the image? Initially, we always named +the image @code{Image}, but then the Netscape browser noticed the name +had @emph{not} changed since the previous request and displayed the +previous image (caching behavior). The server core +is implemented so that browsers are told @emph{not} to cache anything. +Obviously HTTP requests do not always work as expected. One way to +circumvent the cache of such overly smart browsers is to change the +name of the image with each request. These three lines of JavaScript +caused us a lot of trouble. + +The rest can be broken +down into two phases. At first, we check if there are statistical +parameters. When the program is first started, there usually are no +parameters because it enters the page coming from the top menu. +Then, we only have to present the user a form that he can use to change +statistical parameters and submit them. Subsequently, the submission of +the form causes the execution of the first phase because @emph{now} +there @emph{are} parameters to handle. + +Now that we have parameters, we know there will be an image available. +Therefore we insert the JavaScript code here to initiate the opening +of the image in a separate window. Then, +we prepare some variables that will be passed to GNUPlot for calculation +of the probabilities. Prior to reading the results, we must temporarily +change @code{RS} because GNUPlot separates lines with newlines. +After instructing GNUPlot to generate a @file{.png} (or @file{.ps} or +@file{.gif}) image, we initiate the insertion of some text, +explaining the resulting probabilities. The final @samp{plot} command +actually generates the image data. This raw binary has to be read in carefully +without adding, changing, or deleting a single byte. Hence the unusual +initialization of @code{Image} and completion with a @code{while} loop. + +When using this server, it soon becomes clear that it is far from being +perfect. It mixes source code of six scripting languages or protocols: + +@itemize @bullet +@item GNU @command{awk} implements a server for the protocol: +@item HTTP which transmits: +@item HTML text which contains a short piece of: +@item JavaScript code opening a separate window. +@item A Bourne shell script is used for piping commands into: +@item GNUPlot to generate the image to be opened. +@end itemize + +After all this work, the GNUPlot image opens in the JavaScript window +where it can be viewed by the user. + +It is probably better not to mix up so many different languages. +The result is not very readable. Furthermore, the +statistical part of the server does not take care of invalid input. +Among others, using negative variances will cause invalid results. + +@node MAZE, MOBAGWHO, STATIST, Some Applications and Techniques +@section MAZE: Walking Through a Maze In Virtual Reality +@cindex MAZE +@cindex VRML +@c VRML in iX 11/96 134. +@quotation +@cindex Perlis, Alan +@i{In the long run, every program becomes rococo, and then rubble.}@* +Alan Perlis +@end quotation + +By now, we know how to present arbitrary @samp{Content-type}s to a browser. +In this @value{SECTION}, our server will present a 3D world to our browser. +The 3D world is described in a scene description language (VRML, +Virtual Reality Modeling Language) that allows us to travel through a +perspective view of a 2D maze with our browser. Browsers with a +VRML plugin enable exploration of this technology. We could do +one of those boring @samp{Hello world} examples here, that are usually +presented when introducing novices to +VRML. If you have never written +any VRML code, have a look at +the VRML FAQ. +Presenting a static VRML scene is a bit trivial; in order to expose +@command{gawk}'s new capabilities, we will present a dynamically generated +VRML scene. The function @code{SetUpServer} is very simple because it +only sets the default HTML page and initializes the random number +generator. As usual, the surrounding server lets you browse the maze. + +@smallexample +@c file eg/network/maze.awk +function SetUpServer() @{ + TopHeader = "<HTML><title>Walk through a maze</title>" + TopDoc = "\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI><A HREF=" MyPrefix "/AboutServer>About this server</A>\ + <LI><A HREF=" MyPrefix "/VRMLtest>Watch a simple VRML scene</A>\ + </UL>" + TopFooter = "</HTML>" + srand() +@} +@c endfile +@end smallexample + +The function @code{HandleGET} is a bit longer because it first computes +the maze and afterwards generates the VRML code that is sent across +the network. As shown in the STATIST example +(@pxref{STATIST}), +we set the type of the +content to VRML and then store the VRML representation of the maze as the +page content. We assume that the maze is stored in a 2D array. Initially, +the maze consists of walls only. Then, we add an entry and an exit to the +maze and let the rest of the work be done by the function @code{MakeMaze}. +Now, only the wall fields are left in the maze. By iterating over the these +fields, we generate one line of VRML code for each wall field. + +@smallexample +@c file eg/network/maze.awk +function HandleGET() @{ + if (MENU[2] == "AboutServer") @{ + Document = "If your browser has a VRML 2 plugin,\ + this server shows you a simple VRML scene." + @} else if (MENU[2] == "VRMLtest") @{ + XSIZE = YSIZE = 11 # initially, everything is wall + for (y = 0; y < YSIZE; y++) + for (x = 0; x < XSIZE; x++) + Maze[x, y] = "#" + delete Maze[0, 1] # entry is not wall + delete Maze[XSIZE-1, YSIZE-2] # exit is not wall + MakeMaze(1, 1) + Document = "\ +#VRML V2.0 utf8\n\ +Group @{\n\ + children [\n\ + PointLight @{\n\ + ambientIntensity 0.2\n\ + color 0.7 0.7 0.7\n\ + location 0.0 8.0 10.0\n\ + @}\n\ + DEF B1 Background @{\n\ + skyColor [0 0 0, 1.0 1.0 1.0 ]\n\ + skyAngle 1.6\n\ + groundColor [1 1 1, 0.8 0.8 0.8, 0.2 0.2 0.2 ]\n\ + groundAngle [ 1.2 1.57 ]\n\ + @}\n\ + DEF Wall Shape @{\n\ + geometry Box @{size 1 1 1@}\n\ + appearance Appearance @{ material Material @{ diffuseColor 0 0 1 @} @}\n\ + @}\n\ + DEF Entry Viewpoint @{\n\ + position 0.5 1.0 5.0\n\ + orientation 0.0 0.0 -1.0 0.52\n\ + @}\n" + for (i in Maze) @{ + split(i, t, SUBSEP) + Document = Document " Transform @{ translation " + Document = Document t[1] " 0 -" t[2] " children USE Wall @}\n" + @} + Document = Document " ] # end of group for world\n@}" + Reason = "OK" ORS "Content-type: model/vrml" + Header = Footer = "" + @} +@} +@c endfile +@end smallexample + +Finally, we have a look at @code{MakeMaze}, the function that generates +the @code{Maze} array. When entered, this function assumes that the array +has been initialized so that each element represents a wall element and +the maze is initially full of wall elements. Only the entrance and the exit +of the maze should have been left free. The parameters of the function tell +us which element must be marked as not being a wall. After this, we take +a look at the four neighbouring elements and remember which we have already +treated. Of all the neighbouring elements, we take one at random and +walk in that direction. Therefore, the wall element in that direction has +to be removed and then, we call the function recursively for that element. +The maze is only completed if we iterate the above procedure for +@emph{all} neighbouring elements (in random order) and for our present +element by recursively calling the function for the present element. This +last iteration could have been done in a loop, +but it is done much simpler recursively. + +Notice that elements with coordinates that are both odd are assumed to be +on our way through the maze and the generating process cannot terminate +as long as there is such an element not being @code{delete}d. All other +elements are potentially part of the wall. + +@smallexample +@c file eg/network/maze.awk +function MakeMaze(x, y) @{ + delete Maze[x, y] # here we are, we have no wall here + p = 0 # count unvisited fields in all directions + if (x-2 SUBSEP y in Maze) d[p++] = "-x" + if (x SUBSEP y-2 in Maze) d[p++] = "-y" + if (x+2 SUBSEP y in Maze) d[p++] = "+x" + if (x SUBSEP y+2 in Maze) d[p++] = "+y" + if (p>0) @{ # if there are univisited fields, go there + p = int(p*rand()) # choose one unvisited field at random + if (d[p] == "-x") @{ delete Maze[x - 1, y]; MakeMaze(x - 2, y) + @} else if (d[p] == "-y") @{ delete Maze[x, y - 1]; MakeMaze(x, y - 2) + @} else if (d[p] == "+x") @{ delete Maze[x + 1, y]; MakeMaze(x + 2, y) + @} else if (d[p] == "+y") @{ delete Maze[x, y + 1]; MakeMaze(x, y + 2) + @} # we are back from recursion + MakeMaze(x, y); # try again while there are unvisited fields + @} +@} +@c endfile +@end smallexample + +@node MOBAGWHO, STOXPRED, MAZE, Some Applications and Techniques +@section MOBAGWHO: a Simple Mobile Agent +@cindex MOBAGWHO program +@cindex agent +@quotation +@cindex Hoare, C.A.R. +@i{There are two ways of constructing a software design: One way is to +make it so simple that there are obviously no deficiencies, and the +other way is to make it so complicated that there are no obvious +deficiencies.} @* +C. A. R. Hoare +@end quotation + +A @dfn{mobile agent} is a program that can be dispatched from a computer and +transported to a remote server for execution. This is called @dfn{migration}, +which means that a process on another system is started that is independent +from its originator. Ideally, it wanders through +a network while working for its creator or owner. In places like +the UMBC Agent Web, +people are quite confident that (mobile) agents are a software engineering +paradigm that enables us to significantly increase the efficiency +of our work. Mobile agents could become the mediators between users and +the networking world. For an unbiased view at this technology, +see the remarkable paper @cite{Mobile Agents: Are they a good +idea?}.@footnote{@uref{http://www.research.ibm.com/massive/mobag.ps}} + +@ignore +@c Chuck says to take all of this out. +@cindex Tcl/Tk +A good instance of this paradigm is +@cite{Agent Tcl},@footnote{@uref{http://agent.cs.dartmouth.edu/software/agent2.0/}} +an extension of the Tcl language. After introducing a typical +development environment, the aforementioned paper shows a nice little +example application that we will try to rebuild in @command{gawk}. The +@command{who} agent takes a list of servers and wanders from one server +to the next one, always looking to see who is logged in. +Having reached the last +one, it sends back a message with a list of all users it found on each +machine. + +But before implementing something that might or might not be a mobile +agent, let us clarify the concept and some important terms. The agent +paradigm in general is such a young scientific discipline that it has +not yet developed a widely-accepted terminology. Some authors try to +give precise definitions, but their scope is often not wide enough +to be generally accepted. Franklin and Graesser ask +@cite{Is it an Agent or just a Program: A Taxonomy for Autonomous +Agents}@footnote{@uref{http://www.msci.memphis.edu/~franklin/AgentProg.html}} +and give even better answers than Caglayan and Harrison in their +@cite{Agent Sourcebook}.@footnote{@uref{http://www.aminda.com/mazzu/sourcebook/}} + +@itemize @minus +@item +@i{An autonomous agent is a system situated within and a part of +an environment that senses that environment and acts on it, over time, in +pursuit of its own agenda and so as to effect what it senses in the future.} +(Quoted from Franklin and Graesser.) +@item +A mobile agent is able to transport itself from one machine to another. +@item +The term @dfn{migration} often denotes this process of moving. +But neither of the two sources above even mentions this term, while others +use it regularly. +@end itemize + +Before delving into the (rather demanding) details of +implementation, let us give just one more quotation as a final +motivation. Steven Farley published an excellent paper called +@cite{Mobile Agent System Architecture},@footnote{This often +cited text originally appeared as a conference paper here: +@uref{http://www.sigs.com/publications/docs/java/9705/farley.html} +Many bibliographies on the Internet point to this dead link. Meanwhile, +the paper appeared as a contribution to a book called More Java Gems here: +@uref{http://uk.cambridge.org/computerscience/object/catalogue/0521774772/default.htm}} +in which he asks ``Why use an agent architecture?'' + +@quotation +If client-server systems are the currently established norm and distributed +object systems such as CORBA are defining the future standards, why bother +with agents? Agent architectures have certain advantages over these other +types. Three of the most important advantages are: +@cindex CORBA + +@enumerate +@item +An agent performs much processing at the server where local bandwidth +is high, thus reducing the amount of network bandwidth consumed and increasing +overall performance. In contrast, a CORBA client object with the equivalent +functionality of a given agent must make repeated remote method calls to +the server object because CORBA objects cannot move across the network +at runtime. + +@item +An agent operates independently of the application from which the +agent was invoked. The agent operates asynchronously, meaning that the +client application does not need to wait for the results. This is especially +important for mobile users who are not always connected to the network. + +@item +The use of agents allows for the injection of new functionality into +a system at run time. An agent system essentially contains its own automatic +software distribution mechanism. Since CORBA has no built-in support for +mobile code, new functionality generally has to be installed manually. + +@end enumerate + +Of course a non-agent system can exhibit these same features with some +work. But the mobile code paradigm supports the transfer of executable +code to a remote location for asynchronous execution from the start. An +agent architecture should be considered for systems where the above features +are primary requirements. +@end quotation +@end ignore + +When trying to migrate a process from one system to another, +a server process is needed on the receiving side. Depending on the kind +of server process, several ways of implementation come to mind. +How the process is implemented depends upon the kind of server process: + +@itemize @bullet +@item +HTTP can be used as the protocol for delivery of the migrating +process. In this case, we use a common web +server as the receiving server process. A universal CGI script +mediates between migrating process and web server. +Each server willing to accept migrating agents makes this universal +service available. HTTP supplies the @code{POST} method to transfer +some data to a file on the web server. When a CGI script is called +remotely with the @code{POST} method instead of the usual @code{GET} method, +data is transmitted from the client process to the standard input +of the server's CGI script. So, to implement a mobile agent, +we must not only write the agent program to start on the client +side, but also the CGI script to receive the agent on the server side. + +@cindex CGI +@cindex apache +@item +The @code{PUT} method can also be used for migration. HTTP does not +require a CGI script for migration via @code{PUT}. However, with common web +servers there is no advantage to this solution, because web servers such as +Apache +require explicit activation of a special @code{PUT} script. + +@item +@cite{Agent Tcl} pursues a different course; it relies on a dedicated server +process with a dedicated protocol specialized for receiving mobile agents. +@end itemize + +Our agent example abuses a common web server as a migration tool. So, it needs a +universal CGI script on the receiving side (the web server). The receiving script is +activated with a @code{POST} request when placed into a location like +@file{/httpd/cgi-bin/PostAgent.sh}. Make sure that the server system uses a +version of @command{gawk} that supports network access (Version 3.1 or later; +verify with @samp{gawk --version}). + +@example +@c file eg/network/PostAgent.sh +#!/bin/sh +MobAg=/tmp/MobileAgent.$$ +# direct script to mobile agent file +cat > $MobAg +# execute agent concurrently +gawk -f $MobAg $MobAg > /dev/null & +# HTTP header, terminator and body +gawk 'BEGIN @{ print "\r\nAgent started" @}' +rm $MobAg # delete script file of agent +@c endfile +@end example + +By making its process id (@code{$$}) part of the unique @value{FN}, the +script avoids conflicts between concurrent instances of the script. +First, all lines +from standard input (the mobile agent's source code) are copied into +this unique file. Then, the agent is started as a concurrent process +and a short message reporting this fact is sent to the submitting client. +Finally, the script file of the mobile agent is removed because it is +no longer needed. Although it is a short script, there are several noteworthy +points: + +@table @asis +@item Security +@emph{There is none}. In fact, the CGI script should never +be made available on a server that is part of the Internet because everyone +would be allowed to execute arbitrary commands with it. This behavior is +acceptable only when performing rapid prototyping. + +@item Self-Reference +Each migrating instance of an agent is started +in a way that enables it to read its own source code from standard input +and use the code for subsequent +migrations. This is necessary because it needs to treat the agent's code +as data to transmit. @command{gawk} is not the ideal language for such +a job. Lisp and Tcl are more suitable because they do not make a distinction +between program code and data. + +@item Independence +After migration, the agent is not linked to its +former home in any way. By reporting @samp{Agent started}, it waves +``Goodbye'' to its origin. The originator may choose to terminate or not. +@end table + +@cindex Lisp +The originating agent itself is started just like any other command-line +script, and reports the results on standard output. By letting the name +of the original host migrate with the agent, the agent that migrates +to a host far away from its origin can report the result back home. +Having arrived at the end of the journey, the agent establishes +a connection and reports the results. This is the reason for +determining the name of the host with @samp{uname -n} and storing it +in @code{MyOrigin} for later use. We may also set variables with the +@option{-v} option from the command line. This interactivity is only +of importance in the context of starting a mobile agent; therefore this +@code{BEGIN} pattern and its action do not take part in migration: + +@smallexample +@c file eg/network/mobag.awk +BEGIN @{ + if (ARGC != 2) @{ + print "MOBAG - a simple mobile agent" + print "CALL:\n gawk -f mobag.awk mobag.awk" + print "IN:\n the name of this script as a command-line parameter" + print "PARAM:\n -v MyOrigin=myhost.com" + print "OUT:\n the result on stdout" + print "JK 29.03.1998 01.04.1998" + exit + @} + if (MyOrigin == "") @{ + "uname -n" | getline MyOrigin + close("uname -n") + @} +@} +@c endfile +@end smallexample + +Since @command{gawk} cannot manipulate and transmit parts of the program +directly, the source code is read and stored in strings. +Therefore, the program scans itself for +the beginning and the ending of functions. +Each line in between is appended to the code string until the end of +the function has been reached. A special case is this part of the program +itself. It is not a function. +Placing a similar framework around it causes it to be treated +like a function. Notice that this mechanism works for all the +functions of the source code, but it cannot guarantee that the order +of the functions is preserved during migration: + +@smallexample +@c file eg/network/mobag.awk +#ReadMySelf +/^function / @{ FUNC = $2 @} +/^END/ || /^#ReadMySelf/ @{ FUNC = $1 @} +FUNC != "" @{ MOBFUN[FUNC] = MOBFUN[FUNC] RS $0 @} +(FUNC != "") && (/^@}/ || /^#EndOfMySelf/) \ + @{ FUNC = "" @} +#EndOfMySelf +@c endfile +@end smallexample + +The web server code in +@ref{Interacting Service, ,A Web Service with Interaction}, +was first developed as a site-independent core. Likewise, the +@command{gawk}-based mobile agent +starts with an agent-independent core, to which can be appended +application-dependent functions. What follows is the only +application-independent function needed for the mobile agent: + +@smallexample +@c file eg/network/mobag.awk +function migrate(Destination, MobCode, Label) @{ + MOBVAR["Label"] = Label + MOBVAR["Destination"] = Destination + RS = ORS = "\r\n" + HttpService = "/inet/tcp/0/" Destination + for (i in MOBFUN) + MobCode = (MobCode "\n" MOBFUN[i]) + MobCode = MobCode "\n\nBEGIN @{" + for (i in MOBVAR) + MobCode = (MobCode "\n MOBVAR[\"" i "\"] = \"" MOBVAR[i] "\"") + MobCode = MobCode "\n@}\n" + print "POST /cgi-bin/PostAgent.sh HTTP/1.0" |& HttpService + print "Content-length:", length(MobCode) ORS |& HttpService + printf "%s", MobCode |& HttpService + while ((HttpService |& getline) > 0) + print $0 + close(HttpService) +@} +@c endfile +@end smallexample + +The @code{migrate} function prepares the +aforementioned strings containing the program code and transmits them to a +server. A consequence of this modular approach is that the @code{migrate} +function takes some parameters that aren't needed in this application, +but that will be in future ones. Its mandatory parameter @code{Destination} holds the +name (or IP address) of the server that the agent wants as a host for its +code. The optional parameter @code{MobCode} may contain some @command{gawk} +code that is inserted during migration in front of all other code. +The optional parameter @code{Label} may contain +a string that tells the agent what to do in program execution after +arrival at its new home site. One of the serious obstacles in implementing +a framework for mobile agents is that it does not suffice to migrate the +code. It is also necessary to migrate the state of execution of the agent. In +contrast to @cite{Agent Tcl}, this program does not try to migrate the complete set +of variables. The following conventions are used: + +@itemize @bullet +@item +Each variable in an agent program is local to the current host and does +@emph{not} migrate. + +@item +The array @code{MOBFUN} shown above is an exception. It is handled +by the function @code{migrate} and does migrate with the application. + +@item +The other exception is the array @code{MOBVAR}. Each variable that +takes part in migration has to be an element of this array. +@code{migrate} also takes care of this. +@end itemize + +Now it's clear what happens to the @code{Label} parameter of the +function @code{migrate}. It is copied into @code{MOBVAR["Label"]} and +travels alongside the other data. Since travelling takes place via HTTP, +records must be separated with @code{"\r\n"} in @code{RS} and +@code{ORS} as usual. The code assembly for migration takes place in +three steps: + +@itemize @bullet +@item +Iterate over @code{MOBFUN} to collect all functions verbatim. + +@item +Prepare a @code{BEGIN} pattern and put assignments to mobile +variables into the action part. + +@item +Transmission itself resembles GETURL: the header with the request +and the @code{Content-length} is followed by the body. In case there is +any reply over the network, it is read completely and echoed to +standard output to avoid irritating the server. +@end itemize + +The application-independent framework is now almost complete. What follows +is the @code{END} pattern that is executed when the mobile agent has +finished reading its own code. First, it checks whether it is already +running on a remote host or not. In case initialization has not yet taken +place, it starts @code{MyInit}. Otherwise (later, on a remote host), it +starts @code{MyJob}: + +@smallexample +@c file eg/network/mobag.awk +END @{ + if (ARGC != 2) exit # stop when called with wrong parameters + if (MyOrigin != "") # is this the originating host? + MyInit() # if so, initialize the application + else # we are on a host with migrated data + MyJob() # so we do our job +@} +@c endfile +@end smallexample + +All that's left to extend the framework into a complete application +is to write two application-specific functions: @code{MyInit} and +@code{MyJob}. Keep in mind that the former is executed once on the +originating host, while the latter is executed after each migration: + +@smallexample +@c file eg/network/mobag.awk +function MyInit() @{ + MOBVAR["MyOrigin"] = MyOrigin + MOBVAR["Machines"] = "localhost/80 max/80 moritz/80 castor/80" + split(MOBVAR["Machines"], Machines) # which host is the first? + migrate(Machines[1], "", "") # go to the first host + while (("/inet/tcp/8080/0/0" |& getline) > 0) # wait for result + print $0 # print result + close("/inet/tcp/8080/0/0") +@} +@c endfile +@end smallexample + +As mentioned earlier, this agent takes the name of its origin +(@code{MyOrigin}) with it. Then, it takes the name of its first +destination and goes there for further work. Notice that this name has +the port number of the web server appended to the name of the server, +because the function @code{migrate} needs it this way to create +the @code{HttpService} variable. Finally, it waits for the result to arrive. +The @code{MyJob} function runs on the remote host: + +@smallexample +@c file eg/network/mobag.awk +function MyJob() @{ + # forget this host + sub(MOBVAR["Destination"], "", MOBVAR["Machines"]) + MOBVAR["Result"]=MOBVAR["Result"] SUBSEP SUBSEP MOBVAR["Destination"] ":" + while (("who" | getline) > 0) # who is logged in? + MOBVAR["Result"] = MOBVAR["Result"] SUBSEP $0 + close("who") + if (index(MOBVAR["Machines"], "/") > 0) @{ # any more machines to visit? + split(MOBVAR["Machines"], Machines) # which host is next? + migrate(Machines[1], "", "") # go there + @} else @{ # no more machines + gsub(SUBSEP, "\n", MOBVAR["Result"]) # send result to origin + print MOBVAR["Result"] |& "/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080" + close("/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080") + @} +@} +@c endfile +@end smallexample + +After migrating, the first thing to do in @code{MyJob} is to delete +the name of the current host from the list of hosts to visit. Now, it +is time to start the real work by appending the host's name to the +result string, and reading line by line who is logged in on this host. +A very annoying circumstance is the fact that the elements of +@code{MOBVAR} cannot hold the newline character (@code{"\n"}). If they +did, migration of this string did not work because the string didn't +obey the syntax rule for a string in @command{gawk}. +@code{SUBSEP} is used as a temporary replacement. +If the list of hosts to visit holds +at least one more entry, the agent migrates to that place to go on +working there. Otherwise, we replace the @code{SUBSEP}s +with a newline character in the resulting string, and report it to +the originating host, whose name is stored in @code{MOBVAR["MyOrigin"]}. + +@node STOXPRED, PROTBASE, MOBAGWHO, Some Applications and Techniques +@section STOXPRED: Stock Market Prediction As A Service +@cindex STOXPRED program +@cindex Yahoo +@quotation +@i{Far out in the uncharted backwaters of the unfashionable end of +the Western Spiral arm of the Galaxy lies a small unregarded yellow sun.} + +@i{Orbiting this at a distance of roughly ninety-two million miles is an +utterly insignificant little blue-green planet whose ape-descendent life +forms are so amazingly primitive that they still think digital watches are +a pretty neat idea.} + +@i{This planet has --- or rather had --- a problem, which was this: +most of the people living on it were unhappy for pretty much of the time. +Many solutions were suggested for this problem, but most of these were +largely concerned with the movements of small green pieces of paper, +which is odd because it wasn't the small green pieces of paper that +were unhappy.} @* +Douglas Adams, @cite{The Hitch Hiker's Guide to the Galaxy} +@end quotation + +@cindex @command{cron} +Valuable services on the Internet are usually @emph{not} implemented +as mobile agents. There are much simpler ways of implementing services. +All Unix systems provide, for example, the @command{cron} service. +Unix system users can write a list of tasks to be done each day, each +week, twice a day, or just once. The list is entered into a file named +@file{crontab}. For example, to distribute a newsletter on a daily +basis this way, use @command{cron} for calling a script each day early +in the morning. + +@example +# run at 8 am on weekdays, distribute the newsletter +0 8 * * 1-5 $HOME/bin/daily.job >> $HOME/log/newsletter 2>&1 +@end example + +The script first looks for interesting information on the Internet, +assembles it in a nice form and sends the results via email to +the customers. + +The following is an example of a primitive +newsletter on stock market prediction. It is a report which first +tries to predict the change of each share in the Dow Jones Industrial +Index for the particular day. Then it mentions some especially +promising shares as well as some shares which look remarkably bad +on that day. The report ends with the usual disclaimer which tells +every child @emph{not} to try this at home and hurt anybody. +@cindex Dow Jones Industrial Index + +@smallexample +Good morning Uncle Scrooge, + +This is your daily stock market report for Monday, October 16, 2000. +Here are the predictions for today: + + AA neutral + GE up + JNJ down + MSFT neutral + @dots{} + UTX up + DD down + IBM up + MO down + WMT up + DIS up + INTC up + MRK down + XOM down + EK down + IP down + +The most promising shares for today are these: + + INTC http://biz.yahoo.com/n/i/intc.html + +The stock shares to avoid today are these: + + EK http://biz.yahoo.com/n/e/ek.html + IP http://biz.yahoo.com/n/i/ip.html + DD http://biz.yahoo.com/n/d/dd.html + @dots{} +@end smallexample + +@ignore +@c Chuck suggests removing this paragraph +If you are not into stock market prediction but want to earn money +with a more humane service, you might prefer to send out horoscopes +to your customers. Or, once every refrigerator in every household on this side +of the Chinese Wall is connected to the Internet, such a service could +inspect the contents of your customer's refrigerators each day and +advise them on nutrition. Big Brother is watching them. +@end ignore + +The script as a whole is rather long. In order to ease the pain of +studying other people's source code, we have broken the script +up into meaningful parts which are invoked one after the other. +The basic structure of the script is as follows: + +@example +@c file eg/network/stoxpred.awk +BEGIN @{ + Init() + ReadQuotes() + CleanUp() + Prediction() + Report() + SendMail() +@} +@c endfile +@end example + +The earlier parts store data into variables and arrays which are +subsequently used by later parts of the script. The @code{Init} function +first checks if the script is invoked correctly (without any parameters). +If not, it informs the user of the correct usage. What follows are preparations +for the retrieval of the historical quote data. The names of the 30 stock +shares are stored in an array @code{name} along with the current date +in @code{day}, @code{month}, and @code{year}. + +All users who are separated +from the Internet by a firewall and have to direct their Internet accesses +to a proxy must supply the name of the proxy to this script with the +@samp{-v Proxy=@var{name}} option. For most users, the default proxy and +port number should suffice. + +@example +@c file eg/network/stoxpred.awk +function Init() @{ + if (ARGC != 1) @{ + print "STOXPRED - daily stock share prediction" + print "IN:\n no parameters, nothing on stdin" + print "PARAM:\n -v Proxy=MyProxy -v ProxyPort=80" + print "OUT:\n commented predictions as email" + print "JK 09.10.2000" + exit + @} + # Remember ticker symbols from Dow Jones Industrial Index + StockCount = split("AA GE JNJ MSFT AXP GM JPM PG BA HD KO \ + SBC C HON MCD T CAT HWP MMM UTX DD IBM MO WMT DIS INTC \ + MRK XOM EK IP", name); + # Remember the current date as the end of the time series + day = strftime("%d") + month = strftime("%m") + year = strftime("%Y") + if (Proxy == "") Proxy = "chart.yahoo.com" + if (ProxyPort == 0) ProxyPort = 80 + YahooData = "/inet/tcp/0/" Proxy "/" ProxyPort +@} +@c endfile +@end example + +@cindex CSV format +There are two really interesting parts in the script. One is the +function which reads the historical stock quotes from an Internet +server. The other is the one that does the actual prediction. In +the following function we see how the quotes are read from the +Yahoo server. The data which comes from the server is in +CSV format (comma-separated values): + +@example +@c file eg/network/stoxdata.txt +Date,Open,High,Low,Close,Volume +9-Oct-00,22.75,22.75,21.375,22.375,7888500 +6-Oct-00,23.8125,24.9375,21.5625,22,10701100 +5-Oct-00,24.4375,24.625,23.125,23.50,5810300 +@c endfile +@end example + +Lines contain values of the same time instant, whereas columns are +separated by commas and contain the kind of data that is described +in the header (first) line. At first, @command{gawk} is instructed to +separate columns by commas (@samp{FS = ","}). In the loop that follows, +a connection to the Yahoo server is first opened, then a download takes +place, and finally the connection is closed. All this happens once for +each ticker symbol. In the body of this loop, an Internet address is +built up as a string according to the rules of the Yahoo server. The +starting and ending date are chosen to be exactly the same, but one year +apart in the past. All the action is initiated within the @code{printf} +command which transmits the request for data to the Yahoo server. + +In the inner loop, the server's data is first read and then scanned +line by line. Only lines which have six columns and the name of a month +in the first column contain relevant data. This data is stored +in the two-dimensional array @code{quote}; one dimension +being time, the other being the ticker symbol. During retrieval of the +first stock's data, the calendar names of the time instances are stored +in the array @code{day} because we need them later. + +@smallexample +@c file eg/network/stoxpred.awk +function ReadQuotes() @{ + # Retrieve historical data for each ticker symbol + FS = "," + for (stock = 1; stock <= StockCount; stock++) @{ + URL = "http://chart.yahoo.com/table.csv?s=" name[stock] \ + "&a=" month "&b=" day "&c=" year-1 \ + "&d=" month "&e=" day "&f=" year \ + "g=d&q=q&y=0&z=" name[stock] "&x=.csv" + printf("GET " URL " HTTP/1.0\r\n\r\n") |& YahooData + while ((YahooData |& getline) > 0) @{ + if (NF == 6 && $1 ~ /Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec/) @{ + if (stock == 1) + days[++daycount] = $1; + quote[$1, stock] = $5 + @} + @} + close(YahooData) + @} + FS = " " +@} +@c endfile +@end smallexample + +Now that we @emph{have} the data, it can be checked once again to make sure +that no individual stock is missing or invalid, and that all the stock quotes are +aligned correctly. Furthermore, we renumber the time instances. The +most recent day gets day number 1 and all other days get consecutive +numbers. All quotes are rounded toward the nearest whole number in US Dollars. + +@smallexample +@c file eg/network/stoxpred.awk +function CleanUp() @{ + # clean up time series; eliminate incomplete data sets + for (d = 1; d <= daycount; d++) @{ + for (stock = 1; stock <= StockCount; stock++) + if (! ((days[d], stock) in quote)) + stock = StockCount + 10 + if (stock > StockCount + 1) + continue + datacount++ + for (stock = 1; stock <= StockCount; stock++) + data[datacount, stock] = int(0.5 + quote[days[d], stock]) + @} + delete quote + delete days +@} +@c endfile +@end smallexample + +Now we have arrived at the second really interesting part of the whole affair. +What we present here is a very primitive prediction algorithm: +@emph{If a stock fell yesterday, assume it will also fall today; if +it rose yesterday, assume it will rise today}. (Feel free to replace this +algorithm with a smarter one.) If a stock changed in the same direction +on two consecutive days, this is an indication which should be highlighted. +Two-day advances are stored in @code{hot} and two-day declines in +@code{avoid}. + +The rest of the function is a sanity check. It counts the number of +correct predictions in relation to the total number of predictions +one could have made in the year before. + +@smallexample +@c file eg/network/stoxpred.awk +function Prediction() @{ + # Predict each ticker symbol by prolonging yesterday's trend + for (stock = 1; stock <= StockCount; stock++) @{ + if (data[1, stock] > data[2, stock]) @{ + predict[stock] = "up" + @} else if (data[1, stock] < data[2, stock]) @{ + predict[stock] = "down" + @} else @{ + predict[stock] = "neutral" + @} + if ((data[1, stock] > data[2, stock]) && (data[2, stock] > data[3, stock])) + hot[stock] = 1 + if ((data[1, stock] < data[2, stock]) && (data[2, stock] < data[3, stock])) + avoid[stock] = 1 + @} + # Do a plausibility check: how many predictions proved correct? + for (s = 1; s <= StockCount; s++) @{ + for (d = 1; d <= datacount-2; d++) @{ + if (data[d+1, s] > data[d+2, s]) @{ + UpCount++ + @} else if (data[d+1, s] < data[d+2, s]) @{ + DownCount++ + @} else @{ + NeutralCount++ + @} + if (((data[d, s] > data[d+1, s]) && (data[d+1, s] > data[d+2, s])) || + ((data[d, s] < data[d+1, s]) && (data[d+1, s] < data[d+2, s])) || + ((data[d, s] == data[d+1, s]) && (data[d+1, s] == data[d+2, s]))) + CorrectCount++ + @} + @} +@} +@c endfile +@end smallexample + +At this point the hard work has been done: the array @code{predict} +contains the predictions for all the ticker symbols. It is up to the +function @code{Report} to find some nice words to introduce the +desired information. + +@smallexample +@c file eg/network/stoxpred.awk +function Report() @{ + # Generate report + report = "\nThis is your daily " + report = report "stock market report for "strftime("%A, %B %d, %Y")".\n" + report = report "Here are the predictions for today:\n\n" + for (stock = 1; stock <= StockCount; stock++) + report = report "\t" name[stock] "\t" predict[stock] "\n" + for (stock in hot) @{ + if (HotCount++ == 0) + report = report "\nThe most promising shares for today are these:\n\n" + report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \ + tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n" + @} + for (stock in avoid) @{ + if (AvoidCount++ == 0) + report = report "\nThe stock shares to avoid today are these:\n\n" + report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \ + tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n" + @} + report = report "\nThis sums up to " HotCount+0 " winners and " AvoidCount+0 + report = report " losers. When using this kind\nof prediction scheme for" + report = report " the 12 months which lie behind us,\nwe get " UpCount + report = report " 'ups' and " DownCount " 'downs' and " NeutralCount + report = report " 'neutrals'. Of all\nthese " UpCount+DownCount+NeutralCount + report = report " predictions " CorrectCount " proved correct next day.\n" + report = report "A success rate of "\ + int(100*CorrectCount/(UpCount+DownCount+NeutralCount)) "%.\n" + report = report "Random choice would have produced a 33% success rate.\n" + report = report "Disclaimer: Like every other prediction of the stock\n" + report = report "market, this report is, of course, complete nonsense.\n" + report = report "If you are stupid enough to believe these predictions\n" + report = report "you should visit a doctor who can treat your ailment." +@} +@c endfile +@end smallexample + +The function @code{SendMail} goes through the list of customers and opens +a pipe to the @code{mail} command for each of them. Each one receives an +email message with a proper subject heading and is addressed with his full name. + +@smallexample +@c file eg/network/stoxpred.awk +function SendMail() @{ + # send report to customers + customer["uncle.scrooge@@ducktown.gov"] = "Uncle Scrooge" + customer["more@@utopia.org" ] = "Sir Thomas More" + customer["spinoza@@denhaag.nl" ] = "Baruch de Spinoza" + customer["marx@@highgate.uk" ] = "Karl Marx" + customer["keynes@@the.long.run" ] = "John Maynard Keynes" + customer["bierce@@devil.hell.org" ] = "Ambrose Bierce" + customer["laplace@@paris.fr" ] = "Pierre Simon de Laplace" + for (c in customer) @{ + MailPipe = "mail -s 'Daily Stock Prediction Newsletter'" c + print "Good morning " customer[c] "," | MailPipe + print report "\n.\n" | MailPipe + close(MailPipe) + @} +@} +@c endfile +@end smallexample + +Be patient when running the script by hand. +Retrieving the data for all the ticker symbols and sending the emails +may take several minutes to complete, depending upon network traffic +and the speed of the available Internet link. +The quality of the prediction algorithm is likely to be disappointing. +Try to find a better one. +Should you find one with a success rate of more than 50%, please tell +us about it! It is only for the sake of curiosity, of course. @code{:-)} + +@ignore +@c chuck says to remove this +Let us give you one final indication as to what one can expect from +a prediction of stock data, which is sometimes said to contain much +randomness. One theory says that all relevant information to be taken +into account when estimating the price of a stock is contained in the +stock quotes. Every bit of useful information has influenced the +fair price. Therefore (the theory says) temporary changes (i.e., fluctuations +within a minute) have to be purely random. But what is the cause of +short-term changes in stock prices? + +Stock prices are fixed when supply and demand meet each other. +What people are willing to pay reflects human expectations. +Human expectations are not necessarily random. On the Internet, +you can find an elucidating paper about predictability and human +expectations: +@uref{http://it.ucsd.edu/IT/Newsletter/archives/meir/05meir.html, +@cite{Reflections on ``Universal Prediction of Individual Sequences''}} +The authors (Feder, Merhav, Gutman) introduce the reader to the subject +by telling a thrilling anecdote. +@cindex Shannon, Claude +@quotation +In the early 50's, at Bell Laboratories, David Hagelbarger built a +simple ``mind reading'' machine, whose purpose was to play the ``penny +matching'' game. In this game, a player chooses head or tail, while a +``mind reading'' machine tries to predict and match his choice. +Surprisingly, as Robert Lucky tells in his book ``Silicon Dreams'', +Hagelbarger's simple, 8-state machine, was able to match the ``pennies'' +of its human opponent 5,218 times over the course of 9,795 plays. +Random guessing would lead to such a high success rate with a probability +less than one out of 10 billion! Shannon, who was interested in prediction, +information, and thinking machines, closely followed Hagelbarger's +machine, and eventually built his own stripped-down version of the machine, +having the same states, but one that used a simpler strategy at each state. +As the legend goes, in a duel between the two machines, Shannon's machine +won by a slight margin! No one knows if this was due to a superior algorithm +or just a chance happening associated with the specific sequence at that game. +In any event, the success of both these machines against ``untrained'' human +opponents was explained by the fact that the human opponents cannot draw +completely random +bits. +@end quotation +@end ignore + +@node PROTBASE, , STOXPRED, Some Applications and Techniques +@section PROTBASE: Searching Through A Protein Database +@cindex PROTBASE +@cindex NCBI, National Center for Biotechnology Information +@cindex BLAST, Basic Local Alignment Search Tool +@cindex Hoare, C.A.R. +@quotation +@i{Hoare's Law of Large Problems: Inside every large problem is a small + problem struggling to get out.} +@end quotation + +Yahoo's database of stock market data is just one among the many large +databases on the Internet. Another one is located at NCBI +(National Center for Biotechnology +Information). Established in 1988 as a national resource for molecular +biology information, NCBI creates public databases, conducts research +in computational biology, develops software tools for analyzing genome +data, and disseminates biomedical information. In this section, we +look at one of NCBI's public services, which is called BLAST +(Basic Local Alignment Search Tool). + +You probably know that the information necessary for reproducing living +cells is encoded in the genetic material of the cells. The genetic material +is a very long chain of four base nucleotides. It is the order of +appearance (the sequence) of nucleotides which contains the information +about the substance to be produced. Scientists in biotechnology often +find a specific fragment, determine the nucleotide sequence, and need +to know where the sequence at hand comes from. This is where the large +databases enter the game. At NCBI, databases store the knowledge +about which sequences have ever been found and where they have been found. +When the scientist sends his sequence to the BLAST service, the server +looks for regions of genetic material in its database which +look the most similar to the delivered nucleotide sequence. After a +search time of some seconds or minutes the server sends an answer to +the scientist. In order to make access simple, NCBI chose to offer +their database service through popular Internet protocols. There are +four basic ways to use the so-called BLAST services: + +@itemize @bullet +@item +The easiest way to use BLAST is through the web. Users may simply point +their browsers at the NCBI home page +and link to the BLAST pages. +NCBI provides a stable URL that may be used to perform BLAST searches +without interactive use of a web browser. This is what we will do later +in this section. +A demonstration client +and a @file{README} file demonstrate how to access this URL. + +@item +Currently, +@command{blastcl3} is the standard network BLAST client. +You can download @command{blastcl3} from the +anonymous FTP location. + +@item +BLAST 2.0 can be run locally as a full executable and can be used to run +BLAST searches against private local databases, or downloaded copies of the +NCBI databases. BLAST 2.0 executables may be found on the NCBI +anonymous FTP server. + +@item +The NCBI BLAST Email server is the best option for people without convenient +access to the web. A similarity search can be performed by sending a properly +formatted mail message containing the nucleotide or protein query sequence to +@email{blast@@ncbi.nlm.nih.gov}. The query sequence is compared against the +specified database using the BLAST algorithm and the results are returned in +an email message. For more information on formulating email BLAST searches, +you can send a message consisting of the word ``HELP'' to the same address, +@email{blast@@ncbi.nlm.nih.gov}. +@end itemize + +Our starting point is the demonstration client mentioned in the first option. +The @file{README} file that comes along with the client explains the whole +process in a nutshell. In the rest of this section, we first show +what such requests look like. Then we show how to use @command{gawk} to +implement a client in about 10 lines of code. Finally, we show how to +interpret the result returned from the service. + +Sequences are expected to be represented in the standard +IUB/IUPAC amino acid and nucleic acid codes, +with these exceptions: lower-case letters are accepted and are mapped +into upper-case; a single hyphen or dash can be used to represent a gap +of indeterminate length; and in amino acid sequences, @samp{U} and @samp{*} +are acceptable letters (see below). Before submitting a request, any numerical +digits in the query sequence should either be removed or replaced by +appropriate letter codes (e.g., @samp{N} for unknown nucleic acid residue +or @samp{X} for unknown amino acid residue). +The nucleic acid codes supported are: + +@example +A --> adenosine M --> A C (amino) +C --> cytidine S --> G C (strong) +G --> guanine W --> A T (weak) +T --> thymidine B --> G T C +U --> uridine D --> G A T +R --> G A (purine) H --> A C T +Y --> T C (pyrimidine) V --> G C A +K --> G T (keto) N --> A G C T (any) + - gap of indeterminate length +@end example + +Now you know the alphabet of nucleotide sequences. The last two lines +of the following example query show you such a sequence, which is obviously +made up only of elements of the alphabet just described. Store this example +query into a file named @file{protbase.request}. You are now ready to send +it to the server with the demonstration client. + +@example +@c file eg/network/protbase.request +PROGRAM blastn +DATALIB month +EXPECT 0.75 +BEGIN +>GAWK310 the gawking gene GNU AWK +tgcttggctgaggagccataggacgagagcttcctggtgaagtgtgtttcttgaaatcat +caccaccatggacagcaaa +@c endfile +@end example + +@cindex FASTA/Pearson format +The actual search request begins with the mandatory parameter @samp{PROGRAM} +in the first column followed by the value @samp{blastn} (the name of the +program) for searching nucleic acids. The next line contains the mandatory +search parameter @samp{DATALIB} with the value @samp{month} for the newest +nucleic acid sequences. The third line contains an optional @samp{EXPECT} +parameter and the value desired for it. The fourth line contains the +mandatory @samp{BEGIN} directive, followed by the query sequence in +FASTA/Pearson format. +Each line of information must be less than 80 characters in length. + +The ``month'' database contains all new or revised sequences released in the +last 30 days and is useful for searching against new sequences. +There are five different blast programs, @command{blastn} being the one that +compares a nucleotide query sequence against a nucleotide sequence database. + +The last server directive that must appear in every request is the +@samp{BEGIN} directive. The query sequence should immediately follow the +@samp{BEGIN} directive and must appear in FASTA/Pearson format. +A sequence in +FASTA/Pearson format begins with a single-line description. +The description line, which is required, is distinguished from the lines of +sequence data that follow it by having a greater-than (@samp{>}) symbol +in the first column. For the purposes of the BLAST server, the text of +the description is arbitrary. + +If you prefer to use a client written in @command{gawk}, just store the following +10 lines of code into a file named @file{protbase.awk} and use this client +instead. Invoke it with @samp{gawk -f protbase.awk protbase.request}. +Then wait a minute and watch the result coming in. In order to replicate +the demonstration client's behaviour as closely as possible, this client +does not use a proxy server. We could also have extended the client program +in @ref{GETURL, ,Retrieving Web Pages}, to implement the client request from +@file{protbase.awk} as a special case. + +@smallexample +@c file eg/network/protbase.awk +@{ request = request "\n" $0 @} + +END @{ + BLASTService = "/inet/tcp/0/www.ncbi.nlm.nih.gov/80" + printf "POST /cgi-bin/BLAST/nph-blast_report HTTP/1.0\n" |& BLASTService + printf "Content-Length: " length(request) "\n\n" |& BLASTService + printf request |& BLASTService + while ((BLASTService |& getline) > 0) + print $0 + close(BLASTService) +@} +@c endfile +@end smallexample + +The demonstration client from NCBI is 214 lines long (written in C) and +it is not immediately obvious what it does. Our client is so short that +it @emph{is} obvious what it does. First it loops over all lines of the +query and stores the whole query into a variable. Then the script +establishes an Internet connection to the NCBI server and transmits the +query by framing it with a proper HTTP request. Finally it receives +and prints the complete result coming from the server. + +Now, let us look at the result. It begins with an HTTP header, which you +can ignore. Then there are some comments about the query having been +filtered to avoid spuriously high scores. After this, there is a reference +to the paper that describes the software being used for searching the data +base. After a repitition of the original query's description we find the +list of significant alignments: + +@smallexample +@c file eg/network/protbase.result +Sequences producing significant alignments: (bits) Value + +gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733... 38 0.20 +gb|AC021056.12|AC021056 Homo sapiens chromosome 3 clone RP11-115... 38 0.20 +emb|AL160278.10|AL160278 Homo sapiens chromosome 9 clone RP11-57... 38 0.20 +emb|AL391139.11|AL391139 Homo sapiens chromosome X clone RP11-35... 38 0.20 +emb|AL365192.6|AL365192 Homo sapiens chromosome 6 clone RP3-421H... 38 0.20 +emb|AL138812.9|AL138812 Homo sapiens chromosome 11 clone RP1-276... 38 0.20 +gb|AC073881.3|AC073881 Homo sapiens chromosome 15 clone CTD-2169... 38 0.20 +@c endfile +@end smallexample + +This means that the query sequence was found in seven human chromosomes. +But the value 0.20 (20%) means that the probability of an accidental match +is rather high (20%) in all cases and should be taken into account. +You may wonder what the first column means. It is a key to the specific +database in which this occurence was found. The unique sequence identifiers +reported in the search results can be used as sequence retrieval keys +via the NCBI server. The syntax of sequence header lines used by the NCBI +BLAST server depends on the database from which each sequence was obtained. +The table below lists the identifiers for the databases from which the +sequences were derived. + +@ifinfo +@example +Database Name Identifier Syntax +============================ ======================== +GenBank gb|accession|locus +EMBL Data Library emb|accession|locus +DDBJ, DNA Database of Japan dbj|accession|locus +NBRF PIR pir||entry +Protein Research Foundation prf||name +SWISS-PROT sp|accession|entry name +Brookhaven Protein Data Bank pdb|entry|chain +Kabat's Sequences of Immuno@dots{} gnl|kabat|identifier +Patents pat|country|number +GenInfo Backbone Id bbs|number +@end example +@end ifinfo + +@ifnotinfo +@multitable {Kabat's Sequences of Immuno@dots{}} {@code{@w{sp|accession|entry name}}} +@item GenBank @tab @code{gb|accession|locus} +@item EMBL Data Library @tab @code{emb|accession|locus} +@item DDBJ, DNA Database of Japan @tab @code{dbj|accession|locus} +@item NBRF PIR @tab @code{pir||entry} +@item Protein Research Foundation @tab @code{prf||name} +@item SWISS-PROT @tab @code{@w{sp|accession|entry name}} +@item Brookhaven Protein Data Bank @tab @code{pdb|entry|chain} +@item Kabat's Sequences of Immuno@dots{} @tab @code{gnl|kabat|identifier} +@item Patents @tab @code{pat|country|number} +@item GenInfo Backbone Id @tab @code{bbs|number} +@end multitable +@end ifnotinfo + + +For example, an identifier might be @samp{gb|AC021182.14|AC021182}, where the +@samp{gb} tag indicates that the identifier refers to a GenBank sequence, +@samp{AC021182.14} is its GenBank ACCESSION, and @samp{AC021182} is the GenBank LOCUS. +The identifier contains no spaces, so that a space indicates the end of the +identifier. + +Let us continue in the result listing. Each of the seven alignments mentioned +above is subsequently described in detail. We will have a closer look at +the first of them. + +@smallexample +>gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733N23, WORKING DRAFT SEQUENCE, 4 + unordered pieces + Length = 176383 + + Score = 38.2 bits (19), Expect = 0.20 + Identities = 19/19 (100%) + Strand = Plus / Plus + +Query: 35 tggtgaagtgtgtttcttg 53 + ||||||||||||||||||| +Sbjct: 69786 tggtgaagtgtgtttcttg 69804 +@end smallexample + +This alignment was located on the human chromosome 7. The fragment on which +part of the query was found had a total length of 176383. Only 19 of the +nucleotides matched and the matching sequence ran from character 35 to 53 +in the query sequence and from 69786 to 69804 in the fragment on chromosome 7. +If you are still reading at this point, you are probably interested in finding +out more about Computational Biology and you might appreciate the following +hints. + +@cindex Computational Biology +@cindex Bioinformatics +@enumerate +@item +There is a book called @cite{Introduction to Computational Biology} +by Michael S. Waterman, which is worth reading if you are seriously +interested. You can find a good +book review +on the Internet. + +@item +While Waterman's book can explain to you the algorithms employed internally +in the database search engines, most practicioners prefer to approach +the subject differently. The applied side of Computational Biology is +called Bioinformatics, and emphasizes the tools available for day-to-day +work as well as how to actually @emph{use} them. One of the very few affordable +books on Bioinformatics is +@cite{Developing Bioinformatics Computer Skills}. + +@item +The sequences @emph{gawk} and @emph{gnuawk} are in widespread use in +the genetic material of virtually every earthly living being. Let us +take this as a clear indication that the divine creator has intended +@code{gawk} to prevail over other scripting languages such as @code{perl}, +@code{tcl}, or @code{python} which are not even proper sequences. (:-) +@end enumerate + +@node Links, GNU Free Documentation License, Some Applications and Techniques, Top +@chapter Related Links + +This section lists the URLs for various items discussed in this @value{CHAPTER}. +They are presented in the order in which they appear. + +@table @asis + +@item @cite{Internet Programming with Python} +@uref{http://www.fsbassociates.com/books/python.htm} + +@item @cite{Advanced Perl Programming} +@uref{http://www.oreilly.com/catalog/advperl} + +@item @cite{Web Client Programming with Perl} +@uref{http://www.oreilly.com/catalog/webclient} + +@item Richard Stevens's home page and book +@uref{http://www.kohala.com/~rstevens} + +@item The SPAK home page +@uref{http://www.userfriendly.net/linux/RPM/contrib/libc6/i386/spak-0.6b-1.i386.html} + +@item Volume III of @cite{Internetworking with TCP/IP}, by Comer and Stevens +@uref{http://www.cs.purdue.edu/homes/dec/tcpip3s.cont.html} + +@item XBM Graphics File Format +@uref{http://www.wotsit.org/download.asp?f=xbm} + +@item GNUPlot +@uref{http://www.cs.dartmouth.edu/gnuplot_info.html} + +@item Mark Humphrys' Eliza page +@uref{http://www.compapp.dcu.ie/~humphrys/eliza.html} + +@item Yahoo! Eliza Information +@uref{http://dir.yahoo.com/Recreation/Games/Computer_Games/Internet_Games/Web_Games/Artificial_Intelligence} + +@item Java versions of Eliza +@uref{http://www.tjhsst.edu/Psych/ch1/eliza.html} + +@item Java versions of Eliza with source code +@uref{http://home.adelphia.net/~lifeisgood/eliza/eliza.htm} + +@item Eliza Programs with Explanations +@uref{http://chayden.net/chayden/eliza/Eliza.shtml} + +@item Loebner Contest +@uref{http://acm.org/~loebner/loebner-prize.htmlx} + +@item Tck/Tk Information +@uref{http://www.scriptics.com/} + +@item Intel 80x86 Processors +@uref{http://developer.intel.com/design/platform/embedpc/what_is.htm} + +@item AMD Elan Processors +@uref{http://www.amd.com/products/epd/processors/4.32bitcont/32bitcont/index.html} + +@item XINU +@uref{http://willow.canberra.edu.au/~chrisc/xinu.html } + +@item GNU/Linux +@uref{http://uclinux.lineo.com/} + +@item Embedded PCs +@uref{http://dir.yahoo.com/Business_and_Economy/Business_to_Business/Computers/Hardware/Embedded_Control/} + +@item MiniSQL +@uref{http://www.hughes.com.au/library/} + +@item Market Share Surveys +@uref{http://www.netcraft.com/survey} + +@item @cite{Numerical Recipes in C: The Art of Scientific Computing} +@uref{http://www.nr.com} + +@item VRML +@uref{http://www.vrml.org} + +@item The VRML FAQ +@uref{http://www.vrml.org/technicalinfo/specifications/specifications.htm#FAQ} + +@item The UMBC Agent Web +@uref{http://www.cs.umbc.edu/agents } + +@item Apache Web Server +@uref{http://www.apache.org} + +@item National Center for Biotechnology Information (NCBI) +@uref{http://www.ncbi.nlm.nih.gov} + +@item Basic Local Alignment Search Tool (BLAST) +@uref{http://www.ncbi.nlm.nih.gov/BLAST/blast_overview.html} + +@item NCBI Home Page +@uref{http://www.ncbi.nlm.nih.gov} + +@item BLAST Pages +@uref{http://www.ncbi.nlm.nih.gov/BLAST} + +@item BLAST Demonstration Client +@uref{ftp://ncbi.nlm.nih.gov/blast/blasturl/} + +@item BLAST anonymous FTP location +@uref{ftp://ncbi.nlm.nih.gov/blast/network/netblast/} + +@item BLAST 2.0 Executables +@uref{ftp://ncbi.nlm.nih.gov/blast/executables/} + +@item IUB/IUPAC Amino Acid and Nucleic Acid Codes +@uref{http://www.uthscsa.edu/geninfo/blastmail.html#item6} + +@item FASTA/Pearson Format +@uref{http://www.ncbi.nlm.nih.gov/BLAST/fasta.html} + +@item Fasta/Pearson Sequence in Java +@uref{http://www.kazusa.or.jp/java/codon_table_java/} + +@item Book Review of @cite{Introduction to Computational Biology} +@uref{http://www.acm.org/crossroads/xrds5-1/introcb.html} + +@item @cite{Developing Bioinformatics Computer Skills} +@uref{http://www.oreilly.com/catalog/bioskills/} + +@end table + +@node GNU Free Documentation License, Index, Links, Top +@unnumbered GNU Free Documentation License +@center Version 1.1, March 2000 + +@display +Copyright (C) 2000 Free Software Foundation, Inc. +59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +Everyone is permitted to copy and distribute verbatim copies +of this license document, but changing it is not allowed. +@end display +@sp 1 +@enumerate 0 +@item +PREAMBLE + +The purpose of this License is to make a manual, textbook, or other +written document ``free'' in the sense of freedom: to assure everyone +the effective freedom to copy and redistribute it, with or without +modifying it, either commercially or noncommercially. Secondarily, +this License preserves for the author and publisher a way to get +credit for their work, while not being considered responsible for +modifications made by others. + +This License is a kind of ``copyleft'', which means that derivative +works of the document must themselves be free in the same sense. It +complements the GNU General Public License, which is a copyleft +license designed for free software. + +We have designed this License in order to use it for manuals for free +software, because free software needs free documentation: a free +program should come with manuals providing the same freedoms that the +software does. But this License is not limited to software manuals; +it can be used for any textual work, regardless of subject matter or +whether it is published as a printed book. We recommend this License +principally for works whose purpose is instruction or reference. + +@sp 1 +@item +APPLICABILITY AND DEFINITIONS + +This License applies to any manual or other work that contains a +notice placed by the copyright holder saying it can be distributed +under the terms of this License. The ``Document'', below, refers to any +such manual or work. Any member of the public is a licensee, and is +addressed as ``you''. + +A ``Modified Version'' of the Document means any work containing the +Document or a portion of it, either copied verbatim, or with +modifications and/or translated into another language. + +A ``Secondary Section'' is a named appendix or a front-matter section of +the Document that deals exclusively with the relationship of the +publishers or authors of the Document to the Document's overall subject +(or to related matters) and contains nothing that could fall directly +within that overall subject. (For example, if the Document is in part a +textbook of mathematics, a Secondary Section may not explain any +mathematics.) The relationship could be a matter of historical +connection with the subject or with related matters, or of legal, +commercial, philosophical, ethical or political position regarding +them. + +The ``Invariant Sections'' are certain Secondary Sections whose titles +are designated, as being those of Invariant Sections, in the notice +that says that the Document is released under this License. + +The ``Cover Texts'' are certain short passages of text that are listed, +as Front-Cover Texts or Back-Cover Texts, in the notice that says that +the Document is released under this License. + +A ``Transparent'' copy of the Document means a machine-readable copy, +represented in a format whose specification is available to the +general public, whose contents can be viewed and edited directly and +straightforwardly with generic text editors or (for images composed of +pixels) generic paint programs or (for drawings) some widely available +drawing editor, and that is suitable for input to text formatters or +for automatic translation to a variety of formats suitable for input +to text formatters. A copy made in an otherwise Transparent file +format whose markup has been designed to thwart or discourage +subsequent modification by readers is not Transparent. A copy that is +not ``Transparent'' is called ``Opaque''. + +Examples of suitable formats for Transparent copies include plain +ASCII without markup, Texinfo input format, LaTeX input format, SGML +or XML using a publicly available DTD, and standard-conforming simple +HTML designed for human modification. Opaque formats include +PostScript, PDF, proprietary formats that can be read and edited only +by proprietary word processors, SGML or XML for which the DTD and/or +processing tools are not generally available, and the +machine-generated HTML produced by some word processors for output +purposes only. + +The ``Title Page'' means, for a printed book, the title page itself, +plus such following pages as are needed to hold, legibly, the material +this License requires to appear in the title page. For works in +formats which do not have any title page as such, ``Title Page'' means +the text near the most prominent appearance of the work's title, +preceding the beginning of the body of the text. +@sp 1 +@item +VERBATIM COPYING + +You may copy and distribute the Document in any medium, either +commercially or noncommercially, provided that this License, the +copyright notices, and the license notice saying this License applies +to the Document are reproduced in all copies, and that you add no other +conditions whatsoever to those of this License. You may not use +technical measures to obstruct or control the reading or further +copying of the copies you make or distribute. However, you may accept +compensation in exchange for copies. If you distribute a large enough +number of copies you must also follow the conditions in section 3. + +You may also lend copies, under the same conditions stated above, and +you may publicly display copies. +@sp 1 +@item +COPYING IN QUANTITY + +If you publish printed copies of the Document numbering more than 100, +and the Document's license notice requires Cover Texts, you must enclose +the copies in covers that carry, clearly and legibly, all these Cover +Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on +the back cover. Both covers must also clearly and legibly identify +you as the publisher of these copies. The front cover must present +the full title with all words of the title equally prominent and +visible. You may add other material on the covers in addition. +Copying with changes limited to the covers, as long as they preserve +the title of the Document and satisfy these conditions, can be treated +as verbatim copying in other respects. + +If the required texts for either cover are too voluminous to fit +legibly, you should put the first ones listed (as many as fit +reasonably) on the actual cover, and continue the rest onto adjacent +pages. + +If you publish or distribute Opaque copies of the Document numbering +more than 100, you must either include a machine-readable Transparent +copy along with each Opaque copy, or state in or with each Opaque copy +a publicly-accessible computer-network location containing a complete +Transparent copy of the Document, free of added material, which the +general network-using public has access to download anonymously at no +charge using public-standard network protocols. If you use the latter +option, you must take reasonably prudent steps, when you begin +distribution of Opaque copies in quantity, to ensure that this +Transparent copy will remain thus accessible at the stated location +until at least one year after the last time you distribute an Opaque +copy (directly or through your agents or retailers) of that edition to +the public. + +It is requested, but not required, that you contact the authors of the +Document well before redistributing any large number of copies, to give +them a chance to provide you with an updated version of the Document. +@sp 1 +@item +MODIFICATIONS + +You may copy and distribute a Modified Version of the Document under +the conditions of sections 2 and 3 above, provided that you release +the Modified Version under precisely this License, with the Modified +Version filling the role of the Document, thus licensing distribution +and modification of the Modified Version to whoever possesses a copy +of it. In addition, you must do these things in the Modified Version: + +@enumerate A +@item +Use in the Title Page (and on the covers, if any) a title distinct +from that of the Document, and from those of previous versions +(which should, if there were any, be listed in the History section +of the Document). You may use the same title as a previous version +if the original publisher of that version gives permission. + +@item +List on the Title Page, as authors, one or more persons or entities +responsible for authorship of the modifications in the Modified +Version, together with at least five of the principal authors of the +Document (all of its principal authors, if it has less than five). + +@item +State on the Title page the name of the publisher of the +Modified Version, as the publisher. + +@item +Preserve all the copyright notices of the Document. + +@item +Add an appropriate copyright notice for your modifications +adjacent to the other copyright notices. + +@item +Include, immediately after the copyright notices, a license notice +giving the public permission to use the Modified Version under the +terms of this License, in the form shown in the Addendum below. + +@item +Preserve in that license notice the full lists of Invariant Sections +and required Cover Texts given in the Document's license notice. + +@item +Include an unaltered copy of this License. + +@item +Preserve the section entitled ``History'', and its title, and add to +it an item stating at least the title, year, new authors, and +publisher of the Modified Version as given on the Title Page. If +there is no section entitled ``History'' in the Document, create one +stating the title, year, authors, and publisher of the Document as +given on its Title Page, then add an item describing the Modified +Version as stated in the previous sentence. + +@item +Preserve the network location, if any, given in the Document for +public access to a Transparent copy of the Document, and likewise +the network locations given in the Document for previous versions +it was based on. These may be placed in the ``History'' section. +You may omit a network location for a work that was published at +least four years before the Document itself, or if the original +publisher of the version it refers to gives permission. + +@item +In any section entitled ``Acknowledgements'' or ``Dedications'', +preserve the section's title, and preserve in the section all the +substance and tone of each of the contributor acknowledgements +and/or dedications given therein. + +@item +Preserve all the Invariant Sections of the Document, +unaltered in their text and in their titles. Section numbers +or the equivalent are not considered part of the section titles. + +@item +Delete any section entitled ``Endorsements''. Such a section +may not be included in the Modified Version. + +@item +Do not retitle any existing section as ``Endorsements'' +or to conflict in title with any Invariant Section. +@end enumerate + +If the Modified Version includes new front-matter sections or +appendices that qualify as Secondary Sections and contain no material +copied from the Document, you may at your option designate some or all +of these sections as invariant. To do this, add their titles to the +list of Invariant Sections in the Modified Version's license notice. +These titles must be distinct from any other section titles. + +You may add a section entitled ``Endorsements'', provided it contains +nothing but endorsements of your Modified Version by various +parties--for example, statements of peer review or that the text has +been approved by an organization as the authoritative definition of a +standard. + +You may add a passage of up to five words as a Front-Cover Text, and a +passage of up to 25 words as a Back-Cover Text, to the end of the list +of Cover Texts in the Modified Version. Only one passage of +Front-Cover Text and one of Back-Cover Text may be added by (or +through arrangements made by) any one entity. If the Document already +includes a cover text for the same cover, previously added by you or +by arrangement made by the same entity you are acting on behalf of, +you may not add another; but you may replace the old one, on explicit +permission from the previous publisher that added the old one. + +The author(s) and publisher(s) of the Document do not by this License +give permission to use their names for publicity for or to assert or +imply endorsement of any Modified Version. +@sp 1 +@item +COMBINING DOCUMENTS + +You may combine the Document with other documents released under this +License, under the terms defined in section 4 above for modified +versions, provided that you include in the combination all of the +Invariant Sections of all of the original documents, unmodified, and +list them all as Invariant Sections of your combined work in its +license notice. + +The combined work need only contain one copy of this License, and +multiple identical Invariant Sections may be replaced with a single +copy. If there are multiple Invariant Sections with the same name but +different contents, make the title of each such section unique by +adding at the end of it, in parentheses, the name of the original +author or publisher of that section if known, or else a unique number. +Make the same adjustment to the section titles in the list of +Invariant Sections in the license notice of the combined work. + +In the combination, you must combine any sections entitled ``History'' +in the various original documents, forming one section entitled +``History''; likewise combine any sections entitled ``Acknowledgements'', +and any sections entitled ``Dedications''. You must delete all sections +entitled ``Endorsements.'' +@sp 1 +@item +COLLECTIONS OF DOCUMENTS + +You may make a collection consisting of the Document and other documents +released under this License, and replace the individual copies of this +License in the various documents with a single copy that is included in +the collection, provided that you follow the rules of this License for +verbatim copying of each of the documents in all other respects. + +You may extract a single document from such a collection, and distribute +it individually under this License, provided you insert a copy of this +License into the extracted document, and follow this License in all +other respects regarding verbatim copying of that document. +@sp 1 +@item +AGGREGATION WITH INDEPENDENT WORKS + +A compilation of the Document or its derivatives with other separate +and independent documents or works, in or on a volume of a storage or +distribution medium, does not as a whole count as a Modified Version +of the Document, provided no compilation copyright is claimed for the +compilation. Such a compilation is called an ``aggregate'', and this +License does not apply to the other self-contained works thus compiled +with the Document, on account of their being thus compiled, if they +are not themselves derivative works of the Document. + +If the Cover Text requirement of section 3 is applicable to these +copies of the Document, then if the Document is less than one quarter +of the entire aggregate, the Document's Cover Texts may be placed on +covers that surround only the Document within the aggregate. +Otherwise they must appear on covers around the whole aggregate. +@sp 1 +@item +TRANSLATION + +Translation is considered a kind of modification, so you may +distribute translations of the Document under the terms of section 4. +Replacing Invariant Sections with translations requires special +permission from their copyright holders, but you may include +translations of some or all Invariant Sections in addition to the +original versions of these Invariant Sections. You may include a +translation of this License provided that you also include the +original English version of this License. In case of a disagreement +between the translation and the original English version of this +License, the original English version will prevail. +@sp 1 +@item +TERMINATION + +You may not copy, modify, sublicense, or distribute the Document except +as expressly provided for under this License. Any other attempt to +copy, modify, sublicense or distribute the Document is void, and will +automatically terminate your rights under this License. However, +parties who have received copies, or rights, from you under this +License will not have their licenses terminated so long as such +parties remain in full compliance. +@sp 1 +@item +FUTURE REVISIONS OF THIS LICENSE + +The Free Software Foundation may publish new, revised versions +of the GNU Free Documentation License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. See +@uref{http://www.gnu.org/copyleft/}. + +Each version of the License is given a distinguishing version number. +If the Document specifies that a particular numbered version of this +License ``or any later version'' applies to it, you have the option of +following the terms and conditions either of that specified version or +of any later version that has been published (not as a draft) by the +Free Software Foundation. If the Document does not specify a version +number of this License, you may choose any version ever published (not +as a draft) by the Free Software Foundation. + +@end enumerate + +@c fakenode --- for prepinfo +@unnumberedsec ADDENDUM: How to use this License for your documents + +To use this License in a document you have written, include a copy of +the License in the document and put the following copyright and +license notices just after the title page: + +@smallexample +@group + + Copyright (C) @var{year} @var{your name}. + Permission is granted to copy, distribute and/or modify this document + under the terms of the GNU Free Documentation License, Version 1.1 + or any later version published by the Free Software Foundation; + with the Invariant Sections being @var{list their titles}, with the + Front-Cover Texts being @var{list}, and with the Back-Cover Texts being @var{list}. + A copy of the license is included in the section entitled ``GNU + Free Documentation License''. +@end group +@end smallexample +If you have no Invariant Sections, write ``with no Invariant Sections'' +instead of saying which ones are invariant. If you have no +Front-Cover Texts, write ``no Front-Cover Texts'' instead of +``Front-Cover Texts being @var{list}''; likewise for Back-Cover Texts. + +If your document contains nontrivial examples of program code, we +recommend releasing these examples in parallel under your choice of +free software license, such as the GNU General Public License, +to permit their use in free software. + +@node Index, , GNU Free Documentation License, Top +@comment node-name, next, previous, up + +@unnumbered Index +@printindex cp +@bye + +Conventions: +1. Functions, built-in or otherwise, do NOT have () after them. +2. Gawk built-in vars and functions are in @code. Also program vars and + functions. +3. HTTP method names are in @code. +4. Protocols such as echo, ftp, etc are in @samp. +5. URLs are in @url. +6. All RFC's in the index. Put a space between `RFC' and the number. |