START-INFO-DIR-ENTRY * Gawk: (gawk). A text scanning and processing language. END-INFO-DIR-ENTRY START-INFO-DIR-ENTRY * awk: (gawk)Invoking gawk. Text scanning and processing. END-INFO-DIR-ENTRY General Introduction ******************** This file documents `awk', a program that you can use to select particular records in a file and perform operations upon them. Copyright (C) 1989, 1991, 1992, 1993, 1996, 1997, 1998, 1999, 2000, 2001, 2002 Free Software Foundation, Inc. This is Edition 3 of `GAWK: Effective AWK Programming: A User's Guide for GNU Awk', for the 3.1.1 (or later) version of the GNU implementation of AWK. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with the Invariant Sections being "GNU General Public License", the Front-Cover texts being (a) (see below), and with the Back-Cover Texts being (b) (see below). A copy of the license is included in the section entitled "GNU Free Documentation License". a. "A GNU Manual" b. "You have freedom to copy and modify this GNU Manual, like GNU software. Copies published by the Free Software Foundation raise funds for GNU development." To Miriam, for making me complete. To Chana, for the joy you bring us. To Rivka, for the exponential increase. To Nachum, for the added dimension. To Malka, for the new beginning. ...Short Contents... ...Table of Contents... Foreword ******** Arnold Robbins and I are good friends. We were introduced 11 years ago by circumstances--and our favorite programming language, AWK. The circumstances started a couple of years earlier. I was working at a new job and noticed an unplugged Unix computer sitting in the corner. No one knew how to use it, and neither did I. However, a couple of days later it was running, and I was `root' and the one-and-only user. That day, I began the transition from statistician to Unix programmer. On one of many trips to the library or bookstore in search of books on Unix, I found the gray AWK book, a.k.a. Aho, Kernighan and Weinberger, `The AWK Programming Language', Addison-Wesley, 1988. AWK's simple programming paradigm--find a pattern in the input and then perform an action--often reduced complex or tedious data manipulations to few lines of code. I was excited to try my hand at programming in AWK. Alas, the `awk' on my computer was a limited version of the language described in the AWK book. I discovered that my computer had "old `awk'" and the AWK book described "new `awk'." I learned that this was typical; the old version refused to step aside or relinquish its name. If a system had a new `awk', it was invariably called `nawk', and few systems had it. The best way to get a new `awk' was to `ftp' the source code for `gawk' from `prep.ai.mit.edu'. `gawk' was a version of new `awk' written by David Trueman and Arnold, and available under the GNU General Public License. (Incidentally, it's no longer difficult to find a new `awk'. `gawk' ships with Linux, and you can download binaries or source code for almost any system; my wife uses `gawk' on her VMS box.) My Unix system started out unplugged from the wall; it certainly was not plugged into a network. So, oblivious to the existence of `gawk' and the Unix community in general, and desiring a new `awk', I wrote my own, called `mawk'. Before I was finished I knew about `gawk', but it was too late to stop, so I eventually posted to a `comp.sources' newsgroup. A few days after my posting, I got a friendly email from Arnold introducing himself. He suggested we share design and algorithms and attached a draft of the POSIX standard so that I could update `mawk' to support language extensions added after publication of the AWK book. Frankly, if our roles had been reversed, I would not have been so open and we probably would have never met. I'm glad we did meet. He is an AWK expert's AWK expert and a genuinely nice person. Arnold contributes significant amounts of his expertise and time to the Free Software Foundation. This book is the `gawk' reference manual, but at its core it is a book about AWK programming that will appeal to a wide audience. It is a definitive reference to the AWK language as defined by the 1987 Bell Labs release and codified in the 1992 POSIX Utilities standard. On the other hand, the novice AWK programmer can study a wealth of practical programs that emphasize the power of AWK's basic idioms: data driven control-flow, pattern matching with regular expressions, and associative arrays. Those looking for something new can try out `gawk''s interface to network protocols via special `/inet' files. The programs in this book make clear that an AWK program is typically much smaller and faster to develop than a counterpart written in C. Consequently, there is often a payoff to prototype an algorithm or design in AWK to get it running quickly and expose problems early. Often, the interpreted performance is adequate and the AWK prototype becomes the product. The new `pgawk' (profiling `gawk'), produces program execution counts. I recently experimented with an algorithm that for n lines of input, exhibited ~ C n^2 performance, while theory predicted ~ C n log n behavior. A few minutes poring over the `awkprof.out' profile pinpointed the problem to a single line of code. `pgawk' is a welcome addition to my programmer's toolbox. Arnold has distilled over a decade of experience writing and using AWK programs, and developing `gawk', into this book. If you use AWK or want to learn how, then read this book. Michael Brennan Author of `mawk' Preface ******* Several kinds of tasks occur repeatedly when working with text files. You might want to extract certain lines and discard the rest. Or you may need to make changes wherever certain patterns appear, but leave the rest of the file alone. Writing single-use programs for these tasks in languages such as C, C++, or Pascal is time-consuming and inconvenient. Such jobs are often easier with `awk'. The `awk' utility interprets a special-purpose programming language that makes it easy to handle simple data-reformatting jobs. The GNU implementation of `awk' is called `gawk'; it is fully compatible with the System V Release 4 version of `awk'. `gawk' is also compatible with the POSIX specification of the `awk' language. This means that all properly written `awk' programs should work with `gawk'. Thus, we usually don't distinguish between `gawk' and other `awk' implementations. Using `awk' allows you to: * Manage small, personal databases * Generate reports * Validate data * Produce indexes and perform other document preparation tasks * Experiment with algorithms that you can adapt later to other computer languages In addition, `gawk' provides facilities that make it easy to: * Extract bits and pieces of data for processing * Sort data * Perform simple network communications This Info file teaches you about the `awk' language and how you can use it effectively. You should already be familiar with basic system commands, such as `cat' and `ls',(1) as well as basic shell facilities, such as input/output (I/O) redirection and pipes. Implementations of the `awk' language are available for many different computing environments. This Info file, while describing the `awk' language in general, also describes the particular implementation of `awk' called `gawk' (which stands for "GNU awk"). `gawk' runs on a broad range of Unix systems, ranging from 80386 PC-based computers up through large-scale systems, such as Crays. `gawk' has also been ported to Mac OS X, MS-DOS, Microsoft Windows (all versions) and OS/2 PCs, Atari and Amiga microcomputers, BeOS, Tandem D20, and VMS. ---------- Footnotes ---------- (1) These commands are available on POSIX-compliant systems, as well as on traditional Unix-based systems. If you are using some other operating system, you still need to be familiar with the ideas of I/O redirection and pipes. History of `awk' and `gawk' =========================== Recipe For A Programming Language 1 part `egrep' 1 part `snobol' 2 parts `ed' 3 parts C Blend all parts well using `lex' and `yacc'. Document minimally and release. After eight years, add another part `egrep' and two more parts C. Document very well and release. The name `awk' comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger and Brian W. Kernighan. The original version of `awk' was written in 1977 at AT&T Bell Laboratories. In 1985, a new version made the programming language more powerful, introducing user-defined functions, multiple input streams, and computed regular expressions. This new version became widely available with Unix System V Release 3.1 (SVR3.1). The version in SVR4 added some new features and cleaned up the behavior in some of the "dark corners" of the language. The specification for `awk' in the POSIX Command Language and Utilities standard further clarified the language. Both the `gawk' designers and the original Bell Laboratories `awk' designers provided feedback for the POSIX specification. Paul Rubin wrote the GNU implementation, `gawk', in 1986. Jay Fenlason completed it, with advice from Richard Stallman. John Woods contributed parts of the code as well. In 1988 and 1989, David Trueman, with help from me, thoroughly reworked `gawk' for compatibility with the newer `awk'. Circa 1995, I became the primary maintainer. Current development focuses on bug fixes, performance improvements, standards compliance, and occasionally, new features. In May of 1997, Ju"rgen Kahrs felt the need for network access from `awk', and with a little help from me, set about adding features to do this for `gawk'. At that time, he also wrote the bulk of `TCP/IP Internetworking with `gawk'' (a separate document, available as part of the `gawk' distribution). His code finally became part of the main `gawk' distribution with `gawk' version 3.1. *Note Major Contributors to `gawk': Contributors, for a complete list of those who made important contributions to `gawk'. A Rose by Any Other Name ======================== The `awk' language has evolved over the years. Full details are provided in *Note The Evolution of the `awk' Language: Language History. The language described in this Info file is often referred to as "new `awk'" (`nawk'). Because of this, many systems have multiple versions of `awk'. Some systems have an `awk' utility that implements the original version of the `awk' language and a `nawk' utility for the new version. Others have an `oawk' version for the "old `awk'" language and plain `awk' for the new one. Still others only have one version, which is usually the new one.(1) All in all, this makes it difficult for you to know which version of `awk' you should run when writing your programs. The best advice I can give here is to check your local documentation. Look for `awk', `oawk', and `nawk', as well as for `gawk'. It is likely that you already have some version of new `awk' on your system, which is what you should use when running your programs. (Of course, if you're reading this Info file, chances are good that you have `gawk'!) Throughout this Info file, whenever we refer to a language feature that should be available in any complete implementation of POSIX `awk', we simply use the term `awk'. When referring to a feature that is specific to the GNU implementation, we use the term `gawk'. ---------- Footnotes ---------- (1) Often, these systems use `gawk' for their `awk' implementation! Using This Book =============== The term `awk' refers to a particular program as well as to the language you use to tell this program what to do. When we need to be careful, we call the language "the `awk' language," and the program "the `awk' utility." This Info file explains both the `awk' language and how to run the `awk' utility. The term "`awk' program" refers to a program written by you in the `awk' programming language. Primarily, this Info file explains the features of `awk', as defined in the POSIX standard. It does so in the context of the `gawk' implementation. While doing so, it also attempts to describe important differences between `gawk' and other `awk' implementations.(1) Finally, any `gawk' features that are not in the POSIX standard for `awk' are noted. There are subsections labelled as *Advanced Notes* scattered throughout the Info file. They add a more complete explanation of points that are relevant, but not likely to be of interest on first reading. All appear in the index, under the heading "advanced features." Most of the time, the examples use complete `awk' programs. In some of the more advanced sections, only the part of the `awk' program that illustrates the concept currently being described is shown. While this Info file is aimed principally at people who have not been exposed to `awk', there is a lot of information here that even the `awk' expert should find useful. In particular, the description of POSIX `awk' and the example programs in *Note A Library of `awk' Functions: Library Functions, and in *Note Practical `awk' Programs: Sample Programs, should be of interest. *Note Getting Started with `awk': Getting Started, provides the essentials you need to know to begin using `awk'. *Note Regular Expressions: Regexp, introduces regular expressions in general, and in particular the flavors supported by POSIX `awk' and `gawk'. *Note Reading Input Files: Reading Files, describes how `awk' reads your data. It introduces the concepts of records and fields, as well as the `getline' command. I/O redirection is first described here. *Note Printing Output: Printing, describes how `awk' programs can produce output with `print' and `printf'. *Note Expressions::, describes expressions, which are the basic building blocks for getting most things done in a program. *Note Patterns Actions and Variables: Patterns and Actions, describes how to write patterns for matching records, actions for doing something when a record is matched, and the built-in variables `awk' and `gawk' use. *Note Arrays in `awk': Arrays, covers `awk''s one-and-only data structure: associative arrays. Deleting array elements and whole arrays is also described, as well as sorting arrays in `gawk'. *Note Functions::, describes the built-in functions `awk' and `gawk' provide, as well as how to define your own functions. *Note Internationalization with `gawk': Internationalization, describes special features in `gawk' for translating program messages into different languages at runtime. *Note Advanced Features of `gawk': Advanced Features, describes a number of `gawk'-specific advanced features. Of particular note are the abilities to have two-way communications with another process, perform TCP/IP networking, and profile your `awk' programs. *Note Running `awk' and `gawk': Invoking Gawk, describes how to run `gawk', the meaning of its command-line options, and how it finds `awk' program source files. *Note A Library of `awk' Functions: Library Functions, and *Note Practical `awk' Programs: Sample Programs, provide many sample `awk' programs. Reading them allows you to see `awk' solving real problems. *Note The Evolution of the `awk' Language: Language History, describes how the `awk' language has evolved since first release to present. It also describes how `gawk' has acquired features over time. *Note Installing `gawk': Installation, describes how to get `gawk', how to compile it under Unix, and how to compile and use it on different non-Unix systems. It also describes how to report bugs in `gawk' and where to get three other freely available implementations of `awk'. *Note Implementation Notes: Notes, describes how to disable `gawk''s extensions, as well as how to contribute new code to `gawk', how to write extension libraries, and some possible future directions for `gawk' development. *Note Basic Programming Concepts: Basic Concepts, provides some very cursory background material for those who are completely unfamiliar with computer programming. Also centralized there is a discussion of some of the issues surrounding floating-point numbers. The *Note Glossary::, defines most, if not all, the significant terms used throughout the book. If you find terms that you aren't familiar with, try looking them up here. *Note GNU General Public License: Copying, and *Note GNU Free Documentation License::, present the licenses that cover the `gawk' source code and this Info file, respectively. ---------- Footnotes ---------- (1) All such differences appear in the index under the entry "differences in `awk' and `gawk'." Typographical Conventions ========================= This Info file is written using Texinfo, the GNU documentation formatting language. A single Texinfo source file is used to produce both the printed and online versions of the documentation. This minor node briefly documents the typographical conventions used in Texinfo. Examples you would type at the command-line are preceded by the common shell primary and secondary prompts, `$' and `>'. Output from the command is preceded by the glyph "-|". This typically represents the command's standard output. Error messages, and other output on the command's standard error, are preceded by the glyph "error-->". For example: $ echo hi on stdout -| hi on stdout $ echo hello on stderr 1>&2 error--> hello on stderr Characters that you type at the keyboard look `like this'. In particular, there are special characters called "control characters." These are characters that you type by holding down both the `CONTROL' key and another key, at the same time. For example, a `Ctrl-d' is typed by first pressing and holding the `CONTROL' key, next pressing the `d' key and finally releasing both keys. Dark Corners ............ Dark corners are basically fractal -- no matter how much you illuminate, there's always a smaller but darker one. Brian Kernighan Until the POSIX standard (and `The Gawk Manual'), many features of `awk' were either poorly documented or not documented at all. Descriptions of such features (often called "dark corners") are noted in this Info file with "(d.c.)". They also appear in the index under the heading "dark corner." As noted by the opening quote, though, any coverage of dark corners is, by definition, something that is incomplete. The GNU Project and This Book ============================= The Free Software Foundation (FSF) is a nonprofit organization dedicated to the production and distribution of freely distributable software. It was founded by Richard M. Stallman, the author of the original Emacs editor. GNU Emacs is the most widely used version of Emacs today. The GNU(1) Project is an ongoing effort on the part of the Free Software Foundation to create a complete, freely distributable, POSIX-compliant computing environment. The FSF uses the "GNU General Public License" (GPL) to ensure that their software's source code is always available to the end user. A copy of the GPL is included for your reference (*note GNU General Public License: Copying.). The GPL applies to the C language source code for `gawk'. To find out more about the FSF and the GNU Project online, see the GNU Project's home page (http://www.gnu.org). This Info file may also be read from their web site (http://www.gnu.org/manual/gawk/). A shell, an editor (Emacs), highly portable optimizing C, C++, and Objective-C compilers, a symbolic debugger and dozens of large and small utilities (such as `gawk'), have all been completed and are freely available. The GNU operating system kernel (the HURD), has been released but is still in an early stage of development. Until the GNU operating system is more fully developed, you should consider using GNU/Linux, a freely distributable, Unix-like operating system for Intel 80386, DEC Alpha, Sun SPARC, IBM S/390, and other systems.(2) There are many books on GNU/Linux. One that is freely available is `Linux Installation and Getting Started', by Matt Welsh. Many GNU/Linux distributions are often available in computer stores or bundled on CD-ROMs with books about Linux. (There are three other freely available, Unix-like operating systems for 80386 and other systems: NetBSD, FreeBSD, and OpenBSD. All are based on the 4.4-Lite Berkeley Software Distribution, and they use recent versions of `gawk' for their versions of `awk'.) The Info file itself has gone through a number of previous editions. Paul Rubin wrote the very first draft of `The GAWK Manual'; it was around 40 pages in size. Diane Close and Richard Stallman improved it, yielding a version that was around 90 pages long and barely described the original, "old" version of `awk'. I started working with that version in the fall of 1988. As work on it progressed, the FSF published several preliminary versions (numbered 0.X). In 1996, Edition 1.0 was released with `gawk' 3.0.0. The FSF published the first two editions under the title `The GNU Awk User's Guide'. This edition maintains the basic structure of Edition 1.0, but with significant additional material, reflecting the host of new features in `gawk' version 3.1. Of particular note is *Note Sorting Array Values and Indices with `gawk': Array Sorting, as well as *Note Using `gawk''s Bit Manipulation Functions: Bitwise Functions, *Note Internationalization with `gawk': Internationalization, and also *Note Advanced Features of `gawk': Advanced Features, and *Note Adding New Built-in Functions to `gawk': Dynamic Extensions. `GAWK: Effective AWK Programming' will undoubtedly continue to evolve. An electronic version comes with the `gawk' distribution from the FSF. If you find an error in this Info file, please report it! *Note Reporting Problems and Bugs: Bugs, for information on submitting problem reports electronically, or write to me in care of the publisher. ---------- Footnotes ---------- (1) GNU stands for "GNU's not Unix." (2) The terminology "GNU/Linux" is explained in the *Note Glossary::. How to Contribute ================= As the maintainer of GNU `awk', I am starting a collection of publicly available `awk' programs. For more information, see `ftp://ftp.freefriends.org/arnold/Awkstuff'. If you have written an interesting `awk' program, or have written a `gawk' extension that you would like to share with the rest of the world, please contact me (). Making things available on the Internet helps keep the `gawk' distribution down to manageable size. Acknowledgments =============== The initial draft of `The GAWK Manual' had the following acknowledgments: Many people need to be thanked for their assistance in producing this manual. Jay Fenlason contributed many ideas and sample programs. Richard Mlynarik and Robert Chassell gave helpful comments on drafts of this manual. The paper `A Supplemental Document for `awk'' by John W. Pierce of the Chemistry Department at UC San Diego, pinpointed several issues relevant both to `awk' implementation and to this manual, that would otherwise have escaped us. I would like to acknowledge Richard M. Stallman, for his vision of a better world and for his courage in founding the FSF and starting the GNU Project. The following people (in alphabetical order) provided helpful comments on various versions of this book, up to and including this edition. Rick Adams, Nelson H.F. Beebe, Karl Berry, Dr. Michael Brennan, Rich Burridge, Claire Cloutier, Diane Close, Scott Deifik, Christopher ("Topher") Eliot, Jeffrey Friedl, Dr. Darrel Hankerson, Michal Jaegermann, Dr. Richard J. LeBlanc, Michael Lijewski, Pat Rankin, Miriam Robbins, Mary Sheehan, and Chuck Toporek. Robert J. Chassell provided much valuable advice on the use of Texinfo. He also deserves special thanks for convincing me _not_ to title this Info file `How To Gawk Politely'. Karl Berry helped significantly with the TeX part of Texinfo. I would like to thank Marshall and Elaine Hartholz of Seattle and Dr. Bert and Rita Schreiber of Detroit for large amounts of quiet vacation time in their homes, which allowed me to make significant progress on this Info file and on `gawk' itself. Phil Hughes of SSC contributed in a very important way by loaning me his laptop GNU/Linux system, not once, but twice, which allowed me to do a lot of work while away from home. David Trueman deserves special credit; he has done a yeoman job of evolving `gawk' so that it performs well and without bugs. Although he is no longer involved with `gawk', working with him on this project was a significant pleasure. The intrepid members of the GNITS mailing list, and most notably Ulrich Drepper, provided invaluable help and feedback for the design of the internationalization features. Nelson Beebe, Martin Brown, Andreas Buening, Scott Deifik, Darrel Hankerson, Isamu Hasegawa, Michal Jaegermann, Ju"rgen Kahrs, Pat Rankin, Kai Uwe Rommel, and Eli Zaretskii (in alphabetical order) make up the `gawk' "crack portability team." Without their hard work and help, `gawk' would not be nearly the fine program it is today. It has been and continues to be a pleasure working with this team of fine people. David and I would like to thank Brian Kernighan of Bell Laboratories for invaluable assistance during the testing and debugging of `gawk', and for help in clarifying numerous points about the language. We could not have done nearly as good a job on either `gawk' or its documentation without his help. Chuck Toporek, Mary Sheehan, and Claire Coutier of O'Reilly & Associates contributed significant editorial help for this Info file for the 3.1 release of `gawk'. I must thank my wonderful wife, Miriam, for her patience through the many versions of this project, for her proofreading, and for sharing me with the computer. I would like to thank my parents for their love, and for the grace with which they raised and educated me. Finally, I also must acknowledge my gratitude to G-d, for the many opportunities He has sent my way, as well as for the gifts He has given me with which to take advantage of those opportunities. Arnold Robbins Nof Ayalon ISRAEL March, 2001 Getting Started with `awk' ************************** The basic function of `awk' is to search files for lines (or other units of text) that contain certain patterns. When a line matches one of the patterns, `awk' performs specified actions on that line. `awk' keeps processing input lines in this way until it reaches the end of the input files. Programs in `awk' are different from programs in most other languages, because `awk' programs are "data-driven"; that is, you describe the data you want to work with and then what to do when you find it. Most other languages are "procedural"; you have to describe, in great detail, every step the program is to take. When working with procedural languages, it is usually much harder to clearly describe the data your program will process. For this reason, `awk' programs are often refreshingly easy to read and write. When you run `awk', you specify an `awk' "program" that tells `awk' what to do. The program consists of a series of "rules". (It may also contain "function definitions", an advanced feature that we will ignore for now. *Note User-Defined Functions: User-defined.) Each rule specifies one pattern to search for and one action to perform upon finding the pattern. Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in curly braces to separate it from the pattern. Newlines usually separate rules. Therefore, an `awk' program looks like this: PATTERN { ACTION } PATTERN { ACTION } ... How to Run `awk' Programs ========================= There are several ways to run an `awk' program. If the program is short, it is easiest to include it in the command that runs `awk', like this: awk 'PROGRAM' INPUT-FILE1 INPUT-FILE2 ... When the program is long, it is usually more convenient to put it in a file and run it with a command like this: awk -f PROGRAM-FILE INPUT-FILE1 INPUT-FILE2 ... This minor node discusses both mechanisms, along with several variations of each. One-Shot Throwaway `awk' Programs --------------------------------- Once you are familiar with `awk', you will often type in simple programs the moment you want to use them. Then you can write the program as the first argument of the `awk' command, like this: awk 'PROGRAM' INPUT-FILE1 INPUT-FILE2 ... where PROGRAM consists of a series of PATTERNS and ACTIONS, as described earlier. This command format instructs the "shell", or command interpreter, to start `awk' and use the PROGRAM to process records in the input file(s). There are single quotes around PROGRAM so the shell won't interpret any `awk' characters as special shell characters. The quotes also cause the shell to treat all of PROGRAM as a single argument for `awk', and allow PROGRAM to be more than one line long. This format is also useful for running short or medium-sized `awk' programs from shell scripts, because it avoids the need for a separate file for the `awk' program. A self-contained shell script is more reliable because there are no other files to misplace. *Note Some Simple Examples: Very Simple, presents several short, self-contained programs. Running `awk' Without Input Files --------------------------------- You can also run `awk' without any input files. If you type the following command line: awk 'PROGRAM' `awk' applies the PROGRAM to the "standard input", which usually means whatever you type on the terminal. This continues until you indicate end-of-file by typing `Ctrl-d'. (On other operating systems, the end-of-file character may be different. For example, on OS/2 and MS-DOS, it is `Ctrl-z'.) As an example, the following program prints a friendly piece of advice (from Douglas Adams's `The Hitchhiker's Guide to the Galaxy'), to keep you from worrying about the complexities of computer programming (`BEGIN' is a feature we haven't discussed yet): $ awk "BEGIN { print \"Don't Panic!\" }" -| Don't Panic! This program does not read any input. The `\' before each of the inner double quotes is necessary because of the shell's quoting rules--in particular because it mixes both single quotes and double quotes.(1) This next simple `awk' program emulates the `cat' utility; it copies whatever you type on the keyboard to its standard output (why this works is explained shortly). $ awk '{ print }' Now is the time for all good men -| Now is the time for all good men to come to the aid of their country. -| to come to the aid of their country. Four score and seven years ago, ... -| Four score and seven years ago, ... What, me worry? -| What, me worry? Ctrl-d ---------- Footnotes ---------- (1) Although we generally recommend the use of single quotes around the program text, double quotes are needed here in order to put the single quote into the message. Running Long Programs --------------------- Sometimes your `awk' programs can be very long. In this case, it is more convenient to put the program into a separate file. In order to tell `awk' to use that file for its program, you type: awk -f SOURCE-FILE INPUT-FILE1 INPUT-FILE2 ... The `-f' instructs the `awk' utility to get the `awk' program from the file SOURCE-FILE. Any file name can be used for SOURCE-FILE. For example, you could put the program: BEGIN { print "Don't Panic!" } into the file `advice'. Then this command: awk -f advice does the same thing as this one: awk "BEGIN { print \"Don't Panic!\" }" This was explained earlier (*note Running `awk' Without Input Files: Read Terminal.). Note that you don't usually need single quotes around the file name that you specify with `-f', because most file names don't contain any of the shell's special characters. Notice that in `advice', the `awk' program did not have single quotes around it. The quotes are only needed for programs that are provided on the `awk' command line. If you want to identify your `awk' program files clearly as such, you can add the extension `.awk' to the file name. This doesn't affect the execution of the `awk' program but it does make "housekeeping" easier. Executable `awk' Programs ------------------------- Once you have learned `awk', you may want to write self-contained `awk' scripts, using the `#!' script mechanism. You can do this on many Unix systems(1) as well as on the GNU system. For example, you could update the file `advice' to look like this: #! /bin/awk -f BEGIN { print "Don't Panic!" } After making this file executable (with the `chmod' utility), simply type `advice' at the shell and the system arranges to run `awk'(2) as if you had typed `awk -f advice': $ chmod +x advice $ advice -| Don't Panic! Self-contained `awk' scripts are useful when you want to write a program that users can invoke without their having to know that the program is written in `awk'. Advanced Notes: Portability Issues with `#!' -------------------------------------------- Some systems limit the length of the interpreter name to 32 characters. Often, this can be dealt with by using a symbolic link. You should not put more than one argument on the `#!' line after the path to `awk'. It does not work. The operating system treats the rest of the line as a single argument and passes it to `awk'. Doing this leads to confusing behavior--most likely a usage diagnostic of some sort from `awk'. Finally, the value of `ARGV[0]' (*note Built-in Variables::) varies depending upon your operating system. Some systems put `awk' there, some put the full pathname of `awk' (such as `/bin/awk'), and some put the name of your script (`advice'). Don't rely on the value of `ARGV[0]' to provide your script name. ---------- Footnotes ---------- (1) The `#!' mechanism works on Linux systems, systems derived from the 4.4-Lite Berkeley Software Distribution, and most commercial Unix systems. (2) The line beginning with `#!' lists the full file name of an interpreter to run and an optional initial command-line argument to pass to that interpreter. The operating system then runs the interpreter with the given argument and the full argument list of the executed program. The first argument in the list is the full file name of the `awk' program. The rest of the argument list contains either options to `awk', or data files, or both. Comments in `awk' Programs -------------------------- A "comment" is some text that is included in a program for the sake of human readers; it is not really an executable part of the program. Comments can explain what the program does and how it works. Nearly all programming languages have provisions for comments, as programs are typically hard to understand without them. In the `awk' language, a comment starts with the sharp sign character (`#') and continues to the end of the line. The `#' does not have to be the first character on the line. The `awk' language ignores the rest of a line following a sharp sign. For example, we could have put the following into `advice': # This program prints a nice friendly message. It helps # keep novice users from being afraid of the computer. BEGIN { print "Don't Panic!" } You can put comment lines into keyboard-composed throwaway `awk' programs, but this usually isn't very useful; the purpose of a comment is to help you or another person understand the program when reading it at a later time. *Caution:* As mentioned in *Note One-Shot Throwaway `awk' Programs: One-shot, you can enclose small to medium programs in single quotes, in order to keep your shell scripts self-contained. When doing so, _don't_ put an apostrophe (i.e., a single quote) into a comment (or anywhere else in your program). The shell interprets the quote as the closing quote for the entire program. As a result, usually the shell prints a message about mismatched quotes, and if `awk' actually runs, it will probably print strange messages about syntax errors. For example, look at the following: $ awk '{ print "hello" } # let's be cute' > The shell sees that the first two quotes match, and that a new quoted object begins at the end of the command line. It therefore prompts with the secondary prompt, waiting for more input. With Unix `awk', closing the quoted string produces this result: $ awk '{ print "hello" } # let's be cute' > ' error--> awk: can't open file be error--> source line number 1 Putting a backslash before the single quote in `let's' wouldn't help, since backslashes are not special inside single quotes. The next node describes the shell's quoting rules. Shell-Quoting Issues -------------------- For short to medium length `awk' programs, it is most convenient to enter the program on the `awk' command line. This is best done by enclosing the entire program in single quotes. This is true whether you are entering the program interactively at the shell prompt, or writing it as part of a larger shell script: awk 'PROGRAM TEXT' INPUT-FILE1 INPUT-FILE2 ... Once you are working with the shell, it is helpful to have a basic knowledge of shell quoting rules. The following rules apply only to POSIX-compliant, Bourne-style shells (such as `bash', the GNU Bourne-Again Shell). If you use `csh', you're on your own. * Quoted items can be concatenated with nonquoted items as well as with other quoted items. The shell turns everything into one argument for the command. * Preceding any single character with a backslash (`\') quotes that character. The shell removes the backslash and passes the quoted character on to the command. * Single quotes protect everything between the opening and closing quotes. The shell does no interpretation of the quoted text, passing it on verbatim to the command. It is _impossible_ to embed a single quote inside single-quoted text. Refer back to *Note Comments in `awk' Programs: Comments, for an example of what happens if you try. * Double quotes protect most things between the opening and closing quotes. The shell does at least variable and command substitution on the quoted text. Different shells may do additional kinds of processing on double-quoted text. Since certain characters within double-quoted text are processed by the shell, they must be "escaped" within the text. Of note are the characters `$', ``', `\', and `"', all of which must be preceded by a backslash within double-quoted text if they are to be passed on literally to the program. (The leading backslash is stripped first.) Thus, the example seen in *Note Running `awk' Without Input Files: Read Terminal, is applicable: $ awk "BEGIN { print \"Don't Panic!\" }" -| Don't Panic! Note that the single quote is not special within double quotes. * Null strings are removed when they occur as part of a non-null command-line argument, while explicit non-null objects are kept. For example, to specify that the field separator `FS' should be set to the null string, use: awk -F "" 'PROGRAM' FILES # correct Don't use this: awk -F"" 'PROGRAM' FILES # wrong! In the second case, `awk' will attempt to use the text of the program as the value of `FS', and the first file name as the text of the program! This results in syntax errors at best, and confusing behavior at worst. Mixing single and double quotes is difficult. You have to resort to shell quoting tricks, like this: $ awk 'BEGIN { print "Here is a single quote <'"'"'>" }' -| Here is a single quote <'> This program consists of three concatenated quoted strings. The first and the third are single-quoted, the second is double-quoted. This can be "simplified" to: $ awk 'BEGIN { print "Here is a single quote <'\''>" }' -| Here is a single quote <'> Judge for yourself which of these two is the more readable. Another option is to use double quotes, escaping the embedded, `awk'-level double quotes: $ awk "BEGIN { print \"Here is a single quote <'>\" }" -| Here is a single quote <'> This option is also painful, because double quotes, backslashes, and dollar signs are very common in `awk' programs. If you really need both single and double quotes in your `awk' program, it is probably best to move it into a separate file, where the shell won't be part of the picture, and you can say what you mean. Data Files for the Examples =========================== Many of the examples in this Info file take their input from two sample data files. The first, `BBS-list', represents a list of computer bulletin board systems together with information about those systems. The second data file, called `inventory-shipped', contains information about monthly shipments. In both files, each line is considered to be one "record". In the data file `BBS-list', each record contains the name of a computer bulletin board, its phone number, the board's baud rate(s), and a code for the number of hours it is operational. An `A' in the last column means the board operates 24 hours a day. A `B' in the last column means the board only operates on evening and weekend hours. A `C' means the board operates only on weekends: aardvark 555-5553 1200/300 B alpo-net 555-3412 2400/1200/300 A barfly 555-7685 1200/300 A bites 555-1675 2400/1200/300 A camelot 555-0542 300 C core 555-2912 1200/300 C fooey 555-1234 2400/1200/300 B foot 555-6699 1200/300 B macfoo 555-6480 1200/300 A sdace 555-3430 2400/1200/300 A sabafoo 555-2127 1200/300 C The data file `inventory-shipped' represents information about shipments during the year. Each record contains the month, the number of green crates shipped, the number of red boxes shipped, the number of orange bags shipped, and the number of blue packages shipped, respectively. There are 16 entries, covering the 12 months of last year and the first four months of the current year. Jan 13 25 15 115 Feb 15 32 24 226 Mar 15 24 34 228 Apr 31 52 63 420 May 16 34 29 208 Jun 31 42 75 492 Jul 24 34 67 436 Aug 15 34 47 316 Sep 13 55 37 277 Oct 29 54 68 525 Nov 20 87 82 577 Dec 17 35 61 401 Jan 21 36 64 620 Feb 26 58 80 652 Mar 24 75 70 495 Apr 21 70 74 514 If you are reading this in GNU Emacs using Info, you can copy the regions of text showing these sample files into your own test files. This way you can try out the examples shown in the remainder of this document. You do this by using the command `M-x write-region' to copy text from the Info file into a file for use with `awk' (*Note Miscellaneous File Operations: (emacs)Misc File Ops, for more information). Using this information, create your own `BBS-list' and `inventory-shipped' files and practice what you learn in this Info file. If you are using the stand-alone version of Info, see *Note Extracting Programs from Texinfo Source Files: Extract Program, for an `awk' program that extracts these data files from `gawk.texi', the Texinfo source file for this Info file. Some Simple Examples ==================== The following command runs a simple `awk' program that searches the input file `BBS-list' for the character string `foo' (a grouping of characters is usually called a "string"; the term "string" is based on similar usage in English, such as "a string of pearls," or "a string of cars in a train"): awk '/foo/ { print $0 }' BBS-list When lines containing `foo' are found, they are printed because `print $0' means print the current line. (Just `print' by itself means the same thing, so we could have written that instead.) You will notice that slashes (`/') surround the string `foo' in the `awk' program. The slashes indicate that `foo' is the pattern to search for. This type of pattern is called a "regular expression", which is covered in more detail later (*note Regular Expressions: Regexp.). The pattern is allowed to match parts of words. There are single quotes around the `awk' program so that the shell won't interpret any of it as special shell characters. Here is what this program prints: $ awk '/foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C In an `awk' rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for _every_ input line. If the action is omitted, the default action is to print all lines that match the pattern. Thus, we could leave out the action (the `print' statement and the curly braces) in the previous example and the result would be the same: all lines matching the pattern `foo' are printed. By comparison, omitting the `print' statement but retaining the curly braces makes an empty action that does nothing (i.e., no lines are printed). Many practical `awk' programs are just a line or two. Following is a collection of useful, short programs to get you started. Some of these programs contain constructs that haven't been covered yet. (The description of the program will give you a good idea of what is going on, but please read the rest of the Info file to become an `awk' expert!) Most of the examples use a data file named `data'. This is just a placeholder; if you use these programs yourself, substitute your own file names for `data'. For future reference, note that there is often more than one way to do things in `awk'. At some point, you may want to look back at these examples and see if you can come up with different ways to do the same things shown here: * Print the length of the longest input line: awk '{ if (length($0) > max) max = length($0) } END { print max }' data * Print every line that is longer than 80 characters: awk 'length($0) > 80' data The sole rule has a relational expression as its pattern and it has no action--so the default action, printing the record, is used. * Print the length of the longest line in `data': expand data | awk '{ if (x < length()) x = length() } END { print "maximum line length is " x }' The input is processed by the `expand' utility to change tabs into spaces, so the widths compared are actually the right-margin columns. * Print every line that has at least one field: awk 'NF > 0' data This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been removed). * Print seven random numbers from 0 to 100, inclusive: awk 'BEGIN { for (i = 1; i <= 7; i++) print int(101 * rand()) }' * Print the total number of bytes used by FILES: ls -l FILES | awk '{ x += $5 } END { print "total bytes: " x }' * Print the total number of kilobytes used by FILES: ls -l FILES | awk '{ x += $5 } END { print "total K-bytes: " (x + 1023)/1024 }' * Print a sorted list of the login names of all users: awk -F: '{ print $1 }' /etc/passwd | sort * Count the lines in a file: awk 'END { print NR }' data * Print the even-numbered lines in the data file: awk 'NR % 2 == 0' data If you use the expression `NR % 2 == 1' instead, the program would print the odd-numbered lines. An Example with Two Rules ========================= The `awk' utility reads the input files one line at a time. For each line, `awk' tries the patterns of each of the rules. If several patterns match, then several actions are run in the order in which they appear in the `awk' program. If no patterns match, then no actions are run. After processing all the rules that match the line (and perhaps there are none), `awk' reads the next line. (However, *note The `next' Statement: Next Statement., and also *note Using `gawk''s `nextfile' Statement: Nextfile Statement.). This continues until the program reaches the end of the file. For example, the following `awk' program contains two rules: /12/ { print $0 } /21/ { print $0 } The first rule has the string `12' as the pattern and `print $0' as the action. The second rule has the string `21' as the pattern and also has `print $0' as the action. Each rule's action is enclosed in its own pair of braces. This program prints every line that contains the string `12' _or_ the string `21'. If a line contains both strings, it is printed twice, once by each rule. This is what happens if we run this program on our two sample data files, `BBS-list' and `inventory-shipped': $ awk '/12/ { print $0 } > /21/ { print $0 }' BBS-list inventory-shipped -| aardvark 555-5553 1200/300 B -| alpo-net 555-3412 2400/1200/300 A -| barfly 555-7685 1200/300 A -| bites 555-1675 2400/1200/300 A -| core 555-2912 1200/300 C -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sdace 555-3430 2400/1200/300 A -| sabafoo 555-2127 1200/300 C -| sabafoo 555-2127 1200/300 C -| Jan 21 36 64 620 -| Apr 21 70 74 514 Note how the line beginning with `sabafoo' in `BBS-list' was printed twice, once for each rule. A More Complex Example ====================== Now that we've mastered some simple tasks, let's look at what typical `awk' programs do. This example shows how `awk' can be used to summarize, select, and rearrange the output of another utility. It uses features that haven't been covered yet, so don't worry if you don't understand all the details: ls -l | awk '$6 == "Nov" { sum += $5 } END { print sum }' This command prints the total number of bytes in all the files in the current directory that were last modified in November (of any year). (1) The `ls -l' part of this example is a system command that gives you a listing of the files in a directory, including each file's size and the date the file was last modified. Its output looks like this: -rw-r--r-- 1 arnold user 1933 Nov 7 13:05 Makefile -rw-r--r-- 1 arnold user 10809 Nov 7 13:03 awk.h -rw-r--r-- 1 arnold user 983 Apr 13 12:14 awk.tab.h -rw-r--r-- 1 arnold user 31869 Jun 15 12:20 awk.y -rw-r--r-- 1 arnold user 22414 Nov 7 13:03 awk1.c -rw-r--r-- 1 arnold user 37455 Nov 7 13:03 awk2.c -rw-r--r-- 1 arnold user 27511 Dec 9 13:07 awk3.c -rw-r--r-- 1 arnold user 7989 Nov 7 13:03 awk4.c The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the owner of the file. The fourth field identifies the group of the file. The fifth field contains the size of the file in bytes. The sixth, seventh, and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the name of the file.(2) The `$6 == "Nov"' in our `awk' program is an expression that tests whether the sixth field of the output from `ls -l' matches the string `Nov'. Each time a line has the string `Nov' for its sixth field, the action `sum += $5' is performed. This adds the fifth field (the file's size) to the variable `sum'. As a result, when `awk' has finished reading all the input lines, `sum' is the total of the sizes of the files whose lines matched the pattern. (This works because `awk' variables are automatically initialized to zero.) After the last line of output from `ls' has been processed, the `END' rule executes and prints the value of `sum'. In this example, the value of `sum' is 140963. These more advanced `awk' techniques are covered in later sections (*note Actions: Action Overview.). Before you can move on to more advanced `awk' programming, you have to know how `awk' interprets your input and displays your output. By manipulating fields and using `print' statements, you can produce some very useful and impressive-looking reports. ---------- Footnotes ---------- (1) In the C shell (`csh'), you need to type a semicolon and then a backslash at the end of the first line; see *Note `awk' Statements Versus Lines: Statements/Lines, for an explanation. In a POSIX-compliant shell, such as the Bourne shell or `bash', you can type the example as shown. If the command `echo $path' produces an empty output line, you are most likely using a POSIX-compliant shell. Otherwise, you are probably using the C shell or a shell derived from it. (2) On some very old systems, you may need to use `ls -lg' to get this output. `awk' Statements Versus Lines ============================= Most often, each line in an `awk' program is a separate statement or separate rule, like this: awk '/12/ { print $0 } /21/ { print $0 }' BBS-list inventory-shipped However, `gawk' ignores newlines after any of the following symbols and keywords: , { ? : || && do else A newline at any other point is considered the end of the statement.(1) If you would like to split a single statement into two lines at a point where a newline would terminate it, you can "continue" it by ending the first line with a backslash character (`\'). The backslash must be the final character on the line in order to be recognized as a continuation character. A backslash is allowed anywhere in the statement, even in the middle of a string or regular expression. For example: awk '/This regular expression is too long, so continue it\ on the next line/ { print $1 }' We have generally not used backslash continuation in the sample programs in this Info file. In `gawk', there is no limit on the length of a line, so backslash continuation is never strictly necessary; it just makes programs more readable. For this same reason, as well as for clarity, we have kept most statements short in the sample programs presented throughout the Info file. Backslash continuation is most useful when your `awk' program is in a separate source file instead of entered from the command line. You should also note that many `awk' implementations are more particular about where you may use backslash continuation. For example, they may not allow you to split a string constant using backslash continuation. Thus, for maximum portability of your `awk' programs, it is best not to split your lines in the middle of a regular expression or a string. *Caution:* _Backslash continuation does not work as described with the C shell._ It works for `awk' programs in files and for one-shot programs, _provided_ you are using a POSIX-compliant shell, such as the Unix Bourne shell or `bash'. But the C shell behaves differently! There, you must use two backslashes in a row, followed by a newline. Note also that when using the C shell, _every_ newline in your awk program must be escaped with a backslash. To illustrate: % awk 'BEGIN { \ ? print \\ ? "hello, world" \ ? }' -| hello, world Here, the `%' and `?' are the C shell's primary and secondary prompts, analogous to the standard shell's `$' and `>'. Compare the previous example to how it is done with a POSIX-compliant shell: $ awk 'BEGIN { > print \ > "hello, world" > }' -| hello, world `awk' is a line-oriented language. Each rule's action has to begin on the same line as the pattern. To have the pattern and action on separate lines, you _must_ use backslash continuation; there is no other option. Another thing to keep in mind is that backslash continuation and comments do not mix. As soon as `awk' sees the `#' that starts a comment, it ignores _everything_ on the rest of the line. For example: $ gawk 'BEGIN { print "dont panic" # a friendly \ > BEGIN rule > }' error--> gawk: cmd. line:2: BEGIN rule error--> gawk: cmd. line:2: ^ parse error In this case, it looks like the backslash would continue the comment onto the next line. However, the backslash-newline combination is never even noticed because it is "hidden" inside the comment. Thus, the `BEGIN' is noted as a syntax error. When `awk' statements within one rule are short, you might want to put more than one of them on a line. This is accomplished by separating the statements with a semicolon (`;'). This also applies to the rules themselves. Thus, the program shown at the start of this minor node could also be written this way: /12/ { print $0 } ; /21/ { print $0 } *Note:* The requirement that states that rules on the same line must be separated with a semicolon was not in the original `awk' language; it was added for consistency with the treatment of statements within an action. ---------- Footnotes ---------- (1) The `?' and `:' referred to here is the three-operand conditional expression described in *Note Conditional Expressions: Conditional Exp. Splitting lines after `?' and `:' is a minor `gawk' extension; if `--posix' is specified (*note Command-Line Options: Options.), then this extension is disabled. Other Features of `awk' ======================= The `awk' language provides a number of predefined, or "built-in", variables that your programs can use to get information from `awk'. There are other variables your program can set as well to control how `awk' processes your data. In addition, `awk' provides a number of built-in functions for doing common computational and string-related operations. `gawk' provides built-in functions for working with timestamps, performing bit manipulation, and for runtime string translation. As we develop our presentation of the `awk' language, we introduce most of the variables and many of the functions. They are defined systematically in *Note Built-in Variables::, and *Note Built-in Functions: Built-in. When to Use `awk' ================= Now that you've seen some of what `awk' can do, you might wonder how `awk' could be useful for you. By using utility programs, advanced patterns, field separators, arithmetic statements, and other selection criteria, you can produce much more complex output. The `awk' language is very useful for producing reports from large amounts of raw data, such as summarizing information from the output of other utility programs like `ls'. (*Note A More Complex Example: More Complex.) Programs written with `awk' are usually much smaller than they would be in other languages. This makes `awk' programs easy to compose and use. Often, `awk' programs can be quickly composed at your terminal, used once, and thrown away. Because `awk' programs are interpreted, you can avoid the (usually lengthy) compilation part of the typical edit-compile-test-debug cycle of software development. Complex programs have been written in `awk', including a complete retargetable assembler for eight-bit microprocessors (*note Glossary::, for more information), and a microcode assembler for a special-purpose Prolog computer. However, `awk''s capabilities are strained by tasks of such complexity. If you find yourself writing `awk' scripts of more than, say, a few hundred lines, you might consider using a different programming language. Emacs Lisp is a good choice if you need sophisticated string or pattern matching capabilities. The shell is also good at string and pattern matching; in addition, it allows powerful use of the system utilities. More conventional languages, such as C, C++, and Java, offer better facilities for system programming and for managing the complexity of large programs. Programs in these languages may require more lines of source code than the equivalent `awk' programs, but they are easier to maintain and usually run more efficiently. Regular Expressions ******************* A "regular expression", or "regexp", is a way of describing a set of strings. Because regular expressions are such a fundamental part of `awk' programming, their format and use deserve a separate major node. A regular expression enclosed in slashes (`/') is an `awk' pattern that matches every input record whose text belongs to that set. The simplest regular expression is a sequence of letters, numbers, or both. Such a regexp matches any string that contains that sequence. Thus, the regexp `foo' matches any string containing `foo'. Therefore, the pattern `/foo/' matches any input record containing the three characters `foo' _anywhere_ in the record. Other kinds of regexps let you specify more complicated classes of strings. How to Use Regular Expressions ============================== A regular expression can be used as a pattern by enclosing it in slashes. Then the regular expression is tested against the entire text of each record. (Normally, it only needs to match some part of the text in order to succeed.) For example, the following prints the second field of each record that contains the string `foo' anywhere in it: $ awk '/foo/ { print $2 }' BBS-list -| 555-1234 -| 555-6699 -| 555-6480 -| 555-2127 `~' (tilde), `~' operator Regular expressions can also be used in matching expressions. These expressions allow you to specify the string to match against; it need not be the entire current input record. The two operators `~' and `!~' perform regular expression comparisons. Expressions using these operators can be used as patterns, or in `if', `while', `for', and `do' statements. (*Note Control Statements in Actions: Statements.) For example: EXP ~ /REGEXP/ is true if the expression EXP (taken as a string) matches REGEXP. The following example matches, or selects, all input records with the uppercase letter `J' somewhere in the first field: $ awk '$1 ~ /J/' inventory-shipped -| Jan 13 25 15 115 -| Jun 31 42 75 492 -| Jul 24 34 67 436 -| Jan 21 36 64 620 So does this: awk '{ if ($1 ~ /J/) print }' inventory-shipped This next example is true if the expression EXP (taken as a character string) does _not_ match REGEXP: EXP !~ /REGEXP/ The following example matches, or selects, all input records whose first field _does not_ contain the uppercase letter `J': $ awk '$1 !~ /J/' inventory-shipped -| Feb 15 32 24 226 -| Mar 15 24 34 228 -| Apr 31 52 63 420 -| May 16 34 29 208 ... When a regexp is enclosed in slashes, such as `/foo/', we call it a "regexp constant", much like `5.27' is a numeric constant and `"foo"' is a string constant. Escape Sequences ================ Some characters cannot be included literally in string constants (`"foo"') or regexp constants (`/foo/'). Instead, they should be represented with "escape sequences", which are character sequences beginning with a backslash (`\'). One use of an escape sequence is to include a double-quote character in a string constant. Because a plain double quote ends the string, you must use `\"' to represent an actual double-quote character as a part of the string. For example: $ awk 'BEGIN { print "He said \"hi!\" to her." }' -| He said "hi!" to her. The backslash character itself is another character that cannot be included normally; you must write `\\' to put one backslash in the string or regexp. Thus, the string whose contents are the two characters `"' and `\' must be written `"\"\\"'. Backslash also represents unprintable characters such as TAB or newline. While there is nothing to stop you from entering most unprintable characters directly in a string constant or regexp constant, they may look ugly. The following table lists all the escape sequences used in `awk' and what they represent. Unless noted otherwise, all these escape sequences apply to both string constants and regexp constants: `\\' A literal backslash, `\'. `\a' The "alert" character, `Ctrl-g', ASCII code 7 (BEL). (This usually makes some sort of audible noise.) `\b' Backspace, `Ctrl-h', ASCII code 8 (BS). `\f' Formfeed, `Ctrl-l', ASCII code 12 (FF). `\n' Newline, `Ctrl-j', ASCII code 10 (LF). `\r' Carriage return, `Ctrl-m', ASCII code 13 (CR). `\t' Horizontal TAB, `Ctrl-i', ASCII code 9 (HT). `\v' Vertical tab, `Ctrl-k', ASCII code 11 (VT). `\NNN' The octal value NNN, where NNN stands for 1 to 3 digits between `0' and `7'. For example, the code for the ASCII ESC (escape) character is `\033'. `\xHH...' The hexadecimal value HH, where HH stands for a sequence of hexadecimal digits (`0'-`9', and either `A'-`F' or `a'-`f'). Like the same construct in ISO C, the escape sequence continues until the first nonhexadecimal digit is seen. However, using more than two hexadecimal digits produces undefined results. (The `\x' escape sequence is not allowed in POSIX `awk'.) `\/' A literal slash (necessary for regexp constants only). This expression is used when you want to write a regexp constant that contains a slash. Because the regexp is delimited by slashes, you need to escape the slash that is part of the pattern, in order to tell `awk' to keep processing the rest of the regexp. `\"' A literal double quote (necessary for string constants only). This expression is used when you want to write a string constant that contains a double quote. Because the string is delimited by double quotes, you need to escape the quote that is part of the string, in order to tell `awk' to keep processing the rest of the string. In `gawk', a number of additional two-character sequences that begin with a backslash have special meaning in regexps. *Note `gawk'-Specific Regexp Operators: GNU Regexp Operators. In a regexp, a backslash before any character that is not in the previous list and not listed in *Note `gawk'-Specific Regexp Operators: GNU Regexp Operators, means that the next character should be taken literally, even if it would normally be a regexp operator. For example, `/a\+b/' matches the three characters `a+b'. For complete portability, do not use a backslash before any character not shown in the previous list. To summarize: * The escape sequences in the table above are always processed first, for both string constants and regexp constants. This happens very early, as soon as `awk' reads your program. * `gawk' processes both regexp constants and dynamic regexps (*note Using Dynamic Regexps: Computed Regexps.), for the special operators listed in *Note `gawk'-Specific Regexp Operators: GNU Regexp Operators. * A backslash before any other character means to treat that character literally. Advanced Notes: Backslash Before Regular Characters --------------------------------------------------- If you place a backslash in a string constant before something that is not one of the characters previously listed, POSIX `awk' purposely leaves what happens as undefined. There are two choices: Strip the backslash out This is what Unix `awk' and `gawk' both do. For example, `"a\qc"' is the same as `"aqc"'. (Because this is such an easy bug both to introduce and to miss, `gawk' warns you about it.) Consider `FS = "[ \t]+\|[ \t]+"' to use vertical bars surrounded by whitespace as the field separator. There should be two backslashes in the string `FS = "[ \t]+\\|[ \t]+"'.) Leave the backslash alone Some other `awk' implementations do this. In such implementations, typing `"a\qc"' is the same as typing `"a\\qc"'. Advanced Notes: Escape Sequences for Metacharacters --------------------------------------------------- Suppose you use an octal or hexadecimal escape to represent a regexp metacharacter. (See *Note Regular Expression Operators: Regexp Operators.) Does `awk' treat the character as a literal character or as a regexp operator? Historically, such characters were taken literally. (d.c.) However, the POSIX standard indicates that they should be treated as real metacharacters, which is what `gawk' does. In compatibility mode (*note Command-Line Options: Options.), `gawk' treats the characters represented by octal and hexadecimal escape sequences literally when used in regexp constants. Thus, `/a\52b/' is equivalent to `/a\*b/'. Regular Expression Operators ============================ You can combine regular expressions with special characters, called "regular expression operators" or "metacharacters", to increase the power and versatility of regular expressions. The escape sequences described in *Note Escape Sequences::, are valid inside a regexp. They are introduced by a `\' and are recognized and converted into corresponding real characters as the very first step in processing regexps. Here is a list of metacharacters. All characters that are not escape sequences and that are not listed in the table stand for themselves: `\' This is used to suppress the special meaning of a character when matching. For example, `\$' matches the character `$'. `^' This matches the beginning of a string. For example, `^@chapter' matches `@chapter' at the beginning of a string and can be used to identify chapter beginnings in Texinfo source files. The `^' is known as an "anchor", because it anchors the pattern to match only at the beginning of the string. It is important to realize that `^' does not match the beginning of a line embedded in a string. The condition is not true in the following example: if ("line1\nLINE 2" ~ /^L/) ... `$' This is similar to `^', but it matches only at the end of a string. For example, `p$' matches a record that ends with a `p'. The `$' is an anchor and does not match the end of a line embedded in a string. The condition in the following example is not true: if ("line1\nLINE 2" ~ /1$/) ... `.' This matches any single character, _including_ the newline character. For example, `.P' matches any single character followed by a `P' in a string. Using concatenation, we can make a regular expression such as `U.A', which matches any three-character sequence that begins with `U' and ends with `A'. In strict POSIX mode (*note Command-Line Options: Options.), `.' does not match the NUL character, which is a character with all bits equal to zero. Otherwise, NUL is just another character. Other versions of `awk' may not be able to match the NUL character. `[...]' This is called a "character list".(1) It matches any _one_ of the characters that are enclosed in the square brackets. For example, `[MVX]' matches any one of the characters `M', `V', or `X' in a string. A full discussion of what can be inside the square brackets of a character list is given in *Note Using Character Lists: Character Lists. `[^ ...]' This is a "complemented character list". The first character after the `[' _must_ be a `^'. It matches any characters _except_ those in the square brackets. For example, `[^awk]' matches any character that is not an `a', `w', or `k'. `|' This is the "alternation operator" and it is used to specify alternatives. The `|' has the lowest precedence of all the regular expression operators. For example, `^P|[[:digit:]]' matches any string that matches either `^P' or `[[:digit:]]'. This means it matches any string that starts with `P' or contains a digit. The alternation applies to the largest possible regexps on either side. `(...)' Parentheses are used for grouping in regular expressions, as in arithmetic. They can be used to concatenate regular expressions containing the alternation operator, `|'. For example, `@(samp|code)\{[^}]+\}' matches both `@code{foo}' and `@samp{bar}'. (These are Texinfo formatting control sequences.) `*' This symbol means that the preceding regular expression should be repeated as many times as necessary to find a match. For example, `ph*' applies the `*' symbol to the preceding `h' and looks for matches of one `p' followed by any number of `h's. This also matches just `p' if no `h's are present. The `*' repeats the _smallest_ possible preceding expression. (Use parentheses if you want to repeat a larger expression.) It finds as many repetitions as possible. For example, `awk '/\(c[ad][ad]*r x\)/ { print }' sample' prints every record in `sample' containing a string of the form `(car x)', `(cdr x)', `(cadr x)', and so on. Notice the escaping of the parentheses by preceding them with backslashes. `+' This symbol is similar to `*', except that the preceding expression must be matched at least once. This means that `wh+y' would match `why' and `whhy', but not `wy', whereas `wh*y' would match all three of these strings. The following is a simpler way of writing the last `*' example: awk '/\(c[ad]+r x\)/ { print }' sample `?' This symbol is similar to `*', except that the preceding expression can be matched either once or not at all. For example, `fe?d' matches `fed' and `fd', but nothing else. `{N}' `{N,}' `{N,M}' One or two numbers inside braces denote an "interval expression". If there is one number in the braces, the preceding regexp is repeated N times. If there are two numbers separated by a comma, the preceding regexp is repeated N to M times. If there is one number followed by a comma, then the preceding regexp is repeated at least N times: `wh{3}y' Matches `whhhy', but not `why' or `whhhhy'. `wh{3,5}y' Matches `whhhy', `whhhhy', or `whhhhhy', only. `wh{2,}y' Matches `whhy' or `whhhy', and so on. Interval expressions were not traditionally available in `awk'. They were added as part of the POSIX standard to make `awk' and `egrep' consistent with each other. However, because old programs may use `{' and `}' in regexp constants, by default `gawk' does _not_ match interval expressions in regexps. If either `--posix' or `--re-interval' are specified (*note Command-Line Options: Options.), then interval expressions are allowed in regexps. For new programs that use `{' and `}' in regexp constants, it is good practice to always escape them with a backslash. Then the regexp constants are valid and work the way you want them to, using any version of `awk'.(2) In regular expressions, the `*', `+', and `?' operators, as well as the braces `{' and `}', have the highest precedence, followed by concatenation, and finally by `|'. As in arithmetic, parentheses can change how operators are grouped. In POSIX `awk' and `gawk', the `*', `+', and `?' operators stand for themselves when there is nothing in the regexp that precedes them. For example, `/+/' matches a literal plus sign. However, many other versions of `awk' treat such a usage as a syntax error. If `gawk' is in compatibility mode (*note Command-Line Options: Options.), POSIX character classes and interval expressions are not available in regular expressions. ---------- Footnotes ---------- (1) In other literature, you may see a character list referred to as either a "character set", a "character class", or a "bracket expression". (2) Use two backslashes if you're using a string constant with a regexp operator or function. Using Character Lists ===================== Within a character list, a "range expression" consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, using the locale's collating sequence and character set. For example, in the default C locale, `[a-dx-z]' is equivalent to `[abcdxyz]'. Many locales sort characters in dictionary order, and in these locales, `[a-dx-z]' is typically not equivalent to `[abcdxyz]'; instead it might be equivalent to `[aBbCcDdxXyYz]', for example. To obtain the traditional interpretation of bracket expressions, you can use the C locale by setting the `LC_ALL' environment variable to the value `C'. To include one of the characters `\', `]', `-', or `^' in a character list, put a `\' in front of it. For example: [d\]] matches either `d' or `]'. This treatment of `\' in character lists is compatible with other `awk' implementations and is also mandated by POSIX. The regular expressions in `awk' are a superset of the POSIX specification for Extended Regular Expressions (EREs). POSIX EREs are based on the regular expressions accepted by the traditional `egrep' utility. "Character classes" are a new feature introduced in the POSIX standard. A character class is a special notation for describing lists of characters that have a specific attribute, but the actual characters can vary from country to country and/or from character set to character set. For example, the notion of what is an alphabetic character differs between the United States and France. A character class is only valid in a regexp _inside_ the brackets of a character list. Character classes consist of `[:', a keyword denoting the class, and `:]'. Here are the character classes defined by the POSIX standard. `[:alnum:]' Alphanumeric characters. `[:alpha:]' Alphabetic characters. `[:blank:]' Space and TAB characters. `[:cntrl:]' Control characters. `[:digit:]' Numeric characters. `[:graph:]' Characters that are both printable and visible. (A space is printable but not visible, whereas an `a' is both.) `[:lower:]' Lowercase alphabetic characters. `[:print:]' Printable characters (characters that are not control characters). `[:punct:]' Punctuation characters (characters that are not letters, digits, control characters, or space characters). `[:space:]' Space characters (such as space, TAB, and formfeed, to name a few). `[:upper:]' Uppercase alphabetic characters. `[:xdigit:]' Characters that are hexadecimal digits. For example, before the POSIX standard, you had to write `/[A-Za-z0-9]/' to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them, and if your character set collated differently from ASCII, this might not even match the ASCII alphanumeric characters. With the POSIX character classes, you can write `/[[:alnum:]]/' to match the alphabetic and numeric characters in your character set. Two additional special sequences can appear in character lists. These apply to non-ASCII character sets, which can have single symbols (called "collating elements") that are represented with more than one character. They can also have several characters that are equivalent for "collating", or sorting, purposes. (For example, in French, a plain "e" and a grave-accented "e`" are equivalent.) These sequences are: Collating symbols Multicharacter collating elements enclosed between `[.' and `.]'. For example, if `ch' is a collating element, then `[[.ch.]]' is a regexp that matches this collating element, whereas `[ch]' is a regexp that matches either `c' or `h'. Equivalence classes Locale-specific names for a list of characters that are equal. The name is enclosed between `[=' and `=]'. For example, the name `e' might be used to represent all of "e," "e`," and "e'." In this case, `[[=e=]]' is a regexp that matches any of `e', `e'', or `e`'. These features are very valuable in non-English-speaking locales. *Caution:* The library functions that `gawk' uses for regular expression matching currently recognize only POSIX character classes; they do not recognize collating symbols or equivalence classes. `gawk'-Specific Regexp Operators ================================ GNU software that deals with regular expressions provides a number of additional regexp operators. These operators are described in this minor node and are specific to `gawk'; they are not available in other `awk' implementations. Most of the additional operators deal with word matching. For our purposes, a "word" is a sequence of one or more letters, digits, or underscores (`_'): `\w' Matches any word-constituent character--that is, it matches any letter, digit, or underscore. Think of it as shorthand for `[[:alnum:]_]'. `\W' Matches any character that is not word-constituent. Think of it as shorthand for `[^[:alnum:]_]'. `\<' Matches the empty string at the beginning of a word. For example, `/\' Matches the empty string at the end of a word. For example, `/stow\>/' matches `stow' but not `stowaway'. `\y' Matches the empty string at either the beginning or the end of a word (i.e., the word boundar*y*). For example, `\yballs?\y' matches either `ball' or `balls', as a separate word. `\B' Matches the empty string that occurs between two word-constituent characters. For example, `/\Brat\B/' matches `crate' but it does not match `dirty rat'. `\B' is essentially the opposite of `\y'. There are two other operators that work on buffers. In Emacs, a "buffer" is, naturally, an Emacs buffer. For other programs, `gawk''s regexp library routines consider the entire string to match as the buffer. The operators are: `\`' Matches the empty string at the beginning of a buffer (string). `\'' Matches the empty string at the end of a buffer (string). Because `^' and `$' always work in terms of the beginning and end of strings, these operators don't add any new capabilities for `awk'. They are provided for compatibility with other GNU software. In other GNU software, the word-boundary operator is `\b'. However, that conflicts with the `awk' language's definition of `\b' as backspace, so `gawk' uses a different letter. An alternative method would have been to require two backslashes in the GNU operators, but this was deemed too confusing. The current method of using `\y' for the GNU `\b' appears to be the lesser of two evils. The various command-line options (*note Command-Line Options: Options.) control how `gawk' interprets characters in regexps: No options In the default case, `gawk' provides all the facilities of POSIX regexps and the GNU regexp operators described in *Note Regular Expression Operators: Regexp Operators. However, interval expressions are not supported. `--posix' Only POSIX regexps are supported; the GNU operators are not special (e.g., `\w' matches a literal `w'). Interval expressions are allowed. `--traditional' Traditional Unix `awk' regexps are matched. The GNU operators are not special, interval expressions are not available, nor are the POSIX character classes (`[[:alnum:]]', etc.). Characters described by octal and hexadecimal escape sequences are treated literally, even if they represent regexp metacharacters. `--re-interval' Allow interval expressions in regexps, even if `--traditional' has been provided. Case Sensitivity in Matching ============================ Case is normally significant in regular expressions, both when matching ordinary characters (i.e., not metacharacters) and inside character sets. Thus, a `w' in a regular expression matches only a lowercase `w' and not an uppercase `W'. The simplest way to do a case-independent match is to use a character list--for example, `[Ww]'. However, this can be cumbersome if you need to use it often, and it can make the regular expressions harder to read. There are two alternatives that you might prefer. One way to perform a case-insensitive match at a particular point in the program is to convert the data to a single case, using the `tolower' or `toupper' built-in string functions (which we haven't discussed yet; *note String Manipulation Functions: String Functions.). For example: tolower($1) ~ /foo/ { ... } converts the first field to lowercase before matching against it. This works in any POSIX-compliant `awk'. Another method, specific to `gawk', is to set the variable `IGNORECASE' to a nonzero value (*note Built-in Variables::). When `IGNORECASE' is not zero, _all_ regexp and string operations ignore case. Changing the value of `IGNORECASE' dynamically controls the case-sensitivity of the program as it runs. Case is significant by default because `IGNORECASE' (like most variables) is initialized to zero: x = "aB" if (x ~ /ab/) ... # this test will fail IGNORECASE = 1 if (x ~ /ab/) ... # now it will succeed In general, you cannot use `IGNORECASE' to make certain rules case-insensitive and other rules case-sensitive, because there is no straightforward way to set `IGNORECASE' just for the pattern of a particular rule.(1) To do this, use either character lists or `tolower'. However, one thing you can do with `IGNORECASE' only is dynamically turn case-sensitivity on or off for all the rules at once. `IGNORECASE' can be set on the command line or in a `BEGIN' rule (*note Other Command-Line Arguments: Other Arguments.; also *note Startup and Cleanup Actions: Using BEGIN/END.). Setting `IGNORECASE' from the command line is a way to make a program case-insensitive without having to edit it. Prior to `gawk' 3.0, the value of `IGNORECASE' affected regexp operations only. It did not affect string comparison with `==', `!=', and so on. Beginning with version 3.0, both regexp and string comparison operations are also affected by `IGNORECASE'. Beginning with `gawk' 3.0, the equivalences between upper- and lowercase characters are based on the ISO-8859-1 (ISO Latin-1) character set. This character set is a superset of the traditional 128 ASCII characters, which also provides a number of characters suitable for use with European languages. The value of `IGNORECASE' has no effect if `gawk' is in compatibility mode (*note Command-Line Options: Options.). Case is always significant in compatibility mode. ---------- Footnotes ---------- (1) Experienced C and C++ programmers will note that it is possible, using something like `IGNORECASE = 1 && /foObAr/ { ... }' and `IGNORECASE = 0 || /foobar/ { ... }'. However, this is somewhat obscure and we don't recommend it. How Much Text Matches? ====================== Consider the following: echo aaaabcd | awk '{ sub(/a+/, ""); print }' This example uses the `sub' function (which we haven't discussed yet; *note String Manipulation Functions: String Functions.) to make a change to the input record. Here, the regexp `/a+/' indicates "one or more `a' characters," and the replacement text is `'. The input contains four `a' characters. `awk' (and POSIX) regular expressions always match the leftmost, _longest_ sequence of input characters that can match. Thus, all four `a' characters are replaced with `' in this example: $ echo aaaabcd | awk '{ sub(/a+/, ""); print }' -| bcd For simple match/no-match tests, this is not so important. But when doing text matching and substitutions with the `match', `sub', `gsub', and `gensub' functions, it is very important. *Note String Manipulation Functions: String Functions, for more information on these functions. Understanding this principle is also important for regexp-based record and field splitting (*note How Input Is Split into Records: Records., and also *note Specifying How Fields Are Separated: Field Separators.). Using Dynamic Regexps ===================== The righthand side of a `~' or `!~' operator need not be a regexp constant (i.e., a string of characters between slashes). It may be any expression. The expression is evaluated and converted to a string if necessary; the contents of the string are used as the regexp. A regexp that is computed in this way is called a "dynamic regexp": BEGIN { digits_regexp = "[[:digit:]]+" } $0 ~ digits_regexp { print } This sets `digits_regexp' to a regexp that describes one or more digits, and tests whether the input record matches this regexp. When using the `~' and `!~' *Caution:* When using the `~' and `!~' operators, there is a difference between a regexp constant enclosed in slashes and a string constant enclosed in double quotes. If you are going to use a string constant, you have to understand that the string is, in essence, scanned _twice_: the first time when `awk' reads your program, and the second time when it goes to match the string on the lefthand side of the operator with the pattern on the right. This is true of any string-valued expression (such as `digits_regexp', shown previously), not just string constants. What difference does it make if the string is scanned twice? The answer has to do with escape sequences, and particularly with backslashes. To get a backslash into a regular expression inside a string, you have to type two backslashes. For example, `/\*/' is a regexp constant for a literal `*'. Only one backslash is needed. To do the same thing with a string, you have to type `"\\*"'. The first backslash escapes the second one so that the string actually contains the two characters `\' and `*'. Given that you can use both regexp and string constants to describe regular expressions, which should you use? The answer is "regexp constants," for several reasons: * String constants are more complicated to write and more difficult to read. Using regexp constants makes your programs less error-prone. Not understanding the difference between the two kinds of constants is a common source of errors. * It is more efficient to use regexp constants. `awk' can note that you have supplied a regexp and store it internally in a form that makes pattern matching more efficient. When using a string constant, `awk' must first convert the string into this internal form and then perform the pattern matching. * Using regexp constants is better form; it shows clearly that you intend a regexp match. Advanced Notes: Using `\n' in Character Lists of Dynamic Regexps ---------------------------------------------------------------- Some commercial versions of `awk' do not allow the newline character to be used inside a character list for a dynamic regexp: $ awk '$0 ~ "[ \t\n]"' error--> awk: newline in character class [ error--> ]... error--> source line number 1 error--> context is error--> >>> <<< But a newline in a regexp constant works with no problem: $ awk '$0 ~ /[ \t\n]/' here is a sample line -| here is a sample line Ctrl-d `gawk' does not have this problem, and it isn't likely to occur often in practice, but it's worth noting for future reference. Reading Input Files ******************* In the typical `awk' program, all input is read either from the standard input (by default, this is the keyboard, but often it is a pipe from another command) or from files whose names you specify on the `awk' command line. If you specify input files, `awk' reads them in order, processing all the data from one before going on to the next. The name of the current input file can be found in the built-in variable `FILENAME' (*note Built-in Variables::). The input is read in units called "records", and is processed by the rules of your program one record at a time. By default, each record is one line. Each record is automatically split into chunks called "fields". This makes it more convenient for programs to work on the parts of a record. On rare occasions, you may need to use the `getline' command. The `getline' command is valuable, both because it can do explicit input from any number of files, and because the files used with it do not have to be named on the `awk' command line (*note Explicit Input with `getline': Getline.). How Input Is Split into Records =============================== The `awk' utility divides the input for your `awk' program into records and fields. `awk' keeps track of the number of records that have been read so far from the current input file. This value is stored in a built-in variable called `FNR'. It is reset to zero when a new file is started. Another built-in variable, `NR', is the total number of input records read so far from all data files. It starts at zero, but is never automatically reset to zero. Records are separated by a character called the "record separator". By default, the record separator is the newline character. This is why records are, by default, single lines. A different character can be used for the record separator by assigning the character to the built-in variable `RS'. Like any other variable, the value of `RS' can be changed in the `awk' program with the assignment operator, `=' (*note Assignment Expressions: Assignment Ops.). The new record-separator character should be enclosed in quotation marks, which indicate a string constant. Often the right time to do this is at the beginning of execution, before any input is processed, so that the very first record is read with the proper separator. To do this, use the special `BEGIN' pattern (*note The `BEGIN' and `END' Special Patterns: BEGIN/END.). For example: awk 'BEGIN { RS = "/" } { print $0 }' BBS-list changes the value of `RS' to `"/"', before reading any input. This is a string whose first character is a slash; as a result, records are separated by slashes. Then the input file is read, and the second rule in the `awk' program (the action with no pattern) prints each record. Because each `print' statement adds a newline at the end of its output, this `awk' program copies the input with each slash changed to a newline. Here are the results of running the program on `BBS-list': $ awk 'BEGIN { RS = "/" } > { print $0 }' BBS-list -| aardvark 555-5553 1200 -| 300 B -| alpo-net 555-3412 2400 -| 1200 -| 300 A -| barfly 555-7685 1200 -| 300 A -| bites 555-1675 2400 -| 1200 -| 300 A -| camelot 555-0542 300 C -| core 555-2912 1200 -| 300 C -| fooey 555-1234 2400 -| 1200 -| 300 B -| foot 555-6699 1200 -| 300 B -| macfoo 555-6480 1200 -| 300 A -| sdace 555-3430 2400 -| 1200 -| 300 A -| sabafoo 555-2127 1200 -| 300 C -| Note that the entry for the `camelot' BBS is not split. In the original data file (*note Data Files for the Examples: Sample Data Files.), the line looks like this: camelot 555-0542 300 C It has one baud rate only, so there are no slashes in the record, unlike the others which have two or more baud rates. In fact, this record is treated as part of the record for the `core' BBS; the newline separating them in the output is the original newline in the data file, not the one added by `awk' when it printed the record! Another way to change the record separator is on the command line, using the variable-assignment feature (*note Other Command-Line Arguments: Other Arguments.): awk '{ print $0 }' RS="/" BBS-list This sets `RS' to `/' before processing `BBS-list'. Using an unusual character such as `/' for the record separator produces correct behavior in the vast majority of cases. However, the following (extreme) pipeline prints a surprising `1': $ echo | awk 'BEGIN { RS = "a" } ; { print NF }' -| 1 There is one field, consisting of a newline. The value of the built-in variable `NF' is the number of fields in the current record. Reaching the end of an input file terminates the current input record, even if the last character in the file is not the character in `RS'. (d.c.) The empty string `""' (a string without any characters) has a special meaning as the value of `RS'. It means that records are separated by one or more blank lines and nothing else. *Note Multiple-Line Records: Multiple Line, for more details. If you change the value of `RS' in the middle of an `awk' run, the new value is used to delimit subsequent records, but the record currently being processed, as well as records already processed, are not affected. After the end of the record has been determined, `gawk' sets the variable `RT' to the text in the input that matched `RS'. When using `gawk', the value of `RS' is not limited to a one-character string. It can be any regular expression (*note Regular Expressions: Regexp.). In general, each record ends at the next string that matches the regular expression; the next record starts at the end of the matching string. This general rule is actually at work in the usual case, where `RS' contains just a newline: a record ends at the beginning of the next matching string (the next newline in the input), and the following record starts just after the end of this string (at the first character of the following line). The newline, because it matches `RS', is not part of either record. When `RS' is a single character, `RT' contains the same single character. However, when `RS' is a regular expression, `RT' contains the actual input text that matched the regular expression. The following example illustrates both of these features. It sets `RS' equal to a regular expression that matches either a newline or a series of one or more uppercase letters with optional leading and/or trailing whitespace: $ echo record 1 AAAA record 2 BBBB record 3 | > gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" } > { print "Record =", $0, "and RT =", RT }' -| Record = record 1 and RT = AAAA -| Record = record 2 and RT = BBBB -| Record = record 3 and RT = -| The final line of output has an extra blank line. This is because the value of `RT' is a newline, and the `print' statement supplies its own terminating newline. *Note A Simple Stream Editor: Simple Sed, for a more useful example of `RS' as a regexp and `RT'. The use of `RS' as a regular expression and the `RT' variable are `gawk' extensions; they are not available in compatibility mode (*note Command-Line Options: Options.). In compatibility mode, only the first character of the value of `RS' is used to determine the end of the record. Advanced Notes: `RS = "\0"' Is Not Portable ------------------------------------------- There are times when you might want to treat an entire data file as a single record. The only way to make this happen is to give `RS' a value that you know doesn't occur in the input file. This is hard to do in a general way, such that a program always works for arbitrary input files. You might think that for text files, the NUL character, which consists of a character with all bits equal to zero, is a good value to use for `RS' in this case: BEGIN { RS = "\0" } # whole file becomes one record? `gawk' in fact accepts this, and uses the NUL character for the record separator. However, this usage is _not_ portable to other `awk' implementations. All other `awk' implementations(1) store strings internally as C-style strings. C strings use the NUL character as the string terminator. In effect, this means that `RS = "\0"' is the same as `RS = ""'. (d.c.) The best way to treat a whole file as a single record is to simply read the file in, one record at a time, concatenating each record onto the end of the previous ones. ---------- Footnotes ---------- (1) At least that we know about. Examining Fields ================ When `awk' reads an input record, the record is automatically "parsed" or separated by the interpreter into chunks called "fields". By default, fields are separated by "whitespace", like words in a line. Whitespace in `awk' means any string of one or more spaces, tabs, or newlines;(1) other characters, such as formfeed, vertical tab, etc. that are considered whitespace by other languages, are _not_ considered whitespace by `awk'. The purpose of fields is to make it more convenient for you to refer to these pieces of the record. You don't have to use them--you can operate on the whole record if you want--but fields are what make simple `awk' programs so powerful. A dollar-sign (`$') is used to refer to a field in an `awk' program, followed by the number of the field you want. Thus, `$1' refers to the first field, `$2' to the second, and so on. (Unlike the Unix shells, the field numbers are not limited to single digits. `$127' is the one hundred twenty-seventh field in the record.) For example, suppose the following is a line of input: This seems like a pretty nice example. Here the first field, or `$1', is `This', the second field, or `$2', is `seems', and so on. Note that the last field, `$7', is `example.'. Because there is no space between the `e' and the `.', the period is considered part of the seventh field. `NF' is a built-in variable whose value is the number of fields in the current record. `awk' automatically updates the value of `NF' each time it reads a record. No matter how many fields there are, the last field in a record can be represented by `$NF'. So, `$NF' is the same as `$7', which is `example.'. If you try to reference a field beyond the last one (such as `$8' when the record has only seven fields), you get the empty string. (If used in a numeric operation, you get zero.) The use of `$0', which looks like a reference to the "zero-th" field, is a special case: it represents the whole input record when you are not interested in specific fields. Here are some more examples: $ awk '$1 ~ /foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C This example prints each record in the file `BBS-list' whose first field contains the string `foo'. The operator `~' is called a "matching operator" (*note How to Use Regular Expressions: Regexp Usage.); it tests whether a string (here, the field `$1') matches a given regular expression. By contrast, the following example looks for `foo' in _the entire record_ and prints the first field and the last field for each matching input record: $ awk '/foo/ { print $1, $NF }' BBS-list -| fooey B -| foot B -| macfoo A -| sabafoo C ---------- Footnotes ---------- (1) In POSIX `awk', newlines are not considered whitespace for separating fields. Nonconstant Field Numbers ========================= The number of a field does not need to be a constant. Any expression in the `awk' language can be used after a `$' to refer to a field. The value of the expression specifies the field number. If the value is a string, rather than a number, it is converted to a number. Consider this example: awk '{ print $NR }' Recall that `NR' is the number of records read so far: one in the first record, two in the second, etc. So this example prints the first field of the first record, the second field of the second record, and so on. For the twentieth record, field number 20 is printed; most likely, the record has fewer than 20 fields, so this prints a blank line. Here is another example of using expressions as field numbers: awk '{ print $(2*2) }' BBS-list `awk' evaluates the expression `(2*2)' and uses its value as the number of the field to print. The `*' sign represents multiplication, so the expression `2*2' evaluates to four. The parentheses are used so that the multiplication is done before the `$' operation; they are necessary whenever there is a binary operator in the field-number expression. This example, then, prints the hours of operation (the fourth field) for every line of the file `BBS-list'. (All of the `awk' operators are listed, in order of decreasing precedence, in *Note Operator Precedence (How Operators Nest): Precedence.) If the field number you compute is zero, you get the entire record. Thus, `$(2-2)' has the same value as `$0'. Negative field numbers are not allowed; trying to reference one usually terminates the program. (The POSIX standard does not define what happens when you reference a negative field number. `gawk' notices this and terminates your program. Other `awk' implementations may behave differently.) As mentioned in *Note Examining Fields: Fields, `awk' stores the current record's number of fields in the built-in variable `NF' (also *note Built-in Variables::). The expression `$NF' is not a special feature--it is the direct consequence of evaluating `NF' and using its value as a field number. Changing the Contents of a Field ================================ The contents of a field, as seen by `awk', can be changed within an `awk' program; this changes what `awk' perceives as the current input record. (The actual input is untouched; `awk' _never_ modifies the input file.) Consider the following example and its output: $ awk '{ nboxes = $3 ; $3 = $3 - 10 > print nboxes, $3 }' inventory-shipped -| 13 3 -| 15 5 -| 15 5 ... The program first saves the original value of field three in the variable `nboxes'. The `-' sign represents subtraction, so this program reassigns field three, `$3', as the original value of field three minus ten: `$3 - 10'. (*Note Arithmetic Operators: Arithmetic Ops.) Then it prints the original and new values for field three. (Someone in the warehouse made a consistent mistake while inventorying the red boxes.) For this to work, the text in field `$2' must make sense as a number; the string of characters must be converted to a number for the computer to do arithmetic on it. The number resulting from the subtraction is converted back to a string of characters that then becomes field three. *Note Conversion of Strings and Numbers: Conversion. When the value of a field is changed (as perceived by `awk'), the text of the input record is recalculated to contain the new field where the old one was. In other words, `$0' changes to reflect the altered field. Thus, this program prints a copy of the input file, with 10 subtracted from the second field of each line: $ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped -| Jan 3 25 15 115 -| Feb 5 32 24 226 -| Mar 5 24 34 228 ... It is also possible to also assign contents to fields that are out of range. For example: $ awk '{ $6 = ($5 + $4 + $3 + $2) > print $6 }' inventory-shipped -| 168 -| 297 -| 301 ... We've just created `$6', whose value is the sum of fields `$2', `$3', `$4', and `$5'. The `+' sign represents addition. For the file `inventory-shipped', `$6' represents the total number of parcels shipped for a particular month. Creating a new field changes `awk''s internal copy of the current input record, which is the value of `$0'. Thus, if you do `print $0' after adding a field, the record printed includes the new field, with the appropriate number of field separators between it and the previously existing fields. This recomputation affects and is affected by `NF' (the number of fields; *note Examining Fields: Fields.). It is also affected by a feature that has not been discussed yet: the "output field separator", `OFS', used to separate the fields (*note Output Separators::). For example, the value of `NF' is set to the number of the highest field you create. Note, however, that merely _referencing_ an out-of-range field does _not_ change the value of either `$0' or `NF'. Referencing an out-of-range field only produces an empty string. For example: if ($(NF+1) != "") print "can't happen" else print "everything is normal" should print `everything is normal', because `NF+1' is certain to be out of range. (*Note The `if'-`else' Statement: If Statement, for more information about `awk''s `if-else' statements. *Note Variable Typing and Comparison Expressions: Typing and Comparison, for more information about the `!=' operator.) It is important to note that making an assignment to an existing field changes the value of `$0' but does not change the value of `NF', even when you assign the empty string to a field. For example: $ echo a b c d | awk '{ OFS = ":"; $2 = "" > print $0; print NF }' -| a::c:d -| 4 The field is still there; it just has an empty value, denoted by the two colons between `a' and `c'. This example shows what happens if you create a new field: $ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new" > print $0; print NF }' -| a::c:d::new -| 6 The intervening field, `$5', is created with an empty value (indicated by the second pair of adjacent colons), and `NF' is updated with the value six. Decrementing `NF' throws away the values of the fields after the new value of `NF' and recomputes `$0'. (d.c.) Here is an example: $ echo a b c d e f | awk '{ print "NF =", NF; > NF = 3; print $0 }' -| NF = 6 -| a b c *Caution:* Some versions of `awk' don't rebuild `$0' when `NF' is decremented. Caveat emptor. Specifying How Fields Are Separated =================================== The "field separator", which is either a single character or a regular expression, controls the way `awk' splits an input record into fields. `awk' scans the input record for character sequences that match the separator; the fields themselves are the text between the matches. In the examples that follow, we use the bullet symbol (*) to represent spaces in the output. If the field separator is `oo', then the following line: moo goo gai pan is split into three fields: `m', `*g', and `*gai*pan'. Note the leading spaces in the values of the second and third fields. The field separator is represented by the built-in variable `FS'. Shell programmers take note: `awk' does _not_ use the name `IFS' that is used by the POSIX-compliant shells (such as the Unix Bourne shell, `sh', or `bash'). The value of `FS' can be changed in the `awk' program with the assignment operator, `=' (*note Assignment Expressions: Assignment Ops.). Often the right time to do this is at the beginning of execution before any input has been processed, so that the very first record is read with the proper separator. To do this, use the special `BEGIN' pattern (*note The `BEGIN' and `END' Special Patterns: BEGIN/END.). For example, here we set the value of `FS' to the string `","': awk 'BEGIN { FS = "," } ; { print $2 }' Given the input line: John Q. Smith, 29 Oak St., Walamazoo, MI 42139 this `awk' program extracts and prints the string `*29*Oak*St.'. Sometimes the input data contains separator characters that don't separate fields the way you thought they would. For instance, the person's name in the example we just used might have a title or suffix attached, such as: John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139 The same program would extract `*LXIX', instead of `*29*Oak*St.'. If you were expecting the program to print the address, you would be surprised. The moral is to choose your data layout and separator characters carefully to prevent such problems. (If the data is not in a form that is easy to process, perhaps you can massage it first with a separate `awk' program.) Fields are normally separated by whitespace sequences (spaces, tabs, and newlines), not by single spaces. Two spaces in a row do not delimit an empty field. The default value of the field separator `FS' is a string containing a single space, `" "'. If `awk' interpreted this value in the usual way, each space character would separate fields, so two spaces in a row would make an empty field between them. The reason this does not happen is that a single space as the value of `FS' is a special case--it is taken to specify the default manner of delimiting fields. If `FS' is any other single character, such as `","', then each occurrence of that character separates two fields. Two consecutive occurrences delimit an empty field. If the character occurs at the beginning or the end of the line, that too delimits an empty field. The space character is the only single character that does not follow these rules. Using Regular Expressions to Separate Fields -------------------------------------------- The previous node discussed the use of single characters or simple strings as the value of `FS'. More generally, the value of `FS' may be a string containing any regular expression. In this case, each match in the record for the regular expression separates fields. For example, the assignment: FS = ", \t" makes every area of an input line that consists of a comma followed by a space and a TAB into a field separator. (`\t' is an "escape sequence" that stands for a TAB; *note Escape Sequences::, for the complete list of similar escape sequences.) For a less trivial example of a regular expression, try using single spaces to separate fields the way single commas are used. `FS' can be set to `"[ ]"' (left bracket, space, right bracket). This regular expression matches a single space and nothing else (*note Regular Expressions: Regexp.). There is an important difference between the two cases of `FS = " "' (a single space) and `FS = "[ \t\n]+"' (a regular expression matching one or more spaces, tabs, or newlines). For both values of `FS', fields are separated by "runs" (multiple adjacent occurrences) of spaces, tabs, and/or newlines. However, when the value of `FS' is `" "', `awk' first strips leading and trailing whitespace from the record and then decides where the fields are. For example, the following pipeline prints `b': $ echo ' a b c d ' | awk '{ print $2 }' -| b However, this pipeline prints `a' (note the extra spaces around each letter): $ echo ' a b c d ' | awk 'BEGIN { FS = "[ \t\n]+" } > { print $2 }' -| a In this case, the first field is "null" or empty. The stripping of leading and trailing whitespace also comes into play whenever `$0' is recomputed. For instance, study this pipeline: $ echo ' a b c d' | awk '{ print; $2 = $2; print }' -| a b c d -| a b c d The first `print' statement prints the record as it was read, with leading whitespace intact. The assignment to `$2' rebuilds `$0' by concatenating `$1' through `$NF' together, separated by the value of `OFS'. Because the leading whitespace was ignored when finding `$1', it is not part of the new `$0'. Finally, the last `print' statement prints the new `$0'. Making Each Character a Separate Field -------------------------------------- There are times when you may want to examine each character of a record separately. This can be done in `gawk' by simply assigning the null string (`""') to `FS'. In this case, each individual character in the record becomes a separate field. For example: $ echo a b | gawk 'BEGIN { FS = "" } > { > for (i = 1; i <= NF; i = i + 1) > print "Field", i, "is", $i > }' -| Field 1 is a -| Field 2 is -| Field 3 is b Traditionally, the behavior of `FS' equal to `""' was not defined. In this case, most versions of Unix `awk' simply treat the entire record as only having one field. (d.c.) In compatibility mode (*note Command-Line Options: Options.), if `FS' is the null string, then `gawk' also behaves this way. Setting `FS' from the Command Line ---------------------------------- `FS' can be set on the command line. Use the `-F' option to do so. For example: awk -F, 'PROGRAM' INPUT-FILES sets `FS' to the `,' character. Notice that the option uses an uppercase `F' instead of a lowercase `f'. The latter option (`-f') specifies a file containing an `awk' program. Case is significant in command-line options: the `-F' and `-f' options have nothing to do with each other. You can use both options at the same time to set the `FS' variable _and_ get an `awk' program from a file. The value used for the argument to `-F' is processed in exactly the same way as assignments to the built-in variable `FS'. Any special characters in the field separator must be escaped appropriately. For example, to use a `\' as the field separator on the command line, you would have to type: # same as FS = "\\" awk -F\\\\ '...' files ... Because `\' is used for quoting in the shell, `awk' sees `-F\\'. Then `awk' processes the `\\' for escape characters (*note Escape Sequences::), finally yielding a single `\' to use for the field separator. As a special case, in compatibility mode (*note Command-Line Options: Options.), if the argument to `-F' is `t', then `FS' is set to the TAB character. If you type `-F\t' at the shell, without any quotes, the `\' gets deleted, so `awk' figures that you really want your fields to be separated with tabs and not `t's. Use `-v FS="t"' or `-F"[t]"' on the command line if you really do want to separate your fields with `t's. For example, let's use an `awk' program file called `baud.awk' that contains the pattern `/300/' and the action `print $1': /300/ { print $1 } Let's also set `FS' to be the `-' character and run the program on the file `BBS-list'. The following command prints a list of the names of the bulletin boards that operate at 300 baud and the first three digits of their phone numbers: $ awk -F- -f baud.awk BBS-list -| aardvark 555 -| alpo -| barfly 555 -| bites 555 -| camelot 555 -| core 555 -| fooey 555 -| foot 555 -| macfoo 555 -| sdace 555 -| sabafoo 555 Note the second line of output. The second line in the original file looked like this: alpo-net 555-3412 2400/1200/300 A The `-' as part of the system's name was used as the field separator, instead of the `-' in the phone number that was originally intended. This demonstrates why you have to be careful in choosing your field and record separators. Perhaps the most common use of a single character as the field separator occurs when processing the Unix system password file. On many Unix systems, each user has a separate entry in the system password file, one line per user. The information in these lines is separated by colons. The first field is the user's logon name and the second is the user's (encrypted or shadow) password. A password file entry might look like this: arnold:xyzzy:2076:10:Arnold Robbins:/home/arnold:/bin/bash The following program searches the system password file and prints the entries for users who have no password: awk -F: '$2 == ""' /etc/passwd Field-Splitting Summary ----------------------- The following table summarizes how fields are split, based on the value of `FS' (`==' means "is equal to"): `FS == " "' Fields are separated by runs of whitespace. Leading and trailing whitespace are ignored. This is the default. `FS == ANY OTHER SINGLE CHARACTER' Fields are separated by each occurrence of the character. Multiple successive occurrences delimit empty fields, as do leading and trailing occurrences. The character can even be a regexp metacharacter; it does not need to be escaped. `FS == REGEXP' Fields are separated by occurrences of characters that match REGEXP. Leading and trailing matches of REGEXP delimit empty fields. `FS == ""' Each individual character in the record becomes a separate field. (This is a `gawk' extension; it is not specified by the POSIX standard.) Advanced Notes: Changing `FS' Does Not Affect the Fields -------------------------------------------------------- According to the POSIX standard, `awk' is supposed to behave as if each record is split into fields at the time it is read. In particular, this means that if you change the value of `FS' after a record is read, the value of the fields (i.e., how they were split) should reflect the old value of `FS', not the new one. However, many implementations of `awk' do not work this way. Instead, they defer splitting the fields until a field is actually referenced. The fields are split using the _current_ value of `FS'! (d.c.) This behavior can be difficult to diagnose. The following example illustrates the difference between the two methods. (The `sed'(1) command prints just the first line of `/etc/passwd'.) sed 1q /etc/passwd | awk '{ FS = ":" ; print $1 }' which usually prints: root on an incorrect implementation of `awk', while `gawk' prints something like: root:nSijPlPhZZwgE:0:0:Root:/: ---------- Footnotes ---------- (1) The `sed' utility is a "stream editor." Its behavior is also defined by the POSIX standard. Reading Fixed-Width Data ======================== (This minor node discusses an advanced feature of `awk'. If you are a novice `awk' user, you might want to skip it on the first reading.) `gawk' version 2.13 introduced a facility for dealing with fixed-width fields with no distinctive field separator. For example, data of this nature arises in the input for old Fortran programs where numbers are run together, or in the output of programs that did not anticipate the use of their output as input for other programs. An example of the latter is a table where all the columns are lined up by the use of a variable number of spaces and _empty fields are just spaces_. Clearly, `awk''s normal field splitting based on `FS' does not work well in this case. Although a portable `awk' program can use a series of `substr' calls on `$0' (*note String Manipulation Functions: String Functions.), this is awkward and inefficient for a large number of fields. The splitting of an input record into fixed-width fields is specified by assigning a string containing space-separated numbers to the built-in variable `FIELDWIDTHS'. Each number specifies the width of the field, _including_ columns between fields. If you want to ignore the columns between fields, you can specify the width as a separate field that is subsequently ignored. It is a fatal error to supply a field width that is not a positive number. The following data is the output of the Unix `w' utility. It is useful to illustrate the use of `FIELDWIDTHS': 10:06pm up 21 days, 14:04, 23 users User tty login idle JCPU PCPU what hzuo ttyV0 8:58pm 9 5 vi p24.tex hzang ttyV3 6:37pm 50 -csh eklye ttyV5 9:53pm 7 1 em thes.tex dportein ttyV6 8:17pm 1:47 -csh gierd ttyD3 10:00pm 1 elm dave ttyD4 9:47pm 4 4 w brent ttyp0 26Jun91 4:46 26:46 4:41 bash dave ttyq4 26Jun9115days 46 46 wnewmail The following program takes the above input, converts the idle time to number of seconds, and prints out the first two fields and the calculated idle time: *Note:* This program uses a number of `awk' features that haven't been introduced yet. BEGIN { FIELDWIDTHS = "9 6 10 6 7 7 35" } NR > 2 { idle = $4 sub(/^ */, "", idle) # strip leading spaces if (idle == "") idle = 0 if (idle ~ /:/) { split(idle, t, ":") idle = t[1] * 60 + t[2] } if (idle ~ /days/) idle *= 24 * 60 * 60 print $1, $2, idle } Running the program on the data produces the following results: hzuo ttyV0 0 hzang ttyV3 50 eklye ttyV5 0 dportein ttyV6 107 gierd ttyD3 1 dave ttyD4 0 brent ttyp0 286 dave ttyq4 1296000 Another (possibly more practical) example of fixed-width input data is the input from a deck of balloting cards. In some parts of the United States, voters mark their choices by punching holes in computer cards. These cards are then processed to count the votes for any particular candidate or on any particular issue. Because a voter may choose not to vote on some issue, any column on the card may be empty. An `awk' program for processing such data could use the `FIELDWIDTHS' feature to simplify reading the data. (Of course, getting `gawk' to run on a system with card readers is another story!) Assigning a value to `FS' causes `gawk' to use `FS' for field splitting again. Use `FS = FS' to make this happen, without having to know the current value of `FS'. In order to tell which kind of field splitting is in effect, use `PROCINFO["FS"]' (*note Built-in Variables That Convey Information: Auto-set.). The value is `"FS"' if regular field splitting is being used, or it is `"FIELDWIDTHS"' if fixed-width field splitting is being used: if (PROCINFO["FS"] == "FS") REGULAR FIELD SPLITTING ... else FIXED-WIDTH FIELD SPLITTING ... This information is useful when writing a function that needs to temporarily change `FS' or `FIELDWIDTHS', read some records, and then restore the original settings (*note Reading the User Database: Passwd Functions., for an example of such a function). Multiple-Line Records ===================== In some databases, a single line cannot conveniently hold all the information in one entry. In such cases, you can use multiline records. The first step in doing this is to choose your data format. One technique is to use an unusual character or string to separate records. For example, you could use the formfeed character (written `\f' in `awk', as in C) to separate them, making each record a page of the file. To do this, just set the variable `RS' to `"\f"' (a string containing the formfeed character). Any other character could equally well be used, as long as it won't be part of the data in a record. Another technique is to have blank lines separate records. By a special dispensation, an empty string as the value of `RS' indicates that records are separated by one or more blank lines. When `RS' is set to the empty string, each record always ends at the first blank line encountered. The next record doesn't start until the first nonblank line that follows. No matter how many blank lines appear in a row, they all act as one record separator. (Blank lines must be completely empty; lines that contain only whitespace do not count.) You can achieve the same effect as `RS = ""' by assigning the string `"\n\n+"' to `RS'. This regexp matches the newline at the end of the record and one or more blank lines after the record. In addition, a regular expression always matches the longest possible sequence when there is a choice (*note How Much Text Matches?: Leftmost Longest.). So the next record doesn't start until the first nonblank line that follows--no matter how many blank lines appear in a row, they are considered one record separator. There is an important difference between `RS = ""' and `RS = "\n\n+"'. In the first case, leading newlines in the input data file are ignored, and if a file ends without extra blank lines after the last record, the final newline is removed from the record. In the second case, this special processing is not done. (d.c.) Now that the input is separated into records, the second step is to separate the fields in the record. One way to do this is to divide each of the lines into fields in the normal manner. This happens by default as the result of a special feature. When `RS' is set to the empty string, the newline character _always_ acts as a field separator. This is in addition to whatever field separations result from `FS'. The original motivation for this special exception was probably to provide useful behavior in the default case (i.e., `FS' is equal to `" "'). This feature can be a problem if you real