Top (The GNU Awk User’s Guide)

The GNU Awk User’s Guide

Next: Foreword3, Up: (dir)   [Contents][Index]

---

General Introduction

This file documents awk, a program that you can use to select particular records in a file and perform operations upon them.

Copyright © 1989, 1991, 1992, 1993, 1996–2005, 2007, 2009–2020
Free Software Foundation, Inc.

This is Edition 5.1 of GAWK: Effective AWK Programming: A User’s Guide for GNU Awk, for the 5.1.0 (or later) version of the GNU implementation of AWK.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with the Invariant Sections being “GNU General Public License”, with the Front-Cover Texts being “A GNU Manual”, and with the Back-Cover Texts as in (a) below. A copy of the license is included in the section entitled “GNU Free Documentation License”.

1. The FSF’s Back-Cover Text is: “You have the freedom to copy and modify this GNU manual.”

• Foreword3		Some nice words about this Web page.
• Foreword4		More nice words.
• Preface		What this Web page is about; brief history and acknowledgments.
• Getting Started		A basic introduction to using awk. How to run an awk program. Command-line syntax.
• Invoking Gawk		How to run gawk.
• Regexp		All about matching things using regular expressions.
• Reading Files		How to read files and manipulate fields.
• Printing		How to print using awk. Describes the print and printf statements. Also describes redirection of output.
• Expressions		Expressions are the basic building blocks of statements.
• Patterns and Actions		Overviews of patterns and actions.
• Arrays		The description and use of arrays. Also includes array-oriented control statements.
• Functions		Built-in and user-defined functions.
• Library Functions		A Library of awk Functions.
• Sample Programs		Many awk programs with complete explanations.
• Advanced Features		Stuff for advanced users, specific to gawk.
• Internationalization		Getting gawk to speak your language.
• Debugger		The gawk debugger.
• Namespaces		How namespaces work in gawk.
• Arbitrary Precision Arithmetic		Arbitrary precision arithmetic with gawk.
• Dynamic Extensions		Adding new built-in functions to gawk.
• Language History		The evolution of the awk language.
• Installation		Installing gawk under various operating systems.
• Notes		Notes about adding things to gawk and possible future work.
• Basic Concepts		A very quick introduction to programming concepts.
• Glossary		An explanation of some unfamiliar terms.
• Copying		Your right to copy and distribute gawk.
• GNU Free Documentation License		The license for this Web page.
• Index		Concept and Variable Index.
• History		The history of gawk and awk.
• Names		What name to use to find awk.
• This Manual		Using this Web page. Includes sample input files that you can use.
• Conventions		Typographical Conventions.
• Manual History		Brief history of the GNU project and this Web page.
• How To Contribute		Helping to save the world.
• Acknowledgments		Acknowledgments.
• Running gawk		How to run gawk programs; includes command-line syntax.
• One-shot		Running a short throwaway awk program.
• Read Terminal		Using no input files (input from the keyboard instead).
• Long		Putting permanent awk programs in files.
• Executable Scripts		Making self-contained awk programs.
• Comments		Adding documentation to gawk programs.
• Quoting		More discussion of shell quoting issues.
• DOS Quoting		Quoting in Windows Batch Files.
• Sample Data Files		Sample data files for use in the awk programs illustrated in this Web page.
• Very Simple		A very simple example.
• Two Rules		A less simple one-line example using two rules.
• More Complex		A more complex example.
• Statements/Lines		Subdividing or combining statements into lines.
• Other Features		Other Features of awk.
• When		When to use gawk and when to use other things.
• Intro Summary		Summary of the introduction.
• Command Line		How to run awk.
• Options		Command-line options and their meanings.
• Other Arguments		Input file names and variable assignments.
• Naming Standard Input		How to specify standard input with other files.
• Environment Variables		The environment variables gawk uses.
• AWKPATH Variable		Searching directories for awk programs.
• AWKLIBPATH Variable		Searching directories for awk shared libraries.
• Other Environment Variables		The environment variables.
• Exit Status		gawk’s exit status.
• Include Files		Including other files into your program.
• Loading Shared Libraries		Loading shared libraries into your program.
• Obsolete		Obsolete Options and/or features.
• Undocumented		Undocumented Options and Features.
• Invoking Summary		Invocation summary.
• Regexp Usage		How to Use Regular Expressions.
• Escape Sequences		How to write nonprinting characters.
• Regexp Operators		Regular Expression Operators.
• Regexp Operator Details		The actual details.
• Interval Expressions		Notes on interval expressions.
• Bracket Expressions		What can go between ‘[...]’.
• Leftmost Longest		How much text matches.
• Computed Regexps		Using Dynamic Regexps.
• GNU Regexp Operators		Operators specific to GNU software.
• Case-sensitivity		How to do case-insensitive matching.
• Regexp Summary		Regular expressions summary.
• Records		Controlling how data is split into records.
• awk split records		How standard awk splits records.
• gawk split records		How gawk splits records.
• Fields		An introduction to fields.
• Nonconstant Fields		Nonconstant Field Numbers.
• Changing Fields		Changing the Contents of a Field.
• Field Separators		The field separator and how to change it.
• Default Field Splitting		How fields are normally separated.
• Regexp Field Splitting		Using regexps as the field separator.
• Single Character Fields		Making each character a separate field.
• Command Line Field Separator		Setting FS from the command line.
• Full Line Fields		Making the full line be a single field.
• Field Splitting Summary		Some final points and a summary table.
• Constant Size		Reading constant width data.
• Fixed width data		Processing fixed-width data.
• Skipping intervening		Skipping intervening fields.
• Allowing trailing data		Capturing optional trailing data.
• Fields with fixed data		Field values with fixed-width data.
• Splitting By Content		Defining Fields By Content
• More CSV		More on CSV files.
• Testing field creation		Checking how gawk is splitting records.
• Multiple Line		Reading multiline records.
• Getline		Reading files under explicit program control using the getline function.
• Plain Getline		Using getline with no arguments.
• Getline/Variable		Using getline into a variable.
• Getline/File		Using getline from a file.
• Getline/Variable/File		Using getline into a variable from a file.
• Getline/Pipe		Using getline from a pipe.
• Getline/Variable/Pipe		Using getline into a variable from a pipe.
• Getline/Coprocess		Using getline from a coprocess.
• Getline/Variable/Coprocess		Using getline into a variable from a coprocess.
• Getline Notes		Important things to know about getline.
• Getline Summary		Summary of getline Variants.
• Read Timeout		Reading input with a timeout.
• Retrying Input		Retrying input after certain errors.
• Command-line directories		What happens if you put a directory on the command line.
• Input Summary		Input summary.
• Input Exercises		Exercises.
• Print		The print statement.
• Print Examples		Simple examples of print statements.
• Output Separators		The output separators and how to change them.
• OFMT		Controlling Numeric Output With print.
• Printf		The printf statement.
• Basic Printf		Syntax of the printf statement.
• Control Letters		Format-control letters.
• Format Modifiers		Format-specification modifiers.
• Printf Examples		Several examples.
• Redirection		How to redirect output to multiple files and pipes.
• Special FD		Special files for I/O.
• Special Files		File name interpretation in gawk. gawk allows access to inherited file descriptors.
• Other Inherited Files		Accessing other open files with gawk.
• Special Network		Special files for network communications.
• Special Caveats		Things to watch out for.
• Close Files And Pipes		Closing Input and Output Files and Pipes.
• Nonfatal		Enabling Nonfatal Output.
• Output Summary		Output summary.
• Output Exercises		Exercises.
• Values		Constants, Variables, and Regular Expressions.
• Constants		String, numeric and regexp constants.
• Scalar Constants		Numeric and string constants.
• Nondecimal-numbers		What are octal and hex numbers.
• Regexp Constants		Regular Expression constants.
• Using Constant Regexps		When and how to use a regexp constant.
• Standard Regexp Constants		Regexp constants in standard awk.
• Strong Regexp Constants		Strongly typed regexp constants.
• Variables		Variables give names to values for later use.
• Using Variables		Using variables in your programs.
• Assignment Options		Setting variables on the command line and a summary of command-line syntax. This is an advanced method of input.
• Conversion		The conversion of strings to numbers and vice versa.
• Strings And Numbers		How awk Converts Between Strings And Numbers.
• Locale influences conversions		How the locale may affect conversions.
• All Operators		gawk’s operators.
• Arithmetic Ops		Arithmetic operations (‘+’, ‘-’, etc.)
• Concatenation		Concatenating strings.
• Assignment Ops		Changing the value of a variable or a field.
• Increment Ops		Incrementing the numeric value of a variable.
• Truth Values and Conditions		Testing for true and false.
• Truth Values		What is “true” and what is “false”.
• Typing and Comparison		How variables acquire types and how this affects comparison of numbers and strings with ‘<’, etc.
• Variable Typing		String type versus numeric type.
• Comparison Operators		The comparison operators.
• POSIX String Comparison		String comparison with POSIX rules.
• Boolean Ops		Combining comparison expressions using boolean operators ‘	’ (“or”), ‘&&’ (“and”) and ‘!’ (“not”).
• Conditional Exp		Conditional expressions select between two subexpressions under control of a third subexpression.
• Function Calls		A function call is an expression.
• Precedence		How various operators nest.
• Locales		How the locale affects things.
• Expressions Summary		Expressions summary.
• Pattern Overview		What goes into a pattern.
• Regexp Patterns		Using regexps as patterns.
• Expression Patterns		Any expression can be used as a pattern.
• Ranges		Pairs of patterns specify record ranges.
• BEGIN/END		Specifying initialization and cleanup rules.
• Using BEGIN/END		How and why to use BEGIN/END rules.
• I/O And BEGIN/END		I/O issues in BEGIN/END rules.
• BEGINFILE/ENDFILE		Two special patterns for advanced control.
• Empty		The empty pattern, which matches every record.
• Using Shell Variables		How to use shell variables with awk.
• Action Overview		What goes into an action.
• Statements		Describes the various control statements in detail.
• If Statement		Conditionally execute some awk statements.
• While Statement		Loop until some condition is satisfied.
• Do Statement		Do specified action while looping until some condition is satisfied.
• For Statement		Another looping statement, that provides initialization and increment clauses.
• Switch Statement		Switch/case evaluation for conditional execution of statements based on a value.
• Break Statement		Immediately exit the innermost enclosing loop.
• Continue Statement		Skip to the end of the innermost enclosing loop.
• Next Statement		Stop processing the current input record.
• Nextfile Statement		Stop processing the current file.
• Exit Statement		Stop execution of awk.
• Built-in Variables		Summarizes the predefined variables.
• User-modified		Built-in variables that you change to control awk.
• Auto-set		Built-in variables where awk gives you information.
• ARGC and ARGV		Ways to use ARGC and ARGV.
• Pattern Action Summary		Patterns and Actions summary.
• Array Basics		The basics of arrays.
• Array Intro		Introduction to Arrays
• Reference to Elements		How to examine one element of an array.
• Assigning Elements		How to change an element of an array.
• Array Example		Basic Example of an Array
• Scanning an Array		A variation of the for statement. It loops through the indices of an array’s existing elements.
• Controlling Scanning		Controlling the order in which arrays are scanned.
• Numeric Array Subscripts		How to use numbers as subscripts in awk.
• Uninitialized Subscripts		Using Uninitialized variables as subscripts.
• Delete		The delete statement removes an element from an array.
• Multidimensional		Emulating multidimensional arrays in awk.
• Multiscanning		Scanning multidimensional arrays.
• Arrays of Arrays		True multidimensional arrays.
• Arrays Summary		Summary of arrays.
• Built-in		Summarizes the built-in functions.
• Calling Built-in		How to call built-in functions.
• Numeric Functions		Functions that work with numbers, including int(), sin() and rand().
• String Functions		Functions for string manipulation, such as split(), match() and sprintf().
• Gory Details		More than you want to know about ‘\’ and ‘&’ with sub(), gsub(), and gensub().
• I/O Functions		Functions for files and shell commands.
• Time Functions		Functions for dealing with timestamps.
• Bitwise Functions		Functions for bitwise operations.
• Type Functions		Functions for type information.
• I18N Functions		Functions for string translation.
• User-defined		Describes User-defined functions in detail.
• Definition Syntax		How to write definitions and what they mean.
• Function Example		An example function definition and what it does.
• Function Calling		Calling user-defined functions.
• Calling A Function		Don’t use spaces.
• Variable Scope		Controlling variable scope.
• Pass By Value/Reference		Passing parameters.
• Function Caveats		Other points to know about functions.
• Return Statement		Specifying the value a function returns.
• Dynamic Typing		How variable types can change at runtime.
• Indirect Calls		Choosing the function to call at runtime.
• Functions Summary		Summary of functions.
• Library Names		How to best name private global variables in library functions.
• General Functions		Functions that are of general use.
• Strtonum Function		A replacement for the built-in strtonum() function.
• Assert Function		A function for assertions in awk programs.
• Round Function		A function for rounding if sprintf() does not do it correctly.
• Cliff Random Function		The Cliff Random Number Generator.
• Ordinal Functions		Functions for using characters as numbers and vice versa.
• Join Function		A function to join an array into a string.
• Getlocaltime Function		A function to get formatted times.
• Readfile Function		A function to read an entire file at once.
• Shell Quoting		A function to quote strings for the shell.
• Data File Management		Functions for managing command-line data files.
• Filetrans Function		A function for handling data file transitions.
• Rewind Function		A function for rereading the current file.
• File Checking		Checking that data files are readable.
• Empty Files		Checking for zero-length files.
• Ignoring Assigns		Treating assignments as file names.
• Getopt Function		A function for processing command-line arguments.
• Passwd Functions		Functions for getting user information.
• Group Functions		Functions for getting group information.
• Walking Arrays		A function to walk arrays of arrays.
• Library Functions Summary		Summary of library functions.
• Library Exercises		Exercises.
• Running Examples		How to run these examples.
• Clones		Clones of common utilities.
• Cut Program		The cut utility.
• Egrep Program		The egrep utility.
• Id Program		The id utility.
• Split Program		The split utility.
• Tee Program		The tee utility.
• Uniq Program		The uniq utility.
• Wc Program		The wc utility.
• Miscellaneous Programs		Some interesting awk programs.
• Dupword Program		Finding duplicated words in a document.
• Alarm Program		An alarm clock.
• Translate Program		A program similar to the tr utility.
• Labels Program		Printing mailing labels.
• Word Sorting		A program to produce a word usage count.
• History Sorting		Eliminating duplicate entries from a history file.
• Extract Program		Pulling out programs from Texinfo source files.
• Simple Sed		A Simple Stream Editor.
• Igawk Program		A wrapper for awk that includes files.
• Anagram Program		Finding anagrams from a dictionary.
• Signature Program		People do amazing things with too much time on their hands.
• Programs Summary		Summary of programs.
• Programs Exercises		Exercises.
• Nondecimal Data		Allowing nondecimal input data.
• Array Sorting		Facilities for controlling array traversal and sorting arrays.
• Controlling Array Traversal		How to use PROCINFO["sorted_in"].
• Array Sorting Functions		How to use asort() and asorti().
• Two-way I/O		Two-way communications with another process.
• TCP/IP Networking		Using gawk for network programming.
• Profiling		Profiling your awk programs.
• Advanced Features Summary		Summary of advanced features.
• I18N and L10N		Internationalization and Localization.
• Explaining gettext		How GNU gettext works.
• Programmer i18n		Features for the programmer.
• Translator i18n		Features for the translator.
• String Extraction		Extracting marked strings.
• Printf Ordering		Rearranging printf arguments.
• I18N Portability		awk-level portability issues.
• I18N Example		A simple i18n example.
• Gawk I18N		gawk is also internationalized.
• I18N Summary		Summary of I18N stuff.
• Debugging		Introduction to gawk debugger.
• Debugging Concepts		Debugging in General.
• Debugging Terms		Additional Debugging Concepts.
• Awk Debugging		Awk Debugging.
• Sample Debugging Session		Sample debugging session.
• Debugger Invocation		How to Start the Debugger.
• Finding The Bug		Finding the Bug.
• List of Debugger Commands		Main debugger commands.
• Breakpoint Control		Control of Breakpoints.
• Debugger Execution Control		Control of Execution.
• Viewing And Changing Data		Viewing and Changing Data.
• Execution Stack		Dealing with the Stack.
• Debugger Info		Obtaining Information about the Program and the Debugger State.
• Miscellaneous Debugger Commands		Miscellaneous Commands.
• Readline Support		Readline support.
• Limitations		Limitations and future plans.
• Debugging Summary		Debugging summary.
• Global Namespace		The global namespace in standard awk.
• Qualified Names		How to qualify names with a namespace.
• Default Namespace		The default namespace.
• Changing The Namespace		How to change the namespace.
• Naming Rules		Namespace and Component Naming Rules.
• Internal Name Management		How names are stored internally.
• Namespace Example		An example of code using a namespace.
• Namespace And Features		Namespaces and other gawk features.
• Namespace Summary		Summarizing namespaces.
• Computer Arithmetic		A quick intro to computer math.
• Math Definitions		Defining terms used.
• MPFR features		The MPFR features in gawk.
• FP Math Caution		Things to know.
• Inexactness of computations		Floating point math is not exact.
• Inexact representation		Numbers are not exactly represented.
• Comparing FP Values		How to compare floating point values.
• Errors accumulate		Errors get bigger as they go.
• Getting Accuracy		Getting more accuracy takes some work.
• Try To Round		Add digits and round.
• Setting precision		How to set the precision.
• Setting the rounding mode		How to set the rounding mode.
• Arbitrary Precision Integers		Arbitrary Precision Integer Arithmetic with gawk.
• Checking for MPFR		How to check if MPFR is available.
• POSIX Floating Point Problems		Standards Versus Existing Practice.
• Floating point summary		Summary of floating point discussion.
• Extension Intro		What is an extension.
• Plugin License		A note about licensing.
• Extension Mechanism Outline		An outline of how it works.
• Extension API Description		A full description of the API.
• Extension API Functions Introduction		Introduction to the API functions.
• General Data Types		The data types.
• Memory Allocation Functions		Functions for allocating memory.
• Constructor Functions		Functions for creating values.
• Registration Functions		Functions to register things with gawk.
• Extension Functions		Registering extension functions.
• Exit Callback Functions		Registering an exit callback.
• Extension Version String		Registering a version string.
• Input Parsers		Registering an input parser.
• Output Wrappers		Registering an output wrapper.
• Two-way processors		Registering a two-way processor.
• Printing Messages		Functions for printing messages.
• Updating ERRNO		Functions for updating ERRNO.
• Requesting Values		How to get a value.
• Accessing Parameters		Functions for accessing parameters.
• Symbol Table Access		Functions for accessing global variables.
• Symbol table by name		Accessing variables by name.
• Symbol table by cookie		Accessing variables by “cookie”.
• Cached values		Creating and using cached values.
• Array Manipulation		Functions for working with arrays.
• Array Data Types		Data types for working with arrays.
• Array Functions		Functions for working with arrays.
• Flattening Arrays		How to flatten arrays.
• Creating Arrays		How to create and populate arrays.
• Redirection API		How to access and manipulate redirections.
• Extension API Variables		Variables provided by the API.
• Extension Versioning		API Version information.
• Extension GMP/MPFR Versioning		Version information about GMP and MPFR.
• Extension API Informational Variables		Variables providing information about gawk’s invocation.
• Extension API Boilerplate		Boilerplate code for using the API.
• Changes from API V1		Changes from V1 of the API.
• Finding Extensions		How gawk finds compiled extensions.
• Extension Example		Example C code for an extension.
• Internal File Description		What the new functions will do.
• Internal File Ops		The code for internal file operations.
• Using Internal File Ops		How to use an external extension.
• Extension Samples		The sample extensions that ship with gawk.
• Extension Sample File Functions		The file functions sample.
• Extension Sample Fnmatch		An interface to fnmatch().
• Extension Sample Fork		An interface to fork() and other process functions.
• Extension Sample Inplace		Enabling in-place file editing.
• Extension Sample Ord		Character to value to character conversions.
• Extension Sample Readdir		An interface to readdir().
• Extension Sample Revout		Reversing output sample output wrapper.
• Extension Sample Rev2way		Reversing data sample two-way processor.
• Extension Sample Read write array		Serializing an array to a file.
• Extension Sample Readfile		Reading an entire file into a string.
• Extension Sample Time		An interface to gettimeofday() and sleep().
• Extension Sample API Tests		Tests for the API.
• gawkextlib		The gawkextlib project.
• Extension summary		Extension summary.
• Extension Exercises		Exercises.
• V7/SVR3.1		The major changes between V7 and System V Release 3.1.
• SVR4		Minor changes between System V Releases 3.1 and 4.
• POSIX		New features from the POSIX standard.
• BTL		New features from Brian Kernighan’s version of awk.
• POSIX/GNU		The extensions in gawk not in POSIX awk.
• Feature History		The history of the features in gawk.
• Common Extensions		Common Extensions Summary.
• Ranges and Locales		How locales used to affect regexp ranges.
• Contributors		The major contributors to gawk.
• History summary		History summary.
• Gawk Distribution		What is in the gawk distribution.
• Getting		How to get the distribution.
• Extracting		How to extract the distribution.
• Distribution contents		What is in the distribution.
• Unix Installation		Installing gawk under various versions of Unix.
• Quick Installation		Compiling gawk under Unix.
• Shell Startup Files		Shell convenience functions.
• Additional Configuration Options		Other compile-time options.
• Configuration Philosophy		How it’s all supposed to work.
• Non-Unix Installation		Installation on Other Operating Systems.
• PC Installation		Installing and Compiling gawk on Microsoft Windows.
• PC Binary Installation		Installing a prepared distribution.
• PC Compiling		Compiling gawk for Windows32.
• PC Using		Running gawk on Windows32.
• Cygwin		Building and running gawk for Cygwin.
• MSYS		Using gawk In The MSYS Environment.
• VMS Installation		Installing gawk on VMS.
• VMS Compilation		How to compile gawk under VMS.
• VMS Dynamic Extensions		Compiling gawk dynamic extensions on VMS.
• VMS Installation Details		How to install gawk under VMS.
• VMS Running		How to run gawk under VMS.
• VMS GNV		The VMS GNV Project.
• VMS Old Gawk		An old version comes with some VMS systems.
• Bugs		Reporting Problems and Bugs.
• Bug address		Where to send reports to.
• Usenet		Where not to send reports to.
• Maintainers		Maintainers of non-*nix ports.
• Other Versions		Other freely available awk implementations.
• Installation summary		Summary of installation.
• Compatibility Mode		How to disable certain gawk extensions.
• Additions		Making Additions To gawk.
• Accessing The Source		Accessing the Git repository.
• Adding Code		Adding code to the main body of gawk.
• New Ports		Porting gawk to a new operating system.
• Derived Files		Why derived files are kept in the Git repository.
• Future Extensions		New features that may be implemented one day.
• Implementation Limitations		Some limitations of the implementation.
• Extension Design		Design notes about the extension API.
• Old Extension Problems		Problems with the old mechanism.
• Extension New Mechanism Goals		Goals for the new mechanism.
• Extension Other Design Decisions		Some other design decisions.
• Extension Future Growth		Some room for future growth.
• Notes summary		Summary of implementation notes.
• Basic High Level		The high level view.
• Basic Data Typing		A very quick intro to data types.

Short Table of Contents

• ** Foreword to the Third Edition
  • Foreword to the Fourth Edition
  • Preface
• Part I:
  

The awk Language

• • 1 Getting Started with awk
  • 2 Running awk and gawk
  • 3 Regular Expressions
  • 4 Reading Input Files
  • 5 Printing Output
  • 6 Expressions
  • 7 Patterns, Actions, and Variables
  • 8 Arrays in awk
  • 9 Functions
• Part II:
  

Problem Solving with awk

• • 10 A Library of awk Functions
  • 11 Practical awk Programs
• Part III:
  

Moving Beyond Standard awk with gawk

• • 12 Advanced Features of gawk
  • 13 Internationalization with gawk
  • 14 Debugging awk Programs
  • 15 Namespaces in gawk
  • 16 Arithmetic and Arbitrary-Precision Arithmetic with gawk
  • 17 Writing Extensions for gawk
• Part IV:
  

Appendices

• • Appendix A The Evolution of the awk Language
  • Appendix B Installing gawk
  • Appendix C Implementation Notes
  • Appendix D Basic Programming Concepts
  • Glossary
  • GNU General Public License
  • GNU Free Documentation License
  • Index

Table of Contents

• ** Foreword to the Third Edition
  • Foreword to the Fourth Edition
  • Preface
    • History of awk and gawk
    • A Rose by Any Other Name
    • Using This Book
    • Typographical Conventions
      • Dark Corners
    • The GNU Project and This Book
    • How to Contribute
    • Acknowledgments
• Part I:
  The awk Language
  • 1 Getting Started with awk
    • 1.1 How to Run awk Programs
      • 1.1.1 One-Shot Throwaway awk Programs
      • 1.1.2 Running awk Without Input Files
      • 1.1.3 Running Long Programs
      • 1.1.4 Executable awk Programs
      • 1.1.5 Comments in awk Programs
      • 1.1.6 Shell Quoting Issues
        • 1.1.6.1 Quoting in MS-Windows Batch Files
    • 1.2 Data files for the Examples
    • 1.3 Some Simple Examples
    • 1.4 An Example with Two Rules
    • 1.5 A More Complex Example
    • 1.6 awk Statements Versus Lines
    • 1.7 Other Features of awk
    • 1.8 When to Use awk
    • 1.9 Summary
  • 2 Running awk and gawk
    • 2.1 Invoking awk
    • 2.2 Command-Line Options
    • 2.3 Other Command-Line Arguments
    • 2.4 Naming Standard Input
    • 2.5 The Environment Variables gawk Uses
      • 2.5.1 The AWKPATH Environment Variable
      • 2.5.2 The AWKLIBPATH Environment Variable
      • 2.5.3 Other Environment Variables
    • 2.6 gawk’s Exit Status
    • 2.7 Including Other Files into Your Program
    • 2.8 Loading Dynamic Extensions into Your Program
    • 2.9 Obsolete Options and/or Features
    • 2.10 Undocumented Options and Features
    • 2.11 Summary
  • 3 Regular Expressions
    • 3.1 How to Use Regular Expressions
    • 3.2 Escape Sequences
    • 3.3 Regular Expression Operators
      • 3.3.1 Regexp Operators in awk
      • 3.3.2 Some Notes On Interval Expressions
    • 3.4 Using Bracket Expressions
    • 3.5 How Much Text Matches?
    • 3.6 Using Dynamic Regexps
    • 3.7 gawk-Specific Regexp Operators
    • 3.8 Case Sensitivity in Matching
    • 3.9 Summary
  • 4 Reading Input Files
    • 4.1 How Input Is Split into Records
      • 4.1.1 Record Splitting with Standard awk
      • 4.1.2 Record Splitting with gawk
    • 4.2 Examining Fields
    • 4.3 Nonconstant Field Numbers
    • 4.4 Changing the Contents of a Field
    • 4.5 Specifying How Fields Are Separated
      • 4.5.1 Whitespace Normally Separates Fields
      • 4.5.2 Using Regular Expressions to Separate Fields
      • 4.5.3 Making Each Character a Separate Field
      • 4.5.4 Setting FS from the Command Line
      • 4.5.5 Making the Full Line Be a Single Field
      • 4.5.6 Field-Splitting Summary
    • 4.6 Reading Fixed-Width Data
      • 4.6.1 Processing Fixed-Width Data
      • 4.6.2 Skipping Intervening Fields
      • 4.6.3 Capturing Optional Trailing Data
      • 4.6.4 Field Values With Fixed-Width Data
    • 4.7 Defining Fields by Content
      • 4.7.1 More on CSV Files
    • 4.8 Checking How gawk Is Splitting Records
    • 4.9 Multiple-Line Records
    • 4.10 Explicit Input with getline
      • 4.10.1 Using getline with No Arguments
      • 4.10.2 Using getline into a Variable
      • 4.10.3 Using getline from a File
      • 4.10.4 Using getline into a Variable from a File
      • 4.10.5 Using getline from a Pipe
      • 4.10.6 Using getline into a Variable from a Pipe
      • 4.10.7 Using getline from a Coprocess
      • 4.10.8 Using getline into a Variable from a Coprocess
      • 4.10.9 Points to Remember About getline
      • 4.10.10 Summary of getline Variants
    • 4.11 Reading Input with a Timeout
    • 4.12 Retrying Reads After Certain Input Errors
    • 4.13 Directories on the Command Line
    • 4.14 Summary
    • 4.15 Exercises
  • 5 Printing Output
    • 5.1 The print Statement
    • 5.2 print Statement Examples
    • 5.3 Output Separators
    • 5.4 Controlling Numeric Output with print
    • 5.5 Using printf Statements for Fancier Printing
      • 5.5.1 Introduction to the printf Statement
      • 5.5.2 Format-Control Letters
      • 5.5.3 Modifiers for printf Formats
      • 5.5.4 Examples Using printf
    • 5.6 Redirecting Output of print and printf
    • 5.7 Special Files for Standard Preopened Data Streams
    • 5.8 Special File names in gawk
      • 5.8.1 Accessing Other Open Files with gawk
      • 5.8.2 Special Files for Network Communications
      • 5.8.3 Special File name Caveats
    • 5.9 Closing Input and Output Redirections
    • 5.10 Enabling Nonfatal Output
    • 5.11 Summary
    • 5.12 Exercises
  • 6 Expressions
    • 6.1 Constants, Variables, and Conversions
      • 6.1.1 Constant Expressions
        • 6.1.1.1 Numeric and String Constants
        • 6.1.1.2 Octal and Hexadecimal Numbers
        • 6.1.1.3 Regular Expression Constants
      • 6.1.2 Using Regular Expression Constants
        • 6.1.2.1 Standard Regular Expression Constants
        • 6.1.2.2 Strongly Typed Regexp Constants
      • 6.1.3 Variables
        • 6.1.3.1 Using Variables in a Program
        • 6.1.3.2 Assigning Variables on the Command Line
      • 6.1.4 Conversion of Strings and Numbers
        • 6.1.4.1 How awk Converts Between Strings and Numbers
        • 6.1.4.2 Locales Can Influence Conversion
    • 6.2 Operators: Doing Something with Values
      • 6.2.1 Arithmetic Operators
      • 6.2.2 String Concatenation
      • 6.2.3 Assignment Expressions
      • 6.2.4 Increment and Decrement Operators
    • 6.3 Truth Values and Conditions
      • 6.3.1 True and False in awk
      • 6.3.2 Variable Typing and Comparison Expressions
        • 6.3.2.1 String Type versus Numeric Type
        • 6.3.2.2 Comparison Operators
        • 6.3.2.3 String Comparison Based on Locale Collating Order
      • 6.3.3 Boolean Expressions
      • 6.3.4 Conditional Expressions
    • 6.4 Function Calls
    • 6.5 Operator Precedence (How Operators Nest)
    • 6.6 Where You Are Makes a Difference
    • 6.7 Summary
  • 7 Patterns, Actions, and Variables
    • 7.1 Pattern Elements
      • 7.1.1 Regular Expressions as Patterns
      • 7.1.2 Expressions as Patterns
      • 7.1.3 Specifying Record Ranges with Patterns
      • 7.1.4 The BEGIN and END Special Patterns
        • 7.1.4.1 Startup and Cleanup Actions
        • 7.1.4.2 Input/Output from BEGIN and END Rules
      • 7.1.5 The BEGINFILE and ENDFILE Special Patterns
      • 7.1.6 The Empty Pattern
    • 7.2 Using Shell Variables in Programs
    • 7.3 Actions
    • 7.4 Control Statements in Actions
      • 7.4.1 The if-else Statement
      • 7.4.2 The while Statement
      • 7.4.3 The do-while Statement
      • 7.4.4 The for Statement
      • 7.4.5 The switch Statement
      • 7.4.6 The break Statement
      • 7.4.7 The continue Statement
      • 7.4.8 The next Statement
      • 7.4.9 The nextfile Statement
      • 7.4.10 The exit Statement
    • 7.5 Predefined Variables
      • 7.5.1 Built-in Variables That Control awk
      • 7.5.2 Built-in Variables That Convey Information
      • 7.5.3 Using ARGC and ARGV
    • 7.6 Summary
  • 8 Arrays in awk
    • 8.1 The Basics of Arrays
      • 8.1.1 Introduction to Arrays
      • 8.1.2 Referring to an Array Element
      • 8.1.3 Assigning Array Elements
      • 8.1.4 Basic Array Example
      • 8.1.5 Scanning All Elements of an Array
      • 8.1.6 Using Predefined Array Scanning Orders with gawk
    • 8.2 Using Numbers to Subscript Arrays
    • 8.3 Using Uninitialized Variables as Subscripts
    • 8.4 The delete Statement
    • 8.5 Multidimensional Arrays
      • 8.5.1 Scanning Multidimensional Arrays
    • 8.6 Arrays of Arrays
    • 8.7 Summary
  • 9 Functions
    • 9.1 Built-in Functions
      • 9.1.1 Calling Built-in Functions
      • 9.1.2 Numeric Functions
      • 9.1.3 String-Manipulation Functions
        • 9.1.3.1 More about ‘\’ and ‘&’ with sub(), gsub(), and gensub()
      • 9.1.4 Input/Output Functions
      • 9.1.5 Time Functions
      • 9.1.6 Bit-Manipulation Functions
      • 9.1.7 Getting Type Information
      • 9.1.8 String-Translation Functions
    • 9.2 User-Defined Functions
      • 9.2.1 Function Definition Syntax
      • 9.2.2 Function Definition Examples
      • 9.2.3 Calling User-Defined Functions
        • 9.2.3.1 Writing a Function Call
        • 9.2.3.2 Controlling Variable Scope
        • 9.2.3.3 Passing Function Arguments by Value Or by Reference
        • 9.2.3.4 Other Points About Calling Functions
      • 9.2.4 The return Statement
      • 9.2.5 Functions and Their Effects on Variable Typing
    • 9.3 Indirect Function Calls
    • 9.4 Summary
• Part II:
  Problem Solving with awk
  • 10 A Library of awk Functions
    • 10.1 Naming Library Function Global Variables
    • 10.2 General Programming
      • 10.2.1 Converting Strings to Numbers
      • 10.2.2 Assertions
      • 10.2.3 Rounding Numbers
      • 10.2.4 The Cliff Random Number Generator
      • 10.2.5 Translating Between Characters and Numbers
      • 10.2.6 Merging an Array into a String
      • 10.2.7 Managing the Time of Day
      • 10.2.8 Reading a Whole File at Once
      • 10.2.9 Quoting Strings to Pass to the Shell
    • 10.3 Data file Management
      • 10.3.1 Noting Data file Boundaries
      • 10.3.2 Rereading the Current File
      • 10.3.3 Checking for Readable Data files
      • 10.3.4 Checking for Zero-Length Files
      • 10.3.5 Treating Assignments as File names
    • 10.4 Processing Command-Line Options
    • 10.5 Reading the User Database
    • 10.6 Reading the Group Database
    • 10.7 Traversing Arrays of Arrays
    • 10.8 Summary
    • 10.9 Exercises
  • 11 Practical awk Programs
    • 11.1 Running the Example Programs
    • 11.2 Reinventing Wheels for Fun and Profit
      • 11.2.1 Cutting Out Fields and Columns
      • 11.2.2 Searching for Regular Expressions in Files
      • 11.2.3 Printing Out User Information
      • 11.2.4 Splitting a Large File into Pieces
      • 11.2.5 Duplicating Output into Multiple Files
      • 11.2.6 Printing Nonduplicated Lines of Text
      • 11.2.7 Counting Things
    • 11.3 A Grab Bag of awk Programs
      • 11.3.1 Finding Duplicated Words in a Document
      • 11.3.2 An Alarm Clock Program
      • 11.3.3 Transliterating Characters
      • 11.3.4 Printing Mailing Labels
      • 11.3.5 Generating Word-Usage Counts
      • 11.3.6 Removing Duplicates from Unsorted Text
      • 11.3.7 Extracting Programs from Texinfo Source Files
      • 11.3.8 A Simple Stream Editor
      • 11.3.9 An Easy Way to Use Library Functions
      • 11.3.10 Finding Anagrams from a Dictionary
      • 11.3.11 And Now for Something Completely Different
    • 11.4 Summary
    • 11.5 Exercises
• Part III:
  Moving Beyond Standard awk with gawk
  • 12 Advanced Features of gawk
    • 12.1 Allowing Nondecimal Input Data
    • 12.2 Controlling Array Traversal and Array Sorting
      • 12.2.1 Controlling Array Traversal
      • 12.2.2 Sorting Array Values and Indices with gawk
    • 12.3 Two-Way Communications with Another Process
    • 12.4 Using gawk for Network Programming
    • 12.5 Profiling Your awk Programs
    • 12.6 Summary
  • 13 Internationalization with gawk
    • 13.1 Internationalization and Localization
    • 13.2 GNU gettext
    • 13.3 Internationalizing awk Programs
    • 13.4 Translating awk Programs
      • 13.4.1 Extracting Marked Strings
      • 13.4.2 Rearranging printf Arguments
      • 13.4.3 awk Portability Issues
    • 13.5 A Simple Internationalization Example
    • 13.6 gawk Can Speak Your Language
    • 13.7 Summary
  • 14 Debugging awk Programs
    • 14.1 Introduction to the gawk Debugger
      • 14.1.1 Debugging in General
      • 14.1.2 Debugging Concepts
      • 14.1.3 awk Debugging
    • 14.2 Sample gawk Debugging Session
      • 14.2.1 How to Start the Debugger
      • 14.2.2 Finding the Bug
    • 14.3 Main Debugger Commands
      • 14.3.1 Control of Breakpoints
      • 14.3.2 Control of Execution
      • 14.3.3 Viewing and Changing Data
      • 14.3.4 Working with the Stack
      • 14.3.5 Obtaining Information About the Program and the Debugger State
      • 14.3.6 Miscellaneous Commands
    • 14.4 Readline Support
    • 14.5 Limitations
    • 14.6 Summary
  • 15 Namespaces in gawk
    • 15.1 Standard awk’s Single Namespace
    • 15.2 Qualified Names
    • 15.3 The Default Namespace
    • 15.4 Changing The Namespace
    • 15.5 Namespace and Component Naming Rules
    • 15.6 Internal Name Management
    • 15.7 Namespace Example
    • 15.8 Namespaces and Other gawk Features
    • 15.9 Summary
  • 16 Arithmetic and Arbitrary-Precision Arithmetic with gawk
    • 16.1 A General Description of Computer Arithmetic
    • 16.2 Other Stuff to Know
    • 16.3 Arbitrary-Precision Arithmetic Features in gawk
    • 16.4 Floating-Point Arithmetic: Caveat Emptor!
      • 16.4.1 Floating-Point Arithmetic Is Not Exact
        • 16.4.1.1 Many Numbers Cannot Be Represented Exactly
        • 16.4.1.2 Be Careful Comparing Values
        • 16.4.1.3 Errors Accumulate
      • 16.4.2 Getting the Accuracy You Need
      • 16.4.3 Try a Few Extra Bits of Precision and Rounding
      • 16.4.4 Setting the Precision
      • 16.4.5 Setting the Rounding Mode
    • 16.5 Arbitrary-Precision Integer Arithmetic with gawk
    • 16.6 How To Check If MPFR Is Available
    • 16.7 Standards Versus Existing Practice
    • 16.8 Summary
  • 17 Writing Extensions for gawk
    • 17.1 Introduction
    • 17.2 Extension Licensing
    • 17.3 How It Works at a High Level
    • 17.4 API Description
      • 17.4.1 Introduction
      • 17.4.2 General-Purpose Data Types
      • 17.4.3 Memory Allocation Functions and Convenience Macros
      • 17.4.4 Constructor Functions
      • 17.4.5 Registration Functions
        • 17.4.5.1 Registering An Extension Function
        • 17.4.5.2 Registering An Exit Callback Function
        • 17.4.5.3 Registering An Extension Version String
        • 17.4.5.4 Customized Input Parsers
        • 17.4.5.5 Customized Output Wrappers
        • 17.4.5.6 Customized Two-way Processors
      • 17.4.6 Printing Messages
      • 17.4.7 Updating ERRNO
      • 17.4.8 Requesting Values
      • 17.4.9 Accessing and Updating Parameters
      • 17.4.10 Symbol Table Access
        • 17.4.10.1 Variable Access and Update by Name
        • 17.4.10.2 Variable Access and Update by Cookie
        • 17.4.10.3 Creating and Using Cached Values
      • 17.4.11 Array Manipulation
        • 17.4.11.1 Array Data Types
        • 17.4.11.2 Array Functions
        • 17.4.11.3 Working With All The Elements of an Array
        • 17.4.11.4 How To Create and Populate Arrays
      • 17.4.12 Accessing and Manipulating Redirections
      • 17.4.13 API Variables
        • 17.4.13.1 API Version Constants and Variables
        • 17.4.13.2 GMP and MPFR Version Information
        • 17.4.13.3 Informational Variables
      • 17.4.14 Boilerplate Code
      • 17.4.15 Changes From Version 1 of the API
    • 17.5 How gawk Finds Extensions
    • 17.6 Example: Some File Functions
      • 17.6.1 Using chdir() and stat()
      • 17.6.2 C Code for chdir() and stat()
      • 17.6.3 Integrating the Extensions
    • 17.7 The Sample Extensions in the gawk Distribution
      • 17.7.1 File-Related Functions
      • 17.7.2 Interface to fnmatch()
      • 17.7.3 Interface to fork(), wait(), and waitpid()
      • 17.7.4 Enabling In-Place File Editing
      • 17.7.5 Character and Numeric values: ord() and chr()
      • 17.7.6 Reading Directories
      • 17.7.7 Reversing Output
      • 17.7.8 Two-Way I/O Example
      • 17.7.9 Dumping and Restoring an Array
      • 17.7.10 Reading an Entire File
      • 17.7.11 Extension Time Functions
      • 17.7.12 API Tests
    • 17.8 The gawkextlib Project
    • 17.9 Summary
    • 17.10 Exercises
• Part IV:
  Appendices
  • Appendix A The Evolution of the awk Language
    • A.1 Major Changes Between V7 and SVR3.1
    • A.2 Changes Between SVR3.1 and SVR4
    • A.3 Changes Between SVR4 and POSIX awk
    • A.4 Extensions in Brian Kernighan’s awk
    • A.5 Extensions in gawk Not in POSIX awk
    • A.6 History of gawk Features
    • A.7 Common Extensions Summary
    • A.8 Regexp Ranges and Locales: A Long Sad Story
    • A.9 Major Contributors to gawk
    • A.10 Summary
  • Appendix B Installing gawk
    • B.1 The gawk Distribution
      • B.1.1 Getting the gawk Distribution
      • B.1.2 Extracting the Distribution
      • B.1.3 Contents of the gawk Distribution
    • B.2 Compiling and Installing gawk on Unix-Like Systems
      • B.2.1 Compiling gawk for Unix-Like Systems
      • B.2.2 Shell Startup Files
      • B.2.3 Additional Configuration Options
      • B.2.4 The Configuration Process
    • B.3 Installation on Other Operating Systems
      • B.3.1 Installation on MS-Windows
        • B.3.1.1 Installing a Prepared Distribution for MS-Windows Systems
        • B.3.1.2 Compiling gawk for PC Operating Systems
        • B.3.1.3 Using gawk on PC Operating Systems
        • B.3.1.4 Using gawk In The Cygwin Environment
        • B.3.1.5 Using gawk In The MSYS Environment
      • B.3.2 Compiling and Installing gawk on Vax/VMS and OpenVMS
        • B.3.2.1 Compiling gawk on VMS
        • B.3.2.2 Compiling gawk Dynamic Extensions on VMS
        • B.3.2.3 Installing gawk on VMS
        • B.3.2.4 Running gawk on VMS
        • B.3.2.5 The VMS GNV Project
        • B.3.2.6 Some VMS Systems Have An Old Version of gawk
    • B.4 Reporting Problems and Bugs
      • B.4.1 Submitting Bug Reports
      • B.4.2 Please Don’t Post Bug Reports to USENET
      • B.4.3 Reporting Problems with Non-Unix Ports
    • B.5 Other Freely Available awk Implementations
    • B.6 Summary
  • Appendix C Implementation Notes
    • C.1 Downward Compatibility and Debugging
    • C.2 Making Additions to gawk
      • C.2.1 Accessing The gawk Git Repository
      • C.2.2 Adding New Features
      • C.2.3 Porting gawk to a New Operating System
      • C.2.4 Why Generated Files Are Kept In Git
    • C.3 Probable Future Extensions
    • C.4 Some Limitations of the Implementation
    • C.5 Extension API Design
      • C.5.1 Problems With The Old Mechanism
      • C.5.2 Goals For A New Mechanism
      • C.5.3 Other Design Decisions
      • C.5.4 Room For Future Growth
    • C.6 Summary
  • Appendix D Basic Programming Concepts
    • D.1 What a Program Does
    • D.2 Data Values in a Computer
  • Glossary
  • GNU General Public License
  • GNU Free Documentation License
    • ADDENDUM: How to use this License for your documents
  • Index

---

Next: Foreword3, Up: (dir)   [Contents][Index]