AWK is a powerful text analysis tool, particularly adept at data processing and report generation. It reads input line by line, splits each line in to fields by a default delimiter (space), and performs analysis on the segmented parts.

The basic syntax is: awk 'pattern { action }' filenames. Here, pattern specifies what AWK searches for in the data, and action is a series of commands executed when a match is found. Braces {} are used to group commands for a specific pattern.

AWK's core functionality involves scanning files or strings to extract information based on defined rules. A complete AWK script typical formats data from text files. Processing is performed line by line; AWK reads a line, executes the relevant commands, and then moves to the next.

AWK Usage

awk 'BEGIN { commands } pattern { commands } END { commands }'

1. Execute statements in the BEGIN { commands } block.
2. Read a line from the file or standard input. Then run the pattern { commands } block, scanning the file line by line from start to finish.
3. After reading all lines, execute the END { commands } block.

The BEGIN block runs before reading any input (e.g., for variable initialization, printing headers). The END block runs after all input is processed (e.g., for printing summaries). The pattern { commands } block is the main processing block; if omitted, the default action { print } is applied to each line. All three sections are optional.

Built-in Variables

• $0: Entire current line.
• $1 to $n: The nth field of the current line, separated by FS.
• FS: Input field separator (default is space or tab).
• NF: Number of fields in the current record.
• NR: Total number of records read so far (line number across files).
• FNR: Current record number within the current file.
• RS: Input record separator (default is newline).
• OFS: Output field separator (default is space).
• ORS: Output record separator (default is newline).
• FILENAME: Name of the current input file.

Examples

Print each line with its line number:

ls -lh | awk '{print NR " " $1}'

Output:

1 total
2 drwxr-xr-x.
3 drwx------.
4 drwxr-xr-x.
5 drwxrwxr-x.
6 drwxr-xr-x.

Format output using printf for better readability:

awk -F: '{printf("filename:%10s, linenumber:%s, columns:%s, linecontent:%s\n", FILENAME, NR, NF, $0)}' /etc/passwd

Search for lines containing "root":

awk '/root/' /etc/passwd

Output:

root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

Search for "root" and print the shell (7th field):

awk -F: '/root/ {print $7}' /etc/passwd

Output:

/bin/bash
/sbin/nologin

Print the second line of a file:

awk -F: 'NR==2 {print "filename: " FILENAME, $0}' /etc/passwd

Output:

filename: /etc/passwd bin:x:1:1:bin:/bin:/sbin/nologin

List only directories:

ls -lF | awk '/^d/'

Print the first field using colon as delimiter:

awk -F: '{print $1}' /etc/passwd

Print the last field:

awk -F: '{print $NF}' /etc/passwd

Print lines 13 to 30:

awk -F: 'NR>12 && NR<31 {print $1}' /etc/passwd

Using multiple delimiters (colon and slash):

awk -F "[/]" 'NR == 4 {print $0, "\n", $1}' /etc/passwd

Example with BEGIN and END blocks:

cat /etc/passwd | awk -F: 'BEGIN {print "name, shell"} {print $1, $NF} END {print "hello world"}'

Count and sum file sizes larger than 100k:

ls -l | awk '{if($5>100) {count++; sum+=$5}} END {print "Count:" count, "Sum:" sum}'

Count users in /etc/passwd and store names in an array:

awk -F: 'BEGIN {count=0} {name[count] = $1; count++} END {for (i = 0; i < NR; i++) print i, name[i]}' /etc/passwd