Text Processing in Linux: grep, sed, awk, and jq Guide

← Back to Linux & Bash Scripting course

The Unix philosophy — small tools that do one thing well, composable via pipes — reaches its fullest expression in text processing. grep, sed, awk, and jq are the four tools you will use daily for log analysis, configuration management, data extraction, and API response parsing. This lesson covers each in enough depth to handle real production scenarios.

Previous: Lesson 1 — Linux File System & Permissions

grep — Search and Filter

grep searches for patterns in files or stdin and prints matching lines. It understands three regex dialects: basic (grep), extended (grep -E or egrep), and Perl-compatible (grep -P).

Essential Flags

Flag	Effect
`-i`	Case-insensitive match
`-v`	Invert — print non-matching lines
`-n`	Show line numbers
`-r`	Recursive search through directories
`-l`	Print only filenames with matches
`-c`	Count matching lines per file
`-o`	Print only the matched text, not the whole line
`-A N`	Print N lines after each match
`-B N`	Print N lines before each match
`-C N`	Print N lines of context around each match

bash

# Find all lines containing "error" (case-insensitive) in all log files
grep -ri "error" /var/log/

# Count 5xx errors in an nginx access log
grep -c '" 5[0-9][0-9] ' /var/log/nginx/access.log

# Show failed SSH login attempts with 3 lines of context
grep -C 3 "Failed password" /var/log/auth.log

# Extract all IPv4 addresses from a log file
grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' access.log | sort | uniq -c | sort -rn

Extended Regular Expressions

bash

# Match lines starting with a digit or letter (not whitespace)
grep -E '^[[:alnum:]]' file.txt

# Find lines with two or more consecutive spaces
grep -E ' {2,}' file.txt

# Extract email addresses
grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' contacts.txt

# Match ERROR or WARN log levels
grep -E '^(ERROR|WARN)' app.log

sed — Stream Editor

sed reads input line by line, applies one or more editing commands, and writes to stdout. It does not modify files by default — pass -i to edit in place.

Substitution — the Core Command

bash

# Basic substitution: replace first occurrence per line
sed 's/old/new/' file.txt

# Global substitution: replace all occurrences per line
sed 's/old/new/g' file.txt

# Case-insensitive substitution
sed 's/error/ERROR/gI' file.txt

# In-place edit (GNU sed)
sed -i 's/localhost/db.internal/g' config.conf

# In-place edit with backup (safe for production)
sed -i.bak 's/localhost/db.internal/g' config.conf

Addresses — Limiting Which Lines Are Processed

bash

# Apply substitution only to lines 10–20
sed '10,20s/foo/bar/g' file.txt

# Apply substitution only to lines matching a pattern
sed '/ERROR/s/app/APP/g' app.log

# Delete blank lines
sed '/^[[:space:]]*$/d' file.txt

# Print lines between two patterns (inclusive)
sed -n '/BEGIN/,/END/p' file.txt

# Delete comments and blank lines from a config
sed -e '/^#/d' -e '/^[[:space:]]*$/d' nginx.conf

Common sed Recipes

bash

# Remove leading and trailing whitespace
sed 's/^[[:space:]]*//;s/[[:space:]]*$//' file.txt

# Insert a line after a match
sed '/^server_name/a\    listen 443 ssl;' nginx.conf

# Replace an entire line matching a pattern
sed 's/^MAX_CONNECTIONS=.*/MAX_CONNECTIONS=200/' app.conf

# Extract lines N through M without printing the rest
sed -n '5,15p' large_file.txt

awk — Columnar Data Processing

awk is a complete programming language specialised for processing structured text. It splits each input line into fields and executes a program for each line.

Field Splitting

By default, awk splits on any whitespace. $1 is the first field, $NF is the last, $0 is the entire line.

bash

# Print the first and third columns of a file
awk '{print $1, $3}' data.txt

# Print the last column
awk '{print $NF}' data.txt

# Use a custom delimiter (CSV-style)
awk -F',' '{print $2}' data.csv

# Print lines where the 5th field is greater than 1000
awk '$5 > 1000' data.txt

BEGIN and END Blocks

bash

# Print a header before output and a summary after
awk 'BEGIN {print "IP Address\tCount"}
     {count[$1]++}
     END {for (ip in count) print ip "\t" count[ip]}' access.log

# Sum the bytes transferred (field 10 in nginx combined log)
awk 'NR>1 {sum += $10} END {printf "Total: %.2f MB\n", sum/1024/1024}' access.log

awk Conditionals and Loops

bash

# Print lines where response time (field 7) exceeds 1 second
awk '$7 > 1.0 {print $0}' app.log

# Count occurrences of each HTTP status code
awk '{status[$9]++} END {for (s in status) print s, status[s]}' access.log | sort

# Reformat a CSV file
awk -F',' 'NR>1 {printf "Name: %s, Age: %s\n", $1, $2}' users.csv

Combining grep, sed, and awk in Pipelines

The real power emerges when you compose these tools:

bash

# Top 10 IPs generating 5xx errors from nginx log
grep '" 5[0-9][0-9] ' /var/log/nginx/access.log \
  | awk '{print $1}' \
  | sort | uniq -c | sort -rn \
  | head -10

# Extract slow database queries from a MySQL slow query log
grep -A 3 "Query_time: [1-9]" /var/log/mysql/slow.log \
  | grep "^SELECT\|^UPDATE\|^DELETE\|^INSERT" \
  | sed 's/  */ /g' \
  | sort | uniq -c | sort -rn

# Summarise disk usage by top-level directory, human-readable
du -sh /var/log/* 2>/dev/null | sort -rh | head -20

jq — JSON Processing at the Command Line

Modern APIs and tooling output JSON. jq is a lightweight, zero-dependency command-line JSON processor that supports filtering, transformation, and reformatting.

Basic Filtering

bash

# Pretty-print JSON
curl -s https://api.example.com/users | jq '.'

# Extract a top-level field
echo '{"name":"Alice","role":"admin"}' | jq '.name'
# "Alice"

# Extract a nested field
echo '{"user":{"id":42,"email":"alice@example.com"}}' | jq '.user.email'
# "alice@example.com"

# Extract from an array
echo '[{"id":1},{"id":2},{"id":3}]' | jq '.[1].id'
# 2

# Extract all elements of an array
echo '[{"name":"Alice"},{"name":"Bob"}]' | jq '.[].name'
# "Alice"
# "Bob"

Transforming and Filtering

bash

# Select only objects matching a condition
curl -s https://api.example.com/users \
  | jq '[.[] | select(.role == "admin")]'

# Build a new object from selected fields
curl -s https://api.example.com/users \
  | jq '[.[] | {id: .id, email: .email}]'

# Extract a raw string value (no quotes)
curl -s https://api.example.com/config \
  | jq -r '.database.host'

# Convert JSON array to newline-separated values (for shell loops)
curl -s https://api.example.com/servers \
  | jq -r '.[].hostname' \
  | while read -r host; do
      ssh "$host" 'uptime'
    done

Parsing Docker and kubectl Output

bash

# List all running container names
docker inspect $(docker ps -q) | jq -r '.[].Name'

# Get the IP of a specific container
docker inspect mycontainer | jq -r '.[0].NetworkSettings.IPAddress'

# List all pod names in a Kubernetes namespace
kubectl get pods -n production -o json \
  | jq -r '.items[].metadata.name'

# Find pods not in Running state
kubectl get pods -A -o json \
  | jq -r '.items[] | select(.status.phase != "Running") | "\(.metadata.namespace)/\(.metadata.name)"'