Fetching JSON Data with curl

# Print JSON data directly to the terminal
curl -H 'Authorization: Bearer api_key="a1b2c3d4e5f67890"' https://api.example.com/data.json

# Store the curl output in a shell variable
json_data=$(curl -H 'Authorization: Bearer api_key="a1b2c3d4e5f67890"' https://api.example.com/data.json)

# Save JSON output to a file using the -o option
curl -H 'Authorization: Bearer api_key="a1b2c3d4e5f67890"' -o output.json https://api.example.com/data.json

# Alternative: redirect output to a file
# curl -H 'Authorization: Bearer api_key="a1b2c3d4e5f67890"' https://api.example.com/data.json > output.json

Parsing JSON with jq

The jq tool provides powerful JSON parsing capabiilties in shell environments.

Sample JSON structure:

[
  {
    "hostname": "server-01",
    "address": "10.0.1.101",
    "environment": "production",
    "active": true,
    "devices": [
      {
        "local_ip": "192.168.1.100",
        "local_port": 8080,
        "external_port": 80
      },
      {
        "local_ip": "192.168.1.101",
        "local_port": 3306,
        "external_port": 3306
      }
    ]
  },
  {
    "hostname": "server-02",
    "address": "10.0.1.102",
    "environment": "staging",
    "active": true,
    "devices": [
      {
        "local_ip": "192.168.2.100",
        "local_port": 22,
        "external_port": 2222
      }
    ]
  }
]

Extracting specific fields with jq:

jq '.[] | {server_name: .hostname, ip_addr: .address}' data.json

Output:

{
  "server_name": "server-01",
  "ip_addr": "10.0.1.101"
}
{
  "server_name": "server-02",
  "ip_addr": "10.0.1.102"
}

Shell For Loops

host_list=("10.0.1.101" "10.0.1.102" "10.0.1.103")
for host in "${host_list[@]}"; do
  echo "Processing host: $host"
  # Add processing commands here
  ssh admin@$host "hostname"
done

SSH with Password Authentication

Using sshpass for automated password-based SSH connections:

sshpass -p "your_password" ssh username@hostname

Finding and Replacing Text

Display line numbers containing a specific pattern:

sed -n '/MaxStartups/=' /etc/ssh/sshd_config

Replace an entire line at a specific line number:

# Using substitution pattern
sed -i '42s/.*/MaxStartups 50:30:100/' /etc/ssh/sshd_config

# Using change command
sed -i '42c MaxStartups 50:30:100' /etc/ssh/sshd_config

Complete Implementation Example

#!/bin/bash

# Fetch server data from API
server_data=$(curl -H 'Authorization: Bearer api_key="a1b2c3d4e5f67890"' \
  https://api.example.com/servers.json)

# Extract IP addresses
ip_addresses=$(echo "$server_data" | jq -r '.[].address')

# Process each server
for server_ip in $ip_addresses; do
  echo "Updating server: $server_ip"
  
  # Find target line in sshd_config
  target_line=$(sshpass -p "server_pass" ssh admin@$server_ip \
    "sed -n '/MaxSessions/=' /etc/ssh/sshd_config")
  
  echo "Target line found at: $target_line"
  
  # Update the configuration
  sshpass -p "server_pass" ssh admin@$server_ip \
    "sed -i '${target_line}s/.*/MaxSessions 100/' /etc/ssh/sshd_config"
  
  # Verify the change
  updated_value=$(sshpass -p "server_pass" ssh admin@$server_ip \
    "sed -n '${target_line}p' /etc/ssh/sshd_config")
  
  echo "Updated value: $updated_value"
done

sed Command Reference

Basic Syntax

sed [options] 'commands' input_file

Common options:

• -n: Suppress automatic printing
• -e: Add editing commands
• -f: Read commands from file
• -r: Use extended regular expressions
• -i: Edit files in-place

Line Operations

Delete lines 2 through 5:

sed '2,5d' file.txt

Delete line 3 only:

sed '3d' file.txt

Delete from line 3 to end:

sed '3,$d' file.txt

Insert text after line 2:

sed '2a Inserted text' file.txt

Insert text before line 2:

sed '2i Inserted text' file.txt

Search and Display

Find lines containing "error":

sed -n '/error/p' logfile.txt

Display lines 5-7:

sed -n '5,7p' file.txt

Search and Replace

Basic substitution:

sed 's/old_text/new_text/g' file.txt

Extract IP address from ifconfig output:

ifconfig eth0 | grep 'inet ' | sed 's/.*inet \([0-9.]\+\).*/\1/'

Mulitple Operations

sed -e '3,$d' -e 's/root/admin/' /etc/passwd

In-place File Editing

Replace all trailing dots with exclamation marks:

sed -i 's/\.$/!/g' document.txt

Add footer text:

sed -i '$a # Configuration updated on $(date)' config.txt