Understanding SaltStack Job Management

In SaltStack infrastructure, every operation executed generates a unique job ID (jid) on the master node. The minion creates a file in the cache/proc directory using this jid as the filename, which contains the operation's records. This file is automatically deleted once the operation completes.

SaltStack provides two primary methods for job management: salt-run commands and system modules.

Managing Jobs with Salt-Run Commands

The salt-run utility offers several commands for job management. To view available job-related commands:

[root@master ~]# salt-run -d | grep jobs
'jobs.active:'
    Returns a report of all actively running jobs from a job ID perspective
    Example: salt-run jobs.active

'jobs.list_job:'
    Displays details for a specific job ID
    Example: salt-run jobs.list_job 20230916125524463507

'jobs.list_jobs:'
    Lists all detectable jobs with their associated functions
    Example: salt-run jobs.list_jobs

'jobs.lookup_jid:'
    Retrieves return data for a specific job ID
    Example: salt-run jobs.lookup_jid 20230916125524463507
    Example: salt-run jobs.lookup_jid 20230916125524463507 outputter=highstate

'jobs.print_job:'
    Prints formatted information for a specific job
    Example: salt-run jobs.print_job 20230916125524463507

Identifying Job IDs

When executing a command, SaltStack displays the job ID:

[root@master ~]# salt 'minion1' cmd.run 'sleep 30;id'
^CExiting on Ctrl-C
This job's jid is:
20231227213754124155
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later run:
salt-run jobs.lookup_jid 20231227213754124155

Checking Job Results

To retrieve the results of a completed job using its ID:

[root@master ~]# salt-run jobs.lookup_jid 20231227213754124155
minion1:
    root

Viewing Detailed Job Information

The jobs.list_job command provides comprehensive details about a specific job:

[root@master ~]# salt-run jobs.list_job 20231227213754124155
Arguments:
    - sleep 30;id
Function:
    cmd.run
Minions:
    - minion1
Result:
    ----------
    minion1:
        ----------
        return:
            uid=0(root) gid=0(root) groups=0(root)
StartTime:
    2023, Dec 27 21:37:54.124155
Target:
    minion1
Target-type:
    glob
User:
    root
jid:
    20231227213754124155

Managing Jobs with Salt System Modules

Salt system modules provide additional job management capabilities. To view available job-related modules:

[root@master ~]# salt '*' sys.doc saltutil | grep job
'saltutil.find_cached_job:'
    Returns data for a specific cached job ID
    Example: salt '*' saltutil.find_cached_job 

'saltutil.find_job:'
    Returns data for a specific job ID
    Example: salt '*' saltutil.find_job 

'saltutil.kill_job:'
    Sends a SIGKILL signal (9) to a specified salt job's process
    Example: salt '*' saltutil.kill_job 

'saltutil.signal_job:'
    Sends a custom signal to a specified salt job's process
    Example: salt '*' saltutil.signal_job  15

'saltutil.term_job:'
    Sends a SIGTERM signal (15) to terminate a specified salt job's process
    Example: salt '*' saltutil.term_job 

Using System Modules for Job Management

Let's execute a long-running command and then manage it using system modules:

[root@master ~]# salt 'minion1' cmd.run 'sleep 120;whoami'

[root@master ~]# salt 'minion1' saltutil.find_job 20231227220403774993

Terminating Jobs

To forcefully terminate a running job:

[root@master ~]# salt 'minion1' saltutil.kill_job 20231227220403774993
minion1:
    Signal 9 sent to job 20231227220656122140 at pid 143225