Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Automating Client-Server Applications with PyAutoGUI

Tech 1

Environment & Installation

Compatible with Python 3.x. Install via pip using the Douban mirror:

pip install pyautogui -i http://pypi.douban.com/simple --trusted-host pypi.douban.com

Core API Usage Examples

import pyautogui as pag

# Get current display resolution
screen_dims = screen_width, screen_height = pag.size()

# Capture full screen screenshot
full_snap = pag.screenshot()
full_snap.save('full_screenshot.png')

# Capture a specific screen region: format (left, top, width, height)
partial_snap = pag.screenshot(region=(0, 0, 300, 400))
partial_snap.save('partial_screenshot.png')

# Get RGB value of a pixel at given coordinates
pixel_rgb = partial_snap.getpixel((50, 200))
print(pixel_rgb)

# Locate a target image on the screen, returns bounding box if found
target_btn = pag.locateOnScreen('target_button.png')
# Get the center point of the located target and click it
center_x, center_y = pag.center(target_btn)
pag.click(center_x, center_y)
# Note: Returns None if no match is found due to pixel mismatches

# Verify if coordinates fall within the active display bounds
pag.onScreen(0, 0) # Returns True
pag.onScreen(0, -1) # Returns False

# --- Mouse Control Functions ---
# Get current mouse position
curr_x, curr_y = pag.position()
# Right click at current mouse position
pag.click(pag.position(), button='right')

# Hold mouse button down at specified coordinates
pag.mouseDown(x=100, y=100, button='left')
# Release mouse button
pag.mouseUp(x=100, y=100, button='left')

# Double left click
pag.doubleClick(x=100, y=100)
# Right click variants
pag.rightClick(x=1000, y=100)
pag.click(x=1000, y=100, button='right')
# Middle click
pag.middleClick(x=1000, y=100)

# Move mouse to absolute screen coordinates, with 1 second movement duration
pag.moveTo(100, 100, duration=1)
# Move mouse relative to current position
pag.moveRel(100, 100, duration=1)

# Drag mouse to absolute position (keep duration > 0.5s to avoid system recognition issues)
pag.dragTo(x=583, y=590, duration=3)
# Drag mouse relative to current starting position
pag.dragRel(100, 100, duration=2)

# Multiple click configuration
pag.click(pag.position(), clicks=2) # Double left click
pag.click(clicks=2, interval=0.25) # Two clicks with 0.25s pause between clicks
pag.click(button='right', clicks=3, interval=0.25) # Three right clicks with 0.25s interval

# Scroll mouse wheel
pag.scroll(clicks=2, x=435, y=417)

# Easing functions for smooth mouse movement (compatible with all move/drag functions)
pag.moveTo(100, 100, 2, pag.easeInQuad) # Starts slow, gradually accelerates
pag.moveTo(100, 100, 2, pag.easeOutQuad) # Starts fast, gradually decelerates
pag.moveTo(100, 100, 2, pag.easeInOutQuad) # Fast at start/end, slow in the middle
pag.moveTo(100, 100, 2, pag.easeInBounce) # Step-by-step bouncing movement
pag.moveTo(100, 100, 2, pag.easeInElastic) # Larger overshoot past start/end points

# --- Native Dialog Functions ---
pag.alert('Alert dialog: text + OK button')
pag.confirm('Confirm dialog: text + OK + Cancel buttons')
user_input = pag.prompt('Prompt dialog: enter text input')
# Returns user input string, None if no input is provided

# Set global default pause between all PyAutoGUI actions
pag.PAUSE = 2.5

Keyboard Control

# Type a string with 0.25 second delay between each character
pag.typewrite('Hello world', interval=0.25)

Supported Key Names

Key Name Corresponding Physical Key
enter (or return or \n) Enter
esc ESC
shiftleft, shiftright Left/Right Shift
altleft, altright Left/Right Alt
ctrlleft, ctrlright Left/Right Ctrl
tab (\t) Tab
backspace, delete Backspace / Delete
pageup, pagedown Page Up / Page Down
home, end Home / End
up, down, left, right Arrow keys
f1 through f12 F1-F12 function keys
volumemute, volumedown, volumeup Volume control (not available on all keyboards)
pause Pause
capslock, numlock, scrolllock Caps Lock / Num Lock / Scrol Lock
insert Insert
printscreen Print Screen
winleft, winright Windows Logo Key
command Mac OS X Command Key

Available keyboard functions:

  • keyDown(): Press and hold a specified key
  • keyUp(): Release a previously pressed key
  • press(): Complete a full key press (combines keyDown and keyUp)
  • hotkey(): Create keyboard shortcust / combinations, e.g. hotkey('ctrl', 'c') for copy.

Full Interaction Example

import pyautogui as pag

# Capture a test region and locate it on the screen
test_snap = pag.screenshot(region=(0, 0, 500, 500))
test_snap.save('region_capture.png')
located_region = pag.locateOnScreen('region_capture.png')
region_origin = pag.center(located_region)

start_x, start_y = region_origin
end_full = (start_x, start_y + 200)
p1 = (start_x + 30, start_y + 10)
p2 = (p1[0], p1[1] + 30)
mid1 = (start_x, end_full[1] // 2)
mid2 = (p1[0], (end_full[1] // 2) + 10)
bottom_right = (p1[0], int(end_full[1] - 10))

# Sequence of drag operations
pag.mouseDown(x=start_x, y=start_y)
pag.moveTo()
pag.mouseUp(end_full[0], end_full[1])

pag.mouseDown(start_x, start_y)
pag.moveTo()
pag.mouseUp(p1[0], p1[1])

pag.mouseDown(p1[0], p1[1])
pag.moveTo()
pag.mouseUp(p2[0], p2[1])

pag.mouseDown(p2[0], p2[1])
pag.moveTo()
pag.mouseUp(start_x, start_x + 50)

pag.mouseDown(start_x, start_x + 50)
pag.moveTo()
pag.mouseUp(p1[0], start_x + 60)

pag.mouseDown(p1[0], start_x + 60)
pag.moveTo()
pag.mouseUp(bottom_right[0], bottom_right[1])

pag.mouseDown(bottom_right[0], bottom_right[1])
pag.moveTo()
pag.mouseUp(end_full[0], end_full[1])

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.