PYTHON FOR GUI AUTOMATION – PYAUTOGUI

PyAutoGUI is a Python automation library used to click, drag,
scroll, move, etc. It can be used to click at an exact position.

INSTALLATION OF PYTHON (3.X)

For Windows:
Open command prompt/ powershell and type:
~pip install pyautogui
For macOS and Linux:
~python3 –m pip install pyautogui

BASIC MOUSE FUNCTIONS:

In order to perform basic mouse functions:

1. Open any IDE.

2. Import pyautogui

3. Import time.

pyautogui.size() function:

This function doesn’t need any parameter. It returns coordinates of the cursor as the program starts.

1
2
import pyautogui
print(pyautogui.size())

1
2
PS C: \Users\ soura\ OneDrive\ Desktop\ pyautogui > py.\test.py
Size(width = 1366, height = 768)

pyautogui.displayMousePosition() function:

This function also doesn’t need any parameter. It returns the real-time cursor location along with the RGB value of the window.

1
2
import pyautogui
print(pyautogui.displayMousePosition())

1
2
3
PS C: \Users\ soura\ OneDrive\ Desktop\ pyautogui > py.\test.py
Press Ctrl - C to quit.
X: 592 Y: 767 RGB: (30, 30, 30)

pyautogui.moveTo() function:

It moves the cursor from the initial point to the set coordinates in the given duration.
This function takes three parameters:
pyautogui.moveTo(x_coordinate,y_coordinate, duration=’time_in_seconds’)

1
2
import pyautogui
print(pyautogui.moveTo(300, 300, duration = 10))

pyautogui.click() function:

It first moves the cursor to the defined coordinate and click as per given the number of clicks. This function takes five parameters:

1
2
pyautogui.click(x = x_coordinate, y = yoordinate, clicks = num_of_clicks,
  interval = secs_between_clicks, button = 'left')

1
2
3
4
5
import pyautogui
pyautogui.click(x = 300,
  y = 300,
  clicks = 3,
  interval = 2, button = 'right')

pyautogui.dragTo()/dragRel() function:

dragTo(x, y, duration=num_seconds)
Function drags to the given coordinates by selecting everything in between.

dragRel(x, y , duration=num_seconds)
Function drags to the given coordinate relative to current cursor location.

1
2
import pyautogui
pyautogui.dragTo(400, 400, duration = 10)

pyautogui.scroll() function:

This function is used to scroll throughout the page.
Positive amount_to_scroll for scrolling up.
Negative amount_to_scroll for scrolling down.
Here x and y parameters are not mandatory.
pyautogui.scroll(amount_to_scroll, x=moveToX, y=moveToY)

1
2
import pyautogui
pyautogui.scroll(-500)

BASIC KEYBOARD FUNCTIONS

pyautogui.typewrite():

This is used to enter text and to enter a new line.
pyautogui.typewrite(‘Test’, interval=secs_between_keys)

1
2
3
4
5
6
7
8
9
10
import pyautogui
import time
#time is imported to get enough time to
switch between
#the window where the text is to be typed
time.sleep(5)
pyautogui.typewrite('Hello world!', interval = 1)
# List of key names can also be passed through the
function.
pyautogui.typewrite(['a', 'b', 'c', 'left', 'backspace', 'enter', 'f1'])

pyautogui.hotkey():

Hot keys or the combination of keys can also be pressed using hotkey(‘key_name’) name.

1
2
3
4
5
import pyautogui

pyautogui.hotkey('ctrl', 'c') # ctrl - c to copy
pyautogui.hotkey('ctrl', 'v') # ctrl - v to paste
pyautogui.hotkey('ctrl', 'a') # ctrl - a to select all

pyautogui.keyDown()/keyUp():

Individual keys can also be pressed and released.

1
2
3
4
5
6
7
8
9
import pyautogui
import time
time.sleep(3)
time is imported to get enough time to
switch between
#the window where the text is to be typed
pyautogui.keyDown('key_name')
pyautogui.keyDown('key_name')
# key_names such as shift, enter, backspace, tab etc.can be passed.

MESSAGE BOX FUNCTIONS

Use to delay/pause the program until the user clicks OK on something. Or use to display some message to the user.

pyautogui.alert(‘alert_text’):

This function displays an alert box with the alert provided in the alert_text.

1
2
import pyautogui
pyautogui.alert('Your programme is being paused click OK to continue')

pyautogui.confirm()

This function creates a message box with an OK and a Cancel option.

1
2
import pyautogui
pyautogui.confirm('Your programme is being paused click  OK to continue and Cancel to exit.')

pyautogui.prompt():

This function creates a message box with a text field and OK and Cancel Buttons.

1
2
3
import pyautogui

pyautogui.prompt('Please write the feedback here.')

MAKING A RECTANGULAR SPIRAL IN MS PAINT

1
2
3
4
5
6
7
8
9
10
11
12
13
import pyautogui
import time
time.sleep(5)
pyautogui.click()
distance = 400
while distance > 10:
  pyautogui.dragRel(distance, 0, duration = 0.2)
distance = distance - 20
pyautogui.dragRel(0, distance, duration = 0.2)
pyautogui.dragRel(-distance, 0, duration = 0.2)
distance = distance - 20
pyautogui.dragRel()
pyautogui.dragRel(0, -distance, duration = 0.2)

USES OF THIS LIBRARY

AUTOMATE UI TESTING:

To guarantee quality control when a new version of a product is released can be stressful. User interface (UI) testing needs to be provided after each release of the app. Pyautogui can be used to perform automated testing of your UI.

IMAGE RECOGNITION COORDINATES:

PyautoGUI can be used in image recognition to obtain coordinates of image, text or numbers, which can be easily accessed through pyautogui.displayMousePosition().

FOR FUN:

Users can also experiment with fun stuff like automating Chrome’s dinosaur game, piano tiles, aim booster and much more.
Here are the few links for fun projects:

• Dinosaur: Click here

• Aim booster: Click here