Implementing Concurrency with Coroutines in Python
The Global Interpreter Lock (GIL) in CPython prevents true parallel execution with threads. Coroutines offer a lightweight alternative for achieving concurrency within a single thread through controlled switching and state preservation.
Yield-Based Approach
Using yield allows state preservation similar to OS-level thread switching, but at the code level:
import time
def compute_sum():
total = 0
for i in range(10_000_000):
total += i
yield
def counter():
gen = compute_sum()
count = 0
for _ in range(10_000_000):
count += 1
next(gen)
start = time.time()
counter()
print("Concurrent execution time:", time.time() - start)
# Sequential execution comparison
def sequential():
sum_total = 0
for i in range(10_000_000):
sum_total += i
cnt = 0
for _ in range(10_000_000):
cnt += 1
start = time.time()
sequential()
print("Sequential execution time:", time.time() - start)
For CPU-bound tasks, this approach often decreases performance due to switching overhead.
Greenlet Module
The greenlet module providse a cleaner interface for coroutine switching:
from greenlet import greenlet
def process_a(name):
print(f"{name} executing process A")
gr_b.switch(name)
print("Resuming process A")
gr_b.switch()
def process_b(name):
print(f"{name} executing process B")
gr_a.switch()
print("Resuming process B")
gr_a = greenlet(process_a)
gr_b = greenlet(process_b)
gr_a.switch("test")
Gevent for IO-Bound Tasks
Gevent combines coroutines with automatic IO detection:
import gevent
from gevent import monkey
monkey.patch_all()
import time
def fetch_data():
print("Fetching data...")
time.sleep(2)
print("Data received")
def process_logs():
print("Processing logs...")
time.sleep(1)
print("Logs processed")
task1 = gevent.spawn(fetch_data)
task2 = gevent.spawn(process_logs)
gevent.joinall([task1, task2])
Key considerations:
- Always call
join()on coroutine objects - Monkey patching should occur before importing blocking modules
- Coroutines work best for IO-bound operations, not CPU-bound tasks