Overview

The Python Requests library provides a simplified approach to making HTTP requests, making it ideal for API testing scenarios. Built on top of urllib, this Apache2 licensed library offers enhanced usability while maintaining full HTTP compliance.

Major organizations including Twiter, Spotify, Microsoft, and Amazon utilize Requests internally for various applications.

Key features include:

• Persistent connections and connection pooling
• International domain and URL support
• Session-based cookie persistence
• SSL certificate verification
• Automatic content decoding
• Authentication support (Basic/Digest)
• Unicode response handling
• Proxy support
• Multipart file uploads
• Streamed response downloads
• Request timeouts
• .netrc file support

Installation

Install using pip:

pip install requests

Verify installation by importing the module:

import requests

An ImportError indicates failed installation.

Basic HTTP Operations

Different HTTP methods are available through dedicated functions:

base_endpoint = 'http://httpbin.org'

# GET request
response_get = requests.get(base_endpoint + '/get')
print(response_get.status_code)

# POST request
response_post = requests.post(base_endpoint + '/post')
print(response_post.status_code)

Parameter Handling

Query Parameters

For GET requests, parameters can be passed via the params argument:

auth_params = {'username': 'tester', 'token': 'abc123'}
response = requests.get(base_endpoint + '/get', params=auth_params)
print(response.url)
print(response.status_code)

Request Body Data

POST requests typically send data in the request body:

payload = {'account': 'admin', 'secret': 'password123'}
response = requests.post(base_endpoint + '/post', data=payload)
print(response.text)

Custom Headers

HTTP headers are specified using a dictionary passed to the headers parameter:

user_data = {'account': 'admin', 'secret': 'password123'}
custom_headers = {'user-agent': 'Mozilla/5.0'}

response = requests.post(base_endpoint + '/post', data=user_data, headers=custom_headers)
print(response.text)

Web scraping often requires custom headers to avoid detection:

browser_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
response = requests.get("https://www.example.com", headers=browser_headers)
print(response.text)

Response Handling

After sending requests, various response attributes can be accessed:

form_data = {'account': 'admin', 'secret': 'password123'}
headers = {'user-agent': 'Mozilla/5.0'}
response = requests.post(base_endpoint + '/post', data=form_data, headers=headers)

print(response.status_code)  # HTTP status
def print(response.headers)    # Response headers
def print(response.text)      # Response content as string
def print(response.json())    # JSON parsed response

Response attributes include:

Attribute	Description
.status_code	HTTP response status
.content	Raw response bytes with automatic decompression
.headers	Dictionary of response headers
.json()	Parsed JSON response
.url	Final request URL
.encoding	Response encoding
.cookies	Response cookies
.raw	Raw response data
.text	Decoded response string
.raise_for_status()	Raises exception for non-200 responses

Advanced Features

Cookie Management

Setting custom cookies:

session_cookie = {'session_id': 'user123'}
response = requests.get(base_endpoint + '/cookies', cookies=session_cookie)
print(response.text)

Retrieving cookies from responses:

response = requests.get('http://www.example.com')
print(type(response.cookies))
print(response.cookies)

for name, value in response.cookies.items():
    print(f"{name}: {value}")

The cookies attribute returns a RequestCookieJar object that can be iterated to access individual cookie values.

Request Timeouts

Timeouts prevent indefinite waiting for responses:

try:
    response = requests.get('http://example.com', timeout=5)
    print(response.text)
except requests.exceptions.Timeout:
    print("Request timed out")

Setting appropriate timeout values helps manage application responsiveness and resource usage.