HTTP Requests in Python with httpx

I switched from requests to httpx six months ago. Here's everything I learned.

Why httpx Over requests?

When I started Python, everyone used requests. It's the default recommendation everywhere. But after hitting some walls, I discovered httpx and haven't looked back.

The problems I hit with requests:

No async support. My FastAPI app was async, but I was making blocking HTTP calls
No HTTP/2. Some APIs I called supported it, but I couldn't use it
No timeout by default. I learned this the hard way when a request hung forever in production
Inconsistent API. I wanted the same code for sync and async

What httpx gives you:

# Same API for sync and async
import httpx
 
# Sync
response = httpx.get("https://api.example.com/data")
 
# Async
async with httpx.AsyncClient() as client:
    response = await client.get("https://api.example.com/data")

Plus: full type hints, HTTP/2 support, sensible defaults, and active maintenance.

Installation

pip install httpx
 
# For HTTP/2 support
pip install httpx[http2]

Basic GET and POST

The API feels familiar if you've used requests:

import httpx
 
# Simple GET
response = httpx.get("https://api.github.com/users/octocat")
print(response.status_code)  # 200
print(response.json())        # Parse JSON response
 
# GET with query parameters
response = httpx.get(
    "https://api.github.com/search/repositories",
    params={"q": "python httpx", "sort": "stars"}
)
 
# POST with JSON body
response = httpx.post(
    "https://api.example.com/users",
    json={"name": "Owen", "role": "developer"}
)
 
# POST with form data
response = httpx.post(
    "https://api.example.com/login",
    data={"username": "owen", "password": "secret"}
)

Using a Client (The Right Way)

Here's something I wish I knew earlier: always use a Client for multiple requests.

# Don't do this (new connection each time)
for user_id in user_ids:
    response = httpx.get(f"https://api.example.com/users/{user_id}")
 
# Do this instead
with httpx.Client() as client:
    for user_id in user_ids:
        response = client.get(f"https://api.example.com/users/{user_id}")

Why? Connection pooling. The client reuses TCP connections, saving the overhead of new handshakes. In my tests, this made a 10-request loop 3x faster.

Client Configuration

client = httpx.Client(
    base_url="https://api.example.com",
    headers={
        "Authorization": "Bearer my-token",
        "User-Agent": "MyApp/1.0"
    },
    timeout=30.0
)
 
# Now requests are simpler
response = client.get("/users")        # https://api.example.com/users
response = client.get("/posts")        # https://api.example.com/posts
 
# Don't forget to close
client.close()
 
# Or use context manager (better)
with httpx.Client(base_url="https://api.example.com") as client:
    users = client.get("/users").json()

Async Support

This is the killer feature for me. Same API, just async:

import httpx
import asyncio
 
async def fetch_user(client, user_id):
    response = await client.get(f"/users/{user_id}")
    return response.json()
 
async def main():
    async with httpx.AsyncClient(base_url="https://api.example.com") as client:
        user = await fetch_user(client, 123)
        print(user)
 
asyncio.run(main())

Parallel Requests

The real power is concurrent requests:

import httpx
import asyncio
 
async def fetch_all_users(user_ids):
    async with httpx.AsyncClient(base_url="https://api.example.com") as client:
        tasks = [client.get(f"/users/{uid}") for uid in user_ids]
        responses = await asyncio.gather(*tasks)
        return [r.json() for r in responses]
 
# Fetch 100 users concurrently instead of sequentially
users = asyncio.run(fetch_all_users(range(1, 101)))

Sequential: ~30 seconds for 100 requests. Concurrent: ~2 seconds. That's the difference async makes.

Rate Limiting with Semaphore

Real APIs have rate limits. Here's how I handle them:

import httpx
import asyncio
 
async def fetch_with_limit(urls, max_concurrent=10):
    semaphore = asyncio.Semaphore(max_concurrent)
    
    async def fetch(client, url):
        async with semaphore:
            response = await client.get(url)
            return response.json()
    
    async with httpx.AsyncClient() as client:
        tasks = [fetch(client, url) for url in urls]
        return await asyncio.gather(*tasks)
 
# Only 10 requests at a time
results = asyncio.run(fetch_with_limit(my_urls, max_concurrent=10))

Timeouts

This is where httpx shines. It has a default timeout of 5 seconds. requests has no default—it will hang forever.

# Uses default 5 second timeout
response = httpx.get("https://api.example.com/data")
 
# Custom timeout
response = httpx.get("https://api.example.com/data", timeout=30.0)
 
# Granular timeouts
timeout = httpx.Timeout(
    connect=5.0,   # Time to establish connection
    read=30.0,     # Time to receive response
    write=10.0,    # Time to send request
    pool=5.0       # Time to acquire connection from pool
)
client = httpx.Client(timeout=timeout)
 
# Disable timeout (don't do this in production)
response = httpx.get(url, timeout=None)

Per-Request Override

with httpx.Client(timeout=10.0) as client:
    # Uses client default (10s)
    response = client.get("/fast-endpoint")
    
    # Override for slow endpoint
    response = client.get("/slow-report", timeout=120.0)

Retries

httpx doesn't have built-in retry logic, but there are two good approaches.

Using tenacity (My Preference)

import httpx
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type
)
 
@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=1, max=10),
    retry=retry_if_exception_type((httpx.ConnectError, httpx.ReadTimeout))
)
def fetch_with_retry(url):
    response = httpx.get(url, timeout=10.0)
    response.raise_for_status()
    return response.json()
 
# Automatically retries on connection errors and timeouts
data = fetch_with_retry("https://api.example.com/data")

Using httpx Transport

import httpx
 
# Simple retry on connection issues
transport = httpx.HTTPTransport(retries=3)
client = httpx.Client(transport=transport)
 
response = client.get("https://api.example.com/data")

The transport approach only retries connection-level failures, not HTTP errors. For more control, use tenacity.

Async Retry Pattern

import httpx
import asyncio
from tenacity import (
    AsyncRetrying,
    stop_after_attempt,
    wait_exponential
)
 
async def fetch_with_async_retry(client, url):
    async for attempt in AsyncRetrying(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=1, max=10)
    ):
        with attempt:
            response = await client.get(url)
            response.raise_for_status()
            return response.json()

Connection Pooling

Clients manage connection pools automatically:

# Default: 100 connections, 10 per host
client = httpx.Client()
 
# Custom limits
limits = httpx.Limits(
    max_keepalive_connections=20,  # Keep 20 connections alive
    max_connections=100,            # Max 100 total connections
    keepalive_expiry=30.0          # Close idle connections after 30s
)
client = httpx.Client(limits=limits)

For high-throughput applications:

# Aggressive pooling for many requests to same host
limits = httpx.Limits(
    max_keepalive_connections=50,
    max_connections=200
)
 
async with httpx.AsyncClient(limits=limits) as client:
    # Can make many concurrent requests efficiently
    pass

Handling Responses

Status Codes

response = httpx.get("https://api.example.com/users")
 
# Check status
print(response.status_code)        # 200
print(response.is_success)         # True (2xx)
print(response.is_redirect)        # False
print(response.is_client_error)    # False (4xx)
print(response.is_server_error)    # False (5xx)
 
# Raise exception on error
response.raise_for_status()  # Raises HTTPStatusError for 4xx/5xx

Response Content

response = httpx.get(url)
 
# Different formats
response.text          # String (decoded)
response.content       # Bytes (raw)
response.json()        # Parse as JSON
 
# Response headers
response.headers["content-type"]
response.headers.get("x-rate-limit-remaining")
 
# Encoding
response.encoding      # e.g., 'utf-8'

Streaming Large Responses

For large downloads, stream instead of loading into memory:

with httpx.stream("GET", "https://example.com/large-file.zip") as response:
    with open("large-file.zip", "wb") as f:
        for chunk in response.iter_bytes(chunk_size=8192):
            f.write(chunk)
 
# Async version
async with httpx.AsyncClient() as client:
    async with client.stream("GET", url) as response:
        async for chunk in response.aiter_bytes():
            await process_chunk(chunk)

Error Handling

This is crucial for production code. Here's the pattern I use:

import httpx
 
def fetch_user(user_id: int) -> dict | None:
    try:
        response = httpx.get(
            f"https://api.example.com/users/{user_id}",
            timeout=10.0
        )
        response.raise_for_status()
        return response.json()
    
    except httpx.ConnectError:
        print(f"Could not connect to API")
        return None
    
    except httpx.TimeoutException:
        print(f"Request timed out")
        return None
    
    except httpx.HTTPStatusError as e:
        if e.response.status_code == 404:
            print(f"User {user_id} not found")
        elif e.response.status_code == 429:
            print(f"Rate limited, try again later")
        else:
            print(f"HTTP error: {e.response.status_code}")
        return None
    
    except httpx.RequestError as e:
        print(f"Request failed: {e}")
        return None

Exception Hierarchy

httpx.RequestError (base for all request errors)
├── httpx.ConnectError      # Connection failed
├── httpx.ConnectTimeout    # Connection timed out
├── httpx.ReadTimeout       # Read timed out
├── httpx.WriteTimeout      # Write timed out
├── httpx.PoolTimeout       # Couldn't get connection from pool
└── httpx.TimeoutException  # Base for all timeouts

httpx.HTTPStatusError       # 4xx or 5xx response (after raise_for_status)

Comparison: httpx vs requests vs aiohttp

After using all three, here's my honest take:

Feature	requests	httpx	aiohttp
Sync support	✓	✓	✗
Async support	✗	✓	✓
HTTP/2	✗	✓	✗
Default timeout	None	5s	None
Type hints	Partial	Full	Partial
Learning curve	Easy	Easy	Medium
API consistency	—	Same for sync/async	Different

When to use each:

requests: Legacy code, simple scripts, tutorials. It works fine, but I wouldn't start a new project with it.

httpx: New projects. It's modern, has sane defaults, and the same API works for sync and async. This is my default choice.

aiohttp: Maximum async performance. If you're building something like a web scraper making thousands of concurrent requests, aiohttp is slightly faster. But the API is clunkier.

Migration from requests

It's mostly drop-in:

# Before
import requests
response = requests.get(url, timeout=10)
 
# After
import httpx
response = httpx.get(url, timeout=10)

Most code works unchanged. A few differences:

response.ok → response.is_success
Sessions → Clients
requests.exceptions → httpx (exceptions are module-level)

My Production Pattern

Here's the client wrapper I use in most projects:

import httpx
from contextlib import contextmanager
from tenacity import retry, stop_after_attempt, wait_exponential
 
class APIClient:
    def __init__(self, base_url: str, token: str, timeout: float = 30.0):
        self.client = httpx.Client(
            base_url=base_url,
            headers={"Authorization": f"Bearer {token}"},
            timeout=timeout
        )
    
    @retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
    def get(self, path: str, **kwargs) -> dict:
        response = self.client.get(path, **kwargs)
        response.raise_for_status()
        return response.json()
    
    @retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
    def post(self, path: str, data: dict, **kwargs) -> dict:
        response = self.client.post(path, json=data, **kwargs)
        response.raise_for_status()
        return response.json()
    
    def close(self):
        self.client.close()
    
    def __enter__(self):
        return self
    
    def __exit__(self, *args):
        self.close()
 
# Usage
with APIClient("https://api.example.com", "my-token") as api:
    users = api.get("/users")
    new_user = api.post("/users", {"name": "Owen"})

Key Takeaways

Use httpx for new projects. It's the modern choice with better defaults
Always use a Client for multiple requests. Connection pooling matters
Set timeouts explicitly. Even though httpx has defaults, be intentional
Add retry logic for production code. APIs fail. Plan for it
Handle errors properly. Know the exception hierarchy
Use async when it makes sense. Concurrent requests are powerful, but don't add async complexity if you don't need it

The HTTP client might seem like a small detail, but getting it right makes your applications more reliable. httpx makes that easy.

React to this post:

#Why httpx Over requests?

#Installation

#Basic GET and POST

#Using a Client (The Right Way)

#Client Configuration

#Async Support

#Parallel Requests

#Rate Limiting with Semaphore

#Timeouts

#Per-Request Override

#Retries

#Using tenacity (My Preference)

#Using httpx Transport

#Async Retry Pattern

#Connection Pooling

#Handling Responses

#Status Codes

#Response Content

#Streaming Large Responses

#Error Handling

#Exception Hierarchy

#Comparison: httpx vs requests vs aiohttp

#Migration from requests

#My Production Pattern

#Key Takeaways

Keep Reading

Need help shipping fast?