A total timeout for paged API calls with Python requests
The timeout argument in Python requests is not a deadline for the whole call, and it is certainly not a deadline for a loop of calls. It bounds the connection attempt and each wait for data from the socket. When you page through an API, set one deadline before the loop, check it before every page, and pass min(per_request_timeout, time_left) as each request's timeout.
I hit this when an upstream search API started validating its paging parameters, and a tool that had always fetched its results in one request suddenly had to fetch them in several.
What changed and why one request became several
The code in question is a Python tool that an LLM agent calls to find companies matching a set of filters. It used to ask the search API for one large page and return the ids. The API was about to start rejecting page sizes above a small limit with a 400, so the tool had to ask for the same number of results across a few smaller pages instead.
Turning one request into a loop sounds like a five-minute change. It wasn't, because a loop raises four questions a single request never did: when to stop, what to do with duplicates, what to do when page three fails, and how long the whole thing may take.
When to stop paging
There are two reliable stop signals and one unreliable one:
- A short page. If you asked for twenty and got seven, there is nothing after it.
- The reported total. If the response carries a total and
page × page_sizehas reached it, stop, even if the last page was full. Without this you make one extra request that returns nothing. - The cap you set yourself. The caller wanted a bounded number of results, so the loop has a fixed maximum number of pages.
The unreliable one is the total itself. The same API, asked for a page number below one, returned an empty page with a negative total. I now treat any total that isn't a positive integer as "unknown" and fall back to the short-page rule, rather than letting a bad number end the loop early or keep it running.
Why the same id can arrive twice
Offset paging over a live index is not a snapshot. If a document is added or re-ranked between the request for page one and the request for page two, everything shifts, and the last few items of page one reappear at the top of page two. The fix is cheap: keep a set of ids you've seen and append only new ones, so the result keeps the API's order without repeats. It does mean you can end up with slightly fewer results than the cap, which is the right trade for a tool that only needs a bounded list.
What to return when a later page fails
Before the change, a failed request meant no results. After it, a failure on page three would throw away two good pages if the error handling stayed where it was. For an agent tool, a partial answer is far more useful than none: the agent can still reason about the companies it got. So each page sits in its own try, and a failure (an HTTP error, a timeout, or a body that doesn't parse) logs which page failed and breaks out of the loop with what was already collected. A failure on the very first page still returns an empty list, as before.
The trade-off is that a partial result looks like a complete one to the caller. The log line is what tells them apart, so it records how many pages were fetched, the reported total and the elapsed time on every call.
Why a per-request timeout isn't enough
The original single request had a timeout of several seconds. Keeping the same value on each page quietly multiplied the worst case by the number of pages. That is the first problem.
The second is less obvious. In requests, timeout=8 means "give up if connecting takes more than 8 seconds, or if the server sends no bytes for 8 seconds". A server that trickles a response slowly can keep a single request alive much longer than 8 seconds, because the read timer resets every time data arrives. The requests documentation on timeouts says this plainly, but it's easy to read past.
An agent waiting on a tool call shouldn't sit through several slow pages in a row, so what the tool needs is one total budget for all of its pages. The pattern:
- Compute a deadline once, before the first page, from a monotonic clock.
- Before each page, work out the time left. When none is left, stop and keep what you have.
- Pass
min(per_request_timeout, time_left)as that request's timeout, so a late page can't overrun the budget by a full per-request timeout.
This doesn't make a single trickling read impossible, but it bounds every wait to the time remaining, which in practice keeps the loop close to its budget.
A minimal paged fetch with a shared deadline
import time
import requests
def fetch_ids(url, filters, *, page_size=20, max_results=80,
request_timeout=8.0, budget=12.0, http=requests):
deadline = time.monotonic() + budget
ids, seen, total = [], set(), None
for page in range(1, max_results // page_size + 1):
left = deadline - time.monotonic()
if left <= 0:
break
try:
resp = http.get(url, timeout=min(request_timeout, left),
params={**filters, "page": page, "size": page_size})
resp.raise_for_status()
body = resp.json() or {}
items = body.get("items") or []
new = [int(entry["id"]) for entry in items if entry.get("id") is not None]
reported = body.get("total")
except (requests.RequestException, ValueError, KeyError, TypeError):
break # keep the pages we already have
if isinstance(reported, int) and reported > 0:
total = reported
for new_id in new:
if new_id not in seen:
seen.add(new_id)
ids.append(new_id)
if len(items) < page_size or (total and page * page_size >= total):
break
return ids
time.monotonic() is the right clock here because it never jumps when the system time is adjusted. Catching ValueError covers a body that isn't JSON; KeyError and TypeError cover items that don't have the shape you expected.
How to test a deadline without waiting for it
You don't want a test that sleeps. Patch the clock and make each fake request advance it:
from unittest.mock import patch, MagicMock
def page(start, n, total):
resp = MagicMock()
resp.json.return_value = {"items": [{"id": i} for i in range(start, start + n)],
"total": total}
return resp
def test_pages_share_one_deadline():
clock = [0.0]
pages = iter([page(1, 20, 500), page(21, 20, 500), page(41, 20, 500)])
def slow_get(*args, **kwargs):
clock[0] += 5
return next(pages)
http = MagicMock(get=MagicMock(side_effect=slow_get))
with patch("time.monotonic", side_effect=lambda: clock[0]):
result = fetch_ids("https://api.example.test/search", {}, http=http)
assert len(result) == 60
timeouts = [c.kwargs["timeout"] for c in http.get.call_args_list]
assert timeouts == [8.0, 7.0, 2.0]
The timeout list is the useful assertion: it pins that each request is given only what's left of the budget, and that the loop stops once the budget is spent rather than starting a fourth page.
Key takeaways
- The
timeoutin Pythonrequestsbounds connecting and each socket read, not the total time of a request or of a loop of requests. - When paging, set one deadline before the loop and pass the smaller of the per-request timeout and the time remaining to each call.
- Stop on a short page or a reached total, but treat a missing or non-positive total as unknown rather than trusting it.
- Deduplicate ids across pages, because offset paging over a live index can repeat items when the data shifts between requests.
- Keep earlier pages when a later one fails, and log pages fetched and elapsed time so a partial result can be told apart from a complete one.