What’s harder than scraping 10,000 URLs at once is making the resulting 10,000 records safe for an agent to use. Olostep’s real value is not bulk requests themselves, but turning a URL list into a data pipeline that can identify, structure, validate, and reprocess data.
10,000 URLs are a data-contract problem, not a scraping job
Olostep offers single-URL scraping, site crawling, URL maps, bulk batches, search, and answers as one API product suite. Its Product Hunt introduction also highlights processing large URL lists as production data pipelines and organizing results as Markdown, HTML, JSON, and more. If you already have a list of URLs to collect, /v1/batches is the right starting point.
A batch can contain up to 10,000 URLs, and the official documentation estimates processing time at roughly 5–8 minutes regardless of batch size. New accounts may initially be limited to 100 items per batch, however, so check your account limit before attempting a real 10,000-item run. The guidance also says that for fewer than 50 items, sending parallel single-scraping requests can be faster than using a batch. Treat this timing as documentation guidance, not an SLA reflecting your target sites and network conditions.
Don’t start by submitting 10,000 items.
It is safer to validate 20, then 100, then 1,000 items using the same parser and field contract before scaling up. Types that are absent from a small sample—login pages, out-of-stock products, region-specific views, and JavaScript-delayed loading—can become hundreds of silent errors in a bulk run.
The first thing to define is not the URL, but the contract for one output row. For product monitoring, for example, choose the fields an agent will actually use for decisions first: source_url, canonical_url, title, price, currency, availability, and observed_at. In JSON Schema, listing a field in properties does not automatically make it required, so you must specify required separately. To block unexpected keys, you can also declare additionalProperties: false.
Olostep batches have three stages: submit, complete, and retrieve
A batch request contains an items array, and each item receives a url and custom_id. The official example creates the custom_id from part of a URL’s SHA-256 hash. In practice, first confirm whether any paths must retain case, then apply rules such as removing tracking parameters and fragments and normalizing hosts before creating the ID from the result. This reduces duplicate storage of the same page under different IDs.
| Pipeline state | Values to always store | Next decision |
|---|---|---|
| Before submission | custom_id, original URL, normalized URL, schema version | Check duplicates and collection-permission scope |
| Batch created | batch_id, submission time, item count | Receive webhook or check status |
| Item completed | custom_id, retrieve_id, processing status | Retrieve content |
| Validation completed | Validation result, error code, raw-data retention location | Load or move to the reprocessing queue |
If structured JSON is the goal, provide a parser ID suited to the target site when creating the batch. If you instead plan to structure the source content later with separate logic, you can retrieve Markdown or HTML. Split responsibilities so the parser turns page structure into fields and your JSON Schema validator checks whether those results honor your internal contract. The key is not to treat parsing success and passing data-quality checks as the same state.
When a batch finishes, retrieve its item list with GET /v1/batches/{batch_id}/items. The list API supports completed and failed status filters; when there are many results, pass the previous response’s cursor to the next request to paginate through them. The documentation recommends retrieving 10–50 items at a time. To avoid marking 10,000 items as complete after receiving only the first 50, continue until no cursor remains.
Pass each completed item’s retrieve_id to /v1/retrieve to retrieve only the format you need. The response’s json_content is a string, so it needs another round of JSON parsing and schema validation. The official batch example says hosted content is retained for seven days, so instead of storing only links, move the raw content and structured results you need into your own storage immediately after the job completes.
Agents need an additional validation and reprocessing gate
Put only successful rows into the dataset read by agents, and separate collection failures from quality failures into different queues. HTTP errors and timeouts are collection failures; a price represented as a string with no currency, or an empty title, are quality failures. You can retry the former with exponential backoff, but retrying the latter without fixing the parser or schema is likely to repeat the same result.
| Status | Example | How to handle it |
|---|---|---|
| Delivery failure | Connection error, temporary server error | Backoff retries with a retry limit |
| Collection failure | Blocked page, deleted URL | Create a separate batch for failed items only |
| Parsing failure | Invalid JSON syntax, field type mismatch | Preserve the raw content, then fix the parser |
| Semantic quality failure | Price of 0, empty title, stale observation | Validate with business rules or have a person review it |
You can replace polling for completion with webhooks. Olostep sends a batch.completed event and retries failed deliveries up to five times over roughly 30 minutes. Because the same event ID is retained across retries, the receiver must use that ID to prevent duplicate processing. Do not run lengthy validation as soon as the webhook arrives; return a 2xx response quickly and process it through an internal queue.
There is also one security caveat. The current webhook documentation labels cryptographic signature verification as “Coming Soon.” Rather than trusting the webhook body alone to finalize data loading, treat it as a trigger to use the received batch_id to recheck the status and items through the authenticated API.
Being able to collect something does not mean you are allowed to collect it. Review the target site’s terms of use and access permissions, and inspect /robots.txt, which provides path-access rules for automated clients. RFC 9309 defines robots.txt as access rules crawlers should follow, not as an authorization or security mechanism. Personal data, content behind login, and copyrighted materials require separate legal review.
How to build a 10,000-URL pipeline in practice
1. Build the output contract with 20 golden samples
Include not only normal pages, but also out-of-stock, deleted, empty-field, region-restricted, and dynamically loaded pages. Define field types, required status, acceptable empty values, and schema_version, then create JSON Schema validation tests.
2. Create a URL manifest and stable IDs
Store custom_id, the original URL, normalized URL, target domain, and collection purpose in a CSV or table. Remove duplicate URLs and mark only rows that have passed robots.txt, terms, and access-permission checks as eligible for submission.
3. Scale batches gradually, starting with 100 items
Send items, a parser ID, and, if needed, a webhook to POST /v1/batches. Confirm the new-account limit, measure success rate, required-field completion rate, and duplicate rate at 100 items, then scale to 1,000 and 10,000.
4. Traverse every cursor and validate the results
Query /items until the cursor disappears, then retrieve each result using its retrieve_id. Send only rows that pass JSON parsing, schema validation, and business-rule validation to an agent index or database.
5. Operate reprocessing queues by failure cause
Retry delivery errors a limited number of times, hold blocked or deleted URLs, and send field errors to a parser-fix queue with their raw content. On the dashboard, prioritize valid-row rate, required-field completion rate, duplicate rate, and post-reprocessing recovery rate over total item count.
If you want to go deeper
Batch Endpoint - Olostep Docs — The core documentation for checking batch size, new-account limits, parsers, and webhook usage. docs.olostep.com
Get the content of multiple websites in one go - Olostep Docs — Python examples covering batch creation, status checks, item retrieval, and content retrieval. docs.olostep.com
Batch Items - Olostep Docs — See the exact request fields for completed/failed filters and cursor-based pagination. docs.olostep.com
JSON Schema - object — Explains how to build a field contract for structured results with required and additionalProperties. json-schema.org
RFC 9309: Robots Exclusion Protocol — Covers the standard behavior and limits of robots.txt to review before automated collection. rfc-editor.org
