Turn 10,000 URLs into AI reference material with Olostep

Your tabs are piling up, and you have not even started comparing

You have found all the material to compare, but your browser is full of open tabs. As you copy the text page by page and attach the original links, the time you actually have to read and compare keeps shrinking. When you see a summary and wonder, “Where did this come from?” you have to dig through the tabs again.

What you planned to hand off to AI as analysis has turned into manual work preparing material for AI to read. If you will repeat this preparation for the next round of research, you need a way to gather the text and sources from the links you found in one place.

Olostep Batches is a tool that lets you label and submit each URL, then retrieve the text for items once processing is finished. Here, we will start with two public documents and create reference material that lets you trace an AI answer back to the original text. docs.olostep +2

Start with two addresses in the browser, no installation needed

You can run your first collection from the Try it request screen in the official API documentation. No separate code is needed, but you do need an Olostep account, an API token issued from the dashboard, and credits to use for the request. API requests are authenticated in the Authorization: Bearer <API-TOKEN> format. docs.olostep

1. Sign up for or log in to Olostep. Then create a token on the API Keys page and check the credits available to you in the dashboard. docs.olostep
2. Open Try it in the Create Batch documentation. Enter your API token in the authentication field as instructed on screen. If your environment requires entering the entire header manually, leave one space between Bearer and the token. docs.olostep +1

Give your two links names that are easy to find

Create two entries in items and enter a custom_id and url for each. The custom_id is the name that lets you identify which address a retrieved text came from later. For the first request, you can leave optional settings such as parsers blank and first confirm that you can retrieve the Markdown text. docs.olostep +2

Your request data should look like this.

{"items":[
  {"custom_id":"batches-guide","url":"https://docs.olostep.com/features/batches"},
  {"custom_id":"retrieve-guide","url":"https://docs.olostep.com/api-reference/retrieve"}
]}

It is not finished until you receive the text, not just a receipt number

A batch proceeds in the order of submission, completion check, item lookup, and text retrieval. Right after creation, you may receive batch information with an in_progress status instead of the text, so do not stop after receiving only the first response. docs.olostep +1

1. Submit the two URLs in Create Batch. Save the newly returned id from the response. This value is the batch_id used in later requests. It is important not to confuse example values shown in the documentation with the value in your actual response. docs.olostep
2. Check the status in Batch Info. Look it up with the saved batch_id, then move on when status is completed. Processing time can vary by target page and account environment, so do not automate on the assumption of a fixed completion time. docs.olostep
3. Get the per-document identifiers in Batch Items. Completed items include the custom_id, original url, and retrieve_id to use for text retrieval. Look up failed items separately with status=failed. docs.olostep
4. Retrieve the text in Retrieve Content. Query each item using its retrieve_id and formats=markdown. Save the response's markdown_content with that item's custom_id and original URL. docs.olostep

If your request is blocked, classify the error first. 401 or invalid_api_key signals that you should check the token and Bearer format; 402 or credits_exhausted signals that you should check your credits. If a document is missing from the completed list, look up failed items separately. If size_exceeded=true, you need to retrieve and store the content from the provided hosted URL instead of the response body. This URL expires after 7 days. docs.olostep +3

This was written after reviewing the official documentation and public request screens; it is not the result of directly running batch creation and text retrieval with a valid API token.

read -rs OLOSTEP_API_KEY
export OLOSTEP_API_KEY

curl --fail-with-body --silent --show-error \
  https://api.olostep.com/v1/batches \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"items":[{"custom_id":"batches-guide","url":"https://docs.olostep.com/features/batches"},{"custom_id":"retrieve-guide","url":"https://docs.olostep.com/api-reference/retrieve"}]}' \
  > batch.json
batch_id=$(jq -er '.id' batch.json)

curl --fail-with-body --silent --show-error \
  "https://api.olostep.com/v1/batches/$batch_id" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  | jq -r '.status'

curl --fail-with-body --silent --show-error --get \
  "https://api.olostep.com/v1/batches/$batch_id/items" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  --data-urlencode 'status=completed' \
  --data-urlencode 'limit=50' \
  > items.json
jq '{items, cursor}' items.json

retrieve_id=$(jq -er '.items[] | select(.custom_id == "batches-guide") | .retrieve_id' items.json)
curl --fail-with-body --silent --show-error --get \
  https://api.olostep.com/v1/retrieve \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  --data-urlencode "retrieve_id=$retrieve_id" \
  --data-urlencode 'formats=markdown' \
  > content.json
jq -e '.markdown_content | length > 0' content.json

Specify the answer format for the material you collected

Once you have the text, it is better than asking something broad like “Tell me how to use Olostep” to request that AI answer only from the attached material and show the original URLs.

Read only the two attached documents and summarize the batch limit for a new account and the steps for retrieving text. Add the original document URL to each answer, and mark anything not in the documents as unconfirmed.

The answer confirmed in the official documentation is that new accounts have a limit of 100 items per batch, and that you need to contact the provider to raise the limit. If you need structured JSON, you must specify a parser suited to the target when submitting the batch. You can retrieve Markdown or HTML without a parser, but no parser or output quality is guaranteed to work universally across every site. docs.olostep +1

Scale to 10,000 only after the two URLs pass

Olostep's official batch maximum is 10,000 URLs, but the starting limit for new accounts is 100 per batch. So after confirming the pairing of text and source for two documents, it is more realistic to expand to 100 and then to larger groups. docs.olostep

• Check for omissions. Do not look only at the completed list; query the failed list separately and compare results against every submitted custom_id. docs.olostep
• Read through the cursor to the end. If Batch Items returns a cursor, pass it in the next request. Do not save only the first page's maximum of 50 items and decide that the whole batch is complete. docs.olostep
• Do not trust array order. Use custom_id, not result position, to connect the original URL and retrieved text.
• Store the text yourself. Hosted URLs expire after 7 days, so move material you will reuse to separate storage right after the job. docs.olostep

Batches are a good fit when you already have a URL list and repeatedly need to collect many pages at once. If you only need to read one page immediately, the single-scraping approach described in the official documentation may be simpler. docs.olostep

The goal is not to make 10,000 requests succeed. It is to create reference material where you can reopen the original address for every text AI has read, and find missing items to process again.