Using crawl, search, and paginate-api steps

Use these three Context steps to bring outside data into a Flow. Crawl Website collects content from many pages of one site. AI Search searches the web. Paginate API requests every page of a paginated JSON API and combines the results into one array.

Before you begin

Open a flow in the Flow editor. To add any of these steps, click Add step and select the step from the Context group. If an API needs credentials, create a secret in Managing secrets first.

Crawl a website

Crawl Website starts a crawl job with Cloudflare Browser Rendering, waits for the job to finish, and stores the crawled pages in its output variable. The default output variable is crawl_result.

Configure the following fields:

Start crawl from URL: the first page to crawl. You can include flow variables. If the URL has no scheme, Runtype adds https://.
crawl up to and pages with depth: the maximum number of pages to collect and how many links deep to follow from the start page.
using source: All, Sitemaps, or Links. This setting controls how the crawler finds pages.
render pages: turn this on for sites that build their content with JavaScript. It’s off for a step that you add in the Flow editor.
Formats: one or more of Markdown, HTML, Links, and JSON. The default is Markdown.

The Advanced settings add Include patterns and Exclude patterns (for example, /docs/**), Follow external links, Follow subdomains, wait for selector, max age (seconds) for cached pages, and Reject resource types.

Crawl credentials

By default, the step uses Runtype’s Cloudflare account, and Runtype counts each crawled page as platform-key spend. To use your own Cloudflare account, go to Settings > Integrations. In Web Scraping, find Cloudflare Browser Rendering, and enter your Cloudflare Account ID and API Token. For how platform-key spend is billed, see Platform keys vs. BYOK.

Crawl output

The output variable holds an object with these fields:

crawlId: the ID of the crawl job.
status: completed for a successful crawl.
result: the first page of results.
resultPages: every page of results, including the first.

Each results page has a records array. Each record has the page’s url and status, plus the content for the formats that you chose, such as markdown. Runtype stops collecting results when it reaches the crawl up to limit.

Crawl timing

A crawl waits up to 2 minutes for the job to finish, checking its status every 2 seconds. To change these values, set completionTimeoutMs and pollIntervalMs in the flow definition. A crawl that runs longer than completionTimeoutMs fails with Cloudflare crawl timed out.

Batch runs, evals, and Schedules can’t run a Flow that contains a Crawl Website step. For details, see Timeouts and long-running work.

Search the web

AI Search sends a query to a search provider and stores the results. The default output variable is search_results.

Configure the following fields:

Search for: the query. You can include flow variables, such as {{topic}}.
using: the search provider. The list shows the search-capable models that you enabled in Managing AI models. If Exa is available for your account, Exa Search API also appears. If the list is empty, the step shows No search models configured.
returning: the number of results. The default is 10.
with sources: ask the provider to include citations.

The Advanced settings add a Date Range, domain filters (Only search these domains and Exclude these domains), and Model Parameters such as temperature, max tokens, user country, and user region.

The output variable holds an object with query, results, citations, and metadata. What results contains depends on the provider:

Exa Search API returns one result per web page, with title, url, content, and other fields. For Exa options and output fields, see Exa (web search).
A model returns one result whose content is the model’s full written answer. The step fills citations with the URLs that appear in that answer.

A later Prompt step can read the results with {{search_results.results}}.

In a flow definition, an AI Search step has "type": "search". Set provider to exa, or to a model in the form PROVIDER:MODEL_ID, the same value that the Flow editor saves when you select a model.

Collect every page from an API

Paginate API requests one page of a JSON API at a time, extracts the items from each page, and stores all of them as one array. The default output variable is fetched_entities.

Configure the following fields:

Fetch paginated data from: the endpoint URL. The same URL rules as Fetch URL apply: the URL must use HTTPS and point to a public host.
using: GET or POST. For POST, enter a Request body. Runtype sends the same body for every page.
Headers: add key-value pairs, such as Authorization: Bearer {{secret:API_KEY}}.
Pagination: the pagination type, with the page size and the maximum number of pages to request.
extract entities from: a JSONPath to the array of items in each response, such as $.data[*]. Use $[*] when the response body is the array.

The JSONPath supports dot-separated keys, [*] for every item in an array, and a numeric index such as [0]. It doesn’t support filter expressions.

Pagination types

The following table lists what Runtype sends for each pagination type and when it stops requesting pages:

Type	Request parameters	Stops when
Offset/Limit	`offset` and `limit` query parameters	A page returns fewer items than the page size, or the offset reaches the value at `totalPath`
Cursor-based	The `cursor` query parameter, from the second page on	The value at cursor path in response is empty
Page Number	`page` and `per_page` query parameters, starting from page 1	A page returns fewer items than the page size, or the page passes the value at `totalPagesPath`
Link Header	None. Runtype requests the URL from the response’s `rel="next"` link	The response has no `rel="next"` link

You can rename each query parameter in the step. For Page Number, you can also change the start page. totalPath and totalPagesPath are JSONPaths that you set in the flow definition, inside paginationConfig.

For Cursor-based pagination, always set cursor path in response. Without it, Runtype never sends a cursor, so it requests the first page again until it reaches the maximum number of pages.

Limits and retries

A step that you add in the Flow editor requests up to 10 pages. A step in a flow definition with no maxPages requests up to 100. Each request times out after 30 seconds.

The following flow-definition fields control the rest of the step’s behavior:

maxEntities: stop after collecting this many items.
requestDelayMs: wait this many milliseconds between pages.
maxRetries: retry a failed request this many times. The default is 3. Runtype retries only a network error, a timeout, or a 408, 429, or 5xx response. Other 4xx responses fail the step at once. Between retries, Runtype waits for the Retry-After header when the response has one, or else 1, 2, 4 seconds and so on. Either wait is at most 30 seconds, except for a 429 (see retryOnRateLimit).
retryOnRateLimit: when the API returns 429, wait for the time in its Retry-After header (a number of seconds or an HTTP date, at most 10 minutes; 5 seconds if the header is missing), then retry. The default is true.
retryPost: when true, a POST step retries failed requests like a GET step does: network errors, timeouts, and 408, 429, and 5xx responses. The default is false, because the API may already have acted on the failed request, so a POST step fails on the first error.
includeMetadata: when true, also write OUTPUT_metadata with totalEntities, pagesProcessed, and paginationType, where OUTPUT is the output variable name.

Define a paginate-api step in code

This step collects every order from a cursor-paginated API, 50 items per page:

{
  "type": "paginate-api",
  "name": "Fetch all orders",
  "config": {
    "http": {
      "url": "https://api.example.com/orders",
      "method": "GET",
      "headers": { "Authorization": "Bearer {{secret:ORDERS_API_KEY}}" }
    },
    "paginationType": "cursor",
    "paginationConfig": {
      "cursorParam": "after",
      "cursorPath": "$.meta.next_cursor"
    },
    "pageSize": 50,
    "maxPages": 200,
    "entitiesPath": "$.data[*]",
    "outputVariable": "orders",
    "includeMetadata": true
  }
}

Replace ORDERS_API_KEY with the name of your managed secret. The step writes the orders to orders and the counts to orders_metadata. For how to create and update flows from code, see Manage flows as code.

Handle errors

A step that you add in the Flow editor starts with Stop on error. To keep the Flow running when a step fails, open the step’s error handling control and choose Continue on error. The step then writes its default value to the output variable, or one of the following values if no default is set:

Crawl Website: null.
AI Search: { "error": "...", "results": [] }.
Paginate API: an empty array.

In a flow definition with no errorHandling field, Crawl Website and AI Search continue on error, and Paginate API stops the Flow. For more information, see When a step fails.

Troubleshoot these steps

These errors are the most common:

Cloudflare crawl timed out after: the crawl didn’t finish within completionTimeoutMs. Lower crawl up to, or raise completionTimeoutMs.
Cloudflare crawl ended with status: Cloudflare stopped the crawl, for example because of a limit on your Cloudflare account. Check the status in the message.
Exa API key required but not provided: Exa isn’t available for your account. Add an Exa API key in Settings > Integrations.
API key required for: the search model’s provider has no API key. The message says Settings > AI Models, but the section is Settings > Models. Add the key there.
HTTP followed by a status code from Paginate API: the API rejected a page request after every retry. The message includes the API’s error text.
An empty array from Paginate API: extract entities from doesn’t match the response. Run a Fetch URL step against the same URL and check where the items are.

Next steps

Continue with these guides:

Using fetch-url and api-call steps: fetch a single page or call a JSON API once.
Using transform-data steps: filter or reshape the collected results.
Firecrawl (web scraping): scrape pages that block direct requests.
Flow step types overview: compare every step type.