> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Using crawl, search, and paginate-api steps

Use these three Context steps to bring outside data into a Flow. **Crawl Website** collects content from many pages of one site. **AI Search** searches the web. **Paginate API** requests every page of a paginated JSON API and combines the results into one array.

## Before you begin

Open a flow in the **Flow editor**. To add any of these steps, click **Add step** and select the step from the **Context** group. If an API needs credentials, create a secret in [Managing secrets](/user-guide/settings/managing-secrets) first.

## Crawl a website

**Crawl Website** starts a crawl job with Cloudflare Browser Rendering, waits for the job to finish, and stores the crawled pages in its output variable. The default output variable is `crawl_result`.

Configure the following fields:

* **Start crawl from URL**: the first page to crawl. You can include flow variables. If the URL has no scheme, Runtype adds `https://`.
* **crawl up to** and **pages with depth**: the maximum number of pages to collect and how many links deep to follow from the start page.
* **using source**: **All**, **Sitemaps**, or **Links**. This setting controls how the crawler finds pages.
* **render pages**: turn this on for sites that build their content with JavaScript. It's off for a step that you add in the **Flow editor**.
* **Formats**: one or more of **Markdown**, **HTML**, **Links**, and **JSON**. The default is **Markdown**.

The **Advanced** settings add **Include patterns** and **Exclude patterns** (for example, `/docs/**`), **Follow external links**, **Follow subdomains**, **wait for selector**, **max age (seconds)** for cached pages, and **Reject resource types**.

### Crawl credentials

By default, the step uses Runtype's Cloudflare account, and Runtype counts each crawled page as platform-key spend. To use your own Cloudflare account, go to **Settings > Integrations**. In **Web Scraping**, find **Cloudflare Browser Rendering**, and enter your **Cloudflare Account ID** and **API Token**. For how platform-key spend is billed, see [Platform keys vs. BYOK](/user-guide/getting-started/platform-keys-vs-byok).

### Crawl output

The output variable holds an object with these fields:

* `crawlId`: the ID of the crawl job.
* `status`: `completed` for a successful crawl.
* `result`: the first page of results.
* `resultPages`: every page of results, including the first.

Each results page has a `records` array. Each record has the page's `url` and `status`, plus the content for the formats that you chose, such as `markdown`. Runtype stops collecting results when it reaches the **crawl up to** limit.

### Crawl timing

A crawl waits up to 2 minutes for the job to finish, checking its status every 2 seconds. To change these values, set `completionTimeoutMs` and `pollIntervalMs` in the flow definition. A crawl that runs longer than `completionTimeoutMs` fails with `Cloudflare crawl timed out`.

Batch runs, evals, and Schedules can't run a Flow that contains a **Crawl Website** step. For details, see [Timeouts and long-running work](/user-guide/flows/timeouts-and-long-running-work#wait-for-slow-external-work-with-durable-pauses).

## Search the web

**AI Search** sends a query to a search provider and stores the results. The default output variable is `search_results`.

Configure the following fields:

* **Search for**: the query. You can include flow variables, such as `{{topic}}`.
* **using**: the search provider. The list shows the search-capable models that you enabled in [Managing AI models](/user-guide/settings/managing-ai-models). If Exa is available for your account, **Exa Search API** also appears. If the list is empty, the step shows **No search models configured.**
* **returning**: the number of results. The default is 10.
* **with sources**: ask the provider to include citations.

The **Advanced** settings add a **Date Range**, domain filters (**Only search these domains** and **Exclude these domains**), and **Model Parameters** such as **temperature**, **max tokens**, **user country**, and **user region**.

The output variable holds an object with `query`, `results`, `citations`, and `metadata`. What `results` contains depends on the provider:

* **Exa Search API** returns one result per web page, with `title`, `url`, `content`, and other fields. For Exa options and output fields, see [Exa (web search)](/user-guide/integrations/exa-web-search).
* A model returns one result whose `content` is the model's full written answer. The step fills `citations` with the URLs that appear in that answer.

A later Prompt step can read the results with `{{search_results.results}}`.

In a flow definition, an AI Search step has `"type": "search"`. Set `provider` to `exa`, or to a model in the form `PROVIDER:MODEL_ID`, the same value that the **Flow editor** saves when you select a model.

## Collect every page from an API

**Paginate API** requests one page of a JSON API at a time, extracts the items from each page, and stores all of them as one array. The default output variable is `fetched_entities`.

Configure the following fields:

* **Fetch paginated data from**: the endpoint URL. The same URL rules as Fetch URL apply: the URL must use HTTPS and point to a public host.
* **using**: `GET` or `POST`. For `POST`, enter a **Request body**. Runtype sends the same body for every page.
* **Headers**: add key-value pairs, such as `Authorization: Bearer {{secret:API_KEY}}`.
* **Pagination**: the pagination type, with the page size and the maximum number of pages to request.
* **extract entities from**: a JSONPath to the array of items in each response, such as `$.data[*]`. Use `$[*]` when the response body is the array.

The JSONPath supports dot-separated keys, `[*]` for every item in an array, and a numeric index such as `[0]`. It doesn't support filter expressions.

### Pagination types

The following table lists what Runtype sends for each pagination type and when it stops requesting pages:

| Type             | Request parameters                                                   | Stops when                                                                                      |
| ---------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **Offset/Limit** | `offset` and `limit` query parameters                                | A page returns fewer items than the page size, or the offset reaches the value at `totalPath`   |
| **Cursor-based** | The `cursor` query parameter, from the second page on                | The value at **cursor path in response** is empty                                               |
| **Page Number**  | `page` and `per_page` query parameters, starting from page 1         | A page returns fewer items than the page size, or the page passes the value at `totalPagesPath` |
| **Link Header**  | None. Runtype requests the URL from the response's `rel="next"` link | The response has no `rel="next"` link                                                           |

You can rename each query parameter in the step. For **Page Number**, you can also change the start page. `totalPath` and `totalPagesPath` are JSONPaths that you set in the flow definition, inside `paginationConfig`.

For **Cursor-based** pagination, always set **cursor path in response**. Without it, Runtype never sends a cursor, so it requests the first page again until it reaches the maximum number of pages.

### Limits and retries

A step that you add in the **Flow editor** requests up to 10 pages. A step in a flow definition with no `maxPages` requests up to 100. Each request times out after 30 seconds.

The following flow-definition fields control the rest of the step's behavior:

* `maxEntities`: stop after collecting this many items.
* `requestDelayMs`: wait this many milliseconds between pages.
* `maxRetries`: retry a failed request this many times. The default is 3. Runtype retries only a network error, a timeout, or a `408`, `429`, or `5xx` response. Other `4xx` responses fail the step at once. Between retries, Runtype waits for the `Retry-After` header when the response has one, or else 1, 2, 4 seconds and so on. Either wait is at most 30 seconds, except for a `429` (see `retryOnRateLimit`).
* `retryOnRateLimit`: when the API returns `429`, wait for the time in its `Retry-After` header (a number of seconds or an HTTP date, at most 10 minutes; 5 seconds if the header is missing), then retry. The default is `true`.
* `retryPost`: when `true`, a `POST` step retries failed requests like a `GET` step does: network errors, timeouts, and `408`, `429`, and `5xx` responses. The default is `false`, because the API may already have acted on the failed request, so a `POST` step fails on the first error.
* `includeMetadata`: when `true`, also write `OUTPUT_metadata` with `totalEntities`, `pagesProcessed`, and `paginationType`, where `OUTPUT` is the output variable name.

### Define a paginate-api step in code

This step collects every order from a cursor-paginated API, 50 items per page:

```json
{
  "type": "paginate-api",
  "name": "Fetch all orders",
  "config": {
    "http": {
      "url": "https://api.example.com/orders",
      "method": "GET",
      "headers": { "Authorization": "Bearer {{secret:ORDERS_API_KEY}}" }
    },
    "paginationType": "cursor",
    "paginationConfig": {
      "cursorParam": "after",
      "cursorPath": "$.meta.next_cursor"
    },
    "pageSize": 50,
    "maxPages": 200,
    "entitiesPath": "$.data[*]",
    "outputVariable": "orders",
    "includeMetadata": true
  }
}
```

Replace `ORDERS_API_KEY` with the name of your managed secret. The step writes the orders to `orders` and the counts to `orders_metadata`. For how to create and update flows from code, see [Manage flows as code](/user-guide/flows/manage-flows-as-code).

## Handle errors

A step that you add in the **Flow editor** starts with **Stop on error**. To keep the Flow running when a step fails, open the step's error handling control and choose **Continue on error**. The step then writes its default value to the output variable, or one of the following values if no default is set:

* **Crawl Website**: `null`.
* **AI Search**: `{ "error": "...", "results": [] }`.
* **Paginate API**: an empty array.

In a flow definition with no `errorHandling` field, **Crawl Website** and **AI Search** continue on error, and **Paginate API** stops the Flow. For more information, see [When a step fails](/user-guide/flows/flow-step-types-overview#when-a-step-fails).

## Troubleshoot these steps

These errors are the most common:

* **`Cloudflare crawl timed out after`**: the crawl didn't finish within `completionTimeoutMs`. Lower **crawl up to**, or raise `completionTimeoutMs`.
* **`Cloudflare crawl ended with status`**: Cloudflare stopped the crawl, for example because of a limit on your Cloudflare account. Check the status in the message.
* **`Exa API key required but not provided`**: Exa isn't available for your account. Add an Exa API key in **Settings > Integrations**.
* **`API key required for`**: the search model's provider has no API key. The message says **Settings > AI Models**, but the section is **Settings > Models**. Add the key there.
* **`HTTP` followed by a status code** from **Paginate API**: the API rejected a page request after every retry. The message includes the API's error text.
* **An empty array from Paginate API**: **extract entities from** doesn't match the response. Run a Fetch URL step against the same URL and check where the items are.

## Next steps

Continue with these guides:

* [Using fetch-url and api-call steps](/user-guide/flows/using-fetch-url-and-api-call-steps): fetch a single page or call a JSON API once.
* [Using transform-data steps](/user-guide/flows/using-transform-data-steps): filter or reshape the collected results.
* [Firecrawl (web scraping)](/user-guide/integrations/firecrawl-web-scraping): scrape pages that block direct requests.
* [Flow step types overview](/user-guide/flows/flow-step-types-overview): compare every step type.