Firecrawl (web scraping)

Use Firecrawl to scrape and extract content from web pages. Firecrawl is available as a fetch method on the Fetch URL Flow step and as a built-in tool for Agents.

When to use Firecrawl

Firecrawl renders the page, including JavaScript, and returns it in the formats that you request. Use it instead of a Standard HTTP fetch when a site blocks, rate-limits, or challenges automated requests, or when you need the page as Markdown or structured JSON.

The Flow step and the Agent tool support different options. The following table compares them:

To extract structured JSON from a page, use the Fetch URL step.

Use your own Firecrawl API key (optional)

Runtype provides a platform key for Firecrawl, so you can scrape pages without a Firecrawl account. Scrapes that use the platform key count toward your Runtype usage. Scrapes that use your own key are billed by Firecrawl to your Firecrawl account.

To use your own Firecrawl account, follow these steps:

1. Sign up for an account at Firecrawl.
2. Get your API key from the Firecrawl dashboard.
3. In Runtype, open Settings > Integrations.
4. Find Firecrawl.
5. In Firecrawl API Key, enter your API key.
6. Click Save Key.

To go back to the platform key, click Remove Key on the Firecrawl card.

Use Firecrawl in a Flow

To use Firecrawl in a Flow, follow these steps:

1. Open a Flow.
2. Click Add step.
3. Select Fetch URL.
4. In the Fetch selector, choose Firecrawl (Web Scraping).
5. In from URL, enter the page URL or a {{variable}} reference, such as {{_record.metadata.website}}. If the URL has no scheme, Runtype adds https://.
6. In output as, select one or more output formats. The default is Markdown.
7. Optional: To drop navigation, headers, and footers, turn on Extract main content only.
8. If you selected JSON, the JSON Extraction section opens. Enter a JSON Schema, or describe the data in or describe what to extract.
9. Optional: To change caching, location, and language, open Advanced:
   • Cache results: store the scrape in Firecrawl’s cache.
   • cache for: the maximum age, in milliseconds, of a cached copy that Firecrawl can return. For example, 172800000 is 2 days. This field shows only when Cache results is on.
   • country: a country code for the request, such as US.
   • languages: a comma-separated list of language codes, such as en, es.

If you set both a JSON schema and a description, the step uses the schema. If the schema isn’t valid JSON, the step ignores it and doesn’t use the description either. Check that the schema parses before you run the Flow.

Step output

The step writes an object to its output variable. The object has one key for each format that you requested (markdown, html, rawHtml, screenshot, links, or json) and a metadata key. The metadata object includes sourceURL and statusCode.

If the output variable is page, reference the result in a later step:

{{page.markdown}}
{{page.json}}
{{page.metadata.statusCode}}

The result of JSON extraction is under json.

Define the step in code

This flow definition step scrapes a pricing page and extracts the plan names and prices as JSON:

{
  "type": "fetch-url",
  "name": "Scrape pricing",
  "config": {
    "http": { "url": "https://PRICING_PAGE_URL" },
    "fetchMethod": "firecrawl",
    "firecrawl": {
      "formats": ["markdown", "json"],
      "onlyMainContent": true,
      "jsonPrompt": "Extract each plan name and its monthly price."
    },
    "outputVariable": "page"
  }
}

Replace PRICING_PAGE_URL with the page to scrape. The firecrawl object accepts the following fields:

• formats: the output formats. The default is ["markdown"].
• onlyMainContent: when true, returns only the page’s main content.
• jsonSchema: a JSON Schema for JSON extraction, as an object or a string.
• jsonPrompt: a plain-language description of the data to extract.
• storeInCache: when false, Firecrawl doesn’t cache the scrape.
• maxAge: the maximum age of a cached copy, in milliseconds.
• location: an object with country and languages, such as { "country": "US", "languages": ["en"] }.
• actions: a list of click, write, press, wait, or screenshot actions that run on the page before the scrape. The Flow editor doesn’t show this option.

A later prompt step can then use {{page.json}} to compare the plans. For the other Fetch URL settings, see Using fetch-url and api-call steps.

Handle failed scrapes

A failed scrape follows the step’s error handling. With Continue on error, which is the default, the Flow keeps going and the output variable holds the step’s default value, or null if no default is set. With Stop on error, the run ends with an error that starts with Failed to fetch URL with Firecrawl. When you scrape many URLs in a batch, check the output before you use it. For the error-handling modes, see Error handling.

Use Firecrawl as a tool

Add the Firecrawl built-in tool to an Agent. The Agent can then call Firecrawl during an execution to scrape a page. To add it in the dashboard, see Add tools to an Agent. In an Agent definition, add the tool ID builtin:firecrawl to tools.toolIds.

The model passes the following arguments when it calls the tool:

• url (required): the full URL of the page to scrape.
• formats: the output formats. Accepts markdown (default), html, rawHtml, screenshot, and links.
• onlyMainContent: when true, strips navigation, footers, and ads. The default is true.
• waitFor: the time to wait for dynamic content before the scrape, in milliseconds, from 0 to 30000.

The tool returns success, url, metadata, and one field for each format that the model requested.

Troubleshooting

You can see the following errors:

• Firecrawl API key not configured: no Firecrawl key is available. If you use the platform key, check your credit balance in Settings > Billing. When platform credit runs out, the Fetch URL step reports this error. To keep scraping, add your own key in Settings > Integrations.
• Platform key spend credit is exhausted: the Agent tool reports this error when platform credit runs out. Restore your credit balance, or add your own Firecrawl key.
• Firecrawl API error followed by a status code: Firecrawl rejected the request. The message includes Firecrawl’s response, which usually names the problem, such as an invalid key or an unreachable URL.
• The json key is missing or empty: check that the JSON schema is valid JSON, or remove it and use a description.

For more checks, see Firecrawl scraping failed.

Next steps

Choose a guide for your next task:

• Using fetch-url and api-call steps: configure other Fetch URL settings
• Built-in tools: compare Firecrawl with the other built-in tools
• Agent tools: control how an Agent uses its tools
• Massive (web rendering): fetch pages that resist ordinary requests
• Exa (web search): find pages to scrape