Skip to content

Best practices

Fetch

Read whole pages reliably, and only the ones you need, with the options you already have.

These practices help you read pages with Telem reliably, and only as much as you need. Each one uses an option that is described in full on Fetch parameters.

Search first, then fetch the pages you keep

Section titled “Search first, then fetch the pages you keep”

Search returns a snippet for each result. A fetch returns the whole page. Search first, decide which results you need, then send only those URLs to fetch. Asking search for whole pages (include_full_content) returns the full text of every result, and it works only at the extended and max search tiers. See Search best practices.

One call reads up to five URLs at the same time. You get one result for each URL, in the order you sent them.

{
"urls": ["https://example.com/a", "https://example.com/b"]
}
  • Use http and https URLs only.
  • Remove duplicates first. A call that repeats a URL gets a 400 (duplicate_urls).
  • Split longer lists. More than five URLs gets a 400 (over_url_cap).

A URL that fails does not fail the call, and it is not charged. The call returns 200, and the failed URL comes back with status: "failed" and an error that gives the reason. Use the results that succeeded, then retry or drop the others. Pinned providers are the exception: when they fail on every URL, the call returns a 502.

A result that succeeded can still be an error page. Check http_status, when it is present, and warnings: a site’s 404 page can come back as succeeded.

{
"url": "https://example.com/missing-page",
"status": "succeeded",
"http_status": 404,
"title": "Example Domain",
"warnings": [
{
"code": "provider_soft_error",
"message": "firecrawl: the page reported an error (upstream 404, reason: not_found)"
}
]
}

Not every provider reports http_status.

Telem tries several fetch providers, one after another, until one returns the page. It moves on when a page comes back blocked: a login wall, a paywall, a bot check, or a page that needs JavaScript. For some kinds of link, such as YouTube videos, Reddit posts, X posts and TikTok videos, Telem first tries a provider made for that site.

So leave options.providers out unless you need one provider. A pinned list is tried in your order, and no other provider is tried: if the pinned providers fail, the URL fails. See Providers.

Each result’s provider says which provider answered. From the extended tier, provider_chain lists every provider that was tried, and how each one ended.

The tier decides how much each page result includes, and what Telem asks the provider for. The default is default.

Tier Each result adds Cost Use it when
minimalist The page text in content, title, canonical_url, http_status and provider The base price You need the text only
default description, content_length, content_sha256, js_rendered and pdf_mode The base price Most reads
extended summary, links, images, publish_date, author, language, html, raw_html, chunks, screenshot, metadata, provider_chain and usage The base price on most providers. jina returns more text, and it charges by the text You need links, images, the date or the HTML
max media, transcript, excerpt, structured, subpages and enrichments More on exa (a summary, highlights and subpages) and on firecrawl (5 credits instead of 1, for structured) You need structured data, transcripts or media
  • The tier never changes which providers Telem tries. A provider that cannot fill a field still answers: the field is null, and a capability_gap warning says why.
  • A saved copy answers its own tier and the tiers below it. A page that Telem saved at default is read again for an extended request.
  • On the MCP server, and so in Claude Code and Codex, max is lowered to extended.

For the exact fields, see Tier.

  • Tier: use the lowest tier that has the fields you read. See Choose a tier.
  • Metadata only: inline_content: false leaves out the page text in content.
  • Length: each page’s text is cut at 20,000 characters by default, with content_truncated: true. Set inline_max_chars to change the cap.
{
"urls": ["https://example.com/a"],
"options": { "tier": "extended", "inline_max_chars": 8000 }
}

Telem can answer with a saved copy of a page that it read in the last day. When you need the page as it is now, set no_cache: true. To accept a saved copy only up to a certain age, set max_age_seconds. 0 reads the page again.

{
"urls": ["https://example.com/prices"],
"options": { "no_cache": true }
}

The Python and JavaScript SDKs take providers, tier, no_cache, content_format, inline_content and inline_max_chars. JavaScript uses camel case: noCache, contentFormat, inlineContent and inlineMaxChars. The SDKs do not take max_age_seconds yet. To set it, send a direct POST /v1/fetch call.

An agent’s telem_fetch tool takes the URLs only. It removes duplicates and reads up to five pages in one call. The fetch provider, the tier and live reads come from your settings: see Configuration.

Was this page helpful?