Parameters
Fetch parameters
Every option on a Telem page fetch: tiers, the provider chain, the page content settings, batching, and the raw field behind each normalized field.
This page tells you what each fetch option does, and where each field in the
response comes from. You set all fetch options in the request’s options block.
The block itself and a one-line summary of each option are on
Parameters.
Fetch tiers are cumulative, as search tiers are. On every result, the page body
is in the key content. To choose one, see
Fetch best practices.
| Tier | Adds |
|---|---|
minimalist |
content, content_format, canonical_url, title, http_status, warning, provider, cache_state, request_id, latency_ms |
default |
js_rendered, pdf_mode, content_length, content_sha256, description |
extended |
raw_html, html, chunks, screenshot, page_count, content_type, summary, language, author, publish_date, favicon, thumbnail, links, images, metadata, provider_chain, usage |
max |
media, transcript, excerpt, structured, subpages, enrichments |
In the Python SDK, pass tier to fetch() (client.fetch(urls, tier="extended")),
or set TELEM_FETCH_TIER.
Every provider gives the page body and the failure signal. Every provider also
gives title, except these four:
linkupsocialdatax_douyinsocialdatax_kuaishousocialdatax_weibo
Everything else is best-effort. A capability_gap warning names the null fields
that the provider cannot supply.
A paywall, a login wall or a page
that needs JavaScript first makes Telem try the next provider. When no provider gets
the page, the URL usually comes back failed, with the reason, and the rest of the
request still succeeds. A page that is only partly readable can come back as
succeeded, with a partial_content warning. On a pinned chain, when every URL fails
at the provider, the call returns a 502. A site’s own error page can also come
back as succeeded, so check http_status and warnings.
Where each field comes from
Section titled “Where each field comes from”One vocabulary, whoever answered. The chain fails over, so a different provider can answer the same fetch tomorrow. The tabs below show the raw field behind each normalized field, for each provider.
Five fields never come from a provider, so the tabs leave them out:
providerrecords who answered.provider_chainrecords who was tried.content_length,content_sha256andlatency_msare measured by Telem over the body it received.
Two more fields work the same way for every provider:
enrichmentskeeps, verbatim, every provider key that normalization did not use.titleis the one field with a fallback. When the answering provider supplies no title, Telem uses the title it already knows for that URL.
POST api.fetch.tinyfish.ai · document at results[0]
| Normalized | Read from |
|---|---|
content |
text |
content_format |
format. It is markdown when you asked for text, which tinyfish does not offer |
canonical_url / title |
final_url / title |
description |
description |
language / author |
language / author |
publish_date |
published_date |
links / images |
links / image_links (requested only when the tier includes them) |
A page that tinyfish cannot fetch still answers 200. The result list is empty, and the reason is in errors[] as a closed-set code (bot_blocked, timeout, page_not_found, …). That code comes back in a provider_error warning, with the upstream status when tinyfish sent one. Then the chain moves on. etag, last_modified and not_modified go in enrichments.
POST /extract · document at results[0]
| Normalized | Read from |
|---|---|
content |
raw_content, else content / markdown / text |
content_format |
echoes the format you asked for |
canonical_url / title |
url / title |
favicon / images |
favicon / images |
usage / request_id |
usage / request_id (top level) |
A URL that fails goes in failed_results, not in results. Its reason comes back as a warning. The markdown has site-relative links. A relative_links warning flags them, and Telem does not rewrite them. Rewriting the body would make content_sha256 disagree with what tavily sent.
/contents · document at results[0]
| Normalized | Read from |
|---|---|
content |
text |
content_format |
always text |
canonical_url / title |
url / title |
summary |
summary (billed per page, and only requested at max) |
excerpt |
highlights |
author / publish_date |
author / publishedDate |
favicon / thumbnail |
favicon / image |
links / images |
extras.links / extras.imageLinks (max only) |
metadata / subpages |
entities / subpages |
content_format is text whatever you asked for. exa declares no output format. Measurements showed the same call returning plain text for one URL and markdown for another. An echo of the format you asked for could be false, and you could not detect it.
POST /v2/scrape · document at data, page metadata at data.metadata
| Normalized | Read from |
|---|---|
content |
markdown |
content_format |
always markdown |
html / raw_html |
html / rawHtml |
screenshot / summary |
screenshot / summary |
links / images |
links / images |
structured |
json (surcharged, max only) |
canonical_url |
metadata.sourceURL, else metadata.url |
title / description |
metadata.title / metadata.description |
language / author |
metadata.language / metadata.author |
favicon / thumbnail |
metadata.favicon / metadata.og:image |
publish_date |
whichever date key the site itself used |
page_count / content_type |
metadata.numPages / metadata.contentType |
http_status |
metadata.statusCode |
cache_state / request_id |
metadata.cacheState / metadata.scrapeId |
usage |
metadata.creditsUsed |
pdf_mode |
parsed, when the page had a page count |
js_rendered |
always true |
metadata |
every metadata key no row above consumed |
These seven formats bill as one unit: markdown, html, rawHtml, links, images, summary and screenshot. That is why extended can fill most of its row from firecrawl in a single call.
GET r.jina.ai/<url> · document at data, page metadata at data.metadata
| Normalized | Read from |
|---|---|
content |
content, with the reader’s Title: / URL Source: preamble stripped |
content_format |
echoes the format you asked for |
canonical_url / title |
url / title, else the preamble’s |
description |
description |
language / author |
metadata.lang / metadata.author |
publish_date |
the preamble’s Published Time: |
favicon / thumbnail |
favicon (an object, flattened to a string) / metadata.og:image |
links / images / chunks |
links / images / chunks |
http_status |
httpStatus: the upstream status, not the transport’s |
usage |
usage.tokens |
metadata |
every metadata key no row above consumed |
Telem strips the preamble before hashing. So content_sha256 covers the document, without jina’s framing. Watch http_status. jina answers 200 with a body even when the upstream fetch failed. A caller that checks only the transport status would read an error page as content. That case also raises an upstream_error warning.
POST /v1/extract · document at results[0]
| Normalized | Read from |
|---|---|
content |
full_content |
content_format |
always markdown |
canonical_url / title |
url / title |
publish_date |
publish_date |
warning |
warnings[0] (top level) |
request_id |
extract_id (top level) |
usage |
usage: the SKU list (top level) |
Every call asks for full content, at no extra cost. Without it, there is no document. A fetch has no search objective to pass, and parallel says so in a warning on every call. That is why warning is always filled on a parallel document. A URL that parallel cannot extract goes in errors[]. It comes back as a provider_error that names the error type and upstream status.
POST /v1/contents · document is the first element of a top-level array
| Normalized | Read from |
|---|---|
content |
markdown |
content_format |
always markdown |
raw_html |
html |
canonical_url / title |
url / title (a title that is only the URL is dropped) |
favicon |
metadata.favicon_url |
metadata |
every other metadata key |
Every call asks for a fresh crawl (max_age: 0). you.com can answer from its own cache instead. But measurements showed that the cached copy was degraded: a URL for a title and a quarter of the page. A fresh crawl takes some extra seconds, and it gets the whole document.
POST /v1/fetch · the whole body is the document
| Normalized | Read from |
|---|---|
content |
markdown |
content_format |
always markdown |
js_rendered |
always true |
raw_html / content_type |
rawContent / contentType (requested only when the tier includes them) |
canonical_url |
the URL you sent. linkup echoes none |
favicon |
favicon |
images |
images[].url (requested only when the tier includes them) |
linkup returns no title, so title is the one already known for the URL, when there is one. It reports failure only through the HTTP status.
POST /datasets/v3/scrape · one LinkedIn record, the first element of a top-level array
| Normalized | Read from |
|---|---|
content |
the record rendered as markdown: a heading, the key facts, then about, job_summary or post_text |
content_format |
always markdown |
canonical_url |
url |
title |
company name · job job_title · post “user_id on LinkedIn” |
description |
company about (first 300 characters) · job company and location · post text (first 300 characters) |
author / publish_date |
a post’s user_id / date_posted |
structured |
the whole record |
metadata |
record_type (company, job or post), dataset_id, scraped_at, and Bright Data’s own warning when it sent one |
Pin-only, and only for LinkedIn. It takes company and showcase pages (/company/<slug>, /showcase/<slug>), job postings (/jobs/view/<id>) and posts (/posts/<slug>, /feed/update/urn:li:activity:<id>). It declines every other URL before it makes any call. Only a company’s own posts are served. A post by an individual account fails with a provider_error. A dead page and a record too thin to be useful fail in the same way. The record is collected from a fixed field allowlist. So the lists that name other people (employees, commenters, tagged people) never reach Telem.
Providers
Section titled “Providers”Eight providers make up the default chain. Telem tries them one after another. It moves on when a provider fails, or returns a blocked or empty page. The order depends on the URL: for some kinds of page, such as PDFs or Reddit posts, Telem first tries the providers that do best on them, then the rest of the chain.
options.providers pins the chain to a subset, in your order. A pinned chain is
never reordered, and no other provider is tried. A pinned provider can serve a
saved copy of the page, but a page that it fetches is not saved. If no provider in
a pinned chain accepts one of your URLs, you get a 400.
Social, video, review and shopping platforms have their own fetch providers, and
a ninth provider, brightdata, reads LinkedIn company, job and post records. None
of them is in the default chain, but video and post links on many of these
platforms go to their provider first, with no pin. They are listed on
Supported platforms.
The response names your pin in pinned_providers: the options.providers
list, in lower case and without duplicates. It is null when you did not pin a
chain.
| Provider | Notes |
|---|---|
brightdata |
LinkedIn records. Not in the default chain, but LinkedIn company, job and post links go to it first. Takes a LinkedIn company, job or company-post URL and declines every other URL. The record comes back rendered as markdown, and whole in structured. |
exa |
Page text, with links and images at max. Declares no output format, so content_format always comes back text whichever format you asked for. |
firecrawl |
The richest document here: a real browser render, with HTML, links, images, a summary and a screenshot next to the markdown. PDFs are billed by the page. |
jina |
A reader that returns the page as clean markdown. The only one that answers with no key at all. A key only raises its rate limit. |
linkup |
Page markdown, JavaScript-rendered, with the raw HTML and content type from extended up. Echoes no URL and no title. |
parallel |
Full page markdown with a publish date, a request id and a per-call SKU usage list. |
tavily |
One URL in, one document out: page text with the title, favicon and images. Reports no upstream status. A URL either extracts or comes back failed. |
tinyfish |
Page markdown with the title, description, language, author and publish date, plus links and images from extended up. A page it cannot fetch comes back with tinyfish’s own reason code (bot_blocked, timeout, …). |
you |
Always a fresh crawl: page markdown, the raw HTML and the page’s metadata. |
GET /v1/fetch-providers (no auth) is the live list of fetch providers. For each
provider, it shows:
- its name
- whether it is in the chain
- whether it needs a key
- the fields that it can never supply
Page content
Section titled “Page content”Three options shape the body itself:
content_formatpicks markdown or plain text. Those are the only two formats.inline_content: falsereturns every field except the page text. Use this metadata read when you want to know what a page is, without its text in the response.inline_max_charscaps each inline body at 20 000 characters by default. A longer page comes back truncated, with a flag.
A fetch is for whole pages. On search, whole pages are an opt-in. Every fetch pays for a whole page, so fetch the URLs that you already decided to keep.
Batch query
Section titled “Batch query”Pass a list of URLs, and Telem fetches them concurrently in a single
interaction. This is faster and cheaper than calling fetch many times. You get
one result per URL, in the order you sent them. Each result has its own
status. If one URL fails, the call does not fail.
A call takes up to 5 URLs. One more returns a 400 (over_url_cap) that names
the limit.
In the Python SDK, this is client.fetch(["https://a", "https://b"]).
Next steps
Section titled “Next steps”- Search parameters: choose providers, tiers and date windows
- Supported platforms: fetch providers for social, video, review and shopping platforms
- Errors: every error the API returns
- API: Fetch: the generated reference for the fetch endpoint