MeshArc

mesharc

The Python client for the MeshArc API: a URL in, clean content out, and a record of what changed.

• Scrape one page or a batch — markdown, text, HTML, links, structured fields, a screenshot.
• Crawl a whole site with no project to set up first, and keep it as one if it turns out to be worth watching.
• Map what a site declares in its sitemaps before fetching any of it.
• Search the web for pages whose URLs you do not know, and read each result's page in the same call.
• Ask the agent a question: it searches, reads pages and answers, in the shape of a schema if you give one, with the pages its answer rests on. (Not live on mesharc.dev yet.)
• Monitor a search or a question: asked again hourly, daily or weekly, each answer compared with the last, a webhook told when it changed. (Not live on mesharc.dev yet.)
• Watch a site over time: projects, scheduled runs, and a change record — pages added, removed, modified, field by field.

Python 3.10 or newer. One dependency (httpx). Fully typed.

Install

pip install mesharc

Authentication

Every call needs an API key. Create one in the app under Settings → API keys — it is shown once — and give it to the client, or put it in MESHARC_API_KEY and construct the client with nothing:

from mesharc import MeshArc

arc = MeshArc("mesharc_...")

or, with MESHARC_API_KEY in the environment:

arc = MeshArc()

options: MeshArc(api_key, timeout=150.0, max_retries=2)

The key is only ever sent as a bearer header to api.mesharc.dev.

A key carries the scopes it was made with (read, write, admin), optionally a set of projects it may see, an expiry and a rate limit. A route the key may not use answers 403; a project it may not see answers 404.

Quick start

from mesharc import MeshArc

arc = MeshArc("mesharc_...")

page = arc.scrape("https://example.com/pricing") print(page["markdown"]) print(page["verdict"], page["method"], page["credits"]) # ok crawler 1

scrape holds the request open until the page comes back (60 s by default), so there is nothing to poll for an ordinary page.

Reading pages

One page

page = arc.scrape("https://quotes.toscrape.com/js/", config={"render_js": "always"})

config is any setting a project takes, by its API name (render_js, only_main_content, formats, max_tier, wait_for_selector, actions, …). The full list, with defaults, is at mesharc.dev/docs/configuration.

page = arc.scrape_one(

"https://example.com/", formats="markdown,text,cleanHtml", # which bodies to return timeout_s=120, # how long the API holds the request (120 max) idempotency_key="pricing-2026-09-18", # the same key returns the first answer for 24 h )

Many pages

A list of URLs is a batch: grouped by host, fetched in parallel where the config allows, and returned as one row per URL.

batch = arc.scrape(["https://a.com/", "https://b.com/x"], config={"concurrency": 4})

for row in batch["pages"]: print(row["url"], row["httpStatus"], row["verdict"], row["credits"])

arc.scrape(urls, wait=False) returns the batch id at once; arc.batch(id, wait=True) finishes it later. Pass webhook_url= to be told instead of polling (batch.finished, signed with a secret returned once).

What a page looks like

Every page row carries the same fields, whether it came from a scrape, a crawl or a project:

A page the site refused costs 0, and so does a 404.

Crawling a site

job = arc.crawl(

"https://docs.example.com", limit=200, # page budget maxDepth=3, # link hops from the seed includePaths=["/docs/*"], crawlMode="sitemap_first", # what the sitemap declares first, then links maxTier="browser", # how far a refused page may climb scrapeOptions={"formats": ["markdown", "links"]}, config={"crawl_delay_ms": 500}, # any project setting, directly )

for page in job.pages(): # follows the cursor while the crawl runs print(page["url"], page["words"]) print(job.status, job.envelope["counts"], job.envelope["creditsUsed"])

crawl returns a handle immediately; job.pages() yields pages as they land and ends when the crawl does. wait=True blocks until it finishes; job.wait(), job.refresh(), job.cancel() do what they say; arc.get_crawl(id) reattaches to a crawl started elsewhere.

A one-shot crawl expires after 30 days. If the site is worth watching:

project = job.keep(name="Docs", schedule="weekly")

A webhook= in the options ({"url", "events", "metadata", "include"}) is told about crawl.started, crawl.page (fifty pages a message) and crawl.completed; its signing secret comes back once as job.webhook_secret. With "include": ["markdown"] the crawl.page messages carry each page's markdown too, up to about 1 MB a message: a page that does not fit, or was not stored, comes as "markdown": null with markdownOmitted (too_large or not_stored) and a contentUrl, an API path read with your key (job.page(url) reads the same). It needs crawl.page among the events, as it is when events is left out.

Mapping a site

for u in arc.map("https://docs.example.com"):

print(u["url"], u["lastmod"])

details = arc.map_details("https://www.gov.uk/", search="visa", limit=500) print(details["totals"], details["creditsUsed"]) # {'files': 29, 'urls': 508431, …} 29

A map costs one credit per sitemap file read — most sites are one file.

Searching the web

out = arc.web_search(

"python packaging guide", limit=5, # 1 to 10 results freshness="month", # hour | day | week | month | year include_domains=["python.org"], # exclude_domains= too; an exclude wins scrape=True, # each result's page as markdown as well )

if out["status"] == "blocked": # every engine refused; nothing was charged print(out["attempts"]) else: for hit in out["data"]: print(hit["position"], hit["title"], hit["url"], hit["snippet"]) if hit.get("page"): print(hit["page"].get("markdown", "")[:200]) print(out["engine"], out["cached"], out["creditsUsed"])

again = arc.get_search(out["id"]) # a search started earlier, by id for s in arc.searches(q="python"): # the workspace's searches, newest first print(s["id"], s["query"], s["status"], s["resultCount"], s["creditsUsed"])

news = arc.web_search("python release", news=True, freshness="day") # the engines' news results for hit in news["data"]: print(hit["publisher"], hit["age"], hit["title"]) # who published it, and how long ago ('20h') more = arc.web_search("python packaging guide", page=2) # results 11-20

web_search waits for the results and returns the whole envelope, the hits under data. A search is queued, running, done, blocked or error: blocked — every engine refused — is returned, not raised; error raises MeshArcError. A search that scrapes stays running until its pages land. scrape= also takes a dict, sent as given ({"formats": ["markdown", "links"], "maxCredits": 20}); country= and lang= say where and in what language to search; wait=False returns at once, and timeout_s is how long the API holds the request (60 s by default, 120 at most) before the client polls.

news=True searches the engines' news results rather than the web's (one or the other per search); each news hit also carries publisher and age, as the engine put it ("20h"). page= is which results page, 1 to 10: page 2 is results 11–20, and each page is its own search with its own charge. `news=` and `page=` need an API with those features: they are on the API's feature branch, not yet on mesharc.dev.

A search needs a key that can write, and it spends credits. The results page costs the engine that read it; a results page every engine refused is free. An equal search — same query, news, page, country, lang, freshness and domains — within an hour of a finished one (ten minutes for news) comes from the cache (cached is true) with no charge for the results page. Scraped pages are charged, cached or not.

arc.search(project_id, q) is something else: it searches the pages a project has stored, not the web (see below).

Asking the agent

The agent route (`/agent`) is not live on mesharc.dev yet; web search is. The client and the MCP tools are ready for it. The agent's webhook, connection_id, continue_() and field_sources need an API with those features too: they are on the API's feature branch, not yet on mesharc.dev.

The agent takes a question, searches the web, maps sites and reads pages on its own, and answers with the pages it used. Give it a schema and the answer comes back in that shape:

run = arc.agent(

"What does each plan on example.com cost per month?", urls=["https://example.com/pricing"], # pages to start from, up to 20; omit to let it search allowed_domains=["example.com"], # keep it to these sites, up to 20 schema={ # a JSON Schema of type object or array "type": "object", "properties": {"plans": {"type": "array", "items": {"type": "object", "properties": { "name": {"type": "string"}, "monthly": {"type": "number"}}}}}, }, max_credits=500, # the most the run may spend: 1 to 100,000, 2,000 by default max_steps=20, # the most steps it may take: 1 to 100, 40 by default )

out = run.wait() # blocks until it finishes; the envelope print(out["status"], out["creditsUsed"], out["steps"]) print(run.data) # {'plans': [...]}, matching the schema for src in run.sources: # the pages the answer rests on print(src["url"], src["title"]) print(run.field_sources) # {'plans[0].monthly': {'url': ..., 'pageId': ...}, ...}

agent returns an AgentRun at once, queued or running; the prompt is up to 10,000 characters. timeout_s= asks the API to hold the request for the answer (120 s at most). Without a schema the answer is {"text": ...}. field_sources says where each value of the answer came from: its path ("plans[0].monthly", or "[2].name" for a list answer) to {url, pageId}, a page the run read.

run = arc.agent(

"What does each plan on example.com cost per month?", webhook={"url": "https://hooks.example.com/mesharc", "events": ["agent.completed", "agent.failed"]}, connection_id="conn_...", # run on one of the workspace's own LLM connections ) print(run.webhook_secret) # once: the secre