Sitemap URL Extractor

Web & SEO Tools

Operated by Sourcelane✓ verified

A sitemap crawler that extracts a complete URL inventory from a website by auto-discovering and parsing sitemap files (including gzipped XML), following sitemap index references recursively, and returning per-page metadata such as the page URL, last-modified date, change frequency, priority score, and the originating sitemap file. Technical features include robots.txt sitemap discovery with a /sitemap.xml fallback, transparent decompression of.xml.gz files, breadth-first traversal of nested sitemapindex files with sitemap URL de-duplication to handle circular references, configurable recursion depth and sitemap counts, and per-sitemap request timeouts to manage slow responses.

Verified Sep 28, 3:55 AM

$0.0003

per url

Up to 100 per call. Only pay for results returned; failed calls are refunded.

What people use it for

SEO audits: build a full URL inventory before site-wide checks (broken links, missing meta tags, schema validation)
Content migration: compile a complete list of site URLs for platform migrations
Crawl budget planning: quantify exposed URLs and recent update timestamps for crawl prioritization
Competitor research: map every page published by a competitor site
Sitemap validation: verify that published sitemaps contain expected pages
Bulk URL scraping pipelines: generate URL lists for downstream tasks like screenshots, content extraction, or AI summarization

Trust & reliability

7d uptime trend

Window	Uptime	Success rate	p50	p95
24h	—	—	—	—
7d	—	—	—	—
30d	—	—	—	—

Calling contract

Call it through Sourcelane's gateway or MCP server. We run the connector, apply your agent's spend guardrails, and bill only the results returned.

Call it via Sourcelane

curl -X POST https://api.usesourcelane.com/v1/call \
  -H "Authorization: Bearer sl_live_your_agent_key" \
  -H "Content-Type: application/json" \
  -d '{"listing":"sitemap-extractor","params":{"maxItems":5},"maxResults":10}'

Request params

Field	Type	Required	Description
`maxItems`	integer	optional	Maximum number of URLs to return. 0 = unlimited (walks the entire sitemap tree). Capped at 100 per call.
`followSitemapIndex`	boolean	optional	When the top-level sitemap is a sitemap index (a list of child sitemaps), recurse into each child sitemap. Disable to only extract URLs from the top-level file.

Each result contains

Field	Type	Description
`loc`	string	Loc
`lastmod`	null	Lastmod
`changefreq`	null	Changefreq
`priority`	null	Priority
`sourceSitemap`	string	Source Sitemap

Use Sitemap URL Extractor from Claude, ChatGPT or Cursor

Pick your tool and connect in under a minute. Then just ask — for example: “For these 50 domains, find contact emails and what e-commerce platform they run on.”

Connect Claude

Web, desktop and mobile. Paste one URL.

1Copy your personal connector URL
2In Claude open Settings → Connectors → Add custom connector
3Paste the URL and click Add — done

Manual setup (config files, REST, Python) +

Claude Code

Adds the Sourcelane MCP server with your key as a header.

terminal

claude mcp add --transport http sourcelane https://api.usesourcelane.com/mcp \
  --header "Authorization: Bearer sl_live_your_agent_key"

Claude Desktop (config file)

Alternative to the connector URL: add to claude_desktop_config.json, then restart Claude.

claude_desktop_config.json

{
  "mcpServers": {
    "sourcelane": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://api.usesourcelane.com/mcp",
        "--header",
        "Authorization:${AUTH_HEADER}"
      ],
      "env": {
        "AUTH_HEADER": "Bearer sl_live_your_agent_key"
      }
    }
  }
}

Cursor & Windsurf

Add to ~/.cursor/mcp.json (or Windsurf's mcp_config.json).

mcp.json

{
  "mcpServers": {
    "sourcelane": {
      "url": "https://api.usesourcelane.com/mcp",
      "headers": {
        "Authorization": "Bearer sl_live_your_agent_key"
      }
    }
  }
}

VS Code

Save as .vscode/mcp.json. VS Code prompts for your key once.

.vscode/mcp.json

{
  "servers": {
    "sourcelane": {
      "type": "http",
      "url": "https://api.usesourcelane.com/mcp",
      "headers": {
        "Authorization": "Bearer ${input:sourcelane-key}"
      }
    }
  },
  "inputs": [
    {
      "type": "promptString",
      "id": "sourcelane-key",
      "description": "Sourcelane agent key",
      "password": true
    }
  ]
}

ChatGPT Custom GPT (Actions)

Create a GPT → Actions → Import from URL, then Authentication: API Key, Bearer.

OpenAPI schema URL

https://usesourcelane.com/openapi.json

REST

One POST. Pass maxResults to cap cost.

curl

curl -X POST https://api.usesourcelane.com/v1/call \
  -H "Authorization: Bearer sl_live_your_agent_key" \
  -H "Content-Type: application/json" \
  -d '{"listing":"sitemap-extractor","params":{"maxItems":5},"maxResults":10}'

Python, LangChain, CrewAI, OpenAI Agents SDK…

Wrap the REST call as a tool, or point an MCP client at the endpoint.

python

import requests

res = requests.post(
    "https://api.usesourcelane.com/v1/call",
    headers={"Authorization": "Bearer sl_live_your_agent_key"},
    json={
        "listing": "sitemap-extractor",
        "params": {"maxItems":5},
        "maxResults": 10,
    },
    timeout=300,
)
body = res.json()
print(body["receipt"]["chargedMicros"], "micro-USD for", body["receipt"]["results"], "results")
print(body["data"])

Reviews

—

0 reviews

Leave a review

Loading reviews…