Indexing
Indexing is how MCP Studio turns your sources into searchable, citable context. It runs in the background — not inside a single browser request — so closing the dashboard, a timeout, or a deploy mid-run cannot wipe a working index.
| Audience | MCP operators, content owners, and developers troubleshooting source quality |
|---|---|
| Prerequisites | At least one source connected to an MCP server |
What happens during indexing
- MCP Studio queues the source and starts working on it immediately.
- It discovers the pages or files the source contains.
- It fetches each one, retrying transient failures, and skips anything that has not changed since the last run.
- It extracts the meaningful content, splits it into passages that preserve headings and code blocks intact, and prepares them for search.
- New content is written alongside the existing index. Stale pages are only removed once a run finishes successfully, so a failed run leaves your last good index fully searchable.
Large sources are processed incrementally and continue in the background until complete. You do not need to keep the dashboard open.
Indexing status
Each source shows one of these statuses:
| Status | Meaning |
|---|---|
| Pending | Source has been added; work is queued |
| Indexing (dashboard may show Crawling) | Content is being discovered, fetched, and indexed. A percent and page counts appear on the dashboard. |
| Complete | All content has been indexed and is searchable |
| Error | Something went wrong — see the inline error reason below the source |
While a source is indexing you can still click Refresh / Restart indexing. That cancels the current run and starts a fresh one. If a run stops reporting progress, the dashboard marks it stalled and it is automatically retried or failed rather than being left stuck.
Error reasons
When a source ends up in the Error state, the dashboard surfaces a machine-readable reason:
| Reason | Cause | Action |
|---|---|---|
auth_required | Private GitHub repo, but we don't have a valid token for your account | Click Link GitHub next to the source |
not_found | Repo wasn't found with the linked GitHub account (renamed, deleted, or revoked access) | Verify the URL / re-grant org access |
rate_limited | GitHub is temporarily throttling requests for your account | Usually clears on its own; click Refresh to retry |
network | Fetch timed out or the site was unreachable | Click Refresh |
no_content | Pages returned almost no extractable text (often a client-rendered or bot-protected site) | Point at a more specific docs URL, or use the GitHub docs repo |
stalled | The job stopped reporting progress and was reset | Click Restart indexing |
storage_limit | The index is full, so this source could not be written | Remove an unused source or server, then refresh |
storage_budget | The source is larger than your plan allows | Upgrade, or remove another source from this server |
unknown | Generic indexer failure | Click Refresh |
Typical indexing times
- 1–3 sources: Usually 1–6 minutes
- Large docs sites: Can take longer and will continue in the background
- GitHub repos: Often faster for markdown-first repos, longer for broad code repositories
- Unchanged sources on daily refresh: Much faster, because unchanged content is not reprocessed
Using your server during indexing
Your MCP server is usable immediately after deployment, even while sources are still indexing. During this period it falls back to fetching content live when the index does not yet have a match — queries still work, they are just slightly slower.
Once indexing completes, queries are served from the prepared index and are considerably faster.
Refreshing sources
You can refresh sources in two ways:
- Manual refresh: Click Refresh (or Restart indexing) next to any source on the dashboard. This is available even while a source is already indexing.
- Auto-refresh: Sources that have not been indexed recently are re-queued daily at 7:00 AM Central US time. Content that has not changed is detected and skipped.
Closing the browser does not stop a refresh — it continues server-side.
Page limits
- Website and GitHub sources are limited to 5,000 pages (or files) per source
- Very large files and non-text assets are skipped
- If a source is larger than the cap, use a more specific docs URL or the underlying GitHub docs repo
Citations
Answers cite the page they came from, so any claim can be traced back to your documentation. Citations retain section headings where the source provides them, and MCP Studio favours your current documentation when a source publishes several versions side by side.
Deep-link behavior by source type
- Web/docs: Citations resolve to section anchors when available.
- GitHub: Citations can resolve to file line ranges for code-level provenance.
- PDF: Citations include page fragments (
#page=N) when page-level context is available. - Federated MCP sources: Citations include MCP endpoint references and remote tool usage context. These sources are queried at runtime, not crawled.
GitHub access
For GitHub sources, MCP Studio uses the server owner's linked GitHub account to read the repository. Before creating the source it verifies that the repository is readable with that account, so a private repo you cannot access fails immediately with an actionable Link GitHub button rather than silently producing an empty source.
The OAuth scope requested is read:user user:email repo. The repo scope grants read access to private repositories owned by, or shared with, the user. Tokens are stored server-side and are never exposed to the browser.