Appa Tools documentation for MCP Studio, including setup, guides, concepts, and API-related reference content.

Skip to main content

Turn docs into self-evaluating skills

An MCP server answers your assistant's questions. A skill tells it how to do a whole task. docs-to-self-evaluating-skills is an open-source project that writes skills from the sources you index in MCP Studio, grades every skill against those same sources, and keeps the revisions that score better.

AudienceDevelopers and builders who want their assistant to set up an integration correctly
PrerequisitesAn MCP Studio server with an indexed source, an API key, an OpenAI API key, and Node 20 or later

What you get​

Index the Amazon S3 docs and you get skills such as s3-presigned-uploads: a SKILL.md your assistant follows inside your codebase, written from the exact passages in the docs, with a ## Sources list linking each page it came from.

Each skill comes with evals written from the documentation, so it is graded against the docs rather than against itself. Every day the project reports each skill's score with the skill and without it, what changed since the previous run, and which checks a revision fixed.

How it reads your sources​

The project reads your content through List recently indexed chunks, which returns every passage a source holds, newest first. Each passage carries a contentHash, so the project can tell which pages changed after a refresh and rebuild only the skills that depend on them.

curl "https://appatools.com/mcp-studio/api/mcp/<server>/sources/<source-id>/chunks?limit=100" \
  -H "Authorization: Bearer $MCP_STUDIO_API_KEY"

Set it up​

  1. Create a server in MCP Studio and add the documentation you want skills for. Wait for the source to finish indexing.
  2. Find the source ID with Get a server in the Servers reference: it is sources[].id.
  3. Clone the repository, put the server slug and source ID in skills.config.json, and add your keys to .env.
  4. Run npm run daily. The first run plans the skills, writes them, and grades them.
  5. To run it every day, push the repository to GitHub and add MCP_STUDIO_API_KEY and OPENAI_API_KEY as Actions secrets. The included workflow runs after MCP Studio's daily source refresh and commits the skills and the report back.

Copy the skill folders you want into your project's skills directory, such as .cursor/skills/ or .claude/skills/.

The repository's README covers configuration, how revisions are accepted, and what the score does and does not measure.