How It Works

Architecture overview

1. PR opened on GitHub

   You open a pull request in any repo with Agentura installed.

2. GitHub sends a webhook

   GitHub notifies Agentura's webhook endpoint at agentura-ci.vercel.app/api/webhooks/github.

3. Job queued

   Agentura saves the eval run and queues a job in Redis.

4. Worker picks up the job

   A background worker fetches agentura.yaml and datasets from the PR branch.

5. Agent is called

   The worker calls your HTTP endpoint once per test case.

6. Scores computed

   Each response is scored using fuzzy/exact, llm_judge, or performance strategy.

7. Baseline compared

   Suite scores are compared against the last completed run on the baseline branch, usually main.

8. Results posted

   A GitHub Check Run and PR comment are posted with pass/fail and deltas.

Request/response format

Your agent must accept POST requests with this body:

{ "input": "user message or query" }

And return:

{ "output": "agent response" }

That's the full integration contract. No SDK. No library. No extra plumbing.