# Knowledge assistant evaluation kit: Aster team (EN)

This kit is a fictional fixture. The Aster team, Camille and every rule in the documents are invented to test a knowledge assistant. The kit contains no real personal data and describes no real company procedure.

Version française : [../fr/README.md](../fr/README.md)

## Contents

| File | Role | Upload to the assistant? |
|---|---|---|
| `corpus/AST-01-launch-policy-v2.md` | Current launch policy, v2, 2026-11-01 | Yes |
| `corpus/AST-02-launch-policy-v1-archived.md` | Former policy, v1, 2026-09-01, archived | Yes |
| `corpus/AST-03-faq.md` | FAQ: discount, missing information | Yes |
| `answer-key-aster-en.csv` | Expected answers, source document, failure criterion | No, never |
| `scoring-sheet-aster-en.csv` | Blank sheet to fill in by hand | No |

## Steps

1. Open a new workspace or conversation with no other documents loaded.
2. Upload only the three files in the `corpus/` folder.
3. Keep the answer key out of the assistant. Do not attach `answer-key-aster-en.csv` or the scoring sheet: the assistant would read the expected answers and the test would measure nothing.
4. Paste this exact prompt as your first message:

```text
You answer questions for the Aster demo team, a fictional team. Use only the attached documents. For every answer, give the document ID, its version and its date, and quote word for word the sentence that supports the answer. If two documents conflict, apply the most recent current document and say which one is archived. If the documents do not contain the answer, reply "Unknown in the documents, ask the document owner" and do not guess. Never invent rules, exceptions, discounts or figures that are not in the documents.
```

5. Ask the ten questions from the scoring sheet in order, one per message, and copy each answer unedited into the `answer` column.
6. Fill in the sheet by hand. Open the answer key only afterwards, to score `factual_correct` and `abstains_when_required`.

## Filling in the sheet

Score each column separately. An answer can be correct with a dead link, or cite a document that does not contain the sentence: these outcomes do not cancel each other out.

- `answer`: the answer, copied without edits.
- `source_url`: the link or file name the assistant cited; leave empty if it cited nothing.
- `source_version`: the version and date cited, for example `v2, 2026-11-01`.
- `factual_correct`: `yes` or `no`, compared with the answer key.
- `source_opens`: `yes` or `no`, depending on whether the cited source opens in the tool.
- `source_supports`: `yes` or `no`, depending on whether the cited passage actually contains the claim.
- `abstains_when_required`: `yes` or `no` for Q04, Q05 and Q09 (the assistant says the information is missing or declines to invent); `na` for the other questions.
- `review_seconds`: the time in seconds spent checking the answer.
- `notes`: any behaviour worth keeping, such as mixing up v1 and v2.

## What the kit does not test

- Access permissions. Test them separately, in a private workspace you control: create a dummy restricted document, sign in with an account that has no access to it, and check that no answer reveals it.
- Never upload private or confidential files to a public demo, not even to extend this kit.
- The kit provides no reference score and makes no claim about any product. Evaluation is manual: your scores apply only to the tool, plan and date of your own trial.

## File format

Markdown and CSV in UTF-8, comma-separated.
