Test an AI staff assistant before it answers Banner support questions

Choose one staff task, such as finding the approved procedure for investigating a failed file transfer. Give the assistant a defined collection of current runbooks and an agreed list of questions. Start with drafting guidance for an analyst to review; keep the assistant from changing records or executing production steps.

Waterloo's published 2025 accomplishments describe a service-desk support prototype using existing knowledge-base documentation, with further work planned for 2026. For your trial, have a support specialist check the approved answer to each selected question.

Have the Banner service owner and knowledge owner identify the documents permitted in this trial. UBC's assistant service similarly makes technical requirements specific to each project. The trial below uses a proposed institutional or external assistant. Confirm any claimed native feature separately with the vendor.

Ask experienced service-desk staff for approved, redacted examples of recurring questions. Remove student, employee and confidential incident details. For each question, have the responsible support specialist record the expected answer, required qualifications, supporting document and section, and when the question must go to another team.

Label each runbook with its owner, version, review date and applicable environment. A procedure for Banner on-premises may be unsuitable for a contracted SaaS service. For the Banner modernization path, Ellucian uses Ellucian Student powered by Banner. Confirm the institution's actual product, deployment and responsibilities before including instructions.

Include questions where the correct response is to ask for a campus, environment or module. Add cases with no approved answer and with conflicting instructions. Keep a set of questions separate from prompt-tuning work so the team can check whether a correction helps beyond the examples used to make it.

A citation passes only when it supports the answer given. Have the reviewer open the cited section and compare the steps, conditions and escalation contact. Check that the assistant has not combined instructions from different environments or added a step the runbook does not contain.

Have IT test access in both retrieval and the application. A message telling the model to protect restricted material does not replace authorization. OWASP describes how instructions in questions or retrieved files can redirect a model, and recommends limiting privileges, separating untrusted content and testing those boundaries. Use synthetic documents and authorized test accounts for these checks.

Use the table as a starting test set. Keep the question, account role, approved sources, actual answer, failed check and reviewer together. Repeat important cases with different wording and fresh sessions rather than relying on one successful response.

Agree which team receives each unanswered question. A referral should name the approved support queue, explain what remains unresolved and identify the source or test case the analyst checked. Keep the detail within that recipient's access; do not copy restricted content into a wider ticket queue.

In a hypothetical registration-hold question, an assistant may find the approved contact and explain how staff request a review. It should not decide student eligibility or remove the hold. The Registrar or designated functional owner makes that decision through the institution's normal procedure.

Have the service-desk lead walk a referred test case through the proposed handoff. Confirm that someone receives it, understands the question and can respond. During the first limited trial, staff review answers before using them to guide operational work.

Record the model or service version, instructions, document versions and access setup with each test run. Classify failures so the right person can act: wrong source, outdated content, unsupported claim, access problem or failed handoff. Correct the cause and repeat the affected cases.

Have the service owner set release conditions before seeing the results. Do not average an unauthorized disclosure or an unapproved record change into a general answer score. Keep those failures open until IT and the responsible owner have corrected and retested them.

For a limited trial, name the participating staff, supported questions, reviewer and person who can suspend use. Track corrections, referrals and the time staff spend checking answers. Repeat the test set when the documents, model, instructions or permissions change.

Bring the next review a small set of questions, approved answers and failed tests. Altioras Global's AI Subscription Services can help scope the source review, assistant testing and staff handoff for that defined task.