Checking lesson-based AI answers, claim by claim.

THE WORK

The project.

LMS AI Core checks an AI answer against the lesson material before Laravel decides what to show. It reranks the lesson passages supplied by Laravel, then checks individual claims rather than assuming a relevant passage supports everything the model wrote. The response identifies supported chunks, conflicting details and weakly supported content, so Laravel can prune an answer or replace it with review suggestions.

The service uses weighted text heuristics, not a locally hosted language model. Laravel remains the application around it: learner access, source versions, provider generation, citations and saved-guide readiness all stay there. Live, shadow and local fallback routes let the two verification implementations be used and compared without making a service failure mean unchecked output.

ATTEMPTS & CORRECTIONS

What changed along the way.

1. 01Getting the permanent queue worker to start10 June 2026

   At first

   Deployment had two worker commands: a long-running queue:work process for normal and AI queues, and a separate drain for the initial backfill. The permanent command took queue names and numeric settings directly from the environment.

   What needed to change

   The staging record showed the permanent worker repeatedly failing on the unsupported stop-when-empty-for option. The reviewed source no longer contained it, so the repair also needed a regression against its return. A deployment script finishing wasn't proof that the worker had started.

   The change

   The permanent command now uses validated queue names, checked numeric values and supported options. Tests cover both deployment files, the normal queue and named AI queues. They also preserve the separate one-off drain, where stop-when-empty is valid.

   The result

   The closeout record reports the worker-configuration regression passing with 79 assertions. That is a recorded check of the command configuration, not a fresh demonstration of a running worker or a drained queue.

2. 02Showing the ready guide and making the admin filter agree11 June 2026

   At first

   The learner's study-guide status followed an older generation artifact. A newer ready version could already be saved, but it didn't take priority in the readiness calculation.

   What needed to change

   The staging check found the assistant and queue working while the guide still appeared unavailable. The lesson already had a current ready version. After the learner response was fixed, the admin filter still read the older artifact and could disagree with the displayed status.

   The change

   The state service now selects a ready version matching the current provider, model, generation contract and source checksum before falling back to artifact state. The admin filter was then changed to use the same calculated result. Tests cover a valid current version and a stale version with a different checksum.

   The result

   The project record shows the existing guide and quick check appearing on the learner page after the fix. It reports eight assertions passing in the focused readiness test, followed by 87 in the admin operations group. Lessons without enough usable material remained unavailable.

3. 03Using repaired transcripts without accepting empty or stale sources12 June 2026

   At first

   For a remotely hosted lesson, rebuilding the source queued another transcript import before checking for a usable manual transcript, managed captions file or canonical transcript.

   What needed to change

   The remote-import path could skip an administrator's repair and return without projecting the supplied text into segments and chunks. Allowing existing ready transcripts through exposed two further cases: a ready source with no text, and text still attached to an old video or uploaded file.

   The change

   Rebuilds now check usable existing transcript material first and reject ready sources that are empty. Follow-up PR fixes compare imported transcripts with the current video, file checksum or caption path. Guide assembly and assistant retrieval use the shared check to exclude stale material.

   The result

   The June record shows failing regressions before each repair. Afterward, manual transcript reuse passed six assertions, empty-source repair passed three, and four stale-media regressions passed across acquisition, guide assembly and assistant retrieval. The pipeline still doesn't create missing transcripts; lessons without valid material need an authorized repair.

4. 04Fixing service authentication and the extra copies of learner text1 July 2026

   At first

   AI Core checked a shared token but still accepted versioned requests when no token was configured. Laravel redacted query logs, while saved conversations and evaluation exports followed separate storage paths.

   What needed to change

   New regressions reproduced unauthenticated success, exposed default API docs, raw sensitive text in learner messages and fixture or source text in portable reports. Redacting the query log hadn't protected those other outputs.

   The change

   Versioned routes now return unavailable when authentication isn't configured, and default docs and schema routes are disabled. Saved messages use redaction with a longer history allowance than query telemetry. Portable reports replace selected full responses, payloads and errors with summaries. A follow-up regression checks that message redaction doesn't apply the shorter query-log cap.

   The result

   For this staging-only change, the July record reports 12 Python tests passing, 54 focused PHP tests with 360 assertions, and 13 assertions in the message-retention regression after the review fix. One Python dependency warning remained. That evidence step performed no staging or production deployment.

5. 05Keeping an unchanged source rebuild from making a guide stale

   At first

   The Study Pack fingerprint included local row IDs alongside the lesson material. Reprojecting unchanged text could create new source or chunk rows even though the guide's input hadn't changed.

   What needed to change

   The new row IDs changed the fingerprint, making an otherwise current guide appear stale. The freshness check was treating a storage-identity change as a content change.

   The change

   The checksum now uses source type and key, content hash, chunk position, timestamps and a hash of the selected text. Chunk and source-version IDs remain in the returned references, but no longer determine that freshness fingerprint.

   The result

   The regression recreates source, version and chunk rows with the same material and expects the checksum to stay identical. The June local review identifies the false-stale case as fixed. Content, source-key or timing changes can still change the fingerprint; replacing rows alone no longer does.

6. 06Keeping a supported guide when its practice questions fail

   At first

   Laravel's Study Pack readiness required quick-check questions alongside the guide. A supported takeaway and useful lesson points could be withheld because the generated quiz failed verification.

   What needed to change

   The guide and its questions were treated as one all-or-nothing result. Rejecting an unsupported answer or explanation also hid material that had passed its own source checks.

   The change

   Quick checks became optional for readiness through Laravel configuration. The verifier still checks the takeaway and required lesson points, and removes questions whose correct answer or explanation lacks support. Normalization accepts empty question arrays instead of inventing replacement practice content.

   The result

   The regression supplies supported guide text with an unsupported multiple-choice question and expects a supported guide with both question arrays empty. The June browser review records the guide rendering with a quick-check-unavailable message. This correction is in Laravel's verifier and normalizer; the Python implementation retains its separate minimum-quiz rule.

IMPLEMENTATION

Design choices.

1. Retrieval relevance and answer support are checked separately. A passage can rank well for the question while contradicting a particular number or claim in the answer. The reranker selects useful context; the verifier checks generated units and returns the supporting chunks. Its heuristic scores aren't semantic proof.
2. Python works on supplied material and returns explicit outcomes and source references. Laravel retains storage, enrollment, provider calls and citation policy. That makes the service independently packageable and testable without giving it the power to authorize a learner or decide the complete response.
3. Local verification remains available alongside Python live and shadow routes. A service failure therefore has another verification path, and shadow mode can compare outputs without replacing the local answer. The cost is maintaining both implementations; shadow comparisons also wait synchronously for Python.
4. Source versions, generation attempts and saved study guides are separate records. Content and generation fingerprints decide freshness, provider and model checks decide compatibility, and media checks reject transcripts attached to replaced material. A finished job alone cannot make an old guide current.
5. Recommendation targets and their order are selected in Laravel before a provider writes the presentation text. Returned copy is applied by the original candidate index, so the model can change the wording without rebuilding navigation targets or the ranking. This is application ranking with generated copy, not a learned recommendation policy.
6. Operational logs keep route and comparison metadata instead of full request bodies, and selected evaluation exports retain summaries. That leaves useful diagnostics without storing another complete copy of the prompt. It also means those records can't reconstruct the entire request, and minimization is not universal anonymization.

HOW IT FITS TOGETHER

Architecture.

• Laravel authorizes the learner, retrieves permitted lesson chunks and calls the generation provider. It stores source versions, segments and chunks with content hashes, combining lexical retrieval with configured vector or fallback paths. Python receives selected material in the request rather than opening the LMS database.
• The independent Python package exposes versioned FastAPI routes with Pydantic request and response models. Shared bearer authentication protects the routes, and missing token configuration refuses the request. Reranking takes a query, candidate passages and scores; verification takes structured generated content, supporting material and settings.
• The reranker blends incoming retrieval scores with phrase, keyword, title, source and intent signals. The verifier then splits generated content into claims and checks each one for text support and conflicts in details such as numbers, dates, names, quotes, order and negation. It returns outcomes and supporting chunk IDs for Laravel's citation and response checks.
• Laravel's gateway decides which result to use. Live mode can return Python's verification; shadow mode records a comparison but returns the local result; typed timeouts and service errors select local verification. Route telemetry and trace IDs distinguish a successful Python call from a response produced by fallback.

ACCESS & INPUT

Security controls.

• Versioned API routes require the configured bearer token. With no token configured, they return unavailable instead of accepting anonymous requests. Health stays public, while the default documentation and OpenAPI routes are disabled.
• Pydantic rejects unexpected top-level fields and limits the main strings and collections. Candidate extras, metadata and some list members remain flexible, so those models don't provide a complete byte-size or resource-exhaustion limit.
• Laravel checks learner access before sending retrieved material to Python. Ingestion also applies managed-path, file-type, document-size and external-processing rules. A valid service credential doesn't grant access to a learner's course.
• A timeout or typed service error selects local verification rather than bypassing the checks. Laravel still interprets the support outcome and validates citations; the presence of text in a Python response isn't enough to display it unchanged.
• Query telemetry and saved conversations redact sensitive text with separate length limits. Optional HTTP logs retain the route, trace, timing, status, payload keys and candidate counts instead of request bodies.
• Portable evaluation reports replace selected full responses, payloads and raw errors with summaries or presence flags. This reduces extra copies of learner and source text without claiming that every field or provider trace is anonymous.

SELECTED DEVELOPMENT RECORD

Engineering changes.

Implementation dates come from project records. “Recorded” is the date this portfolio entry was written.

1. Protecting service requests and retained learner textfixedRecorded 11 September 2026

   Missing authentication configuration now blocks versioned requests. Default docs are disabled, saved messages are redacted and evaluation exports retain less text. The recorded checks were local, not a deployment.

   Source observedChanged 1 July 2026

2. Reusing transcript repairs and excluding replaced mediafixedRecorded 11 September 2026

   Rebuilds check for usable existing text before importing again. Guide assembly and assistant retrieval also reject transcript chunks tied to replaced media.

   Source observedChanged 12 June 2026

3. Letting a current ready guide take precedencefixedRecorded 11 September 2026

   A compatible saved version now supplies learner readiness ahead of an older failed artifact. The admin filter uses that same calculated status.

   Source observedChanged 11 June 2026

4. Getting the permanent worker command into a valid formfixedRecorded 11 September 2026

   Removed the unsupported option, checked the command inputs and kept the long-running worker separate from the one-off backfill drain.

   Source observedChanged 10 June 2026

CHECKS & RESULTS

What was checked.

A source review shows what the code does. It is not a fresh test of the running app.

1. Code and development records reviewedReviewed11 September 2026

   The code, tests and available development records were checked. Reported test results keep their original scope. No fresh app or live deployment check was run for this write-up.

   Scope of this check

   Code review, not a fresh run of the app