An AI intake pilot can pass while underwriters still spend time correcting its daily work. Review a defined sample of submissions each week, including outputs nobody changed, and trace disagreements back to the information and rules available when the assistant ran. That tells you whether to repair an input, clarify a rule or restrict an assistant behavior.
We helped an insurance underwriting group add $1.96M in net recurring profit through a broader intake and carrier-reporting engagement. We connected both tasks through one submission record. Their pilot passed its preparation-time and agreement requirements, and we monitored usage and accuracy weekly as teams began using the workflow. The finance result belongs to that wider engagement.
The method below turns that monitoring principle into a weekly disagreement log for a managing general agent, or MGA. You'll finish with a specific correction, an accountable owner and a test for restoring the affected behavior. The worked submission and log are teaching examples.
Start with the work the assistant actually saw
A correction tells you that an underwriter changed something. It doesn't yet tell you why. If the underwriter had a later attachment that the assistant never received, the two were working from different evidence. Changing the model before checking that difference could leave the missing attachment problem untouched.
For example, let's say an assistant flags submission S-104 as a poor fit for a program. The underwriter changes the route to further review because a supplemental attachment describes an activity the program can consider. Before calling that a model error, save the original email, attachment list, extracted fields, program-rule version and assistant proposal. Also record the model, prompt and workflow versions so your technology owner can reproduce the run.
The underwriter adds the reference route, its reason and the source supporting it. Preserve when each source became available. That lets the reviewer distinguish a wrong answer to the original evidence from a reasonable answer that needs updating after new information arrives.
In the source engagement's pilot, underwriters marked reference decisions on past submissions. The held-out test in one program group achieved at least 33% less non-underwriting time per submission with no drop in agreement against those references. Weekly monitoring continued after the pilot. The same principle carries into live review: the comparison needs a defined reference and a known set of inputs.
Include work nobody corrected
If your sample contains only overrides, it can show why corrections happen but can't show how often reviewed outputs need correction. It also misses mistakes that passed through unchanged. Choose a program, task and week, record all eligible outputs, and draw a sample before looking at which ones were edited. Have an underwriter check both changed and unchanged outputs against the applicable evidence and rules.
Keep a separate list of reported serious errors so the team can investigate them immediately. Don't wait for a weekly meeting when a known failure could misroute a submission. Don't mix that deliberately selected list into the regular sample's rate, because its selection makes errors more likely by design.
For an illustrative weekly sample of 20 outputs, suppose the underwriter identifies four disagreements. Record four disagreements out of 20 reviewed outputs, or 20%. If only three are resolved and one lacks enough evidence, retain all four in the disagreement count and show the unresolved one separately. A disagreement rate describes this reviewed sample. It isn't automatically a model-error rate or a loss measure.
That distinction helps you direct repair work toward a known cause. For the wider growth decision, the same discipline means investigating where effort is holding the business back before committing more resources.
If you’re an MGA operations or underwriting leader and want to grow your firm without growth turning into more intake and reporting work, we’ll identify the biggest bottleneck standing in your way, build the plan to remove it, and implement it in eight weeks* or less - guaranteed or we continue to work for free and you don’t pay.
Find my bottleneck* Eight-week delivery, or twelve weeks when substantial data foundations are required, as agreed before work begins. If a covered stage fails its agreed benchmarks, we provide up to four additional weeks of remedy work at no charge, then waive or refund that stage’s fee if it still fails. Successfully delivered stages remain payable. Agreed client-caused delays pause the clock. Delivery guarantee terms.
Find the missing attachment before changing the model
Return to S-104. The original email lists a supplement, but the assistant's captured attachment list doesn't contain it. The underwriter's review does. Your first finding is an input gap. The operations owner checks whether the file arrived after the assistant ran or whether the intake connection failed to capture a file that was already there.
Those findings require different repairs. A later arrival needs a clear update route so the submission gets reviewed again when new evidence arrives. A capture failure needs the technology owner to repair attachment ingestion and confirm that the file reaches the assistant. In either case, preserve the original run. Replacing its inputs would erase the evidence explaining the disagreement.
The poor-fit proposal still needs a separate check. If the approved rule says missing information requires a request or manual review, a poor-fit flag based on absence also breaks that rule. One submission can expose both an input failure and an incorrect response to missing information. Record both so repairing the attachment feed doesn't hide a remaining routing error.
Underwriters keep the decision. The assistant can structure information, flag gaps and prepare a route for review, but it doesn't gain authority to decline or bind a risk because the weekly sample looks better. In our source engagement, every underwriting decision remained with an underwriter acting within carrier-delegated authority.
Give each finding to the owner who can change it
The review should end in work someone can do. Use the evidence to distinguish these causes, and keep the finding unresolved if the record can't yet support one.
| What the reviewer finds | Why the distinction matters | Repair owner and next test |
|---|---|---|
| A source arrived after the run | The assistant couldn't use information it hadn't received | Operations owner checks the update route and confirms that a late attachment triggers another review |
| An available source never reached the assistant | The input path lost evidence before reasoning began | Technology owner repairs ingestion and verifies the captured attachment manifest |
| The source reached the assistant, but a field was copied incorrectly | The evidence was available, but extraction changed its meaning | Technology owner tests that field against the original passage and similar document layouts |
| The input is correct, but the approved rule is unclear | Repeated prompt edits would encode an unresolved business choice | Underwriting or risk owner resolves the rule within existing authority before the workflow changes |
| The rule is clear, but the proposal conflicts with it | The assistant applied a known instruction incorrectly | Technology owner repairs the behavior and underwriter checks both failure cases and previously accepted cases |
| Qualified reviewers disagree | There isn't yet a stable reference for grading the proposal | Accountable underwriting owner resolves the case or retains manual review without forcing a model-error label |
A spreadsheet or your existing issue tracker can hold these records. Start there if it preserves the evidence and makes ownership clear. A new monitoring platform won't resolve an unclear program rule, and a new model won't retrieve an attachment your connection never passes through.
Copy a log that carries the decision through to retesting
Use one record per reviewed output, including outputs accepted without correction. Keep sensitive submission material in your approved environment and link to it through existing access controls. The log needs enough detail for an authorized reviewer to reproduce the comparison without copying the whole submission into another system.
| Field | Worked S-104 record | Blank record |
|---|---|---|
| Population and sample | One program's intake outputs for the review week. Regular sample, not the serious-error list | [fill in] |
| Submission and run | S-104, original run R1 | [fill in] |
| Input and system versions | Email E1, manifest M1, rules P1, workflow W1, prompt V1, model release recorded | [fill in] |
| Assistant proposal | Poor fit | [fill in] |
| Underwriter reference and reason | Further review, supported by supplement A2 | [fill in] |
| Evidence timing | A2 was present in the original email but absent from M1 | [fill in] |
| Finding and ambiguity | Capture failure. Also check whether missing evidence should have triggered manual review under P1 | [fill in] |
| Correction effort | Record actual minutes to investigate and prepare the corrected work | [fill in] |
| Owner and repair | Technology owner repairs capture. Underwriting owner confirms the missing-evidence route | [fill in] |
| Temporary restriction | Affected submissions follow the existing manual intake route | [fill in] |
| Retest and release decision | Reproduce R1, test capture and missing-evidence behavior separately, then obtain owner acceptance | [fill in] |
| Next check | Record the next review date and inspect the same failure type in the next live sample | [fill in] |
The worked row now says the supplement was already present, which selects the capture-failure repair. If timestamp evidence instead shows a later arrival, change the finding and repair to the update route. Keep that evidence change in the history so the team can see why the decision changed.
Restore the behavior only after the repair passes
A repaired connection should first prove that the previously missing attachment reaches the assistant. Then test the assistant with complete and incomplete versions of the submission. The complete version checks whether it uses the supplement. The incomplete version checks whether it asks for missing evidence or sends the submission to manual review under the approved rule.
Also rerun previously accepted examples from the same program so a repair to one route doesn't silently break another. Agree the acceptance conditions before testing, including which failures require continued restriction. Have the underwriting owner accept the proposed routes and the technology owner record the tested version. If either condition remains unresolved, keep the affected behavior on the manual route.
Track the review effort as well as the disagreement count. If a lower disagreement rate depends on underwriters doing more preparation before every run, the work may have moved rather than disappeared. Compare the same task and program, record changes in the submission mix, and measure total preparation and checking effort before calling it an improvement.
Your first completed log should support one decision: keep the behavior, investigate a named gap or restrict it until a specified test passes. Start with the correction that repeatedly consumes underwriter time, and trace it to the evidence needed to assign the right owner. That gives the team a repair it can verify instead of another week of unexplained overrides.
A reliable intake assistant can release attention for underwriting, but your next growth goal may be limited elsewhere. Start the wider investment decision with the result your business needs over the next 3–18 months, then investigate where people, time and money will do the most to move it forward.
If you’re an MGA operations or underwriting leader and want to grow your firm without growth turning into more intake and reporting work, we’ll identify the biggest bottleneck standing in your way, build the plan to remove it, and implement it in eight weeks* or less - guaranteed or we continue to work for free and you don’t pay.
Find my bottleneck* Eight-week delivery, or twelve weeks when substantial data foundations are required, as agreed before work begins. If a covered stage fails its agreed benchmarks, we provide up to four additional weeks of remedy work at no charge, then waive or refund that stage’s fee if it still fails. Successfully delivered stages remain payable. Agreed client-caused delays pause the clock. Delivery guarantee terms.