Evaluating AI claims in inspection software

Every inspection software vendor now has an AI story. Most of them sound the same on a slide: ask a question in plain language, get an answer, save hours. The trouble is that an inspection record is evidence. It is what you hand a jurisdictional inspector, an insurer, or an incident investigator. An assistant that produces a fluent wrong answer about a vessel's remaining life is worse than no assistant, because it looks like work that was done. This paper is a checklist for evaluating AI claims the way you would evaluate a new NDE method: by asking what it measures, how it fails, and what evidence supports the claim.

What is the AI actually reading?

The first question is about scope. A general-purpose model answers from what it learned during training, which includes public standards commentary, forum posts, and marketing copy. None of that is your plant. A grounded assistant answers from a retrieval step over your own records and documents, and only from that. The difference shows up immediately in the kind of wrong answer you get. An ungrounded model will confidently tell you a corrosion rate that sounds right. A grounded one will tell you that no thickness readings exist for that CML.

Questions to put to the vendor:

• Which data does the assistant retrieve from? Inspection records, attached PDFs, procedures, drawings, or only some of these?
• Is retrieval scoped to the tenant, and does it respect the same data entitlements the user already has? A technician who cannot open a site's records should not be able to ask the assistant about them.
• What happens when the retrieval finds nothing? The correct behavior is a stated refusal, not a general answer.
• Does the assistant see the current record or a stale copy from an index built last night?

Spectra, the grounded layer in intelliSPEC, retrieves over the tenant's own documents and records, and every answer carries a citation to the source record and the collection it came from. That is the behavior to look for. The claim is checkable: open the cited record and confirm it says what the answer says. See Spectra for how the citation path is built.

Does every answer carry a citation you can open?

Citations are the only practical defense against hallucination in day-to-day use, because nobody will re-derive every answer. The requirement is not a footnote that names a document. It is a link that opens the exact record, so the reviewer can see the thickness reading, the finding, or the procedure paragraph that the answer rests on.

Test this during evaluation, with your own data loaded:

• Ask for the last three thickness readings at a CML and open every citation. Do the numbers match the record?
• Ask a summary question across an asset ("what damage mechanisms are recorded on this exchanger?") and check that each mechanism named traces to a record.
• Ask about a document that was never uploaded. The assistant should say it has no source, not paraphrase what a typical document of that kind would say.

Ask what a citation resolves to. A page reference inside a PDF is useful for a document library, but in an inspection system the unit of truth is the record, and the citation should resolve to the record itself so it survives the next revision of the document.

Can the AI change data, and under what controls?

Reading is one risk class. Writing is another. Many products now offer agentic actions: the assistant creates a work item, updates a finding status, schedules an inspection. Each of those is a change to a controlled record, and the question is whether it is governed like one.

The controls to look for, in order of importance:

• Approval by default for creates and updates. The action is proposed, and a named user approves it before it lands.
• Destructive actions blocked, not merely gated. Bulk delete should be impossible for the assistant, full stop.
• Time-boxed approvals. An approval request that sits for an hour is a request nobody is looking at. It should expire, and it should escalate to someone else before it expires.
• An undo log for every mutation the assistant performs, so a bad approval can be reversed without a database restore.
• The same audit trail entries a human change would produce, with the actor recorded as the approving person.

In intelliSPEC, Spectra's actions are generated from the platform's own metadata (59 actions in 9 categories today), create and update actions require approval by default, bulk delete is blocked outright, approvals expire after 10 minutes and escalate after 5, and every mutation lands in an undo log. Those specific numbers are less important than the shape: every write path has a human in it and a way back.

How does the AI resist manipulation?

Prompt injection is the attack where text in a document tells the model to do something the user did not ask for. In an inspection context the vectors are obvious: an uploaded vendor report, a contractor's notes, a pasted email. If the assistant retrieves that text and follows instructions inside it, an attacker who can upload a PDF can alter your records or pull data out.

Ask three things:

1. Is user input checked for injection patterns before it reaches the model?
2. Is retrieved content scanned as well? This is the one most vendors miss. Injection hidden in a document is more dangerous than injection typed by a user, because the user is trusted and the document may not be.
3. What is the blast radius if an injection succeeds? If writes need approval and deletes are blocked, a successful injection can at most propose something a human then sees.

The third answer matters most. Detection will never be perfect. Containment can be.

Where is the line between AI and engineering judgment?

An assistant can retrieve, summarize, draft, and flag. It should not derate a vessel, set an inspection interval, or declare a defect acceptable. Those are engineering decisions with a code basis, and they belong in deterministic calculators that show the design code, the formula, and the inputs, with an engineer's verification on the result.

The evaluation test is to ask the assistant for an engineering conclusion and watch what it does. The right response is to run or retrieve the calculation and present it as provisional pending an engineer's review, or to decline. The wrong response is a confident remaining-life figure with no visible arithmetic.

Two honesty signals worth noticing in any product:

• Calculators that return provisional results until a qualified person verifies them, and never auto-derate the asset.
• Vendors who say plainly what they have not implemented. In intelliSPEC, the tMin engine returns the design code and formula with the number, ASME B31G results are provisional until an engineer verifies them, and API 579-1 Part 5 Level 1 is deliberately not implemented pending licensed-engineer verification. A vendor who tells you what the software will not do is easier to trust on what it will.

What about voice and vision features?

Voice-to-form and photo analysis are where AI is most useful in the field and least dangerous, because the output is a draft the technician sees before it is saved. Even so, apply the same standard.

• Voice to structured fields. The value is not a transcript. It is that "east nozzle, pitting, three millimeters deep, photo attached" becomes a finding record with location, mechanism, and dimension populated. Check that the technician can correct each field before submitting.
• Vision analysis on photos and video. Useful for flagging suspect areas and drafting condition notes. Check that the output is labeled as a draft assessment, that it is attached to the component and finding rather than a loose camera roll, and that a human classification is still the record of truth.
• Offline behavior. Most model calls need connectivity. Ask what happens on a unit with no signal. The honest answer is usually that capture works offline and analysis runs at sync, which is fine if it is stated.

Spectra offers speech to structured fields and vision analysis on photos and video condition assessment, and the underlying module for visual work is intelliINSPECT. In every case the field record is what the technician confirms, not what the model guessed.

What evaluation evidence should you demand?

Any vendor can run a demo on curated data. Ask instead for the test artifacts a serious team would keep:

• A groundedness suite: a set of questions over realistic record shapes with the expected cited sources, and the pass rate over time.
• A red-team suite: injection attempts, exfiltration attempts, and permission-boundary probes, with what happened.
• A description of how retrieved content is scanned and what is stripped.
• The approval-gate configuration, including default settings and whether a tenant admin can weaken them.
• A statement of what model vendors or components are used and how tenant data is or is not used to train anything.

The last point is a data-handling question rather than an AI question, and it belongs in the same security review as encryption and audit logging. The Trust Center at /security is where intelliSPEC publishes its posture; the same page should exist for any vendor you evaluate.

How intelliSPEC handles it

The checklist above describes the design of Spectra, so the fair thing is to state the facts plainly and let you test them.

• Retrieval runs over the tenant's own documents and records, and every answer cites the source record and collection.
• Actions are generated from the platform's metadata rather than hand-coded: 59 actions in 9 categories today. Create and update actions require approval by default, bulk delete is blocked outright, approvals expire after 10 minutes and escalate after 5, and every mutation has an undo log.
• User input is checked for prompt injection and retrieved content is scanned before it reaches the model.
• Voice-to-form converts speech to structured fields; vision analysis runs on photos and video condition assessment. Both produce drafts the technician confirms.
• Groundedness and red-team evaluation suites are maintained and can be walked through in a technical evaluation.
• Engineering calculations stay deterministic and inspectable. Results carry the design code and formula, B31G results stay provisional until an engineer verifies them, and API 579 Part 5 is not implemented until it can be verified by a licensed engineer.
• Every action, human or assisted, lands in a hash-chained audit trail with actor, tenant, request id, and timestamp.

Bring your own records to the evaluation, load them, and ask the assistant hard questions. Then open every citation. If the terms above are unfamiliar, the glossary defines them the way this paper uses them.