How to choose AI audit trail software: eleven questions to put to any vendor
AI audit trail software records what an AI system decided, when, on which model version, and whether a human overrode it, then holds those records in a form a third party can inspect and test later. This guide is the evaluation checklist: eleven questions, in the order you should ask them, that separate a searchable log from evidence you would be willing to hand to an examiner.
Auditrek is not open yet, so anything on this page about what it does is what it is being built to do.
What AI audit trail software is
AI audit trail software captures a durable record of each decision an AI system produced, and keeps it in a structure someone outside your team can read after the fact. A record that earns the name usually names the system, the model identifier and version, the time, the output, a confidence value where the model emits one, a reference to the input, and whether a human overrode the result. The audit trail part is not the capture, it is the custody: the assurance that the record you show in month eighteen is the record that was written in month one.
Three quite different product categories answer to the same search term, and the demos blur them deliberately.
- Governance platforms organise policy, risk registers, model inventories and questionnaires. They describe your programme. They rarely hold per-decision facts.
- ML observability and monitoring watches drift, latency and output quality. It is built for the team operating the model, and its retention windows are usually tuned to debugging, not to examination.
- Evidence layers hold one record per inference event, make that record tamper-evident by a named mechanism, and export it in a structure an auditor or supervisory authority can read.
If you searched for audit trail software rather than for a governance platform, you are almost certainly shopping in the third group, and the questions below are the ones that distinguish members of it.
The vocabulary around the search has hardened over the past two years. Compliance teams in the EU talk about record-keeping and logging obligations under Article 12 of the AI Act and about transparency and provision of information to deployers under Article 13. Teams in US financial services reach for the model risk management language they already use with their examiners. Internal audit asks a blunter version of the same thing: can we prove it, and to whom.
None of that tells you which tool to buy. It tells you what the tool has to survive, which is a person who was not present when the decision was made, reading the record and looking for reasons to doubt it.
The decision actually in front of you
Most teams that reach this page are not starting from nothing. Inference events are already landing somewhere: a logging pipeline, a warehouse table, a monitoring vendor. The gap is rarely capture. The gap is provenance and export.
So the real choice is between three options, and it is worth naming them before you take any calls.
Extend what you have
Add fields to the events you already emit, and write them to a table with update permissions revoked. This is cheap and it is honest about its own limits. What it cannot do is answer the follow-up question, which is how you know nobody with database credentials changed a row. Ordering is not integrity.
Build the evidence layer yourself
A hash chain is not exotic engineering. Computing each record's digest from the previous digest plus a canonical payload, inside one transaction, is a week of careful work and then a long tail of questions about canonicalisation, replay, backfills, key rotation, report rendering and who maintains it when the person who wrote it changes teams. Teams that build it usually underestimate the report, not the chain.
Buy a layer that sits beside the inference path
The reason to buy is not that hashing is hard. It is that the report, the verification interface and the ingestion controls are the parts you do not want to be inventing in the week before an examination. The reason to hesitate is that you are handing custody of evidence to a third party, which is exactly why questions five through eight below exist.
One trade-off cuts across all three. A tool that sits inline, with your inference call routed through it, gives you the most complete capture and puts a vendor in your latency path and your incident blast radius. A tool that receives events after the decision, over an API or a webhook, keeps production clean and makes capture completeness your responsibility. Decide which of those you can defend before you compare feature grids, because it narrows the field faster than anything else.
Questions one to four: what gets captured, and where from
What is the unit of record, and what is it scoped to?
Ask whether one record is one inference event, one API call or one batch job, and what records are grouped under. The answer shapes both your evidence structure and your invoice. Auditrek stores records per ai_system as append-only entries, and bills per connected AI system across starter and growth tiers, so the grouping and the commercial unit are the same thing. If a vendor cannot tell you plainly what one row represents, later answers about integrity will not mean much.
Which fields are in the record, and are model id and version among them?
The question examiners ask most often is which version of the model produced this output on this date. A record without a model version cannot answer it, and adding the field later does not repair the months already logged. Auditrek's ingestion payload takes an input hash, model id and version, output, confidence, timestamp and a human override flag. Ask any vendor for the exact field list in writing before the demo, and check that human oversight is a first-class field rather than a free-text note someone remembers to fill in.
Where does capture sit relative to the production inference path?
This decides how much of your platform team's year you are spending. Some tools ask you to route inference through them. Others accept events afterwards. Auditrek takes events over a REST endpoint or a webhook, with documented payload mapping guides for AWS SageMaker and Azure ML rather than bespoke platform SDKs, so no model migration is involved. Whichever shape you choose, ask the vendor what happens to the decision when the logging call fails, and what happens to the chain when events arrive out of order.
What happens to the input itself?
The prompt or feature vector is often the most sensitive part of the event and the least useful to keep verbatim. Ask whether the tool stores the input, hashes it, or both, and who can read it afterwards. Auditrek's payload field is an input hash, which lets you show later that a specific input produced a specific output without carrying the input itself into the ledger. If a vendor stores raw inputs by default, that is not automatically wrong, but it is a data protection conversation you now own.
Questions five to eight: tamper evidence you can test
By what mechanism are the records tamper-evident?
Immutable is a marketing word until somebody names a mechanism. Make the vendor pick one and describe it: append-only writes with update permissions revoked, retention locks on object storage, digital signatures over each record, or a hash chain where each record's digest covers the previous digest. Auditrek writes SHA-256 hash-chained records, computing each hash from the previous hash plus the payload and storing sequence number, previous hash and hash atomically in one transaction. Specificity here is the whole signal: a vendor who cannot describe the mechanism in one sentence does not have one.
Is tamper status recomputed on request, or read from a stored flag?
This is the question that separates evidence from theatre. A column that says verified was written by the same system that wrote the record, so a privileged user or a careless migration that edits one can edit the other. Ask whether verification walks the chain and recomputes hashes at the moment you ask. Auditrek's verification endpoint recomputes the chain from the first sequence up to the record in question and returns a tamper status, with the first broken sequence number when the chain does not hold, rather than returning a stored flag alone.
Which fields are inside the digest, and which sit outside it?
A hash chain protects only what it hashes. Ask for the exact list of fields in the canonical payload, ask specifically about the timestamp, and ask what verification returns if you edit a field that sits outside the digest. Then ask whether timestamps are attested by anything external, such as a trusted timestamping token, or whether they are the server clock. A vendor who answers this precisely is telling you where their guarantee ends, which is far more useful in a filing than one who repeats the word immutable.
Can someone who does not trust the vendor verify the chain?
Your evidence should not depend on the vendor being available, solvent or honest at the moment of scrutiny. Ask whether records and their hashes can be exported in bulk, whether the hashing rule is documented closely enough that one of your engineers could recompute the chain independently, and whether an external auditor can call the verification interface directly. If the only path to verification is a green tick inside the vendor's own screen, you have bought a claim, not a proof.
Questions nine to eleven: what you let in, what comes out, and how you leave
What stops sensitive fields entering the ledger in the first place?
Inference payloads from credit, hiring and clinical pipelines routinely carry data you never intended to copy into a second system, and an append-only store is the worst possible place to discover that. Redaction after ingestion is a cleanup job. A gate at ingestion is a control. Auditrek rejects payloads carrying recognisable raw sensitive field patterns, including social security number formats, ICD-10 style diagnosis codes and unredacted credit bureau fields, returning a 422 before any database write, and pseudonymises subject identifiers by default. It also refuses to issue an API key or webhook secret for a system until a data processing agreement confirmation with a signed_at date is recorded. Ask each vendor which of those three controls is enforced server side, and which is a note
What does the report actually contain, and what does it disclaim?
Read a sample before you buy anything. Ask which regulation the structure follows, which output formats you get, whether you can choose one system and one date range, and whether the report asserts anything about your legal position. Auditrek generates PDF and HTML output mapped to EU AI Act Article 12 and 13 fields for a chosen system and date range, with a single regulation and format set at launch, and embeds a fixed disclaimer on every page stating that the report does not constitute legal advice and has not been reviewed by qualified regulatory counsel. A sample report that tells you that you are in good standing is making a judgement your counsel gets to make, not your vendor.
Who in your organisation can reach the evidence, and how do you get out?
Evidence that lives behind one person's login is a single point of failure during an examination, and an evidence store with no export path is a slow trap. Ask how many people can be in the workspace and how they authenticate, ask what an export of raw records and hashes looks like, and ask what happens to the chain when you stop paying. Auditrek provides auth-gated multi-user organisation access with email and password sign-in, is hosted on Amazon Web Services, and handles per-organisation billing through Stripe based on the number of connected AI systems.
Four ways teams hold AI decision evidence, and what each one actually shows
| Approach | What it can show | What it cannot show | Where it tends to fail under scrutiny |
|---|---|---|---|
| Application and platform logs | That an inference happened, with whatever fields a developer chose to emit at the time | That an entry was not edited, or that the field set was complete on the date in question | Retention windows expire, schemas drift between releases, and whoever operates the log store can rewrite it |
| Append-only table in your own database | An ordered sequence of records under your own keys and your own access policy | That no privileged user or migration performed an update, because ordering is not integrity | Nobody can demonstrate the absence of an out-of-band edit, which is the exact question asked |
| Write-once object storage with retention settings | That a stored object was not overwritten while its retention period was in force | Whether the contents were correct when written, or how one decision relates to the next | Evidence is per file rather than per decision, so reconstructing a decision history means parsing archives |
| Hash-chained decision records with server-side recomputation | That the sequence is internally consistent, and which sequence number is the first to break | Anything about fields excluded from the digest, or facts that were never sent into the record | Field coverage of the digest, and whether anyone outside the vendor can recompute the chain |
How to run the evaluation in two weeks
- Write down the systems, then pick the hardest one
Start from your AI use-case registry, or build a first draft of one in an afternoon. Pick the system you would least like to explain to an examiner, usually the one making decisions about people. Evaluating on your easiest pipeline produces a pleasant pilot and tells you nothing.
- Fix the payload contract before you send anything
Take one real inference event and decide, field by field, what leaves your perimeter. Hash the input, pseudonymise the subject reference, drop anything that is special-category data unless your own lawful basis is documented. Write this down as a contract, because it is the artefact your data protection colleague will ask for.
- Wire capture into a staging path first
Send events from a staging or shadow path over the vendor's API or webhook and confirm that the production decision is unaffected when the logging call is slow or fails. Log a few hundred events, including at least one human override, so you have a sequence rather than a single row.
- Break a record on purpose
This is the step vendors are least prepared for and the one that decides the purchase. Edit a stored record, then call verification and see what comes back. Repeat with a field you suspect is outside the digest, such as the timestamp. Record both results verbatim in your evaluation notes, including the sequence number reported as the first break.
- Generate the report for a real date range and read it as a stranger
Produce the output in every format offered, for a range that includes your override event, and read it as somebody who was not in the room. Check that the record count matches what you logged, that nothing overflows or truncates, and that the structure maps to the Article 12 and Article 13 fields your counsel has flagged.
- Hand the report to counsel with the disclaimer intact
Ask your internal or external counsel one question: what is missing from this for our purposes. Their gap list, not the vendor's feature list, is your requirements document. Keep the non-legal-advice disclaimer in the copy you circulate, so nobody downstream mistakes a structural record for a legal opinion.
What no audit trail tool can do for you
Three limits are worth stating plainly, because the category's marketing tends to leave them out.
A tool cannot tell you which of your systems are in scope, or which obligations attach to them. That determination is legal work, and it depends on your role in the value chain, your sector and your jurisdiction. Any product that answers it for you is selling you a judgement it is not positioned to make.
A tool cannot record what you never sent. Completeness of capture is an integration property, owned by whoever wires the events, and no amount of hashing repairs a pipeline that silently stopped emitting in March. Put a check on event volume per system next to your other operational alerts.
A tool cannot make a report into a finding. A structured record of what an AI system did, when, on which model version, and where a human intervened, is evidence. Whether that evidence satisfies a given obligation is decided by a supervisory authority or an auditor reading it, with your counsel's work alongside it.
What a good tool does is narrower and still valuable: it makes the record hard to change quietly, and it makes the export cheap to produce on a deadline.
From here, three companion pages in this pillar go deeper than a buyer's guide can. The page on the compliance dashboard covers the day-to-day view of decisions logged, systems connected, last report generated and tamper-verification status. The page for the compliance officer who has to produce the evidence works through the examination scenario itself. The page on multi-user organisation access covers how compliance, risk and engineering share one workspace without sharing one login.
Questions
What is AI audit trail software?
Software that captures one durable record per AI decision, holds it in a form that resists quiet editing, and exports it in a structure a third party can read. In practice a record names the system, the model id and version, the time, the output, a confidence value where one exists, a reference to the input, and whether a human overrode the result.
Is an audit trail the same as ML monitoring?
No. Monitoring is built for the team operating the model and answers questions about drift, latency and output quality, usually with retention tuned to debugging. An audit trail is built for a reader who was not present, and its central property is custody: that the record shown later is the record written at the time.
Do hash-chained records require a blockchain?
No. A hash chain is simply each record's digest computed over the previous digest plus that record's canonical payload, so any later edit breaks every hash after it. Auditrek uses SHA-256 hash-chained records with a verification endpoint that recomputes the chain server side and returns a tamper status. A distributed ledger adds consensus across parties, which is a different problem from detecting edits in your own store.
Can we just use our cloud provider's logging instead?
Sometimes, if you can answer three questions: whether the retention window covers the period you may be asked about, whether the field set includes model version and human override, and how you would demonstrate that entries were not edited by whoever administers the log store. If any answer is uncomfortable, that is the gap an evidence layer fills.
How long should we retain AI decision records?
The period comes from the obligations your counsel identifies plus your own records policy, and it is not something a vendor should set for you. The question to put to a vendor is mechanical rather than legal: can the store hold records for the period you set, and can you export them in bulk at the end of it.
What does Auditrek do today, and what does it cost?
Auditrek captures inference events over a REST endpoint or webhook, stores them as SHA-256 hash-chained append-only records per AI system, recomputes the chain on request through a verification endpoint, rejects payloads carrying recognisable raw sensitive fields, requires a signed data processing agreement confirmation before issuing credentials, and generates PDF and HTML reports mapped to EU AI Act Article 12 and 13 fields with a non-legal-advice disclaimer on every page. Billing is per connected AI system across starter and growth tiers through Stripe, and sign-in is email and password. The product is pre-launch, so pricing is not published yet and access starts from the waitlist.