Buying Explainability: An Evidence Contract for Family-Office AI Vendors

The procurement problem

A vendor can make an investment dashboard appear understandable without making its judgments dependable. A feature-importance chart may identify variables associated with a prediction while saying little about whether the model is accurate, whether the explanation is faithful, or whether the proposed action is sensible. A family office buying AI should therefore specify an evidence contract before discussing interface polish. The contract names the decision, the responsible person, the permissible model role and the evidence needed to challenge an output. It should cover a narrow operating use, such as screening documents for review, rather than grant a broad entitlement to decide where capital goes.

What the review contributes

Weber, Carl and Hinz screened 2,022 publications and analyzed 60 relevant articles on explainable AI in finance [1]. Their synthesis describes both transparent models and explanations applied after a complex model has produced an output. The reviewed applications and explanatory goals were unevenly distributed. For a buyer, the implication is to avoid treating explainability as one interchangeable feature. A risk analyst, beneficiary and independent reviewer may require different evidence. The review maps a literature rather than testing a procurement intervention. Its coverage also precedes many recent systems, so a contemporary vendor must supply its own version-specific evaluation.

Specify three separate acceptance questions

The first question is predictive validity: what task does the model perform, against which reference labels, on which population and period? Require out-of-sample evaluation, class-specific errors and a plain description of known failure conditions. The second question is explanatory fidelity: does the explanation track what the system actually used, and does it remain stable under changes that should not matter? A persuasive natural-language rationale is not sufficient evidence. The third question is decision usefulness: can a trained reviewer identify unsupported inputs, detect a consequential failure and choose an appropriate next step? Record these questions separately. High accuracy cannot compensate for an explanation that misrepresents the model, and a lucid explanation cannot rescue a badly performing predictor.

Make the vendor demonstrate a challenge

Select a small set of authorized, representative cases and a few difficult cases the vendor has not optimized for. Include missing evidence, changed economic conditions, contradictory document versions and records close to a decision boundary. Ask the vendor to expose the input provenance, model version, explanation method and output limitations. An internal specialist should formulate a plausible alternative interpretation before seeing the vendor explanation. Compare the two and retain disagreements. The exercise should reveal the circumstances in which a reviewer must stop, seek more information or use another method. It is a demonstration of inspectability, not a prediction of investment returns. Keep confidential documents out of unapproved services and use a sanitized test set when necessary.

Treat explanation changes as operating changes

An explanation can change because the data changed, the model changed, or the method describing the model changed. These events have different implications and should not be hidden inside a routine software update. Require a version register and a notice for material changes to the approved use. The office should reproduce a stable reference suite after an update and examine failed cases before expanding access. Compare user comprehension with actual error detection, since users may feel more confident without becoming better judges. Give reviewers a route to challenge an explanation without being penalized for slowing the workflow. The desired outcome is justified reliance, not maximal adoption or an attractive confidence score.

Implementation ownership and review

Owner: Technology diligence lead. Action: Define the approved task and three distinct acceptance questions before vendor selection. Evidence: Task specification, reference cases, error categories and version record. Measure: Coverage of material failure conditions and unresolved acceptance questions.

Owner: Independent domain reviewer. Action: Test fidelity and usefulness on authorized cases withheld from vendor tuning. Evidence: Source-grounded challenges, reviewer disagreements and corrected outputs. Measure: Material errors detected, unsupported explanations and comprehension gaps.

Owner: Investment governance chair. Action: Authorize a bounded scope and rehearse suspension and export. Evidence: Approval conditions, update policy and successful record export. Measure: Unreviewed scope changes and time to suspend an affected use.

A decision and an exit route

The procurement committee should finish with a bounded authorization: approved tasks, users, data classes, review obligations and withdrawal conditions. A failed explanation-fidelity check or an unexplained material error should pause the affected use until the cause is understood. A vendor that cannot export input provenance and review records creates an additional dependency; test the exit process before signing a long agreement. Measure recurring disagreement, unresolved source gaps, specialist review time and the proportion of high-consequence outputs independently checked. These measures are proposed operating indicators, not standards validated by the review. A family office earns efficiency when it can explain why a decision is warranted and preserve the ability to change its mind.

Source, attribution and limits

[1] Patrick Weber; K. Valerie Carl; Oliver Hinz. Applications of Explainable Artificial Intelligence in Finance—a systematic review of Finance, Information Systems, and Computer Science literature. Management Review Quarterly (2023-02-28). https://doi.org/10.1007/s11301-023-00320-0. CC BY 4.0.

Review coverage concerns research from 2010–2021 and its selected outlets/search strategy. Findings do not establish current model rankings, causal effects of explanations on decision quality, or legal compliance for a specific family office.

Prepared by the Family Office Private Wealth Journal editorial research desk with AI assistance at the direction of Dr. Freedom Cheteni. Original research authors retain full credit and are not authors or endorsers of this companion. This proposed process has not been evaluated in a new empirical study; no independent peer-review acceptance is claimed. This is educational analysis, not individualized investment, legal or tax advice.

Implementation record

Open the cited sources

1. [1] Patrick Weber; K. Valerie Carl; Oliver Hinz. Applications of Explainable Artificial Intelligence in Finance—a systematic review of Finance, Information Systems, and Computer Science literature. Management Review Quarterly (2023-02-28). ↗