Surfacing problems of explainability and tracing the priors that shape evidence-based reasoning.
-
Data lineage is a solved problem in principle. The W3C PROV family specifies how to record that an entity was generated by an activity attributed to an agent, and the standards work. What is not solved is provenance of reasoning — the priors, assumptions, source judgements and conditions that shape an output, whether the output came from a person or a machine.
Most current work approaches this through explainability, which produces an account of a decision after the decision has been made. An explanation generated after the fact is a reconstruction, and its relationship to what actually happened is an open question. A record made at the time is a different kind of object.
Related programmes: Governance Frameworks, Epistemic Assurance.
-
Lineage standards exist, are mature, and interoperate.
Records can be made tamper-evident at low cost, through content addressing and append-only structures.
Retrospective reconstruction of a decision from logs is unreliable where the logs were not designed for the purpose.
-
Whether post-hoc explanation of a statistical model carries evidential weight, or only reassurance.
Whether the provenance of human judgement can be recorded without imposing overhead that practitioners will not accept.
Whether explainability and provenance are complementary or in competition for the same governance budget.
-
What minimum record makes a decision reconstructable by someone who was not present.
How priors and assumptions get recorded at all, given that they are frequently not articulated even by the person holding them.
Whether evidential quality degrades measurably across a chain of transformations, or only arguably.