Method research

Do independent assessors reach similar conclusions?

We do not yet know how consistently different people assess the same case. In v0.8, we first investigate what ArtiProof adds to existing practice on new cases. A direct comparison between independent assessors requires a separate study.

Another assessor should be able to follow how you reached your conclusion. Whether independent assessors also reach similar conclusions needs a separate study. Give them the same evidence, scope, guidance and information about possible consequences. Record their first judgements before they discuss or calibrate their assessments. This is what allows you to assess their initial agreement.

Working hypothesis. Five assurance properties plus explicit context, consequence and proportionate evidence may be sufficient for a defensible fitness-for-reliance judgement without adding a sixth property or one aggregate score.

What the v0.8 challenge records

whether a good existing approach already finds the same problem;
whether ArtiProof repeatedly finds a problem that the existing approach misses;
whether that finding affects the decision;
whether the method flags problems that are not there or requires too much extra work;
whether the designer is needed to interpret the conclusions.

What remains for later reproducibility research

Compare independent assessors who receive the same evidence and instructions, within the same scope and with the same information about possible consequences. If their conclusions still differ significantly, we need to examine where the method is unclear or inadequate.

This research still needs to establish how reliably assessors agree.

See the current research status

ArtiProof is ready for research, but has not yet been broadly validated by independent researchers. The first independent replication is still pending. More about the research.