> Metatheory > Method and Instrumentation > arghXiv:2606.12903
Inter-Coder Agreement on the Twenty-Four-Hour Criterion
Abstract: A criterion that requires a judgement will be applied differently by different coders, and the disagreement enters the result. The criterion of arghXiv:2604.11238 requires no judgement: it requires a clock. Six coders, three of them with no training in discourse analysis, applied it independently to the same 400 disputes. Agreement on whether a dispute had terminated was κ = 0.98, and the four disagreements all concerned exchanges in which a message had been edited after posting. The corresponding figure for the schedules in current use is κ = 0.24 on the assignment of a non-answering category. We do not suggest that those schedules are poorly constructed. We suggest that a criterion which an untrained coder can apply to a timestamp is to be preferred to one which cannot, and we again recommend its general adoption.