Independent central reading or off-site reading of imaging endpoints is increasingly used in clinical trials. Clinicianreported\noutcomes, such as endoscopic disease activity scores, have been shown to be subject to bias and random\nerror. Central reading attempts to limit bias and improve accuracy of the assessment, two factors that are critical to\ntrial success. Whether one central reader is sufficient and how to best integrate the input of more than one central\nreader into one output measure, is currently not known.\nIn this concept paper we develop the theoretical foundations of a reading algorithm that can achieve both\nobjectives without jeopardizing operational efficiency We examine the role of expert versus competent reader,\nframe scoring of imaging as a classification task, and propose a voting algorithm (VISA: Voting for Image Scoring\nand Assessment) as the most appropriate solution which could also be used to operationally define imaging gold\nstandards. We propose two image readers plus an optional third reader in cases of disagreement (2 + 1) for\nordinary scoring tasks. We argue that it is critical in trials with endoscopically determined endpoints to include the\nscore determined by the site reader, at least in endoscopy clinical trials. Juries with more than 3 readers could\ndefine a reference standard that would allow a transition from measuring reader agreement to measuring reader\naccuracy. We support VISA by applying concepts from engineering (triple-modular redundancy) and voting theory\n(Condorcetââ?¬â?¢s jury theorem) and illustrate our points with examples from inflammatory bowel disease trials,\nspecifically, the endoscopy component of the Mayo Clinic Score of ulcerative colitis disease activity. Detailed\nflow-diagrams (pseudo-code) are provided that can inform program design.\nThe VISA ââ?¬Å?2 + 1ââ?¬Â reading algorithm, based on voting, can translate individual reader scores into a final score in a\nfashion that is both mathematically sound (by avoiding averaging of ordinal data) and in a manner that is\nconsistent with the scoring task at hand (based on decisions about the presence or absence of features, a\nsubjective classification task). While the VISA 2 + 1 algorithm is currently being used in clinical trials, empirical data\nof its performance have not yet been reported.
Loading....