Kappa Statistics

<body> <b><font face="Arial" size="4"><p align="center"></font><font size="4">Agreement between Categorical Measurements</font></b></p> <hr> <b><p>Kappa Statistics: </b>an index to correct chance agreement.</p> <b><p>Hypothetical Example: </b>There are two doctors to diagnosis the same 29 patients. </p> <table border="1" cellPadding="7" cellSpacing="1" width="638"> <tbody> <tr> <td colSpan="2" rowSpan="2" vAlign="top" width="26%"> <p> </td> <td colSpan="2" vAlign="top" width="50%"><b><p align="center">Doctor A</b></td> <td vAlign="top" width="25%"> </td> </tr> <tr> <td vAlign="top" width="25%"><b><p align="center">No</b></td> <td vAlign="top" width="25%"><b><p align="center">Yes</b></td> <td vAlign="top" width="25%"><b><p align="center">Total</b></td> </tr> <tr> <td rowSpan="2" vAlign="center" width="13%"><b>Doctor B</b></td> <td vAlign="top" width="13%"><b><p align="center">No</b></td> <td vAlign="top" width="25%"><b><p align="center">10 (34.5%)</b></td> <td vAlign="top" width="25%"><b><p align="center">7 (24.1%)</b></td> <td vAlign="top" width="25%"><b><p align="center">17 (58.6%)</b></td> </tr> <tr> <td vAlign="top" width="13%"><b><p align="center">Yes</b></td> <td vAlign="top" width="25%"><b><p align="center">0 (0.0%)</b></td> <td vAlign="top" width="25%"><b><p align="center">12 (41.4%)</b></td> <td vAlign="top" width="25%"><b><p align="center">12 (41.4%)</b></td> </tr> <tr> <td colSpan="2" vAlign="top" width="26%"><b><p align="center">Total</b></td> <td vAlign="top" width="25%"><b><p align="center">10 (34.5%)</b></td> <td vAlign="top" width="25%"><b><p align="center">19 (65.5%)</b></td> <td vAlign="top" width="25%"><b><p align="center">29</b></td> </tr> </tbody> </table> <b><p>Definition:</p> <p>Kappa = (Observed agreement - Chance agreement)/(Maximum possible agreement - Chance agreement)</p> <p>Agreement</b> = (10 + 12)/29 = 0.76 <b>(observed agreement)</p> <p>Chance agreement</b> = 0.586 * 0.345 + 0.655 * 0.414 = 0.474</p> <b><p>Kappa</b> = (0.76 - 0.474)/(1 - 0.474) = 0.54</p> <p><b> </p> <p>What is good agreement? </b>(from Landis and Koch, 1977)</p> <p>A rough guide is provided, but this should not be taken too seriously.</p> <table border="1" cellPadding="7" cellSpacing="1" width="638"> <tbody> <tr> <td vAlign="top" width="50%"><b>Kappa</b></td> <td vAlign="top" width="50%"><b>Strength of agreement</b></td> </tr> <tr> <td vAlign="top" width="50%"><b>0.00</b></td> <td vAlign="top" width="50%"><b>Poor</b></td> </tr> <tr> <td vAlign="top" width="50%"><b>0.01-0.20</b></td> <td vAlign="top" width="50%"><b>Slight</b></td> </tr> <tr> <td vAlign="top" width="50%"><b>0.21-0.40</b></td> <td vAlign="top" width="50%"><b>Fair</b></td> </tr> <tr> <td vAlign="top" width="50%"><b>0.41-0.60</b></td> <td vAlign="top" width="50%"><b>Moderate</b></td> </tr> <tr> <td vAlign="top" width="50%"><b>0.61-0.80</b></td> <td vAlign="top" width="50%"><b>Substantial</b></td> </tr> <tr> <td vAlign="top" width="50%"><b>0.81-1.00</b></td> <td vAlign="top" width="50%"><b>Almost perfect</b></td> </tr> </tbody> </table> <p>The above Kappa of 0.54 is estimated from a fairly small sample of patients. What is the standard error of the kappa coefficient? The required formula is rather complicated and will not be presented here. Just let you know the <b>standard error </b>of the above kappa is 0.134. The <b>95% confidence interval</b> of kappa value is (0.279, 0.805). So in terms of the statements in the above guide, agreement here is somewhere between fair and almost perfect.</p> <b><p> </p> <p>Calculator: Please fill out the following four required non-negative integers. </p> <form> </b><table border="1" borderColor="#000000" cellPadding="7" cellSpacing="1" width="484"> <tbody> <tr> <td colSpan="2" rowSpan="2" vAlign="top" width="39%"> <p> </td> <td colSpan="2" vAlign="top" width="61%"><b><div align="center"><center><p>Doctor A</b></td> </tr> <tr align="center"> <td vAlign="top" width="28%"><b><div align="center"><center><p>No</b></td> <td vAlign="top" width="33%" align="center"><b><div align="center"><center><p>Yes</b></td> </tr> <tr align="center"> <td rowSpan="2" vAlign="center" width="23%"><b>Doctor B</b></td> <td vAlign="top" width="16%"><b><div align="center"><center><p>No</b></td> <td vAlign="top" width="28%" align="center"><div align="center"><center><p><input name="a" size="10"> </td> <td vAlign="top" width="33%" align="center"><div align="center"><center><p><input name="b" size="10"> </td> </tr> <tr align="center"> <td vAlign="top" width="16%"><b><div align="center"><center><p>Yes</b></td> <td vAlign="top" width="28%" align="center"><div align="center"><center><p><input name="c" size="10"> </td> <td vAlign="top" width="33%" align="center"><div align="center"><center><p><input name="d" size="10"> </td> </tr> </tbody> </table> <div align="center"><center><p>Press the button when you're ready to see your results: <input name="calc" onclick="Kappa(this.form,this.form.a.value,this.form.b.value,this.form.c.value, this.form.d.value)" type="button" value="Calculate"> </p> </center></div><div align="center"><center><p>Agreement: <input name="observed" size="20"> </p> </center></div><div align="center"><center><p>Kappa: <input name="kappa" size="20"> </p> </center></div><div align="center"><center><p>Standard Error: <input name="se" size="20"> </p> </center></div><div align="center"><center><p>95% Confidence Interval: <br> Lower Limit: <input name="lower" size="20"> <br> Upper Limit: <input name="upper" size="20"> </p> </center></div><div align="center"><center><p> </p> </center></div> </form> <b><p align="center">SAS Program: </b></p> <div align="center"><center><pre>data A; input a b c; cards; 1 1 12 1 0 7 0 1 0 0 0 10 ; proc freq; tables a*b/agree; weight c; run;</pre> </center></div><p align="center"><b> </p> <p align="center">SAS Output: </b></p> <div align="center"><center><pre> Simple Kappa Coefficient ------------------------ 95% Confidence Bounds Kappa = 0.542 ASE = 0.134 0.279 0.805 Sample Size = 29</pre> </center></div><p align="center"> </p> <b><p align="center">Reference: </b></p> <ol> <li><p align="left">Dunn G, Everitt B. Clinical Biostatistics: An Introduction to Evidence-Based Medicine. London: Edward Arnold 1995.</p> </li> <li><p align="left">Everitt BS. Statistical Methods for Medical Investigations, 2nd ed. London: Edward Arnold 1994.</p> </li> </ol> <hr align="center"> <p align="center">By <a href="mailto:jc641@columbia.edu">Jen-Hsiang Chuang</a><font face="Times New Roman">, M.D., M.S.<br> Department of Medical Informatics<br> Columbia University</font></p> <p align="center">Last modified: 10/22/99</p> </body>