Applied Psychological Measurement

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Click here to sign up for SAGE Journal Email Alerts today!

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Huiqin Hu,
Right arrow Articles by Vukmirovic, Z.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
Applied Psychological Measurement, Vol. 32, No. 4, 311-333 (2008)
DOI: 10.1177/0146621606292215

Investigation of IRT-Based Equating Methods in the Presence of Outlier Common Items

Huiqin Hu

Data Recognition Corporation, hhu{at}datarecognitioncorp.com

W. Todd Rogers

University of Alberta, Canada

Zarko Vukmirovic

Harcourt Assessment, Inc.

Common items with inconsistent b-parameter estimates may have a serious impact on item response theory (IRT)—based equating results. To find a better way to deal with the outlier common items with inconsistent b-parameters, the current study investigated the comparability of 10 variations of four IRT-based equating methods (i.e., concurrent calibration, separate calibration with test characteristic curve [TCC] and mean/sigma [M/S] transformations, and calibration with fixed common item parameters [FCIP]) when outliers were either ignored or considered. Simulated data were generated for the common-item nonequivalent groups matrix design to reflect the manipulated factors: group ability differences and nonequivalent groups, number/score points of outliers, and types of outliers. When no outliers were present, the TCC and M/S transformations performed the best. When there were outliers, overall, the methods that considered them (except the M/S transformation with outliers weighted) resulted in a vast improvement compared to the methods that ignored them.

Key Words: item response theory • equating • outliers • calibration • transformation.

References

  • Angoff, W.H. (1984). Scales, norms, and equivalent scores. Princeton, NJ: Educational Testing Service.
  • Baker, F.B. (1996). An investigation of the sampling distributions of equating coefficients. Applied Psychological Measurement, 20(1), 45-57.[Abstract]
  • Bejar, I., & Wingersky, M.S. (1981). An application of item response theory to equating the Test of Standard Written English (College Board Report No. 81-8). Princeton, NJ: Educational Testing Service. (ETS No. 81-35)
  • Bolt, D.M. (1999). Evaluating the effects of multidimensionality on IRT true-score equating. Applied Measurement in Education, 12(3), 383-407.[CrossRef][ISI]
  • Childs, R.A., & Chen, W.-H. (1999). Obtaining comparable item parameter estimates in MULTILOG and PARSCALE for two polytomous IRT models. Applied Psychological Measurement, 23(4), 371-379.[Abstract/Free Full Text]
  • Cohen, A.S., & Kim, S.-H. (1998). An investigation of linking methods under the graded response model. Applied Psychological Measurement, 22(2), 116-130.[Abstract/Free Full Text]
  • Cook, L.L., Eignor, D.R., & Hutton, L.R. (1979, April). Considerations in the application of latent trait theory to objective-based criterion-referenced tests. Paper presented at the meeting of the American Educational Research Association, San Francisco.
  • Cook, L.L., & Petersen, N.S. (1987). Problems related to the use of conventional and item response theory equating methods in less than optimal circumstances. Applied Psychological Measurement, 11, 225-244.[Abstract]
  • De Champlain, A.F. (1996). The effect of multidimensionality on IRT true-score equating for subgroups of examinees. Journal of Educational Measurement, 33(2), 181-201.[CrossRef][ISI]
  • Dorans, N.J., & Kingston, N.M. (1985). The effects of violations of unidimensionality on the estimation of item and ability parameters and on item response theory equating of the GRE verbal scale. Journal of Educational Measurement, 22(4), 249-262.[CrossRef][ISI]
  • Gifford, J.A., & Swaminathan, H. (1990). Bias and the effect of priors in Bayesian estimation of parameters of item response models. Applied Psychological Measurement, 14, 33-43.[Abstract]
  • Haebara, T. (1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22, 144-149.
  • Hambleton, R.K., & Murray, L. (1983). Some goodness of fit investigations for item response models. In R. K. Hambleton (Ed.), Applications of item response theory (pp. 71-94). British Columbia: Educational Research Institute of British Columbia.
  • Hanson, B.A., & Beguin, A.A. (2002). Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26(1), 3-24.[Abstract/Free Full Text]
  • Hanson, B.A., & Feinstein, Z.S. (1997). Application of a polynomial log linear model to assessing differential item functioning for common items in the common-item equating design (ACT Research Report Series 97-1). Iowa City, IA: American College Testing.
  • Harris, D.J. (1991, April). Equating with nonrepresentative common item sets and non-equivalent groups. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
  • Harwell, M., Stone, C.A., Hsu, T.-C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101-125.[Abstract/Free Full Text]
  • Hills, J.R., Subhiyah, R.G., & Hirsch, T.M. (1988). Equating minimum-competency tests: Comparison of methods. Journal of Educational Measurement, 25(3), 221-231.[CrossRef][ISI]
  • Ironson, G.H. (1983). Using item response theory to measure bias. In R. K. Hambleton (Ed.), Applications of item response theory (pp. 155-174). British Columbia, Canada: Educational Research Institute of British Columbia.
  • Klein, L.W., & Jarjoura, D. (1985). The importance of content representation for common-item equating with non-random groups. Journal of Educational Measurement, 22, 197-206.[CrossRef][ISI]
  • Kolen, M.J., & Brennan, R.L. (2004). Test equating, scaling, and linking: Methods and practices. New York: Springer.
  • Kromrey, J.D., Parshall, C.G., & Yi, Q. (1998, April). The effects of content representativeness and differential weighting on test equating: A Monte Carlo study. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
  • Lee, G., Kolen, M.J., Frisbie, D.A., & Ankenmann, R.D. (2001). Comparison of dichotomous and polytomous item response models in equating scores from tests composed of testlets. Applied Psychological Measurement, 25(4), 357-372.[Abstract/Free Full Text]
  • Lehman, R.S., & Bailey, D.E. (1968). Digital computing: Fortran IV and its applications in behavioural science. New York: John Wiley.
  • Li, Y.H., Lissitz, R.W., & Yang, Y.-N. (1999, April). Estimating IRT equating coefficients for tests with polytomously and dichotomously scored items. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
  • Linn, R.L., Levine, M.V., Hastings, C.N., & War-drop, J.L. (1980). An investigation of item bias in a test of reading comprehension (Tech. Rep. No. 163). Urbana: Center for the Study of Reading, University of Illinois.
  • Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
  • Loyd, B.H., & Hoover, H.D. (1980). Vertical equating using the Rasch model. Journal of Educational Measurement, 17, 179-193.[CrossRef][ISI]
  • Marco, G.L. (1977). Item characteristic curve solutions to three intractable testing problems. Journal of Educational Measurement, 14, 139-160.[CrossRef][ISI]
  • Minnesota Comprehensive Assessments Grade 3 & 5 Technical Manual. (2002). Retrieved November 26, 2005, from http://education.state.mn.us/mde/static/001879.pdf
  • Muraki, E., & Bock, R.D. (1999). PARSCALE: IRT Item Analysis and Test Scoring for Rating-scale Data (Version 3.5) [Computer software]. Chicago: Scentific Software.
  • Petersen, N.C., Marco, G.L., & Stewart, E.E. (1982). A test of the adequacy of linear score equating models. In P. W. Holland & D. B. Rubin (Eds.), Test equating (pp. 71-135). New York: Academic Press.
  • Psychometric Society. (1979). Publication policy regarding Monte Carlo studies. Psychometrika, 44, 133-134.[CrossRef]
  • Stocking, M.L., & Lord, F.M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.[Abstract]
  • Tate, R. (2000). Performance of a proposed method for the linking of mixed format tests with constructed response and multiple choice items. Journal of Educational Measurement, 37(4), 329-346.[CrossRef]
  • Thissen, D. (1991). MULTILOG user's guide: Multiple, categorical item analysis and test scoring using item response theory (Version 6.0). New York: Springer.
  • 2001 MCAS technical report. (2001). Retrieved November 26, 2005, from http://www.doe.mass.edu/mcas/2002/news/01techrpt.pdf
  • 2002 MCAS technical report. (2002). Retrieved November 26, 2005, from http://www.doe.mass.edu/mcas/2003/news/02techrpt.pdf
  • Vukmirovic, Z., Hu, H., & Turner, J.C. (2003, April). The effects of outliers on IRT equating with fixed common item parameters. Paper presented at the meeting of the National Council on Measurement in Education, Chicago.
  • Wang, T.-Y., Hanson, B.A., & Harris, D.J. (2000). The effectiveness of circular equating as a criterion for evaluating equating. Applied Psychological Measurement, 24(3), 195-210.[Abstract/Free Full Text]
  • Yang, W. (1997, April). The effects of content mix and equating method on the accuracy of test equating using anchor-item design. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
  • Yen, W.M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125-145.[Abstract]
  • Zeng, L. (1991). Standard errors of linear equating for the single-group design (ACT Research Report 91-4). Iowa City, IA: American College Testing.
  • Zenisky, A.L. (2001, October). Investigating the accumulation of equating error in fixed common item parameter linking: A simulation study. Paper presented at the annual meeting of the Northeastern Educational Research Association, Kerhonkson, NY.
  • Zimowski, M.F., Muraki, E., Mislevy, R.J., & Bock, R.D. (1996). BILOG-MG: Multiple group IRT analysis and test maintenance for binary items [Computer program]. Chicago: Scientific Software International.

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Huiqin Hu,
Right arrow Articles by Vukmirovic, Z.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?