Gerçek ve Üretilmiş Veri Setlerinde Çok Boyutlu Bireyselleştirilmiş Bilgisayarlı Test Uygulama Sonuçlarının İncelenmesi

Abstract

This study aims to develop a multi-dimensional computerized adaptive test (MCAT) that measures grammar and vocabulary knowledge as well as general ability of the students attending Anadolu University School of Foreign Languages by using the data from Proficiency Exams administered for these student at the end of preparatory school education. To achieve this purpose, an item pool was created by using four different data sets each of which consists of 50 questions used in the previous proficiency exams administered at Anadolu Uniersity School of Foreign Languages. According to IRT-based analyses applied to the data, it was found that three-dimensional bifactor model was the most appropriate model for all item sets. Following the development of the item pool, a hybrid simulation was applied –by taking the missing matrix into consideration – in order to determine the algorithm to be used in real-time MCAT application. In this application, a total of 36 conditions were created in which different ability estimation methods (EAP and MAP), different item selection methods (D-rule, KL, W-rule, T-rule, weighted W-rule and weighted T-rule) and different termination rules (standard error, θ convergence, fixed number of items) were used. In these conditions, the correlation between estimated θ value and real value for each dimension; bias, RSMD and standard error values were obtained. Being an important indicator of measurement accuracy in CAT applications in addition to correlation values and error statistic, the number of items replied was also reported under the variable-length simulation conditions. According to the results obtained, it was found that D-rule item selection method and MAP ability prediction method give the best statistics for each different termination rule. Following the comparisons of best conditions for each termination rule, it was decided that the termination rules based on standard error as well as D-rule item selection method and MAP ability prediction method should be used for real-time MCAT application. By using this algorithm, a correlation was obtained between general ability levels obtained from the real-time application and the scores 32 students out of 99 received from paper-pencil test, who were applied real-time application. In addition, the total number of items administered to each individual and how frequently the items in item pool are used were reported based on the results obtained from the application carried out for 99 students. According to the results obtained from real-time application, it was found that 30% of the item pool, which includes a total of 200 items were used. It was also observed that the average number of items replied was 12.3 and nearly 75% of the participants replied 10 to 12 items.