Development and Reliability-Validity Testing of a Knowledge, Attitude, and Behavior (KAB)-Based Scale for University Students' Academic Use of AI Tools

Yangli Ou; Nancy A. Aguila; Lulu Xiao; Zhiyuan Zhang; Fangzhu Ouyang

doi:10.62051/ijphmr.v3n3.02

Authors

Yangli Ou
Nancy A. Aguila
Lulu Xiao
Zhiyuan Zhang
Fangzhu Ouyang

DOI:

https://doi.org/10.62051/ijphmr.v3n3.02

Keywords:

College students, Generative AI tools, Academic misconduct, KAB theory

Abstract

To develop and validate a scale for assessing the knowledge and behaviors related to academic use norms of AI tools among university students, focusing on reliability and validity. Grounded in the Knowledge-Attitude-Behavior (KAB) theory, a pool of scale items was constructed through a review of the literature, semi-structured interviews, and Delphi expert consultations. A pilot test was conducted with 140 undergraduate nursing students from a certain university in China, employing item analysis and exploratory factor analysis to refine the scale items. A formal assessment was then conducted with 220 nursing undergraduates who had used generative AI tools for at least three months. Reliability analysis and confirmatory factor analysis were performed using SPSS and AMOS. The scale consists of 2 subscales, 3 dimensions, and 14 items. Exploratory factor analysis extracted three common factors, with a cumulative variance explanation of 70.878%. The Cronbach's α coefficients for the subscales were 0.925 and 0.855, respectively, while the split-half reliability coefficients were 0.891 and 0.860. The fit indices for the confirmatory factor model were as follows: χ2/df=2.978, GFI=0.859, PGFI=0.606, RMSEA=0.098, RMR=0.048, CFI=0.920, and PCFI=0.748. The scale demonstrates strong reliability and validity, making it suitable for assessing university students' knowledge and behaviors regarding the academic use of AI tools.

References

[1] Tertiary Education Quality and Standards Agency, (2022). What is academic integrity? https://www.teqsa.gov.au/students/understanding-academic-integrity/what-academic-integrity

[2] Jereb, E., Perc, M., Lämmlein, B., Jerebic, J., Urh, M., Podbregar, I., & Šprajc, P. (2018). Factors influencing plagiarism in higher education: A comparison of German and Slovene students. PLOS ONE, 13(8), e0202252. https://doi.org/10.1371/journal.pone.0202252

[3] Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., & Smith, D. M. (2023). Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, 183(6), 589. https://doi.org/10.1001/jamainternmed.2023.1838

[4] Liu, H., Azam, M., Bin Naeem, S., & Faiola, A. (2023). An overview of the capabilities of ChatGPT for medical writing and its implications for academic integrity. Health Information & Libraries Journal, 40(4), 440–446. https://doi.org/10.1111/hir.12509

[5] Chris Westfall (2023). Educators Battle Plagiarism As 89% Of Students Admit To Using OpenAI’s ChatGPT For Homework [N/OL]. Forbes. https://www.forbes.com/sites/chriswestfall/2023/01/28/educators-battle-plagiarism-as-89-of-students-admit-to-using-open-ais-chatgpt-for-homework/?sh=12e01a42750d

[6] Sun, G. H., & Hoelscher, S. H. (2023). The ChatGPT Storm and What Faculty Can Do. Nurse Educator, 48(3), 119–124. https://doi.org/10.1097/NNE.0000000000001390

[7] Ministry of Foreign Affairs (2022). China's Position Paper on Strengthening Ethical Governance of Artificial Intelligence. https://www.mfa.gov.cn/web/ziliao_674904/zcwj_674915/202211/t20221117_10976728.shtml

[8] Yang, J.F., Chu, J., Zhang B.X.(2024). Research on Constructing Ethical Norm Indicators for the Application of Artificial Intelligence in Education [J]. E-education Research. DOI:10.13811/j.cnki.eer.2024.10.003

[9] Hong, W. C. H., Chi, C., Liu, J., Zhang, Y., Lei, V. N.-L., & Xu, X. (2023). The influence of social education level on cybersecurity awareness and behaviour: A comparative study of university students and working graduates. Education and Information Technologies, 28(1), 439–470. https://doi.org/10.1007/s10639-022-11121-5

[10] Fresán, A., Domínguez-Martínez, T., Castilla-Peón, M. F., Robles, R., Hernández, O., Vélez, T., & Muñoz, C. (2022). Confirmatory Factor Analysis of the Transgender Knowledge, Attitudes, and Beliefs (T-KAB) Scale for the Mexican Population. Archives of Sexual Behavior, 51(4), 1959–1966. https://doi.org/10.1007/s10508-021-02259-3

[11] Chory, A., Nyandiko, W., Beigon, W., Aluoch, J., Ashimosi, C., Munyoro, D., Scanlon, M., Apondi, E., & Vreeman, R. (2021). Perspectives of education sector stakeholders on a teacher training module to reduce HIV/AIDS stigma in Western Kenya. BMC Public Health, 21(1), 1281. https://doi.org/10.1186/s12889-021-11331-5

[12] Chen Xiangming (1999). Classification and Analysis of Data [J]. Social Science Front, 223-229.

[13] Wu, M.L. (2010). Practical Statistical Analysis of Questionnaires: SPSS Operations and Applications [M]. Chongqing: Chongqing University Press.

[14] Nachalon, Y., Broer, M., & Nativ-Zeltzer, N. (2024). Using ChatGPT to Generate Research Ideas in Dysphagia: A Pilot Study. Dysphagia, 39(3), 407–411. https://doi.org/10.1007/s00455-023-10623-9

[15] Nanzhong, W., Xianzhang, C., Yong, F.(2023). From “Disordered” to “Ordered”: The Transformation and Its Generation Mechanism of Generative Artificial Intelligence Education Application.Journal of Distance Education, 6:42-51. https://doi.org/10.15881/j.cnki.cn33-1304/g4.2023.06.005