Development of AI-Assisted STEM Learning Tools to Improve Students' Scientific Literacy and Problem-Solving Skills

Faiq Makhdum Noor; Monera A. Salic-Hairulla

doi:10.37630/bijee.v4i1.4447

Authors

Faiq Makhdum Noor Institut Agama Islam Negeri Kudus
Monera A. Salic-Hairulla Mindanao State University-Iligan Institure of Technology

DOI:

https://doi.org/10.37630/bijee.v4i1.4447

Keywords:

Artificial Intelligence, STEM, Scientific Literacy, Problem-Solving Skills

Abstract

Indonesian students continue to perform below the OECD average in scientific literacy and problem-solving on PISA 2022, and existing classroom interventions rarely combine STEM integration with Artificial Intelligence (AI) scaffolds within a single coherent design. This study developed, validated, and tested the effectiveness of an AI-assisted STEM learning tool (STEMI-AI) for improving the scientific literacy and problem-solving skills of seventh-grade students at a junior secondary madrasah in Kudus Regency, Central Java. The Plomp development model was applied across three iterative phases: preliminary research, prototyping, and assessment. Participants were 32 students of class VIIA (experimental group) and 32 students from a parallel class (control group), purposively sampled. Data were collected through expert validation sheets, teacher and student practicality questionnaires, a scientific literacy test adapted from the PISA framework, and a problem-solving test based on Polya's four-step rubric. Data were analyzed using V Aiken for content validity, percentage analysis for practicality, N-gain and analysis of covariance for effectiveness, with Cohen's d as the effect-size index. Expert validation produced a mean V Aiken of 0.91 (highly valid), practicality reached 88 percent (teachers) and 84 percent (students), and the experimental class showed significantly greater gains than the control class on both scientific literacy (N-gain = 0.58, medium) and problem-solving (N-gain = 0.54, medium), with Cohen's d of 1.12 and 1.05 respectively. The integrated AI-STEM design is feasible and effective for junior secondary madrasah classrooms when AI components are embedded as pedagogical scaffolds rather than as standalone aids.

References

Akgun, S., & Greenhow, C. (2021). Artificial intelligence in education: Addressing ethical challenges in K-12 settings. AI and Ethics, 2(3), 431–440. https://doi.org/10.1007/s43681-021-00096-7

Akker, J. v. d., Bannan, B., Kelly, A. E., Nieveen, N., & Plomp, T. (Eds.). (2013). Educational design research: Part A: An introduction. SLO Netherlands Institute for Curriculum Development.

Antonopoulou, H., Halkiopoulos, C., & Gkintoni, E. (2023). Educational neuroscience and its contribution to math learning. Technium Education and Humanities, 4. https://doi.org/10.47577/teh.v4i.8237

Azzahra, W., Diana, S., & Nuraeni, E. (2024). Unraveling the evolution of brain-based learning in Indonesia: An in-depth exploration through systematic literature review. International Journal of Educational Reform, 33(4), 483–502. https://doi.org/10.1177/10567879241258134

Bang, H. J., Li, L., & Flynn, K. (2023). Efficacy of an adaptive game-based math learning app to support personalized learning and improve early elementary school students' learning. Early Childhood Education Journal, 51(4), 717–732. https://doi.org/10.1007/s10643-022-01332-3

Bruner, J. S. (1966). Toward a theory of instruction. Harvard University Press.

Chiu, T. K. F. (2024). Future research recommendations for transforming higher education with generative AI. Computers and Education: Artificial Intelligence, 6, 100197. https://doi.org/10.1016/j.caeai.2023.100197

Cichy, I., Kaczmarczyk, M., Wawrzyniak, S., Kruszwicka, A., Przybyla, T., Klichowski, M., & Rokita, A. (2020). Participating in physical classes using Eduball stimulates acquisition of mathematical knowledge and skills by primary school students. Frontiers in Psychology, 11, 2194. https://doi.org/10.3389/fpsyg.2020.02194

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Coppi, M., Fialho, I., & Cid, M. (2023). Developing a scientific literacy assessment instrument for Portuguese 3rd cycle students. Education Sciences, 13(9), 941. https://doi.org/10.3390/educsci13090941

Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024). Impact of AI assistance on student agency. Computers & Education, 210, 104967. https://doi.org/10.1016/j.compedu.2023.104967

Debets, T., Banihashem, S. K., Joosten-Ten Brinke, D., Vos, T. E., de Buy Wenniger, G. M., & Camp, G. (2025). Chatbots in education: A systematic review of objectives, underlying technology and theory, evaluation criteria, and impacts. Computers & Education, 234, 105323. https://doi.org/10.1016/j.compedu.2025.105323

Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2024). Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers & Education, 227, 105224. https://doi.org/10.1016/j.compedu.2024.105224

Dwiputra, D. F. K., Azzahra, W., & Heryanto, F. N. (2023). A systematic literature review on enhancing the success of independent curriculum through brain-based learning innovation implementation. Indonesian Journal on Learning and Advanced Education, 5(3), 262–276. https://doi.org/10.23917/ijolae.v5i3.22318

Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146

Goldin, G. A., & Shteingold, N. (2001). Systems of representations and the development of mathematical concepts. In A. A. Cuoco & F. R. Curcio (Eds.), The roles of representation in school mathematics (pp. 1–23). National Council of Teachers of Mathematics.

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64–74. https://doi.org/10.1119/1.18809

Hartoyo, S. R., Nahdi, D. S., & Cahyaningsih, U. (2025). Pengaruh pendekatan realistic mathematics education (RME) berbantuan media ARCA terhadap pemahaman konsep siswa sekolah dasar. STRATEGY: Jurnal Inovasi Strategi dan Model Pembelajaran, 5(3), 348–356. https://doi.org/10.51878/strategi.v5i3.6843

Hidayat, R., & Wardat, Y. (2023). A systematic review of augmented reality in science, technology, engineering and mathematics education. Education and Information Technologies, 28, 11521–11556. https://doi.org/10.1007/s10639-023-12157-x

Howard-Jones, P. A. (2014). Neuroscience and education: Myths and messages. Nature Reviews Neuroscience, 15(12), 817–824. https://doi.org/10.1038/nrn3817

Issholikhah, L. N., Oktradiksa, A., & Shalikhah, N. D. (2024). Realistic mathematics education (RME) model to improve mathematics learning outcomes for MI Muhammadiyah Sriwedari students, Magelang Regency. In Proceedings of the 5th Borobudur International Symposium on Humanities and Social Science 2023 (pp. 793–801). Atlantis Press. https://doi.org/10.2991/978-2-38476-273-6_83

Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi. (2022). Kurikulum Merdeka [Independent curriculum]. Government of Indonesia.

Krause, E., Iordan, A. D., Polizzi, A., & Patriotis, D. (2025). The impact of Artificial Intelligence (AI) on students' academic development. Education Sciences, 15(3), 343. https://doi.org/10.3390/educsci15030343

Kurniawan, E. S., Mundilarto, M., & Istiyono, E. (2024). Improving student higher order thinking skills using Synectic-HOTS-oriented learning model. International Journal of Evaluation and Research in Education, 13(2), 1132–1140. https://doi.org/10.11591/ijere.v13i2.25002

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863

Liu, D., Tan, X., Yan, H., & Li, W. (2024). Improving mental arithmetic ability of primary school students with schema teaching method: An experimental study. PLOS ONE, 19(4), e0297013. https://doi.org/10.1371/journal.pone.0297013

Liu, M., Ren, Y., Nyagoga, L. M., Stonier, F., Wu, Z., & Yu, L. (2025). The impact of ChatGPT on students' academic achievement: A meta-analysis. Journal of Computer Assisted Learning, 41, e70096. https://doi.org/10.1111/jcal.70096

McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.

Mukuka, A., & Alex, J. K. (2024). Review of research on microteaching in mathematics teacher education: Promises and challenges. Eurasia Journal of Mathematics, Science and Technology Education, 20(1), em2381. https://doi.org/10.29333/ejmste/13941

Ng, D. T. K., Tan, C. W., & Leung, J. K. L. (2024). Empowering student self-regulated learning and science education through ChatGPT: A pioneering pilot study. British Journal of Educational Technology, 55(4), 1328–1353. https://doi.org/10.1111/bjet.13454

Niss, M., & Højgaard, T. (2019). Mathematical competencies revisited. Educational Studies in Mathematics, 102(1), 9–28. https://doi.org/10.1007/s10649-019-09903-9

Nur, S., & Ikhsan, J. (2024). Implementation of STEM integrated problem based learning model to improve problem solving skills and learning motivation of grade X vocational high school students on the material of substances and their changes. Jurnal Penelitian Pendidikan IPA, 10(11), 8882–8891. https://doi.org/10.29303/jppipa.v10i11.9121

Organisation for Economic Co-operation and Development. (2023). PISA 2022 results (Volume I): The state of learning and equity in education. OECD Publishing. https://doi.org/10.1787/53f23881-en

Plomp, T., & Nieveen, N. (Eds.). (2013). Educational design research. SLO Netherlands Institute for Curriculum Development.

Polya, G. (1957). How to solve it: A new aspect of mathematical method (2nd ed.). Princeton University Press.

Risdiyanti, I., Zulkardi, Z., Putri, R. I. I., Prahmana, R. C. I., & Nusantara, D. S. (2024). Ratio and proportion through realistic mathematics education and pendidikan matematika realistik Indonesia approach: A systematic literature review. Jurnal Elemen, 10(1). https://doi.org/10.29408/jel.v10i1.24445

Roediger, H. L., III, & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x

Rohana, S., Irianto, A., & Rachmadtullah, R. (2023). Project-based learning model on critical thinking ability seen from cognitive style in elementary schools. Studies in Learning and Teaching, 4(2), 245–259.

Rojas, M., Nussbaum, M., Chiuminatto, P., Guerrero, O., Greiff, S., Krieger, F., & Van Der Westhuizen, L. (2021). Assessing collaborative problem-solving skills among elementary school students. Computers & Education, 175, 104313. https://doi.org/10.1016/j.compedu.2021.104313

Säfström, A. I., Lithner, J., Palm, T., Palmberg, B., Sidenvall, J., Andersson, C., Boström, E., & Granberg, C. (2024). Developing a diagnostic framework for primary and secondary students' reasoning difficulties during mathematical problem solving. Educational Studies in Mathematics, 115(1), 125–149. https://doi.org/10.1007/s10649-023-10278-1

Samritin, S., Natsir, S. R., Manaf, A., & Sari, E. R. (2023). The effect of realistic mathematics education implementation in mathematics learning in elementary school. Formatif: Jurnal Ilmiah Pendidikan MIPA, 13(1), 81–88. https://doi.org/10.30998/formatif.v13i1.16522

Schoenfeld, A. H. (1985). Mathematical problem solving. Academic Press.

Sembiring, R. K., Hadi, S., & Dolk, M. (2008). Reforming mathematics learning in Indonesian classrooms through RME. ZDM, 40, 927–939. https://doi.org/10.1007/s11858-008-0125-9

Suanto, E., Maat, S. M., & Zakaria, E. (2023). The effectiveness of the implementation of three dimensions geometry KARA module on higher order thinking skills (HOTS) and motivation. International Journal of Instruction, 16(3), 95–116. https://doi.org/10.29333/iji.2023.1636a

Suharja, S., Mustadi, A., & Oktari, V. (2024). Examining brain based learning models assisted open-ended approach to mathematics understanding concept. Jurnal Prima Edukasia, 12(1), 19–29. https://doi.org/10.21831/jpe.v12i1.67303

Susanti, E. (2025). Enhancing problem-solving skills in elementary students through realistic mathematics education. SCIENCE: Jurnal Inovasi Pendidikan Matematika dan IPA, 5(1), 48. https://doi.org/10.51878/science.v5i1.4344

Vuong, Q. A., Bui, D. T., Dang, H. T. T., Le, A. V., & Do, D. L. (2025). Teachers' perspectives on the application of technology in mathematics education in primary schools: A dataset from Vietnam. Data in Brief, 60, 111473. https://doi.org/10.1016/j.dib.2025.111473

Wahdaniyah, N., Agustini, R., & Tukiran, T. (2023). Analysis of effectiveness PBL-STEM to improve student's critical thinking skills. IJORER: International Journal of Recent Educational Research, 4(3), 365–382. https://doi.org/10.46245/ijorer.v4i3.312

Wahid, S. H. (2024). Exploring the intersection of Islam and digital technology: A bibliometric analysis. Social Sciences & Humanities Open, 10, 101085. https://doi.org/10.1016/j.ssaho.2024.101085

Wang, J., & Fan, W. (2025). The effect of ChatGPT on students' learning performance, learning perception, and higher-order thinking: Insights from a meta-analysis. Humanities and Social Sciences Communications, 12, 621. https://doi.org/10.1057/s41599-025-04787-y

Wasehudin, W., Hidayat, T., Hadijah, S., Hairullah, H., & Kosim, A. (2024). Artificial Intelligence's impact on the development of Islamic Religious Education learning at a public junior high school of Cilegon, Indonesia. Hanifiya: Jurnal Studi Agama-Agama, 7(2), 193–198.

Wen, T., He, J., & Yang, Y. (2023). Development and validation of an instrument for assessing scientific literacy from junior to senior high school. Disciplinary and Interdisciplinary Science Education Research, 5, 21. https://doi.org/10.1186/s43031-023-00093-2

Zhao, X., Feng, G. C., Ao, S. H., & Liu, P. L. (2022). Interrater reliability estimators tested against true interrater reliabilities. BMC Medical Research Methodology, 22, 232. https://doi.org/10.1186/s12874-022-01707-5

Zuhriyeh, S., Ali, M., & Hidayat, A. (2025). Digital transformation of Islamic education: An artificial intelligence-based teaching module development study. Edunesia: Jurnal Ilmiah Pendidikan, 6(2), 1113–1126. https://doi.org/10.51276/edu.v6i2.1255