How Well Do LLMs Understand Indonesian Textbooks? An Evaluation on Informatics Question Answering

Abstract

The emergence of Large Language Models (LLMs) presents significant opportunities for developing automatic question answering (QA) systems in education. This study compares the text generation quality of GPT-4o-mini and DeepSeek-v4-flash on context-grounded QA tasks using official Informatics textbooks for Senior High School (SMA/MA) grades X, XI, and XII sourced from Indonesia's National Textbook System (SISPBK). A dataset of 1,760 context-question-answer triples was constructed, covering six major topic areas aligned with the Merdeka Curriculum. Both models were evaluated under identical zero-shot prompting conditions without any task-specific fine-tuning. Automated evaluation was conducted using an LLM-as-a-Judge framework with DeepSeek as the evaluator, assessing three quality dimensions: answerability (contextual grounding), fluency (linguistic quality), and relevance (topical alignment) on a 1-5 Likert scale. Results show that both models achieved the highest scores in the fluency dimension, with GPT-4o-mini scoring between 4.60 and 5.00 and DeepSeek-v4-flash scoring between 4.00 and 5.00 across all grade levels, demonstrating strong capability in producing grammatically correct and natural Bahasa Indonesia. GPT-4o-mini outperformed in answerability, while DeepSeek-v4-flash showed a slight advantage in relevance. However, consistently low answerability scores below 3.00 for both models reveal limitations in meeting the strict requirements of closed-book, curriculum-based QA settings. These findings highlight the need for improved prompt design and domain-specific fine-tuning to enhance LLM effectiveness in Indonesian secondary education.

Keywords: Informatics, Large Language Model, LLM-as-a-Judge, Merdeka Curriculum, Question Answering