Evaluating the Effectiveness of Generative AI Models as Tutoring Systems: Domain-Specific Approaches vs. ChatGPT-3.5

Abstract

Due to the variety of chatbot types and classifications, students and advisers may struggle to select a trusted and effective chatbot. Chatbot classification depends on various factors, including task complexity, response style, and domain specificity. To explore the most suitable option for high school advising, this study employed semi-structured interviews with eight high school students to assess their perspectives on seven generative responses from a domain-specific chatbot, HSGAdviser, in comparison with ChatGPT-3.5. The advising-related questions covered topics such as university applications, admission tests, and academic majors. Transcribed data were analyzed using thematic analysis. Findings indicate that most students preferred HSGAdviser for its ease, brevity, and speed, especially for Yes/No questions. However, when dealing with complex or high-impact decisions, some students favored ChatGPT-3.5 for its detailed responses. A key limitation of the study is the small sample size.