Defining Dimension Metrics for Evaluating Overall Prompting Effectiveness

The integration of Large Language Models (LLMs) into education, particularly in software engineering and IT, presents opportunities and challenges. While LLMs support problem-solving and code generation following the specifications, their effectiveness often depends on students’ ability to formulate precise and effective prompts. To address this, frameworks known as meta-prompts have been proposed. However, the impact of specific frameworks on students’ prompt-writing skills and learning outcomes remains underexplored. This study examines two didactic frameworks–Iterative Feedback and Reflection (IFR) and Adaptive Learning Progression (ALP)—to assess their effectiveness in enhancing prompt-writing skills and learning engagement. We propose complementary metrics within the Overall Prompting Effectiveness (OPE) framework, defined through three key dimensions: Adaptability, Relevance, and Efficiency. These dimensions encapsulate essential components for effective interaction with LLMs in educational contexts. The design of controlled experiment involves IT-engineering students divided into two groups, each using one of the two different didactical meta-prompt-enhanced frameworks. The IFR group engages in iterative cycles of prompt refinement and self-reflection, while the ALP group utilizes adaptive meta-prompts that dynamically adjust task complexity based on performance. Data collection focuses on OPE-aligned metrics, including the number of prompt iterations, time efficiency, response alignment, and learning progress self-rating, allowing for a comparative analysis of the frameworks’ impacts on learning outcomes. Our work establishes and evaluates these metrics, contributing to research in LLM-assisted learning. It addresses gaps in prompt engineering by showing how IFR and ALP frameworks can be utilized to enhance skill development and offers guidance on integrating LLMs into educational contexts for better interactive learning.