OSADE: Coaching Self-Regulation Skills
A mentorship application that helps mentors diagnose underlying self-regulatory skill gaps of students’ struggles and formulate tailored strategies. By addressing underlying emotional and metacognitive blockers traditional cognitive approaches often overlook, OSADE helps students overcome obstacles and achieve lasting improvement.
2022, Winter of my sophomore year
I joined the Design, Technology, and Research Program. Back then, OSADE was called:
Orchestration Scripting Environments (OSE)
originally conceived as a visual programming interface for authoring Orchestration Scripts (Garg et al., 2023) that automate situated coaching support for complex work.
Through needfinding, iterative design, and user testing, I gained a key insight that simple automated scripts were insufficient, and that effective coaching much address the “root cause” of students’ struggles—what mentees are struggling with, why they are struggling with it, and their ineffective ways of working.
OSE Interface, Winter 2022 (pretty monstrous…I know)
Built my first compiler…The blocked-based programming interface allowed mentors to author Orchestration Scripts (OS) in a visually-intuitive way. I wrote a compiler that turns the block-based script on the left into an executable OS, which gets executed by the OS engine and slackbots to automatically detect students’ ineffective practices and suggest strategies for common root causes associated with the practice.
While both the interface and our understanding of the problem space had a long way to go, the decision to start embedding root causes and strategies tailored to address them into the scripts started stirring us to a more profound approach that helps student become effective learners.
2023, Spring of my sophomore year
Further pursing this "root cause” analysis, the key breakthroughs we made this quarter were
1) Splitting the system into a scripting interface and a diagnosis interface. The scripting interface focuses on modeling expert mentor’s mental model of potential root causes related to a struggle, relevant context from work tools data that would aid diagnosing the root cause, and tailored strategy for each root cause. When work issues are detected, the diagnosis interface populates the pre-defined context with actual data from work tools and resurfaces the root cause model to quickly identify “prime suspect” causes.
2) Embedding the RootCause-Context-Strategy model in authored scripts and resurfacing the relevant model when a specific work issue has been detected to support diagnosis during in-person meetings.
The OSE Interface, Spring 2023 (still pretty monstrous…)
Deployment Study: To understand how well OSE supports mentoring interactions within a work ecosystem, we ran a 2-week deployment study in a research learning community where we ran scripts that mentors created to monitor for mentee struggles and asked the mentor to use pre-authored root cause model during in-person meetings to help diagnose and address the underlying cause.
Root Cause Study: During the deployment study, we felt that we lacked a concrete understanding of what makes a root cause analysis effective, which made it difficult to assess the helpfulness of our tool. To address this gap, we ran a study with two mentors in which we presented each mentor with a detector alert, context, and a list of root causes for a problem that we were facing. We observed how they approached eliciting information from students and how effective each was in uncovering the underlying issues leading up to this problem, resulting in a map of the diagnosis process:
2023, Fall of my Junior year
With a deeper understanding of root-cause diagnosis, we continued to design and improve the scripting and diagnosis interfaces. We renamed OSE to Orchestration Scripting and Diagnosis Environment (OSADE).
The Scripting Interface, Fall 2023
The Diagnosis Interface, Fall 2023
2023, Spring of my Junior year
OSADE underwent a major pivotal moment—when we learned through literature review and needfinding studies with mentors in Problem-Based Learning STEM courses.
What we threw into the bucket of “root causes” was, in fact, underlying gaps in students’ self-regulation skills that prevented them from achieving effective learning outcomes.
We learned that in project-based learning (PBL) environments across design, research, STEM, and entrepreneurship domains, students are often tasked with tackling complex, open-ended innovation challenges. Research shows that regulation skills are crucial for effective learning and innovation, as students must self-direct their efforts to assess their knowledge, focus on key areas, develop strategies, execute plans, seek help, self-reflect, collaborate, and adapt to challenges and uncertainties.
However, many students struggle with ineffective regulation practices and face difficulties in developing effective strategies on their own. To become successful innovators, students need experienced PBL coaches who can help resolve issues with their products and processes, and, more importantly, diagnose and address the underlying regulation gaps that impact their behavior. Effective coaching can target a range of regulation skills, including cognitive regulation (managing thinking), metacognitive regulation (monitoring and controlling learning), and emotional regulation (managing emotions such as frustration or anxiety).
By addressing regulation gaps, coaches can not only help students resolve current challenges but also equip them with new strategies, mental models, and attitudes that improve their ability to innovate in diverse situations. Without effective regulation skills, students may struggle to implement coach suggestions, and their regulation gaps may continue to hinder their progress . Despite the critical role regulation skills play in innovation, studies of university engineering design coaches, STEM research mentors, and entrepreneurship coaches reveal that coaches often struggle to effectively teach these skills, even with good intentions.
Running out of time towards the end of the quarter, we started exploring design directions for a mentor-in-the-loop support tool for regulation coaching with paper prototypes:
2024, Fall of my Senior year - Present
Struggling with the lack of clarity on obstacles mentors face in regulation coaching, we took a step back this quarter to study regulation coaching processes in PBL classrooms in more detail. We proposed a qualitative study to understand why mentors are often ineffective at coaching regulation skills, drafted research proposals to secure funding, and submitted IRB protocol for IRB approval, in preparation for conducting the study in Winter 2025.
Based on need-finding with 3 mentors in 3 STEM PBL classrooms and a pilot study with 3 students and 2 mentors, we proposed the Task, Practice, and Regulation Framework, which breaks down the PBL coaching process into three distinct levels: task-level, practice-level, and regulation-level. This structured framework allows for the systematic analysis of coaching practices at different stages of the learning process.
We learned that effective regulation coaching is a complex and time-consuming process that is not well-studied in existing literature. By proposing to observe and analyze multiple coaching sessions over the course of an academic term—and studying students’ regulatory processes through interviews and contextual inquiry—this study will provide in-depth insights into how PBL coaches (may struggle to) elicit, model, and facilitate the development of regulation skills and how students engage in regulation-processes as they self-direct innovation work.
By identifying and mapping the specific strategies PBL coaches use (and the areas where they may struggle), this study aims to provide actionable recommendations for improving the coaching process. This research is significant not only for PBL but also for broader educational practices for teaching self-regulation. The findings could be applied to a range of contexts where students need to manage complex, uncertain tasks and improve their ability to self-monitor and adapt strategies. The insights into effective coaching methods could thus influence how educators approach skill-building in self-regulation across disciplines.