Case Study - Recruiting Students into Recommended Courses
Overview
Project Background
Company Overview
Abl believes every student deserves the opportunity to maximize their potential by making informed and meaningful choices about their lives and futures. Unfortunately, many students aren’t given the same opportunities due to systemic inequities perpetuated by outdated, unresponsive systems. Abl offers advanced, data-driven software and services to help K12 school and district leaders identify and remove those barriers to equity. The Abl team is made up of education leaders and experts working together with engineers, designers, and data scientists to build the tools schools districts need to shape pathways for each student to achieve success.
Testing Focused Approach
The design work for this project was strategically centered around testing, aligning with both the long-term vision and immediate needs of the company.
Forward-Thinking Alignment
With a multi-year vision for a sophisticated course recommendation algorithm, we adopted a forward-thinking approach by designing for anticipated capabilities a year from now, delivering incremental value at each milestone.
Iterative Value Delivery
Acknowledging the dynamic project timeline, our focus was on delivering tangible value incrementally. Rather than waiting for the algorithm's completion, we prioritized solutions for the smallest practical milestone, ensuring continuous progress.
Testing Unknowns
Embracing uncertainties in algorithm development, testing served as our compass to navigate unknowns and make informed decisions.
-
Quality Assessment: Rigorous testing evaluated the accuracy and effectiveness of the algorithm-generated course recommendations, enabling iterative improvements.
-
Full Problem Space Understanding: Testing extended beyond the algorithm to explore the full problem space, identifying key pain points hindering year-round value delivery.
-
Full Solution Space Exploration: Testing various approaches allowed us to seamlessly integrate recommendations into the product, addressing identified pain points and promoting equitable outcomes for enrolled students.
​
In essence, our testing-focused approach was a strategic imperative, guiding us through uncertainties, addressing unknowns, and paving the way for a future where intelligent course recommendations significantly contribute to educational success.
Project Goals and Objectives
Goal 1: Define Future Vision and Provide Immediate Solutions
Objective:
To anticipate the future capabilities of the course recommendation algorithm and align them with the immediate needs of K12 school and district leaders.
Key Deliverables:
-
Develop a multi-year vision for a sophisticated course recommendation algorithm that aligns with Abl's mission of promoting equity and maximizing student potential.
-
Identify incremental milestones to deliver tangible value, ensuring continuous progress and alignment with evolving needs.
-
Integrate recommendations seamlessly into Abl Analytics, making it a valuable year-round tool for school and district leaders to drive positive outcomes.
-
Ensure actionable insights by providing clear recommendations for students' academic pathways, empowering counselors and school leaders to make informed decisions.
Goal 2: Ensure High-Quality Recommendations and Minimize Risks
Objective:
To ensure the accuracy, relevance, and safety of algorithm-generated course recommendations while minimizing potential risks.
Key Deliverables:
-
Develop algorithms that provide accurate and relevant course recommendations based on comprehensive data analysis and machine learning.
-
Enhance data quality through improvements in Abl's Manage Data app, ensuring the algorithm learns from accurate and complete datasets.
-
Implement safeguards to minimize the risk of harmful recommendations, such as ensuring recommendations align with graduation requirements and student goals.
-
Continuously monitor and evaluate recommendation quality to increase user confidence and reduce the risk of adverse outcomes.
By focusing on these goals and objectives, we aim to empower K12 school and district leaders with advanced, data-driven tools that promote equity, maximize student potential, and drive positive educational outcomes.
Role
As the sole designer on the project, I played a central role in:
-
Leading the design of the Manage Data App for seamless data ingestion.
-
Utilizing my expertise in School District data to bridge design and technical teams.
-
Introducing interactive prototyping for efficient user testing and rapid iteration.
-
Collaborating closely with Product Managers, Data Scientists, and Engineers to align design with project goals and technical requirements, ensuring user-centric solutions.
Research and Discovery
Our user research approach was meticulously crafted to gather insights aimed at addressing the unique challenges faced by educational institutions. We focused on testing scenarios crucial for keeping students on-track for their desired outcomes and acting on historic and current student data analysis.
Customer Interviews for Pain Point Identification
To initiate the user research phase, we conducted comprehensive interviews with a diverse range of stakeholders, including school leaders, guidance counselors, and students. Our primary goal was to unearth the foremost pain points and challenges experienced in guiding students towards their academic goals. These interviews provided invaluable qualitative data, offering a nuanced comprehension of the intricacies surrounding course selection and assessment.
Key Insights:
-
Emphasis on actionable insights alongside analytical data.
-
Strong demand for consistent, impactful interventions throughout the academic year.
Design Partnership with a School District
Building on insights from customer interviews, we established a design partnership with a school district. This collaborative effort involved working closely with educators, administrators, and stakeholders to co-create and refine design concepts. The partnership facilitated a deeper understanding of the district's challenges and provided real-time feedback on proposed solutions.
Key Insights:
-
Validated pain points identified in customer interviews through real-world scenarios within the school district.
-
Received iterative feedback during design co-creation sessions, ensuring proposed solutions resonated with practical needs and workflows.
Prototyping for Testing
In the pursuit of refining our solutions and harnessing the full potential of the algorithm, we implemented an interactive prototyping approach. The use of interactive prototypes served multiple strategic purposes, focusing on honing the solution space, evaluating the course recommendations algorithm, and iteratively improving the user experience based on evolving algorithm capabilities.
Design Process
Ideation and Prototyping
Our ideation process was guided by the dual objectives of addressing immediate needs while also laying the groundwork for our multi-year vision. Central to our design decisions was the imperative to create solutions that could be effectively tested, ensuring that each iteration brought us closer to our goals.
Designed for the Multi-Year Vision
Our multi-year vision centered around an algorithm capable of recommending the best set of courses for each student, considering various objectives such as academic success, graduation requirements, and college eligibility. We envisioned a system that could efficiently handle scheduling needs and limitations, automating much of the puzzle for the following year's schedule. This approach aimed to save guidance counselors time while ensuring students remained on-track for their desired outcomes. Importantly, our design philosophy emphasized providing solutions in the same sentence as the problem, making it easy for counselors to understand and act upon recommendations. Additionally, we aimed to facilitate seamless communication with students to address or inform them of any necessary changes.
The First-Value Milestone
For our initial deliverable, we focused on developing an algorithm capable of recommending individual courses to students for each desired outcome. This milestone allowed counselors to view student academic journeys, evaluate their progress towards specific outcomes, and use recommended courses to guide them back on track. By achieving this milestone, we aimed to provide immediate value to our users while setting the stage for further enhancements aligned with our broader vision.
Interactive Prototypes and Iterations
In our ideation process, we used interactive prototypes that allowed us to quickly explore various design concepts and iterate based on feedback. These prototypes enabled us to visualize the user journey, test different interaction patterns, and refine our solutions iteratively. By starting with low-fidelity prototypes, we could rapidly iterate through ideas, incorporating user feedback and refining our designs to better meet user needs and preferences.
Throughout the ideation and prototyping phase, our focus remained on creating solutions that could be effectively tested and validated, ensuring that each design decision brought us closer to achieving our project goals and vision.
Results and Next Steps
Testing Outcomes
Test 1: User Test with LAUSD Guidance Counselors
Purpose:
Evaluate the effectiveness of personalized course recommendations during Scheduling Season and within the Scheduler app, and assess the quality of algorithm-generated recommendations.
Method:
Conducted user tests with LAUSD guidance counselors using an interactive prototype demonstrating personalized course recommendations for their student caseload.
Key Findings:
-
Personalized course recommendations aided in hand scheduling during Scheduling Season but lacked alignment with broader school or district goals.
-
Integrating recommendations within the Scheduler app facilitated decision-making based on available resources, but separating them from Abl Analytics limited their potential value.
-
Algorithm-generated recommendations showed promise but require further development to enhance accuracy and relevance.
Example 1
-
Recommended courses in the list of students in the Scheduler
-
The user can filter for students missing Outcome requirements and schedule them by hand into recommended courses that would allow them to fulfill their missing requirements
Example 2
-
Recommended courses in the context of an Academic Journey for a single student
-
The user can evaluate if the student is off-track for an outcome and scan a list of multiple course options supported by scheduling data that would get them back on-track
Test 2: Improved Analysis and Actionability of Data
Purpose:
Assess the impact of providing analysis and access to students in need of recruitment on school and district leaders' planning capabilities.
Method:
Utilized an interactive prototype to showcase the potential use of currently enrolled student data with year-over-year data visualizations and calls to action.
Key Findings:
-
Providing analysis alone was insufficient; actionable recommendations were necessary for effective planning.
-
Recommendations should include actionable steps to address identified needs and improve outcomes, such as suggesting interventions or allocating resources based on trends.
Example 3
-
Year-over-year change monitoring for currently enrolled students to identify trends and needs
-
The user can quickly identify cohorts or strategies that need attention, evaluate positive or negative change over time, and access affected students
Test 3: 4-Year Plans with Guidance Counselors, Students
Purpose:
Confirm the viability of implementing 4-year plans aligned with the ultimate vision of the solution, considering future algorithm maturity and optimization.
Method:
Conducted user tests with guidance counselors and students to assess the effectiveness and feasibility of generating full courseloads for students optimizing their potential.
Key Findings:
-
Implementing 4-year plans aimed to establish a foundational pathway for students, saving counselors time by minimizing the need for manual scheduling adjustments.
-
“Opt-out” mechanisms for advanced courses aimed to encourage students to explore challenging courses, aligning with their potential for success.
-
4-year plans alleviated the burden on guidance counselors, providing students with guidance on course selection and ensuring alignment with their desired outcomes
-
A clearer picture of future course demand and resource planning, aids in equitable resource allocation and school or district goal alignment.
Example 4
-
Full secondary school career courseload generated to meet the student's goals.
-
The user is alerted if any changes put them off track for their desired outcomes, and is provided with options that would get them back on track
Example 5
-
Pathway explorer to aid in the evaluation of course pathways
-
The user can assess their top recommended course sequence and accompanying information about why it was selected and be confident in their decision
Test 4: Proposal for Course Recommendation Designs (Internal)
Purpose:
Align on understanding, product direction, and identify the first deliverable value regarding course recommendation designs, considering current and future algorithm capabilities.
Method:
Internal proposal supported by an interactive prototype to outline assumptions, features, and constraints for course recommendation designs based on existing insights and anticipated algorithm capabilities.
Key Findings:
-
Examined algorithm capabilities and proposed features for future development, such as providing one set of courses per student considering all outcomes.
-
Data limitations hindered optimal recommendations, proposing ongoing algorithm development based on available data and exploring obtaining missing data from school districts.
-
Proposed limiting initial recommendations to outcome-specific ones to ensure user safety and reduce the risk of harmful recommendations for the first delivered feature.
Example 6
-
Exploring the possibilities of the algorithm. In this scenario students are automatically sorted into lists for recruitment for the best course to keep them on track for their outcomes.
-
Bulk actions significantly reduce counselor workload, allowing quick sorting into desired courses for the following year.
-
Centering the lists around courses aids in resourcing for the upcoming year by surfacing the demand for each course earlier than is currently possible.
Example 7
-
Roll-up view for school or district leaders to identify equity trends and enable easy prioritization for student recruitment to help meet equity goals.
Test 5: First Deliverable Value User Test
Purpose:
To assess the implementation of the first deliverable value, including displaying recommendations in the Academic Journey, providing the ability to accept or dismiss recommendations, and allowing users to provide feedback to aid in algorithm development.
Method:
User tests were conducted with an interactive prototype to evaluate the workflow and usability of the implemented features.
Key Findings:
​
-
Counselors expressed enthusiasm for the potential value provided by an easy way to evaluate if a student is on-track for their outcomes.
-
Identified the desire to provide students and their families access to this information in the fall to help get/keep students on track earlier and reduce workload during course request collection and quality assurance.
-
Initiated discovery regarding if/how to evaluate students' on-trackness for graduation due to complex data collection requirements.
-
Confirmed the value of filterable lists of students and their on-trackness in Abl Analytics to assess progress towards goals.
​
Next Steps:
-
Release first deliverable to Design Partner school district for early access and feedback.
-
Gather quantitative data assessing the effectiveness and usage of the course recommendations
-
Continue to develop the underlying technology necessary to support the future product direction.
-
Product design iteration for the next release toward the ultimate product vision
Example 8
-
Academic Journey showing coursetaking history and recommendations for a student
-
The user can evaluate a recommendation and confirm that it is appropriate for the student, as well as give feedback if they notice bad recommendations
Conclusion
In conclusion, the project aimed to address the pressing challenges faced by educational institutions in guiding students towards academic success and equitable outcomes. By adopting a testing-focused approach and aligning design decisions with both immediate needs and long-term visions, we embarked on a journey to revolutionize course recommendation systems for secondary school and district leaders.
Through rigorous research and iterative prototyping, we gained valuable insights into the pain points and preferences of stakeholders, laying the foundation for user-centric design solutions. Our collaborative efforts with school districts and continuous testing allowed us to validate our assumptions, refine our prototypes, and deliver tangible value at each milestone.
The testing outcomes provided invaluable feedback on the effectiveness of personalized course recommendations, data analysis tools, and 4-year planning features. While the journey has been challenging, it has also been rewarding, as we witnessed the potential impact of our solutions in empowering counselors, administrators, and students to make informed decisions and achieve their educational goals.
As we move forward, our focus remains on refining the course recommendation algorithm, gathering quantitative data on solution effectiveness, and iterating on design enhancements. By releasing the first deliverable to our design partner school district and gathering feedback, we are poised to continue our journey towards realizing the ultimate product vision.
In summary, this project exemplifies our commitment to leveraging advanced data-driven tools to promote equity, maximize student potential, and drive positive educational outcomes. We are excited about the possibilities that lie ahead and remain dedicated to shaping pathways for each student to achieve success.