During Spring 2018, RadGrad "onboarding" sessions were held in several ICS classes, growing the active student user base to over 100 students. Each onboarding session was accompanied by a presentation which highlighted the different aspects and usage of RadGrad. Students logged in to RadGrad and used it to build a preliminary degree plan, but there were no explicit follow-on activities to encourage subsequent use of the system.
To observe how often students utilized RadGrad after the initial onboarding session, we analyzed User Interaction data through the Student Summary and Overhead Analysis tools. The chart below illustrates user behaviors from mid-March to June 2018:
The chart shows significant usage of RadGrad only during the weeks where onboarding occurred. Most students did not revisit RadGrad after their initial onboarding sesssion.
The data thus far from the Student Summary and Overhead Analysis tools reveals that the majority of students who have registered for RadGrad rarely revisit the application after onboarding. The data reveals that there are only a couple of students who are relatively consistent in using RadGrad, and most of the observed activity outside of the registration days can be attributed to these few students.
The goal of RadGrad is to help students to have a high quality undergraduate degree experience, which we believe involves:
- Exploring interests and career goals, and determining which ones are relevant;
- Completing courses and opportunities relevant to chosen interests and career goals;
- Planning a degree program that leads to 100 earned Innovation, Competency, and Experience points;
- Completing the degree program and earning 100+ points in each of the three categories;
- Achieving a high Level (5 or 6) in RadGrad, which indicates knowledge of and participation in other RadGrad services beyond degree planning (Course Reviews, MentorSpace)
Unfortunately, our initial experiences show that a single onboarding session is insufficient by itself to achieve this goal. In 25 minutes, students do not have enough time to explore all of the interests and career goals. They cannot review all of the relevent courses and opportunities. Most do not have time to plan a degree program leading to 100+ points in Innovation, Competency, and Experience. And if they do not return to the system, then they will not become aware of new interests, career goals, courses, and opportunities, nor might they even remember the opportunities they originally planned to take part in during a future semester.
The design of the RadGrad newsletter is motivated by the realization that, for RadGrad to be effective in improving a student's undergraduate degree experience, they must return to the system periodically to review and update their interests, career goals, degree plan, and other services. We do not believe that students must use RadGrad frequently in order for it to be effective in improving their degree experience: one or twice a semester should be sufficient. This motivation leads to the following primary goals for the RadGrad newsletter:
- The newsletter should motivate students to visit RadGrad at least once a semester;
- The newsletter should help students to use RadGrad effectively (not just visit the site, surf around, and leave).
The goal of the newsletter email is to provide relevant, actionable information to each student that they can use to improve their degree experience. To achieve this, the newsletter is divided into two sections:
Section 1: Message from the admins
This is the opening message sent to all students, written by the admins, which contains general information and announcements about RadGrad relevent to all student users. For example,
Welcome to the September edition of the RadGrad newsletter! We have implemented several bugfixes and performance optimizations this month that we hope will improve your user experience. Note that the Agathon and HACC Opportunities are coming up this month. As always, contact us at email@example.com if you have any questions about RadGrad.
Section 2: What to do right now
This section is the meat of the newsletter and provides personalized recommendations for each student based upon their current degree plan and other aspects of their current RadGrad state. Students will receive a maximum of three recommendations, in decreasing urgency. The current list of recommendations are as follows:
Finish your degree plan. If the student has not yet reached 100 Innovation, Competency, and Experience points, then this section will detail where their fall short, explain what they need to add, and provide an example addition (selected based upon their current interests).
Verify your completed opportunities. If the student has opportunities a current or previous semester that have not yet been verified, list them here along with instruction on how to submit them for verification.
Level up and upgrade your RadGrad sticker. This section will provide hints about what a student needs to do to make it to the next level. If the student has also recently achieved a new Level, this section will remind them that they can go to an advisor or admin to pick up a new sticker.
Complete your academic plan. If students have specified an academic plan, but have not yet finished planning out all their coursework in their degree planner, then this section will indicate that issue and provide a list of the remaining coursework.
Review completed courses. RadGrad functions better if all courses have recent reviews. If the student has recently completed a course that has few reviews, this section will request that they make a review.
Review completed opportunities. Similarly, it is useful for all Opportunities to have reviews. If a student has a verified Opportunity from a recent semester that does not already have many reviews, then this section will request a review.
Check out Mentor Space. If User Interaction data indicates that this student has not spent time at Mentor Space, then this can serve as a reminder.
Below are two newsletters generated for two different students. The first newsletter is sent to an entering sophomore that is lacking in the Innovation and Experience department. To improve upon these two areas, the newsletter recommends that the student reach 100 ICE points and to also level up, both which requires that the student adds opportunities. The degree plan recommendation in particular places emphasis on increasing their Innovation and Experience, and also provides sample opportunities that the student may be interested in. To round out their degree planning, the newsletter also provides a list of the remaining coursework required for the student to complete their academic plan.
The second newsletter is sent to a junior that has accomplished quite a lot during their academic journey and have already reflected it within their degree plan. Their RadGrad experience now consists of verifying opportunities that they have completed and returning back to the community by providing reviews for courses and opportunities.
Shown below is the admin interface to generate student emails:
The interface is divided into two sections:
This section of the interface provides admins with the option to create and format the opening message, add bcc recipients, and add the subject line for the email. The admin message input box accepts markdown format, and the message is required before sending out newsletters.
This section of the interface provides three different methods of sending newsletters to students:
Send newsletter targeted for specific students to the admin email. This method is normally used for testing purposes so admins can view how a specific student newsletter will look like when viewed from within an email client. There is also a checkbox to select to send to the respective students.
Send newsletter to students of a specified level. This method provides a dropdown that lists each RadGrad level, along with the number of students that are currently at that level
Send newsletter to all students. This method allows admins to send newsletters to all students that are registered within RadGrad.
Outcomes and evaluation
If the newsletter is successful, then we should expect to see the following impacts:
- Most students in the system will visit RadGrad at least once a semester.
- There will be an increase in User Interaction Behaviors corresponding to an improved Degree Experience. These behaviors include: Change Outlook, Exploration, Level Up, and Complete Plan Achieved.
To evaluate the newsletter, we will use User Interaction data to test whether the above impacts are present. In addition, we could perform a survey of selected undergraduates to see if they find the newsletter to be useful and what improvements they would like to it.
Many undergraduates participate in the ACM Manoa discord server as a way of keeping up with the department. A future extension to the newsletter is a Discord Bot that communicates with the community in general, and individual users, via Discord posts to channels and DMs. Eventually, users could be given the option to receive RadGrad communications either by email or by Discord.