During Spring 2018, RadGrad "onboarding" sessions were held in several ICS classes, growing the active student user base to over 100 students. Each onboarding session was accompanied by a presentation which highlighted the different aspects and usage of RadGrad. Students logged in to RadGrad and used it to build a preliminary degree plan, but there were no explicit follow-on activities to encourage subsequent use of the system.
To observe how often students utilized RadGrad after the initial onboarding session, we analyzed User Interaction data through the Student Summary and Overhead Analysis tools. The chart below illustrates user behaviors from mid-March to June 2018:
The chart shows significant usage of RadGrad only during the weeks where onboarding occurred. Most students did not revisit RadGrad after their initial onboarding sesssion.
The data thus far from the Student Summary and Overhead Analysis tools reveals that the majority of students who have registered for RadGrad rarely revisit the application after onboarding. The data reveals that there are only a couple of students who are relatively consistent in using RadGrad, and most of the observed activity outside of the registration days can be attributed to these few students.
The goal of RadGrad is to help students to have a high quality undergraduate degree experience, which we believe involves:
- Exploring interests and career goals, and determining which ones are relevant;
- Completing courses and opportunities relevant to chosen interests and career goals;
- Planning a degree program that leads to 100 earned Innovation, Competency, and Experience points;
- Completing the degree program and earning 100+ points in each of the three categories;
- Achieving a high Level (5 or 6) in RadGrad, which indicates knowledge of and participation in other RadGrad services beyond degree planning (Course Reviews, MentorSpace)
Unfortunately, our initial experiences show that a single onboarding session is insufficient by itself to achieve this goal. In 25 minutes, students do not have enough time to explore all of the interests and career goals. They cannot review all of the relevent courses and opportunities. Most do not have time to plan a degree program leading to 100+ points in Innovation, Competency, and Experience. And if they do not return to the system, then they will not become aware of new interests, career goals, courses, and opportunities, nor might they even remember the opportunities they originally planned to take part in during a future semester.
The design of the RadGrad newsletter is motivated by the realization that, for RadGrad to be effective in improving a student's undergraduate degree experience, they must return to the system periodically to review and update their interests, career goals, degree plan, and other services. We do not believe that students must use RadGrad frequently in order for it to be effective in improving their degree experience: one or twice a semester should be sufficient. This motivation leads to the following primary goals for the RadGrad newsletter:
- The newsletter should motivate students to visit RadGrad at least once a semester;
- The newsletter should help students to use RadGrad effectively (not just visit the site, surf around, and leave).
The system design involves two interfaces. One "interface" is the newsletter email seen by students. The second interface is used by admins to generate and send the emails.
Newsletter email design
The goal of the newsletter email is to provide relevant, actionable information to each student that they can use to improve their degree experience. To achieve this, the newsletter has three sections:
Section 1: Message From The Admins
This is the opening message sent to all students, written by the admins, which contains general information and announcements about RadGrad relevent to all student users. For example,
Welcome to the September edition of the RadGrad newsletter! We have implemented several bugfixes and performance optimizations this month that we hope will improve your user experience. Note that the Agathon and HACC Opportunities are coming up this month. As always, contact us at firstname.lastname@example.org if you have any questions about RadGrad.
Section 2: Improve your degree experience
This section is provides personalized recommendations for each student based upon their current degree plan and other aspects of their current RadGrad state. One or more of the following recommendations will appear in this section depending on their relevency:
Finish your degree plan. If the student has not yet reached 100 Innovation, Competency, and Experience points, then this section will detail where their fall short, explain what they need to add, and provide an example addition (selected based upon their current interests).
Upgrade your RadGrad laptop sticker. If the student has recently achieved a new Level, this section will remind them that they can go to an advisor or admin to pick up a new sticker.
Level Up. This section will provide hints to students about what they need to do to make it to the next Level.
Verify your completed opportunities. If the student has opportunities a current or previous semester that have not yet been verified, list them here along with instruction on how to submit them for verification.
Complete your academic plan. If students have specified an academic plan, but have not yet planned out the coursework required to complete it, this section will indicate that issue and explain how to finish it.
Review completed courses. RadGrad functions better if all courses have recent reviews. If the student has recently completed a course that has few reviews, this section will request that they make a review.
Review completed opportunities. Similarly, it is useful for all Opportunities to have reviews. If a student has a verified Opportunity from a recent semester that does not already have many reviews, then this section will request a review.
Check out Mentor Space. If User Interaction data indicates that this student has not spent time at Mentor Space, then this can serve as a reminder.
Based on your interests.... This section highlights Opportunities and/or Courses that match this student's interests but which is not included in their degree plan.
Section 3: RadGrad Updates
This section provides a listing of changes to the state of RadGrad since the last newsletter was sent out. These updates can include:
- Newly added opportunities.
- Newly added courses.
- Newly added course reviews.
- The numbers of students who have recently achieved a new Level. For example, "In the last month, 23 students have achieved Level 2, 5 have achieved Level 3, 2 have achieved Level 4, 6 have achieved Level 5, and 3 students have become RadGrad Ninjas! (Level 6)"
Below are newsletter mockups for two different students. The first mockup is sent to an entering sophomore that is lacking in the Innovation and Experience department. Emphasis is put upon achieving 100 ICE points, and there is a push to add and verify opportunities. Because the student has completed a lot of coursework, there is no recommendation to add courses, and instead, there is a small tip provided to leave a course review.
The second mockup is sent to a junior that has verified the recommended number of opportunities and has accomplished quite a bit along their academic journey. Leveling up in RadGrad is emphasized for the student as it is normally a long term goal. Any unverified opportunities are also mentioned, along with a list of courses to review. These tips are also helpful in expediting the leveling process.
The admin interface enables the admin to write a generic header section to the email, then have a set of checkboxes indicating who will receive the email. One checkbox is "admin only" along with a text field that enables the admin to enter a comma-separated list of usernames (i.e. email addresses). The result of this selection is to generate the newsletter for each of the users and send it to the admin instead. The other checkbox is to send it to all students.
When an email is generated and sent to a student, the admin email (i.e. email@example.com) is cc'd.
We may need to "throttle" email sending, such as having each email sent five seconds apart, in order to avoid problems with meteor, mailgun, and/or gmail.
Outcomes and evaluation
If the newsletter is successful, then we should expect to see the following impacts:
- Most students in the system will visit RadGrad at least once a semester.
- There will be an increase in User Interaction Behaviors corresponding to an improved Degree Experience. These behaviors include: Change Outlook, Exploration, Level Up, and Complete Plan Achieved.
To evaluate the newsletter, we will use User Interaction data to test whether the above impacts are present. In addition, we could perform a survey of selected undergraduates to see if they find the newsletter to be useful and what improvements they would like to it.
Many undergraduates participate in the ACM Manoa discord server as a way of keeping up with the department. A future extension to the newsletter is a Discord Bot that communicates with the community in general, and individual users, via Discord posts to channels and DMs. Eventually, users could be given the option to receive RadGrad communications either by email or by Discord.