Analytics for Engagement

Motivation

In order to obtain a better understanding of RadGrad's impact on a student's academic career, we must observe how, and how often, RadGrad is utilized by that user. Ideally, students will incorporate the DEP/RadGrad within their undergraduate journey and use it as a resource to enrich their learning experience. Simply introducing students to RadGrad may not be enough incentive, however, to continue using it. To encourage its use, various 'interventions' are being developed as way to increase exposure and engagement of RadGrad. These interventions will allow us to communicate with students and not only update them on their current DEP, but also provide information on related courses and opportunities. To assess whether or not these interventions have any impact on RadGrad engagement, analytical tools have been developed as a way to view when students are using RadGrad and how they interact with their DEP. This will help us observe any trends in RadGrad usage after an intervention has been released to a target student population. These trends could be as simple as a change in frequency of RadGrad usage, or perhaps even a change in their DEP (interests, courses, career goals, opportunities, etc).

Design

The core of these analytics are what is defined as User Interactions. Whenever a student updates their DEP, that information, or interaction, is stored within the 'UserInteractions' collection as a document (see the Data Model documentation page for more information on RadGrad's collections). The document provides four pieces of information: the username belonging to the student, the type of interaction that was updated, data associated with the interaction, and a timestamp. Types of interactions include adding or removing a course/opportunity, changes to the student's interests and career goals, leveling up, and completing their academic plan. Page views are also considered an interaction, and provides us a way to observe which area of the DEP students are spending the most time on. The following are two examples of interaction documents, as viewed in the MongoDB shell:

{ "_id" : "SCwSToH6KTcrFvhwd", "username" : "weng@hawaii.edu", "type" : "pageView",
    "typeData" : [ "home/aboutme" ], "timestamp" : ISODate("2018-08-18T10:50:46.943Z") }
{ "_id" : "eoGa4RGQQXNBpGFtf", "username" : "weng@hawaii.edu", "type" : "addCourse",
    "typeData" : [ "ics_111" ], "timestamp" : ISODate("2018-08-20T14:01:05.758Z") }

From this collection of interaction documents, analytical tools have been created to analyze the usage of RadGrad from different perspectives. These tools can be accessed from the Analytics tab when logged in as an Admin, with the usage of each tool detailed below.

Page Overview

The analytics page is presented above. The following tools are accessible from this page:

Logged In Users: This is the default page for Analytics, and shows which users are currently logged in.

Newsletter: The RadGrad newsletter can be generated and sent to students from this page.

Overhead Analysis: Provides overhead numbers for the amount of interaction documents created from student sessions.

Student Summary: Provides a summary of students' sessions by highlighting the different interactions. Individual sessions are in the form of a timeline.

User Interactions: All interaction documents for students can be viewed here.

The RadGrad newsletter is a personalized engagement tool used to help guide students through RadGrad and maximize their degree planning experience. See the Newsletter documentation page for more information regarding the newsletter and the newsletter admin interface.

Overhead Analysis

Due to the high number of interaction documents being generated by students, the User Interactions collection needs to be monitored for any potential overhead issues. The Overhead Analysis tool provides a way to measure the number of interaction documents created during student sessions. Admins first select a date range from which to view interaction activity. Overhead information is then provided in two separate tabs: the User Session Overhead tab which contains numeric data on documents created per student, and the Overall Server Load tab which contains a graph of the overall number of documents being created per minute. Data from the User Session Overhead tab is shown below for the time period of November 1st to November 10th:

A session is defined as a period of activity in which no more than ten minutes has passed between two interactions for a student. The following information is provided for each student that has generated at least one interaction within the specified time period:

Number of sessions: This is the total number of sessions that a student has been active in for the specified time period.
Number of documents: This is the total number of user interaction documents created among all sessions within the time period.
Documents per minute: This is the average number of documents generated per minute over all sessions within the time period.
Total time: This is the total time spent among all sessions within the time period.
Show documents: Clicking on the Show button within this column will bring up a modal that lists all interactions that were generated by the student within the specified time period.

The Overall Server Load tab provides overhead information in the form of the following graph for a specified time period:

The data visualized here groups the number of documents generated per minute into bucket ranges. So for example, during the time period represented by the graph above, there were approximately 190 minutes where 0-9 documents were generated per minute. This provides a quick way to observe any unusual spikes in interaction generation during a time period.

Student Summary

The Student Summary tool provides a way of observing how students utilized RadGrad during a specified period of time. The information it provides is divided into two parts: an aggregate student summary and individual student timelines. When a date range is selected, an aggregate student summary will initially be returned. This summary provides information on how many students have logged into the system at least once during the specified time, and also the amount of users that has exhibited at least one of (currently) nine behaviors. These behaviors are defined by interaction types, and are as follows:

Change Outlook: This corresponds to a change in interests, career goals, or degree plan.
Exploration: This corresponds to viewing entries in the Explorer, such as interests, opportunities, and courses.
Planning: This corresponds to work with the Degree Planner, such as adding or removing a course/opportunity.
Verification: When a student requests a verification for an opportunity.
Reviewing: When a student submits a review of a course.
Mentorship: When a student visits the MentorSpace page, and/or submits a question to the mentors.
Level Up: When a student achieves a new level.
Complete Plan Achieved: The moment when a student has first created a degree plan that, if executed to completion, will result in at least 100 points in all three ICE categories.
Profile: When a student updates their profile picture or their personal website URL.

Below is an example of the aggregate student summary for the month of March 2018:

As shown, each behavior section can be expanded to provide a list of all the students that have exhibited that particular behavior. Clicking on a student will bring up a modal that shows the timeline for that student during the specified time. The timeline is divided into sessions, and a session is defined the same as within the Overhead Analysis tool, in which no more than ten minutes has passed between two interactions. Similar to the aggregate summary, each session is divided into behaviors that the student has exhibited during that time.

Below is an example of an individual student timeline:

The aggregate student summary can also be viewed in the form of a chart, which can be accessed from the Timeline Chart tab next to the Student Summary tab. The timeline chart differs from the aggregate student summary in that it provides information on student behavior over time. The chart below shows behavior-related activity from October 22nd to November 2nd:

The percentage of students to exhibit a particular behavior per day within the selected range is visualized. Although all behavior activity is shown by default, data points for specific behaviors can be hidden by clicking on their respective labels within the legend. In the chart above, only the Exploration, Planning, Level Up, and Complete Plan behaviors are selected to be shown.

User Interactions

The User Interactions tool provides a means of viewing interactions at a low level for each student. The page provides a list of all current students, and clicking on each student will bring up a section that lists all interactions performed by that user.

Below is an example of the interaction list for a particular student:

Initial findings

Starting on the 27th of November 2018, newsletters were sequentially sent out to all RadGrad students by level over the course of several days. Below shows the timeline chart for that specific week:

Immediately following the release of the newsletter to lower level students, there was an increase in student activity. Most notably, there was an increase in the behaviors associated with log ins, outlook changes, degree planning, and explorer page views. Although the percentage increase in student activity was only in the order of a few percent at most, this brief insight show that there is potential to increase engagement with intervention tools. To stronger correlate the increase in RadGrad activity with the newsletter, individual student timelines will need to be analyzed to see if session activity matches that of the recommendations provided by the student's respective newsletter. At the very least, the current set of analytical tools will allow us to measure the effectiveness of future engagement tools, and further tuning can be performed accordingly.