Analytics for Engagement

Motivation

In order to obtain a better understanding of RadGrad's impact on a student's academic career, we must observe how, and how often, RadGrad is utilized by that user. Ideally, students will incorporate the DEP/RadGrad within their undergraduate journey and use it as a resource to enrich their learning experience. Simply introducing students to RadGrad may not be enough incentive, however, to continue using it. To encourage its use, various 'interventions' are being developed as way to increase exposure and engagement of RadGrad. These interventions will allow us to communicate with students and not only update them on their current DEP, but also provide information on related courses and opportunities. To assess whether or not these interventions have any impact on RadGrad engagement, analytical tools have been developed as a way to view when students are using RadGrad and how they interact with their DEP. This will help us observe any trends in RadGrad usage after an intervention has been released to a target student population. These trends could be as simple as a change in frequency of RadGrad usage, or perhaps even a change in their DEP (interests, courses, career goals, opportunities, etc).

Design

The core of these analytics are what is defined as User Interactions. Whenever a student updates their DEP, that information, or interaction, is stored within the 'UserInteractions' collection as a document (see the Data Model section for more information on RadGrad's collections). The document provides four pieces of information: the username belonging to the student, the type of interaction that was updated, data associated with the interaction, and a timestamp. Types of interactions include adding or removing a course/opportunity, changes to the student's interests and career goals, and also any changes to their profile (including their level). Page views are also considered an interaction, and provides us a way to observe which area of the DEP students are spending the most time on. The following are two examples of interaction documents, as viewed in the MongoDB shell:

{ "_id" : "SCwSToH6KTcrFvhwd", "username" : "weng@hawaii.edu", "type" : "pageView",
    "typeData" : [ "home/aboutme" ], "timestamp" : ISODate("2018-08-18T10:50:46.943Z") }
{ "_id" : "eoGa4RGQQXNBpGFtf", "username" : "weng@hawaii.edu", "type" : "addCourse",
    "typeData" : [ "ics_111" ], "timestamp" : ISODate("2018-08-20T14:01:05.758Z") }

From this collection of interaction documents, analytical tools have been created to analyze the usage of RadGrad from different perspectives. These tools can be accessed from the Analytics tab when logged in as an Admin, with the usage of each tool detailed below.

Page Overview

The analytics page is presented above. The following tools are accessible from this page:

Logged In Users: This is the default page for Analytics, and shows which users are currently logged in.

Student Summary: Provides a summary of students' sessions by highlighting the different interactions. Individual sessions are in the form of a timeline.

User Interactions: All interaction documents for students can be viewed here.

Student Summary

The Student Summary tool provides a way of observing how students utilized RadGrad during a specified period of time. The information it provides is divided into two parts: an aggregate student summary and individual student timelines. When a date range is selected, an aggregate student summary will initially be returned. This summary provides information on how many students have logged into the system at least once during the specified time, and also the amount of users that has exhibited at least one of (currently) nine behaviors. These behaviors are defined by interaction types, and are as follows:

Change Outlook: This corresponds to a change in interests, career goals, or degree plan.

Exploration: This corresponds to viewing entries in the Explorer, such as interests, opportunities, and courses.

Planning: This corresponds to work with the Degree Planner, such as adding or removing a course/opportunity.

Verification: When a student requests a verification for an opportunity.

Reviewing: When a student submits a review of a course.

Mentorship: When a student visits the MentorSpace page, and/or submits a question to the mentors.

Level Up: When a student achieves a new level.

Complete Plan Achieved: The moment when a student has first created a degree plan that, if executed to completion, will result in at least 100 points in all three ICE categories.

Profile: When a student updates their profile picture or their personal website URL.

Below is an example of the aggregate student summary for the month of March 2018:

As shown, each behavior section can be expanded to provide a list of all the students that have exhibited that particular behavior. Clicking on a student will bring up a modal that shows the timeline for that student during the specified time. The timeline is divided into sessions, and a session is defined as a period of activity in which no more than ten minutes has passed between two interactions. Similar to the aggregate summary, each session is divided into behaviors that the student has exhibited during that time.

Below is an example of an individual student timeline:

Currently being implemented within this tool is the number of interventions that have occurred during the specified date range. Interventions will also be listed within the student timeline should they have been released during that period of time. This will allow us to see if there are any changes in behavior after an intervention has been released. This will be fully implemented by the time the first intervention has been sent out.

User Interactions

The User Interactions tool provides a means of viewing interactions at a low level for each the student, and is useful for debugging purposes. The page provides a list of all current students, and clicking on each student will bring up a section that lists all interactions performed by that user.

Below is an example of the interaction list for a particular student:

Initial findings

Initial findings are still being evaluated.