On their face, early alert systems seem promising for community colleges and their overworked employees.
Students who do poorly on assignments, miss classes or act out can be flagged in the system. Then support staff can take action, nudging students with text messages or intervening more intensively, depending on the situation. Supports offered could range from tutoring to mental healthcare to child care services or help with housing and food.
Those actions aim to boost retention and graduation rates. But surprisingly little empirical evidence exists that they're effective, even though they have been around for years, according to New America, a left-leaning think tank. And some of the research that does exist looks at four-year institutions, meaning even less is known about the systems' successes or failures at community colleges.
A new report from New America sheds light on how community college leaders see early alert systems, or EAS. The think tank interviewed community college leaders, vendors, and sector experts from November to April. Most community college leaders interviewed were at institutions that had hired vendors to put early alert systems in place, although a small number developed a system internally or heavily personalized one.
New America found several areas of concern, including that early alert systems could contribute to racial discrimination and existing biases.
Many community college participants had been implementing an EAS for at least six years, according to the report.
"But it was not until our conversations that they had the opportunity and space to think about the racial implications of their EAS tool," the report said. "Many lack the capacity to evaluate their EAS data to ensure the predictive analytic tool, fed by faculty input, is not computing systemic discrimination."
The think tank recommended considering student perspectives while designing and implementing early alert systems on campus. Community colleges need to view procurement, implementation and evaluation critically to make sure they mitigate bias, it said.
New America identified five other challenges and recommended ways to address them. They're summarized below.
Procurement is a struggle
It takes time to pick an early alert system, and the process is difficult, New America found. Purchasing a system is expensive, and institutions can be stuck with their picks for years, even if they don’t work very well.
Vendors' salespeople don't always help.
"Many college leaders believe salespeople pitch a product during the initial conversation that is good in principle but often falls short in practice,” the report said.
For instance, community colleges told the think tank that third-party EAS vendors cast many processes as automatic even though they required manual integrations.
Leaders have many different needs to meet when picking products. They need to work for their student bodies, have enough data transparency, keep data secure and plug into existing systems.
How to address it: The report recommends a strong decision-making process when picking a product. That means bringing in different stakeholders to consider how new tools will fit into workflows, what functions they should have, how they might need to be customized, and whether colleges should buy systems or build their own.
Faculty don't always buy in
Community colleges often struggle to get faculty to buy in when they put early alert systems in place, New America found. Some faculty members wanted more proof the systems boost academic performance. Others were worried about the use of student data or didn't want to be accountable to systems they didn't know. And some didn't know what happened to students once faculty members flagged their performance.
Without faculty buy-in, the systems won't work, New America said. Faculty are the ones who engage with students firsthand.
How to address it: Use faculty feedback when procuring and piloting an early alert system, New America said. It also recommended training faculty on using the systems, as well as considering how they affect faculty workload.
Support services are often lacking
Even when early alert systems are in place, community colleges sometimes struggle to connect students with the services they need. Student support staff often have too much on their plates.
"The reality is that it is usually one or two staff members managing the alerts, and they often have additional responsibilities outside of EAS," the report said. "As a result of this limited staffing, flags sometimes fall through the cracks, with many students not receiving the support they need."
The messaging itself can also be problematic if not crafted effectively. New America recounted a message that used the term "probation" in relation to students' academic standing and finances. Students viewed the term negatively, which likely cut their levels of engagement with the assistance offered.
How to address it: Colleges should build up staff capacity to monitor alerts, coordinate among different offices on campus, and make sure they're communicating inclusively and effectively, New America said.
Early alert data isn't adequately evaluated
New America found evidence that institutions only analyze early alert system data superficially. For example, the majority of interviewed colleges and vendors focus evaluations on faculty participation.
"The frequency of flags and the rate of faculty participation do not dig deep enough to understand the process of streamlining flags from faculty alerts to the connection of services to students," the report said. "Looking only at these factors also does not help college leaders understand the impact interventions have on students’ academic and non-academic outcomes."
Nearly all community colleges and vendors that spoke with New America weren't examining flag frequency across important student characteristics like race, gender or whether students were enrolled part time.
"To not disaggregate data is to admit ignorance," New America's report said.
How to address it: Do deeper analysis, including disaggregating data by demographics, intervention type and student outcomes, New America said. For example, ask if certain students are more likely to receive flags and alerts than others — and why.
Community colleges struggled to use systems during the COVID-19 pandemic
As the COVID-19 pandemic eroded enrollment and retention rates, community colleges struggled to adopt and use early alert systems, New America found.
Institutions wrestled with monitoring attendance in virtual settings. Faculty members had difficulty identifying problematic academic behaviors, and students may have faced new challenges, such as COVID-19 infections, caring for family members and difficulty accessing online classes.
"Many college administrators grappled with how to swiftly identify and connect students to appropriate interventions in a remote context," the report said. "Furthermore, many colleges struggled with defining what a successful intervention looked like virtually."
Even after in-person classes returned, the lessons of the pandemic are important. Hybrid learning is expected to stick around or even grow.
How to address it: Community colleges can modify their student attendance policies, boost their support services and keep a personal touch as they use early alert systems, New America said.
"Colleges must redefine what constitutes an alert in different learning modalities," the report said. "Although EAS can be effective in promoting student success, students may be discouraged to engage with a notification or an intervention if they believe the notification came from a bot rather than a person.”
