Case Study
Major Automotive Company Supports Agents with AI that Performs 10X Better
Brands consistently analyze and tap into large quantities of data provided by social media platforms using keyword-based systems. One major automotive company recognized this approach was not scalable for monitoring across multiple Facebook pages, Twitter handles and forums. By implementing the Interactions Digital Roots social media engagement platform, agents utilizing the system saw a 46% increase in the speed of response under twenty minutes and a 54% increase in unique customers engaged. The Interactions platform performed ten times better than the previous manual query-based system.
The Problem
In order to identify engagement opportunities, one major auto manufacturer’s social care agents were spending 60% of their time sifting through false positive search matches—that is, posts thought to be relevant that aren’t actually relevant—leading to loss of productivity and efficiency. Agents were spending far too much time crafting searches to find user content and missing real opportunities to connect with customers interested in making a purchase or those who had concerns needing to be addressed.
The Solution
Interactions focused on two areas: technology and the social moderation process. First, we structured a technical study to compare the performance between the query-based system and our Interactions social media engagement platform. To do this, we established an F1 score—the harmonic mean between precision (P) and recall (R), a standard metric for classification evaluation. The F1 Score = 2 x P x R / (P+R). Ultimately, we wanted to know the accuracy of the classification system in measuring marketing or assistance opportunities (precision) and the number of posts missed because of keywords used in filtering data (recall).
The F1 metric, after the three-month study, showed the Interactions platform performed ten times better than a query
system. Interactions was able to increase response times and free up agent time to connect with more unique customers than before.
The Results
A keyword-based approach to match search results works when the user is not concerned about the volume of results returned by the searched term. For example, internet users expect search engines to bring in highly relevant content on the first page of results and care less about what’s on page 99. However, this automaker’s agents were spending almost 60% of their productive time in identifying posts that they could respond to—like a sales lead or customer concern—using keywords. They needed a much more precise way of getting and using the information they needed.
The complete findings are summarized into three categories:
- Accuracy of classification when a post was labeled a marketing opportunity or assistance opportunity (Precision)
- Number of missed posts because the keywords used in filtering data being collected for agents to review (Recall)
- Overall response, engagement, and performance increases
PRECISION FINDINGS:
Greater precision: For every 100 relevant posts, the Interactions platform read 423 posts to the query-based system’s 4,748 posts.
RECALL FINDINGS:
Greater efficiency: The Interactions platform collected 15% more relevant data than the query-based system, which lost over 43% of the conversations captured by Interactions.
OVERALL FINDINGS:
Faster response time: Agents saw a 46% increase in the speed of response under twenty minutes.
Increased engagement: Agents experienced a 54% increase in unique customers engaged.
Better performance: The F1 metric showed the Interactions platform performed ten times better than the query-based system.
