Tethr Blog

QA measures customers actually care about

Our last post introduced a new way to connect QA measures with business outcomes we all care about, such as retention, share of wallet and advocacy. Tethr’s latest scoring algorithm, the Agent Impact Score (AIS) isolates the agent’s impact on perceived customer effort, by focusing on potentially damaging effects the agent may have on customer loyalty.

Too often our quality assurance processes rely heavily on very binary, black and white measures. What agents are often left with is guidance rooted in what may be easy to observe but in practice feels hard to change. Imagine a few common QA outputs:

QA Guidance: talk less
Agent reaction: ok…how much less?

QA Guidance: say “I’m sorry” more often
Agent reaction: sometimes when I do that the customer seems to get more frustrated…what do I do then?

QA Guidance: make sure to ask the customer at the end if they need any further assistance
Agent reaction: I did that and the customer still filled out a survey indicating lack of resolution…how can that be my fault?

If we’re being really honest, these traps appear in part because we start with us. Surely resolution is most important, right? True, customers want resolution, but we like it as much or more because it means more efficient service with less repeat calls, bringing cost-of-service down. That us-first view extends into how we might we go about trying to spot issues with respect to resolution:

  1. First, we’d rely on what limited data we do have. Perhaps we have some post-interaction surveys indicating customer-perceived “resolution.” Maybe we used some keyword spotting to indicate the agent or customer did NOT say words such as “fully resolved.”
  2. Then, we’d go look at a few calls and look for defects and gaps. What was missing in those calls where resolution was lacking? This is a game of working back from a structured view on what should be there so that I can know when it is not.
  3. What’s the best way to work back from what should be there? Let’s rely on what is easiest to observe! So we go back time and again to old standbys, looking for the same stuff we’ve always looked for: empathy, listening skills, handle time, etc.

The common thread, of course, is those lenses are often very focused on what we think is healthy and a perfunctory view that most of what customers care about revolves around resolution. In this perspective, customers care less about how we get to resolution than they do that we get there.  

But this flies in the face of decades worth of evidence around what drives customer disloyalty: actual and perceived effort – both within and beyond service interactions – remain the most reliable predictor of churn and even impact upsell and cross-sell potential. Customers care a lot about how issues are resolved.

Real call scenarios aren’t monochromatic

Measuring effort in all its various forms introduces shades of gray to the old black and white world of QA, filling in important context around the how part of getting to resolution. As one example, we can measure the relationship of call duration with perceived effort. Is an additional two minutes on the call worth it if it means that two minutes is spent proactively solving a next anticipated, downstream issue that prevents the next call? Handle time is up, but effort is down. Feedback can reflect that particular shade of gray: handle time increases aren’t always bad.

The Tethr Agent Impact Score takes QA to a new height, colorizing the entire experience by isolating those effort variables directly in control of the agent. AIS (as part of the Tethr Effort Index) predicts the relationship between perceived effort and hundreds of real-valued and syntax-based variables. Scores reflect how the conversation ensued (e.g. who was talking when, indicators of active conversation, etc.) along with explicitly- and implicitly-articulated effort drivers presented either as part of the interaction or about how the agent handles concerns regarding other parts of the customer journey. 

Whereas in the past QA might rely only on binary metrics, using AIS enables quality to be viewed on a spectrum that is both observably important to the customer and actionable on a per-call basis. Agents can learn to spot specific phrases associated with complex emotional concepts such as decision uncertainty. QA scores can reflect how the agent handled such expressions. And business leaders can make informed investments in training or tools knowing the relationship between agent actions and customer outcomes.

So what happens when you colorize QA by adding in AIS as a filter? Early findings indicate that some of the things companies have long assumed to improve quality actually don’t move the needle; even more surprisingly, some of the things we’ve long assumed to be detrimental to quality are actually positive drivers of the customer experience. 

A multidimensional approach to quality

Using AIS as a lens for agent performance, we are able to easily compare specific behaviors on several dimensions: a) how often is it a driver of major unwanted friction, and b) the relative degree of contribution on a per-call basis. 

This last dimension is super unique and important. Lots of things happen on a customer call.  It’s easy to cherry pick individual things and assume, if removed or improved, it is *the* thing on which to focus. But this dimension tells us that some things happen and they don’t really matter. They may be unwanted and, in a vacuum, we’d prefer they didn’t happen. But all things equal they didn’t move the outcome all that much.

But there are certain occurrences that matter a LOT more than others. These situations are often represented as sequences across the course of the conversation – one thing happens, and another thing happens afterwards, sometimes far afterwards. And now we have a way of isolating this relative contribution toward perceived customer effort.

To illustrate, consider the following framework:

This matrix represents contribution toward effort reduction, filtered by AIS scores. Here, we’re looking at scores of less than 4. So these are bad situations where the agent is either not helping or making things worse.

The x-axis, moving from left to right, indicates how often these variables were contributing to AIS scores being pulled down.

The y-axis starts quantifying how much it was pulled down. So the y-axis measures the relative size of contribution. The higher the contribution, the bigger the effect that variable had on the score – and, in this case, a bigger effect means pulling scores down the most.

Specifically, the vertical axis is looking at the average level of contribution. So those at the top are basically above average contributors; and those below are basically below average contributors. Anything on here is bad, but those towards the bottom are less bad on a relative basis.

So, as you look at the top left quadrant, these are infrequent drivers of “bad AIS” (scores of <4) but when they are, it’s often the dispositive driver. Bottom right is the opposite, agent behaviors that often contribute to low AIS scores but individually aren’t all that damaging. Top right is the main concern – both frequently large drivers and very damaging to the experience we want to deliver for customers. 

Now let’s see what happens when we examine real calls from several companies (worth noting: individual companies may see some variation in these data points depending on their specific situation, but the findings are worthy of examination for all companies).

Some agent behaviors we would expect do bubble to the top. Those following our Effort research won’t be surprised to see Powerless to Help – where agents often hide behind policy and shirks action – as both a frequent and damaging agent behavior. But several others benefit from this colorized view of quality assessment, including but not limited to:

  • Redirect – this is where the agent passes the buck, for instance saying “I suggest you try [another contact]” or “I can’t help you but maybe they can.” Here, it’s clear that on average it is not often a main driver of (high) effort. But in certain instances, paired with specific other events, it’s one of the most damaging things an agent can do. So coaching is situational, where outliers are of main concern.
  • Rep Confusion – nobody wants our teams to act confused with customers. But it’s unrealistic to think it will never happen. The main thing to avoid is pairing it with specific situations that takes something only somewhat bothersome and makes it overly frustrating. Again, coaching can be situational, where combinations of events matter more than the confusion itself on its own.

Now let’s look at the opposite, where customers are generally happy with how the agent performed:

Perhaps not surprising, the power of an agent offering guidance in a proactive manner turns out to be a large driver of low effort situations. Other behaviors such as an agent probing issues with customers (“Probing Questions” on the left) are quite positive if more rarely direct drivers of low effort. Several others obviously benefit from this colorized view of quality assessment, including:

  • Escalations and Holds – these are outcomes we generally prefer to avoid, but customers appear to appreciate them if placed in the right manner. Even better when paired with some degree of guidance or advocacy as a way to either circumvent the need for escalation or identify additional issues that might otherwise drive a call back. 
  • Overtalk – this is measuring how often an agent speaks at the same time as the customer (a close cousin to the number of interruptions detected within the audio). Most agents are taught to optimize listening ratios – and, therefore, one might assume lots of overtalk would be perceived poorly. Here it’s the opposite, where some degree of overtalk appears to be positive. Again, coaching is colorized, as some amount of overtalk is likely viewed as an indicator of “active conversation” and therefore forgiven or preferred versus more passive behavior.

Tethr customers are learning every day new ways AIS impacts their teams. What’s already clear is that adding AIS to QA scorecards allows teams to view situations in full color, understanding the relationship of specific behaviors and customer preferences.

Check out our announcement of AIS as well as our on-demand webinar on aligning agent performance to business outcomes with AIS. Learn more about how one of our customers is using AIS for QA here. Of course, if you’re interested in seeing AIS in action, simply request a demo.