What equation are you using for calculating "usage" of your product?

  • 1
  • Question
  • Updated 2 months ago
  • In Progress
We are re-doing our health scores and I'm grappling with how best to represent "usage" in a health score. So far I've decided to roll up all of our pageviews, and API calls across 3 different products into an overall usage aggregate number. Then, I tried to use the percentage change function to compare usage of each customer to themselves over the last 4 weeks. Then we can see generally if they are using it more or less than they had been. More == better, right? If they continue to adopt, and grow over time their health scores will remain good. Longer term reductions in overall usage will tell us that something may be going wrong.

My issue with this approach is that if a customer has a big spike in usage for a short period of time, it skews their scores for the next month, dragging them down and making it look like they are less healthy. It also does not take into account any comparison between accounts. ie. customer A fluctuates between 30,000 and 10,000 actions will look less healthy than customer b who has a consistent 10 actions. Which isn't right...

I was thinking about trying to calculate the distribution of customers and try to force them into quintiles of usage. I remember an old scoring 1.0 feature that did this, but not in 2.0...

Wondering what others are doing.
Photo of Jason Metzler

Jason Metzler

  • 540 Points 500 badge 2x thumb
  • oh, fine.

Posted 3 months ago

  • 1
Photo of Dan Ahrens

Dan Ahrens, Official Rep

  • 24,802 Points 20k badge 2x thumb
Hi Jason, it usually helps to determine how you will score customer usage health using a excel doc where you can set filters and calculations and determine the impact to your customer community. In your specific example, usage health might be a function of (number of page views or API calls / number of licenses or connections). This way you have a scalable ratio that works for small customers and large ones. 

As usage varies from week to week and month to month, you will keep track of the score for each time period but as you noted it can be problematic to actually change the ranges of what is considered red, yellow, or green by incorporating historical information into the mix. 

A more methodical approach might be to re-evaluate your health score ranges on a periodic basis (either quarterly or bi-annually) to ensure that a customer that scores a green is still considered in good health. 
Photo of Tom Wix

Tom Wix, Champion

  • 2,446 Points 2k badge 2x thumb
In addition, (perhaps you were stating this indirectly), you may want to look at segmenting the customer base into general type of users , assuming you have some historical data.  Score each segment with a different formula.  As you mentioned, users that are "spikey" users and those that are more consistent and perhaps there is one other kind.

Tag/Segment the customers, manually score (Excel or outside of the tool) the segment based on some known healthy and non-healthy to ensure the segmented scoring is working (as Dan mentioned) and then put it into play.   Be sure to review in 4 to 6 months.
Photo of Jason Metzler

Jason Metzler

  • 540 Points 500 badge 2x thumb
I guess I was looking for the actual equation folks are using.

For example, I am taking our pageviews and api calls and adding them all together, then comparing the previous week to the average of the past 10 weeks, ignoring missing data.

(System1 pageviews + System1 Sis1 api calls + System2 pageviews + System2 api calls) = combined usage number

Then, to calculate the score I tie it to this value and scale it -200 to 25, to account for swings in usage. Any increase in usage will lead to higher scores, while a decrease in usage will lead to neutral scores, provided there is some usage.

I'm looking for ideas on how to better score 'usage', as this isn't an ideal solution. Our highest usage customers are in the tens of millions of total actions per day, while other customers may only use 10 pages, and both could be deemed equally 'healthy'.