A/B Testing for Copilot

Related products: None

We are looking for an easy way to split a power list in half to send a message to one sub group and a slightly different one to another sub group to compare the results. Of course, we can set up a flag and populate it. I am just looking to see if we can do it easier than that. Is this something we have considered as a feature for copilot?
Could you use the variant feature (with different message content instead of language)?
I was thinking the same Dan.  You would want to pick an attribute to use to split the list.
Beauty of supporting A/B testing is the powerlist is randomized in system. I do think variant is the right mechanism to leverage though!
Totally agreed to all of your request, however today we don't have randomizing email feature available. We have this as our roadmap items.
Hi Praneet, is there an ETA for adding this feature? We're eager to move more of our customer emails to Gainsight vs the email marketing automation platform our Marketing team uses, but the lack of easy, random A/B testing functionality is a major obstacle to this.
We also have a use case for an A/B testing feature combined with the Survey tool and advanced outreaches.  It would be great to have the ability to configure a fork in the AO workflow so participants receive one of two possible email template/survey combinations.  Perhaps build this into the conditional wait configuration.
So here's an idea for doing some A/B style testing using automation. This leverages the LENGTH(field) and MEDIAN (field) functions in rules. https://support.gainsight.com/Product_Documentation/Rules_Engine/Admin_Configuration/Formula_Builder...





Steps below as follows:




  1. Fetch the Account ID and Account Name of each account for consideration (along with any other fields you'd want to carry forward)



  2. Use a transformation task to add "Account Name Length" to your data set (using LENGTH(field) function).

  3. Use a second transformation task to bring forward the data in the second task and add a formula field using the MEDIAN(field) function to calculate the mid point length of characters for all your customer names

  4. Use a third transformation task to add a calculated field that is equal to the Account Name Length - Median Account Name Length. 

  5. You'll now have a field for each account that you can use that will be a positive number if the account name length is greater than the median, negative if the account name length is less than the median and a few that will be zero where the account name length equals the median.

  6. This will split your data set into two roughly equal groups (with a few in the zero spot). 
Then just use the compare field you created in step 5 to segregate your email variants where those where the compare field is greater or equal to 0 gets one variant and those where the compare is less than zero get the other variant. 





This is just one example. You could use the basic principle on any field data that is present on all accounts and I'm sure some may even devise more complex methods using the other statistical functions like Correlation, Covariance, Std Deviation/Variance and Median. 





Average should be used with caution as a single large outlier can skew the entire dataset's mid point. 

@dan_ahrens

Would you have a minute to look at my transformations? For some reason this survey failed to send… was trying to use this A/B suggestion.

 

I’ve included the sources attached here

 


HI @scott_sigsci - at your last step you are aggregating by company ID name, which instead of pulling in the email address and other relevant data of your contacts, is showing the count and sum of those values. You’ll want to remove that aggregation.


So close... I’ll remove that and have another go. Thank you @dan_ahrens


@dan_ahrens now I remember why I put that in… I keep getting this error. Any thoughts? (Note: I didn’t see this error in the first two transformations)

 


Hi Scott, without working in your org specifically, it’s hard to dig very deep on troubleshooting and advising using screenshots, but you might find it helpful to do that field length calculation in it’s own step and then merging that data back in with your other company data. Your Gainsight Customer Success Manager can probably work 1 on 1 with you to tune this query. 


Thank you @dan_ahrens , much appreciated.


@dan_ahrens I figured out what the issue is. In this case, you don’t use a “calculated field,” you use “numeric expression.”


@dan_ahrens: Thanks a lot for the idea with LENGTH and MEDIAN above. Very interesting angle to look at it! Is it, however,  – statistically speaking – random enough? Have you maybe considered adding a distinct random function, i.e. similar to RAND() function in MySQL? We currently use exactly that, just with a bit of extra complexity and pushing it to Gainsight from our internal data warehouse.

In any case, I think it would be great to have the fork functionality that @dale_parent suggested above. Having that option directly in the flow and, ideally, also being able to compare the results directly in the Analytics would be absolutely amazing!


Do posts tagged ‘Work around available’ ever get revisited? This would be nice to have as an actual feature and not a 3 task transformation. It also doesn’t solve for powerlists or segments, as that feature grows in adoption.


We are looking at some options to do this in Journey Orchestrator and might have some more updates closer to the start of the new year. 


Is there any update to make this a feature? All of our account IDs are exactly the same length, so this crazy complicated workaround is really not feasible for us. It would be really helpful to have this as a feature with similar parity to other email systems where I can just configure the A/B settings and let it deploy with randomization. 

This would also help with company conversion. My marketing team is showing more interest in using JO rather than Marketo for certain drip campaigns and surveys, but this is a basic functionality they use all the time. 


This is a feature that is being planned for 1H 2021. 


Hello! I wanted to follow up and see if there was still hope for this to be a feature in JO? Our company would love to start doing some A/B testing as we bring more complicated programs into Gainsight and adding the additional transformations would complicate things a bit leaving more room for error when trying to complete the testing. 


Hello! I wanted to follow up and see if there was still hope for this to be a feature in JO? Our company would love to start doing some A/B testing as we bring more complicated programs into Gainsight and adding the additional transformations would complicate things a bit leaving more room for error when trying to complete the testing. 

@PavanCh could you comment?


@PavanCh any comments on this?


@vmallya could you pl address the latest with this? 


Hello @anirbandutta and everyone on this thread!! 

Despite being on our backlog, this requirement was not picked up due to other conflicts. Although this is a valuable addition to the platform, we will not be able to implement it anytime soon.

If it helps, I’d like to suggest an alternative solution (can share more details if needed) - 

  1. We can create a custom dropdown field in the Company object (can have A, B.. values as needed)
  2. We then create a rule to update the field value of this field based on some conditions
    • SUBSTRING(field, position, length) - finds the substring based on the position and length.
      • Last digit of GSID (Alphabet or number)
      • If number (odd or even)
      • etc.
  3. Create a data set filtered on this field value (A, B, ..etc.)
  4. Then use appropriate data sets for respective JO programs.

Thank you.