Question

Service Incident: Sponsor Tracking Outage

  • 7 December 2016
  • 4 replies
  • 32 views

Userlevel 4
SUMMARY 

Early this morning we identified an issue with Sponsor Tracking where erroneous data was being passed through the API from our vendor resulting in false sponsor tracking updates. In an effort to prevent any further false updates, we've suspended the Sponsor Tracking service for all customers.

IMPACT

If your environment was directly impacted by false updates, our support team will be reaching out to you first thing today to discuss the impact and path to resolution.

NEXT STEPS

Gainsight is working urgently with our vendor to resolve this and restore service as soon as possible. Once service is restored, all sponsor changes that were missed during the outage will be pushed to your environment.

POST-MORTEM

Will be shared once we have more information.

FOR MORE INFORMATION

Please subscribe to this article for regular updates until the issue is resolved. If you have questions about this issue, please open a ticket with us by sending a note to support@gainsight.com.


4 replies

Userlevel 4

Update to Sponsor Tracking Service Incident:

We have confirmed that we are now receiving the correct information from our vendor regarding Sponsor Tracking updates.  We have been monitoring the updates and have validated on our end that the updates are correct that are being sent to our Gainsight environment.  

Root Cause

Our vendor received incorrect data via their API.  The incorrect data included multiple repeated connections associated with nearly every email address in their system.  This caused this data to be sent to Gainsight, which caused notifications to be falsified with incorrect data, as the email address was tied to an incorrect profile.  
  

Next Steps
  • If you have turned off your Sponsor Tracking Rules, you can now turn them back on and reschedule them
  • Once the rules are turned back on, we will re-enable Sponsor Tracking for our customers
  • Once the rules run, it will only pickup changes in the last 24 hours (will not be retroactive for the days that were missed)
Sponsor Tracking Moving Forward

Our vendor has implemented a staging environment to validate the data before sending it to Gainsight.  This staging environment will detect any "super nodes" profiles that exceed the expected number of connections.  If there is a major variance, that data will not be sent to Gainsight, which will prevent these false notifications.

If you need any assistance with enabling your rules, or have any questions on next steps, please reach out to support@gainsight.com
Userlevel 4
Update to Sponsor Tracking Service Incident:

We have not yet re-enabled Sponsor Tracking, as we are still validating the data.  Currently, we are validating the data received from our vendor, and we have seen a large amount of inaccurate changes.  

Next Steps:

We are currently working with our vendor to identify the root cause on the ~30% of inaccurate data received.  We will need to get to the root cause in order to proceed with re-enabling Sponsor Tracking.

If you have any questions on this, please reach out to support@gainsight.com.  We will continue to update this community thread as we learn more information on when this will be resolved.
Userlevel 4
Update to Sponsor Tracking Service Incident:

We have been working with our vendor on the quality of data they are sending, and we are back up to seeing accurate updates on (>80%) the data.  

Next Steps:

We will need to push an MDA patch to re-enable sponsor tracking to all customers.  Our Engineering team is currently working with our Release team to get a scheduled release date for this.  It is reasonable that this will be re-enabled in the next week.

If you have any questions on this, please reach out to support@gainsight.com.  We will continue to update this community thread as we learn more information on when this will be resolved.
Userlevel 4
Update to Sponsor Tracking Service Incident:

We have verified the changes with our vendor, and we are back up to accurate data that we are receiving.  Sponsor tracking will be re-enabled on 1/19.

Next Steps:

If you have turned off your Sponsor Tracking rules, you should now turn them on.  Once the scheduler is enabled on 1/19, you may see a high number of CTAs when your rules run, as this is pulling changes for the past 40 days.  

If you have any questions on this, please reach out to support@gainsight.com.

Reply