19 Posts


1.8K Points

Wed, Oct 11, 2017 5:30 PM


getting duplicates in S3 data upon ingest

A customer contacted me via ticket 28381 regarding an issue that he was having with one of his S3 jobs creating duplicate entries of selected key fields upon ingest. He wanted to see if there was a way within Gainsight to remove these entries by default within Gainsight. 

Here is the initial description of the issue per the customer:

I'm working with the S3 connectors, and for each ingest job, there is a section that says: "Select key fields to identify unique records". 

With the data that I have getting exported into the S3 bucket that Gainsight pulls to ingest, I have some duplicates, so I don't want those getting written as new rows in the upsert to the MDA.

With the "Select key fields to identify unique records" function, I was hoping these fields would essentially combine to create a composite unique key (I have 3 fields chosen for this function) like you could create in a regular MySQL database.However, when I run the ingest, I still get duplicate rows where all 3 values are the same between the duplicate rows.

The last development on this issue was that the client was going to try and remove these duplicate entries before ingest. What is the possibility of adding a duplicate checker (or something similar) into the UI for the S3 connector?