Import a file from S3 to Bionic Rules directly

  • 11
  • Idea
  • Updated 3 months ago
  • Under Consideration
I've been hearing multiple use cases / suggestions around being able to import the data file from S3 into Bionic Rules. 

1) Eliminate the need for setting up S3 jobs and then running the rule separately and potentially avoid 24 hours of delay before data is processed and available for consumption.

2) Loading data to SFDC directly without needing to store in MDA as a staging area

3) Without having to think about missing schedules. With Bionic Rules, reading the data from S3 and processing can be one rule.

4) Avoid storing unwanted data. Dimensionality reduction, filtering the bad quality records, applying the transformations.

5) Merging multiple files into one data set before loading. 

What are the other supporting use cases that you can think of?
Photo of Ashok Dugaputi

Ashok Dugaputi, Employee

  • 1,490 Points 1k badge 2x thumb

Posted 12 months ago

  • 11
Photo of Scott Drost

Scott Drost, Senior Solutions Architect

  • 4,088 Points 4k badge 2x thumb
Love the general idea!  Turns Bionic rules into a 1-stop shop for data ingest and transformation.

In particular, #'s 1, 3, and 5 get my vote for the strongest use cases in the list. Basically sets up a way to allow data transformation, if nothing else, to occur immediately following the arrival of new data.

I'd want to talk through #2, 'load directly to SFDC' in a very detailed fashion - a lot of implications there for consequences to the SFDC-side data that may not have any bearing on Gainsight, which always makes me nervous - basically would want that to be a feature added later.

One other use case:

-- Allowing an opportunity to load custom field mappings (static data points like "load date" or "data source") as part of a file ingest as well - formula fields with Today(), for instance, for Load Date, etc.
Photo of Sundar

Sundar, Employee

  • 11,270 Points 10k badge 2x thumb
Thanks Ashok for bringing this up! Can't agree more, I would love to hear more perspectives.
Photo of Roja Kode

Roja Kode, Employee

  • 930 Points 500 badge 2x thumb
Thanks Ashok for the idea!
(1-3) will address missing connect flight scenarios and in turn reduces data issues. I believe, it's one of the awaiting feature for many of the customers.
Photo of Roja Kode

Roja Kode, Employee

  • 930 Points 500 badge 2x thumb
Another use case is that for standalone customers, for loading Customers list, currently I have to load to MDA object (staging), load to Customer then load to Company via Gainsight Connect. If we have S3 as source, then directly I can load to Customer Info.
Photo of Sri Gali (IBM)

Sri Gali (IBM)

  • 100 Points 100 badge 2x thumb
Thanks Ashok for the ideas!
Can't wait for #1,4,5 to happen. There would be tremendous productivity benefits with these ideas. I believe some of them may be in the pipeline. But overall, I like the way Gainsight (You) take client feedback seriously.
Keep up with the good work!
Photo of Jitin Mehndiratta

Jitin Mehndiratta, Product Manager

  • 2,862 Points 2k badge 2x thumb
Thanks everyone for your inputs.

With Winter release, you will be able to import data directly from S3 file and be able to do transformations on that.

There is a new type of task(s3) in bionic rules. This task will read an s3 file (user will get the flexibility to define the format of the s3 file similar to export to S3 functionality that we have today) and load show fields based on the columns in the s3 file. Once the file is loaded, the show fields will be available for selection by user.

Once a file is loaded, show fields cannot be synced with the S3 file again. It always has to be a new task. 
(Edited)
Photo of Tanya Strauss

Tanya Strauss, Employee

  • 438 Points 250 badge 2x thumb
@Ashok - would this solution allow us to connect to a client's own S3 bucket? If so, that's a use case one of my clients would love to see. They are concerned about data at rest and outside their custody.
Photo of Ashok Dugaputi

Ashok Dugaputi, Employee

  • 1,490 Points 1k badge 2x thumb
@Tanya - Current release does not support reading from client's own S3 bucket but this is something being planned in the near term roadmap.
Photo of Jeff Kirkpatrick

Jeff Kirkpatrick, Champion

  • 23,868 Points 20k badge 2x thumb
I tried this today, selecting the "GainsightManaged" and clicking either Load Column Details or Preview, but I keep getting an error "Error occured while loading the file. The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 6C0DCF3B759BA46B)"

Meanwhile, S3 jobs in the Connectors tab are running successfully without issue.
Photo of Jeff Kirkpatrick

Jeff Kirkpatrick, Champion

  • 23,868 Points 20k badge 2x thumb




Photo of Kevin Ly

Kevin Ly, Employee

  • 3,104 Points 3k badge 2x thumb
Hey Jeff, 

Try like what I have in my dev which works.


I placed in the same input directory as you, except I have different file name. In case the picture looks a little small, may path is.

MDA-Data-Ingest/input/Export from Bionic-2018-02-20.csv
Photo of Jeff Kirkpatrick

Jeff Kirkpatrick, Champion

  • 23,868 Points 20k badge 2x thumb
Tried that too.  Gives me a little different error though.



PS I don't have an Export directory - is that necessary?

Your example in the S3 Dataset Task in Bionic Rules documentation doesn't show having to add all the folders in the path...just the filename (I took screenshots from the gif on that doc):





Photo of Kevin Ly

Kevin Ly, Employee

  • 3,104 Points 3k badge 2x thumb
Thanks for the update Jeff.

I see that you have a support ticket opened therefore I shall look into that ticket. Doc has been flagged for update due to the same reason you have come upon, FYI.

Best,

Kevin Ly
Photo of Jeff Kirkpatrick

Jeff Kirkpatrick, Champion

  • 23,868 Points 20k badge 2x thumb
Well dang it I was moving so fast trying to juggle multiple things I didn't realize I failed to select the Gainsight Bucket in the last attempt where I added the MDA-Data-Ingest/input path. 

Thanks for the catch.  And thanks for flagging the support doc to have it updated.  I was originally going off that doc which is why it was failing to begin with.  An explicit reference in the verbiage to say "You must enter the full path" would be good.
Photo of Zach Davis

Zach Davis, Official Rep

  • 1,740 Points 1k badge 2x thumb
I noticed that this no longer supports the MM/DD/YYYY format. for instance my dates are stored as 03/13/2018 and it is giving me an error unless I trim off the leading zeros. Is this a known issue? THis is a standard date format used by most US customers.