Notification when data file doesn't show up on S3 as expected
CS Rules & Permissions
We have several files that, (generally) with great consistency, show up on our S3 input folder sometime while I'm (generally) asleep. And then sometime in the past few weeks there was a change to an underlying report server and we stopped getting some data. A CSM noticed it when trying to look into some customer usage.
Idea: As part of the ingest setup I would like to specify a "notify if not received by" time.
Question: Anyone have a clever way to detect this lack of data (other than, you know, me looking at a dashboard ever day? 🙂
Idea: As part of the ingest setup I would like to specify a "notify if not received by" time.
Question: Anyone have a clever way to detect this lack of data (other than, you know, me looking at a dashboard ever day? 🙂
Sign up
If you ever had a profile with us, there's no need to create another one.
Don't worry if your email address has since changed, or you can't remember your login, just let us know at community@gainsight.com and we'll help you get started from where you left.
Else, please continue with the registration below.
Welcome to the Gainsight Community
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
We just added this notification so you could see confirms when data is loaded (or if a data load fails). Not exactly the same but could help in your use case.
Denise
Users receive two messages at the endpoint:
As Jeffrey DaSilva suggested, if you you use Time based scheduler, when S3 job is triggered, since It can't find the file in "Input" folder, it throws an error. In the Winter release, as Denise mentioned, you can also have Webhook notification to trigger your ETL jobs directly In addition to email notification.
But "Post file upload" does not give the same flexibility since the S3 job gets triggered only after the file is uploaded. But "Post file upload" is usually preferred as it can handle the scenarios of late arrivals of the file where scheduler works only as per the specified time.
Unfortunately with the current product offering the decision lies whether to give importance to late arrival vs no arrival scenarios to handle.
So In order to support this from the product side in the future release, here are a couple of Ideas that I can think of immediately, please feel free to share your thoughts.
1) Give an additional option to current "Post File upload" to "notify me if the file does not arrive by <<Specified time in a day>>" Here similar to scheduler setup, we will have to capture the frequency of the file upload happens so that a notification can be triggered only when the file has not arrived as per the schedule.
2) Expose the Job Logs object: We've also received requests around exposing the Job logs object so that trend line reports can be created to understand the performance of each job (Successful & Failed records) or report on error types over period. So you can get the report of each job last triggered. In addition to reporting, one can also consider to define a rule (may be a OOTB rule) to notify for all the data Ingest options (S3, API, Mixpanel, Segment and GA).
Any recommendations on either of these options or any other ideas?