header
Deprecation Planned

Email notifications for S3 drops should read better

  • 3 December 2018
  • 4 replies
  • 60 views

Userlevel 4
Badge
When I drop a file manually in S3, sometimes I forget a small step and the file fails. That is fine, but the email notification says successful. It should read better to say the error. I have to go look at the execution logs to see what was the issue. It is misleading and hard to filter out errors vs successes.


4 replies

Userlevel 7
Badge
Hi Anthony, 

 

Are you performing the task from Rules engine or Connectors or do you face this issue in both the places?

 

What kind of steps are you missing which caused these errors?

Userlevel 4
Badge
Hi Sai,

This is only in the Connectors not Rules Engine. The setup was done by someone else, so sometimes I miss a step such as a date creeps into the file that isn't formatted correctly. Our raw data files need manipulation before dropping them in. I usually fix them using other code but its not always perfect.

Userlevel 5
Hi Anthony,

I would recommend using Rules engine to ingest data from S3 for a variety of advantages as listed below:

Advantages of using Rules Engine for S3 ingest

1. Linking data ingest with the business processes: S3 Dataset task helps in performing business actions only after successful data ingest. E.g. If the file does not arrive on time, the rule will fail.

2. Eliminating the need for storing raw data -

(a) If a customer has concerns over data-security and data-storage, they can just enable data for processing and create actions on top of it.

(b) Ingest selective data and aggregate during ingest process itself: For example, if the CSV exports (in customer's bucket) has data for all page views, a business rule can filter data from this file using advanced logic.

3. Advanced lookup during ingest: The functions and merges in Bionic rules help in performing complex lookups and resolutions. E.g. resolving `Email Address -> Domain -> Account ID/Name`

4. Increased limits for file size: S3 Dataset task will support file as large as 6 GB in size.

Regards,

Jitin

Userlevel 4
Badge
Thank you for the reply and helpful feedback. I was hoping more of someone actually making fixes than make sense instead of redoing our current process beacuse notification messages are not very good. :(

Reply