[Feature Request] Have total S3 Failure Files move into Error Folder instead of Archive on the bucket.


I am posting on behalf of a customer:



I noticed that given a complete failure of an S3 file upload, it gets transferred to the "Archive" folder in the bucket. I can see how that can be misleading if we were to believe successful loads only make it to Archive. It would be a good idea to move it to Error instead and give indication in the Execution History that file failed due to header issues or etc.
Kevin,



Here are the details on your / customer suggestion.



Archive folder: Archive folder is designed to hold the copy of the file Ingested irrespective of quality of the file. This is not the typical scenario of having one folder for Good and another for Bad quality records. This is very handy to reload the files with historical data and we can also to set the data retention period on top it.



Error folder: This is designed to hold purely error records. The suggestion given by you/customer is valid and Team confirmed to handle this soon (most likely in Spring Release). We will also plan to convey the same message in Execution history especially in the place of "Successful records" & "Failure records" about the file level failure message.