HI Team,
Posting these 2 ideas on behalf of a customer, Please review and update here if these can be included in our road map.
- Currently our scheduled job runs by taking the last job run datestamp as the start date and the scheduled date as the end date. So here according to their use case it would be great if we use the last loaded date time for the particular field as the start date in the next schedule run because at the time of data run they have 1 hr of data missing which will be loaded into their snowflake later. As of now we have suggested them to create one system generated datetime field in their snowflake instance which can be used as a last modified timestamp.
- Is there a possibility to introduce all data sync as a scheduled run ?
We can sync to discuss on this further.
I can answer on the 2nd question.. When we talk about scheduled job runs of all data , it will result into scheduled syncing of all data which sometimes can be > 200k records . When data sync for such hugh records happens, it usually takes lot of time to execute, which will result into eating up lot of infra and will also eats up time for other jobs .
Also, syncing all data one time and then syncing delta data via schedule job runs will solve most of the use cases.
But yes, having a manual option to sync all data is better option so that admins can sync all data on as per their needs. which is already present in product
Hi@inagar thanks for addressing the second part. I’m the customer that Jitesh wrote about. I understand the concerns around “All Data” jobs taking long and eating up infrastructure. It’s a valid concern as it could have wider impact both for the customer and for Gainsight.
However, it seems that “All Data” is the only type of job that actually looks at the “Unique Identifier” fields. Currently Gainsight only uses its own “GS Last Modified” field when starting a run. Whereas the “All Data” looks at the field specified by the user as “Reference for Last Modified”.
Is there a way to get around this limitation in your opinion?