Failure notifications on Real Time rules

How...is this not a thing? 

Hi @darkknight 

Real Time Rules runs every time there is an update by any one on Company/CTA. Imagine if you have 100 updates, this would result in 100 emails. For some customers, this number is in order of hundreds of thousands. This makes what we have for notifications in Rules or Connectors not work 1:1 with Real Time rules.

Some questions for you -

How frequently do you monitor for these failures / how soon you need to know about this? Assume you see a failure report once a day, is that a good frequency?

Email vs In-App Notifications - Is there a preference? With In-App, we might be able to group them better (One notification per rule). Or would you prefer to see all the failures in one place in Rules Engine? 

@rakesh 

For those 100 updates in your example, am I expecting every one of them to fail?  I would hope not.

Then it wouldn’t be 100 emails - it would only be the number of emails = number of failures.

I need to know when anything fails. Period end of sentence. If I don’t get a notification when something fails, then I have to manually check for failures. Because if I don’t, and something gets out of whack between systems, then I’ll have VPs breathing down my neck asking why Gainsight didn’t do what it was supposed to do, and depending on how long it takes for them to reach out to me, I may not have any log details to go check because it’s da 16 since it was supposed to have happened and the logs rolled off.

I need to know immediately when something fails, particular where data synchronization is involved (i.e. Gainsight couldn’t write to SFDC because of a record lock).  Subsequent processes may depend on the accuracy of the data.

I send my notifications to a Slack channel. It’s quicker/easier to reference and to collaborate on vs. in-app notifications or emails.

I think some type of failure notification is super important, and I do see the concern of possibly getting tons of alerts.

Is it possible for some type of middle ground, where you get an alert either per rule (or per rule action) that there are one or more failures, and either the alert itself contains a list of all said failures or points you exactly to where a list is and until you dismiss that particular alert you wouldn’t receive another unless you configure it otherwise? Your list would need to update in real time if more errors are happening so you can see more than one failure per alert.

Obviously that’s not a fully developed idea, but maybe opens a third door here.

What I think as an admin at least for me would be important, would be:

• The ability to receive some type of alert for any failure
• Some way to throttle the number of alerts for the same rule in a certain amount of time, while not losing visibility on the amount of errors in any one alert. (e.g, if there were 12 errors but I only received one alert, I would still know there were 12 errors on that rule)
• I think for the second point to be possible, there’d need to be some way to ‘acknowledge’ a failure since there’s no ‘resolve failure’ option for rules, it’s basically just fix and re-run it.
• Since the rules are real time, then the errors should be real time, at least that first one. As @darkknight mentioned the result of the action may have dependencies so being able to intervene in a timely manner could be critical. 

Hopefully that helps.

@bradley I don’t know Bradley, I think this may be the first time we somewhat disagree on something.  That seems awfully complicated - moreso probably than just a simple notification process (something akin to Connectors posted below).  And the occurrence of multiple failures doesn’t necessarily mean its same issue that caused each failure. 

I do believe admins should have the autonomy to choose real-time notifications for any failure or maybe if they want a “daily digest” as well as the ability to identify the type of failures to notify, but I wouldn’t want the delivery of an MVP solution to hinge upon more granular throttling capabilties.

@bradley I don’t know Bradley, I think this may be the first time we somewhat disagree on something.  That seems awfully complicated - moreso probably than just a simple notification process (something akin to Connectors posted below).  And the occurrence of multiple failures doesn’t necessarily mean its same issue that caused each failure. 

I do believe admins should have the autonomy to choose real-time notifications for any failure or maybe if they want a “daily digest” as well as the ability to identify the type of failures to notify, but I wouldn’t want the delivery of an MVP solution to hinge upon more granular throttling capabilties.

 

I don’t know that we disagree so much as I was trying to think of some alternative if the options were “NO alerts”, “one aggregated alert per day” or “alerts for every interaction” (the last of which seems like it wouldn’t happen as it was all but cited for the reason we don’t have alerts today). Never said they were good ideas either :)

I think that level of flexibility for alerts like in the screenshot would be rad for rules in general too!

@rakesh any update here? I’m having to check Real Time Rules multiple times daily to keep an eye out for failures (and yes, there are failures with indeterminate reasons, i.e “"Failed to query data from Query API").  

I shouldn’t have to manually check for failures. 

Hi are there any updates here? I recently had to create a separate bionic rule to “catch” the records that failed with the real-time rule (we have a real-time rule to push the CSM value from Gainsight to Salesforce). This seems to kind of defeat the purpose of the real-time rule…it is nice that most of the records are pushed over instantly, but we need to be alerted when these fail.

we have this as well - we have nightly rules that run to ‘clean-up’ any misses from the RTRs.

Writing in to share my enthusiasm for this request as well. Failure notifications are necessary for RTR.

Why is this still only “Under Consideration”? Failure notifications are mandatory for rules

To add to this, it would be good if someone could confirm what is keeping Realtime Rules moving from beta to GA and what the expected timescales are for this

And when we will have access to more objects, such as SPs and CTAs.

Success lans, I think we have CTAs today.*

@Wayne we do have CTAs, but only limited actions. No “Custom Updates” which are sorely needed.

Success Plans needed too!

Error logs could also be stored on a Gainsight table that we have access to. Much like email logs. I know some admins like to create CTAs for example when an email bounces. Why couldn’t we do the same for the RTR failure log? There there is also some flexibility. You could have it on an admin dashboard, or create a CTA rule that notifies the admin (not exactly immediately) when there was a failure with a given RTR.

Again, immediate notifications are ideal for most folks, but it would be also nice to have options for admins who don’t want instant emails.

@rakesh - Any update on this? Having failure alerts should be mandatory. 

This would be super helpful to us… we use real-time rules to update fields in Salesforce.
But because there are other systems trying to update Salesforce at the same time, these rules often fail.
The only work around for this today is checking the rule execution logs daily… having automated failure notifications would be a huge time savings to admins.

if you don’t already, it might be worth doing a nightly rule run (or more often if you want) of “regular” rules with the same type of criteria so if any RTR failed, you can correct them regularly. This is what we do with our RTRs that write to SFDC.

Hi @bradley!

Thank you for the idea! We currently do that for most of our GS to SFDC fields.
However, a few of those fields are bi-directional. So we can’t mass update from GS to SFDC if it doesn’t sync up with our SFDC to GS sync or we are risking overwriting old data back into the field.

I see this is changed to “Discovery” - does that mean it’s actively on the roadmap now?  Do we have a target ETA?

 @anirbandutta may you help us get an update from PM here? Thanks!

Yes… the earlier ‘Under Consideration’ stage is now ‘Discovery’.

@rakesh would be able to share light on this request.