0

We Have an Alerting Problem

 2 years ago
source link: https://blog.serverfault.com/2011/08/03/we-have-an-alerting-problem/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

We Have an Alerting Problem

Hello, our name is Stack Exchange and we have an alerting problem. It hurts us, our friends, and our family. We are not sure how we got here. Sure, we get some extraneous alerts, but everyone does right? Then one day we woke up and had an inbox full of alerts. We wrote it off. We told ourselves that it happens to everyone. But then it happened again, and then again. We don’t want this. We don’t want to live like this anymore. We are ready to pick ourselves up. We are ready to face this problem and live a new, and better life.

Don’t know if you have a problem? Here are some of the signs:

  • You get alerts that you just don’t care about, because of this you maybe don’t see the ones you do care about.
  • The more serious ones wake us up in the middle of the night when they don’t need to because someone else is already dealing with them.
  • When something major happens your inbox is flooded.
  • You set up email rules to handle them.
  • You are ashamed.

If like us, you have an alerting problem and you have admitted it, I believe finding the righteous path starts with one rule:

Every Alert Requires Action

Every alert requires action. The problem we have right now at Stack Exchange is that alerts don’t require that we do anything. If we are to address our alerting problem — I believe this, more than anything, needs to be fixed. When I mean every alert, I do mean every alert. So what sort of actions can we take:

  • If it is real problem and you are dealing with it — acknowledge the alert.
  • If it is a false alert, acknowledge and adjust the threshold level
  • If the alert was a flood of alerts, acknowledge them and set up dependencies.

In order to do this we need a few things. We need a system that allows to effectively acknowledge alerts without too much friction. We need cooperation from everyone to use this system once it is in place. Lastly, we need to accept that we don’t have the power to cure our alerting problem. We can, however, through constant vigilance, get it under control.


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK