this post was submitted on 29 Aug 2023
35 points (100.0% liked)

Programming

423 readers
13 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 1 year ago
MODERATORS
 

Let’s share stories where your automation efforts have been rejected and you can’t quite understand why! Here’s mine.

you are viewing a single comment's thread
view the rest of the comments
[–] otl@lemmy.sdf.org 2 points 1 year ago (1 children)

Thanks for sharing. I did a bit of work for a NOC and know exactly what you mean about letting real work slip through your hands. I wasn’t directly responsible for managing the alarms, but it felt strange to be writing software streamlining the workflow. All the time I spent I felt like I could have just helped the technicians actually solving problems they faced in their day to day - to stop the alarms going off in the first place!

To be fair, most of the work that you have to do in a NOC is total bullshit. About 30% of the time you will be working on technical issues, and for most other people in the NOC, that would mean escalating the technical issues to me. Unfortunately, I had to earn the stripes, which means I had to work harder than everyone else, which meant doing their work as well as handling all escalations. Eventually, I was promoted to a supervisor for my efforts, but I did not want to be in a managerial role.

The real bulk of NOC work that is tiresome is the amount of alarms that are unnecessary. Managing SNMP is a nightmare, and configuring it properly involves a deep level of engineering knowledge. You can either tune the alarm board to only show certain alarms(which means parsing through many alarms to find out what is necessary and what isn't), or you make sure that devices that are onboarded are configured locally for what SNMP traps they will alert for. Typically, the devices' SNMP settings are not configured, so all alarms get sent to the SNMP server, and the SNMP server was never tuned to know which alarms it should show or it shouldn't, so there are alarms which don't really "mean anything" and alarms that "could potentially mean something if it's correlated with this other alarm," but most of the work is sifting through so much shit, to then have to troubleshoot a network issue for a network that was never documented in the first place.