Degrees and certs are poor indicators of performance

Degrees and certs are poor indicators of performance

I just love certifications and degrees. They make excellent kindling.

Back when I was working for a school district, one of the "tech people" at a certain elementary school wanted more from life. Apparently, the "network engineers" working at the district level with their MCSEs seemed really awesome to him, and he strove to emulate them. He decided to get one as well.

He wanted to take the shortest path to this sort of thing, and so enrolled in a "boot camp" which promised they would be able to pass their exams after a week or two of "intense training". Whatever they did clearly left him able to pass his exam, since before long, he was a newly-minted MCSE.

Not long after that, he quit the school district and went on to a "real job" by using his shiny new cert to get his foot in the door. Maybe two weeks into his new job, one of the "network engineers" who sat near my desk got a phone call. It was this guy, calling from his new job. The two of them talked for a bit, and when it was over, my co-worker turned to me and told me what had happened.

"Hey, it was so and so. He ... asked me how to partition a hard drive in NT."

So, somehow, this guy had managed to get a certification and then get a job based on it, but didn't even know the basic things required to get a system going? Did they even do OS installs in those classes or tests? How could you possibly claim to be an expert without knowing about partitions or at least having the ability to figure it out yourself?

By way of reference, I was a thoroughly anti-Windows bigot at the time, and refused to touch them unless absolutely necessary, but even I could figure out how to set up a new disk. One time, I was the only person around with admin powers and slapped a new disk into a machine for the Windows people. Even though I had no idea where the tools lived or how they would work, I had no problems figuring it out.

But okay, it's easy to pick on MCSEs from the 1990s. Everyone did it. So now I'll turn up the flames a few notches and tell you about someone who had gone through multiple Computer Science degree programs. By that, I mean he had secured a BS, a MS, and even a Ph. D. in CompSci. I only found this out after trying to figure out what was wrong with this guy.

I had been working with him on a project and had run into some truly strange problems getting him to understand some Unix basics. I wondered what his story was and decided to do a search for his username in the vast piles of random mail stored in my archives. I subscribed to a whole bunch of internal mailing lists which were auto-archived for my own perusal later, and figured I might find something.

Oh, did I ever. I discovered that this guy was a regular poster to a particularly interesting mailing list which existed for new employees. It was a mailing list where "RTFM" was off limits, since they were genuinely new and needed help getting set up with all of the new things they had to learn to be effective at this company. In other words, if you asked a question, you'd actually get a non-snarky and honest answer, because people assumed they were genuinely helping you.

At this point you're probably thinking that this guy was one of the people who posted a bunch of replies and helped others out. Nope! Indeed, this guy was quite the opposite. He posted tons of questions for dumb things that anyone could have figured out by searching the intranet pages. There were plenty of pages which would expand an acronym or explain what various kinds of Secret Sauce did. You just had to dig around a bit.

This guy would just go straight to this mailing list with his questions. Somehow, nobody noticed that he kept mailing it and couldn't possibly be a new employee any more. Apparently the "keep it nice" rules really got to people, and they had turned off their suspicion circuitry.

One of his questions was classic. He had a unit test which wasn't working properly. It was trying to compare two variables, call them "a" and "b", and it wasn't returning true. One of them was built from one calculation, like "a = 10 / 2", while the other came from another direction, like "b = 2 * 2.5". It wasn't exactly that, but you get the idea. He couldn't figure it out.

"a" and "b" were floating point numbers.

Somehow, with all of his computer science classwork and then immersion in the field at this company for 2 or 3 years, plus who knows what else before it, he had never heard about the wonkiness of how floats are stored. Even if he had missed out on that, you would think that someone who was in charge of writing a bunch of testing code which dealt with floating point numbers would have encountered the caveats about trying to compare floats (or doubles, or whatever).

Another time, I ran into a different sort of bug in some of his Python test handling code. We had this antagonist process which would "warm up" the backend caches with a few queries before it started the actual test. His code parsed the stdout from this test (sigh...) and basically looked to make sure the query count was greater than some value. This is because it was seeing things like this:

warming up...
query 1000: (result)
query 2000: (result)
query 3000: ...
...
query 100000: ...
warmup DONE
running actual test...
query 1000: ...
query 2000: ...
query 3000: ...
...
query 100000
query 110000
...
query 500000
test DONE

Rather than looking for "warmup DONE" or "running actual test" or anything of the sort, he just ignored any number below 100000 since it might be part of the warmup. Or, well, at least, that's what he thought he was doing.

The actual Python code looked like this:

for line in lines:
  # ... [split line into query_count and result data] ...
 
  # ignore lines which are part of the warmup period
  if query_count < 100000:
    continue
 
  # graph the rest
  graph_data(query_count, result)

Trouble is, "line" was a string, and whatever he was doing to split it up meant query_count was also a string. That made it actually do this comparison: '1000' < 100000. For those of you fortunate enough to not have to know what Python does in this scenario, I will explain:

$ python -c "print '1000' < 100000"
False

That's just how Python rolls. If you don't mind your types, it may bite you.

Since "graph_data" was being called twice for query_count values between 0 and 100000, the resulting graph had two Y values for the X values between 0 and 100000 as a result. Actually looking at the graphic showed the usual dot, line, dot, line thing going on, then it would skew all the way back to the left and would start over. Somehow, nobody else ever noticed or cared.

If this person seems familiar, it's because he was the star of my story about runaway benchmarks from last summer. Now you know why I was trying to figure out what made him tick.

Am I saying degrees and certificates make you a bad person? No. I'm also not saying they make you a good person, either. Rather, they seem to inhabit a completely different axis. You might encounter a complete idiot with tons of wallpaper or a genius with none or any other mix. You just can't tell one from the other with any reliability in my experience.

Now, if you have a company which hires and promotes based on the one measurement thinking that it's some predictor of the other... well, you can guess what will happen then. Or hey, maybe it already has happened and you're starting to see the results on the outside world now. Wait and see!