​021 Code Coverage Scandal

Vanilla Pop: So how about it team? Can we do TDD for a Sprint and see how it goes?
Horst: I’ve backed off my no micro test policy. Ja? So I’m open to letting the team decide how to deliver to me features and quality.
Code Dog: But I don’t believe in TDD. You can’t write tests before you write the code. You what? Really? Well no one really does it. Come one. After a few weeks, you’ll go back to what I call, Bug Driven Development. It’s what all of us debugger ninjas do.
Architect: I’ve changed. My consoler has advised me to try and have a more flexible mindset. I’m trying to be a better man. To be less bossy. To let teams decide rather than me order them about. It’s kumbiyah, everyone! Let’s sing. No? Ok then. So Mr Pop is right. He’s shown me the way. Now here is the thing. We all need to start doing this. The management wants us to lead the others and achieve 100% code coverage with micro tests. The VP has signed off on it. I’ve got a Continuous Integration server running them. Now we have to do it! It’s the law.
Good luck.
Jose: (whisper) Well, that flexible mindset lasted about 5 minutes.
Junior Joe: (whisper) Would you believe that’s a new record?
Code Dog: Come on! This is not happening. People aren’t really doing that. That’s what QA is for.
Jose: I don’t know. Vanilla Pop has done some nice work. Have you looked at those tests?
Code Dog: Well… no. That’s Pop’s thing.
Junior Joe: I’ve seen the light. I’m on board.
Vanilla Pop: thank you.
Architect: Vanilla Pop and Junior have got the coverage up to 40%. This is a small new single page app. With Code Dog also doing TDD, by my calculations we could get to 100% by this time next week. Let’s get it there boys!
(Time passes.) Tuesday, 50% Wednesday, 60%, Thursday 65%, Friday 85%, (go! Go! Go!) Monday 90%.
Architect: not bad! I’m quite impressed with the turn around from Code Dog. He’s really caught the TDD gestalt. In fact, the code areas he works in have 100% code coverage. So I’ve nominated him for an award.
Code Dog: (shocked) wow. Thanks!
Devs: (pat Code Dog’s back)
Junior: good work,
Vanilla: I’m proud to know you,
Jose: not bad for a developer
Vanilla: He really turned it around.
Horst: Was ist dass? Why is it when I swipe this way I no longer see the screen? Ja?
Code Dog: (embarrassed) I know what that is. I’ll fix it with a bit of debugger foo.
(Door closes)
Jose: so that’s weird. His code has 100% code coverage but sounds like he got a bug.
Junior: perhaps it’s at an integration point? I reviewed his code and told him to stop putting I.O. calls in his view layer. You can’t micro test that without a lot of mock objects.
Architect: where’s Code Dog? I want him to refactor his code a bit. Maybe add caching. His code is doing too much network bandwidth.
Vanilla: he’s working on some problem Horst had found.
Architect: What? Impossible! His code has 100% code coverage. Look at this beautiful code coverage report.
Vanilla: yeah. That is a wonder. Junior could you bring up his micro tests.
Junior: Ah ha!
Vanilla: Oh my dear. The ghost of Alan Turing is turning over in his grave.
Jose: You see Pop? He’s not a mutant developer like you too.
Junior: would you believe that he has 50 micro tests that assert nothing? Now why didn’t I think of that?
Vanilla: Those tests are fakes.
Architect: that man is a monster! He’s a TDD cheat!
Jose:
Oh my god! He finally really did it. You manic! You blew it up. Oh damn you! God damn you all to hell!
Narrator: code coverage is a great way to get a lot of code analyzed to see how your team is doing with automation. But if over incentivized or policy forced, it can be easily faked. Code coverage measures what lines of code were executed during a test run, but code coverage tools can’t tell you if the test automation is doing its job. Think of a function that returns the sum of two numbers. If the function is executed while measuring code coverage, it will report 100% whether or not it’s test check that the return value for one plus one does indeed equal two.
Pair programming or code reviewing the test code will put a stop to test fakery. If you’re going to review code, it’s actually more important review the micro test code than the product code. The micro tests spec out the developers intent. If they are high quality, then one can infer that if the tests pass then the product code is high quality too.
​