Beta

Data projects are all about trust and building trust is all about meeting and exceeding expectations.

I was shocked, scared and distressed. A week ago I just finished a new report showing individual performance of all the sales reps. Not even seven days later, the company used this report to made half of the sales team redundant.

It was a matter of life and death for the company and they had to act fast in order to survive but I had no idea what was coming and how critical my insights would be. I hadn’t realised the insights would decide who would keep their job and who would have to find a new one.

When I found out, I got quite worried. Was the report absolutely correct?

The nature of the Agile approach is that testing is fairly limited, if there are issues they will get discovered fairly quickly by the end users. Good enough and fast is much better than perfect and slow.

But there is a reason Agile is not used in high risk environments where an error might have catastrophic consequences. Ok, this wasn’t quite a catastrophic level but it was significantly impacting people’s lives.

I spent good half a day afterwards checking the report was error free. Fortunately that was the case but I decided to avoid similarly stressful situation in the future.

From that point onwards I added a beta flag to all my new reports and I forbade directors of the company to lay anyone off based on a report with Beta on it. It was fun.

In fact, I took it one step forward. I started to use the beta flag to turn users of my reports into beta testers. I taught them that when they see the beta flag it’s a sign for them to look out for bugs — data discrepancies, missing values, suspicious numbers — and report them to me to investigate and fix.

This completely transformed the relationship between me and my users. Instead of feeling attacked by each other we got on the same side of a table and became collaborators.

All that changed was their expectations. And all I needed to do that was to say: “When it’s in beta expect some issues there. I can’t find them all so I need your help so we can clean them up together.” It has worked perfectly.

I only remove the beta flag when the end users sign off a report. That means they are happy with the numbers and from that point onward they are not allowed to complain about data any more as they had their beta period to validate them.

As a result ownership, understanding and usage of reports skyrocketed.

Such a simple thing and yet so powerful.