The Data Iceberg

Embarking on a project to deliver data visualisations? Data Iceberg

Do not underestimate the data iceberg which will capsize your delivery.

In my experience, getting your data right is the biggest challenge, building the data visualisation is the easy bit!

Common issues in corporations include:

  • You need to jump through several “hoops” to get access and connect to the data. This can end up taking several months if you need to get changes into a release cycle.
  • The data you need is actually spread across several databases and you need to align and bring all this data together.
  • There are often a number of “off-system” data sets you need to uncover and map into your data set, eg budget holder mappings, reporting hierarchies.
  • There maybe issues with the quality of your data, either data is missing or been incorrectly/inconsistently entered. This might even require you to enact some level of business change to get the data sorted.
  • You will need to prepare the data into a sensible structure to visualise with. Consider how you set-up your dimensions and measures, particularly with respect to time dimensions and hierarchies. Often I get prior periods pre-calculated in the data source so it simple to reference when building the visualisation.
  • An iterative situation, but generally you will want to push calculations and business logic into your dataset so that you don’t take the performance hit on the visualisation side.
  • Business users apply additional adjustments or logic on top of the data before presenting it. Often this can be to workaround limitations of the source system or just to deal other situations they have accommodated for over the years. Be prepared to dive into end user Excel spreadsheets and unravel something that was set-up years ago and the original creator isn’t about anymore.

There is no easy solution to dealing with these, but if you go in preparing for such issues and anticipating that you will find them (for you will!), then you can be more realistic with your approach and timescales.

Best of luck and I hope your delivery doesn’t sink!

 

One thought on “The Data Iceberg

Leave a comment