SQL-based solutions have been evolving for decades. They become more powerful, handle expanding data volumes and answer questions in ever-growing complexity. All of this built on the simplest and most intuitive of data structures, right?
The world of data analytics in early 2018 is pretty exciting. We are witnessing the emergence of commercial AI applications, Machine Learning is commonplace, and - after over 40 years - we've established SQL as a universal language.
SQL is everywhere
The label "universal" may feel too strong, but we see it everywhere in big data analytics. Every major organization on the globe leverages SQL. Year after year, SQL ranks #1 among in-demand technical skills. The data market supports a multitude of SQL engines, hundreds of SQL-based BI tools, and is arguably a key feature enabling Hadoop success in the enterprise.
And let's make sure we give credit to the underlying data structures. After all, SQL is just an interface, an interface to the TABLE. Every SQL database implements a table. Some might argue that the CSV, the simplest of all tables, is the most common data structure on the planet. Rows and columns. It's so simple! So intuitive! Right?
Tables are the intuitive dataset, right?
Why just the other day, I asked my 8-year-old about her day at school. She smiled and handed me a table. I immediately understood her day - pizza for lunch, too cold to go outside for recess and she received a love note from a secret admirer. Those rows and columns made it so clear, and such an efficient way to communicate!
Okay, maybe not. I really don't know a single grade-schooler would hand you rows and columns to describe her day. In fact, I don't know anyone of any age that would answer the question this way. Why? Tables aren't really that intuitive. It's a MACHINE dataset. A very convenient way of organizing data that was easy for our programs and tools to ingest.
Some of us are quite adept with tables. Like learning a new language, we've spent years training ourselves to think of data with a rectangular shape. We've built very popular tools to help us create, manage, store and query these tables. Despite all of that success, the vast majority of us don't naturally think in rows and columns
Since the beginning of recorded history…
But there is a more intuitive data format. For thousands of years, since before the invention of paper, even back to paintings on cave walls, the human race has been sharing its history, knowledge and ideas through STORIES. Even my fictional example about my 8-yr-old was a STORY.
It is through stories that we establish relationships and convey meaning to one another. So why should we manage data any differently?
Stories in the board room
So let's embrace this, for a moment, in regards to our executive audiences. Storytelling as a presentation format is not a new thing, even in the analytics space. Effective communicators have been using stories in their presentations for decades. The team works a project, one person stands before the executive audience, and she tells the teams' story supported by charts and graphs. Some questions are asked, the team answers. Decisions are made. Fantastic job!
But some of us are encouraging something more. Our executive audiences in the C-Suite are becoming more hands-on. Increasingly, it's not enough for this audience to hear the story of our analytic efforts. The executive may not need your personal story of how hard or easy the project was. More and more, they want to interact with the story within the data.
What does this look like, to share analytics as a story? Well, maybe we should be analyzing stories.
The right tools at the right times
To be clear, I don't believe that we should abandon tables entirely. I think they have their place. We can't throw away decades of tooling and analytic strategies just because they aren't perfect. But I also don't think it's cost effective to translate naturally occurring event-based data to tables, perform our analytics, and again translate it back to a story format for presentation purposes. Especially not when every data transformation interjects someone's personal interpretation, assumptions and bias of some kind.
We need to stop bending our brains to a format tuned for a machine, and let's have the machines do the work to make it easier for us. Isn't that a machine's purpose?
Building the story analysis tools
As we begin working with stories, we find that data has shape. Is it a table? Sure - if it has to be; probably several tables. Is it time series? Maybe. A story is all of those, and something else. Natural language processing, graph databases, photo and video classification. Each of these tools use mere shadows of the full object. Stories are complex, multi-dimensional objects.
Our industry needs better tools. Tools that our analysts (not just data scientists!) and our audiences - including your executive audiences - can ask questions of and to provide shape to their business stories without leaving that intuitive data format. Filtering, sequence analysis, noise reduction, performed directly on a truly HUMAN dataset.
Having worked with stories (a.k.a "journeys") for years, I don't believe we have the right story analytics tools … yet. We need to build them. We need our machines - our software, applications and programs - to finally bend themselves to our intuitive frameworks.
It's going to take some work. Some of us have started. Join the Journey Sciences movement.