IDC’s Digital Universe study, “Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East,” monitors the growth of digital data, and shows that data continues to grow, both in its raw number and in its rate.  Even more so than a year ago, we are seeing the explosion of digital information, driven by social data, sensor data, and the Internet of Things, accelerate at an ever-increasing pace.

This proliferation of data represents huge opportunity for change.  As Anthony Goldblum, the CEO of Kaggle recently said in Forbes, “I’ll go into a company and say, ‘What data problems can we solve?’ We get blank looks. [When he asks, instead, what things can help a company lose money and make money, usually two out of three are] problems that data can solve.”

The explosion and availability of data demonstrated in the Digital Universe study is not just about having more and more information; it is an opportunity to drive change.  Here are three quick takeaways on this point:

1)      Data is growing fast.  The digital universe is growing more quickly than expected and will hit 40 zettabytes – or 40 trillion gigabytes – by 2020.  This growth is heavily driven by the increasingly huge amounts of data produced by the Internet of Things, sensor data, and social media data.  These three things are pervading many, many areas of our business and our lives, as evidenced by the recent Human Face of Big Data project. This has potential to revolutionize everything from the way we drive cars (think of Progressive Insurance’s real-time sensors that gauge how you drive and give discounts to safe drivers) to healthcare and personalized medicine (Human Genome project  and crowdsourced medicine) , or even education (crowdsourced education with Coursera, Udacity, and others) and government.

2)      Big Data = Big Opportunity.   As mentioned in the Digital Universe study, “Massive amounts of useful data are getting lost in the Digital Universe. For instance, only 0.5% of the Digital Universe was analyzed in 2010.” If Big Data is the new gold that people are mining for, this means that most of the data out there is waiting to be mined.   According to the study, only 3% of the potentially useful data is tagged, and even less is analyzed. This is the Big Data Gap: untapped information ready to have its hidden value extracted.   Consider for a minute how this breaks down for 2012 (data shown in exabytes):

 This means that nearly 895 exabytes this year were not analyzed, so there is plenty of opportunity for people to find creative ways to extract new value from this.

3)    As the data deluge grows, so does the skills gap.  This year’s Digital Universe study projects that by 2020, the number of servers will grow 10x and information managed by enterprise data centers will grow 14x, yet the number of IT professionals will grow by less than 1.5x, creating a huge technology skills gap.    As Paul Barth showed in the Harvard Business Review article “There’s No Panacea for the Big Data Talent Gap,” it continues to be difficult to find Data Scientists:

These findings underscore that the real challenge is not just where to put all of the data, but what to do with it.  Extracting hidden insights out of a massive amount of information requires better tools, powerful infrastructure, and talented people, such as Data Scientists.   As cited in the same issue of HBR,  DJ Patil and Tom Davenport talk about how to find and retain Data Scientists, which are a scarce commodity these days.  In fact, as I mentioned in my blog post after the Strata/Hadoop World conference, I believe we are still early in this adoption cycle and this will only get bigger, as more people wake up to the opportunity provided by Big Data. As a result, the demand for Data Scientists will only continue to increase.  Last year at Strata, every speaker ended their talk by saying that if anyone was looking for work as a Data Scientist, they were hiring.  However, because of the shortage of available Data Scientists, we are realizing that most companies are, instead, trying to grow their own data science teams. EMC recognized this emerging trend, and as such, created the industry’s first “open” course focused on Data Science & Big Data Analytics to help fill the gap.  It is also critical for corporations to join with academia fill the gap and skill the next generation of Data Scientists.  Programs like the EMC Academic Alliance, which recently announced more than 1,000 partner colleges and universities to date, are a great example of this.

In our Data Science & Big Data Analytics course (also mentioned in Davenport and Patil’s HBR article referenced above, “Data Scientist:  The Sexiest Job of the 21^st Century”), we teach people a structured, lifecycle approach to Big Data projects, how to perform advanced analytics, and also how to communicate findings.  My point is that it’s not enough to help people become strong at writing algorithms; they also need to be good at communicating – either verbally, in writing, or with data visualizations.   As someone commented in a recent class I taught, “Data science is a team sport.”   It’s not that the need for reporting and a solid process go away, but you need everyone working together to do the things that need to get done every day.  You also need vision to see the world differently and to find hidden opportunities in data.  This is the real value provided by Data Scientists—they help extract value that enables organizations do things they never could before.

 In other words, this is not just about the shortage of Data Scientists, which is growing more challenging all the time. It is also about the need to enrich the skills of more traditional IT roles, such as Database Administrators (DBA), Business Intelligence Analysts, and Data Engineers, as well.

InFocus » David Dietrich