BIG DATA - Q&A

How is Big Data different from other tools that were been used 5 and 10 years ago?

As you can see from this picture there is a lot of DATA been produce, which is growing at really fast rates. BIG DATA allows you to take into account ALL these information, Terabytes of information, which could is limited to other programs such as Excel.  Another difference is the input of External information that is available to companies, social media, blogs, etc. Which BIG DATA takes into consideration while other programs will find it more challenging to include.

How expensive it is to implement BIG DATA into a company?

The price varies depending on the industry, size, and sophistication of the information. However, research has shown that the ROI of investing in BIG DATA is large. According to IBM: “Plenty of customers are seeing tangible ROI using IBM solutions to address their big data challenges:

           

- Healthcare: 20% decrease in patient mortality by analyzing streaming patient data.

- Telco: 92% decrease in processing time by analyzing networking and call data

- Utilities: 99% improved accuracy in placing power generation resources by analyzing 2.8 petabytes of untapped data” [1]

Note: Remember to take into consideration that you must invest in security to protect your information.

How do I know what information is useful and what’s not?

There is a Human Factor associate to the implementation of Big Data and the Variety of information that is available. If you want to see the best results it is really important for you to Process the Data in-house in order for you to see positive results. If you plug random data you will get random results. You can hire a consulting firm to help you process the Data, but try to get the managers of every department involve in this selection part.

Can you give some examples of the uses of BIG DATA in other industries?

You can check some example in the follow link:

http://www-01.ibm.com/software/data/bigdata/industry.html

 References:

http://www.marketingpilgrim.com/2012/11/just-how-big-is-big-data.html

http://www-01.ibm.com/software/data/bigdata/industry.html

---

[1] http://www-01.ibm.com/software/data/bigdata/industry.html

SMALL DATA BEFORE BIG DATA

How did humankind evolve all the way to the Big Data Revolution?

At first there was the Age of statistical sampling  and extrapolation of Small Data…

… which although with some degree of bias, could give considerable support to essential predictive decision-making…

…but only now we have the luxury and the precision of interconnected massive data-sets that allows us to make accurate analysis and selections…

…although again with some degree of risks when trying to orient in the Big Data ocean…

In short, Big Data is a Revolution and a big step forward as it saved us from analyzing random samples full of individually biased errors by drowning them under huge data-sets. However, distinguishing useful data from useless data, building quick analytical processes and ensuring customer privacy represent the next challenges for Big Data statisticians. It is also important not to enthusiastically rely on data as something that never fails as it could turn into some significant unfair decisions such as refusing insurance or credit to a potential customer only because an algorithm says so.

Risks associated with Big Data

Every second social networking sites like Facebook, Flickr, Foursquare, Google+ etc. are collecting enormous amount of personal data about us. 

In theory most of the web browsers track our online web browsing behavior and are owner of this information. In fact the current regulation allows these companies to use the information as per their need, which lead to big privacy issues for end users and is a matter of great concern. 

In the video bellow, the specialist Rick Smolan discusses risks associated with big data.

Steps to build a Big Data Security Program:

a)    Setup a holistic cyber security strategy: Organization should setup a security strategy customized for the specifics risks, threats and needs applicable to them.

b)    Establish a shared data architecture:  Since big data requires data to be captured from multiple sources each having their own format and data encryption methodology using a single architecture with multiple level encryption will enhance the overall security and allow all the information to be captured, shared and analyzed in a logical manner.

c)    Migrate from point product to unified security architecture: Traditionally it has been seen that the benefit of unifying the individual data architecture for security analytics outweighs the cost of preserving individual architecture of existing point products.  

d)    Look for open and big data security tool: Organization should constantly invest in security products using agile analytic based approaches which will allow the architectural flexibility required to make quick changes incases of security threats.

e)    Leverage External Threat Intelligence. One should augment internal threat analysis with external threat analysis services. Often external intelligence feeds can prevent security breaches and allow companies the necessary time to fix the gaps.

 References: 

http://www2013.org/

http://www.emc.com/

http://www.informatica.com/

An innovative way that Big Data is changing hiring

If you thought that being tapped on LinkedIn for a new job opportunity was the extent of your professional footprint on the web, then you had better think again. There is a very innovative use of Big Data happening now in a field that you might not originally think of when Big Data comes to mind - Human Resources and Recruiting.

Here is a quick infographic summarizing some of the recent findings which Big Data is allowing HR professionals to remove biais from their decision-making process:

There are a number of new companies which are helping companies to fill their very hard to find niche Software Developer roles by combing the entire internet history of individuals and creating a list of candidates and a combined "score" of how well they might succeed int he role. One company I checked out was called Gild and their product is called Gild Source - here is the description from their website:

Built for hiring teams facing the challenge of identifying developers in today’s competitive landscape. Gild Source is tech recruiting software that leverages powerful data aggregation and analysis to provide you with comprehensive profiles of millions of candidates. Unlike job boards, professional networks, or standard social recruiting tools, Gild Source is powered by proprietary code evaluation technology and a team of data scientists that work to uncover the real indicators of developer talent.

In fact, this is part of a very fast-growing field called work-force science. Breaking down what were once "soft" and "gut-feeling" fields of HR and employee management and turning them into a hard science using the massive amounts of data available on the workforce and individual employees.

Here is a video from the Economist which will give you a great introduction to this innovative concept of applying Big Data to the HR field.



Makes you think twice about your existing web footprint (the collection of all the data that exists from all your previous actions (postings, photos, comments, accounts, etc) and how it might be seen by future employers!

So what are your thoughts about de-humanizing the HR field by using Big Data to try and better predict success? Certainly it raises a number of privacy concerns - but what about the fact of taking intuition or personal judgement out of the process? Is this necessarily a good thing or is there something to be said for such measures which machines and data will never be able to measure?



Sources:

http://www.nytimes.com/2013/04/28/technology/how-big-data-is-playing-recruiter-for-specialized-workers.html?pagewanted=all&_r=0

http://online.wsj.com/article/SB10000872396390443890304578006252019616768.html

http://www.huffingtonpost.com/phil-simon/big-data-a-big-opportunit_b_3032789.html

http://www.gild.com/products/gild-source-tech-recruiting/

http://www.nytimes.com/2013/04/21/technology/big-data-trying-to-build-better-workers.html?_r=0&adxnnl=1&ref=stevelohr&adxnnlx=1367519084-+9vOzaHP9i629H5S1KCYwQ