Menu

Big Data

What is Big Data?

Big data refers to data that is so large, fast or complex that it’s difficult or its too expensive to process using traditional methods.

The Big Data Technology is associated with 5 V’s namely;

Volume

It refers to the size of Data storage and processing. The Big Data platforms handles few dozens of terabytes to hundreds of petabytes of data.

Velocity

 It refers to the to the pace with which the Data is generated and processed. Many big data platforms even record and interpret data in real-time. 

Variety

It refers to the type and nature of the data. Big data can include structured, unstructured, semi-structured or combinations of structured and unstructured data.

Veracity

It refers to the reliability of the data i.e the data quality and the data value. Big data must not only be large in size, but also must be reliable in order to achieve value in the analysis of it.

Value

It refers to the worth in information that can be achieved by the processing and analysis of the data.

How does big data analytics work?

Data analysts, data scientists, predictive modelers, statisticians and other analytics professionals collect, process, clean and analyze growing volumes of structured transaction data as well as other forms of data not used by conventional BI and analytics programs. 

Here is an overview of the four steps of the data preparation process

1. Data collection

Data professionals collect data from a variety of different sources. Often, it is a mix of semi-structured and unstructured data. While each organization will use different data streams, some common sources include:  

  • Internet clickstream data 
  • Web server logs 
  • Cloud applications 
  • Mobile applications 
  • Social media content 
  • Text from customer emails and survey responses 
  • Mobile phone records; and 
  • Machine data captured by sensors connected to the internet of things (IoT)

2. Data Processing

Data is processed. After data is collected and stored in a data warehouse or data lake, data professionals must organize, configure and partition the data properly for analytical queries. Thorough data processing makes for higher performance from analytical queries. 

3. Data Cleansing

Data is cleansed for quality. Data professionals scrub the data using scripting tools or enterprise software. They look for any errors or inconsistencies, such as duplications or formatting mistakes, and organize and tidy up the data.

4. Data Analyzing

The collected, processed and cleaned data is analyzed with analytics software. This includes tools for:  

  • Data mining, which shifts through data sets in search of patterns and relationships  
  • Predictive analytics, which builds models to forecast customer behavior and other future developments 
  • Machine learning, which taps algorithms to analyze large data sets 
  • Deep learning, which is a more advanced offshoot of machine learning  
  • Text mining and statistical analysis software  
  • Artificial intelligence (AI) 
  • Mainstream business intelligence software 
  • Data visualization tools 

Examples of Big Data Technologies

HADOOP

APACHE HIVE 

APACHE SPARK

APACHE KAFKA