Understanding Big Data

In Simple words Big Data is "Huge Volume of related/non-related data"


Wiki's explanation

Big Data usually includes data sets with sizes beyond the ability of commonly used software tools to capture,curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set.

What is the need of Big Data?

If in case a continuous monitoring and analysis of data obtained from a source is required then that is the place for Big Data. eg: when a rockets propels to the space, each and every second the data about the condition within the rocket "Pressure, Temperature, etc" has to be monitored" if in case there is a slight change in the condition other than the accepted limit the control room has to be intimated immediately to make precautionary measures else that may even cause the destruction of the rocket. So the analysis of the huge volume of data is really important in many such conditions like Patient monitoring, Volcano eruption analysis, Heart Beat analysis etc.

How do we gather Big Data?

We gather these big data in the data base. These data has to be crunched without loosing the major thresholds (changes - Up's and Down's).These threshold will be the deciding factor to analyse the big data collected. All the data's of the Big data need not be maintained in a single data base. There may be a cluster of master database with the reference of a particular data pointing to different tables(**Pointers) available within the cluster. 


What is Flat Files ?

When the data from the Big data is requested by any application then the data is shared as "Flat Files". 
Wiki: A flat file database describes any of various means to encode a database model (most commonly a table) as a single file.

A flat file can be a plain text field or a binary file. There are usually no structural relationships between the records.

Data thus requested from the "Big Data" is first encrypted at the sender end as Flat File. The same is later Decrypted at the application which has requested it eg: SharePoint application for patient monitoring system etc.