Testing Big Data

There will be huge chunk of data entering into the Database (Individual or cluster of DB's) in case of Big data. Testing of all the data which enters into the database is really hard so "Sampling method" or "A/B Testing" method can be performed to verify the data entering into the Database

What is Sampling or A/B method ?

This is just like a manual method where the factor(A) is compared with the entry in the Database(B) for the factor A. To make it clear if the factor A is considered as Temperature of the rocket then manually the temperature of the environment within the rocket is moderated to different limits to check whether the same moderated or adjusted temperature is recorded as B in the Database. Basically in the above entry case A=B.

This method is done to find the accuracy or the correctness of the data capturing sense of the Database.

Now once when the accuracy of the data collected in the data base is tested we need to move on to test the data retrieved from the Database to the application where the big data is used.

Understanding Big Data

In Simple words Big Data is "Huge Volume of related/non-related data"


Wiki's explanation

Big Data usually includes data sets with sizes beyond the ability of commonly used software tools to capture,curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set.

What is the need of Big Data?

If in case a continuous monitoring and analysis of data obtained from a source is required then that is the place for Big Data. eg: when a rockets propels to the space, each and every second the data about the condition within the rocket "Pressure, Temperature, etc" has to be monitored" if in case there is a slight change in the condition other than the accepted limit the control room has to be intimated immediately to make precautionary measures else that may even cause the destruction of the rocket. So the analysis of the huge volume of data is really important in many such conditions like Patient monitoring, Volcano eruption analysis, Heart Beat analysis etc.

How do we gather Big Data?

We gather these big data in the data base. These data has to be crunched without loosing the major thresholds (changes - Up's and Down's).These threshold will be the deciding factor to analyse the big data collected. All the data's of the Big data need not be maintained in a single data base. There may be a cluster of master database with the reference of a particular data pointing to different tables(**Pointers) available within the cluster. 


What is Flat Files ?

When the data from the Big data is requested by any application then the data is shared as "Flat Files". 
Wiki: A flat file database describes any of various means to encode a database model (most commonly a table) as a single file.

A flat file can be a plain text field or a binary file. There are usually no structural relationships between the records.

Data thus requested from the "Big Data" is first encrypted at the sender end as Flat File. The same is later Decrypted at the application which has requested it eg: SharePoint application for patient monitoring system etc.