What is Big Data?

big data

The key to understanding big data technologies is unlocking its past, its present and its future.

Information is data, and data only started to make an impact within our global economy with the earliest introduction of a data centre for financial institutions in 1955, for the Bank of America. With the introduction of the ERMA Mark 2, banks were able to handle their accounting and cheque systems around the clock. In a way this could be considered one of the first instances of automation within the banking world during the 1900’s, mere kilobytes of data used to manage whole accounting systems compared to petabytes (1000 TB) of data we have access today.

big data

Ok now let’s switch back to the present

With the global advent of IoT (Internet of Things), there has been ever rising surge of devices connected to the internet, gathering and collating huge amount of user data not just for the benefit of advertising companies but also for a variety of industries such as manufacturing and software development to better understand customers but also an efficient manner to build better products and services. This is driven by collating and dissecting big data, transforming it into dashboards through ERP systems that give usage patterns and performance data on your current product line.

Big Data

The future for Big Data is here, and it is through Machine learning

Analyzing big data through Machine Learning has not only produced more data but has learned as the name suggests, to use this data in an incredibly efficient manner. Machine Learning, is a manner of data analytics that learns computer algorithms that improve based on experiential data and by identifying patterns, machine learning can effectively make decisions without human intervention.

Here’s an easy-to-understand example of how machine learning works in your day-to-day devices, like your smartphone such as a virtual assistant software. Be it Google Voice or Siri, the major key to these assistants is voice recognition, however most individuals have varying degrees of tone, language and also manner of speech. By using big data and specifically your user data as a point of input, your chosen virtual assistant is able to better understand over time, becoming increasingly accurate over time with the gradual eventuality of becoming more efficient and effective than real assistants who can not work at the speed and effectiveness of a service such as Google voice or Siri.

Big Data

How do you identify or describe big data?

You can further characterise big data through the following four characteristics

(i) Volume: Well volume is key, as big data is as the name suggest enormous, the size is crucial to identifying the value within the data. For example: The New York Stock Exchange (NYSE) generates 1TB (Terabyte) of trade data each day, if you collate these over months or weeks, this can generate a lot of information through which analysts can determine market behaviour.

(ii) Variety: when handling any amount of enormous information you can expect a variety through analysis, as such Big Data has different types of data you can further dichotomise. These can be such as:

1. Structured: Data that is processed, stored and accessed is through a stringent fixed format giving it a form of structure. Ok, if it’s a bit harder to understand here is a very simple example of structured data

big data

2. Unstructured: This is data that is in its raw form hence being unstructured, it’s even bigger but isn’t actively managed. A simple example of this would be through social media, while it is semi-structured through the use of hashtags if you were to search for interests, however anything related to the hashtags depending on the platform is most times unrelated, but companies are overcoming this by using many strategies such as combining mentions, hashtags and also tags within their network.

(iii) Velocity: This is the speed at which data is created hence ‘velocity’. However, the value of this data depends how the data generation is processed. This can be through the flow of data through financial processes, but an easier example would be again through social media. For example, twitter generates 500 million tweets a day, an incredible amount of data within the span of just 24 hours, this stat could be constantly changing as its users grow at a rate of 8.4% annually.

(iv) Variability: As with any enormous amount of information there is bound to be faults/inconsistencies within the information provided. This same principle is applied to identifying big data, being a variable in itself, there are multiple types of data and sources as well and due to data having constant changes with new information.  As a result companies have started to use of advanced software not only to sort through the information but also to contextualize them.

Big Data

Big Value through Big Data

Data is intrinsically valuable for all businesses. Today, data is capital as the more data a business acquires, the more value it can output not just to effectively manage business processes, but also to analyse customers to better develop existing or new products.

What is big data used for?

So, the key to unlocking the incredible benefits Big Data is the ability to process it, which is done through a Data Base Management System or DBMS as the tech wizards call it. Another way to look at it is to see Big Data as the cow and the DBMS as the butcher who processes the ‘data’.

This gives your business the ability to improve customer service through analysing data from social media services that give you analytics and the ability to strategize to target and achieve higher retention with customers.

It doesn’t stop here you can further use this to analyse consumer behaviour through consumer responses and identify any defects in products (this can be through BETA testing software) and also better operational efficiency in terms of employee management.

Some of the biggest companies today are based on the value they offer through data storage but also processing big data. We can simply look at tech companies such as Google, with the majority of their value driven through data processing, analytics and storage.

Big Data

Reaching scalability through the Cloud

With massive improvements in cloud technology, not only in terms of architecture, with the application of Moore’s law the cost of storage and computation of data has become increasingly more commercially viable, be it outsourcing data storage centres through providers such as Amazon AWS, Azure or even Google, these services often cost less and offer more security than an inbuilt on premise data center, as a business owner you do not have to worry about maintenance costs and more importantly data security.

The possibilities today with the improvement of cloud computing architecture is seemingly endless. Big data has the ability to scale and also be accessible from a variety of devices. As a business if you have invested in cloud computing such as a cloud-based ERP solution for your business, you and your employees can access your customer and company data for your product/service line anywhere, this is especially beneficial with hybrid work environments that are present in the workplace today.

Big Data


Just to refresh your memory, Big data is essentially data that can be identified by its incredible volume, velocity (in terms of information) and like any huge amount of information, its variety. These data sets are so huge they are classified as big data, however using regular processing software is not feasible.

The use of an ERP system to drive solutions through business intelligence in an effort to solve problems is a much more succinct approach towards driving value through big data. This is further explained in the next leg of this big data series, to be released soon.