Skip to main content

Big Data Introduction

I am beginning to put up some technical topics on my blog henceforth. 

As a beginner post, here is an introduction to the concepts of Big Data. The content here is based on books I read, articles I read on the internet and also based on looking at demos of certain software that I or other people use. The effort I make is to put perspectives about the domain into a crisp readable mind map which is concise and puts forth a big picture for readers.

I use mind maps. If you are uncomfortable with mind maps, please look elsewhere for a better way to understand these topics. 



This mind map covers the facts about Big Data, the challenges we face with the amount of data we are generating in this world today. There are constraints to using hardware, constraints on the nature of the data that we aim to process, etc.

When discussing drawbacks, it is also useful to understand the alternatives of how problems can be solved and the risks of using such alternatives. When we try and do some operations on Big Data they can be done entirely online or offline - so it is important to understand the choices we have to handle data in this manner.

Further data is also processed in different ways depending on where the data is coming from, what exactly we want to do with the data (index, search, etc). There are tools meant for usage for specific types of processing which are also listed in the map.

Hadoop being the popular (until now) distributed data processing framework means that it has thrown its weight around considering how many companies invested into it, to use it, to enhance it and produce outcomes on large amounts of data for purposes such as business analytics, or data sciences areas.

Finally the map also covers the related tools/projects in conjunction with Hadoop that exist in the ecosystem today and what they are capable of doing to help the Big Data domain. Obviously then what you would need to do is, based on your interest dwell deeper into an area that you are interested to know more.

I will leave you with a quote I saw in a particular book that defines the reason we need a different way to attack the problems Big Data poses.

If you need an ox for heavier pulling you don't grow a larger ox, you put more oxen

Comments

Popular posts from this blog

Car Rentals in Kuala Lumpur - SoCar vs GoCar

IntroductionAs I was headed to Malaysia for a trip for 10 days, I needed an option to commute within the country. While renting cabs with drivers was definitely an option, there was no privacy enough, and there was no way to make my own decisions on where I wished to go when. I was not so keen to rent a cab outside of KL city.We had plans to visit multiple places such as Cameron, Penang, Langkawi so I needed a vehicle which I was willing to drive myself. It was the best option as it would give me the convenience without burning a hole in my pocket. The trip itself was turning out to be expensive and I wanted a good option for inside the country travel.Things that I was looking forAs any first time car rental customer, I had some concerns, so I needed theseEasy rental mechanism, including driver sign up and validation of
licensePreferably use local Karnataka (state license from india) - without the need for IDPApp based car handlingEasy fuelingEasy breakdown service if needed anywhere …

How to cover Malaysia in 10 days - full details on a self planned trip

Some history Ever since 2016 my wife and I decided that we want to do one international trip (though not necessarily one country) every year. We made it to Singapore in 2016, and Dubai in 2017. It was time to explore and take my family to Malaysia this time around in 2018.

The fact that I was working in Malaysia in 2002 and could not get time in the short one year that I stayed there to see many places only meant the urge to go there was pretty strong in order that I could get another chance to cover all the places I did not see this time around.

Also 15 years down the line with a lot of instrastructure and convenience improvements happening around the world, surely the trip would seemingly put me at more ease than earlier on. So it was time to decide how to plan the trip.

My logic for planning trips I usually do my own booking and planning for everything for my trip. I look through places to visit, I look through multiple sources of reviews, I look through the best times to travel t…

Veg Biriyani Recipe

IngredientsOne Potato - dices into small cube sized pieces and soak in waterBeans - about 10 of them chopped into medium pieces, soak in waterCarrot - about 2 of them chopped into medium pieces, and soak in waterOnions - big ones, 2 of them sliced thin and longGreen peas - a handfulMint and Coriander leaves - one medium sized bowlGhee - 2 to 3 teaspoonsCurd - 4 tea spoonsGroundnut oil - 4 to 6 teaspoons for the onions to caramelize (same can be later used for vegetables to be cooked), 4 teaspoons for the spices to be cooked along with riceBasmati rice - One cup upto brim (standard cup measure)Multi spices pack - 2 packs (contains cloves, elaichi, cinnamon, star anise)Bay leaves - 3-4 for vegetables, 2-3 for riceShahi jeera - for vegetables (2-3 teaspoons)Turmeric powder, chilli powder, garam masala, biriyani masala, salt, cumin powderGinger garlic paste - 2-3 teaspoonsMethi leaves - half a handfulChillies to taste (I used one long chilli, you can desire to add 2) - diced into medium p…