A little about Apache Hadoop

Apache Hadoop is one of the earliest and most influential open-source tools for storing, processing and managing the colossal amount of readily-available digital data that has aggregate with the rise of the World Wide Web (WWW). It unfolded from a project called Nutch, which attempted to find a better open source way to crawl the web. Nutch's creators were heavily determined by the thinking in two key papers from Google and basically fused them into Nutch, but hereafter the storage and processing work split into the Hadoop project, while continuing to develop Nutch as its own web crawling project.

In this article, we'll succinctly consider data systems and some precise, differentiating needs of big data systems. Then we’ll look at how Hadoop has evolved to address those needs.

Data Systems

Data endure all over the place: on scraps of paper, in books, photos, multimedia files, servers, logs, and on web sites. When that data is assiduously collected, it enters a data system.

Just imagine a school project where students measure the water level of a nearby ditch every day. They record or track their measurements in the field on a clipboard or paper sheet etc., return to their classroom, and enter that data in a spreadsheet. When they've collected an enough amount, they begin to examine it. They might compare the same months from different years, sort from the highest to lowest water level. They might build graphs to look for trends.

This school project highlight a data system:

Information present in different locations (the field notebooks of different students)It is collected into a system (hand-entered into a spreadsheet)It is stored (saved to disk on the classroom computer; field notebooks might be copied or retained to verify the honesty of the data)It is analysed (aggregated, sorted, or otherwise manipulated)Processed data is displayed (tables, charts, graphs)

This project is on the small end of the spectrum. A single computer can store, analyse, and display the daily water level measurements of one creek. Toward the other end of the spectrum, all the content on all the web pages in the world form a much larger dataset. At its most basic this is big data: so much information that it can't fit on a single computer.

The Hadoop’s framework itself is mainly written in Java programming language, with some native code in C’s  and command line utilities written as shell scripts. Though MapReduce Java codes are common, many programming languages can be used with "Hadoop Streaming" to implement the map and reduce parts of the user's program.

Jion now - hadoop training in gurgaon