Amazon does Hadoop: Adds Web-Scale Processing Engine to its Cloud
By Tarry Singh at 2 April, 2009, 2:22 pm
Slowly but surely, Amazon keeps adding capabilities to its cloud computing services. What started out as pay-by-the-drink storage (S3) and computational processing (EC2), now includes a simple database (SimpleDB), a content delivery network (CloudFront), and computer-to-computer messaging (SQS). And today Amazon added a web-scale file system data processing engine with Amazon Elastic MapReduce. (It is a framework for accessing data stored in file systems and databases).
This is actually a big deal because it allows developers to better take advantage of the massive computing power Amazon has to offer and create applications which process huge reservoirs of data (conveniently stored in Amazon S3) in parallel. MapReduce is the name of the data processing framework Google created to index and search the Web. It literally breaks up huge computational tasks and spreads them to different servers. This is called mapping the data. Once each processor is done with its portion of the math problem, it sends the result back so that all the different partial answers can be combined and then “reduced” into one final answer.
Amazon is using Hadoop, which is the open-source version of MapReduce. Yahoo also started using Hadoop last year. While Google and Yahoo use this technique for searching the Web, it can be used for any data-intensive computational problem. Amazon lists the following examples: “web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research.” Indeed, Hadoop is also the underlying technology used by IBM in its Blue Cloud initiative.
Related posts:
- Cloudera launches commercial distro of Hadoop; Stellar line up has Diane Greene and others involved! I was talking to a couple of folks and...
- Amazon pushes Cloud to new level with EBC Amazon’s new Elastic Block Computing initiative raises the bar for...
- Hyperic monitors Google App Engine Open source web infrastructure management provider Hyperic Inc. today announced...
- While you were asleep: How Amazon turned Cloud Computing into big business Honestly Amazon really and truly gets it. I have...
- Motley Fool : Amazon, The true Cloud Computing Gods! So, what the heck is “cloud computing”?Even after reading...
- Data Center buildout continues - Amazon is building another one in Oregon Amazon buys a large volume of servers to support...
- Cloudera = Cloud Computing what Red Hat = Linux? I am preparing for my keynote speech at the Cloud...
- Amazon, the King of the Cloud Computing Amazon.com (Nasdaq: AMZN) is the king of cloud computing....
No comments yet.