Our team has been discussing how we want to handle programming in this environment. This is due to the fact that while we all have a programming background, none of us has ever worked with Java. So for the benefit of getting this project up and running as quickly as possible, we decided to pursue the severely under-documented streaming python method.
So to get started I reviewed a great blog by Michael Gnoll who goes over the basics: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
Now in our environment, Hadoop will be consuming vast amount of serialized or delimited data. So we're fortunate enough to have some "structure" on the input. This also means we can easily deserialize on the map phase. So for my first test I wanted to take a simple pipe (|) delimited file and get a count of one of the column values in this case the 6th column.
So I first constructed my super simple mapper.py and reducer.py files:
Mapper.py
Reducer.py
So I loaded my test data up to HDFS (~ 50 gigs worth outta do it). I then loaded my python up to my local directory. I ran a quick test file (same file layout obviously, but fewer records) to ensure my python worked.
This returned some data, so I was ready to go.
So I ran the following command to initiate a stream mapreduce:
And..... FAILURE. sigh
Yup... not very helpful. However, after some digging around, I discovered what the issue was.
dos2unix
/facepalm
Gotta love mixed environments! Anyway after running simple "dos2unix mapper.py" and "dos2unix reducer.py", I reran the execution code and:
Success! While the performance wasn't incredibly awesome (~18 minutes to process 458,848,279 rows, compared to 5 minutes in SQL Server), its a start. Going to start messing around with more advanced python methods and see what we can do.
