Stepping into Python

Many things happen in the last one month. I will cover one of them. This new thing is called Python. I know its not something new many have used it but it is new to me. Yes, I taught myself python and stepping out beyond, the still powerful, Java.

I started with simple Python programming language such as syntax, control flow, etc. And then step into its frameworks like Django, Flask, etc. Its a normal step when we learn new language.

  

What I want to tell about is not python’s language or its frameworks. But some of its libraries for machine learning. There are  four main libraries that are useful for conducting machine learning activities using python. Those are numpy, scipy, pandas and Scikit-learn.

Surprisingly these libraries have been used in production by many startups, including Tellapart that has been bought by Twitter for 500 Millions dollars.

My short research shown that these libraries can be used with Hadoop technology stacks. Simple example is incorporating these libraries in Map Reduce program. As we all know that we can create a map reduce using python thanks to Hadoop streaming.

But as any other machine learning implementation on Big Data it is not as obvious as the sun in mid summer. We need to clean the data, select the proper algorithm, make some adjustment on the parameter, analyze, interprest the results and finally display the result using visualization library. I am still working on that for my personal project. I will inform you about the progress.

Feel free to work on these libraries in python and let me know the results.