Dec 31, 2012

Software engineer's guide to getting started with data science

Many of my software engineer friends ask me about learning data science. There are many articles on this subject from renowned data scientists (DatasporaGigaomQuora, Hilary Mason). This post captures my journey (a software engineer) on learning Statistics and Data Visualization.

I'm mid-way in my 5 year journey to become proficient in data science and my learning program has included self-learning (books, blogs, toy problems), projects at work, class-room training (Stanford), teaching/presentations, conferences (UseR, Strata). Here's what I've done so far and what worked and what didn't...


a) Self-learning (2 - 4 months)

Explore if data science is for you

This is the key to getting started. Two years ago some of us at work formed a study group to review Stats 202 class material. This is what got me excited and started with data analytics. Only 2 of the 5 members of our study group chose to dive deeper into this field (data science is not for everyone).

b) Class-room training (9 - 12 months)

If you're serious about learning, enroll into a formal program

If you're serious about picking this skill, then opt for a course. The rigor of the class ensured that I didn't slack. Stanford offers great coursework to get started. They are far superior compared to many week-long training courses I've been to...


a) Spend 100% of my time on data science

  • Once I was hooked on data science, it was difficult to spend only 20% of my time on it to build expertise. I needed to spend 100% of my time on it, so I found work problems related to data science (big data analysis, healthcare, marketing & sales and retail analytics, optimization problems). 

b) Work on interesting problems

  • I aligned my learning goal with my passion. I found it energizing and engaging to solve interesting problems while learning new techniques. I was interested in retail, healthcare and sports (cricket) data analysis. 

c) Accelerate learning: 

d) Learn business domains

I'm lucky to have access to internal and external experts in data science, and they've helped me understand their approach to data science problems (how they think, hypothesize and test/assess/reject solutions). I've learned from them the importance of "Hypothesis-driven data analysis" rather than "blind/brute-force data analysis". This highlighted the importance of understanding the business domains really well before trying to extract meaningful insights from the data. This led me to understand operations research and marketing topics, retail, travel & logistics (revenue management) and healthcare industries. NY Times recently published an article highlighting the need for intuition.



  • Learning multiple Statistical tools: A year ago, I started getting some work requests for SAS programming, so I wanted to learn it. I tried to learn it for a month or so but could not do it. The main reason was learning inertia and my love for the statistical tool I knew already - R. I really didn't need another statistical tool. I could solve most of my data science problems with R and other software tools I knew. So my advice is that if you already know SAS, Stata, Matlab, SPSS, Statistica very well, stick to it. However if you're learning a new statistical tool, pick R. R is open source while most others are commercial software (expensive and complex).
  • Auditing courses: I tried to follow self-paced coursework from Coursera and other MOOCs but it wasn't effective for me. I needed the routine, the pressure of a formal course with proper grading to go through the rigor
  • Increasing academic workload: Manage work-life balance and work-commitments well. Earlier this year, I tried to take multiple difficult courses at the same time and quickly realized that I wasn't enjoying and learning as I should.
  • Sticking to course text book only: Many of the books in these classes are too "dense" for me (a software engineer). So I used other material to understand the concepts. E.g. regression from Carnegie Mellon notes

Comments, questions, suggestions are welcome!



  2. Columbia has a great program on data science. See the list of their coursework in Stats dept here -> Checkout the notes from their Intro to Data Science course here ->

  3. Instead, I need a "Statistician's guide to getting started with software engineering".

    1. I'm in the same boat. I have the business acumen and stats background;just need to get more comfortable with SQL and Python to round out my programming skills.

      Fortunately, i think that most of these recommendations apply, with a heavier emphasis on the programming languages up-front.

  4. I'm hoping someone with Stats background can list their experience of picking up software engineering skills to serve as "Statistician's guide to getting started with software engineering." Here's what I've taught my colleagues in the past for data science training. It might be helpful to review it.

    I'm teaching another batch in January 2013 in Mumbai and will post the curriculum and my notes soon.

  5. Hi Kaizer, I will post the training agenda soon. The trip to Mumbai was productive and hectic. We identified 10 engineers from the batch of 20 to work with our team on data analysis.

  6. As I reflect on some books I've read, I realize that some stats books were too "dense" for me, a software engineer. The trick that worked for me was to look for alternative text. The book/text that I liked was often not the one suggested/listed by course professor. Often the course material/lecture notes on Carnegie Melon, MIT, Columbia courses are helpful.

  7. how could we access stanford class without an sunet id? Thanks

  8. Thanks for all this great information. It is helping me develop my plan. I especially like the part about you finding problems at work that let you use your DS skills... I am doing that too :)

  9. Great helpful information. Thanks for sharing.

  10. @sas study, I don't think its possible to see Stanford class material without sunet id. However, many of the lecture notes and tests are available on course websites (public).

  11. Thanks @Lillian and @Gaurav. Here's a vendor offered course that I found helpful in the last few months -- Clourdera's Hadoop developer, Pig/Hive training.

  12. Good structured information. The sequence is truly helpful. Thanks for the post. Stanford is offering its STATS202 course as part of a resident summer session program (June-July-August). Now I am stuck between undertaking this class at Stanford Summer Session vs undertaking their online Data Mining & Applications certificate program. Can you provide some inputs on this? Thanks.

  13. @Pronojit, nothing like taking Stats 202 on the campus. Go for it! The interaction in the class and team projects will make it much more enriching. cheers!

    1. Thanks for that input. Apart from the STATS 202 what would be some of the good sources to dive deeper into Data Science in San Francisco area?

  14. Great information that has helped me plan my training program. When are you teaching next?

  15. Thank you for all the information, very helpful. I was looking at this program at Stanford : Mining Massive Data Sets Graduate Certificate. Do you know if they offer any placement assistance after completing the program?

  16. Stanford has multiple, effective ways to assist with placement. Best to call the graduate office to get this information.

  17. AnonymousJune 29, 2014

    thank u sir, I am 12 pass out, can u please tell me will in statistics help in becoming a data scientist?