Dec 31, 2012

Software engineer's guide to getting started with data science


Many of my software engineer friends ask me about learning data science. There are many articles on this subject from renowned data scientists (DatasporaGigaomQuora, Hilary Mason). This post captures my journey (a software engineer) on learning Statistics and Data Visualization.

I'm mid-way in my 5 year journey to become proficient in data science and my learning program has included self-learning (books, blogs, toy problems), projects at work, class-room training (Stanford), teaching/presentations, conferences (UseR, Strata). Here's what I've done so far and what worked and what didn't...



1. GETTING STARTED


a) Self-learning (2 - 4 months)

Explore if data science is for you

This is the key to getting started. Two years ago some of us at work formed a study group to review Stats 202 class material. This is what got me excited and started with data analytics. Only 2 of the 5 members of our study group chose to dive deeper into this field (data science is not for everyone).

b) Class-room training (9 - 12 months)

If you're serious about learning, enroll into a formal program

If you're serious about picking this skill, then opt for a course. The rigor of the class ensured that I didn't slack. Stanford offers great coursework to get started. They are far superior compared to many week-long training courses I've been to...




2. GETTING FOCUSED


a) Spend 100% of my time on data science

  • Once I was hooked on data science, it was difficult to spend only 20% of my time on it to build expertise. I needed to spend 100% of my time on it, so I found work problems related to data science (big data analysis, healthcare, marketing & sales and retail analytics, optimization problems). 


b) Work on interesting problems

  • I aligned my learning goal with my passion. I found it energizing and engaging to solve interesting problems while learning new techniques. I was interested in retail, healthcare and sports (cricket) data analysis. 


c) Accelerate learning: 


d) Learn business domains

I'm lucky to have access to internal and external experts in data science, and they've helped me understand their approach to data science problems (how they think, hypothesize and test/assess/reject solutions). I've learned from them the importance of "Hypothesis-driven data analysis" rather than "blind/brute-force data analysis". This highlighted the importance of understanding the business domains really well before trying to extract meaningful insights from the data. This led me to understand operations research and marketing topics, retail, travel & logistics (revenue management) and healthcare industries. NY Times recently published an article highlighting the need for intuition.




3. DATA SCIENCE BOOKS I FOUND USEFUL





4. WHAT DIDN'T WORK FOR ME

  • Learning multiple Statistical tools: A year ago, I started getting some work requests for SAS programming, so I wanted to learn it. I tried to learn it for a month or so but could not do it. The main reason was learning inertia and my love for the statistical tool I knew already - R. I really didn't need another statistical tool. I could solve most of my data science problems with R and other software tools I knew. So my advice is that if you already know SAS, Stata, Matlab, SPSS, Statistica very well, stick to it. However if you're learning a new statistical tool, pick R. R is open source while most others are commercial software (expensive and complex).
  • Auditing courses: I tried to follow self-paced coursework from Coursera and other MOOCs but it wasn't effective for me. I needed the routine, the pressure of a formal course with proper grading to go through the rigor
  • Increasing academic workload: Manage work-life balance and work-commitments well. Earlier this year, I tried to take multiple difficult courses at the same time and quickly realized that I wasn't enjoying and learning as I should.
  • Sticking to course text book only: Many of the books in these classes are too "dense" for me (a software engineer). So I used other material to understand the concepts. E.g. regression from Carnegie Mellon notes



Comments, questions, suggestions are welcome!

126 comments:

  1. VERY HELPFUL. THANKS. HOW ABOUT COLUMBIA UNIVERSITY'S COURSE?

    ReplyDelete
  2. Columbia has a great program on data science. See the list of their coursework in Stats dept here -> http://statistics.columbia.edu/content/course-descriptions. Checkout the notes from their Intro to Data Science course here -> http://columbiadatascience.com.

    ReplyDelete
  3. Instead, I need a "Statistician's guide to getting started with software engineering".

    ReplyDelete
    Replies
    1. I'm in the same boat. I have the business acumen and stats background;just need to get more comfortable with SQL and Python to round out my programming skills.

      Fortunately, i think that most of these recommendations apply, with a heavier emphasis on the programming languages up-front.

      Delete
  4. I'm hoping someone with Stats background can list their experience of picking up software engineering skills to serve as "Statistician's guide to getting started with software engineering." Here's what I've taught my colleagues in the past for data science training. It might be helpful to review it.

    I'm teaching another batch in January 2013 in Mumbai and will post the curriculum and my notes soon.

    ReplyDelete
  5. Hi Kaizer, I will post the training agenda soon. The trip to Mumbai was productive and hectic. We identified 10 engineers from the batch of 20 to work with our team on data analysis.

    ReplyDelete
  6. As I reflect on some books I've read, I realize that some stats books were too "dense" for me, a software engineer. The trick that worked for me was to look for alternative text. The book/text that I liked was often not the one suggested/listed by course professor. Often the course material/lecture notes on Carnegie Melon, MIT, Columbia courses are helpful.

    ReplyDelete
  7. how could we access stanford class without an sunet id? Thanks

    ReplyDelete
  8. Thanks for all this great information. It is helping me develop my plan. I especially like the part about you finding problems at work that let you use your DS skills... I am doing that too :)

    ReplyDelete
  9. Great helpful information. Thanks for sharing.

    ReplyDelete
  10. @sas study, I don't think its possible to see Stanford class material without sunet id. However, many of the lecture notes and tests are available on course websites (public).

    ReplyDelete
  11. Thanks @Lillian and @Gaurav. Here's a vendor offered course that I found helpful in the last few months -- Clourdera's Hadoop developer, Pig/Hive training.

    ReplyDelete
  12. Good structured information. The sequence is truly helpful. Thanks for the post. Stanford is offering its STATS202 course as part of a resident summer session program (June-July-August). Now I am stuck between undertaking this class at Stanford Summer Session vs undertaking their online Data Mining & Applications certificate program. Can you provide some inputs on this? Thanks.

    ReplyDelete
  13. @Pronojit, nothing like taking Stats 202 on the campus. Go for it! The interaction in the class and team projects will make it much more enriching. cheers!

    ReplyDelete
    Replies
    1. Thanks for that input. Apart from the STATS 202 what would be some of the good sources to dive deeper into Data Science in San Francisco area?

      Delete
  14. Great information that has helped me plan my training program. When are you teaching next?

    ReplyDelete
  15. Thank you for all the information, very helpful. I was looking at this program at Stanford : Mining Massive Data Sets Graduate Certificate. Do you know if they offer any placement assistance after completing the program?

    ReplyDelete
  16. Stanford has multiple, effective ways to assist with placement. Best to call the graduate office to get this information.

    ReplyDelete
  17. AnonymousJune 29, 2014

    thank u sir, I am 12 pass out, can u please tell me will b.sc in statistics help in becoming a data scientist?

    ReplyDelete
  18. I have no experience with coursework in B.Sc Stats, so very difficult to comment. Perhaps best to tailor your coursework to include key skills needed for data science - stats + computer science

    ReplyDelete
  19. its training varies from kids instruction in that grown-ups grabessays.com have aggregated learning and work experience which can add to the learning background.

    ReplyDelete
  20. For the original trip entrance, they will be thinking about the fitness of the house, the way modern it's, what exactly changes happen to be built to the idea, and also how big is it's. Various other factors, just like metres, cellars . as well as swimming pools will even come into engage in. They are hunting at this time issue in addition to record with the property or home order coursework is best one and really awesome custom essay writing service providers.

    ReplyDelete
  21. Thank you so much for sharing this wonderful information. I find this very useful and informative. keep up this good work.

    SAS training chennai

    ReplyDelete
  22. AnonymousJune 23, 2015

    My education is in math and statistics and it still comes easily to me. My work has been as a software engineer for the last 30 years. I fear that I would never be able to get work as a data scientist. Things have changed so dramatically.

    ReplyDelete
  23. Awesome article, I am a big fan of this website, keep on posting that great content, and I'll be a regular visitor for a long time..http://bachataclassesinnortheastphiladelphia.weebly.com/

    ReplyDelete
  24. Your article about software testing is awesome. It helped me to understand the career prospects in software testing industry. software testing training in Chennai

    ReplyDelete
  25. Thanks for your article. You have stated the most certain things about latest technologies. This would be very useful for my future career. This blog has awesome contents to read. Looking forward for more articles like this. Thanks for sharing.

    Testing training in chennai | Salesforce training in chennai | JAVA Training in Chennai | Salesforce Training in Chennai

    ReplyDelete
  26. Thank you for the informative post. It was thoroughly helpful to me. Keep posting more such articles on software testing training institute in Chennai and enlighten us.

    ReplyDelete
  27. Thanks for sharing such informative article on Loadrunner Automation testing tool. This load testing tool will provide most precise information about the quality of software. Loadrunner Course in Chennai | Loadrunner training institute in Chennai

    ReplyDelete
  28. I agree with your post, the Introduction of automation testing product shortens the development life cycle. It helps the software developers and programmers to validate software application performance and behavior before deployment. You can choose testing product based on your testing requirements and functionality. QTP Course in Chennai | QTP training

    ReplyDelete
  29. Thanks for sharing those useful basic programming links, it helps me to explore my knowledge in programming...if you want to switch your career in developing area you should know the basic of programming’s for that you have to read python because it was the first programming language for more details contact the link.
    Regards,
    Python Training in Chennai|Python Training|Python Training Institutes in Chennai

    ReplyDelete
  30. This is the key to getting started. Two years ago some of us at work formed a study group to review Stats 202 class material. This is what got me excited and started with data analytics. Only 2 of the 5 members of our study group chose to dive deeper into this field (data science is not for everyone).ccde written

    ReplyDelete
  31. Testing is the only way to deliver reliable products in the Information Technology market. Articles like this are vital in improvising one's ability as a software testing professional. Thank you so much for sharing this information in here. Keep blogging.

    Software testing training in chennai | Automation testing courses in chennai | Software testing institute in chennai

    ReplyDelete
  32. Automation will make any work to be completed so soon(Selenium training in chennai), in addition to this recording the actions is possible with automated tool. Your content explicitely states the same(Selenium testing course in chennai). Thanks for sharing this worth able content in here. This was very useful to me as well. Keep blogging like this(Selenium training institutes in chennai).

    ReplyDelete
  33. Hi, am a big follower of your blog. I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post. keep update your blog. Big Data Training in Chennai

    ReplyDelete
  34. Technology place a vital part in humans ecosystem. So in order to survive one must be up to date. Thanks for sharing this information in here. Keep blogging article like this. I have bookmarked this page for future reference.


    Hadoop Training Chennai | Big Data Course in Chennai | JAVA training in Chennai

    ReplyDelete
  35. Hi, Thanks for sharing such an informative blog about big data. You have done really a great job. I gathered lot of new information about big data from your blog. Keep posting.
    Big Data Training in Chennai | Big Data Training in Chennai

    ReplyDelete
  36. The Information which you provided is really useful for big data hadoop learners. Thank You for Sharing Valuable Information. I like this blog and this is very informative. Big Data Training in Chennai

    ReplyDelete
  37. Dear admin, The way you have explained the concept is mezmerizing. Thank you so much for sharing tis worth able content with us. The concept taken here will be useful for my future programs and i will surely implement them in my study. Keep blogging article like this.


    JAVA J2EE Training Institutes in Chennai | JAVA Training | JAVA Course in Chennai | Android training in chennai

    ReplyDelete
  38. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post. QTP Training in Chennai

    ReplyDelete
  39. Great post. Just wanted to mention that if you're interested in learning more about very effective enterprise-level software you should check out some of the predictive analytics solutions from Modern analytics. They provide valuable services and tools that help in making informed business decisions. Hope this helps!

    ReplyDelete
  40. Really interesting content which is unique which provided me the required information.
    Selenium Training in Chennai | best selenium testing training institute in Chennai.

    ReplyDelete
  41. Testing is very important before launching a web application or a mobile application because it can detect the error at an early stage, and it reduces the work of the developer.
    manual testing training institute in chennai | mobile application training in chennai | FITA Academy Chennai

    ReplyDelete
  42. The best thing about HTML5 is that it allows the developers to embed the video files, audio files, and high quality graphics without any third party applications.
    html5 training in chennai | html5 training institutes in chennai | FITA Academy reviews

    ReplyDelete
  43. Very interesting and informative blog.Thanks for sharing.Please keep updating.

    Informatica training in chennai

    ReplyDelete
  44. If you are willing to develop a website but you dont know web development or coding then relax wordpress CMS platform is just for you. Where you can create website all by yourself.
    wordpress training in chennai | Wordpress course in chennai

    ReplyDelete
  45. 100% Safe and Authentic Weight Loss Pills
    Buy PhenQ Now

    Naturally melt down the excessive fat without affecting your muscle mass.
    http://www.phenqreview.com.au/

    ReplyDelete
  46. PhenQ Reviews
    Make the fat storage away from the body.
    Supports to maintain the perfect body weight in which you look stunning.
    http://www.phenqreview.com.au/index.php/phenq-reviews/

    .phenqreview

    ReplyDelete
  47. Wonderful post its tell us how to make a software for since education and how to launch with easy way thanks for share it help with java programming .

    ReplyDelete
  48. Nice guide. It is very useful for engineering students. Thanks for sharing.

    Direct Admission in Engineering Colleges

    ReplyDelete
  49. Your blog is awesome..You have clearly about it ...Its very useful for me to know about new things..Keep on blogging..
    Java training in chennai

    ReplyDelete

  50. These provided information was really so nice,thanks for giving that post and the more skills to develop after refer the article .our giving articles really impressed for me,because of all information so nice.


    INFORMATICA Training in Chennai

    ReplyDelete
  51. This blog is having the general information. Got a creative work and this is very different one. We have to develop our creativity mind. This blog helps for this. Thank you for this blog. this is very interesting and useful.
    Informatica Training in Chennai

    ReplyDelete
  52. Digital Marketing is one of the most promising field which is giving a tough tie for traditional marketing methods.
    digital marketing course in chennai | Digital Marketing training in chennai | digital marketing courses in chennai

    ReplyDelete
  53. This comment has been removed by the author.

    ReplyDelete
  54. Selenium is a testing tool which is an open source which can work very well for the professionals and for testers. If a profesional is knowing manual testing then if he learn seenium then there are good chance for career growth.
    selenium training in chennai | selenium training institute in chennai | selenium course in chennai

    ReplyDelete
  55. Informatica is one of the most trusted and effective ETL software which is known globally. Informatica is one data warehousing ETL tool. Informatica ha very go opportuities for job across the world.
    informatica training in chennai | informatica training institute in Chennai | informatica classes in Chennai

    ReplyDelete
  56. This is a wonderful post. I enjoyed the information lot. I will bookmark this page. Thanks for sharing this important information.largemouth bass fishing tips

    ReplyDelete
  57. Nice useful article for small businesses like us. thank you so much for the detail information.ecommerce web designing

    ReplyDelete
  58. This comment has been removed by the author.

    ReplyDelete
  59. Java is programing language which is used in almost all the applications and games which are on the web. Java is being used extensively and it will be used extensively in near future. So getting trained in Java will surely be helpful.
    Java training in Chennai | Java training institute in Chennai | Java course in Chennai

    ReplyDelete
  60. Manual testing is the process of testing the software manually in order to detect the software defects. It requires the role of tester to deliver the product efficiently to the customer and use all of its application to ensure that it works correctly
    manual testing training institute in chennai | manual testing training

    ReplyDelete
  61. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work..Please keep update..

    Salesforce Training in Chennai

    ReplyDelete
  62. Nice guide, This is very useful. Thanks for sharing.

    Internet Marketing Company in chennai

    ReplyDelete
  63. Best SEO training in hyderabad all modules are clearly solved and practice with guide.seo training in hyderabadAnd clearly getting knowledge of seo.

    ReplyDelete

  64. this is valuable information for learners.thanks
    Data Scientist Course in Hyderabad

    ReplyDelete
  65. Angular JS is a structural framework for dynamic web application and angular's data binding eliminates much of the code that you would otherwise write it.
    Angularjs Training in Chennai | Angularjs training Chennai

    ReplyDelete
  66. This is very useful guide. Thanks for sharing.

    Wordpress development in chennai

    ReplyDelete
  67. nice posts..
    Informatica training, in the recent times has acquired a wide scope of popularity amongst the youngsters at the forefront of their career.
    Informatica online training in hyderabad


    ReplyDelete
  68. Excellant post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    cloud computing training in chennai | cloud computing courses in chennai

    ReplyDelete
  69. Excellant post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    Node JS training in chennai | Node JS training institute in chennai

    ReplyDelete
  70. Great post....Thank you for posting the great content……I found it quiet interesting, hopefully you will keep posting such blogs…
    If you Want more seo course chennai

    ReplyDelete
  71. Nice information!!! I prefer Loadrunner automation testing tool to validate the performance of software application/system under actual load.
    Regards,
    Loadrunner Training|Loadrunner course in Chennai|Qtp training |Qtp training institutes in chennai

    ReplyDelete
  72. My Arcus offer java training with 100% placement. Our java training course that includes fundamentals and advance java training program with high priority jobs. java j2ee training with placement having more exposure in most of the industry nowadays in depth manner of java

    java training in chennai

    ReplyDelete
  73. Great post. Thanks for sharing such a useful post.

    Web designing training in Chennai

    ReplyDelete
  74. Your software wonderful i use it for practice thanks for share it personal statement engineering .

    ReplyDelete
  75. Really valuable guidance, these kind of tips really helpful. thanks
    SEO Company Melbourne | SEO Services Melbourne | SEO Consultant Melbourne

    ReplyDelete
  76. software engineers nice posts..
    Hadoop training in hyderabad.All the basic and get the full knowledge of hadoop.
    hadoop training in hyderabad

    ReplyDelete
  77. Wonderful post. Thanks for sharing such a useful post.

    digital marketing training in chennai

    ReplyDelete
  78. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
    oracle training in chennai | oracle training institutes in chennai

    ReplyDelete
  79. R is a programming dialect and programming environment for measurable registering and design bolstered by the R Foundation for Statistical Computing. DataStage Training | SQL Training | Android Training | SharePoint Training

    ReplyDelete
  80. Thanks for sharing this informative content that guided me to know the details about the training offered in different technology.
    digital marketing course in chennai | digital marketing training in Chennai

    ReplyDelete
  81. software engineers guide to getting ..


    informatica training

    ReplyDelete
  82. Thanks for Sharing the valuable information and thanks for sharing the wonderful article..We are glad to see such a wonderful article..
    QTP Training in Chennai | QTP Training Institute in Chennai | QTP Training

    ReplyDelete
  83. This is a topic that’s near to my heart… Many thanks!
    Exactly where are your contact details though?

    Here My web
    - Grosir Jaket Parka
    - Grosir Jaket Parka
    - Grosir Jaket Parka
    - Grosir Jaket Parka
    - Grosir Jaket Parka
    - Grosir Jaket Parka

    ReplyDelete
  84. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
    PHP Training in Chennai | PHP Course in Chennai

    ReplyDelete
  85. Wonderful blog! I found it while searching on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Many thanks.
    2048 game | five nights at freddy's 4 | five nights at freddy's 3 | fireboy and watergirl | fireboy and watergirl 4||red ball | age of war

    ReplyDelete
  86. below youll discover the link to some websites that we believe you ought to visit


    neat huge : Souvenir Pernikahan Unik - Souvenir Pernikahan Unik

    ReplyDelete
  87. we like to honor lots of other internet web sites around the internet, even when they arent linked to us, by linking to them. Under are some webpages worth checking out
    Here My Website :

    - Grosir Jaket Parka
    - Harga Grosir Jaket Parka
    - Grosir Jaket Parka Wanita
    - Grosir Jaket Parka Murah
    - Grosir Jaket Parka Wanita

    ReplyDelete
  88. nice and really helpful article to everyone... thanks for sharing
    oracle training in chennai

    ReplyDelete
  89. Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    salesforce training in chennai | salesforce training institute in chennai

    ReplyDelete
  90. Which animal has bigger eyes than brain?
    [url=www.quester.pk/question/which-animal-has-bigger-eyes-than-brain/]https://www.quester.pk/question/which-animal-has-bigger-eyes-than-brain/[/url]
    Which animal has longest lifespan?
    [url=www.quester.pk/question/which-animal-has-longest-lifespan/]https://www.quester.pk/question/which-animal-has-longest-lifespan/[/url]
    Does Ufone Have 3G Coverage In yazman?
    [url=www.quester.pk/question/does-ufone-have-3g-coverage-in-yazman/]https://www.quester.pk/question/does-ufone-have-3g-coverage-in-yazman/[/url]
    Does Ufone Have 3G Coverage In Wazirabad?
    [url=www.quester.pk/question/does-ufone-have-3g-coverage-in-wazirabad/]https://www.quester.pk/question/does-ufone-have-3g-coverage-in-wazirabad/[/url]

    ReplyDelete