Programmers learn by implementing techniques from scratch.

It is a type of learning that is perhaps slower than other types of learning, but fuller in that all of the micro decisions involved become intimate. The implementation is owned from head to tail.

I recently finished reading the paperback version and I think it might be one of my favorite beginner machine learning books for the year. Grab a copy!

Overview of the Book

Let’s take a birds eye look at this book.

The Author: Joel Grus

The author of this book is Joel Grus, a software engineer at Google.

In previous roles he’s been a Data Scientist and Analyst at startups and engineer at Google. He got his PhD from Caltech. A very fine background.

Learn more about Joel on his LinkedIn profile and blog and Twitter.

The Target Audience: Beginners

The target audience for the book is intermediate programmers interested in getting started in data science and machine learning.

Python is not a prerequisite to read this book (there is a Python crash course in Chapter 2), but it would speed things up if you were already a Python programmer.

The book does not assume any mathematical background in machine learning (there is a crash course in Chapters 4-7), but again, some background in stats, probability and algebra would speed things along.

The Book Approach: Write Code (in Python)

This is an introductory book to data science and machine learning.

The majority of the text focuses on the implementation of machine learning algorithms. There is a brief introduction to Python and the coverage of some basic math, data visualization and data gathering subjects.

It will take you from an a beginner programmer to being able to implement machine learning algorithms to address various data science problems.

Code From Scratch

The approach taken in the book is to describe the concepts and then to implement them in Python from scratch. This means without the use of machine learning and data handling libraries (e.g. scikit-learn).

The stated goal by the author of implementing algorithms from scratch is:

…building tools and implemented algorithms by hand in order to better understand them.

Good code examples must be readable first and efficient and effective second. They are written for understandability as a teaching aid, not production level code. Take note that the programming you will be doing from scratch will be instructional only, not operationalizable.

I put a lot of thought into creating implementations and examples that are clear, well commented, and readable. In most case, the tools we build will be illuminating but impractical.

Book Contents

The book is 311 pages long and contains 25 chapters. It’s a classic O’Reilly book and is the perfect form factor to have open in front of you while you bash away at the keyboard implementing the code examples.

In this section we take a look at the table of contents:

a data scientist is someone who extracts insights from messy data

• Chapter 1: Introduction (What is data science?)
• Chapter 2: A Crash Course in Python (syntax, data structures, control flow, and other features)
• Chapter 3: Visualizing Data (bar, line and scatter plots with matplotlib)
• Chapter 4: Linear Algebra (vectors and matricies)
• Chapter 5: Statistics (central tendency and correlations)
• Chapter 6: Probability (Bayes’ Theorem, Random Variables, Normality)
• Chapter 7: Hypothesis and Inference (confidence intervals, P values, Bayesian inference)
• Chapter 8: Gradient Descent (gradients, steps, stochastic variation)
• Chapter 9: Getting Data (scraping HTML, JSON APIs)
• Chapter 10: Working with Data (basic viz, data transforms)

machine learning… [refers] to creating and using models that are learned from data […] this might be called predictive modeling or data mining

• Chapter 11: Machine Learning (fitting, bias-variance, feature selection)
• Chapter 12: k-Nearest Neighbors (also curse of dimensionality)
• Chapter 13: Naive Bayes
• Chapter 14: Simple Linear Regression (also gradient descent)
• Chapter 15: Multiple Regression (also bootstrap, regularization)
• Chapter 16: Logistic Regression (also SVM)
• Chapter 17: Decision Trees (also random forest)
• Chapter 18: Neural Networks (perceptron and back-prop)
• Chapter 19: Clustering (k-Means)

Natural language process (NLP) refers to computational techniques involving language.

• Chapter 20: Natural Language Processing (n-gram, grammars, Gibbs sampling)
• Chapter 21: Network Analysis (Centrality and PageRank)
• Chapter 22: Recommender Systems (user- and item-based)
• Chapter 23: Databases and SQL (basic usage)
• Chapter 24: MapReduce (various worked examples)
• Chapter 25: Go Forth and Do Data Science (libs you should use)

Implementing things “from scratch” is great for understanding how they work. But it’s generally not great for performance …, ease of use, rapid prototyping, or error handling. In practice, you’ll want to use well-designed libraries that solidly implement the fundamentals.

Opinions of the Book

I generally liked the table of contents, except I would make some changes.

I would drop some of the later chapters like NLP, Network Analysis, and so on (Chapters 20-24) and rename the book “Machine Learning Algorithms from Scratch“. It would be a less sexy but a more honest and accurate title.

Data Science is about formulating the questions then gathering the data and building the models to answer them. We don’t really need a data science from scratch book unless it was a bunch of business case studies plus the modeling. From scratch in data science really means the algorithms part.

I’m not upset, in fact I had a great time reading this book, but I could imagine someone expecting systematic processes for formulating and working through business-data problems in addition the modeling feeling a little bit misleading.

I did not implement all of the algorithms from scratch. I read the whole book, studied all of the examples, but I only implemented a few for fun.

I found the code easy to read, commented just enough. I think going vanilla Python (over NumPy) was a good move. It lowered the bar just enough so that all you need is some basic Python syntax and away you go.

Resources

I’ve gathered up some additional resources related to the book if you’re interested in diving deeper.

Final Thoughts

I like the book. I had fun, I think primarily because I have always liked working through programming books and because I’ve written a book just like this myself (i.e. Clever Algorithms).

If you’ve been around the block and you’re hard core into scikit-learn or R right now and not interested in the distraction, this book is probably not for you. But remember, the learning never ends and it can be fun to go over the beginner stuff again and tighten up the screws.

If you know some Python (or you’re a solid dev and want to get into Python) and you want to get intimate with machine learning algorithms by implementing them, then this book is for you.

Did you read Data Science From Scratch? What did you think? Leave a comment.