In order to pursue my interest in machine learning I want to be comfortable with the knowledge assumed by resources such as Machine Learning: a Probabilistic Perspective and The Elements of Statistical Learning, which are aimed at later-year undergraduate or beginning graduate students.
The following is a list of resources for reviewing maths, starting from the beginning and covering undergraduate-level calculus, linear algebra, probability and statistics.
Other lists and advice
Best Math Books – A Comprehensive Reading List (StumblingRobot.com)
This guide's recommendations are consistent with the advice I found elsewhere. It describes and compares several of the best books on each topic. It's aimed at people interested in pure maths, but the basics section seems very relevant to my interests, and it intentionally chooses challenging texts.
Chicago undergraduate mathematics bibliography
This well-respected list is the one I saw most commonly referenced. Its comments are briefer, but come from several contributors and cover many topics and books.
Programming books you might want to consider reading (Dan Luu)
This list of programming books includes a math section. Dan Luu also has an interesting post asking How much math do programmers need?
What are some good books available for learning probability and statistics? (Quora)
Several good answers.
Recommended books (UC Berkeley Statistics Graduate Student Association)
A list of books in many categories, without much commentary.
The World of Math (Khan Academy)
Khan Academy covers all of school maths and a little more. It starts from counting and goes up to calculus, differential equations and linear algebra, with short video lectures and comprehensive online exercises. The explanations are very clear, but the format makes it slower than other resources for refreshing prior knowledge.
Basic Mathematics (Lang)
This book is aimed at high school students or beginning university students and includes everything needed to prepare a student for going on to calculus, linear algebra and other topics. It makes a point of including some vector geometry earlier than is commonly done and "deals with mathematics on both a manipulative (or computational) level and the theoretical level."
Engineering Mathematics (Stroud & Booth)
This was highly recommended by a colleague, for building practical maths skills from the beginning. It stops at Fourier transforms (covered in Advanced Engineering Mathematics) and focuses on practice rather than theory, lacking formal definitions. It is designed for self study, with checklists and exercises included.
Mathematical thinking and proof writing
How to Study as a Mathematics Major (Alcock)
A useful and easy to read explanation of what university-level mathematics is about and how to approach studying it. It goes a deeper than Devlin in the discussion, and refers users to other resources (including Velleman) for practice of the proof-writing techniques it introduces. The general study advice won't be of much interest unless you're really fresh out of high-school.
How to Prove It: A Structured Approach (Velleman)
The most widely recommended book for learning how to work with proofs.
Introduction to Mathematical Thinking (Devlin)
This book starts with an interesting discussion of how modern mathematics differs from its past, and from what you might have learned in school. The second half introduces proofs and assumes you're following along with all the exercises. My preference is the previous two books, but this is a good alternative for both. This material is also covered by the author in a Coursera course.
For calculus I'm interested in Apostol rather than Spivak. I found a particularly helpful Mathematics Stack Exchange answer comparing the two. Apostol covers integration before differentiation, introduces linear algebra before multi-variable calculus, and has been particularly recommended for older students coming back to mathematics and as the best calculus book for building mathematical maturity.
These two (and Wasserman, below) were recommended in Machine Learning: a Probabilistic Perspective:
Statistics (Freedman et al.)
Highly recommended in reviews and by a colleague. Focuses on fundamental ideas and their application rather than equations or proofs.
All of Statistics: A Concice Course in Statistical Inference (Wasserman)
From the publisher: "This book covers a much wider range of topics than a typical introductory text on mathematical statistics. It includes modern topics like nonparametric curve estimation, bootstrapping and classification, topics that are usually relegated to follow-up courses. The reader is assumed to know calculus and a little linear algebra. No previous knowledge of probability and statistics is required. The text can be used at the advanced undergraduate and graduate level."
The course Basic Statistics (Coursera) is nice and clear, but aimed at humanities students.
After I've reviewed the topics above, it might be nice to take a broader look at mathmatics. These two books are often recommended for that purpose.
Mathematics: Its Content, Methods and Meaning (Aleksandrov et al.)
I found this survey by Soviet mathematicians via a highly upvoted Mathematics Stack Exchange answer, and noticed it recommended in several other places.
What is Mathematics? An Elementry Approach to Ideas and Methods (Courant)
Einstein called it "a lucid representation of the fundamental concepts and methods of the whole field of mathematics."
There are a lot of online courses on these topics. Here is a list of many of them:
I was quite inspired by reading about Scott Young's MIT challenge, in which he went through 4 years worth of MIT computer science materials in 1 year. MIT has a lot of course materials freely available online, especially in computer science and mathematics. Sometimes it's video lectures, other times it's written notes, of varying detail. This is a great resource, but I think I will focus my learning on textbooks.