Maths foundations for statistics and machine learning
While (and after) being a COVID hermit, I took/re-took a journey from basic maths to try to use the momentum to get better at the mathematics of statistics and machine learning. It was a winding path but, with hindsight, the list below shows the steps I would have taken in an ideal world (in order, for the first two sub-lists) with the resources I found most helpful along the way (or, in some cases - including Lang, Devlin, Kline, and Jones & Jones - ones I didn't use myself originally but found when looking for a reference to fill out a point on the list).The first sub-list could be good preparation for statistics modules in e.g. an undergraduate psychology course. The idea of the second sub-list is to interweave general maths with the probability maths that's probably the motivating material for the intended reader of this sub-list; it's what I think should be covered at latest by the end of a social science PhD, so you have a good enough foundation to understand typical statistics and read up on further topics and methods relatively efficiently. (If you have the opportunity to follow undergrad maths courses for proofs, calculus, and linear algebra, that can be a huge boost, if they're well-taught; I finally did that myself around the end of my PhD. The suggested sources here might still be useful as complementary material.) Further sub-lists are things I found interesting that built on that foundation.
Note: While I've excluded almost all books that are such didactic garbage they're pretty much sabotaging their readers (I've made allowances for what happens in "starred" sections, and there's one noted exception because the good bits are really good), only a few are really good about helping readers select necessary subsets of exercises. So there's some judgement needed; don't let ego get in the way of being efficient with time, e.g., by only doing some exercises to avoid getting stuck in prerequisite-land that's maybe not inherently motivating for everyone, or by looking up information or solutions online at some points (it can be fun and maybe even important to heroically struggle and transcend, but not if it turns out you're just being blocked by some "cute" exercise you need some unmentioned, non-obvious factoid for). You can always revisit topics if you find out you're shaky on them, as long as you've made a good-faith attempt at a good chunk of exercises at least - not doing any exercises is like going to a gym and just gazing at the fitness machines rather than using them.
The list
- General prerequisites
- Back-to-basics to make sure you're not missing anything that'll trip you up later: The first four parts of the MathTrackX series on edX: Polynomials, Functions and Graphs; Special Functions; Differential Calculus; Integral Calculus. These is about having the "school maths" and numeracy that everything else will assume is known, similarly to literacy. If you find it's needed to go back further, to arithmetic or wherever you need to start, then that's where to start instead!
- Any introduction to very basic linear algebra and matrices. E.g., Savov's No Bullshit Guide to Linear Algebra, chapters 2 and 3. This is mostly to learn the simple but critical language of how data are typically represented and organized - in columns and rows - and later on there will also be necessary maths building on these basic concepts.
- An introduction to the language of proofs and sets: Section 1.16 of Savov.
- Introduction to Probability (STAT110x) on edX: Really well designed and accessible online introduction to foundational probability maths. While statistics isn't the same as probability, statistics is fundamentally about probabilities, so having this basis will make your life infinitely easier in stats modules.
- Fundamental probability and statistics
- I found out late about Lang's Basic Mathematics as a great source for all the foundational, pre-calculus maths that could trip you up trying to get further if you're missing any of it. It could be worth interweaving sections of it between the other topics (doing the whole book at once might be too dry). I've made some provisional suggestions on that. To start, I'd suggest doing Part 1 to start with as a first step here. Using Lang probably replaces some of the original suggestions here, kept here in case they're still helpful as complements or alternatives:
- Just for awareness at this point, to avoid confusion since they're briefly alluded to here and there: Section 1.14 of Savov covering complex numbers. Also, depending on how secure you are in the "school maths" from the first sub-list, it could be worth running through all of Chapter 1 to avoid gaps.
- An introduction to logic, proofs, and set theory and Boolean algebra (including truth tables); e.g., chapters 1 - 3 of Devlin's Sets, Functions, and Logic: An Introduction to Abstract Mathematics. This is the more formal mathematical language used in sources in further steps below; statistics modules and textbooks might also implicitly assume at least some familiarity with the concepts.
- Introduction to Probability by Hwang & Blitzstein, chapters 1 - 4. This is the book the STAT110x online course is based on, but the next step after completing the course is to really work through the book - including the (standard) exercises - you can trust the authors that they're doable; more difficult exercises are clearly marked. It's a time investment but I found it very worth it, as someone who wanted to start properly understanding scientific methods. Subsequent statistical topics in this list will assume a good grasp of probability concepts covered in the book (although which specific ones will vary per topic). At this point, before continuing with further chapters, I've suggested building foundations, but it could be worth having a glance through chapter 5 already to know what you're doing it all for, or to check whether you're already OK.
- Quick digression just for the arithmetic series, since that'll be assumed to be known below. This Intro video on Khan academy covers it, and it's also covered in Lang's chapter 16 on Induction and Summations along with more material.
- Continue up to and including chapter 10 of Lang as preparation for the start of calculus.
- Revisit and consolidate calculus, since the next bits on probability will heavily involve that. I haven't found anything I've fully vetted and been satisfied with but Kline's Calculus: An Intuitive and Physical Approach gives a great start with chapters 1 through 9 (the non-starred subsections), explaining a lot of points in a helpful conversational/lecture style.
- Continue up to and including chapter 13 of Lang, as this covers the trigonometric and exponential functions Kline continues with.
- Continue through chapter 12 of Kline, which will cover what's needed for the next part of Hwang & Blitzstein. (My main issue with Kline concerns the chapters on trigonometry, 10 and 11, which are a bit of a slog that doesn't quite feel optimal or necessary to me, with all the secants and cosecants etc in addition to just sines and cosines; but the chapters aren't skippable since they introduce general points about calculus.)
- Back to Introduction to Probability by Hwang & Blitzstein, chapter 5.
- Further calculus up to the basic idea of multiple integrals. Finishing Kline's Calculus: An Intuitive and Physical Approach is currently still the best single option I'm aware of (again, just the standard, non-starred sections). It's a fair bit of material but I think you can get away with reading-through more than doing many exercises for current purposes.
- And back to Introduction to Probability, chapter 6 onward. I'd suggest studying up to at least conditional expectations, since those will come up a lot in basic statistical methods.
- Further linear algebra, needed for subsequent techniques like regression or Principal Component Analysis. I'd recommend starting on the great Strang lectures here, up to projections and least squares; but be aware that the best edition of the associated book might not be the latest (6th) one (the online course materials refer to the 4th and 5th, most consistently the 4th which is what the assignment numbers refer to, although the assignments are repeated in the solution PDFs anyway). See also chapters 4 through 6.6 of Savov's No Bullshit Guide to Linear Algebra.
- Regression by Bingham & Fry, fully covers linear regression including the contents of the usual mathematical black box where Psychology statistics teaching ends.
- Finish the Strang lectures.
- To avoid gaps around geometry that could come up in further reading: Sections 1 through 3 of Uscka-Wehlou's Calculus 3 (multivariable calculus), part 1 of 2, which cover some remedial bits of geometry very clearly; and finding the area of a triangle from coordinates. (Lang possibly covers this already, but worth being sure.)
- Finish up Lang.
- I found out late about Lang's Basic Mathematics as a great source for all the foundational, pre-calculus maths that could trip you up trying to get further if you're missing any of it. It could be worth interweaving sections of it between the other topics (doing the whole book at once might be too dry). I've made some provisional suggestions on that. To start, I'd suggest doing Part 1 to start with as a first step here. Using Lang probably replaces some of the original suggestions here, kept here in case they're still helpful as complements or alternatives:
- Further statistics and machine learning
- Programming for Computations - Python: A Gentle Introduction to Numerical Simulations with Python 3.6. Very nice introduction to computational methods using Python, including numerical methods for integration and differential equations.
- Principal Component Analysis (PCA): e.g., chapter 15 in Shalizi's Advanced Data Analysis from an Elementary Point of View. PCA is used a lot inside other methods and serves as an examplar of dimension reduction in general.
- The Elements of Statistical Learning by Hastie, Tibshirani and Friedman. This isn't particularly accessibly written (although that's all relative! but let's say it won't handhold and will assume foreknowledge or willingness to look up further information) but it's the Bible of machine learning - I recognized a lot of the book in other texts on machine learning after reading it. I just read this through for the concepts and used relevant parts for reference when working in detail on something.
- Sutton & Barto's Reinforcement Learning. The Bible of reinforcement learning, working up to, e.g., actor-critic models.
- Continued maths
- Probability and Measure by Billingsley. One I just skim-read versus doing exercises, but even that's worth it to see what the fuss is about and to not be intimidated by talk about "sigma algebras".
- Mendelson's Introduction to Topology. Maybe beyond what's strictly needed for basic data analysis purposes so more for general interest and a foundation for getting into more advanced maths, but I thought this was an amazingly well-written book, especially the didactic thought that clearly went into exercises (the very last chapter maybe a little spottier). It covers ideas about the "skin" connecting elements of sets to turn them into spaces with a concept of nearness; lots of vaguely familiar concepts get fully explained. (For awareness, based on reviews I read the specific terminology is dated.)
- An introduction to the very basics of number theory: Jones and Jones' Elementary Number Theory, chapters 1 through 3. This is included here mainly for completeness and since it could pop up in exercises in other books as expected knowledge.
- Spivak's Calculus. This is one you might want to read at some point but only when you're ready for it (I found it had some kind of didactic landmines in it, for me at the time, although with hindsight mostly in sections marked as appendices) - basically when you've already done "normal" calculus courses but now want everything to be really solidly mathematically grounded, and won't get sabotaged by questionable sections.
- Probably missing here - a basic introduction to "standard" multivariate calculus. Currently planning to check out this freely available online course from MIT, or Lang's Calculus of Several Variables.
- H. M. Edwards' Advanced Calculus: A Differential Forms Approach. Building on the foundations above: Another pre-21st century gem in terms of actually "humanly" explaining stuff; it starts off with a different (to me) approach to the symbols around differentials and integrals, so the maths you apparently need for flows and forcefields in arbitrary shapes, centering on Stokes' theorem. It (exceptionally amongst maths books, sadly) gives comprehensive answers to the exercises, which made their difficulty forgivable (although I think it's missing "normal" consolidating exercises in some cases). At the time of writing, I worked up through chapter 6 (and then did it again, which really helped); towards the end of the process that started giving I think at least a decent gist of the concepts, which is really what I wanted for now.
- H. M. Schey's Div, Grad, Curl, and All That: An Informal Text on Vector Calculus. A really lovely little book, as with others mentioned above written in a very conversational, human style. Gives an introduction to the concepts in the title, both deriving them and explaining their meaning and relevance. Useful figures as well and great exercises - really helped build understanding and en passant exposed a lot of points of calculus I was still iffy on. (In principle this probably should come before Edwards, but I quite liked this order personally; possibly because I was missing a traditional multivariate step.)
- Lagrange multipliers - amazingly clear video explaining Lagrange multipliers from Serpentine Integral.