There are lots of competent textbooks discussing
mathematical concepts that are useful for our course.
The following selection is by no means exhaustive.
B. V. Gnedenko, A. Ya. Khinchin: An Elementary Introduction to the Theory of Probability. W. H. Freeman and co., 1961
Translation from a russian standard: this small book covers the bases with a lot of unusual examples.
P. Billingsley: Probability and Measure, 3rd Edition. Wiley-Interscience, 1995.
The other end of the spectrum in probability theory. A definite reference for all kinds of sophisticated material.
L. N. Trefethen and D. Bau: Numerical Linear Algebra. SIAM, 1997.
A highly rated graduate textbook on numerical algorithms for linear algebra.
P. G. Ciarlet: Introduction to Numerical Linear Algebra and Optimization. Cambridge University Press, 1989.
This book keeps things short and precise. Do not expect to turn the pages quickly. Excellent on optimization.