This week we had our second guest lecture. Max Shron presented a live demo of using Google Transit data to analyze the effects of budget cuts on passenger wait times, adapting his original analysis for Chicago to more current New York City MTA data. Max highlighted several useful python modules, including csv.DictReader for easily parsing [...]
In this lecture we discussed recommendation systems in general, and collaborative filtering in particular. With the Netflix Prize as a motivating example, we saw that simple memory-based methods (e.g., nearest-neighbors) are surprisingly effective in practice, although model-based methods (e.g., matrix factorization) often have a number of advantages in scalability and performance. See the slides for [...]
We had our first guest lecture this week. John Myles White presented recent work on modeling data from functional MRI experiments to understand the relationship between various mental states and physical activity in the brain. For more details, see John’s slides and this review paper on “mind reading”. Using Data to Understand the Brain View [...]
In this lecture we extended our toolbox of linear classification methods to include support vector machines (SVMs). We began with a unifying view of loss functions for classification, including squared, logistic, and misclassification loss. We then introduced the perceptron loss as a relaxation of misclassification error and discussed Rosenblatt’s perceptron algorithm. We concluded with an [...]
In this lecture we looked at non-linear feature transformations to accomodate more complex decision boundaries, introduced regularization to avoid overfitting, and covered the kernel trick for learning non-linear predictors. See sections 1.2.5 and 3.1.4 of Bishop along with section 3.4 of Hastie for discussions of regularized least squares. Chapter 6 of Bishop covers kernel methods. [...]
In this lecture we studied maximum likelihood inference for linear classifiers. We saw that ordinary least squares regression can be phrased as maximum likelihood inference under the assumption of additive Gaussian noise. We then derived the closed-form solution to the normal equations for small-scale problems and discussed alternative optimization methods, namely gradient descent, for larger-scale [...]
Our previous discussion of naive Bayes led us to the problem of overfitting, specifically in dealing with rare words for text classification. We investigated this problem a bit more formally in the context of probabilistic modeling and discussed maximum likelihood, maximum a posteriori, and Bayesian methods for parameter estimation. With a Bernoulli model for word [...]
With document classification (e.g., spam filtering) as a motivating example, we reviewed issues with kNN for high-dimensional classification problems—namely the curse of dimensionality—and explored linear methods as an alternative. We first reviewed Bayes’ rule for inverting conditional probabilities via a simple, but perhaps counterintuitive, medical diagnosis example and then adapted this to an (extremely naive) [...]
Last week we looked at the problem of digit recognition and saw that k-nearest neighbors (kNN), while extremely simple, performs reasonably well for this task. We covered two (of many possible) feature space representations of image data, introduced the idea of generalization error, and discussed the benefits and drawkbacks of kNN. The relevant material is [...]
Our first lecture was last Monday. We began with a high-level overview and introduction to the course. Data-driven modeling: Lecture 01 A few main points here. First, for many tasks—e.g. spam classification, image recognition, etc.—we (people) are able to learn relatively quickly and easily, such that we can generalize well after seeing few, loosely-structured examples. [...]