Computationally challenging training sets appear when one want to emulate intelligence: biological brains learn quite efficiently from the continuous streams of perceptual data generated by our six senses, using limited amounts of sugar as a source of power.

Computationally challenging training sets also appear when one want to analyze the masses of data that describe the life of our computerized society. The more data we understand, the more we enjoy competitive advantages.

The first part of the presentation clarifies the relation between the design of learning algorithms, their statistical efficiency, and their computational cost.

The second part makes a detailed exploration of specific learning algorithms and of their implementation, with both simple and complex examples.

The third part considers algorithms that learn with a single pass over the data. Certain algorithms have optimal properties but are often too costly. Workarounds are discussed.

This work is discussed in the two following papers:

Learning Using Large Datasets

On-line Learning for Very Large Datasets