First and second order training algorithms for both shallow and deep neural networks. Initialization lemmas. Approximation of continuous functions and Bayes discriminants using feedforward networks. Structure and training of convolutional networks, and their relationship to conventional pattern recognition systems. Analyses of drop out and mini-batches. Methods for evaluating network performance. Applications in pattern recognition, estimation and forecasting.