Statistical Foundations of Data Science

Chapter I: Introduction

• • Rise of Big Data and Dimensionality
• • Biological Sciences
• • Health Sciences
• • Computer and Information Sciences
• • Economics and Finance
• • Business and Program Evaluation
• • Earth Sciences and Astronomy
• • Impact of Big Data
• • Impact of Dimensionality
• • Computation
• • Noise Accumulation
• • Spurious Correlation
• • Statistical theory
• • Aim of High-dimensional Statistical Learning
• • What big data can do
• • Scope of the book

Chapter 2: Multiple and Nonparametric Regression

• • Introduction
• • Multiple Linear Regression
• • The Gauss-Markov Theorem
• • Statistical Tests
• • Weighted Least-Squares
• • Box-Cox Transformation
• • Model Building and Basis Expansions
• • Polynomial Regression
• • Spline Regression
• • Multiple Covariates
• • Ridge Regression
• • Bias-Variance Tradeo
• • Penalized Least Squares
• • Bayesian Interpretation
• • Ridge Regression Solution Path
• • Kernel Ridge Regression
• • Regression in Reproducing Kernel Hilbert Space
• • Leave-one-out and Generalized Cross-validation
• • Exercises

Chapter 3: Introduction to Penalized Least-Squares

• • Classical Variable Selection Criteria
• • Subset selection
• • Relation with penalized regression
• • Selection of regularization parameters
• • Folded-concave Penalized Least Squares
• • Orthonormal designs
• • Penalty functions
• • Thresholding by SCAD and MCP
• • Risk properties
• • Characterization of folded-concave PLS
• • Lasso and L Regularization
• • Nonnegative garrote
• • Lasso
• • Adaptive Lasso
• • Elastic Net
• • Dantzig selector
• • SLOPE and Sorted Penalties
• • Concentration inequalities and uniform convergence
• • A brief history of model selection
• • Bayesian Variable Selection
• • Bayesian view of the PLS
• • A Bayesian framework for selection
• • Numerical Algorithms
• • Quadratic programs
• • Least angle regression_
• • Local quadratic approximations
• • Local linear algorithm
• • Penalized linear unbiased selection_
• • Cyclic coordinate descent algorithms
• • Iterative shrinkage-thresholding algorithms
• • Projected proximal gradient method
• • ADMM
• • Iterative Local Adaptive Majorization and Minimization
• • Other Methods and Timeline
• • Regularization parameters for PLS
• • Degrees of freedom
• • Extension of information criteria
• • Application to PLS estimators
• • Residual variance and refitted cross-validation
• • Residual variance of Lasso
• • Refitted cross-validation
• • Extensions to Nonparametric Modeling
• • Structured nonparametric models
• • Group penalty
• • Applications
• • Bibliographical notes
• • Exercises

Chapter 4: Penalized Least Squares: Properties

• • Performance Benchmarks
• • Performance measures
• • Impact of model uncertainty
• • Bayes lower bounds for orthogonal design
• • Minimax lower bounds for general design
• • Performance goals, sparsity and sub-Gaussian noise
• • Penalized L Selection
• • Lasso and Dantzig Selector
• • Selection consistency
• • Prediction and coefficient estimation errors
• • Model size and least squares after selection
• • Properties of the Dantzig selector
• • Regularity conditions on the design matrix
• • Properties of Concave PLS
• • Properties of penalty functions
• • Local and oracle solutions
• • Properties of local solutions
• • Global and approximate global solutions
• • Smaller and Sorted Penalties
• • Sorted concave penalties and its local approximation
• • Approximate PLS with smaller and sorted penalties
• • Properties of LLA and LCA
• • Bibliographical notes
• • Exercises

Chapter 5: Generalized Linear Models and Penalized Likelihood

• • Generalized Linear Models
• • Exponential family
• • Elements of generalized linear models
• • Maximum likelihood
• • Computing MLE: Iteratively reweighed least squares
• • Deviance and Analysis of Deviance
• • Residuals
• • Examples
• • Bernoulli and binomial models
• • Models for count responses
• • Models for nonnegative continuous responses
• • Normal error models
• • Sparest solution in high confidence set
• • A general setup
• • Properties
• • Variable Selection via Penalized Likelihood
• • Algorithms
• • Local quadratic approximation
• • Local linear approximation
• • Coordinate descent
• • Iterative Local Adaptive Majorization and Minimization
• • Tuning parameter selection
• • An Application
• • Sampling Properties in low-dimension
• • Notation and regularity conditions
• • The oracle property
• • Sampling Properties with Diverging Dimensions
• • Asymptotic properties of GIC selectors
• • Properties under Ultrahigh Dimensions
• • The Lasso penalized estimator and its risk property
• • Strong oracle property
• • Numeric studies
• • Risk properties
• • Bibliographical notes
• • Exercises

Chapter 6: Penalized M-estimators

• • Penalized quantile regression
• • Quantile regression
• • Variable selection in quantile regression
• • A fast algorithm for penalized quantile regression
• • Penalized composite quantile regression
• • Variable selection in robust regression
• • Robust regression
• • Variable selection in Huber regression
• • Rank regression and its variable selection
• • Rank regression
• • Penalized weighted rank regression
• • Variable Selection for Survival Data
• • Partial likelihood
• • Variable selection via penalized partial likelihood and its properties
• • Theory of folded-concave penalized M-estimator
• • Conditions on penalty and restricted strong convexity
• • Statistical accuracy of penalized M-estimator with folded concave penalties
• • Computational accuracy
• • Bibliographical notes
• • Exercises

Chapter 7: High Dimensional Inference

• • Inference in linear regression
• • Debias of regularized regression estimators
• • Choices of weights
• • Inference for the noise level
• • Inference in generalized linear models
• • Desparsified Lasso
• • Decorrelated score estimator
• • Test of linear hypotheses
• • Numerical comparison
• • An application
• • Asymptotic efficiency
• • Statistical efficiency and Fisher information
• • Linear regression with random design
• • Partial linear regression
• • Gaussian graphical models
• • Inference via penalized least squares
• • Sample size in regression and graphical models
• • General solutions_
• • Local semi-LD decomposition
• • Data swap
• • Gradient approximation
• • Bibliographical notes
• • Exercises

Chapter 8: Feature Screening

• • Correlation Screening
• • Sure screening property
• • Connection to multiple comparison
• • Iterative SIS
• • Generalized and Rank Correlation Screening
• • Feature Screening for Parametric Models
• • Generalized linear models
• • A unified strategy for parametric feature screening
• • Conditional sure independence screening
• • Nonparametric Screening
• • Additive models
• • Varying coefficient models
• • Heterogeneous nonparametric models
• • Model-free Feature Screening
• • Sure independent ranking screening procedure
• • Feature screening via distance correlation
• • Feature screening for high-dimensional categorial data
• • Screening and Selection
• • Feature screening via forward regression
• • Sparse maximum likelihood estimate
• • Feature screening via partial correlation
• • Refitted Cross-Validation
• • RCV algorithm
• • RCV in linear models
• • RCV in nonparametric regression
• • An Illustration
• • Bibliographical notes
• • Exercises

Chapter 9: Covariance Regularization and Graphical Models

• • Basic facts about matrix
• • Sparse Covariance Matrix Estimation
• • Covariance regularization by thresholding and banding
• • Asymptotic properties
• • Nearest positive definite matrices
• • Robust covariance inputs
• • Sparse Precision Matrix and Graphical Models
• • Gaussian graphical models
• • Penalized likelihood and M-estimation
• • Penalized least-squares
• • CLIME and its adaptive version
• • Latent Gaussian Graphical Models
• • Technical Proofs
• • Proof of Theorem
• • Bibliographical notes
• • Exercises

Chapter 10: Covariance Learning and Factor Models

• • Principal Component Analysis
• • Introduction to PCA
• • Power Method
• • Factor Models and Structured Covariance Learning
• • Factor model and high-dimensional PCA
• • Extracting latent factors and POET
• • Methods for selecting number of factors
• • Covariance and Precision Learning with Known Factors
• • Factor model with observable factors
• • Robust initial estimation of covariance matrix
• • Augmented factor models and projected PCA
• • Asymptotic Properties
• • Properties for estimating loading matrix
• • Properties for estimating covariance matrices
• • Properties for estimating realized latent factors
• • Properties for estimating idiosyncratic components
• • Technical Proofs
• • Proof of Theorem
• • Bibliographical Notes
• • Exercises

Chapter 11: Applications of Factor Models and PCA

• • Factor-adjusted Regularized Model Selection
• • Importance of factor adjustments
• • FarmSelect
• • Application to forecasting bond risk premia
• • Application to a neuroblastoma data
• • Asymptotic theory for FarmSelect
• • Factor-adjusted robust multiple testing
• • False discovery rate control
• • Multiple testing under dependence measurements
• • Power of factor adjustments
• • FarmTest
• • Application to neuroblastoma data
• • Factor Augmented Regression Methods
• • Principal Component Regression
• • Augmented Principal Component Regression
• • Application to Forecast Bond Risk Premia
• • Applications to Statistical Machine Learning
• • Community detection
• • Topic model
• • Matrix completion
• • Item ranking
• • Gaussian Mixture models
• • Bibliographical Notes
• • Exercises

Chapter 12: Supervised Learning

• • Model-based Classifiers
• • Linear and quadratic discriminant analysis
• • Logistic regression
• • Kernel Density Classifiers and Naive Bayes
• • Nearest Neighbor Classifiers
• • Classification Trees and Ensemble Classifiers
• • Classification trees
• • Bagging
• • Random forests
• • Boosting
• • Support Vector Machines
• • The standard support vector machine
• • Generalizations of SVMs
• • Sparse Classifiers via Penalized Empirical Loss
• • The importance of sparsity under high-dimensionality
• • Sparse support vector machines
• • Sparse large margin classifiers
• • Sparse Discriminant Analysis
• • Nearest shrunken centroids classifier
• • Features annealed independent rule
• • Selection bias of sparse independence rules
• • Regularized optimal affine discriminant
• • Linear programming discriminant
• • Direct sparse discriminant analysis
• • Solution path equivalence between ROAD and DSDA
• • Feature Augmention and Sparse Additive Classifiers
• • Feature augmentation
• • Penalized additive logistic regression
• • Semiparametric sparse discriminant analysis
• • Bibliographical notes
• • Exercises

Chapter 13: Unsupervised Learning

• • Cluster Analysis
• • K-means clustering
• • Hierarchical clustering
• • Model-based clustering
• • Spectral clustering
• • Data-driven choices of the number of clusters
• • Variable Selection in Clustering
• • Sparse clustering
• • Sparse model-based clustering
• • Sparse mixture of experts model
• • An Introduction to High Dimensional PCA
• • Inconsistency of the regular PCA
• • Consistency under sparse eigenvector model
• • Sparse Principal Component Analysis
• • Sparse PCA
• • An iterative SVD thresholding approach
• • A penalized matrix decomposition approach
• • A semidefinite programming approach
• • A generalized power method
• • Bibliographical notes
• • Exercises

Chapter 14: An Introduction to Deep Learning

• • Rise of Deep Learning
• • Feed-forward neural networks
• • Model setup
• • Back-propagation in computational graphs
• • Popular models
• • Convolutional neural networks
• • Recurrent neural networks
• • Vanilla RNNs
• • GRUs and LSTM
• • Multilayer RNNs
• • Modules
• • Deep unsupervised learning
• • Autoencoders
• • Generative adversarial networks
• • Sampling view of GANs
• • Minimum distance view of GANs
• • Training deep neural nets
• • Stochastic gradient descent
• • Mini-batch SGD
• • Momentum-based SGD
• • SGD with adaptive learning rates
• • Easing numerical instability
• • ReLU activation function
• • Skip connections
• • Batch normalization
• • Regularization techniques
• • Weight decay
• • Dropout
• • Data augmentation
• • Example: image classification
• • Bibliography notes