Volume 71, Issue 5

Sparse additive models

Pradeep Ravikumar

University of California, Berkeley, USA

Search for more papers by this author
John Lafferty

Carnegie Mellon University, Pittsburgh, USA

Search for more papers by this author
Han Liu

Carnegie Mellon University, Pittsburgh, USA

Search for more papers by this author
Larry Wasserman

Carnegie Mellon University, Pittsburgh, USA

Search for more papers by this author
First published: 19 October 2009
Citations: 151
Larry Wasserman, Department of Statistics, 232 Baker Hall, Carnegie Mellon University, Pittsburgh, PA 15213‐3890, USA.
E‐mail: larry@stat.cmu.edu

Abstract

Summary. We present a new class of methods for high dimensional non‐parametric regression and classification called sparse additive models. Our methods combine ideas from sparse linear modelling and additive non‐parametric regression. We derive an algorithm for fitting the models that is practical and effective even when the number of covariates is larger than the sample size. Sparse additive models are essentially a functional version of the grouped lasso of Yuan and Lin. They are also closely related to the COSSO model of Lin and Zhang but decouple smoothing and sparsity, enabling the use of arbitrary non‐parametric smoothers. We give an analysis of the theoretical properties of sparse additive models and present empirical results on synthetic and real data, showing that they can be effective in fitting sparse non‐parametric models in high dimensional data.

Number of times cited according to CrossRef: 151

  • Designing Reactive Power Control Rules for Smart Inverters Using Support Vector Machines, IEEE Transactions on Smart Grid, 10.1109/TSG.2019.2942850, 11, 2, (1759-1770), (2020).
  • Conditional density estimation tools in python and R with applications to photometric redshifts and likelihood-free cosmological inference, Astronomy and Computing, 10.1016/j.ascom.2019.100362, (100362), (2020).
  • Group Sparse Additive Machine with Average Top-k Loss, Neurocomputing, 10.1016/j.neucom.2020.01.104, (2020).
  • Correlation and association analyses in microbiome study integrating multiomics in health and disease, , 10.1016/bs.pmbts.2020.04.003, (2020).
  • Explaining the Neural Network: A Case Study to Model the Incidence of Cervical Cancer, Information Processing and Management of Uncertainty in Knowledge-Based Systems, 10.1007/978-3-030-50146-4_43, (585-598), (2020).
  • A projection-based conditional dependence measure with applications to high-dimensional undirected graphical models, Journal of Econometrics, 10.1016/j.jeconom.2019.12.016, (2020).
  • undefined, 2020 American Control Conference (ACC), 10.23919/ACC45564.2020.9147515, (126-131), (2020).
  • Comparative transcriptome analysis digs out genes related to antifreeze between fresh and frozen–thawed rooster sperm, Poultry Science, 10.1016/j.psj.2020.01.022, (2020).
  • Learning in Reproducing Kernel Hilbert Spaces, Machine Learning, 10.1016/B978-0-12-818803-3.00022-2, (531-594), (2020).
  • Ultrahigh dimensional feature screening for additive model with multivariate response, Journal of Statistical Computation and Simulation, 10.1080/00949655.2019.1703371, (1-25), (2020).
  • Sparse additive machine with ramp loss, Analysis and Applications, 10.1142/S0219530520400011, (1-20), (2020).
  • A sparse additive model for treatment effect-modifier selection, Biostatistics, 10.1093/biostatistics/kxaa032, (2020).
  • Kernel Meets Sieve: Post-Regularization Confidence Bands for Sparse Additive Model, Journal of the American Statistical Association, 10.1080/01621459.2019.1689984, (1-16), (2020).
  • Spike-and-Slab Group Lassos for Grouped Regression and Sparse Generalized Additive Models, Journal of the American Statistical Association, 10.1080/01621459.2020.1765784, (1-14), (2020).
  • AdaBoost Semiparametric Model Averaging Prediction for Multiple Categories, Journal of the American Statistical Association, 10.1080/01621459.2020.1790375, (1-15), (2020).
  • Rank reduction for high-dimensional generalized additive models, Journal of Multivariate Analysis, 10.1016/j.jmva.2019.05.005, (2019).
  • Epidemiologic Studies of Medical Devices, Pharmacoepidemiology, 10.1002/9781119413431, (496-523), (2019).
  • Improving Prediction Efficacy Through Abnormality Detection and Data Preprocessing, IEEE Access, 10.1109/ACCESS.2019.2930257, 7, (103794-103805), (2019).
  • Block HSIC Lasso: model-free biomarker detection for ultra-high dimensional data, Bioinformatics, 10.1093/bioinformatics/btz333, 35, 14, (i427-i435), (2019).
  • Consistent Variable Selection for a Nonparametric Nonlinear System by Inverse and Contour Regressions, IEEE Transactions on Automatic Control, 10.1109/TAC.2018.2867252, 64, 7, (2653-2664), (2019).
  • Additive modeling and prediction of transient stability boundary in large-scale power systems using the Group Lasso algorithm, International Journal of Electrical Power & Energy Systems, 10.1016/j.ijepes.2019.05.068, 113, (963-970), (2019).
  • , High-Dimensional Statistics, 10.1017/9781108627771, (2019).
  • Variable selection of high-dimensional non-parametric nonlinear systems by derivative averaging to avoid the curse of dimensionality, Automatica, 10.1016/j.automatica.2018.11.019, 101, (138-149), (2019).
  • Sure independence screening in ultrahigh dimensional generalized additive models, Journal of Statistical Planning and Inference, 10.1016/j.jspi.2018.04.004, 199, (126-135), (2019).
  • undefined, AIAA Propulsion and Energy 2019 Forum, 10.2514/6.2019-4088, (2019).
  • Machine Learning Approach for Graphical Model-Based Analysis of Energy-Aware Growth Control in Plant Factories, IEEE Access, 10.1109/ACCESS.2019.2903830, (1-1), (2019).
  • Testing if a nonlinear system is additive or not, Automatica, 10.1016/j.automatica.2019.02.053, 104, (134-140), (2019).
  • Early Detection and Risk Assessment for Chronic Disease with Irregular Longitudinal Data Analysis, Journal of Biomedical Informatics, 10.1016/j.jbi.2019.103231, (103231), (2019).
  • Robust Variable Selection and Estimation Based on Kernel Modal Regression, Entropy, 10.3390/e21040403, 21, 4, (403), (2019).
  • Learning General Sparse Additive Models from Point Queries in High Dimensions, Constructive Approximation, 10.1007/s00365-019-09461-6, (2019).
  • Spatio-temporal additive regression model selection for urban water demand, Stochastic Environmental Research and Risk Assessment, 10.1007/s00477-019-01682-2, (2019).
  • Integrated Framework for Selection of Additive and Nonadditive Genetic Markers for Genomic Selection, Journal of Computational Biology, 10.1089/cmb.2019.0223, (2019).
  • Variable selection in multivariate linear models for functional data via sparse regularization, Japanese Journal of Statistics and Data Science, 10.1007/s42081-019-00055-x, (2019).
  • Multiresolution Functional ANOVA for Large-Scale, Many-Input Computer Experiments, Journal of the American Statistical Association, 10.1080/01621459.2019.1595630, (1-23), (2019).
  • Generalized additive model with embedded variable selection for bankruptcy prediction: Prediction versus interpretation, Cogent Economics & Finance, 10.1080/23322039.2019.1597956, 7, 1, (2019).
  • Functional Horseshoe Priors for Subspace Shrinkage, Journal of the American Statistical Association, 10.1080/01621459.2019.1654875, (1-14), (2019).
  • Uncertainty Quantification for High-Dimensional Sparse Nonparametric Additive Models, Technometrics, 10.1080/00401706.2019.1665591, (1-12), (2019).
  • Partially Linear Functional Additive Models for Multivariate Functional Data, Journal of the American Statistical Association, 10.1080/01621459.2017.1411268, 114, 525, (406-418), (2018).
  • Semiparametric method for model structure discovery in additive regression models, Econometrics and Statistics, 10.1016/j.ecosta.2017.02.005, 5, (124-136), (2018).
  • Generalized Additive Models, Semiparametric Regression with R, 10.1007/978-1-4939-8853-2_3, (71-128), (2018).
  • Empirical Asset Pricing via Machine Learning, SSRN Electronic Journal, 10.2139/ssrn.3281018, (2018).
  • Empirical Asset Pricing Via Machine Learning, SSRN Electronic Journal, 10.2139/ssrn.3159577, (2018).
  • How to Make Causal Inferences with Time-Series Cross-Sectional Data Under Selection on Observables, SSRN Electronic Journal, 10.2139/ssrn.3176910, (2018).
  • From Black-Box to White-Box: Interpretable Learning with Kernel Machines, Machine Learning and Data Mining in Pattern Recognition, 10.1007/978-3-319-96136-1_18, (213-227), (2018).
  • Regression with stagewise minimization on risk function, Computational Statistics & Data Analysis, 10.1016/j.csda.2018.12.011, (2018).
  • Compositional Stochastic Average Gradient for Machine Learning and Related Applications, Intelligent Data Engineering and Automated Learning – IDEAL 2018, 10.1007/978-3-030-03493-1_77, (740-752), (2018).
  • Nonparametric Methods for Big Data Analytics, Handbook of Big Data Analytics, 10.1007/978-3-319-18284-1_5, (103-124), (2018).
  • Scattered Data and Aggregated Inference, Handbook of Big Data Analytics, 10.1007/978-3-319-18284-1_4, (75-102), (2018).
  • Ultra High-Dimensional Nonlinear Feature Selection for Big Biological Data, IEEE Transactions on Knowledge and Data Engineering, 10.1109/TKDE.2018.2789451, 30, 7, (1352-1365), (2018).
  • Error analysis for coefficient-based regularized regression in additive models, Statistics & Probability Letters, 10.1016/j.spl.2017.10.001, 134, (22-28), (2018).
  • High-dimensional supervised feature selection via optimized kernel mutual information, Expert Systems with Applications, 10.1016/j.eswa.2018.04.037, 108, (81-95), (2018).
  • Variable selection for high dimensional Gaussian copula regression model: An adaptive hypothesis testing procedure, Computational Statistics & Data Analysis, 10.1016/j.csda.2018.03.003, 124, (132-150), (2018).
  • PAC-Bayesian risk bounds for group-analysis sparse regression by exponential weighting, Journal of Multivariate Analysis, 10.1016/j.jmva.2018.12.004, (2018).
  • Feature screening in ultrahigh-dimensional additive Cox model, Journal of Statistical Computation and Simulation, 10.1080/00949655.2017.1422127, 88, 6, (1117-1133), (2018).
  • Nonparametric independence screening for ultra-high-dimensional longitudinal data under additive models, Journal of Nonparametric Statistics, 10.1080/10485252.2018.1497797, 30, 4, (884-905), (2018).
  • Bayesian Regression Trees for High-Dimensional Prediction and Variable Selection, Journal of the American Statistical Association, 10.1080/01621459.2016.1264957, 113, 522, (626-636), (2018).
  • Distribution-Free Predictive Inference for Regression, Journal of the American Statistical Association, 10.1080/01621459.2017.1307116, 113, 523, (1094-1111), (2018).
  • Bayesian Neural Networks for Selection of Drug Sensitive Genes, Journal of the American Statistical Association, 10.1080/01621459.2017.1409122, 113, 523, (955-972), (2018).
  • How to Make Causal Inferences with Time-Series Cross-Sectional Data under Selection on Observables, American Political Science Review, 10.1017/S0003055418000357, (1-16), (2018).
  • Nonparametric regression with adaptive truncation via a convex hierarchical penalty, Biometrika, 10.1093/biomet/asy056, (2018).
  • Nonlinear sparse feature selection algorithm via low matrix rank constraint, Multimedia Tools and Applications, 10.1007/s11042-018-6909-1, (2018).
  • Improving the prediction performance of the LASSO by subtracting the additive structural noises, Computational Statistics, 10.1007/s00180-018-0849-0, (2018).
  • ABC–CDE: Toward Approximate Bayesian Computation With Complex High-Dimensional Data and Limited Simulations, Journal of Computational and Graphical Statistics, 10.1080/10618600.2018.1546594, (1-20), (2018).
  • RANK: Large-Scale Inference With Graphical Nonlinear Knockoffs, Journal of the American Statistical Association, 10.1080/01621459.2018.1546589, (1-43), (2018).
  • Feature Screening for Nonparametric and Semiparametric Models with Ultrahigh-Dimensional Covariates, Journal of Systems Science and Complexity, 10.1007/s11424-017-6310-6, 31, 5, (1350-1361), (2017).
  • High-dimensional multivariate additive regression for uncovering contributing factors to healthcare expenditure, Biostatistics, 10.1093/biostatistics/kxx043, 19, 3, (359-373), (2017).
  • Algorithms for learning sparse additive models with interactions in high dimensions*, Information and Inference: A Journal of the IMA, 10.1093/imaiai/iax008, 7, 2, (183-249), (2017).
  • Error Variance Estimation in Ultrahigh-Dimensional Additive Models, Journal of the American Statistical Association, 10.1080/01621459.2016.1251440, 113, 521, (315-327), (2017).
  • Sparse Additive Gaussian Process with Soft Interactions, Open Journal of Statistics, 10.4236/ojs.2017.74039, 07, 04, (567-588), (2017).
  • SLANTS: Sequential Adaptive Nonlinear Modeling of Time Series, IEEE Transactions on Signal Processing, 10.1109/TSP.2017.2716898, 65, 19, (4994-5005), (2017).
  • undefined, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 10.1109/ICASSP.2017.7952981, (4366-4370), (2017).
  • undefined, 2017 IEEE International Conference on Big Data (Big Data), 10.1109/BigData.2017.8258067, (1357-1362), (2017).
  • Feature Selection with Annealing for Computer Vision and Big Data Learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 10.1109/TPAMI.2016.2544315, 39, 2, (272-286), (2017).
  • A Bayesian Additive Model for Understanding Public Transport Usage in Special Events, IEEE Transactions on Pattern Analysis and Machine Intelligence, 10.1109/TPAMI.2016.2635136, 39, 11, (2113-2126), (2017).
  • LogitBoost autoregressive networks, Computational Statistics & Data Analysis, 10.1016/j.csda.2017.03.010, 112, (88-98), (2017).
  • Multi-Penalty Regularization for Detecting Relevant Variables, Recent Applications of Harmonic Analysis to Function Spaces, Differential Equations, and Data Science, 10.1007/978-3-319-55556-0_15, (889-916), (2017).
  • Sparse kernel canonical correlation analysis for discovery of nonlinear interactions in high-dimensional data, BMC Bioinformatics, 10.1186/s12859-017-1543-x, 18, 1, (2017).
  • Variable Selection in Kernel Regression Using Measurement Error Selection Likelihoods, Journal of the American Statistical Association, 10.1080/01621459.2016.1222287, 112, 520, (1587-1597), (2017).
  • Network Reconstruction From High-Dimensional Ordinary Differential Equations, Journal of the American Statistical Association, 10.1080/01621459.2016.1229197, 112, 520, (1697-1707), (2017).
  • Two stage smoothing in additive models with missing covariates, Statistical Papers, 10.1007/s00362-017-0896-6, (2017).
  • Additive model building for spatial regression, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 10.1111/rssb.12195, 79, 3, (779-800), (2016).
  • A unified penalized method for sparse additive quantile models: an RKHS approach, Annals of the Institute of Statistical Mathematics, 10.1007/s10463-016-0566-9, 69, 4, (897-923), (2016).
  • Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions, Mathematical Programming, 10.1007/s10107-016-1017-3, 161, 1-2, (419-449), (2016).
  • Variable selection via additive conditional independence, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 10.1111/rssb.12150, 78, 5, (1037-1055), (2016).
  • Non-asymptotic oracle inequalities for the Lasso and Group Lasso in high dimensional logistic model, ESAIM: Probability and Statistics, 10.1051/ps/2015020, 20, (309-331), (2016).
  • Automatic Component Selection in Additive Modeling of French National Electricity Load Forecasting, Nonparametric Statistics, 10.1007/978-3-319-41582-6_14, (191-209), (2016).
  • Structured variable selection via prior-induced hierarchical penalty functions, Computational Statistics & Data Analysis, 10.1016/j.csda.2015.10.011, 96, (87-103), (2016).
  • Electricity Forecasting Using Multi-Stage Estimators of Nonlinear Additive Models, IEEE Transactions on Power Systems, 10.1109/TPWRS.2015.2504921, 31, 5, (3665-3673), (2016).
  • Structured Dimensionality Reduction for Additive Model Regression, IEEE Transactions on Knowledge and Data Engineering, 10.1109/TKDE.2016.2525996, 28, 6, (1589-1601), (2016).
  • Additive model selection, Statistical Methods & Applications, 10.1007/s10260-016-0357-8, 25, 4, (519-564), (2016).
  • Bias-corrected inference for multivariate nonparametric regression: Model selection and oracle property, Journal of Multivariate Analysis, 10.1016/j.jmva.2015.08.016, 143, (71-93), (2016).
  • Flexible variable selection for recovering sparsity in nonadditive nonparametric models, Biometrics, 10.1111/biom.12518, 72, 4, (1155-1163), (2016).
  • Generalized varying index coefficient models, Journal of Computational and Applied Mathematics, 10.1016/j.cam.2015.11.025, 300, (1-17), (2016).
  • Continuously dynamic additive models for functional data, Journal of Multivariate Analysis, 10.1016/j.jmva.2016.05.003, 150, (1-13), (2016).
  • Fused Lasso Additive Model, Journal of Computational and Graphical Statistics, 10.1080/10618600.2015.1073155, 25, 4, (1005-1025), (2016).
  • Sparse Partially Linear Additive Models, Journal of Computational and Graphical Statistics, 10.1080/10618600.2015.1089775, 25, 4, (1126-1140), (2016).
  • Bayesian Additive Machine: classification with a semiparametric discriminant function, Journal of Statistical Computation and Simulation, 10.1080/00949655.2015.1028937, 86, 4, (682-695), (2015).
  • Robust variable selection method for nonparametric differential equation models with application to nonlinear dynamic gene regulatory network analysis, Journal of Biopharmaceutical Statistics, 10.1080/10543406.2015.1052496, 26, 4, (712-724), (2015).
  • A selective overview of feature screening for ultrahigh-dimensional data, Science China Mathematics, 10.1007/s11425-015-5062-9, 58, 10, (1-22), (2015).
  • undefined, 2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 10.1109/CAMSAP.2015.7383814, (373-376), (2015).
  • See more