CAAM 654: Sparse Optimization
Monday and Wednesday, 4:15PM5:30PM, Duncan 1042.
Instructor: Wotao Yin / CoInstructor: Ming Yan
Tentative schedule
We meet every Monday and Wednesday at 4:15pm except for Sep.3 (Labor day), Oct.10 (Friday courses on Wednesday), Oct. 29 (instructors out of town).
Lecture notes are passwordprotected. If you are auditting the class, email yanm@rice.edu for password.
Convex optimization background (12 lectures)
S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2006.
*Compressive sensing theory including sparse representation and robust recovery conditions: NSP, RIP, SSP, RIPless, etc. (2 lectures)
RIP: E. Candes and T. Tao. Decoding by linear programming. IEEE Transactions on Information Theory, 51:42034215, 2005.
RIP shart bound: T. Cai and A. Zhang. Sharp RIP bound for sparse signal and lowrank matrix recovery. Applied and Computational Harmonic Analysis, to appear.
SSP (spherical section property): Y. Zhang. A simple proof for recoverability of l1minimization: go over or under? Rice University CAAM Technical Report TR0509, 2005.
RIPless: E. Candes and Y. Plan. A probabilistic and RIPless theory of compressed sensing. Information Theory, IEEE Transactions on, 57(11):72357254, 2010.
L1 solution uniqueness: H. Zhang, W. Yin, and L. Cheng. Necessary and sufficient conditions of solution
uniqueness in l1 minimization. Rice CAAM Report TR1218.
*Sparse optimization models and classic solvers (2 lectures)
*Shrinkage (softthresholding), primal (proxlinear and higherorder) algorithms (12 lectures)
P. L. Combettes and V. R. Wajs, Signal recovery by proximal forwardbackward splitting, SIAM Journal on Multiscale Modeling and Simulation, vol. 4, no. 4, pp. 11681200, 2005.
A. Beck, and M. Teboulle, 'A Fast Iterative ShrinkageThresholding Algorithm for Linear Inverse Problems’, SIAM Journal on Imaging Sciences, 2 (2009), 183202.
P.L. Combettes, and J.C. Pesquet, 'Proximal Splitting Methods in Signal Processing’, FixedPoint Algorithms for Inverse Problems in Science and Engineering (2011), 185212.
S. Wright, R. Nowak, and M. Figueiredo, 'Sparse Reconstruction by Separable Approximation’, 2009.
S. Becker, J. Bobin, and E. Candes, 'Nesta: A Fast and Accurate FirstOrder Method for Sparse Recovery’, Arxiv preprint arXiv:0904.3367 (2009).
K.C. Toh, and S. Yun, 'An Accelerated Proximal Gradient Algorithm for Nuclear Norm Regularized Linear Least Squares Problems’, Pacific Journal of Optimization, 6 (2010), 15.
S. Ma, D. Goldfarb, and L. Chen, 'Fixed Point and Bregman Iterative Methods for Matrix Rank Minimization’, Mathematical Programming, 128 (2011), 32153.
S. Ma, W. Yin, Y. Zhang, and A. Chakraborty, 'An Efficient Algorithm for Compressed MR Imaging Using Total Variation and Wavelets’, 2008.
L. Xiao and T. Zhang, A ProximalGradient Homotopy methods for the Sparse LeastSquares Problem, arXiv:1203:2003v1, 2012
Dual algorithms (Review, Augmented Lagrangian, Bregman, linearized Bregman) (1 lecture) Lecture 5b Lecture 5c Lecture 5d
R. Glowinski and P. Le Tallec, Augmented Lagrangian and OperatorSplitting Methods in Nonlinear Mechanics. 1989.
S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, An iterative regularization method for total variationbased image restoration, 2005.
W. Yin, S. Osher, D. Goldfarb, and J. Darbon, Bregman Iterative Algorithms for l1Minimization with Applications to Compressed Sensing, 2008.
W. Yin and S. Osher, Error forgetting of bregman iteration, 2012.
*Splitting and ADMM (2 Lectures)
Spingarn, J.E. “Applications of the Method of Partial Inverses to Convex Programming: Decomposition.” 1985
G. Chen and M. Teboulle. A proximalbased decomposition method for convex minimization problems. 1994.
B. He, L.Z. Liao, D. Han, and H. Yang. A new inexact alternating directions method for monotone variational inequalities. 2002
Y. Wang, J. Yang, W. Yin, and Y. Zhang. A new alternating minimization algorithm for total variation image recon
struction. 2008
E. Esser. Applications of lagrangianbased alternating direction methods and connections to split bregman. 2009.
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, 2011
X. Zhang, M. Burger, and S. Osher. A unified primaldual algorithm framework based on bregman iteration. 2011
B. He and X. Yuan, “On nonergodic convergence rate of DouglasRachford
alternating direction method of multipliers”, 2012
W. Deng and W. Yin, On the global linear convergence of the alternating direction method of multipliers, 2012.
*Primal versus daul algorithms (code) (12 lectures)
Alternating minimization and block coordinate descent (1 lecture)
P. Tseng and S. Yun, A coordinate gradient descent method for nonsmooth separable minimization, Mathematical Programming, vol. 117, no. 1, pp. 387423, 2009.
Y. Li and S. Osher, Coordinate descent optimization for L1 minimization with applications to compressed sensing: a greedy algorithm, UCLA CAM Report 0917, 2009.
M. Razaviyayn, M. Hong, and Z. Luo, A unified convergence analysis of coordinatewise successive minimization methods for nonsmooth optimization, Report of University of Minnesota, Twin Cites, 2012.
X. Wei, Y. Yuan, and Q. Ling, Doa estimation using a greedy block coordinate descent algorithm, Report of University of Science and Techonolgy of China, 2012.
Homotopy algorithms and parametric quadratic programming (1 lecture)
B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, Annals of Statistics, vol. 32, pp. 407499, 2004.
M. Best, An algorithm for the solution of the parametric quadratic programming problem, CORR Report 8284, University of Waterloo, 1982.
Nonconvex approaches for sparse optimization (1 lecture) Lecture 10b Lecture 10c
E. Candes, M. Wakin, and S. Boyd, Enhancing sparsity by reweighted 1 minimization, Journal of Fourier Analysis and Applications, vol. 14, no. 5, pp. 877905, 2008.
Rick Chartrand, “Nonconvex splitting for regularized lowrank + sparse decomposition”, To appear in IEEE Transactions on Signal Processing, 2012
*Greedy algorithms, greedy versus optimization (12 lectures)
J. Tropp and A. Gilbert, Signal recovery from partial information via orthogonal matching pursuit, IEEE Transactions on Information Theory, vol. 53, no. 12, pp. 46554666, 2007.
D. Needell and J.A. Tropp. “Cosamp: Iterative signal recovery from in complete and inaccurate samples,” Applied and Computational Harmonic Analysis, vol. 26, no. 3, pp. 301321, 2009.
T. Blumensath and M. Davies, Iterative hard thresholding for com pressed sensing, Applied and Computational Harmonic Analysis, vol. 27, no. 3, pp. 265274, 2009.
Y. Wang and W. Yin, Sparse signal reconstruction via iterative support detection, SIAM Journal on Imaging Sciences, vol. 3, no. 3, pp. 462491, 2010.
S. Foucart, Hard thresholding pursuit: An algorithm for compressive sensing, SIAM Journal on Numerical Analysis, vol. 49, no. 6, pp. 25432563, 2011.
*Algorithms for groupsparse vectors, lowrank matrices, modelbased signals (1 lecture)
*Project presentations (68 lectures)
