CAAM 654: Sparse Optimization

Monday and Wednesday, 4:15PM-5:30PM, Duncan 1042.
Instructor: Wotao Yin / Co-Instructor: Ming Yan

Tentative schedule

We meet every Monday and Wednesday at 4:15pm except for Sep.3 (Labor day), Oct.10 (Friday courses on Wednesday), Oct. 29 (instructors out of town).

Lecture notes are password-protected. If you are auditting the class, email for password.

  1. Convex optimization background (1-2 lectures)

    1. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2006.

  2. *Compressive sensing theory including sparse representation and robust recovery conditions: NSP, RIP, SSP, RIPless, etc. (2 lectures)

    1. RIP: E. Candes and T. Tao. Decoding by linear programming. IEEE Transactions on Information Theory, 51:4203-4215, 2005.

    2. RIP shart bound: T. Cai and A. Zhang. Sharp RIP bound for sparse signal and low-rank matrix recovery. Applied and Computational Harmonic Analysis, to appear.

    3. SSP (spherical section property): Y. Zhang. A simple proof for recoverability of l1-minimization: go over or under? Rice University CAAM Technical Report TR05-09, 2005.

    4. RIPless: E. Candes and Y. Plan. A probabilistic and RIPless theory of compressed sensing. Information Theory, IEEE Transactions on, 57(11):7235-7254, 2010.

    5. L1 solution uniqueness: H. Zhang, W. Yin, and L. Cheng. Necessary and sufficient conditions of solution uniqueness in l1 minimization. Rice CAAM Report TR12-18.

  3. *Sparse optimization models and classic solvers (2 lectures)

  4. *Shrinkage (soft-thresholding), primal (prox-linear and higher-order) algorithms (1-2 lectures)

    1. P. L. Combettes and V. R. Wajs, Signal recovery by proximal forward-backward splitting, SIAM Journal on Multiscale Modeling and Simulation, vol. 4, no. 4, pp. 1168-1200, 2005.

    2. A. Beck, and M. Teboulle, 'A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems’, SIAM Journal on Imaging Sciences, 2 (2009), 183-202.

    3. P.L. Combettes, and J.C. Pesquet, 'Proximal Splitting Methods in Signal Processing’, Fixed-Point Algorithms for Inverse Problems in Science and Engineering (2011), 185-212.

    4. S. Wright, R. Nowak, and M. Figueiredo, 'Sparse Reconstruction by Separable Approximation’, 2009.

    5. S. Becker, J. Bobin, and E. Candes, 'Nesta: A Fast and Accurate First-Order Method for Sparse Recovery’, Arxiv preprint arXiv:0904.3367 (2009).

    6. K.C. Toh, and S. Yun, 'An Accelerated Proximal Gradient Algorithm for Nuclear Norm Regularized Linear Least Squares Problems’, Pacific Journal of Optimization, 6 (2010), 15.

    7. S. Ma, D. Goldfarb, and L. Chen, 'Fixed Point and Bregman Iterative Methods for Matrix Rank Minimization’, Mathematical Programming, 128 (2011), 321-53.

    8. S. Ma, W. Yin, Y. Zhang, and A. Chakraborty, 'An Efficient Algorithm for Compressed MR Imaging Using Total Variation and Wavelets’, 2008.

    9. L. Xiao and T. Zhang, A Proximal-Gradient Homotopy methods for the Sparse Least-Squares Problem, arXiv:1203:2003v1, 2012

  5. Dual algorithms (Review, Augmented Lagrangian, Bregman, linearized Bregman) (1 lecture) Lecture 5b Lecture 5c Lecture 5d

    1. R. Glowinski and P. Le Tallec, Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. 1989.

    2. S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, An iterative regularization method for total variation-based image restoration, 2005.

    3. W. Yin, S. Osher, D. Goldfarb, and J. Darbon, Bregman Iterative Algorithms for l1-Minimization with Applications to Compressed Sensing, 2008.

    4. W. Yin and S. Osher, Error forgetting of bregman iteration, 2012.

  6. *Splitting and ADMM (2 Lectures)

    1. Spingarn, J.E. “Applications of the Method of Partial Inverses to Convex Programming: Decomposition.” 1985

    2. G. Chen and M. Teboulle. A proximal-based decomposition method for convex minimization problems. 1994.

    3. B. He, L.Z. Liao, D. Han, and H. Yang. A new inexact alternating directions method for monotone variational inequalities. 2002

    4. Y. Wang, J. Yang, W. Yin, and Y. Zhang. A new alternating minimization algorithm for total variation image recon- struction. 2008

    5. E. Esser. Applications of lagrangian-based alternating direction methods and connections to split bregman. 2009.

    6. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, 2011

    7. X. Zhang, M. Burger, and S. Osher. A unified primal-dual algorithm framework based on bregman iteration. 2011

    8. B. He and X. Yuan, “On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers”, 2012

    9. W. Deng and W. Yin, On the global linear convergence of the alternating direction method of multipliers, 2012.

  7. *Primal versus daul algorithms (code) (1-2 lectures)

  8. Alternating minimization and block coordinate descent (1 lecture)

    1. P. Tseng and S. Yun, A coordinate gradient descent method for non-smooth separable minimization, Mathematical Programming, vol. 117, no. 1, pp. 387-423, 2009.

    2. Y. Li and S. Osher, Coordinate descent optimization for L1 minimization with applications to compressed sensing: a greedy algorithm, UCLA CAM Report 09-17, 2009.

    3. M. Razaviyayn, M. Hong, and Z. Luo, A unified convergence analysis of coordinatewise successive minimization methods for nonsmooth optimization, Report of University of Minnesota, Twin Cites, 2012.

    4. X. Wei, Y. Yuan, and Q. Ling, Doa estimation using a greedy block coordinate descent algorithm, Report of University of Science and Techonolgy of China, 2012.

  9. Homotopy algorithms and parametric quadratic programming (1 lecture)

    1. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, Annals of Statistics, vol. 32, pp. 407-499, 2004.

    2. M. Best, An algorithm for the solution of the parametric quadratic programming problem, CORR Report 82-84, University of Waterloo, 1982.

  10. Non-convex approaches for sparse optimization (1 lecture) Lecture 10b Lecture 10c

    1. E. Candes, M. Wakin, and S. Boyd, Enhancing sparsity by reweighted 1 minimization, Journal of Fourier Analysis and Applications, vol. 14, no. 5, pp. 877-905, 2008.

    2. Rick Chartrand, “Nonconvex splitting for regularized low-rank + sparse decomposition”, To appear in IEEE Transactions on Signal Processing, 2012

  11. *Greedy algorithms, greedy versus optimization (1-2 lectures)

    1. J. Tropp and A. Gilbert, Signal recovery from partial information via orthogonal matching pursuit, IEEE Transactions on Information Theory, vol. 53, no. 12, pp. 4655-4666, 2007.

    2. D. Needell and J.A. Tropp. “Cosamp: Iterative signal recovery from in- complete and inaccurate samples,” Applied and Computational Harmonic Analysis, vol. 26, no. 3, pp. 301-321, 2009.

    3. T. Blumensath and M. Davies, Iterative hard thresholding for com- pressed sensing, Applied and Computational Harmonic Analysis, vol. 27, no. 3, pp. 265-274, 2009.

    4. Y. Wang and W. Yin, Sparse signal reconstruction via iterative support detection, SIAM Journal on Imaging Sciences, vol. 3, no. 3, pp. 462-491, 2010.

    5. S. Foucart, Hard thresholding pursuit: An algorithm for compressive sensing, SIAM Journal on Numerical Analysis, vol. 49, no. 6, pp. 2543-2563, 2011.

  12. *Algorithms for group-sparse vectors, low-rank matrices, model-based signals (1 lecture)

  13. *Project presentations (6-8 lectures)