A Sharp Restricted Isometry Constant Bound of Orthogonal Matching Pursuit

0.07 MB·English

by  Qun Mo

1
A Sharp Restricted Isometry Constant Bound
of Orthogonal Matching Pursuit
Qun Mo
5
1
0
2 Abstract

n
a
J We shall show that if the restricted isometry constant (RIC) δs+1(A) of the measurement matrix

8 A satisfies
1
] δs+1(A)< ,
T √s+1
I
. then the greedy algorithm OrthogonalMatching Pursuit(OMP) will succeed. That is, OMP can recover
s
c
every s-sparse signal x in s iterations from b=Ax. Moreover, we shall show the upper bound of RIC
[

1 is sharp in the following sense. For any given s N, we shall construct a matrix A with the RIC
∈
v
1
8 δs+1(A)=
0 √s+1
7
1 such that OMP may not recover some s-sparse signal x in s iterations.
0
.
1
0 Index Terms
5
1
Compressedsensing,restrictedisometryproperty,orthogonalmatchingpursuit,sparsesignalrecon-
:
v
i struction.
X
r
a
I. INTRODUCTION
Weshallgivea sharpRIC boundofOMPin thispaper. Firstlet us reviewsomebasicconcepts.
Suppose we have a signal x in RN where N is very large. For instance, x represents a digit
photo of 1024 1024 pixels and N = 10242 106. Denote x to be the number of nonzero
0
× ≈ k k
entries of x. We say x is s-sparse if x s; and x is sparse if s N. In real world, there
0
k k ≤ ≪
ResearchsupportedinpartbytheNSFofChinaundergrant10971189and11271010, andbythefundamental researchfunds
for the Central Universities.
Q.MoiswiththeDepartmentofMathematics,ZhejiangUniversity,Hangzhou, 310027, China(e-mail:[email protected] ).
January9,2015 DRAFT
2
are many signals are sparse or can be well approximated by sparse signals, either under the
canonical basis or other special basis/frames. For simplicity, we assume that x is sparse under
the canonical basis in RN.
Suppose we have some linear measurements of a signal x. That is, we have b = Ax where
A Rm N and b Rm are given. The key idea of compressed sensing [3], [8] is that it is
×
∈ ∈
highly possible to retrieve the sparse signal x from those linear measurements, while the number
of measurements is far less than the dimension of the signal, i.e., m N.
≪
To retrieve such a sparse signal x, a natural method is to solve the following l problem
0
min x subject to Ax = b (1)
0
x k k
where A and b are known. To ensure the s–sparse solution is unique, we would like to use the
restricted isometry property (RIP) which was introduced by Cande`s and Tao in [4]. A matrix A
satisfies the RIP of order s with the restricted isometry constant (RIC) δ = δ (A) if δ is the
s s s
smallest constant such that
(1 δ ) x 2 Ax 2 (1+δ ) x 2 (2)
− s k k2 ≤ k k2 ≤ s k k2
holds for all s-sparse signal x.
If δ (A) < 1, the l problem has a unique s-sparse solution [4]. The l problem is equivalent
2s 0 0
to the l minimization problem when δ (A) < √2/2, please see [2], [13], [1] and the references
1 2s
therein.
OMP is an efficient greedy algorithm to solve the l problem [7], [16], [17]. To introduce this
0
algorithm, we denote A by the matrix A with indices of its columns in Ω and we denote x as
Ω Ω
the similar restriction of some vector x. The following iterative algorithm shows the framework
of OMP.
Input: A, b
Set: Ω = , r = b, k = 1
0 0
∅
while not converge
– Ω = Ω argmax r ,Ae
k k 1 i k 1 i
− ∪ |h − i|
– x = argmin A z b
k zk Ωk − k2
– r = b A x
k − Ωk k
– k = k +1
January9,2015 DRAFT
3
end while
xˆ = x , xˆ = 0
Ωk k ΩCk
Return xˆ
Now let us consider the conditions in terms of RIC for OMP to exactly recover any s-sparse
signal in s iterations. Davenport and Wakin [6] haveproven that δ (A) < 1/(3√s) is sufficient.
s+1
Later, Liu and Temlyakov [12] have improved the condition to δ (A) < 1/((√2+1)√s).
s+1
Furthermore, Mo and Shen [14] have pushed the sufficient condition to δ (A) < 1/(√s+1)
s+1
and have shown by detailed examples that a necessary condition is δ (A) < 1/√s. Later, Yang
s+1
and Hoog [10] improved the sufficient condition to δ (A)+√sδ (A) < 1.
s s+1
There is a small gap open between the sufficient condition and the necessary condition of the
best results. And we would like to fill out the gap in this paper.
The content of this paper is consisted by two parts.
We shall prove that the condition
•
δ (A) < 1
s+1 √s+1
is sufficient for OMP to exactly recover any s-sparse x in s iterations.
For any positive integer s, we shall construct a matrix with δ = 1/√s+1 such that
s+1
•
OMP may fail for at least one s-sparse signal in s iterations.
II. PRELIMINARIES
Before going further, let us introduce some notations. For k = 1,2, ,N, define e to be
k
···
the k-th element in the canonical basis of RN. That is, e is the 1-sparse vector in RN such that
k
only the k-th entry of e is 1. For a given matrix A and a given signal x, define
k
S := Ax,Ae , k = 1,...,N.
k k
h i
Denote S := max S . The following two lemmas are useful in our analysis.
0 k 1,...,s k
∈{ }| |
Lemma II.1. For any s > 0, define t := (√s+1 1)/√s. Then we have t2 < 1 and
± −
A(x+te ) 2 A(t2x te ) 2 =
k k k2 −k − k k2 (3)
(1 t4) Ax,Ax √s Ax,Ae k = 1, ,N.
k
− h i± h i ∀ ···
(cid:0) (cid:1)
January9,2015 DRAFT
4
Proof: By the definition of t, t2 = (√s+1 1)2/s = (√s+1 1)/(√s+1 + 1) < 1.
− −
Moreover, by direct expanding and simplification, we have
2t
A(x+te ) 2 A(t2x te ) 2 = (1 t4) Ax,Ax + Ax,Ae .
k k k2 −k − k k2 − h i 1 t2h ki
(cid:18) − (cid:19)
Now we only need to verify that 2t/(1 t2) = √s. This is indeed true since
− ±
2t √s+1 1 √s+1 1
= 2 − / 1 − = √s.
1 t2 ± √s − √s+1+1 ±
− (cid:18) (cid:19)
Lemma II.2. If the restricted isometry constant δ (A) satisfies
s+1
1
δ (A) < (4)
s+1
√s+1
and the support of the signal x is a non-empty subset of 1,2,...,s , then S > S for k > s.
0 k
{ } | |
Proof: Notice this lemma keeps unchanged if we replace x by cx with some non-zero scalar
c. Therefore, we can assume that x = 1. For the given s-sparse x, we obtain
2
k k
s
Ax,Ax = A x e ,Ax
k k
h i
* +
k=1
X
s
= x Ae ,Ax
k k
h i
k=1
X
s
= x S
k k
k=1
X
S x
0 1
≤ k k
S √s x
0 2
≤ k k
= √sS .
0
Define t := (√s+1 1)/√s. Then by the above inequality and Lemma II.1, we have
− −
(1 t4)√s(S S ) (1 t4)( Ax,Ax √s Ax,Ae )
0 k k
− − ≥ − h i− h i (5)
= A(x+te ) 2 A(t2x te ) 2.
k k k2 −k − k k2
January9,2015 DRAFT
5
Moreover, for any k > s, by the definition of δ (A), we obtain
s+1
A(x+te ) 2 A(t2x te ) 2
k k k2 −k − k k2
(1 δ (A))( x 2 +t2) t2(1+δ (A))(t2 x 2 +1)
≥ − s+1 k k2 − s+1 k k2 (6)
= (1+t2) 1 t2 (1+t2)δ (A)
s+1
− −
= (1+t2)2(cid:0) (1 t2)/(1+t2) δ (cid:1)(A) .
s+1
− −
By the definition of t, we hav(cid:0)e (cid:1)
√s+1 1 √s+1 1
(1 t2)/(1+t2) = 1 − / 1+ − = 1/√s+1. (7)
− − √s+1+1 √s+1+1
(cid:18) (cid:19) (cid:18) (cid:19)
Now combining (5), (6), (7) and condition (4), we obtain
(1 t4)√s(S S )
0 k
− −
A(x+te ) 2 A(t2x te ) 2
≥ k k k2 −k − k k2
(1+t2)2 (1 t2)/(1+t2) δ (A)
s+1
≥ − −
= (1+t2)2(cid:0)1/√s+1 δ (A) (cid:1)
s+1
−
(cid:16) (cid:17)
> 0.
Therefore, we have S > S for all k > s. Similarly we can prove that S > S for all k > s.
0 k 0 k
−
Hence, S > S for all k > s.
0 k
| |
III. MAIN RESULTS
Now we are ready to show the main results of this paper.
Theorem III.1. Suppose that A satisfies
1
δ (A) < , (8)
s+1
√s+1
then for any s-sparse signal x, OMP will recover x from b = Ax in s iterations.
Proof:Withoutlossofgenerality,weassumethatthesupportofxisasubsetof 1,2,...,s .
{ }
Thus, the sufficient condition for OMP choosing an index from 1,...,s in the first iteration
{ }
is
S > S for all k > s.
0 k
| |
January9,2015 DRAFT
6
By Lemma II.2, δ < 1 guarantees the success of the first iteration of OMP. OMP makes
s+1 √s+1
an orthogonal projection in each iteration. Hence, it can be proved by induction that OMP always
selects an index from the support of x in s-iterations.
Theorem III.2. For any given positive integer s, there exist a s-sparse signal x and a matrix
A with the restricted isometry constant
1
δ (A) =
s+1
√s+1
such that OMP may fail in s iterations.
Proof: For any given positive integer s, let
1
√s(s+1)
 . 
s I ..
A = s+1 s .
 
 1 
p
 √s(s+1) 
 
 
0 ... 0 1
 
 (s+1) (s+1)
  ×
By simple calculation, we get
1
s+1
.
 s I .. 
ATA = s+1 s ,
 1 
 s+1 
 
 1 ... 1 1+ 1 
 s+1 s+1 s+1 (s+1) (s+1)
  ×
where AT denotes the transpose of A. By direct calculation, we can verify that ATAx = sx/(s+
1) for all x Rs+1 satisfying x = 0 and s x = 0. Therefore, s/(s+1) is an eigenvalue
∈ s+1 k=1 k
of ATA with multiplicity of s 1. MoreovPer, by direct calculation, we can see that ATAx =
−
(1 1/√s+1)xwithx = (1,1,...,1,1 √s+1)T.Therefore,ATAhasanothertwoeigenvalues
± ±
1 1/√s+1. Thus, the restricted isometry constant δ (A) is equal to 1 . Now let
± s+1 √s+1
x = (1,1,...,1,0)T Rs+1.
∈
We have
S = Ae ,Ax = ATAe ,x = s/(s+1) for all k 1,...,s+1 .
k k k
h i h i ∈ { }
January9,2015 DRAFT
7
This implies OMP may fail in the first iteration. Since OMP chooses one index in each iteration,
we conclude that OMP may fail in s iterations for the given matrix A and the s-sparse signal
x.
Remark III.3. For the case of measurements with noise, please see [9], [11], and the references
therein.
REFERENCES
[1] T.TonyCaiandAnruZhang,SparseRepresentationofaPolytopeandRecoveryofSparseSignalsandLow-RankMatrices,
IEEE Trans. Inform. Theory, 60(1), 122 - 132, 2014.
[2] E.J. Cande`s, The restricted isometry property and its implications for compressed sensing, C. R. Math. Acad. Sci. Paris,
Ser. I, 346 589-592, 2008.
[3] E.J. Cande`s, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete
frequency information, IEEE Trans. Inform. Theory, 52(2) 489-509, 2006.
[4] E.J. Cande`s and T. Tao, Decoding by linear programming, IEEE Trans. Inform. Theory, 51(12) 4203-4215, 2005.
[5] W. Dai and O. Milenkovic, Subspace pursuit for compressive sensing signal reconstruction, IEEE Trans. Inform. Theory,
55(5) 2230-2249, 2009.
[6] M.A. Davenport and M.B. Wakin, Analysis of orthogonal matching pursuit using the restricted isometry property, IEEE
Trans. Inform. Theory, 56(9) 4395-4401, 2010.
[7] G. Davis, S. Mallat, and M. Avellaneda, Adaptive greedy approximation, J. Constr. Approx., 13, 57-98, 1997.
[8] D. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, 52(4) 1289-1306, 2006.
[9] Y.ShenandS.Li,SparseSignalsRecoveryfromNoisyMeasurementsbyOrthogonal MatchingPursuit,Inverseproblems
and imaging, to appear.
[10] M. Yang, and F.Hoog, New Coherence and RIPAnalysis for Greedy AlgorithmsinCompressive Sensing, arXivpreprint
arXiv:1405.3354(2014).
[11] R. Wu, H. Wei,and Di-RongChen, TheExact Support Recovery of Sparse SignalsWithNoise viaOrthogonal Matching
Pursuit, IEEE Signal Process. Lett. 20.4 (2013): 403-406.
[12] E. Liu and V.N. Temlyakov, Orthogonal Super Greedy Algorithm and Applications in Compressed Sensing, IEEE Trans.
Inform. Theory 58(4), 2040-2047, 2012.
[13] Q.MoandL.Song,Newboundsontherestrictedisometryconstantδ2k,Appl.Comput. Harmon.Anal.,31(3),460C468,
2011.
[14] Q. Mo and Y. Shen, A remark on the restricted isometry property in orthogonal matching pursuit algorithm, IEEE Trans.
Inform. Theory, 58(6) 3654-3656, 2012.
[15] D. Needell and J.A. Tropp, CoSaMP: Iterative signal recovery from incomplete and inaccurate samples, Appl. Comp.
Harmonic Anal., 26(3) 301-321, 2009.
[16] Y.C. Pati, R. Rezaiifar and P.S. Krishnaprasad, Orthogonal Matching Pursuit: Recursive function approximation with
applicationstowaveletdecomposition.inProc.27thAnn.AsilomarConf.onSignals,SystemsandComputers,Nov.1993.
[17] J.A.Tropp, Greedisgood: Algorithmicresultsfor sparseapproximation, IEEETrans. Inform.Theory, 50(10) 2231-2242,
2004.
January9,2015 DRAFT
8
Qun Mo was born in 1971 in China. He has obtained a B.Sc. degree in 1994 from Tsinghua University, a M.Sc. degree in
1997 from Chinese Academy of Sciences and a Ph.D. degree in 2003 from University of Alberta in Canada.
HeiscurrentanassociateprofessorinmathematicsinZhejiangUniversity.Hisresearchinterestsincludecompressedsensing,
wavelets and their applications.
January9,2015 DRAFT