$\newcommand{\R}{\mathbb R }$ $\newcommand{\N}{\mathbb N }$ $\newcommand{\Z}{\mathbb Z }$ $\newcommand{\bfa}{\mathbf a}$ $\newcommand{\bfb}{\mathbf b}$ $\newcommand{\bff}{\mathbf f}$ $\newcommand{\bfu}{\mathbf u}$ $\newcommand{\bfx}{\mathbf x}$ $\newcommand{\ep}{\varepsilon}$
This is an abbreviated version of material in Sections 1.4 and 1.5 of Folland's Advanced Calculus. For these sections, students will be responsible for only the material covered in these notes.
A common technique for solving a problem (think for example of minimizing a function $\bff: \R^n\to \R^n$) is the following:
start with an initial guess $\bfx_1\in \R^n$.
devise some method for improving the guess, to get an updated guess (let's call it $\bfx_2$) that ideally comes closer to solving the problem.
Repeatedly apply the method for improving the guess, calling the new points $\bfx_3$, then $ \bfx_4$ and so on ...
This generates a sequence $\bfx_1, \bfx_2, \bfx_3, \bfx_4, \bfx_5, \ldots$.
One can then try to prove that the sequence converges to a limit, and that the limit solves the problem we are interested in.
To carry out the last point, we would like to have theorems that allow us to conclude that a sequence has a limit (under certain hypotheses).
If we want such a theorem, the best thing we can imagine would be a theorem saying that Every sequence in $\R^n$ converges to a limit
. But this is clearly false. (Consider for example the sequence $\bfx_j = j {\bf e}_1$.)
The next best thing we can imagine would be a theorem saying that Every bounded sequence in $\R^n$ converges to a limit
. But this is also clearly false. (Consider for example the sequence $\bfx_j = (-1)^j {\bf e}_1$.)
However, if we change the statement a little, we get a theorem that is in fact true, that enables us to find convergent sequences. To do this, we will need to introduce the notion of a subsequence - see below. With this, we will prove
Theorem 1: Bounded Sequence Theorem. Every bounded sequence in $\R^n$ has a subsequence that converges to a limit.
This is an excellent theorem if you like convergent sequences. It is also very useful for proving that certain kinds of problems (for example, minimization problems) have solutions, as we will see.
The theorem is connected to the completeness property of $\R^n$, as we will discuss below.
From MAT137 you are familiar with sequences of real numbers. A sequence may be written in various ways, including $$ \{ a_j \}_{j=1}^\infty,\qquad \{ a_j \}_j,\qquad \{ a_j \}, \quad \ldots $$
A sequence in $\R^n$ may be written as $$ \{ \bfa_j \}_{j=1}^\infty,\qquad \{ \bfa_j \}_j,\qquad \{ \bfa_j \}, \quad \ldots $$ It is exactly like a sequence of real numbers, except that the terms $\bfa_1,\bfa_2, \ldots$ in the sequence all elements of $\R^n$ rather than real numbers.
(Thus, if we like, we can mimic the formal definition from MAT137 and say that a sequence in $\R^n$ is a function $\N\to \R^n$, or more generally, a function whose domain has the form $\{ j\in \Z : j\ge j_0\}$ and whose range is a subset of $\R^n$.)
We say that a sequence $\{ \bfa_j \}_j$ in $\R^n$ converges to the limit ${\bf L}\in \R^n$ if \begin{equation}\label{conv.seq} \forall \ep>0,\exists J>0 \ \ \mbox{ such that } \ \ j \ge J \ \ \Rightarrow \ \ |\bfa_j - {\bf L}|< \ep. \end{equation}
This is exactly like the definition of the limit of a sequence of real numbers except that (as usual) we replace the absolute value in the old definition by the Euclidean norm in the new one. Note also that we can reformulate definition \eqref{conv.seq} in terms of the sequence of real numbers $\{ |\bfa_j - {\bf L}|\}_j$, as follows: $$ \lim_{j\to \infty}\bfa_j ={\bf L} \qquad\iff \qquad \lim_{j\to\infty} |\bfa_j - {\bf L}| = 0. $$
This being the case, you (as usual) already understand many aspects of limits of sequences in $\R^n$, including limit laws for sequences.
If we have a sequence $\{ \bfa_j\}_j$ of elements of $\R^n$, then as usual we will write the components of a typical element $\bfa_j$ of the sequence as $(a_{j1}, \ldots, a_{jn})$. If we consider some $k\in \{1,\ldots, n\}$, then the sequence of $k$th components, that is, $\{ a_{jk} \}_{j=1}^\infty$ is a sequence of real numbers.
Theorem 2: Let $\{ \bfa_j\}_j$ be a sequence in $\R^n$, and let ${\bf L} = (L_1,\ldots, L_n)\in \R^n$. Then $$ \lim_{j\to \infty}\bfa_j = {\bf L} \quad \iff \quad \lim_{j\to \infty}a_{jk} = L_k \mbox{ for all }k=1,\ldots, n. $$
Proof. This is almost exactly like the proof of the corresponding theorem about limts of functions. Details omitted.
One potentially confusing aspect of this theorem is the notation. Considering a concrete example, see below, may help make it digestible.
Example
Cosider the sequence in $\R^3$,
$$
\bfa_j = ( \frac 1j , \frac{j^3-1}{j^3+j^2}, \frac{\sin(j^2)}{2j}), \qquad j = 1,2,3, \ldots
$$
The sequence of first components is $\{a_{j1}\}_{j=1}^\infty$, where
$
a_{j1} = \frac 1j .
$
The sequence of second components is $\{a_{j2}\}_{j=1}^\infty$, where $ a_{j2} = \frac {j^3-1}{j^3+j^2} = 1 - \frac{j^2+1}{j^3+j^2} . $
The sequence of third components is $\{a_{j3}\}_{j=1}^\infty$, where $ a_{j3} =\frac{\sin(j^2)}{2j}. $
We can check using first-year calculus that these sequences converge to $0$, $1$, and $0$, respectively.
So the theorem implies that $\bfa_j \to {\bf L} = (0,1,0)$ as $j\to \infty$.
Recall the following facts from MAT137.
A basic property of the real numbers is the Completeness Axiom, also known as the Least Upper Bound Property, which states that
Also, recall the definition:
a number $a$ is an upper bound for a set $S\subset \R$ if $a\ge x$ for every $x\in S$.
A number $a$ is the least upper bound for $S$ if $a$ is an upper bound for $S$, and $$ \mbox{ for every upper bound $b$ for $S$}, \qquad a\le b. $$
The least upper bound of a set $S$ is often called the supremum of $S$, written $\sup S$.
The Completeness axiom also implies that every nonempty bounded set $S$ has a greatest lower bound, or infimum, denoted $\inf S$.
Using the Completeness Axiom, one can prove the following:
Theorem 3: Monotone Sequence Theorem. Every bounded nondecreasing sequence of real numbers converges to a limit.
(Recall that a sequence $\{ x_j\}_j$ is nondecreasing if $x_k \ge x_j$ whenever $k\ge j$.)
A presentation of the proof with a lot of discussion of ideas and strategies can be found in a MAT137 Youtube Video A concise proof is below.
Proof
Let $\{ x_j\}_j$ be a bounded nondecreasing sequence.
Let $\ell := \sup \{ x_j\}$. The idea of the proof is to show that $x_j \rightarrow \ell$ as $j\to \infty$.
To do this, fix $\ep>0$. Since $\ell - \ep <\ell$, and since $\ell$ is the least upper bound for the sequence, clearly $\ell - \ep$ cannot be an upper bound. Thus there exists some $J$ such that $x_J>\ell-\ep$. Then, since the sequence is nondecreasing, it follows that $x_j>\ell-\ep$ for all $j\ge J$. Also, since $\ell$ is an upper bound, it is clear that $x_j \le \ell$ for all $j$.
We have thus shown that $$ \ell-\ep < x_j \le \ell \qquad\mbox{ for all }j\ge J. $$ Since $\ep>0$ was arbitrary, this shows that $x_j\to \ell$ as $j\to \infty$.
Assume that $\{ \bfa_j\}_j$ is a sequence in $\R^n$, for $n\ge 1$.
A subsequence of $\{ \bfa_j\}_j$ is a new sequence, obtained by deleting some terms from the original sequence.
For example, consider the sequence of real numbers defined by $$ a_j = (1- \frac 1j) \sin( \frac{ j \pi}4), \qquad j\ge 1. $$ The first few terms in this sequence are $$ 0, \quad \frac 12 , \quad \frac 23\frac{\sqrt 2}2, \quad 0,\quad -\frac 45\frac{\sqrt 2}2, \quad -\frac 56,\quad -\frac 67\frac{\sqrt 2}2, \quad 0,\quad \frac 89\frac{\sqrt 2}2, \quad \frac 9{10},\quad \ldots $$ It is pretty easy to see that this sequence does not converge. But suppose we throw out every $a_j$, except those for which $j = 2, 10, 18, 26, \ldots$. That is, we keep only $j$ of the form $8k+2$, where $k$ is a nonnegative integer. The first few terms in the resulting sequence would be $$ \frac 12, \quad \frac 9{10},\quad \frac {17}{18}, \quad \frac {25}{26}, \quad \frac {33}{34}, \quad \frac {41}{42}, \quad \ldots $$ We can write the subsequence as $\{ a_{8j-6}\}$, since the $j$th term of the new sequence is the $(8j-6)$th term of the original sequence. We can also write is as $\{ a_{k_j}\}_{j\ge 1}$ where $k_j = 8j-6$.
This is an example of a subsequence. It illustrates how, starting with a sequence that does not converge, it may be possible to find a subsequence that does converge to a limit. (The subsequence converges to $1$. We can see by inspection that this is true, and also, by substituting $k_j = 8j-6$ into the definition of the sequence, we can check that $a_{k_j} = 1 - \frac 1{8j-6}$, which we know converges to $1$. Note also in this example we could have chosen a different subsequence, for example by setting $k_j = 4j$, and gotten a different limit.)
We now give the formal definition of subsequence, which you will need if you ever have to prove anything about subsequences.
Definition. A subsequence of a sequence $\{ \bfa_j \}_{j\ge j_0}$ in $\R^n$, is a new sequence, denoted $\{ \bfa_{k_j}\}_j$, where $\{ k_j \}$ is an increasing sequence of integers such that $k_j \ge j_0$ for every $j$. Thus, the $j$th term $\bfa_{k_j}$ of the subsequence is the $k_j$th term of the original sequence. (Also, increasing means that $k_{j+1} > k_j$ for every $j$).
Fact. If $\{ \bfa_j \}_j$ is a sequence in $\R^n$ that converges to a limit ${\bf L}\in \R^n$, then any subsequence of $\{ \bfa_j \}_j$ converges to the same limit.
This says that if we have a sequence that converges to a limit, and we delete some terms, then the terms that are left will converge to the same limit. The proof is (will be) one of the practice problems. It's not very hard -- some of you may be able to do it in your head.
Next we prove a version of Theorem 1 (the Bounded Sequence Theorem) for sequences of real numbers. We will use this to later prove the general case of the theorem.
Theorem. Every bounded sequence of real numbers has a convergent subsequence.
There are several possible proofs. One of them can be found in the textbook (Section 1.4). A different one can be found below. The idea can be understood by drawing the right picture, but the details are a little bit complicated for this class, and you should only read it if you like this kind of thing. (In fact, our hard-working team of typists almost regrets taking the trouble to include it here, since the proof in the textbook may be easier.)
Proof.
Let $\{ a_j\}_j$ be a bounded sequence of real numbers.
For each $j$, let $b_j := \inf \{ a_k : k > j\}$.
Claim 1. $\{ b_j\}_j$ is a bounded, nondecreasing sequence. To check this, let's introduce the notation $S_j := \{ a_k : k > j \}$. Thus $b_j = \inf S_j$. Note that if $j_2 > j_1$ then $S_{j_2}\subset S_{j_1}$. So any lower bound for $S_{j_1}$ is also a lower bound for $S_{j_2}$. In particular, $b_{j_1} = \inf S_{j_1}$ is a lower bound for $S_{j_1}$ and hence $S_{j_2}$. It is thus less than or equal to $b_{j_2}=\inf S_{j_2} = $ the greatest lower bound for $S_{j_2}$. This shows that $b_{j_2} \ge b_{j_1}$ whenever $j_2> j_1$, and hence proves that $\{ b_j\}_j$ is nondecreasing. It is bounded because by assumption there exists some $M\in \R$ such that $|a_j|\le M$ for every $j$, so clearly $|b_j| \le M$ for every $j$. This proves the claim.
(Remark: it is generally true, by exactly the same argument given above, that if $A \subset B \subset \R$, then $\inf A \ge \inf B$ and $\sup A \le \sup B$.)
In view of Claim 1 and the Monotone Sequences Theorem, $\{ b_j\}_j$ is a convergent sequence. Let $\ell$ denote its limit.
Claim 2. $\{ a_j \}_j$ has a subsequence that converges to $\ell$.
When we prove this, we will have completed the proof of the theorem.
To prove this, we will describe how to choose the subsequence.
Let $k_1 = 1$.
Assume that we have chosen $k_1, k_2, \ldots, k_{j}$. We will describe how to choose $k_{j+1}$. The idea is to choose it so that $a_{k_{j+1}}$ is (nearly) the smallest term, among all those in the sequence after $a_{k_{j}}$.
In practice, to do this we note that the definition of $b_{k_j} = \inf \{ a_k : k > k_j \}$ and of infimum imply that we can choose $k_{j+1} > k_j$ such that $$ b_{k_j} \le a_{k_{j+1}} < b_{k_j} + \frac 1j \le \ell+ \frac 1j. $$ This describes how to choose $k_{j+1}$. By repeating this procedure indefinitely, we generate the subsequence $\{ a_{k_j} \}_j$.
Finally, we know that $\{ b_{k_j}\}_j$ converges to the same limit $\ell$ as $\{ b_j\}_j$ (by the Fact mentioned above), and clearly $ \ell+ \frac 1j$ also converges to $\ell$, so $\{ a_{k_j} \}_j$ converges to the same limit, by the Squeeze Theorem for sequences (familiar from MAT137).
Finally, we consider the Bounded Sequence Theorem, which is our main goal for this discussion. We recall the statement.
Theorem 1: Bounded Sequence Theorem. Every bounded sequence in $\R^n$ has a convergent subsequence.
Before giving the proof, we illustrate the idea by considering an example. First some notation. For a real number $x$, let $\lfloor x\rfloor$ denote the largest integer less than or equal to $x$, and let $\mbox{frac}(x)$ denote the fractional part
of $x$, that is,
$$
\mbox{frac}(x) = x - \lfloor x \rfloor.
$$
Let $\{ \bfa_j\}_j$ be the sequence in $\R^2$ defined by
$$
\bfa_j = ( \mbox{frac} (j/3), \mbox{frac}(j/7)).
$$
Thus the first few terms in the sequence (starting with $\bfa_1$) are
\begin{equation}\label{ss1}
(\frac 13, \frac 17), \
(\frac 23, \frac 27), \
(0, \frac 37), \
(\frac 13, \frac 47), \
(\frac 23, \frac 57), \
(0, \frac 67), \
(\frac 13, 0), \
(\frac 23, \frac 17), \
(0, \frac 27), \ \ldots
\end{equation}
We can easily find a subsequence such that the first components
converge. For example, we can consider the subsequence $\{ \bfa_{k_j} \}$ for $k_j = 3j$. This amounts to discarding all $\bfa_j$ in the original sequence where $j$ is not divisible by $3$. The resulting
subsequence is
\begin{equation}\label{ss11}
(0, \frac 37), \
(0, \frac 67), \
(0, \frac 27), \
(0, \frac 57), \
(0, \frac 17), \
(0, \frac 47), \
(0, 0), \ \ldots
\end{equation}For this subsequence the first component definitely converges (since it is always equal to zero) but the second component does not.
Similarly, we could consider the subsequence $\{ \bfa_{k_j} \}$ for $k_j = 7j$. For this subsequence, the second component converges (since it is always equal to zero) but the first component does not.
However, there is an easy way out of this difficulty.
We first choose a subsequence such that the first component converges, as in \eqref{ss1} above, for example getting the subsequence in \eqref{ss11}.
Then, we choose a subsequence of that subsequence to arrange that the second component converges, without changing the fact that the first component converges. In the example, at this stage we could choose the subsequence of \eqref{ss11} consisting of every $7$th term. The resulting sequence is convergent (in fact every term equals $(0,0)$.) Also, it is still a subsequence of the original sequence -- in fact it is the subsequence $\{ \bfa_{k_j} \}$ for $k_j = 21j$.
The proof of the theorem follows this same strategy of choosing subsequences of subsequences. See below for details.
Proof of Theorem 1.
Assume that $\{ \bfa_j\}_j$ is a bounded sequence in $\R^n$. This means that there exists some $R>0$ such that $$ |\bfa_j| < R \qquad\mbox{ for every }j. $$ It follows that for every $\ell\in \{ 1,\ldots, n\}$, and every $j$, $$ |a_{j\ell}| = |\bfa_j \cdot {\bf e}_\ell| \le |\bfa_j|\, |{\bf e}_\ell| < R. $$ Here $a_{j\ell}$ denotes the $\ell$th component of $\bfa_j$, as usual.
Thus for every $\ell$, the sequence $\{ a_{j\ell}\}_j$ of $\ell$th components is a bounded sequence. We now proceed as follows:
Using the one-dimensional case of the Bounded Sequence Theorem, we can choose a subsequence of $\{ \bfa_j\}_j$ such that the first components of the subsequence converge to a limit $L_1$. The other components (clearly) remain bounded.
Again using the one-dimensional case of the Bounded Sequence Theorem, we can choose a subsequence of the first subsequence, such that the second components converge to a limit $L_2$. Clearly, this is still a subsequence of $\{ \bfa_j\}_j$, the first components still converge to $L_1$, and the other components remain bounded.
repeat this procedure $n-2$ more times. At the end we will arrive at a subsequence of $\{ \bfa_j\}_j$ such that the $\ell$th components converge to a limit $L_\ell$ for $\ell = 1,\ldots, n$. According to Theorem 2 above, this implies that the subsequence is convergent, so we have finished the proof. $\quad \Box$
The material in this section does not really fall into the category of basic skills
, in the sense that it does not involve any methods that can be applied in a staightforward way to solve problems. However, we could ask a question testing whether you know the main theorems, such as
Justify your answercould mean: give the name of a relevant theorem, and check that its hypotheses are satisfied.)
We could also ask a question testing whether you know the definition of limit of a sequence.
Assume that $\{ \bfa_j \}_j$ is a convergent sequence in $\R^n$. Prove that any subsequence $\{ \bfa_{k_j} \}_j$ converges to the same limit.
Assume that ${\bfa_j}$ and ${\bfb_j}$ are sequences in $\R^n$ such that $\bfa_j\to \bfa$ and $\bfb_j\to \bfb$.
Assume that $\{ \bfa_j \}_j$ is a sequence in $\R^n$
and that $\lim_{j\to \infty}\bfa_j = \bfa$.
Also assume that $\bff:\R^n\to \R^k$ is a function that is continuous at $\bfa$.
Prove that $\lim_{j\to \infty} \bff(\bfa_j) = \bff(\bfa)$.
Find a sequence in $\R^n$ with no convergent subsequences. Prove that your answer is correct.
Review of calculus: Assume that $\{ x_j \}_j$ is a sequence
of real numbers, and let $d_j := |x_{j+1} - x_j|$, i.e. the distance between successive terms. Is it true that if $d_j\to 0$
as $j\to \infty$, then $x_j$ must converge to a limit as $j\to \infty$?
Hint: Nnnn.....
Assume that $\{ \bfa_j \}_j$ is a sequence
in $\R^n$, and let $d_j := |\bfa_{j+1} - \bfa_j|$, i.e. the distance between successive terms. Is it true that if $d_j\to 0$
as $j\to \infty$, then $\bfa_j$ must converge to a limit as $j\to \infty$?
Hint: This looks a lot like the previous problem.