$\newcommand{\R}{\mathbb R }$ $\newcommand{\N}{\mathbb N }$ $\newcommand{\Z}{\mathbb Z }$ $\newcommand{\bfa}{\mathbf a}$ $\newcommand{\bfb}{\mathbf b}$ $\newcommand{\bfc}{\mathbf c}$ $\newcommand{\bff}{\mathbf f}$ $\newcommand{\bfg}{\mathbf g}$ $\newcommand{\bfG}{\mathbf G}$ $\newcommand{\bfh}{\mathbf h}$ $\newcommand{\bfu}{\mathbf u}$ $\newcommand{\bfv}{\mathbf v}$ $\newcommand{\bfx}{\mathbf x}$ $\newcommand{\bfp}{\mathbf p}$ $\newcommand{\bfy}{\mathbf y}$ $\newcommand{\ep}{\varepsilon}$

Optimization

1. Lagrange multipliers for constrained optimization
2. Other optimization problems
3. Problems

Lagrange multipliers for constrained optimization

Consider the problem \begin{equation} \left\{\begin{array}{r} \mbox{minimize/maximize }\ \ \ f(\bfx)\qquad \\ \mbox{ subject to the constraint: }\ \ g(\bfx)=0. \end{array}\right. \label{con1}\end{equation}

A point $\bfx$ is said to be a local maximum of $f$ subject to the constraint $g= 0$ if $$ \exists r>0\mbox{ such that }f(\bfx)\ge f(\bfx')\mbox{ for all }\bfx' \mbox{ such that } |\bfx'-\bfx|<r \mbox{ and }g(\bfx')=0. $$ This is the same as saying that $$ \exists \mbox{ an open set }U\mbox{ containing }\bfx,\mbox{ and such that }f(\bfx)\ge f(\bfx')\mbox{ for all }\bfx'\in U. $$ The definition of a local minimum of a constrained variational problem is essentially the same, except (of course) $\ge$ is changed to $\le$.

Theorem 1. Assume that $f$ and $g$ are functions $S\to \R$ of class $C^1$, where $S$ is an open subset of $\R^n$.
If $\bfx$ is a local minimum point or local maximum point of $f$ subject to the constraint $g=0$, and if $\nabla g(\bfx) \ne {\bf 0}$, then there exists $\lambda\in \R$ such that the following system of equations is satisfied by $\bfx$ and $\lambda$: \begin{equation} \left\{ \begin{array} {r} \nabla f(\bfx) +\lambda \nabla g(\bfx) = \bf0,\\ g(\bfx) = 0. \end{array} \right. \label{lm1}\end{equation}

In particular, if $\bfx$ is any solution of \eqref{con1}, then the equations \eqref{lm1} hold (unless $\nabla g(\bfx)={\bf 0}$, in which case they might not.)

We will probably prove this theorem later in the class.

For many constrained optimization problems, it is the case that \begin{equation}\label{nondeg} \nabla g \ne {\bf 0} \mbox{ on the set } \{ \bfx\in \R^n : g(\bfx) = 0\}. \end{equation} If this holds, then every local minimum or maximum point satisfies \eqref{lm1}.

Note that \eqref{lm1} is a system of $n+1$ equations $$ \left\{ \begin{array} {r} \partial_1 f(\bfx) +\lambda \partial_1 g(\bfx) = 0,\\ \partial_2 f(\bfx) +\lambda \partial_2 g(\bfx) = 0,\\ \vdots\qquad\\ \partial_n f(\bfx) +\lambda \partial_n g(\bfx) = 0,\\ g(\bfx) = 0. \end{array} \right. $$ with $n+1$ unknowns, the $n$ components $x_1, \ldots, x_n$ of $\bfx$ and the number $\lambda$, which is called the Lagrange multiplier.

Here are some examples:

Example 1.Consider the problem $$ \left. \begin{array}{rl} \mbox{minimize/maximize}\ \ &f(x,y) = y\\ \mbox{ subject to the constraint:}\ \ &g(x,y) = 0 \end{array}\right. $$ where the set $S := \{ (x,y) : g(x,y) = 0\}$ is the curve shown in the picture below. We assume that $g$ satisfies condition \eqref{nondeg}.

We can informally think of this problem as asking us to find the northernmost point on the curve (maximum of $f(x,y) = y$) and the southernmost point on the curve (minimum of $f(x,y)= y$.) It is easy to see where these are.

What do the Lagrange multipler equations say? Clearly, $\nabla f = \binom 0 1$ everywhere. These vectors are pictured in green below.

So the Lagrange multiplier equations say \begin{align} \binom 0 1 + \lambda \nabla g(\bfx) &=0 \nonumber \\ g(\bfx) &= 0.\nonumber \end{align} The first equation cannot be satisfied if $\lambda = 0$. Therefore we can divide by $\lambda$, and the first equation says that $\nabla g(\bfx) = -\lambda^{-1}\binom 0 1 = $ a multiple of the vector $\binom 01$.

To understand what this means, recall the important fact that $$ \boxed {\nabla g(\bfx)\mbox{ is orthogonal to the level set of }g\mbox{ passing through }\bfx.} $$ The directions of $\nabla g$ at points on the curve $S$ are thus as shown in red in the picture below:

(In the picture we are assuming for concreteness that $g$ increases as one moves from the interior to the exterior of the curve -- otherwise the arrows would point inward.)

So the Lagrange multiplier equations are satisfied at points where the red vector (the normal to the curve) is parallel to the green vector. This is the same as saying that the level set of $g$ (i.e. the curve where the constraint is satisfied) is orthogonal to $\binom 01$. These points are shown yellow in the picture below. The northnmost and southernmost points are the maximum and minimum respectively.

From this example, we can understand more generally the meaning of the Lagrange multiplier equations, and we can also understand why the theorem makes sense. The equations imply for example that at the northernmost point on the curve, the curve is oriented exactly in an east-west direction. This is intuitively clear, since at any point where the curve is not oriented exactly east-west, we could get a little farther to the north by moving in one or the other direction alog the curve.

Generally, the same considerations apply to any $C^1$ functions $f$ and $g$. That is,
$$ \boxed{\begin{array}{l} \mbox{the Lagrange muliplier equations}\ \ \Longrightarrow \\ \hspace{5em} \nabla f(\bfx) \mbox{ is orthogonal at }\bfx\mbox{ to the set of points satisfying the constraint.} \end{array}} $$ You may be able to convince yourself that if $\nabla f$ is not orthogonal to the level set of $g$, then it should be possible to move a little bit, in a way that makes $f$ larger or smaller but continues to satisfy the constraint. The underlying idea is exactly the same as in Example 1.

Example 2. Solve the problem $$ \left. \begin{array}{rl} \mbox{minimize/maximize}\ \ &f(x,y) := x(1-y^2)\\ \mbox{ subject to the constraint:}\ \ &g(x,y) := x^2+ y^2 - 1 = 0. \end{array}\right. $$

Solution. First, note that $f$ is continuous, and the set defined by the constraint is compact. Thus, the Extreme Value Theorem guarantees that the min and max are achieved. Also, $\nabla g(x,y) = (2x, 2y)$. Thus $\nabla g = {\bf 0}$ only at the origin, which does not satisfy the constraint, so condition \eqref{nondeg} holds. Thus the Lagrange multiplier equations are guaranteed to be satisfied at the points where the min and max occur.

The Lagrange multiplier equations are \begin{align} (1-y^2) + \lambda2x &= 0 \nonumber \\ -2xy+ \lambda 2y &=0 \nonumber \\ x^2+y^2&=1.\nonumber \end{align} The middle equation states that $2 y (\lambda-x) = 0$. This can only hold if $y=0,$ or $x=\lambda$. We consider both cases:

Case 1. $x = \lambda$. Then the first equation implies that $1-y^2 +2x^2=0 $. Then the constraint imples that $x=0$. $$ \mbox{solutions of the lagrange multiplier equations: } \lambda = 0, \bfx = (0,\pm 1). $$

Case 2. $y=0$. Then the constraint implies that $x=\pm 1$. We can then solve the first equation for $\lambda$ (if we care). This leads to $$ \mbox{solutions of the lagrange multiplier equations: } \bfx = (\pm 1, 0), \ \lambda = \mp \frac 12. $$

Overall, the candidate solutions of the extreme value problem are $(\pm1, 0)$ and $(0,\pm 1)$, and this set of points must include the points where the min and max are attained. By evaluating $f$ at these points, we find that \begin{align} \mbox{the minimum is }-1, \qquad &\mbox{ and the only global minimum point is }(-1,0)\nonumber \\ \mbox{the maximum is }1, \qquad &\mbox{ and the only global maximum point is }(1,0).\nonumber \end{align}

Example 3. Solve the problem $$ \left. \begin{array}{rl} \mbox{minimize/maximize}\ \ &x(1-y^2)\\ \mbox{ subject to the constraint:}\ \ &(x^2+ y^2 - 1)^2 = 0. \end{array}\right. $$

Note that this is exactly the same problem as in Example 2, since we are considering the same function $f$, and in both cases the constraint says that we are minimizing/maximizing over the unit sphere $$ \{ (x,y)\in \R^2 : x^2+y^2=1 \}. $$ The only difference is that in this version of the problem, I seem to have chosen to write the constraint in an unncessarily complicated way. Let's see what happens when we try to solve it: The Lagrange multiplier equations are \begin{align} (1-y^2) + \lambda 4x(x^2+y^2-1) &= 0\nonumber \\ -2xy+ \lambda 4y(x^2+y^2-1) &=0\nonumber \\ x^2+y^2&=1.\nonumber \end{align} Now, using the last equation, we can simplify the first two equations, rewriitng them as \begin{align} (1-y^2) &= 0\nonumber \\ -2xy&=0.\nonumber \end{align} It is easy to see that the only solutions are $\bfx = (0, \pm 1)$, and that $\lambda$ can be any real number. But $f(\bfx) = 0$ at $\bfx = (0,\pm 1)$, and $0$ is neither the maximum nor the minimum value of $f$ on the unit circle.

Thus, in this example, the Lagrange multiplier method does not work. The problem is that we have written the constraint in a silly way. Indeed, for the function $g(\bfx) = (x^2+y^2-1)^2$, it is easy to see that $\nabla g(\bfx) = 0$ at every point where $g(\bfx) = 0$, so condition \eqref{nondeg} is violated. So the failure of the method does not contradict the theorem.

Here is another example. This may be harder than anything we would ask you to do by hand.

Example 4. Consider the problem $$ \left\{\begin{array}{r} \mbox{minimize }\ \ \ \frac{x+y}{1+x^2+y^2} \qquad \\ \mbox{ subject to the constraint: }\ \ x^2 + y^2 - R^2 = 0. \end{array}\right. $$ (Of course this problem depends on $R$; we will solve it for every possible choice of $R$.)

Let's call the function we are minimizing $f$. We have already seen this function in Example 2 of Section 2.7, where we computed its derivatives and found its critical points. There we found that $$ \partial_x f(x,y) = \frac{1 - x^2 - 2xy +y^2 }{(1+x^2+y^2)^2}, \qquad \partial_y f(x,y) = \frac{1 + x^2 - 2xy -y^2 }{(1+x^2+y^2)^2}, $$ Using this, the Lagrange multiplier equations are \begin{align} \frac{1 - x^2 - 2xy +y^2 }{(1+x^2+y^2)^2}+ 2\lambda x &=0 \nonumber \\ \frac{1 + x^2 - 2xy -y^2 }{(1+x^2+y^2)^2}+2\lambda y &= 0 \nonumber \\ x^2+y^2 &= R^2 \nonumber \end{align} Using the third equation, we can rewrite the first two equations as \begin{align} 1 - x^2 - 2xy +y^2 + 2\lambda(1+R^2)^2 x &=0 \nonumber \\ 1 + x^2 - 2xy -y^2 +2\lambda (1+R^2)^2 y &= 0. \nonumber \end{align} Let's simplify things by writing $K = \lambda (1+R^2)^2$. Then by subtracting the second equation above from the first and rearranging, we get \begin{equation}\label{h1}0 = y^2 - x^2 - K(y-x) = (y+x-K)(y-x) \end{equation} On the other hand , by adding the two equations and rearranging, we get \begin{equation}\label{h2} K(x+y) - 2xy + 1 = 0. \end{equation} Now we can solve the system \eqref{h1}, \eqref{h2}: Clearly, \eqref{h1} is satisfied if either $x=y$ or $x+y=K$.

• If $x=y$, then the constraint implies that \begin{equation} (x,y) = (\frac R{\sqrt 2},\frac R{\sqrt 2})\qquad \mbox{ or }\qquad(x,y) = (-\frac R{\sqrt 2}, -\frac R{\sqrt 2}). \label{h3}\end{equation}

• If $x+y=K$, then we can substitute into \eqref{h2} and rewrite to find that $x^2+y^2+1 = 0$. This clearly has no solutions, so the only solutions are the ones found in \eqref{h3}.

By evaluating $f$ we conclude that $(x,y) = (-\frac R{\sqrt 2},-\frac R{\sqrt 2})$ minimizes $f$ with the constraint $g(\bfx)=0$, and $f(-\frac R{\sqrt 2},-\frac R{\sqrt 2}) = \frac {-\sqrt 2 R}{1+R^2}$.

Another technique

Sometimes constrained minimization problems can be reduced to unconstrained problems by parametrizing the set of points where the constraint is satisfied. We illustrate this with an example.

Example 4 revisited. Note that a point satisfies $x^2+y^2 = R^2$ if and only if it can be written in the form $(R\cos\theta, R\sin \theta)$ for some $\theta\in \R$.

So the problem $$ \mbox{ minimize }f\mbox{ (from Example 4), subject to the constraint }x^2+y^2 = R^2 $$ is equivalent to the problem: minimize $f(R\cos\theta, R\sin \theta)$ among all $\theta\in \R$. Also, from the formula for $f$, it is clear that $$ f(R\cos\theta, R\sin \theta) = \frac{R}{1+R^2} (\cos\theta+\sin \theta). $$ This is much easier than using the Lagrange multipler technique as above.

Remark 1. Despite this example,
the Lagrange multiplier technique is used more often. The reason is that applications often involve high-dimensional problems, and the set of points satisfying the constraint may be very difficult to parametrize. If you are programming a computer to solve the problem for you, Lagrange multipliers are typically more straightforward to program.

Even if you are solving a problem with pencil and paper, for problems in $3$ or more dimensions, it can be awkward to parametrize the constraint set, and therefore easier to use Lagrange multipliers.

Also, Lagrange multipliers have theoretical advantages in some situations, as illustrated in homework 1.4.

problems with multiple constraints

One can also use the Lagrange mutiplier method to address problems with more than one constraint. We will write it down for problems with $2$ constraints, which have the form \begin{equation}\label{mt1c} \left\{\begin{array}{r} \mbox{minimize/maximize} \ \ f(\bfx),\qquad \\ \mbox{ subject to the constraints: }\ \ g_1(\bfx)=0\ \\ \mbox{ and }\ \ g_2(\bfx)=0. \end{array}\right. \end{equation} (It will be clear how to generalize it to problems with $k$ constraints, if one wishes to do so.)

As with \eqref{nondeg} the case of single constraint, there is an annoying condition that limits the applicability of the method, see \eqref{nd2} below. In many problems one does not need to worry about it.

Theorem 2. Assume that $f, g_1$ and $g_2$ are functions $\R^n\to \R$ of class $C^1$. Assume also that \begin{equation}\label{nd2} \{\nabla g_1(\bfx), \nabla g_2(\bfx)\}\mbox{ are linearly independent at all $\bfx$ where $g_1(\bfx) = g_2(\bfx) = 0$.}% \ne 0 \mbox{ on the set } \{ \bfx\in \R^n : g(\bfx) = 0\}. \end{equation} Then if $\bfx$ is any solution of \eqref{mt1c}, there exists $\lambda_1,\lambda_2\in \R$ such that the following system of equations is satisfied by $\bfx, \lambda_1$ and $\lambda_2$: \begin{equation} \left\{ \begin{array} {r} \nabla f(\bfx) +\lambda_1 \nabla g_1(\bfx)+\lambda_2 \nabla g_2(\bfx) = \bf0,\\ g_1(\bfx) = 0\ \\ g_2(\bfx) = 0. \end{array} \right. \label{lm1a}\end{equation}

This too may be proved later in the course.

Note that the Lagrange multiplier equations \eqref{lm1a} are now a system of $n+2$ equations, with $n+2$ unknowns, $x_1,\ldots, x_n$ and $\lambda_1,\lambda_2$.

Example 5. $$ \left\{\begin{array}{r} \mbox{minimize } \ \ \ \quad \ xy+ xz+yz \\ \mbox{ subject to the constraints: }\ \ x^2+y^2+z^2 = 2\\ \mbox{ and }\qquad\qquad\ \ \ \ \ z = 1. \end{array}\right. $$ First note that the set defined by the constraints is compact, and that $\{ \nabla g_1(\bfx), \nabla g_2(\bfx)\}$ are linearly independent at except where $x=y=0$. So they are always linearly independent at points where the constraint is satisfied. Thus a minimum point is guaranteed to exist, and the Lagrange multiplier equations are guaranteed to hold at the minimum point.

The Lagrange multiplier equations are: \begin{align} \left(\begin{array}{c} y+z \\ x+z \\ x+y \end{array} \right) +\lambda_1 \left(\begin{array}{c} 2x\\2y\\2z\end{array} \right) +\lambda_2 \left(\begin{array}{c} 0\\0\\1\end{array} \right) &= \left(\begin{array}{c} 0\\0\\0\end{array} \right) \\ x^2+y^2+z^2 &=2\\ z&=1. \end{align} This is a system of 5 equations with 5 unknowns. Fortunately, the equation $z=1$ is trivial and allows us to eliminate $z$ from the other equations, which then become a system of 4 equations and 4 unknowns: \begin{align} y+1+2\lambda_1 x &=0,\nonumber \\ x+1+2\lambda_1 y &=0.\nonumber \\ x+y+2\lambda_1+\lambda_2 &=0.\nonumber \\ x^2+y^2&=1\nonumber \end{align} Note that $\lambda_2$ appears only in the third equation, so if the other three equations determine the other three unknowns $x,y,\lambda_1$, then the third equation can always be satisfied by choosing $\lambda_2 = -x-y-2\lambda_1$. So we can focus on the system \begin{align} y+1+2\lambda_1 x &=0,\nonumber \\ x+1+2\lambda_1 y &=0.\nonumber \\ x^2+y^2&=1\nonumber \end{align}

We multiply the second equation by $2\lambda_1$ and use the first equation to eliminate $x$, yielding $$ -1-y + 4\lambda_1^2 y = -2\lambda_1, $$ which we can rewrite as $$ 0 = (4\lambda_1^2-1)y + (2\lambda_1-1) = (2\lambda_1-1)[ (2\lambda_1+1)y +1]. $$

This says that $$ \mbox{EITHER }\ \ \lambda_1=\frac 12 \qquad\mbox{ OR}\ \ (2\lambda_1+1)y = -1\qquad $$ We consider both cases. If $\lambda_1= \frac 12$, then the equations become $$ x+y+1 = 0, \qquad x^2+y^2 = 1. $$ This has two solutions, $(x,y) = (-1,0)$ or $(x,y) = (0,-1)$.

If $\lambda_1\ne \frac 12$, then $y= -1/(2\lambda_1+1)$. Also, one could go through exactly the same argument as above, with the roles of $x$ and $y$ reversed, to find that $$ x=-1/(2\lambda_1+1) \qquad\mbox{ also}. $$ Thus $x=y$ in this case, and it follows from the constraint that the only solutions of the Lagrange multiplier equations are $(x,y) = \pm (\frac 1{\sqrt 2}, \frac 1 {\sqrt 2})$.

In summary, the solutions of the Largange multiplier equations are:

• $(x,y) = (-1,0)$, with $f(x,y) = -1$.
• $(x,y) = (0,-1)$, with $f(x,y) = -1$.
• $(x,y) = (\frac 1{\sqrt 2}, \frac 1 {\sqrt 2})$, with $f(x,y) = \frac 12 +\sqrt 2$
• $(x,y) = (-\frac 1{\sqrt 2}, -\frac 1 {\sqrt 2})$, with $f(x,y) = \frac 12 - \sqrt 2 \approx -.914$ (At this stage we no longer need to worry about the value of $\lambda_1$.)

So we can see that the minimum value is $-1$, and it occurs at $(-1,0)$ and $(0,-1)$.

other optimization problems

Minimizing or maximizing a function in an open set

Assume that $S$ is an open subset of $\R^n$, and that $f:S\to \R$ is continuous, and maybe $C^1$ or $C^2$. Consider the problem $$ \mbox{minimize }f \mbox{ in }S. $$ As we know:

1. there may be no solution, depending of course on $f$ and $S$.

2. However, sometimes we can guarantee there is a solution. (See Example 3 and some of the Basic Skills problems in Section 1.4).

In any case, keeping that in mind, we know that a maximum/minimum point in an open set is always a critical point, so we can find all candidates by finding all of the critical points in the open set. If we like, we can also try to classify the critical points (determine which are local max/local mins) to get more information.

Inequality constraints

Finally, consider the problem \begin{equation}\label{ineqc} \left\{\begin{array}{r} \mbox{minimize/maximize }\ \ \ f(\bfx)\qquad \\ \mbox{ subject to the constraint: }\ \ g(\bfx)\le 0. \end{array}\right. \end{equation} where we assume that $g$ is $C^2$, say, and that $\nabla g(\bfx) \ne {\bf 0}$ on the set $\{ \bfx\in \R^n : g(\bfx)=0\}$.

tl;dr: we can reduce this to problems we already know how to solve.

For problems such as \eqref{ineqc}, it is often the case that the Extreme Value Theorem guarantees that the problem has a solution (though this depends on properties of $f$ and $g$). How can we find them? Well, there are exactly 2 cases:

Case 1. The max or min occurs in the set $\{ \bfx \in \R^n : g(\bfx)<0\}$.

Then it is a critical point, which we know how to find and classify.

Case 2. The max or min occurs in the set $\{ \bfx \in \R^n : g(\bfx)= 0\}$.

Then we can find it by the Lagrange Multipler technique.

Since one of these two cases must hold if the min or max is attained, we can:

1. find all critical points (if any exist) of $f$ in $\{ \bfx \in \R^n : g(\bfx)<0\}$. Find the max or min of $f$ among these critical points.
2. Use the Lagrange multiplier technique to find the max or min of $f$ with the constraint $g(\bfx)= 0$.
3. choose the smallest / largest value of $f$ (and the point where that value is attained) from among all the candidates found in steps 1 and 2 above.

Remark 2. This situation is analogous to problems in first-year calculus, where you are asked to minimize a function in a closed interval $[a,b]$. There too, you have to consider two cases:
Case 1. The min occurs in the interior. To address this possibility, you can find all critical points in the interior $(a,b)$, and if you like you can also use a second derivative test to get more information. This is complete analogous to Case 1 above.
Case 2. The minimum occurs at the boundary, ie in the set $\{a,b\}$. As in the multi-variable case, this requires separate consideration. It is easier for functions of a single variable, because we only have to worry about the two points $a$ and $b$. For functions of several variables this is where we need Lagrange Multipliers (or some other technique).

Remark 3. There are more sophisticated ways of solving problems of this type, or (more generally) problems with more than one inequality cnstrint, but we will not discuss them in this class.

Example 6. Revisit Example 1 (minimize/maximize $f(x,y) = y$, with $g$ described by a picture, see above), but with the constraint changed to $g(x,y)\le 0$, where we assume that $g<0$ in the region enclosed by the curve, and $g>0$ outside the curve.

Here $\nabla f = (0,1)$ everywhere, so there are no critical points of $f$ in the set where $g<0$. Thus this problem has the same solutions as Example 1.

Example 7. Consider the problem $$ \left\{\begin{array}{r} \mbox{minimize }\ \ \ \frac{x+y}{1+x^2+y^2} \qquad \\ \mbox{ subject to the constraint: }\ \ x^2 + y^2 - R^2 \le 0. \end{array}\right. $$ (Of course this problem depends on $R$; we will solve it for every possible choice of $R$.)

We simply follow the procedure discussed above.

Step 1. Find all critical points $(x,y)$ such that $x^2+y^2 <R^2$.

In fact, we already found all critical points (on all of $\R^2$) for this function in Example 2 of Section 2.7. There we found that the only critical points are $$ (x,y) = \pm ( \sqrt{1/2}, \sqrt{1/2} ). $$ These satisfy $x^2+y^2 < R^2$ if and only if $R>1$

Step 2. Solve the Lagrange multiplier equations for the equality constraint. We have already done this in Example 4 above. There we found that the solutions are $(\frac R{\sqrt 2},\frac R{\sqrt 2})$ (the maximum point) and $(-\frac R{\sqrt 2},-\frac R{\sqrt 2})$ (the minimum point).

Step 3. Conclude. We see that there are two cases:

1. If $R\le 1$, then there are no interior critical points, and so the minimizer with the inequality constraint $g\le 0$ must be the same as the minimizer with the equality constraint, $g=0$. Thus

   • if $R\le 1$, then the minimum occurs at $(x,y) = (-\frac R{\sqrt 2},-\frac R{\sqrt 2})$.

2. If $R>1$, then there are both interior critical points and solutions of the Lagrange multiplier problem. By evaluating $f$ at the various points, we conclude that

   • if $R > 1$, then the minimum occurs at $(x,y) = (-\frac 1{\sqrt 2},-\frac 1{\sqrt 2})$.

Problems

more questions may be added later.

Basic skills

1. Solve problems of the form \begin{equation} \left\{\begin{array}{r} \mbox{minimize/maximize }\ \ \ f(\bfx)\qquad \\ \mbox{ subject to the constraint: }\ \ g(\bfx)=0. \end{array}\right. \label{con1a}\end{equation}
   You might be specifically asked to use the Lagrange multiplier technique to solve problems of the form \eqref{con1a}.
   Techniques such as Lagrange multipliers are particularly useful when the set defined by the constraint is compact.
   Here are some sample problems.

   • Minimize $f(x,y,z) = x^2+2y^2+4z^2$, subject to the constraint $$ x^2+y^2+z^2 = 1 $$

   • Given $\bfy\in \R^n$ and $\bfv\in \R^n,b\in \R$, minimize $f(\bfx) = |\bfx-\bfy|^2$ subject to the constraint $$ \bfv \cdot \bfx - b = 0 $$ What is the minimum value? (This says: find the closest point to $\bfy$ in the hyperplane $\{ \bfx\in \R^n : \bfv \cdot\bfx = b\}$. This can be solved by geometric reasoning, but it is instructive to solve with Lagrange multipliers.)

   • For points $\bfx = (x,y)$ and $\bfu = (u,v)$ in $\R^2$, minimize $f(\bfx, \bfu) = |\bfx - \bfu|$ among all pairs of points such that $\bfx$ belongs to the plane $\{ \bfx\in \R^2 : \bfx \cdot (1,2) = -10\}$ and $\bfu$ belongs to the parabola $\{ \bfu \in \R^2 : v = u^2\}$.
     

     Hints. We suggest that you only click after trying to get started.

     It is easier to minimize $|\bfx - \bfu|^2$ than $|\bfx - \bfu|$.
     Also, you may be able to simplify this problem by using the fact that you already know a formula for the distance from a point to a plane, if you have solved the previous problem.

   • Find the maximum volume of a rectangular box in $\R^3$, with sides parallel to the coordinate axes, whose vertices are all a distance $R$ from the origin.

   • Redo the same question for an $n$-dimensional box in $\R^n$, where the volume is of course the product of the lengths of the sides.

   • Heron's Formula for the area of a triangle with sides of length $x,y,z$ is $$ \mbox{ area } = \sqrt{ s(s-x)(s-y)(s-z)},\quad\mbox{ where }s = \frac 12(x+y+z). $$ Prove that a triangle that maximizes area for given perimeter is equilateral.

   • Find the largest and smallest values of $f(x,y,z) = x+2y+3z$ on the set of points such that $x^2+y^2 = 1$ and $y-z=2$.

   • Find the largest and smallest values of $x^3+y^3+z^3$ on the set of points where $x^2+y^2+z^2=1$.

   • Find the largest and smallest values of $x^3+y^3+z^3$ on the set of points where $x^4+y^4+z^4=1$.

2. Show that the function $f = ...$ has a absolute minimum/maximum on the noncompact set $S = ...$ (see Section 1.4) and find it.
   This involves combining ideas from Section 1.4 and techniques for finding and classifying critical points, as in Section 2.7.

   • Consider the problem $$ \left\{\begin{array}{r} \mbox{minimize/maximize }\ \ \ 2x^2+y^2+z^2\qquad \\ \mbox{ subject to the constraint: }\ \ xyz-16=0. \end{array}\right. $$ with the additional constraint that $x,y,z$ are all positive. Prove that a minimizer exists, and find it.

   • Some of the sample problems above involve optimization on noncompact sets. Determine which problems have this character, and prove that the optimization problem has a solution -- that the minimum or maximum (whatever the problem asks about) is attained. Then solve the problem, if you didn't already do it.

   • Given points $(x_1,y_1),\ldots, (x_k, y_k)\in \R^2$ with $x_i\ne x_j$ for $i\ne j$, consider the function $$ f(a,b) = \sum_{j=1}^k (y_j - ax_j - b)^2 $$
     This is a measure of how close the line $\ell(x) = ax+b$ comes to passing through the points $(x_j, y_j)$. Show that the the absolute minimum of $f$ is attained on $\R^2$, and prove that it is given by $$ a = \frac { k^{-1}\sum_{j=1}^k x_jy_j - \bar x \bar y} { k^{-1}\sum_{j=1}^k x_j^2 - \bar x^2}, \quad b = \bar y - a \bar x $$ where $$ \bar x = \frac 1 k \sum_{j=1}^k x_j, \qquad \bar y = \frac 1 k \sum_{j=1}^k y_j. $$

3. In principle we could ask you to solve an optimization problem of the form $$ \left\{\begin{array}{r} \mbox{minimize/maximize }\ \ \ f(\bfx)\qquad \\ \mbox{ subject to the constraint: }\ \ g(\bfx)\le 0. \end{array}\right. $$ However, these problems typically take a while to solve, so in this class we are not very likely to ask such a question in a situation where you are faced with rigid time constraints. But we could!

   • for practice, change $\ldots = 0$ to $\ldots \le 0$ on some of the above practice problems involving equality constraints.

More conceptual questions

1. Referring to the picture below, assume that $g(x,y)$ is a function such that $g=0$ on the curve, and $\nabla g \ne {\bf 0}$ on the curve.

   Consider the problem $$ \left\{\begin{array}{r} \mbox{minimize/maximize }\ \ \ f(\bfx) = |\bfx - {\bf P}|^2 \\ \mbox{ subject to the constraint: }\ \ g(\bfx)=0.\qquad\ \ \end{array}\right. $$
   • Find the points where the maximum and minimum are attained.
   • Find all solutions of the Lagrange multiplier equations.
   • Consider the same question, wbut instead minimize/maximize
     • $f(x,y) = x+y$
     • $f(x,y) = xy$, if $\bf P$ is the origin.
   • Suppose $f(x,y) = |\bfx - {\bf P}|^2 + x+y$. Is the question Minimize $f$, subject to the constraint $g=0$ any more or less reasonable than the other questions asked here? Why or why not?

   $\Leftarrow$  $\Uparrow$  $\Rightarrow$