This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 Canada License.
\(\renewcommand{\R}{\mathbb R }\)
\(\Leftarrow\) \(\Uparrow\) \(\Rightarrow\)
Suppose that \(S\) is an open subset of \(\R^n\), and let \(f\) be a function \(f:S\to \R\). Our first goal is to define what it means for \(f\) to be differentiable. Since our definition should generalize what we know from functions of a single variable, let’s first recall how that goes.
Note that our domain is an open set. To generalize this to \(\mathbf f: S\to \R\) with \(S \subseteq \R^n\), we will need to replace \(h\) with a vector, and use the vector length in the denominator. But this won’t be enough. Consider \(f: \R^2\to \R\) defined by \(f(x,y)=x\). This function should be differentiable. Writing \(\mathbf h=(h,k)\), we have \[f(\mathbf x+\mathbf h)-f(\mathbf x)=h,\] and \(|\mathbf h|=\sqrt{h^2+k^2}\), but \(\displaystyle \lim_{\mathbf h\to \mathbf 0} \frac h{\sqrt{h^2+k^2}}\) does not exist, as we have seen by looking at the values along \(k=mh\). Just like completeness, we will need to look at alternative definitions of differentiable functions to find one that generalizes to \(\R^n\) in a way that makes sense.
If the derivative at \(x\) is \(m\), then \[\lim_{h\to 0} \frac{ f(x+h) - f(x)-mh}{h} = 0,\] and conversely, if this limit is \(0\) for a real number \(m\), then the derivative at \(x\) is \(m\). Thus, an equivalent definition isThe equivalent definition \(\eqref{der2}\) can be understood as follows: temporarily fix \(x\), treat \(h\) as a variable, and view \(f(x+h) - f(x)\) as a function of \(h\). Then \(f\) is differentiable at \(x\) if \(f(x+h)-f(x)\) is approximately \(mh\), with an error that is smaller than linear as \(h\to 0\). When this holds, \(f'(x)=m\).
It’s rare that \(f(x+h)-f(x)=mh\) exactly. Instead, it will be \(mh+E(h)\) for some error term \(E(h)\), with \(\lim_{h\to 0} \frac{E(h)}h=0\). Thus, a third way of defining the derivative is:To see that these definitions are all the same, note that \[\begin{align*} &\lim_{h\to 0} \frac{f(x+h) - f(x)}{h} = m \\ &\qquad \iff \qquad \lim_{h\to 0} \frac{ f(x+h) - f(x)-mh}{h} = 0 \\ &\qquad \iff \qquad f(x+h) - f(x)-mh =E(h) \quad \text{ satisfies } \lim_{h\to 0}\frac{E(h)}{h}=0 \\ &\qquad \iff \qquad \text{$\eqref{der3}$ holds} . \end{align*}\]
Suppose that \(S\) is an open subset of \(\R^n\) and consider a function \(f:S\to \R\).
We will define differentiability in a way that generalizes definition \(\eqref{der2}\). The idea is that \(f\) is differentiable at a point \(\mathbf x\in S\) if \(f\) can be approximated near \(\mathbf x\) by a linear map \(\R^n\to \R\), with errors that are smaller than linear near \(x\).
To make this precise, we will suppose that \(\mathbf x\in S\) is fixed, and we will consider the function \(\mathbf h \mapsto f(\mathbf x+\mathbf h)-f(\mathbf x)\) of a variable \(\mathbf h\in \R^n\).
Remark. The notation \(a \mapsto b\) defines the function \(f(a)=b\).
We want \(f\) to be differentiable at \(\mathbf x \iff f(\mathbf x+\mathbf h) - f(\mathbf x)\) is approximately a linear function of \(\mathbf h\), for \(\mathbf h\) near \(\mathbf 0\).
Recall from linear algebra:Another way of saying this is: a function \(\ell:\R^n\to \R^m\) is linear if \[ \ell(a\mathbf x+b\mathbf y) = a \ell (\mathbf x) + b \ell(\mathbf y)\quad\text{ for all }a,b\in \R\text{ and } \mathbf x,\mathbf y \in \R^n. \] If a function has the form \(f(\mathbf x) = M\mathbf x + \mathbf b\), we will say that it is affine. We may also sometimes call it a first-order polynomial or a polynomial of degree 1.
In general, we know from linear algebra that if \(\ell:\R^n\to\R\) is a linear function, then \(\ell\) can be written in the form \[ \ell(\mathbf h) = \mathbf m \cdot \mathbf h\qquad\text{ for some vector }\mathbf m\in \R^n. \]
So, combining all these, we are led to the following basic definition:
Note that \(\nabla f(\mathbf x)\) is uniquely determined by condition \(\eqref{der.Rn}\). Thus, the gradient \(\nabla f\) (when it exists) is characterized by the property that \[ \lim_{\mathbf h \to {\mathbf 0}}\frac{ f(\mathbf x+\mathbf h) - f(\mathbf x) - \nabla f(\mathbf x) \cdot \mathbf h} {|\mathbf h|} = 0. \]
We can also write a definition in the style of \(\eqref{der3}\):
These definitions can be understood as follows: temporarily fix \(\mathbf x\), view \(\mathbf h\) as a variable, and view \(f(\mathbf x+\mathbf h) - f(\mathbf x)\) as a function of \(\mathbf h\). Then \(f\) is differentiable at \(\mathbf x\) if the linear function \(\mathbf m\cdot \mathbf h\) approximates \(f(\mathbf x+\mathbf h)-f(\mathbf x)\), with errors that are smaller than linear as \(\mathbf h\to {\mathbf 0}\).
We will soon see that there is a simple computational way of finding out what the gradient of \(f\) must be, if it exists. First, let’s do one example the hard way. This will help us to appreciate the theorems that will be proved shortly.
Consider \(f:\R^2\to \R\) defined by \[ f(x,y) = (x-1)^3(y+2). \] Is \(f\) differentiable at the origin?
The next important property of differentiability generalizes a familiar fact about functions of a single variable.
Recall that we have defined \({\mathbf e}_j\) to be the unit vector in \(\R^n\) in the \(j\)th coordinate direction. Thus \[ {\mathbf e}_1 = (1, 0, \ldots ,0), \quad {\mathbf e}_2 = (0,1, \ldots ,0), \quad \ldots,\quad {\mathbf e}_n = (0, 0, \ldots ,1). \]
To see what it means, let’s consider a function \(f\) of \(2\) variables. In this case we usually write \(\frac{\partial f}{\partial x}\) and \(\frac{\partial f}{\partial y}\), instead of \(\frac{\partial f}{\partial x_j}\) for \(j=1\) or \(2\). The definition says that \[\begin{align*} \frac{\partial f}{\partial x} (x,y) &= \lim_{h\to 0} \frac {f(x+h, y) - f(x,y)}h \\ \frac{\partial f}{\partial y} (x,y) &= \lim_{h\to 0} \frac {f(x, y+h) - f(x,y)}h. \end{align*}\] So for example, \(\frac{\partial f}{\partial x}\) is computed by “freezing \(y\)” – that is, considering it to be a constant – and differentiating with respect to the variable \(x\).
In other words, if we want to compute \(\frac{\partial f}{\partial x}\) at a point \((x,y)\), we can temporarily define a function \(g(x) = f(x,y)\), that is, \(f\) with the \(y\) variable “frozen.” Then \[ \frac{\partial f}{\partial x} (x,y) = g'(x). \] Both sides of the above equality are limits, so it means that the limit on the left-hand side exists if and only if the limit on the right-hand side exists, and when they exist, they are equal. Similarly, \[ \frac{\partial f}{\partial y} (x,y) = g'(y) \qquad\text{ for }g(y) = f(x,y) \text{ with }x \text{ ``frozen''}. \]
Consider \(f:\R^2\to \R\) defined by \[ f(x,y) = (x-1)^3(y+2), \] the same function as in Example 1. What are the partial derivatives of \(f(x,y)\)?
For functions of 3 or more variables, the principles are exactly the same.
Note that partial derivatives at a point \(\mathbf x\) only tell us about the behaviour of \(f\) on lines through \(\mathbf x\) parallel to the coordinate axes. For example,
Let \(f_1:\R^3\to \R\) be the function defined by \[ f_1(x,y,z) = \begin{cases}0 &\text{ if }xyz=0 \\ 1 &\text{ otherwise}. \end{cases} \] Since \(xyz=0\) if and only if at least one component \(x,y,z\) equals zero, we can see that \(f=0\) on the union of the \(xy\)-plane, the \(yz\)-plane, and the \(xz\)-plane. It is straightforward to check that all partial derivatives exist at the origin, and in fact that \[ \frac{\partial f_1}{\partial x}=\frac{\partial f_1}{\partial y} =\frac{\partial f_1}{\partial z} = 0 \text{ at }(0,0,0). \] Similarly, let’s define \[ f_2(x,y,z) = \begin{cases}0 &\text{ if }xyz=0 \\ \cos(x-2 yz)e^{xz} &\text{ otherwise}. \end{cases} \] Although the behaviour of \(f_2\) is complicated if all components \(x,y,z\) are nonzero, the partial derivatives of \(f_2\) at \((0,0,0)\) do not “see” this behaviour, and again, all partial derivatives of \(f_2\) at \((0,0,0)\) exist and equal \(0\).
Notation. It is important to know that partial derivatives are often written in many ways. For example, \(\partial_j f, \partial_{x_j} f,\) and \(f_{x_j}\) are all alternate ways of writing \(\frac{\partial f}{\partial x_j}\) so that they fit on one line. Similarly, \(\partial_x f,\) and \(f_x\) are alternate ways of writing \(\frac{\partial f}{\partial x}\), with corresponding notation for \(\frac{\partial f}{\partial y}\), \(\frac{\partial f}{\partial z}\).
We will now investigate the relationship between differentiability and partial differentiability.
That is, the partial derivatives are the components of the gradient vector.
The converse is not true! It can happen that all partial derivatives \(\frac{\partial f}{\partial x_j}\) exist at \(\mathbf a\) but that \(f\) is not differentiable at \(\mathbf a\). This is the case for the functions \(f_1\) and \(f_2\) in Example 3. These functions cannot be differentiable at the origin, since differentiability implies continuity (by Theorem 1) and these functions are not continuous at the origin. But as we have noted, all partial derivatives exist at \((0,0,0)\).
Assume that \(f\) is differentiable at \(\mathbf x\in S\). Consider any \(j \in \{1,\ldots, n\}\), and define \[ g(\mathbf h) = \frac { f(\mathbf x + \mathbf h) - f(\mathbf x) - \nabla f(\mathbf x)\cdot \mathbf h}{|\mathbf h|} \] whenever \(\mathbf h \ne {\mathbf 0}\) and \(\mathbf x+\mathbf h \in S\). Then \[\begin{equation}\label{ddd1} \lim_{\mathbf h\to {\mathbf 0}} g(\mathbf h) = 0 \end{equation}\] by the definition of differentiability. It follows, see problem 9, that for any \(j\in \{1,\ldots,n\}\), \[\begin{equation}\label{ddd2} \lim_{h\to 0} g(h {\mathbf e}_j) = 0. \end{equation}\]
This says that \[\begin{equation}\label{ddd3} \lim_{h\to 0} \frac { f(\mathbf x + h {\mathbf e}_j ) - f(\mathbf x) - \nabla f(\mathbf x)\cdot (h {\mathbf e}_j) }{|h|} = 0 \end{equation}\] It follows that \[\begin{align}\label{ddd4} 0 &= \lim_{h\to 0} \frac { f(\mathbf x + h {\mathbf e}_j ) - f(\mathbf x) - \nabla f(\mathbf x)\cdot (h{\mathbf e}_j )} {h} \\\ &= \lim_{h\to 0} \frac { f(\mathbf x + h {\mathbf e}_j ) - f(\mathbf x)}{h} - \nabla f(\mathbf x)\cdot{\mathbf e}_j \nonumber \end{align}\]
This says that \(\frac{\partial f}{\partial x_j}(\mathbf x)\) exists and equals \(\nabla f(\mathbf x)\cdot {\mathbf e}_j\), which is what we had to prove. You should try to fill in the details of \(\eqref{ddd1} \implies \eqref{ddd2}\) and $ \(\eqref{ddd4}\) in the last two problems of this section.If you need to determine whether a function \(f\) is differentiable at a point \(\mathbf x\), then Theorem 2 can simplify your life. It tells you
if any partial derivatives \(\frac {\partial f}{\partial x_j}\) do not exist at \(\mathbf x\), then \(f\) is not differentiable there, and
if all partial derivatives exist, then the vector \({\mathbf m} =(\partial f/\partial x_1, \ldots, \partial f/\partial x_n)\) is the only possible vector that can work in the definition \(\eqref{der.Rn}\) of differentiability.
Consider the function \(f:\R^2\to \R\) defined by \[ f(x,y) = \begin{cases}\frac{y^3-x^8y}{x^6+y^2}&\text{ if }(x,y)\ne (0,0) \\\ 0&\text{ if }(x,y) = (0,0). \end{cases} \] Is \(f\) differentiable at \((0,0)\)?
Theorem 2 can be used to simplify some complicated problems. The next theorem is even better in this respect — it makes it simple to check differentiability of many functions of several variables, by using single variable derivatives and properties of continuous functions.
Suppose \(f\) is a function \(S\to \R\) for some open \(S\subseteq\R^n\). If all partial derivatives of \(f\) exist and are continuous at every point of \(S\), then \(f\) is differentiable at every point of \(S\).
This theorem motivates the following definition.
Thus by Theorem 3, any \(C^1\) function is differentiable everywhere in \(S\).
For notational simplicity, we will present the proof when \(n=2\). The idea is exactly the same in the general case.
Let \(\mathbf x\) be any point in \(S\). Since \(S\) is open, there exists \(r>0\) such that \(\mathbf x+\mathbf h \in S\) whenever \(|\mathbf h|<r\). Below, we will always assume that \(|\mathbf h|<r\).
Consider a vector \(\mathbf h = (h,k)\). We start by writing \[\begin{align} f(\mathbf x +\mathbf h) - f(\mathbf x) &= f(x+h, y+k) - f(x,y) \label{ppp1} \\ & = [f(x+h,y+k) - f(x+h,y)] + [f(x+h,y)- f(x,y)] \nonumber \end{align}\] Let’s temporarily write \(g(x) = f(x,y)\). Then \[\begin{align*} f(x+h,y)- f(x,y) &= g(x+h) - g(x) \\ &= h \, g'(x+\theta_1 h)\quad\text{ for some }\theta_1\in (0,1) \end{align*}\] by the single variable Mean Value Theorem from MAT 137. Rewriting this in terms of partial derivatives of \(f\), it says that \[ f(x+h,y)- f(x,y) = h \frac{\partial f}{\partial x}(x+\theta_1 h, y)\text{ for some }\theta_1 \in (0,1). \] A very similar argument shows that \[ f(x+h, y+k) - f(x+h, y) = k \frac{\partial f}{\partial y}(x+ h, y+\theta_2 k)\text{ for some }\theta_2 \in (0,1). \] Combining these with \(\eqref{ppp1}\), we see that \[ f(\mathbf x +\mathbf h) = f(\mathbf x) + (h,k)\cdot (\frac{\partial f}{\partial x},\frac{\partial f}{\partial y})(x,y) + E(h,k), \] where \[ E(h,k) = h \left( \frac{\partial f}{\partial x}(x+\theta_1 h, y) -\frac{\partial f}{\partial x}(x,y) \right) + k \left( \frac{\partial f}{\partial y}(x+h, y+\theta_2 k) - \frac{\partial f}{\partial y}(x,y) \right) \] Finally, since \(|h| \le |\mathbf h | = \sqrt{h^2+k^2}\) and \(|k|\le |\mathbf h|\), it follows that
\[ \lim_{\mathbf h\to {\mathbf 0}} \frac {E(\mathbf h)} {|\mathbf h|} \le \lim_{\mathbf h\to {\mathbf 0}}\left| \frac{\partial f}{\partial x}(x+\theta_1 h, y) -\frac{\partial f}{\partial x}(x,y) \right| + \lim_{\mathbf h\to {\mathbf 0}}\left| \frac{\partial f}{\partial y}(x+h, y+\theta_2 k) - \frac{\partial f}{\partial y}(x,y) \right|. \] The right-hand side equals zero by an \((\varepsilon, \delta)\) argument, using our assumption that the partial derivatives are continuous. Hence \[ \lim_{\mathbf h\to {\mathbf 0}} \frac {E(\mathbf h)} {|\mathbf h|} = 0 , \] which proves the differentiability of \(f\) at \(\mathbf x\). Since \(\mathbf x\) was an arbitrary point of \(S\), this completes the proof.Let \(f(x,y) = (x-1)^3(y+2)\). At which points of \(\R^2\) is \(f\) differentiable?
Let \(f:\R^n\to \R\) be a polynomial of total degree \(d\). At which points of \(\R^n\) is \(f\) differentiable?
Contrast this with the example using a naive, incorrect definition for differentiable. The correct definition of differentiable functions eventually shows that polynomials are differentiable, and leads us towards other concepts that we might find useful, like \(C^1\). The incorrect naive definition leads to \(f(x,y)=x\) not being differentiable. Although it looks more complicated, the correct version does two important things that we look for from mathematical definitions: it includes the functions that we intuitively believe it should, and it leads us to new interesting properties.
A direction in \(\R^n\) is naturally represented by a unit vector. In general a vector has a direction and a magnitude; if we are only interested in directions, we can just consider vectors with magnitude equal to \(1\), i.e. unit vectors.
Thus, given a unit vector \(\mathbf u\) and a point \(\mathbf x\in \R^n\), the point \(\mathbf x+ h \mathbf u\) is the point reached by starting at \(\mathbf x\) and traveling a distance \(h\) in the direction \(\mathbf u\). So \(f(\mathbf x+h\mathbf u) - f(\mathbf x)\) represents the change in \(f\) if we start at \(\mathbf x\) and move a distance \(h\) in the direction \(\mathbf u\). This motivates the following definition:
Based on our knowledge of first-year calculus, we can see that \(\partial_\mathbf u f(\mathbf x)\) represents the instantaneous rate of change of \(f\) as we move in the direction \(\mathbf u\) through the point \(\mathbf x\).
By comparing the definitions of directional derivative and partial derivative, we see that for any \(j\in \{1,\ldots, n\}\), \[ \frac{\partial f}{\partial x_j} = \partial_{{\mathbf e}_j} f \] where \(\mathbf e_j\) denotes the unit vector in the \(j\)th coordinate direction.
The proof of this is almost exactly like the proof of Theorem 2. This is not surprising, since partial derivatives are just a special case of directional derivatives.
The proof does not require \(\mathbf u\) to be a unit vector; this is only needed for the interpretation of \(\partial_{\mathbf u}f\) as a directional derivative. For any vector \(\mathbf v\), whether or not it is a unit vector, it is generally true that if \(f\) is differentiable at a point \(\mathbf x\), then \[\begin{equation}\label{qd} \lim_{h\to 0} \frac{ f(\mathbf x + h {\mathbf v}) - f(\mathbf x)}{h} = {\mathbf v} \cdot \nabla f(\mathbf x). \end{equation}\] The proof again is essentially the same as that of Theorem 2. It is left as an exercise.
The converse of Theorem 4 is not true. One can find functions \(f\) such that at some point \(\mathbf x\), the directional derivative \(\partial _\mathbf u f(\mathbf x)\) exists for every unit vector \(\mathbf u\), but \(f\) is not differentiable at \(\mathbf x\). See the exercises below for some examples.
Assume that \(S\) is an open subset of \(\R^n\) and that \(f:S\to \R\) is differentiable. At a point \(\mathbf x\in S\), determine the direction \(\mathbf u^*\) in which \(f\) is increasing most rapidly, in the sense that \[ \partial_{\mathbf u^*}f(\mathbf x) = \max \{ \partial_\mathbf u f(\mathbf x) : \mathbf u\text{ a unit vector} \}. \]
Solution
By Theorem 4 and basic properties of the dot product, we know that for any unit vector \(\mathbf u\), \[\begin{equation}\label{qp2}
\partial_\mathbf u f(\mathbf x) = \mathbf u \cdot \nabla f(\mathbf x) = |\mathbf u| \, |\nabla f(\mathbf x)| \, \cos \theta = |\nabla f(\mathbf x)| \, \cos \theta,
\end{equation}\] where \(\theta\) is the angle between \(\nabla f(\mathbf x)\) and \(\mathbf u\).
We consider two cases:
Case 1. If \(\nabla f(\mathbf x) = \mathbf 0\) then \(\partial_\mathbf u f(\mathbf x)= 0\) for all \(\mathbf u\), so every unit vector maximizes (and minimizes) \(\partial_\mathbf u f(\mathbf x)\).
Case 2. If \(\nabla f(\mathbf x) \ne \mathbf 0\), then according to \(\eqref{qp2}\) we have to choose \(\mathbf u\) so that \(\cos\theta\) is as large as possible. This happens when \(\cos \theta = 1\), that is, when \(\mathbf u\) is the unit vector pointing in the same direction as \(\nabla f(\mathbf x)\). That is, the directional derivative is maximized in the direction \[\begin{equation}\label{fp} \mathbf u^* = \frac{\nabla f(\mathbf x)} {|\nabla f(\mathbf x)|}. \end{equation}\]This example tells us what the gradient of a real-valued function means. If you remember only one thing from MAT 237, it should be this:
This is the principle that allows machine learning by gradient descent, determines seam carving for image resizing, and will play an important role in the rest of this course.
Let \(f(x,y,z) = \frac 12 x^2 + \frac 14 y^4 + \frac 16 z^6\). Find the direction in which \(f\) has the greatest increase at the point \((1,1,1)\).
Let \(f(x,y,z) = xyz^2\). Find the direction in which \(f\) is decreasing most rapidly at the point \((1,1,1)\).
Solution
If \(f\) has the greatest decrease in the direction \(\mathbf u\), then \(-f\) is increasing most rapidly in the direction \(\mathbf u\). By linearity of the dot product and gradient, \(\mathbf u \cdot \nabla(-f)=-\mathbf u \cdot \nabla(f)\), so \(f\) is decreasing most rapidly in the direction of \(-\nabla f\).
Problems like this are normally solved by using Theorem 3 and properties of continuous functions which allow us to recognize partial derivatives as continuous. Examples might be
Given a function \(f\), find the direction in which \(f\) is increasing/decreasing most rapidly at the point \(\mathbf x\). For example, consider any of the functions in the above question, at the point \((1,1)\) or \((1,1,1)\), depending on the dimension.
If \(f\) is a complicated function like Example 4, determining whether \(f\) is differentiable at a given point using the defintion rather than using Theorem 3 is difficult. But determining the directional derivatives at a point using their definition is not. For example
For any unit vector \(\mathbf u = (u_1, u_2)\), determine \(\partial_\mathbf u f({\mathbf 0})\).
Prove that \(f\) is not differentiable at the origin.
Hint
If \(f\) were differentiable at the origin, then necessarily \(\partial_\mathbf u f({\mathbf 0}) = u_1\partial_1 f({\mathbf 0}) + u_2 \partial_2 f({\mathbf 0})\).
Is \(f\) continuous at \((0,0)\)? Determine the answer using material from Section 1.2.
Show that for every unit vector \({\mathbf u} = (u_1, u_2)\), the directional derivative \(\partial_{\mathbf u}f(0,0)\) exists and equals zero.
Prove that \(f\) is not differentiable at \((0,0)\).
Hint
Consider \(\dfrac { f(t, t^2) - f(0,0) }{|(t,t^2)|}\) for small values of \(t\). Also, note that if \(|t|<1\), then \(|t| \le |(t, t^2)| \le \sqrt 2 |t|\).
Prove that if \(f\) is differentiable away from the origin and homogeneous of degree \(d\), then for every unit vector \(\mathbf u\), the directional derivative \(\partial_\mathbf u f\) is homogeneous of degree \(d-1\).
In particular this imples that all partial derivatives are homogeneous of degree \(d-1\), since partial derivatives are a special case of directional derivatives.
Give a detailed proof of \(\eqref{qd}\), and hence of Theorem 4 (by small modifications of the proof of Theorem 2.)
Suppose that \(S\) is an open subset of \(\R^n\) that contains the origin, and that \(g\) is a function \(S\to \R\). Prove that if \(\lim_{\mathbf x\to {\mathbf 0}} g(\mathbf x) = L,\) then \(\displaystyle\lim_{h\to 0} g(h\mathbf u) = L\) for every unit vector \(\mathbf u\).
This was used in the proof of Theorem 2. It is good practice to write out the proof. You should be able to supply all relevant definitions from memory. This would be considered an easy proof question for a test.
\(\Leftarrow\) \(\Uparrow\) \(\Rightarrow\)
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 Canada License.