## Introduction to Numerical Analysis: Taylor Series

### Definitions

Let be a smooth (differentiable) function, and let , then a Taylor series of the function around the point is given by:

In particular, if , then the expansion is known as the Maclaurin series and thus is given by:

### Taylor’s Theorem

Many of the numerical analysis methods rely on Taylor’s theorem. In this section, a few mathematical facts are presented (mostly without proof) which serve as the basis for Taylor’s theorem.

#### Extreme Values of Smooth Functions

##### Definition: Local Maximum and Local Minimum

Let . is said to have a local maximum at a point if there exists an open interval such that and . On the other hand, is said to have a local minimum at a point if there exists an open interval such that and . If has either a local maximum or a local minimum at , then is said to have a local extremum at .

##### Proposition 1

Let be smooth (differentiable). Assume that has a local extremum (maximum or minimum) at a point , then .
This proposition simply means that if a smooth function attains a local maximum or minimum at a particular point, then the slope of the function is equal to zero at this point.
As an example, consider the function with the relationship . In this case, is a local maximum value for attained at and is a local minimum value of attained at . These local extrema values are associated with a zero slope for the function since

and are locations of local extrema and for both we have . The red lines in the next figure show the slope of the function at the extremum values.

View Mathematica Code that Generated the Above Figure

Clear[x]
y = x^3 - 3 x;
Plot[y, {x, -3, 3}, Epilog -> {PointSize[0.04], Point[{-1, 2}], Point[{1, -2}], Red, Line[{{-3, 2}, {1.5, 2}}], Line[{{3, -2}, {-1.5, -2}}]}, Filling -> Axis, PlotRange -> All, Frame -> True, AxesLabel -> {"x", "y"}]


“Smoothness” or “Differentiability” is a very important requirement for the proposition to work. As an example, consider the function defined as . The function has a local minimum at , however, is not defined as the slope as from the right is different from the slope as from the left as shown in the next figure.

View Mathematica Code that Generated the Above Figure
Clear[x]
y = Abs[x];
Plot[y, {x, -1, 1}, Epilog -> {PointSize[0.04], Point[{0, 0}]}, PlotRange -> All, Frame -> True, AxesLabel -> {"x", "y"}]

##### Proposition 2

Let be smooth (differentiable). Assume such that . Then,

• If , then is constant around .
• If , then is a local minimum.
• If , then is a local maximum.

This proposition is very important for optimization problems when a local maximum or minimum is to be obtained for a particular function. In order to identify whether the solution corresponds to a local maximum or minimum, the second derivative of the function has to be evaluated. Considering the same example given above,

We have already identified and as locations of the local extremum values. To know whether they are local maxima or local minima, we can evaluate the second derivative at these points. . Therefore, is the location of a local minimum, while implying that is the location of a local maximum.

#### Extreme Value Theorem

Statement: Let be continuous. Then, attains its maximum and its minimum value at some points and in the interval .

The theorem simply states that if we have a continuous function on a closed interval , then the image of contains a maximum value and a minimum value within the interval . The theorem is very intuitive. However, the proof is highly technical and relies on the definitions of real numbers and on continuous functions. For now, we will just illustrate the meaning of the theorem using an example. Consider the function defined as:

The theorem states that has to attain a maximum value and a minimum value at a point within the interval. In this case, is the maximum value of attained at and is the minimum value of attained at . Alternatively, if with the same relationship as above, is the minimum value of attained at and is the maximum value of attained at
The following figure shows the graph of the function on the specified intervals.

View Mathematica Code that Generated the Above Figures

Clear[x]
y = x^3 - 3 x;
Plot[y, {x, -1.5, 1.5}, Epilog -> {PointSize[0.04], Point[{-1, 2}], Point[{1, -2}]}, Filling -> Axis, PlotRange -> All, Frame -> True, AxesLabel -> {"x", "y"}]
Plot[y, {x, -3, 3}, Epilog -> {PointSize[0.04], Point[{-3, y /. x -> -3}], Point[{3, y /. x -> 3}]}, Filling -> Axis, PlotRange -> All, Frame -> True, AxesLabel -> {"x", "y"}]


#### Rolle’s Theorem

Statement: Let be differentiable. Assume that , then there is at least one point where .

The proof of Rolle’s theorem is straightforward from Proposition 1 and the Extreme Value Theorem above. The Extreme Value Theorem ensures that there is a local maximum or local minimum within the interval, while proposition 1 ensures that at this local extremum, the slope of the function is equal to zero. As an example, consider the function defined as . . This ensures that there is a point with . Indeed, and the point is the location of the local minimum. The following figure shows the graph of the function on the specified interval along with the point .

View Mathematica Code that Generated the Above Figure
Clear[x]
y = 20 (x - 1/2)^3 - 20 (x - 1/2) + 5;
Expand[y]
y /. x -> 1.5
y /. x -> 0.5
y /. x -> (1/2 + 1/Sqrt[3])
D[y, x] /. x -> (1/2 + 1/Sqrt[3])
Plot[y, {x, 0.5, 1.5}, Epilog -> {PointSize[0.04], Point[{1/2 + 1/Sqrt[3], y /. x -> 1/2 + 1/Sqrt[3]}], Red, Line[{{-3, y /. x -> 1/2 + 1/Sqrt[3]}, {1.5, y /. x -> 1/2 + 1/Sqrt[3]}}]}, Filling -> Axis, PlotRange -> All, Frame -> True, AxesLabel -> {"x", "y"}]


#### Mean Value Theorem

Statement: Let be differentiable. Then, there is at least one point such that .

The proof of the mean value theorem is straightforward by applying Rolle’s theorem to the function defined as:

Clearly, satisfies the conditions of Rolle’s theorem (Differentiable and ). We also have:

Therefore, by Rolle’s theorem, there is a point such that .

The mean value theorem states that there is a point inside the interval such that the slope of the function at is equal to the average slope along the interval. The following example will serve to illustrate the main concept of the mean value theorem. Consider the function defined as:

The slope or first derivative of is given by:

The average slope of on the interval is given by:

The two points and have a slope equal to the average slope:

The figure below shows the function on the specified interval. The line representing the average slope is shown in black connecting the points and . The red lines show the slopes at the points and .

View Mathematica Code that Generated the Above Figure
Clear[x]
y = x^3 - 3 x;
averageslope = ((y /. x -> 3) - (y /. x -> -3))/(3 + 3)
dydx = D[y, x];
a = Solve[D[y, x] == averageslope, x]
Point1 = {x /. a[[1, 1]], y /. a[[1, 1]]}
Point2 = {x /. a[[2, 1]], y /. a[[2, 1]]}
Plot[y, {x, -3, 3},  Epilog -> {PointSize[0.04], Point[{-3, y /. x -> -3}], Point[{3, y /. x -> 3}], Line[{{-3, y /. x -> -3}, {3, y /. x -> 3}}], Point[Point1],Point[Point2], Red, Line[{Point1 + {-1, -averageslope}, Point1, Point1 + {1, averageslope}}], Line[{Point2 + {-1, -averageslope}, Point2, Point2 + {1, averageslope}}]}, Filling -> Axis, PlotRange -> All, Frame -> True, AxesLabel -> {"x", "y"}]


#### Taylor’s Theorem

As an introduction to Taylor’s Theorem, let’s assume that we have a function that can be represented as a polynomial function in the following form:

where is a fixed point and is a constant. The best way to find these constants is to find and its derivatives when . So, when we have:

Therefore, . The derivatives of have the form:

The derivatives of when have the form:

Therefore, :

The above does not really serve as a rigorous proof for Taylor’s Theorem but rather an illustration that if an infinitely differentiable function can be represented as the sum of an infinite number of polynomial terms, then, the Taylor series form of a function defined at the beginning of this section is obtained. The following is the exact statement of Taylor’s Theorem:

Statement of Taylor’s Theorem: Let be times differentiable on an open interval . Let . Then, between and such that:

Explanation and Importance: Taylor’s Theorem has numerous implications in analysis in engineering. In the following we will discuss the meaning of the theorem and some of its implications:

• Simply put, Taylor’s Theorem states the following: if the function and its derivatives are known at a point , then, the function at a point away from can be approximated by the value of the Taylor’s approximation :

The error (difference between the approximation and the exact is given by:

The term is bounded since is a continuous function on the interval from to . Therefore, when , the upper bound of the error can be given as:

While, when , the upper bound of the error can be given as:

The above implies that the error is directly proportional to . This is traditionally written as follows:

where . In other words, as gets smaller and smaller, the error gets smaller in proportion to . As an example, if we choose and then , then, . I.e., if the step size is halved, the error is divided by .

• If the function is infinitely differentiable on an interval , and if , then is the limit of the sum of the Taylor series. The error which is the difference between the infinite sum and the approximation is called the truncation error as defined in the error section.
• There are many rigorous proofs available for Taylor’s Theorem and the majority rely on the mean value theorem above. Notice that if we choose , then the mean value theorem is obtained. For a rigorous proof, you can check one of these links: link 1 or link 2. Note that these proofs rely on the mean value theorem. In particular, L’HÃ´pital’s rule was used in the Wikipedia proof which in turn relies on the mean value theorem.

The following code illustrates the difference between the function and the Taylor’s polynomial . You can download the code, change the function, the point , and the range of the plot to see how the Taylor series of other functions behave.

View Mathematica Code
Taylor[y_, x_, a_, n_] := (y /. x -> a) +
Sum[(D[y, {x, i}] /. x -> a)/i!*(x - a)^i, {i, 1, n}]
f = Sin[x] + 0.01 x^2;
Manipulate[
s = Taylor[f, x, 1, nn];
Grid[{{Plot[{f, s}, {x, -10, 10}, PlotLabel -> "f(x)=Sin[x]+0.01x^2",
PlotLegends -> {"f(x)", "P(x)"},
PlotRange -> {{-10, 10}, {-6, 30}},
ImageSize -> Medium]}, {Expand[s]}}], {nn, 1, 30, 1}]


The Mathematica function Series can also be used to generate the Taylor expansion of any function:
View Mathematica Code

Series[Tan[x],{x,0,7}]
Series[1/(1+x^2),{x,0,10}]


The following tool shows how the Taylor series expansion around the point , termed in the figure, can be used to provide an approximation of different orders to a cubic polynomial, termed in the figure. Use the slider to change the order of the series expansion. The tool provides the error at , namely . What happens when the order reaches 3?

### Applications and Examples

#### MacLaurin Series

The following are the MacLaurin series for some basic infinitely differentiable functions:

#### Numerical Differentiation

Taylor’s Theorem provides a means for approximating the derivatives of a function. The first-order Taylor approximation of a function is given by:

where is the step size. Then, :

##### Forward finite-difference

If we use the Taylor series approximation to estimate the value of the function at a point knowing the values at point then,we have:

(1)

where . In this case, the forward finite-difference can be used to approximate as follows:

##### Backward finite-difference

If we use the Taylor series approximation to estimate the value of the function at a point knowing the values at point then, we have:

(2)

where . In this case, the backward finite-difference can be used to approximate as follows:

##### Centred Finite Difference

The centred finite difference can provide a better estimate for the derivative of a function at a particular point. If the values of a function are known at the points and , then, we can use the Taylor series to find a good apprxoimation for the derivative as follows:

Subtracting the above two equations and dividing by gives the following:

#### Building Differential Equations

See momentum balance and beam approximation for examples where the Taylor’s approximation is used to build a differential equation.

#### Approximating Continuous Functions

Taylor’s theorem essentially discusses approximating differentiable functions using polynomials. The approximation can be as close as needed by adding more polynomial terms and/or by ensuring that the step size is small enough. It is important to realize that polynomial approximations are valid for continuous functions that are not necessarily differentiable at every point. Let , then the Stone-Weierstrass Theorem states that for any small number a polynomial function such that the biggest difference between . In other words, we can always find a polynomial function that approximates any continuous function on the interval within any degree of accuracy sought! This fact is extremely important in all engineering applications. Essentially, it allows modelling any continuous functions using polynomials. In other words, if a problem wishes to find the distribution of a variable (stress, strain, velocity, density, etc…) as a function of position, and if this variable is continuous, then, in the majority of applications we can assume differentiability and a Taylor approximation for the unknown function.

#### Examples

##### Example 1

Apply Taylor’s Theorem to the function defined as to estimate the value of . Use . Estimate an upper bound for the error.

##### Solution

The Taylor approximation of the function around the point is given as follows:

If terms are used (including ), then the upper bound for the error is:

With , an upper bound for the error can be calculated by assuming . That’s because the absolute values of the derivatives of attain their maximum value in the interval at 4. The derivatives of have the following form:

The derivatives of the function evaluated at the point can be calculated as:

If two terms are used, then:

In this case, the upper bound for the error is:

Using Mathematica, the square root of approximated to 4 decimal places is . Therefore, the error when two terms are used is:

which satisfies that the actual error is less the upper bound:

If three terms are used, then:

In this case, the upper bound for the error is:

The actual error in this case is indeed less than the upper bound:

The following code provides a user-defined function for the Taylor series having the following inputs: a function , the value of , the value of , and the number of terms including the constant term.
View Mathematica Code

Clear[x]
Taylor[f_, xi_, a_, n_] := Sum[(D[f, {x, i}] /. x -> a)/i!*(xi - a)^i, {i, 0, n - 1}]
f = (x)^(0.5);
a = Table[{i, Taylor[f, 5, 4, i]}, {i, 2, 21}];
a // MatrixForm

##### Example 2

Apply Taylor’s Theorem to the function defined as to estimate the value of and . Use . Estimate an upper bound for the error.

##### Solution

First, we will calculate the numerical solution for the and :

The Taylor approximation around is given as:

If terms are used (including ) and if , then the upper bound of the error is:

If , then, the upper bound of the error is:

The derivatives of the function are given by:

when evaluated at these have the values:

For , and using two terms:

Using three terms:

Using four terms:

The terms are getting closer to each other and four terms provide a good approximation. The error term in the theorem gives an upper bound for as follows:

The maximum value would be obtained for . Therefore:

Indeed, the actual value of the error is less than the upper bound:

For , the Taylor series for this function around doesn’t give a very good approximation as will be shown here but rather keeps oscillating. First, using two terms:

Using three terms:

Using four terms:

Using five terms:

Using six terms:

In fact, the following table gives the values up to 21 terms. It is clear that the Taylor series is diverging.

The error term in the theorem gives an upper bound for when six terms are used as follows:

The maximum value will be obtained when :

This is a large upper bound and indicates that using six terms is not giving a good approximation. In general, the Taylor series works best if the distance between and is as small as possible. For some functions, like , , and , the Taylor series always converges. However, for functions with square roots, the Taylor series converges when is relatively close to . There are some analytical conditions that would indicate the radius of convergence (x-a) of a Taylor series; however, this is beyond the scope of this course!

The following code provides a user-defined function for the Taylor series having the following inputs: a function , the value of , the value of , and the number of terms including the constant term.
View Mathematica Code

Clear[x]
Taylor[f_, xi_, a_, n_] := Sum[(D[f, {x, i}] /. x -> a)/i!*(xi - a)^i, {i, 0, n-1}]
f = (1 - x)^(0.5);
a = Table[{i, Taylor[f, -2, 0, i]}, {i, 2, 21}];
a // MatrixForm
a = Table[{i, Taylor[f, 0.1, 0, i]}, {i, 2, 21}];
a // MatrixForm

##### Example 3

Use zero through fourth order Taylor’s series expansion to approximate the value of the function defined as at . Use . Calculate the error associated with each expansion.

##### Solution

The true value of the function at is given by:

The zero order Taylor series expansion around has the following form:

The error in this case is given by:

The first order Taylor series expansion around has the following form:

The error in this case is given by:

The second order Taylor series expansion around has the following form:

The error in this case is given by:

The third order Taylor series expansion around has the following form:

The error in this case is given by:

The fourth order Taylor series expansion around has the following form:

The error in this case is given by:

It is important to note that the error reduces to zero when using the fourth order Taylor series approximation. That is because the fourth order Taylor series approximation of a fourth order polynomial function is identical to the function itself. You can think of this as follows, the zero order Taylor approximation provides a “constant” function approximation. The second order Taylor approximation provides a parabolic function approximation while the third order provides a cubic function approximation. The nth Taylor series approximation of a polynomial of degree “n” is identical to the function being approximated!

#### Problems

1. Use the Taylor series for the function defined as to estimate the value of . Use once and for another time. Estimate an upper bound for the error for each approximation. Comment on the behaviour of the Taylor series of this function. (hint: For this particular function using a Taylor expansion around should not give a proper approximation for because 10 and 4 are far from each other)
2. Using the Taylor series and setting , derive the polynomial forms of the functions listed in the MacLaurin series section.
3. Use Taylor’s Theorem to find an estimate for at with . Employ the zero-, first-,
second-, and third-order versions and compute the truncation error for each case.
4. Using the MacLaurin series expansion for , find an approximation for as a function of the number of terms up to 5 terms. For each case, find the relative error and the relative approximate error .
5. Use zero- through third-order Taylor series expansions to predict for

assuming that . Compute for each approximation.

6. Derive the MacLaurin series for the following functions:

Note: The resulting series are only convergent for

7. Derive the MacLaurin series for the following functions:

Note: The resulting series are only convergent for