Linear Systems of Equations: Linear Algebra
Definitions
Linear Vector Spaces
In this course, by a linear vector space we mean the set
![Rendered by QuickLaTeX.com \[ \mathbb{R}^n=\mathbb{R}\times\mathbb{R}\times\cdots\mathbb{R}=\left\{\left(\begin{array}{c}x_1\\x_2\\\vdots\\x_n\end{array}\right)\bigg| x_1,x_2,\cdots,x_n\in\mathbb{R} \right\} \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-24f1dd9048735c5e685d89acf32998ab_l3.png)
The dimension of this vector space is
. A vector in this space
has
components
each of which is a real number. The zero vector is the vector whose components are all equal to zero.
For example
is the two-dimensional linear vector space. Each vector in this space has two components.
can be represented geometrically by a plane.
is a three-dimensional linear vector space. Each vector in this space has three components. The vector
is the zero vector in
. The vector
is the zero vector in
.
Linear Functions between Vector Spaces(Matrices)
Given a linear vector space
, and another linear vector space
, a linear function
has the following form,
:
![Rendered by QuickLaTeX.com \[ f(x)=\left(\begin{array}{c} a_{11}x_1+a_{12}x_2+\cdots +a_{1m}x_m\\ a_{21}x_1+a_{22}x_2+\cdots +a_{2m}x_m\\ \vdots \\ a_{n1}x_1+a_{n2}x_2+\cdots +a_{nm}x_m \end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-ba40af3313fff7b462d0652e85ab5ce3_l3.png)
The function
takes a vector
with
components and returns a vector
which has
components. Using matrix notation, the above can be written in the following more convenient form:
![Rendered by QuickLaTeX.com \[ f(x)=\left(\begin{matrix}a_{11}&a_{12}&\cdots&a_{1m}\\a_{21}&a_{22}&\cdots&a_{2m}\\\vdots&\vdots&\vdots&\vdots\\a_{n1}&a_{n2}&\cdots&a_{nm}\end{matrix}\right)\left(\begin{array}{c}x_1\\x_2\\\vdots\\x_m\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-75d94f3f14067090d2468eb340812bf9_l3.png)
If we denote the
matrix of numbers by
, then the linear function
can be written in the following simple form:
![]()
We noted that
is an element of the linear vector space
. Similarly, we view
as an element of the possible linear functions from
to
which is denoted: ![]()
If
, then for simplicity, we say that
. We also use the symbol
to denote the set of square matrices that live in the space
.
For a visual representation showing the action of the matrix
on the space
, see the tool shown here.
Matrix Transpose
If
, then the transpose of
, namely
is defined as the matrix
whose columns are the rows of
.
Examples
Example 1
Consider the function
defined as
. Following the above definition,
is a one-dimensional vector and the matrix
has dimensions
. In this case
and
.
Example 2
Consider the function
defined as follows:
:
![]()
In matrix form, this function can be represented as follows:
![Rendered by QuickLaTeX.com \[ f(x)=\left(\begin{array}{ccc}5&3&2\\0&2&4\end{array}\right)\left(\begin{array}{c}x_1\\x_2\\x_3\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-7a7c60fcca8faf2a14181322554e70d1_l3.png)
This function takes vectors that have three components and gives vectors that have two components. The matrix
associated with this linear function has dimensions
, lives in the space
, and has the form:
![]()
In this case,
is simply:
![Rendered by QuickLaTeX.com \[ A^T=\left(\begin{array}{cc}5&0\\3&2\\2&4\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-4fbd748d1805a648d6a7fe6d75a15e4c_l3.png)
Example 3
Consider the function
defined as follows:
:
![Rendered by QuickLaTeX.com \[ f(x)=\left(\begin{array}{c}5x_1+3x_2+2x_3\\2x_2+4x_3\\2x_1-3x_3\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-f7033cba98b31392418999f7b76b3a52_l3.png)
In matrix form, this function can be represented as follows:
![Rendered by QuickLaTeX.com \[ f(x)=\left(\begin{array}{ccc}5&3&2\\0&2&4\\2&0&-3\end{array}\right)\left(\begin{array}{c}x_1\\x_2\\x_3\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-2271b9bf899f90717ecbd22f8eb2504b_l3.png)
This function takes vectors that have three components and gives vectors that have three components. The matrix
associated with this linear function has dimensions
, lives in the space
, and has the form:
![Rendered by QuickLaTeX.com \[ A=\left(\begin{array}{ccc}5&3&2\\0&2&4\\2&0&-3\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-e826d3bbf2de048b79a214b821202bb7_l3.png)
In this case,
is simply:
![Rendered by QuickLaTeX.com \[ A^T=\left(\begin{array}{ccc}5&0&2\\3&2&0\\2&4&-3\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-d6b5b3e5b9b968a03392545ff230a722_l3.png)
Example 4
Consider the function
defined as follows:
:
![]()
In matrix form, this function can be represented as follows:
![Rendered by QuickLaTeX.com \[ f(x)=\left(\begin{array}{ccc}5&0&2\end{array}\right)\left(\begin{array}{c}x_1\\x_2\\x_3\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-f34beff8b9b7796aeafc9ce579a9a735_l3.png)
This function takes vectors that have three components and gives vectors that have one component. The matrix
associated with this linear function has dimensions
, lives in the space
, and has the form:
![]()
In this case,
is simply:
![Rendered by QuickLaTeX.com \[ A^T=\left(\begin{array}{c}5\\0\\2\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-cd85db17de5b90d65049d99aa81973c4_l3.png)
Linear System of Equations
Let
and given a vector
, a linear system of equations seeks the solution
to the equation:
![]()
In a matrix component form the above equation is:
![Rendered by QuickLaTeX.com \[ \left(\begin{matrix}a_{11}&a_{12}&\cdots&a_{1m}\\a_{21}&a_{22}&\cdots&a_{2m}\\\vdots&\vdots&\vdots&\vdots\\a_{n1}&a_{n2}&\cdots&a_{nm}\end{matrix}\right)\left(\begin{array}{c}x_1\\x_2\\\vdots\\x_m\end{array}\right)=\left(\begin{array}{c}b_1\\b_2\\\vdots\\b_n\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-996d7a8461f241a2cd64059b884e8b39_l3.png)
Another way of viewing this set of equations is as follows:
![Rendered by QuickLaTeX.com \[\begin{split} a_{11}x_1+a_{12}x_2+\cdots +a_{1m}x_m&=b_1\\ a_{21}x_1+a_{22}x_2+\cdots +a_{2m}x_m&=b_2\\ &\vdots \\ a_{n1}x_1+a_{n2}x_2+\cdots +a_{nm}x_m&=b_n \end{split} \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-f0478bbb8ab7de073b43f700e69555d6_l3.png)
The solution for a linear system of equations is the numerical value for the variables
that would satisfy the above equations. There are three possible scenarios:
Scenario 1
If
then we have more unknown components of the
-dimensional vector
than we have equations. The system is then termed underdetermined. This results in having many possible solutions, or no solution. For example, consider the linear system of equations
defined as:
![Rendered by QuickLaTeX.com \[ \left(\begin{array}{ccc}5&3&2\\0&2&4\end{array}\right)\left(\begin{array}{c}x_1\\x_2\\x_3\end{array}\right)=\left(\begin{array}{c}1\\2\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-02848490b93104db9e3f07a700add2ad_l3.png)
There are many possible solutions that would satisfy the above equation. One solution can be obtained by setting
. Then:
![]()
which results in
and
. Therefore one possible solution is:
![Rendered by QuickLaTeX.com \[ x=\left(\begin{array}{c}\frac{2}{5}\\-1\\1\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-e6d8ae83e908d48df24c6d1cdd837494_l3.png)
Another possible solution can be obtained by setting
. Then:
![]()
which results in
and
. Therefore another possible solution is:
![Rendered by QuickLaTeX.com \[ x=\left(\begin{array}{c}\frac{6}{5}\\-3\\2\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-de6c4648ba21520e83471a1373d5a133_l3.png)
In fact, there are infinite possible solutions that depend on what value you choose for
. So, if
is set such that
, then we have:
![]()
Therefore:
![]()
and
![]()
Therefore:
![Rendered by QuickLaTeX.com \[ x=\left(\begin{array}{c}\frac{-2+4\alpha}{5}\\1-2\alpha\\\alpha\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-5283dbe5a2e2df17bfb9a1132669428e_l3.png)
Scenario 2
If
and the
equations are linearly independent, then we have more equations than we have unknown components of the
-dimensional vector
. This results in having too many restrictions which prevents us from finding a solution. In this case, the system is termed overdetermined. For example, consider the linear system of equations
defined as:
![Rendered by QuickLaTeX.com \[ \left(\begin{array}{cc}1&3\\0&2\\-1&1\end{array}\right)\left(\begin{array}{c}x_1\\x_2\end{array}\right)=\left(\begin{array}{c}1\\2\\2\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-ef451fd19e267354b2a30d5210556124_l3.png)
In this system, we have three independent equations and two unknowns
and
. If the first two equations are used to find a solution, then the third equation will present a contradiction. The first two equations have the following form:
![]()
which result in the solution
, and
. If we substitute this solution into equation 3 we get:
![]()
Scenario 3
If
and the
equations are independent, i.e., the rows of the matrix are linearly independent, then one unique solution exists and the matrix
that forms the equation has an inverse. In this case, there exists a matrix
such that:
![]()
To know whether the equations are independent or not the determinant function defined below is sufficient. If the determinant of a matrix is equal to 0, then the rows of the matrix are linearly dependent, and so, an inverse cannot be found. If the determinant of a matrix on the other hand is not equal to 0, then the rows of the matrix are linearly independent and an inverse can be found.
For example, consider the system
defined as:
![]()
The rows of the matrix
are clearly linearly independent. The inverse of
can be found (which will be discussed later in more detail). For now, we can simply solve the two equations as follows. From the second equation we get:
![]()
Substituting into the first equation we get:
![]()
Therefore, the system has a unique solution.
If
and the
equations are dependent, i.e., the rows of the matrix are linearly dependent, then it is not possible to find a unique solution for the linear system of equations. For example, consider the system
defined as:
![]()
The rows of the matrix
are clearly linearly dependent. The inverse of
can not be found (which will be discussed later in more detail). The system presents a contradiction and cannot be solved because the first equation predicts that
while the second equation predicts that
!
As a different example, consider the system
defined as:
![]()
The rows of the matrix
are clearly linearly dependent. The inverse of
can not be found (which will be discussed later in more detail). The system has infinite number of possible solutions. The first and second equations give us the relationship
. So, by assuming any value for
, we can find a value for
. For example, if
, then
. Another possible solution is
and
.
We will focus in the remaining part of this section on how to find a solution for a linear system of equations when
.
Special Types of Square Matrices
Symmetric Matrix
A symmetric matrix
is a matrix that satisfies
. For example:
![]()
is a symmetric matrix.
Identity Matrix
The identity matrix
is the matrix whose diagonal components have the value of 1 while its off diagonal components have a value of zero. For example,
has the form:
![Rendered by QuickLaTeX.com \[ I=\left(\begin{array}{ccc}1&0&0\\0&1&0\\0&0&1\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-e4bf02689c68eec80797a7aa349ba72c_l3.png)
This is called the identity matrix because it takes every vector
and returns the same vector. For example, consider the general vector
with components
,
, and
. The same vector is obtained after the operation
:
![Rendered by QuickLaTeX.com \[ Ix=\left(\begin{array}{ccc}1&0&0\\0&1&0\\0&0&1\end{array}\right)\left(\begin{array}{c}x_1\\x_2\\x_3\end{array}\right)=\left(\begin{array}{c}x_1\\x_2\\x_3\end{array}\right)=x \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-adc1aea4f436c37e0e6591076776e750_l3.png)
Diagonal Matrix
A matrix
is called diagonal if all its off diagonal components are zero. The identity matrix is a diagonal matrix. Similarly, the following matrix is diagonal:
![Rendered by QuickLaTeX.com \[ M=\left(\begin{array}{cccc}5&0&0&0\\0&-1&0&0\\0&0&0&0\\0&0&0&-3\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-ae78cb2f5bfb508b610e49836c52c97d_l3.png)
Upper Triangular Matrix
Let
.
is an upper triangular matrix if all the components below the diagonal components are equal to zero. For example, the following matrix is an upper triangular matrix:
![Rendered by QuickLaTeX.com \[ M=\left(\begin{matrix}1&2&3\\0&1&4\\0&0&5\end{matrix}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-459558072df006119fcbcb283bad63a0_l3.png)
Lower Triangular Matrix
Let
.
is a lower triangular matrix if all the components above the diagonal components are equal to zero. For example, the following matrix is a lower triangular matrix:
![Rendered by QuickLaTeX.com \[ M=\left(\begin{matrix}1&0&0&0\\5&2&0&0\\2&3&1&0\\1&2&0&3\end{matrix}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-5af11b0024f182098c46ff0d850d5007_l3.png)
Banded Matrix
A matrix
is called banded if all its components are zero except the components on the main diagonal and one or more diagonals on either side. To quantify how banded a matrix is, the bandwidth of a matrix is defined. The bandwidth of a matrix indicates the width of the element band out of which all other elements are zero. To define the bandwidth, first we define the lower or left half-bandwidth, and the upper or right half-bandwidth as follows.
The upper bandwidth
is the smallest positive integer such that
when
. This finds the last diagonal, above the main diagonal, that contains non-zero elements.
The lower bandwidth
is the smallest positive integer such that
when
. This finds the last diagonal, below the main diagonal, that contains non-zero elements.
The bandwidth of
can be defined as:
.
Note that the bandwidth may have different definition, e.g. maximum of
and
, in some text.
Examples:
![Rendered by QuickLaTeX.com \[ M=\left(\begin{matrix}1&0&0&0&0\\1&2&3&0&0\\0&3&1&2&0\\0&0&1&2&3\\0&0&0&4&4\end{matrix}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-bd56ed3872f8ceb05819f3efa4db1eb9_l3.png)
is banded with a bandwidth of
.
![Rendered by QuickLaTeX.com \[ M=\left(\begin{matrix}1&1&0&0&0\\1&1&1&0&0\\0&0&1&0&0\\0&0&0&1&1\\0&0&0&1&1\end{matrix}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-ef08c778ee92d7b2b9fb846c788e433f_l3.png)
is banded with a bandwidth of
.
The bandwidth of a diagonal matrix is ![]()
An upper triangular matrix
is not a banded matrix, but the definition of the bandwidth leads to a bandwidth of
for this matrix.
Matrix Operations
Matrix Addition
Let
. Consider the function
and the function
. Then, the function
is given as:
![]()
where
is computed by adding each component in
to the corresponding component in
. For example, consider the two matrices:
![]()
Then, the matrix
can be computed as follows:
![]()
Two matrices can only be added if they have the same dimensions.
Matrix Multiplication
Consider the matrix
which takes vectors of dimensions
and returns vectors of dimensions
. The matrix
has dimensions
. Consider the matrix
which takes vectors of dimension
and gives vectors of dimension
. The matrix
has dimensions
. The combined operation
takes vectors of dimensions
and gives vectors of dimension
. This matrix
corresponding to the combined operation can be obtained by taking the dot product of the row vectors of the matrix
with the column vectors of the matrix
. This can only be done if the inner dimensions are equal to each other. The product
would have dimensions of
and it is such that
:
![]()
For example, consider the two matrices:
![Rendered by QuickLaTeX.com \[ N=\left(\begin{array}{cc}1&4\\2&2\\3&5\end{array}\right) \qquad M=\left(\begin{array}{cccc}2&3&1&5\\0&-1&2&4\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-b7c5b68ba2677c6f17cb0b2926a6c7c6_l3.png)
We can only perform the operation
but the opposite is not defined. The matrix multiplication
results in:
![Rendered by QuickLaTeX.com \[ L=\left(\begin{array}{cccc} 1\times 2+ 4\times 0 & 1\times 3-4\times 1 & 1\times 1+4\times 2 & 1\times 5 + 4\times 4\\ 2\times 2+ 2\times 0 & 2\times 3-2\times 1 & 2\times 1+2\times 2 & 2\times 5 + 2\times 4\\ 3\times 2+ 5\times 0 & 3\times 3-5\times 1 & 3\times 1+5\times 2 & 3\times 5 + 5\times 4 \end{array}\right)=\left(\begin{array}{cccc}2&-1&9&21\\4&4&6&18\\6&4&13&35\end{array}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-d110cf659e42e1260cb9653cc9a73769_l3.png)
You can use “.” in Mathematica to multiply matrices, or you can use “MMULT” in excel to multiply matrices.
View Mathematica CodeNn = {{1, 4}, {2, 2}, {3, 5}};
M = {{2, 3, 1, 5}, {0, -1, 2, 4}};
L = Nn.M;
L // MatrixForm
import numpy as np Nn = [[1, 4], [2, 2], [3, 5]] M = [[2, 3, 1, 5], [0, -1, 2, 4]] np.matmul(Nn,M)
It is important to note that when the dimensions are appropriate, then matrix multiplication is associative and distributive, i.e., given the three matrices
then:
![]()
However, matrix multiplication is not necessarily commutative, i.e.,
. For example, consider the matrices
![]()
Then:
![]()
Matrix Determinant
If
such that:
![]()
then
. If
such that:
![Rendered by QuickLaTeX.com \[ M=\left(\begin{matrix}a_1 &a_2&a_3\\b_1 & b_2&b_3\\c_1&c_2&c_3\end{matrix}\right) \]](https://engcourses-uofa.ca/wp-content/ql-cache/quicklatex.com-df8b63f15d08eb31ef618b1ca650b5e1_l3.png)
then
.
Let
. The matrix determinant can be defined using the recursive relationship:
![]()
where
and is formed by eliminating the 1st row and
column of the matrix
. It can be shown that
the rows of
are linearly dependent. In other words,
if and only if the matrix cannot be inverted. The function “Det[]” in Mathematica can be used to calculate the determinant of a matrix. In Excel, you can use the function “MDETERM()” to calculate the determinant of a matrix as well.
The following function in Mathematica computes the determinant of the matrix using the above recursive relationship and then compares the solution with the built-in function “Det[]” in Mathematica.
View Mathematica Codefdet[M_] := (n = Length[M]; If[n == 2, (M[[1, 1]]*M[[2, 2]] - M[[1, 2]]*M[[2, 1]]),Sum[(-1)^(i + 1) M[[1, i]]*fdet[Drop[M, 1, {i}]], {i, 1, n}]]);
M = {{1, 1, 3, 7}, {4, 4, 5, 0}, {1, 5, 5, 6}, {1, -4, 3, -2}};
fdet[M]
Det[M]
import numpy as np
def fdet(M):
n = len(M)
if n == 2:
return M[0][0]*M[1][1] - M[0][1]*M[1][0]
else:
return sum([(-1)**(i + 1)*M[0][i]*fdet(np.delete(M[1:],i,1)) for i in range(n)])
M = [[1, 1, 3, 7], [4, 4, 5, 0], [1, 5, 5, 6], [1, -4, 3, -2]]
print("fdet(M):",fdet(M))
print("np.linalg.det(M):",np.linalg.det(M))
You can check the section on the determinant of a matrix for additional information on some of the properties of the determinant of a matrix.
Solving Systems of Linear Equations
Solving systems of linear equations has been the subject of study for many centuries. In particular, for every engineering discipline and almost every engineering application, systems of linear equations appear naturally with large numbers of unknowns. For example, the stiffness method in structural analysis as well as the finite element analysis method seek to find the solution of the linear system of equations
where
is a symmetric matrix of dimensions
,
is a vector of external forces and
is the vector of the displacement of the structure nodes. The software used need to implement a method by which
can be obtained given particular values for
. An appropriate method to solve for a linear system of equations has to be computationally efficient. In the following, we will present a few methods to solve systems in which the number of equations is equal to the number of unknowns along with their pros and cons. The methods are categorized into “Direct Methods” and “Iterative Methods”

In the banded matrix, how to understand “The band width is the maximum number of columns on the same row that have non-zero entries.” If I have a matrix [1,1,0,0,0][1,1,0,0,0][0,0,1,0,0][0,0,0,1,1][1,1,0,1,1], the band width is two or three?
The matrix you provided isn’t a good example as the bottom row contains nonzero values at the first and last element.
If the last row was [0,0,0,1,1] then the bandwidth would be 3.
Yes, Dr. Adeeb. That was my mistake. If my matrix is [1,1,0,0,0][1,1,0,0,0][0,0,1,0,0][0,0,0,1,1][0,0,0,1,1], I agree that the bandwidth for this matrix would be 3. But the maximum number of columns on the same row that have non-zero entries is only 2?
Hi Lei,
You are right. We will modify the definition to account for examples like the one you presented.
Samer