Monthly Archives: August 2011

Bellman-Ford Algorithm

I just finished a script which executes the Bellman-Ford Algorithm on a randomly generated directed graph (possibly with negative arcs). It can be accessed here.

The Bellman Ford algorithm is another shortest path algorithm. Unlike Dijkstra’s algorithm, though, this algorithm can handle edges with negative weights and detect negative cycles in a graph. Negative cycles are important because in a graph with negative cycles, there is no shortest path to some nodes since a negative cycle can be iterated an infinite number of times.

The algorithm works by initially setting the distance from the source node to all other nodes to be infinity. The distance to the source is then updated to be 0 since we are already there.
Next we proceed through each edge (u, v) and check the condition that if the distance to u plus the weight of the edge (u, v) is less than the distance to the node v, then we have just found a shorter route to v.

This condition on the edges only needs to be checked at most |V| times for each edge since any path will contain at most |V| edges, which means that the distance of a node can be updated at most |V| times.

If after |V| times there is still a node whose weight can be updated, it signifies the existence of a negative weight cycle and the Bellman-Ford will show that this cycle exists, but will be unable to find shortest paths in this problem as shortest paths no longer exist.

 

Dijkstra’s Algorithm

I’ve just written a script that executes Dijkstra’s algorithm that seeks to find the shortest path tree on a randomly generated graph.

Given a weighted graph G and a vertex s, the single node shortest path problem seeks to find the shortest path from s to every other vertex in this graph such that the sum of the weights of the edges along these paths is minimized. One famous algorithm for this problem is Dijkstra’s Algorithm, which constructs this shortest path tree using techniques of greedy algorithms and dynamic programming.

Dijkstra’s Algorithm works as follows.

  • Initially we have an empty path tree T, and assume that the distance to every vertex in the graph has some maximum cost, say infinity, i.e. w(v) = infinity for all v in V.
  • Add the node s to the tree, and give the associated path cost a value of zero, i.e. w(s) = 0.
  • Find the edge which is adjacent to T that adds the vertex whose cost is minimum, i.e. we look for an e = (u, v) such that u is in T, and v is not in T and w(u) + cost(u, v) < w(v) is minimum. If no such edge exists go to 5.
  • Add the corresponding edge and vertex to the tree, and update the weight vector of the vertex v. Go to 3.
  • The path tree T now represents the shortest path from the vertex s to all other vertices reachable in the graph G. The weight vector w represents the costs of these paths.

For an example of Dijkstra’s algorithm executed on the Graph with the following adjacency matrix:

0 1 2 3 4 5 6 7
0 - - 2 4 - - 20 -
1 - - - 11 11 7 5 -
2 2 - - - 10 - 7 -
3 4 11 - - - - - 2
4 - 11 10 - - - - -
5 - 7 - - - - - 14
6 20 5 7 - - - - -
7 - - - 2 - 14 - -

Graph with 8 Nodes

Suppose we are interested in discovering the shortest paths from the node 0.

Initially Dijkstra’s algorithm constructs an empty path tree, T = {}.

Because we want to discover the shortest paths from the node 0, our first step will be to add this node to the tree and update the weight vector.
T = {0}
w(0) = 0.

Now we will consider the edges adjacent to T, {(0, 2), (0, 3), and (0, 6)}. Our assumption is that the shortest path of every vertex in T has already been computed correctly and we will seek the edge that minimizes the value w(u) + c(u, v), where u is a member of T and (u, v) is an edge adjacent to T. The edge that does that currently is (0, 2) since w(0) = 0 and c(0, 2) = 2. We add the node 2 to T and update the weight vector.
T = {0, 2}
w(2) = 2.

The edges adjacent to T are now {(0, 3), (0, 6), (2, 4), (2, 6)}.
The associated path costs are:
w(0) + c(0, 3) = 0 + 4 = 4
w(0) + c(0, 6) = 0 + 20 = 20
w(2) + c(2, 4) = 2 + 10 = 12
w(2) + c(2, 6) = 2 + 7 = 9
We can see that the edge that minimizes the value w(u) + c(u, v), where u is a member of T and (u, v) is an edge adjacent to T is (0, 3). We add the node 3 to T and update the weight vector.
T = {0, 2, 3}
w(3) = 4

The edges adjacent to T are now {(0, 6), (2, 4), (2, 6), (3, 1), (3, 7)}.
The associated path costs are:
w(0) + c(0, 6) = 0 + 20 = 20
w(2) + c(2, 4) = 2 + 10 = 12
w(2) + c(2, 6) = 2 + 7 = 9
w(3) + c(3, 1) = 4 + 11 = 15
w(3) + c(3, 7) = 4 + 2 = 6
We can see that the edge that minimizes the value w(u) + c(u, v), where u is a member of T and (u, v) is an edge adjacent to T is (3, 7). We add the node 7 to T and update the weight vector.
T = {0, 2, 3, 7}
w(7) = 6

The edges adjacent to T are now {(0, 6), (2, 4), (2, 6), (3, 1), (5, 7)}.
The associated path costs are:
w(0) + c(0, 6) = 0 + 20 = 20
w(2) + c(2, 4) = 2 + 10 = 12
w(2) + c(2, 6) = 2 + 7 = 9
w(3) + c(3, 1) = 4 + 11 = 15
w(7) + c(5, 7) = 6 + 14 = 20
We can see that the edge that minimizes the value w(u) + c(u, v), where u is a member of T and (u, v) is an edge adjacent to T is (2, 6). We add the node 6 to T and update the weight vector.
T = {0, 2, 3, 6, 7}
w(6) = 9

The edges adjacent to T are now {(2, 4), (3, 1), (5, 7) (6, 1)}.
The associated path costs are:
w(2) + c(2, 4) = 2 + 10 = 12
w(3) + c(3, 1) = 4 + 11 = 15
w(7) + c(5, 7) = 6 + 14 = 20
w(6) + c(6, 1) = 9 + 5 = 14
We can see that the edge that minimizes the value w(u) + c(u, v), where u is a member of T and (u, v) is an edge adjacent to T is (2, 4). We add the node 4 to T and update the weight vector.
T = {0, 2, 3, 4, 6, 7}
w(4) = 12

The edges adjacent to T are now {(3, 1), (5, 7), (6, 1), (4, 1)}.
The associated path costs are:
w(3) + c(3, 1) = 4 + 11 = 15
w(7) + c(5, 7) = 6 + 14 = 20
w(6) + c(6, 1) = 9 + 5 = 14
w(4) + c(4, 1) = 12 + 11 = 23
We can see that the edge that minimizes the value w(u) + c(u, v), where u is a member of T and (u, v) is an edge adjacent to T is (6, 1). We add the node 1 to T and update the weight vector.
T = {0, 1, 2, 3, 4, 6, 7}
w(1) = 14

The edges adjacent to T are now {(5, 7), (1, 5)}.
The associated path costs are:
w(7) + c(5, 7) = 6 + 14 = 20
w(1) + c(1, 5) = 14 + 7 = 21
We can see that the edge that minimizes the value w(u) + c(u, v), where u is a member of T and (u, v) is an edge adjacent to T is (5, 7). We add the node 5 to T and update the weight vector.
T = {0, 1, 2, 3, 4, 5, 6, 7}
w(5) = 20

Now that T contains all the nodes from the tree, we know the shortest path from node 0 to all other nodes and and have solved the problem. The associated weight vector is w = [0, 14, 2, 4, 12, 20, 9, 6].

To learn more and see more examples, view Dijkstra’s Algorithm at LEARNINGlover.com

Kruskal’s Algorithm

I’ve just written a script that executes Kruskal’s algorithm on a randomly generated graph.

Given a weighted graph, many times we are interested in finding a minimum spanning tree (MST) for that graph. These have several applications in areas like transportation and the network simplex method. We already discussed Prim’s algorithm. Another method for generating minimum spanning trees is Kruskal’s algorithm. A spanning tree is a subset of the edges of a graph that connects to every vertex, but contains no cycles. This spanning tree is called a minimum spanning tree if in addition the sum of the weights of the edges included in this tree is less than or equal to the sum of the weights of the edges of any other spanning tree for this graph.

Kruskal’s algorithm works by the following procedure.
1. Initially each vertex is a stand-alone tree, so for each v in V, we define the tree Treev = {v}. The set of selected edges E* is initially empty.
2. Find the edge e = (uv) of minimum weight such that u and v belong to different trees. If no such edge exists, go to 6.
3. Merge the trees Tlookup(u) and Tlookup(v).
4. Add the edge e to E* and remove the edge e from the graph.
5. If the size of E* is less than n – 1, go to step 2. Else go to step 7.
6. If you reached this step. then the graph is not connected.
7. If you reached this step, then E* is a minimum spanning tree.

For example, consider the graph represented by the following adjacency matrix:

0 1 2 3 4
0 - - - 13 12
1 - - - - 16
2 - - - - 24
3 13 - - - 13
4 12 16 24 13 -

Graph with 5 nodes

Initially we have 5 distinct trees and E* = {}/
T0 = {0}
T1 = {1}
T2 = {2}
T3 = {3}
T4 = {4}.

The first step of Prim’s algorithm says to find the cheapest edge such that its two endpoints belong to different trees. This will be the edge (0, 4) with a cost of 12. So E* = {(0, 4)}. We then merge the two trees so that our trees are now:
T0 = {0, 4}
T1 = {1}
T2 = {2}
T3 = {3}

Again, we look for the cheapest edge such that the endpoints of the two edges are in different trees. There are two edges with a cost of 13 (either (0, 3) or (3, 4)) so we will arbitrarily choose (0, 3) and add it to our tree. So E* = {(0, 4), (0, 3)}. We again merge the associated trees and it results in the following trees:
T0 = {0, 3, 4}
T1 = {1}
T2 = {2}

The cheapest edge that has endpoints in distinct trees will be the edge (1, 4) with a cost of 16. We add this edge to our tree. So E* = {(0, 4), (0, 3), (1, 4)}. Once we merge the associated trees we have the following:
T0 = {0, 1, 3, 4}
T2 = {2}

The cheapest remaining edge that has endpoints in distinct trees will be the edge (2, 4) with a cost of 16. This makes E* = {(0, 4), (0, 3), (1, 4), (2, 4)}. We merge the associated trees and arrive at:
T0 = {0, 1, 2, 3, 4}

Because T0 contains all the nodes in the graph it is a spanning tree. Its total cost is 12 + 13 + 16 + 24 = 65.

To learn more and see more examples, view Kruskal’s Algorithm at LEARNINGlover.com

Prim’s Algorithm

I have just written a script that executes Prim’s Algorithm that finds the minimum spanning tree on a randomly generated graph.

Given a weighted graph, many times we are interested in finding a minimum spanning tree (MST) for that graph. This has many applications including the very important network simplex method. Prim’s algorithm is a greedy method which does finds this MST. A spanning tree is a subset of the edges of a graph that connects every vertex, but contains no cycles. This spanning tree is called a minimum spanning tree if in addition the sum of the weights of the edges included in this tree is less than or equal to the sum of the weights of the edges of any other spanning tree for this graph.

Prim’s algorithm works by the following procedure.
1. Let Treev be the set of vertices included in the tree, and TreeE be the set of edges included in the tree. Initially Treev and TreeE are empty.
2. Add an arbitrary vertex to Treev (TreeE is still empty).
3. Find the edge e of minimum weight such that one vertex is in Treev and vertex is not in Treev. Add the associated vertex to Treev, and add e to TreeE.
4. If edge was found in step 3, goto 5, else go to 6.
5. If the number of vertices in Treev is less than the number of vertices in the original graph, then the graph is not connected and thus does not contain a minimum spanning tree. Goto 8.
6 If the number of vertices in Treev is less than the number of vertices in the original graph, go to 2, else go to 7.
7. Output “The Minimum Spanning Tree is “, TreeE.
8. Output “This graph does not have a minimum spanning tree because it is not connected. ”

For example, consider the graph represented by the following adjacency matrix:

0 1 2 3 4
0 - - - 13 12
1 - - - - 16
2 - - - - 24
3 13 - - - 13
4 12 16 24 13 -

Graph with 5 nodes

Initially our tree (Tv is empty). The first step says to choose a random vertex and add it to the tree, so lets choose vertex 2.

Iteration 1: Now our tree contains the vertex 2 (i.e. Tv = {2}) and likewise TE contains the edges coming from Tv. Thus TE = {(2, 4)}.
We want to choose the cheapest edge that has one endpoint in Tv and one endpoint not in Tv. These edges are represented by TE. Notice that TE only contains one edge, so we select this
edge, which has a cost of 24.

Iteration 2: Our tree thus contains the vertices 2 and 4 (i.e Tv = {2, 4}) and likewise TE contains the edges coming from Tv. Thus TE = {(0, 4), (1, 4), (3, 4)}.
Again, we want to choose the cheapest edge that has one endpoint in Tv and one endpoint not in Tv. This will be the edge (0, 4) which has a cost of 12.

Iteration 3: Now Tv = {0, 2, 4} and TE = {(1, 4), (3, 4), (0, 3)}. The cheapest of these three edges is the edge (0, 3) with a cost of 13, which means we will add it to our tree.

Iteration 4: Now Tv = {0, 2, 3, 4} and TE = {(1, 4)}. Since (1, 4) is the only edge connected to our tree we add it and it has a cost of 16.

Iteration 5: Now Tv = {0, 1, 2, 3, 4} and TE = {}. Because our tree contains all the vertices of the graph it is now spanning tree. The cost of this spanning tree is 24 + 12 + 13 + 16 = 65.

To learn more and see more examples, view Prim’s Algorithm at LEARNINGlover.com

Gaussian Elimination

I have just written a script which executes the Gaussian Elimination Algorithm.

When we have a collection of lines we wish to know if they all intersect at some point. Many times we are interested in determining what that point is. In order to calculate this information, we first need an understanding of the lines themselves. The way the Gaussian Elimination Algorithm works is that the collection of lines are input using a notation of Ax = b, where the matrix A is called the coefficient matrix, as the nth row of it corresponds to the coefficients for the nth line being considered. The vector b represents the right hand side vector (in two dimensions, we would call these constants the y-intercepts of the lines. In higher dimensions they hold a similar property). The vector x represents the point where the lines intersect. It is this quantity which Gaussian Elimination seeks to determine.

The basic procedure of Gaussian Elimination is to use \”elementary row operations\” on the matrix (A|b), which is called the augmented matrix, to transform A into upper triangular form. Once this is done, a procedure called back-substitution can find the solution (x) to this problem.

The elementary row operations that we are allowed to perform are:

  • Interchange two rows.
  • Multiply a row by a nonzero number.
  • Add a row to another one multiplied by a number.
  • For the last property listed above, we will determine this number by dividing the coefficient of the term we which to eliminate by the negative of the coefficient of the element on the main diagonal of the same column of the matrix. This will have the property of cancelling out, or producing a desired zero in the resulting row.

    If this algorithm produces an upper triangular matrix from which we can solve for x using back-substitution. This procedure of back-substitution is simply solving for the vector x from the bottom of the matrix to the top. If the algorithm does not produce an upper triangular matrix (because somewhere along the line, we are unable to obtain a ratio because we have zero’s on the diagonal and all zeros below the diagonal), then we say the matrix is singular. This means that there is no unique point where the lines all intersect.

    The Euclidean Algorithm

    I remember being in school and learning about fractions. In particular, I remember the problems we had when trying to add and subtract fractions. This problem also presented itself when we tried to multiply fractions, although we still received partial credit if we couldn’t reduce fractions to their lowest terms.

    What I’m referring to is the problem of finding the Greatest Common Divisor (GCD) of two numbers. The GCD of two numbers is the largest number that divides into both numbers with a remainder of zero each time. This problem has many applications, but for most of us we can relate to it because of our frustrations with fractions.

    The algorithm that I’m writing about and have written a script for is called the Euclidean Algorithm, which solves precisely this problem. To be more precise, the Euclidean Algorithm finds the greatest common divisor between two integer numbers.

    There are different versions of the algorithm, but the one I have implemented finds the GCD by subtracting the smaller number from the larger number, and if the result is greater than 0, the procedure repeats itself with the two lower numbers. Otherwise, the result (the GCD) is the final number that was greater than 0.

    Lets see an example:
    Consider the number 9 and 21.
    21 – 9 = 12. 12 is greater than 0, so we repeat the procedure with 9 and 12.
    12 – 9 = 3. 3 is greater than 0, so we repeat the procedure with 3 and 9.
    9 – 3 = -6. -6 is less than 0, so we have finished the procedure.
    Since 3 was the last positive number that we arrived at in this procedure, we see that the GCD of 9 and 12 is 3.

    To learn more and see more examples, check out My Script on The Euclidean Algorithm.

    Sieve of Eratosthenes

    Prime numbers are an important concept in Number Theory and Cryptography which often uses the difficulty of finding prime numbers as a basis for building encryption systems that are difficult to break without going through all (or a very large number of) possible choices.

    Remember that a prime number is a number greater than 1 whose only divisors are 1 and that number itself. One of the most famous algorithms for searching for prime numbers is the Sieve of Eratosthenes. I added a script which implements the Sieve of Eratosthenes to my Examples page.

    This algorithm prints out all prime numbers less than a given number by first canceling out all multiples of 2 (the smallest prime), then all multiples of 3 (the second smallest prime), then all multiples of 5 (the third smallest prime – multiples of 4 do not need to be considered because they are also multiples of 2), etc until we have reached a number which cannot be a divisor of this maximum number.

    So if we are given a number, n, the first step of the algorithm is write out a table that lists all the numbers that are less than n. For example lets run this Sieve on 50. So all numbers less than 50 are

    1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50.

    So since 1 is not a prime number (by the definition of prime numbers), we cancel that number out.

    1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50.

    Next, we look at the list and the first number that is not crossed out is a prime. That number is 2. We will put a mark by 2 and cancel out all of 2′s multiples.

    1, 2*, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50.

    Again, we look at the list and the first number that is not marked or crossed out is 3, so that number is prime. We will put a mark by 3 and cancel out all of 3′s multiples.

    1, 2*, 3*, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50.

    Once again, we look at the list and the first number that is not marked or crossed out is 5, so that number is prime. We will put a mark by 5 and cancel out all of 5′s multiples.

    1, 2*, 3*, 4, 5*, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50.

    We look at the list and the first number that is not marked or crossed out is 7, so that number is prime. We will put a mark by 7 and cancel out all of 7′s multiples.

    1, 2*, 3*, 4, 5*, 6, 7*, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50.

    Now, we look and the first number that is not crossed out is 11. However, since 11 is greater than sqrt(50) we know that each of 11′s multiples that are less than 50 will have been cancelled out by a previous prime number. So we have finished the algorithm.

    Check out my script which implements the Sieve of Eratosthenes for more examples.

    The Cost of Learning

    I often hear phrases like “the value of a college education” or “the importance of a college education”. These are phrases that I grew up on. These and many similar others have been a part of my life for as long as I can remember. What was not ingrained in my head was the cost of this education. Unfortunately, the concept of students or recent graduates struggling with debt due to student loans is not uncommon in this day and age. Generally, there are things like scholarships and fellowships available to a select few who prove themselves the cream of the crop in a matter of speaking. For the rest, though, the concept of education seems heavily related to the size of one’s pocket books.

    With this in mind, I created this site to use as a resource to those who are interested in learning. I truly believe in the power of education. This belief originates from a concept that education equals empowerment. There are several examples of people throughout history using education as an asset that is only available to a select few. To those whom education was not available, neither was power.

    This site is meant to empower those who wish to learn. This site is not meant to stand in place of any course covering any topic listed here. Instead, it is meant to grant exposure to those who do not have a chance to learn the fundamentals of these courses in an academic setting. This site is meant to empower workers looking to increase their skills when applying for a new job. This site is meant to serve as a refresher to those who may have learned these concepts long ago, or those students who do not understand their professor’s lectures.

    I will go through a variety of topics, updating and adding to them regularly. I will cover these topics by introducing different software that I have written.

    Examples Page

    Sometimes, the most effective way to understand a new concept is to actually see it in action. On My Examples Page, I have implemented a variety of scripts to help teach many different concepts.This is the page that is the easiest for me to update, so you will regularly see changes to this page along with an accompanying blog entry at My Blog Page.

    This page is focused on teaching individual concepts and/or algorithms. In particular, with the HTML5 Canvas element, I’ve been able to visualize many of these concepts. Generally I try to provide a script that executes the given algorithm (or concept) and allows for users to view these concepts on random instances. When possible, I provide a button labeled “New Problem” (or something similar) which will allow the user to view a different instance of the algorithm.

    Flash Cards Page

    One of the most effective ways I remember studying for vocabulary quizzes in high school was through flash cards. During my time in college, I wondered why a similar method was not used to help study for other things, particularly in math and math-like classes where new definitions and concepts are introduced everyday. As a result of this, I wrote an initial flash cards script to help me with my advanced calculus class. Because of my success in this class, I continued developing flash cards to help study many other concepts. Here,  I have decided to share some of those concepts with you.

    The general concept behind how I develop the flash cards is simple. Each flash card consists of a question on one side and the answer to that question on the other. For definitions, the question is generally of the form “What is the definition of _____” and the answer will generally go “The definition of _____ is _____”. This may vary a bit, but I try to keep it in this context to allow for searching easier. The other things I generally put onto flash cards are Lemmas/Theorems/Corollaries. These are a bit more difficult to put on flash cards though. The flash cards are still in a question and answer format, but I tried to make the question center around what the question being asked that led to the development of the Lemma/Theorem/Corollary was. Sometimes I may be off-base with these questions, and I’ve had to go back and re-do these questions a number of times. However, I hope they can be helpful.

    For algorithms, which are a large part of this site, I generally asked to include the pseudocode for the given algorithm. However since pseudocode is far from formal, this can lead to a bit of a debate among whether you feel that your answer is the same as mine.

    The AMS Graduate Student Blog lists some other sites where you can get math generated flash cards.