why do neurons make networks
On the logical operations page, I showed how single neurons can perform simple logical operations, but that they are unable to perform some more difficult ones like the XOR operation (shown above). and I described how an XOR network can be made, but didn't go into much detail about why the XOR requires an extra layer for its solution. This page is about using the knowledge we have from the formalising & visualising page to help us understand why neurons need to make networks. The only network we will look at is the XOR, but at the end you will play with a network that visualises the XOR problem as a pair of lines through input space that you can adjust by changing the parameters of the neurons.
In Formalising & visualising, we saw that there is no way that a single neuron can solve this problem because it is only able to draw one single straight line through input space in order to divide the input space into two categories.
The minimum number of lines that we need to draw through the input space to solve this problem is two.
On the right there is an example of how this can be done. --> --> --> In this case, the area between the lines equates to desired outputs of 1, while both the area above the blue line and the area below the orange line equate to desired outputs of 0. Easy question: if one neuron defines one line through input space, what do we need in order to draw two lines? The answer is, of course, to have two neurons working in parallel (next to each other rather than in different layers). ---> Notice that both neurons are connected to THE SAME input sensors. This is because they are both drawing lines on the same input space in order to makes decisions about the same inputs. They need to work together to decide if the single thing that they are looking at is a member of category 1 or category 0. The other thing to notice is that we now have two output commands rather than one. This actually poses a much bigger problem than it first seems. We want the neural network to categorise the objects it sees into just two groups, but having two output commands that can each be either 1 or 0 gives us four possible combinations: [00, 01, 10, 11] |
You might notice that the result of drawing two lines through input space is actually three distinct categories rather than the two that we wanted. This is a problem, but we won't worry about it until later.
|
If you're on the ball, you might notice that these four options can be arranged to make a logic table, just like the one at the top of the page. So we have two neurons, each performing a logical function described by a logic table, and then the two neurons feed their results forward into a third neuron that again performs a logical function described by a logic table. It has a beautiful simplicity to it!
On the left I've added the output neuron.
To summarise:
Let's see graphically what the space defined by the two first layer neurons looks like, and how the second layer neuron divides it... |
On the right is the same graph that we saw at the top of the page. Be sure you understand what is happening here perfectly (see my formalising & visualising page if you don't know the relevance of lines to neurons). |
The way we do this is to have one neuron calculate the bottom line, as shown in the graph on the left and the logic table below.... Notice that the only combination of inputs that isn't allowed by this neuron is (0,0), just as shown in the graph.
... and the second neuron determine the upper line, as shown in the graph on the left and the logic table below... |
The single neuron in the output (second) layer uses the outputs of the two neurons in the previous layer as its input. In logic table terms, this means the third columns of the two first layer neurons become the first two columns of the second layer neuron.
Below I have shown how the logic table for the single neuron in the second layer is formed from the third columns (outputs) of the two neurons in the first layer.
Below I have shown how the logic table for the single neuron in the second layer is formed from the third columns (outputs) of the two neurons in the first layer.
Then we use the logic table that we've just made for the single second layer neuron to draw its graph, with input 1 (from first layer neuron 1's output) as the horizontal axis, and input 2 (from first layer neuron 2's output) as the vertical axis. The red squares (output = 0) and blue circle (output = 1) taken from the third column. This is shown on the right.
You can see that although there are four rows in the logic table, there are only three points on the graph. This is because the two points that require an output of 1 are situated in the same place (1,0). You can also see that it's possible to draw a single line that separates the two kinds of output, as I have done in green, and as the single output neuron in the second layer does. |
something to play with...Here's an Excel file I made to demonstrate how the weights control the orientation of the line, and how the network will behave properly as long as the lines defined by the neurons in the first layer correctly divide up the input space and the line defined by the neuron in the second layer correctly divides up the space defined by the outputs of the first layer neurons.
The network is initially set up as shown on the right, which is the same as the one we created on a previous page. |
|
Below is a picture of what it looks like when it's open. The first layer neurons are coloured in blue and orange and both receive inputs from the yellow cells; B1 and C1. The second layer neuron is coloured green and uses the outputs from the first layer neurons (cells C5 and F5) as its inputs.
The lines on the graphs are coloured to match, with the top graph being the input space and the bottom graph being the space defined by the outputs of the first layer neurons.
The logic table is shown in yellow at the bottom as a single table rather than the three that are shown above.
The lines on the graphs are coloured to match, with the top graph being the input space and the bottom graph being the space defined by the outputs of the first layer neurons.
The logic table is shown in yellow at the bottom as a single table rather than the three that are shown above.
To play with the file, just change the weights around and see how it affects the lines and whether it gives rise to an error (red cell). You can also change the threshold if you like, as this also affects the line (see formalising & visualising).
One effect of having the second layer neuron dividing the space which is defined by the outputs of the first layer neurons is that if the first layer neurons are not behaving correctly, the second layer neuron cannot behave correctly. So when making similar networks that learn (this one doesn't), it's important that the first layer learns (settles) faster.
One effect of having the second layer neuron dividing the space which is defined by the outputs of the first layer neurons is that if the first layer neurons are not behaving correctly, the second layer neuron cannot behave correctly. So when making similar networks that learn (this one doesn't), it's important that the first layer learns (settles) faster.
conclusion
A single neuron has just one axon to send outputs with, and the output it sends are the all or nothing spikes of action potentials - they are either active or not. In other words, there are only two possible outputs for any single neuron: on or off (1 or 0, yes or no, true or false, firing or quiet). Because there are only two possibilities, a single neuron can only categorise its inputs into two groups. This is irrespective of how many inputs there are into the neuron (inputs give you more information to help make the decision, but don't add different possibilities for what the decision will be). Moreover, the neuron's method of making this binary categorisation is to draw a straight line through its input space.
Since a neuron can only draw a single straight line through its input space, there are some categorisations that are impossible for a single neuron to perform; an example being the XOR logical operation (and there are many more).
It doesn't help to add more neurons in the same layer because we still need a way of combining all of their outputs to get a single conclusion. The only solution is, therefore, to add another layer of neurons.
Adding extra layers of neurons allows the network to break down the problem into sub-problems that can be solved, and then combine the results later (via another neuron)..
Since a neuron can only draw a single straight line through its input space, there are some categorisations that are impossible for a single neuron to perform; an example being the XOR logical operation (and there are many more).
It doesn't help to add more neurons in the same layer because we still need a way of combining all of their outputs to get a single conclusion. The only solution is, therefore, to add another layer of neurons.
Adding extra layers of neurons allows the network to break down the problem into sub-problems that can be solved, and then combine the results later (via another neuron)..