Some advantages of artificial neural networks
In cognitive science artificial neural networks often come under the banner of 'connectionism' or 'connectionist systems'. This distinguishes them from the symbol processing systems that have traditionally been the foundation of computational work in cog science. It also distinguishes them from ANNs used for purposes other than cognitive science, since ANNs can be applied to all sorts of problems in industry, finance, etc.
Connectionist systems have several advantages over symbolic systems. The ones we'll focus on here are graceful degradation and generalisation to data which was not seen in training. Both of these arise because of the distributed nature of the processing in a neural network. To demonstrate these two phenomena, we'll build a simple little network that is able to distinguish fruits based upon their features.
The six green circles are the six feature sensors. You can think of them as something like photoreceptors in the eye. When a fruit is presented to the system, they become active or not depending upon whether the fruit has the feature that they are specialised to detect. The first sensory layer neuron will detect a value of 1 if the fruit has yellowness, and 0 if it doesn't, the second a value of 1 if the fruit has greenness, and 0 if it doesn't, and so on for the six features listed.
All of the six feature sensors are connected to both of the two neurons in the associative layer. In this case, associative neuron 1 will become more active the more similar the fruit is to a banana, while associative neuron 2 will become more active the more similar the fruit is to an apple.
The table in the bottom right-hand corner shows the weights (weights = synapses = connection strength between two neurons) for the network. The rows (labelled i) are the two neurons and the columns (labelled j) are the six feature sensors - so, the orange square is the link from feature sensor 1 to neuron 1, and the value of the weight is 0.5 (though you can't see that on the diagram to avoid it looking cluttered), and the green square is the weight between feature sensor 4 and neuron 2. Now, let's put this into Excel...
Connectionist systems have several advantages over symbolic systems. The ones we'll focus on here are graceful degradation and generalisation to data which was not seen in training. Both of these arise because of the distributed nature of the processing in a neural network. To demonstrate these two phenomena, we'll build a simple little network that is able to distinguish fruits based upon their features.
The six green circles are the six feature sensors. You can think of them as something like photoreceptors in the eye. When a fruit is presented to the system, they become active or not depending upon whether the fruit has the feature that they are specialised to detect. The first sensory layer neuron will detect a value of 1 if the fruit has yellowness, and 0 if it doesn't, the second a value of 1 if the fruit has greenness, and 0 if it doesn't, and so on for the six features listed.
All of the six feature sensors are connected to both of the two neurons in the associative layer. In this case, associative neuron 1 will become more active the more similar the fruit is to a banana, while associative neuron 2 will become more active the more similar the fruit is to an apple.
The table in the bottom right-hand corner shows the weights (weights = synapses = connection strength between two neurons) for the network. The rows (labelled i) are the two neurons and the columns (labelled j) are the six feature sensors - so, the orange square is the link from feature sensor 1 to neuron 1, and the value of the weight is 0.5 (though you can't see that on the diagram to avoid it looking cluttered), and the green square is the weight between feature sensor 4 and neuron 2. Now, let's put this into Excel...
There's no need to have this network learning, so we can use a a very simple basic set-up.
There are six inputs sensors, so we need to have six weights going into each of the two associative layer neurons. That makes 12 weights in all, but we'll just do one neuron to start with - associative neuron 1, which becomes more active the more similar the fruit is to a banana. So, set up your neuron as such:
The input spots are empty at the moment because the network isn't currently looking at anything. The six weights are set to the values in row one of the table we saw above. The activation of a neuron is the sum of all inputs*weights (input1 times weight1 plus input2 times weight2,... etc), so we need to make these connections in Excel. You can do this by clicking on the association neuron's cell (cell C4 in the picture) and typing in or copy & pasting the sum below.
=(A3*B3+A4*B4+A5*B5+A6*B6+A7*B7+A8*B8)
. After you've done this, when you double click cell C4 it should bring up the following:
=(A3*B3+A4*B4+A5*B5+A6*B6+A7*B7+A8*B8)
. After you've done this, when you double click cell C4 it should bring up the following:
That's neuron 1 set up. Now we do the same for neuron 2, bearing in mind that the weights should be the values in row 2 of the table from the first picture.
When it comes to creating the formula for the activation, remember that the inputs to both neurons are the same, so the formula to enter into cell C11 is:
=(A3*B10+A4*B11+A5*B12+A6*B13+A7*B14+A8*B15)
Double clicking the cell that contains the activation for neuron 2 should lead to the following display:
When it comes to creating the formula for the activation, remember that the inputs to both neurons are the same, so the formula to enter into cell C11 is:
=(A3*B10+A4*B11+A5*B12+A6*B13+A7*B14+A8*B15)
Double clicking the cell that contains the activation for neuron 2 should lead to the following display:
And that is our two neuron network finished!
The next step is to create the inputs. I'm going to create a little table below the network to hold these. When we present a fruit to the network, it will be a case of telling the feature sensors to refer to one of the lines of this table. Mine looks like this:
The next step is to create the inputs. I'm going to create a little table below the network to hold these. When we present a fruit to the network, it will be a case of telling the feature sensors to refer to one of the lines of this table. Mine looks like this:
From this table we can see that line 18 is the banana because it stimulates the feature sensors for yellowness, longness, white flesh and thick peelable skin. The line below, line 19, is the apple because it stimulates the feature sensors for greenness, roundness and white flesh.
When a banana is presented to the network, cell A3 (the first input neuron) will be get a value from cell A18 (the yellowness feature), cell A4 will get a value from cell B18, and so on.
When a banana is presented to the network, cell A3 (the first input neuron) will be get a value from cell A18 (the yellowness feature), cell A4 will get a value from cell B18, and so on.
So, just to sum up: when a banana is presented to the network, the values in row 18 are put into the cells that represent the input neurons (A3 to A8), and we expect the upper of the two association neurons to be more active than the lower one. Conversely, when the green apple is presented to the network, the values in row 19 are put into the very same input neuron cells (A3 to A8) and we expect the lower of the two association neurons to be the more active.
I don't want to have to change all six inputs every time I present a different fruit to the system, so I'm going to automate the presentation process.
I don't want to have to change all six inputs every time I present a different fruit to the system, so I'm going to automate the presentation process.
- Click on cell E14, and enter a number 1. This will refer to line 1 of the input table we've just created (row 18 in Excel, meaning a banana). When we've finished the following few steps, we'll be able to change all the inputs from banana to apple just by changing this to a 2.
- Click on the first of our feature sensors - cell A3.
- Go to the Formulas tabbed menu and click the Insert Function button on the left.
- In the floating window that pops up, select the CHOOSE function (if it's not there, you may need to select 'All' in the drop down category menu).
- In the new floating window that appears for the CHOOSE function, type E14 in the Index_num field. This means that the feature sensor checks cell E14 to find out which row of the inputs table to look at.
- In the box for value1, type A18. This is the yellowness feature of the banana. In the box for value2, type A19. This is the yellowness feature of the apple. Press OK.
- Open a new CHOOSE box for our second feature sensor - cell A4. This should again reference cell E14, but this time the two values should be B18 and B19, the greenness features of the banana and apple.
- Repeat this process for the other four feature sensors, cells A5, A6, A7 and A8.
That's the network finished. If you present different fruits to the network, it should tell you whether it's an apple or banana by outputting a greater value from either the banana cell (neuron 1 - cell C4) or the apple cell (neuron 2 - cell C11). Just to make it pretty, I like to add a final little bit of code. Click on cell D7 and input the following:
=IF(C4>C11,"banana","apple")
You'll see what this does if you play with the system!
Just in case you have got stuck anywhere along the way, here's a video that shows what's in each of the cells:
Playing with the network...
Ok, that's the network. Now we can go ahead and demonstrate generalisation and graceful degradation.
Generalisation first...
We've seen that the network recognises normal bananas and apples with no problems, but what about an abnormal one. Bananas are green before they become ripe. Let's add an input for a green banana. Modify your input table as such:
Generalisation first...
We've seen that the network recognises normal bananas and apples with no problems, but what about an abnormal one. Bananas are green before they become ripe. Let's add an input for a green banana. Modify your input table as such:
You'll also need to update the CHOOSE functions in the six feature sensor cells (A3:A8). Do this by simply adding a comma and the appropriate feature of the green banana to the function. So the CHOOSE function for feature sensor 1 (cell A3) will look like this:
=CHOOSE(E14,A18,A19,A20)
where the final input, A20, is the part I've just added. Do this for all six feature sensors.
Now test the network by changing the value in cell E14 to a 3. What is the output; apple or banana? Check all three inputs again. Is the difference in output values the same between an banana and an apple as between a green banana and an apple? Why or why not?
What features would these things have: a golden delicious apple, a peeled banana, apple sauce?
Design your own inputs for the above, add them to the Excel program and test the outputs. Does the network perform as expected on all of them? Pay attention to the difference between the output values in all cases.
My apple sauce was zero for all of the features except for white flesh and the output was 0.5 from both neurons. This would mean that the network has no idea whether it's an apple or a banana. If you'd never experienced apple sauce before and were unable to smell or taste it, do you think you would be able to tell the difference?
From the above tests, you should gather that this simple little network is able to generalise from the things that it's had experience of before to things it's never experienced by using the knowledge it has about normal apples and bananas.
Next onto graceful degradation...
If a system exhibits graceful degradation, it doesn't completely break suddenly because a little bit of it is damaged or because the input to the system is noisy (not perfect). Instead performance decreases a little bit for every little bit of damage or extra noise.
Firstly, we'll simulate a noisy input. In the picture below you can see that I've changed the value of the 'white flesh' feature from a 1 to a 0 (in the orange cell). You could think of this as being caused by a speck on the camera lens, or a cataract on the animal's eye.
=CHOOSE(E14,A18,A19,A20)
where the final input, A20, is the part I've just added. Do this for all six feature sensors.
Now test the network by changing the value in cell E14 to a 3. What is the output; apple or banana? Check all three inputs again. Is the difference in output values the same between an banana and an apple as between a green banana and an apple? Why or why not?
What features would these things have: a golden delicious apple, a peeled banana, apple sauce?
Design your own inputs for the above, add them to the Excel program and test the outputs. Does the network perform as expected on all of them? Pay attention to the difference between the output values in all cases.
My apple sauce was zero for all of the features except for white flesh and the output was 0.5 from both neurons. This would mean that the network has no idea whether it's an apple or a banana. If you'd never experienced apple sauce before and were unable to smell or taste it, do you think you would be able to tell the difference?
From the above tests, you should gather that this simple little network is able to generalise from the things that it's had experience of before to things it's never experienced by using the knowledge it has about normal apples and bananas.
Next onto graceful degradation...
If a system exhibits graceful degradation, it doesn't completely break suddenly because a little bit of it is damaged or because the input to the system is noisy (not perfect). Instead performance decreases a little bit for every little bit of damage or extra noise.
Firstly, we'll simulate a noisy input. In the picture below you can see that I've changed the value of the 'white flesh' feature from a 1 to a 0 (in the orange cell). You could think of this as being caused by a speck on the camera lens, or a cataract on the animal's eye.
Try playing with noisy inputs by changing the values of different features of the banana and apple. You can also try changing the values to values other than one or zero - how about 2? 5? -1? -3? How many features can be noisy before the system becomes unable to distinguish apples and bananas?
Now onto what happens when we damage the system a little bit. In this case, damage refers to tampering with the weights since this is where the information about how to perform is stored. In the picture below you can see that I've changed the weight from the first feature sensor to neuron 1 from a 0.5 to a 0.
Now onto what happens when we damage the system a little bit. In this case, damage refers to tampering with the weights since this is where the information about how to perform is stored. In the picture below you can see that I've changed the weight from the first feature sensor to neuron 1 from a 0.5 to a 0.
How does the network perform if you do this? How much can you change the weight value before the network is unable to perform correctly? How many weights can you change just a little before the network can't perform correctly?
You should find that performance decreases proportionately to the damage you incur on the weights. This is strongly equivalent with the way animals brains are capable of incurring damage without breaking completely: Parkinson's, Alzheimer's, lesions, etc.
You should find that performance decreases proportionately to the damage you incur on the weights. This is strongly equivalent with the way animals brains are capable of incurring damage without breaking completely: Parkinson's, Alzheimer's, lesions, etc.