Friday, April 09, 2010

Found my neural network bug (three years later)

After the university of Waterloo AI Challenge I went back to a bit of AI programming in Haskell. Three years ago I had written some code to implement a neural network, and it didn't seem to learn as fast as the book said it should. I had not worked on that any more, but I decided to look at it again. And I found a bug!! I mixed up the deltas to pass on from one run to the next. Changing two lines of code makes my network learning much faster.
So now onto something a bit more meaty than exclusive or: very simple character recognition. I want to be able to recognize numbers written as a digital alarm clock would display them (at least mine): you have 7 lights, four vertical and three horizontal, and various patterns make the numbers. If all lights are on, it's 8, if only the middle horizontal light is off we have zero, etc...
I just test things with a network that has 7 input neurons (one for each light), 10 output neurons (one for each number) and 8 hidden neurons (the average between input and output)

g<-getStdGen
n<-network 7 8 10
let trainingSet=[
([1,1,1,-1,1,1,1],[1,0,0,0,0,0,0,0,0,0]), -- 0
([-1,-1,1,-1,-1,1,-1],[0,1,0,0,0,0,0,0,0,0]), -- 1
([1,-1,1,1,1,-1,1],[0,0,1,0,0,0,0,0,0,0]), -- 2
([1,-1,1,1,-1,1,1],[0,0,0,1,0,0,0,0,0,0]), -- 3
([-1,1,1,1,-1,1,-1],[0,0,0,0,1,0,0,0,0,0]), -- 4
([1,1,-1,1,-1,1,1],[0,0,0,0,0,1,0,0,0,0]), -- 5
([1,1,-1,1,1,1,1],[0,0,0,0,0,0,1,0,0,0]), -- 6
([1,1,1,-1,-1,1,-1],[0,0,0,0,0,0,0,1,0,0]), -- 7
([1,1,1,1,1,1,1],[0,0,0,0,0,0,0,0,1,0]), -- 8
([1,1,1,1,-1,1,1],[0,0,0,0,0,0,0,0,0,1]) -- 9
]
let (LNS(n',err,ds),e) = run (initialState n) defaultLearningRate trainingSet (1,1000)

And after something like 150 iterations I get a network that has a low error rate. A little helper method to massage the results:

runNumber n inputs=do
let
(_, output)=exec n inputs
normalized=map (\v->if (1-v)>0.5 then 0 else 1) output
return $ elemIndex 1 normalized

Et voila!

Main> runNumber nn [1,-1,1,1,-1,1,1]
Just 3

Now, things are starting to get interesting! Hopefully I won't wait another three years before trying maybe more complex character recognition.

1 comment:

Anonymous said...
This comment has been removed by a blog administrator.